Introduction
Learning a policy for long-horizon planning requires choosing how to collect the experience used to improve it [41 ] . In inventory control, an order changes the pipeline of future deliveries, and its consequences depend on subsequent demand [4 ] . Decisions therefore shape not only future costs but also the states encountered during learning. This makes the behavior policy that collects data an important intervention point: if the behavior policy is itself learned, what should it be optimized to do for the learning process as a whole? Concretely, can heuristic guidance applied to it and behavior policy persistently influence the target policy, which is the policy finally want to optimize, to achieve a better performance?
Existing designs give reason to doubt that this mechanism happens in most RL frameworks. Importantly, failures are observed in long-horizon planning tasks. In a studied multi-echelon inventory network, PPO converged consistently only after inventory transitions were constrained during training [10 ] , and our comparisons show high-cost runs for PPO, Discrete SAC, and adapted GPS (§ 5 ). These methods share a default: experience is collected by the task policy or a recent snapshot of it [34 , 15 ] . Off-policy learning permits a distinct behavior policy [7 ] , but leaves open how that policy should improve and how its deviations should inform target updates. Behavior–target mismatch can impair approximate off-policy updates, and fitting a policy to a teacher’s actions need not preserve the teacher’s long-horizon performance. Our focus is therefore to construct explicit coupling updating rules for the transmission path itself: which behavioral deviations should a separately updated target reinforce or counteract, and when should planning-derived guidance enter its updates?
We introduce Heuristic Behavior Policy Optimization (HBPO) , an off-policy coupled framework that routes heuristic guidance through a persistent behavior policy to a separately and selectively optimized target policy. HBPO forms an explicit delayed heuristic–behavior–target transmission: heuristic guidance, constructed from the current generation target policy, first shapes the successor behavior policy then reaches two-next target updates through four combined optimizations. This transmission is based on data collected by behavior, residual-based advantage estimation supporting both policies and emphatic weighting carrying heuristic confidence forward. Under same interaction budgets on DSIS and DSIS-RB, HBPO recovers many high-cost initial policies, lowers mean costs across most initialization sources, and improves most non-collapse comparison groups. Cold starts achieve competitive costs without cost collapse across all tested seeds. We identify the transmission of delayed heuristic–behavior–target coupling as a key driver of recovery and improvement. An anonymized implementation is included in the supplementary material and will be made publicly available upon acceptance.
Related Work
Behavior learning and selective transfer. Off-policy learning permits the collector to differ from the target [7 ] . On-policy methods remove the difference by collecting only from the current policy (policy-gradient, natural-gradient, trust-region, and asynchronous families) [42 , 37 , 33 , 20 , 32 , 34 , 27 ] , so the data source cannot outgrow the target. Common algorithms derive their evolving behavior distributions from learned value functions or actors and exploration rules [26 , 39 , 31 , 24 , 9 , 15 ] . Correction schemes persistently track the behavior (truncated importance sampling, Retrace, V-trace) reconcile it with target updates without shaping it [40 , 29 , 8 , 14 ] , so correction grows with the gap. Heuristic shaping (optimism, count-based bonuses, curiosity, learned novelty policies, variance-optimized behavior) can improve data collection, but these gains need not translate into consistent target-policy improvement [2 , 3 , 30 , 5 , 41 , 43 , 25 , 12 ] . HBPO couples three in one loop: a temporary heuristic controller guides a persistent behavior learner, and behavior data plus an advantage-filtered transfer term weighting signed target-advantage scores by absolute behavior–target disagreement feed target updates. ACER uses a one-sided residual to correct importance-weight truncation bias [40 ] ; HBPO uses absolute behavior–target disagreement to weight an additional advantage-based transfer objective. Our experiments support the contribution of this heuristic–behavior–target coupling (Section 5).
Coupled updates and transmission. With the heuristic, behavior, and target roles fixed, the key question is how improvement travels through their coupled updates. GPS fits local controllers directly into the global policy [22 ] ; conservative and regularized policy iteration fit an improved action distribution into the actor [19 , 28 , 1 ] ; TRPO, PPO, and GRPO make each improved policy the immediate data collector [32 , 34 , 35 ] ; and TPO fits the policy to a softmax-improved target distribution [18 ] . HBPO instead claims a delayed coupling: information and policy impacts enter the next generation separately, rather than the current one, making the transmission π g → q H , g → μ g + 1 → π g + 2 \pi_g \to q_{H,g} \to\mu_{g+1} \to\pi_{g+2} π g → q H , g → μ g + 1 → π g + 2 explicit; § 5 examines how this transmission operates during recovery and improvement.
Emphatic state weighting. Emphatic temporal-difference and actor–critic methods relate behavior sampling to the state weighting of an off-policy learning objective [38 , 16 , 13 , 44 ] , and expected emphatic traces and adaptive interest extend this framework to learned state emphasis [17 , 21 ] . HBPO adapts this perspective through separate learned masses for target-return learning and behavior transfer, with heuristic-improvement confidence supplying interest for the transfer mass.
Inventory control as a testbed. Deep RL has proven effective across lost-sales, dual-sourcing, and multi-echelon inventory problems [11 ] , though training conditions materially affect learning [10 ] . Böttcher et al. [4 ] optimize dual-sourcing policies by embedding inventory dynamics in recurrent neural networks. We adapt their problem formulation for our testbed. We choose inventory control as a testbed because poor decisions carry severe downside risk, and the systems admit a compact Markov formulation whose known or learnable dynamics support planning-derived guidance. § 3 defines our inventory control problem.
Problem Settings
We adapt the single-item dual-sourcing model of Böttcher et al. [4 ] into DSIS, with regular deliveries after four periods and costlier, immediate expedited deliveries. DSIS has IID demand and uninterrupted supply; our extension, DSIS-RB, adds persistent demand and supply disruptions.
We formulate both benchmarks as underlying Markov decision processes ( X , A , P , r , γ ) (\mathcal{X},\mathcal{A},\mathcal{P},r,\gamma) ( X , A , P , r , γ ) . At period t t t , DSIS observations s t = ( I t , p t , B t ) s_t = (I_t,p_t,B_t) s t = ( I t , p t , B t ) contain net inventory, a four-order pipeline, and current supply availability. DSIS-RB adds previous demand d t − 1 d_{t-1} d t − 1 , increasing the dimension from six to seven. Actions a t = ( q t r , q t e ) ∈ { 0 , … , 8 } 2 a_t = (q_t^r,q_t^e) \in\{0,\ldots,8\}^2 a t = ( q t r , q t e ) ∈ { 0 , … , 8 } 2 specify regular and expedited orders. DSIS-RB training pools trajectories across demand disruption-risk levels ρ \rho ρ , fixed within each trajectory and withheld from all learners and simulation workers. All learned networks in HBPO use only the current observation s t s_t s t , with fixed preprocessing. The full environment state is x t = ( s t , ρ ) x_t = (s_t,\rho) x t = ( s t , ρ ) for DSIS-RB and x t = s t x_t = s_t x t = s t for DSIS. Risk variables and their averaging are omitted for DSIS throughout. The environment kernel P \mathcal{P} P governs order acceptance, arrivals, demand, supply availability, and inventory carryover through x t + 1 ∼ P ( ⋅ ∣ x t , a t ) x_{t+1} \sim\mathcal{P}(\cdot\mid x_t,a_t) x t + 1 ∼ P ( ⋅ ∣ x t , a t ) (Appendix C). Period cost c t c_t c t comprises expediting, holding, and backlogging. We maximize expected discounted return using r t = − c t / 495 r_t = -c_t/495 r t = − c t /495 and γ = 0.995 \gamma= 0.995 γ = 0.995 , where 495 is the per-unit, per-period backlog penalty. All shared information access and real-interaction budgets. DSIS dynamics are public; HBPO and GPS construct DSIS-RB models from shared public simulation specification P ^ g \hat{\mathcal{P}}_g P ^ g and their observed data, without access to true stochastic parameters (Appendix C). Delayed deliveries, accumulated backlogs, and severe shortage penalties make these tasks a testbed for long-horizon decision-making under uncertainty, where failures can incur extreme costs.
Heuristic Behavior Policy Optimization Methods
Figure 1. HBPO Learning Framework. Red highlights the delayed heuristic transmission.
HBPO maintains two separately parameterized, persistent policies, μ g = μ g θ μ \mu_g = \mu_g^{\theta_\mu} μ g = μ g θ μ and π g = π g θ π \pi_g = \pi_g^{\theta_\pi} π g = π g θ π . The behavior policy μ g \mu_g μ g collects interaction data D g \mathcal{D}_g D g , while the target policy π g \pi_g π g is evaluated for task performance and defines the continuation policy for model evaluation. Under the simulation specification P ^ g \hat{\mathcal{P}}_g P ^ g , frozen within generation g g g , long-horizon comparisons of alternative first actions followed by π g \pi_g π g construct a temporary heuristic policy q H , g q_{H,g} q H , g . A shared advantage face scores policy changes and behavior–target discrepancies using discounted target-policy advantages, defining the A , J , P A,J,P A , J , P objectives. Together with heuristic guidance H H H , these yield four gradients: behavior combines target-advantage learning g A g_A g A with heuristic guidance g H g_H g H , while target combines its own advantage learning g J g_J g J with selective transfer from behavior g P g_P g P .
The frozen reference v g ( s ) = 1 5 ∑ j = 1 5 V g , j ( s ) v_g(s) = \frac{1}{5}\sum_{j=1}^{5}V_{g,j}(s) v g ( s ) = 5 1 ∑ j = 1 5 V g , j ( s ) , averaging five separately parameterized scalar critics, supports heuristic evaluation and residual-completed estimation of the discounted model action value Q g π g ( s , a ) Q_g^{\pi_g}(s,a) Q g π g ( s , a ) ; successor critics are fitted by minimizing squared TD error. Positive masses M g χ M_g^\chi M g χ provide mean-one emphatic weights w g χ ( s ) w_g^\chi(s) w g χ ( s ) , χ ∈ { J , P } \chi\in\{J,P\} χ ∈ { J , P } , for the state-weighted parts of learning; their construction and the critic updates are specified in Section 4.3.
Heuristic Objectives Guidance
To construct a temporary heuristic policy from π g \pi_g π g , anchor states are sampled from D g \mathcal{D}_g D g , with each anchor state s s s defining local time k = 0 k=0 k = 0 . We compare the long-horizon consequences of candidate first actions a a a over T = 1000 T=1000 T = 1000 periods, with all subsequent actions drawn from the frozen parent target π g \pi_g π g . Let S k S_k S k , A k A_k A k , and R k R_k R k denote the observation, action, and normalized reward at local decision stage k k k . Writing E H \mathbb{E}_{\mathrm{H}} E H for expectation under P ^ g \widehat{P}_g P g , including the synthetic-risk draw for each DSIS-RB tape, we define the expected return G T , g ( s , a ) G_{T,g}(s,a) G T , g ( s , a ) and the centered estimated utility u g ( s , a ) u_g(s,a) u g ( s , a ) as
G T , g ( s , a ) = E H [ ∑ k = 0 T − 1 R k ∣ S 0 = s , A 0 = a A k ∼ π g ( ⋅ ∣ S k ) ( 1 ≤ k < T ) ] , u g ( s , a ) = G ^ T , g ( s , a ) − ∑ a ′ ∈ A π g ( a ′ ∣ s ) G ^ T , g ( s , a ′ ) . G_{T,g}(s,a)=\mathbb{E}_{\mathrm{H}}\left[\left.\sum_{k=0}^{T-1}R_k\right|_{\substack{S_0=s,\ A_0=a\\ A_k\sim\pi_g(\cdot\mid S_k)\ (1\leq k<T)}}\right],\qquad u_g(s,a)=\widehat{G}_{T,g}(s,a)-\sum_{a'\in\mathcal{A}}\pi_g(a'\mid s)\widehat{G}_{T,g}(s,a'). G T , g ( s , a ) = E H k = 0 ∑ T − 1 R k S 0 = s , A 0 = a A k ∼ π g ( ⋅ ∣ S k ) ( 1 ≤ k < T ) , u g ( s , a ) = G T , g ( s , a ) − a ′ ∈ A ∑ π g ( a ′ ∣ s ) G T , g ( s , a ′ ) .
Here, G ^ T , g \widehat{G}_{T,g} G T , g is a Monte Carlo estimate of G T , g G_{T,g} G T , g , constructed using the frozen critic v g v_g v g as a control variate and 32 simulation tapes shared across the candidate first-action branches at each distinct query observation. All queries are performed within the model, requiring no additional environment interactions. Appendices § 11 .1.3 and § 11 .§ 4.3 detail the simulation specification and the utility estimator.
For a candidate action distribution q q q at state s s s , ⟨ q , u g ( s , ⋅ ) ⟩ = ∑ a ( q a − π g ( a ∣ s ) ) G ^ T , g ( s , a ) \langle q,u_g(s,\cdot)\rangle=\sum_a(q_a-\pi_g(a\mid s))\widehat{G}_{T,g}(s,a) ⟨ q , u g ( s , ⋅ )⟩ = ∑ a ( q a − π g ( a ∣ s )) G T , g ( s , a ) measures the estimated return gain from replacing the π g ( ⋅ ∣ s ) \pi_g(\cdot\mid s) π g ( ⋅ ∣ s ) with q q q and following π g \pi_g π g thereafter. Maximizing this gain alone would concentrate probability on utility-maximizing actions. We therefore construct a temporary stochastic policy for heuristic guidance through Hellinger regularization
q H , g ( ⋅ ∣ s ) = arg max q ∈ Δ ( A ) { ⟨ q , u g ( s , ⋅ ) ⟩ − 2 τ ∑ a ( q a − π g ( a ∣ s ) ) 2 } . (1) q_{H,g}(\cdot\mid s)=\underset{q\in\Delta(\mathcal{A})}{\arg\max}\left\{\langle q,u_g(s,\cdot)\rangle-2\tau\sum_a\left(\sqrt{q_a}-\sqrt{\pi_g(a\mid s)}\right)^2\right\}. \tag*{(1)} q H , g ( ⋅ ∣ s ) = q ∈ Δ ( A ) arg max { ⟨ q , u g ( s , ⋅ )⟩ − 2 τ a ∑ ( q a − π g ( a ∣ s ) ) 2 } . (1)
The utility term favors higher estimated returns, while the symmetric, bounded squared Hellinger penalty moderates departures from the target prior to help stabilize heuristic guidance. The coefficient τ > 0 \tau>0 τ > 0 controls this trade-off and remains fixed throughout each run after calibration under the shared hyperparameter-selection protocol HPO. With a strictly positive target prior, scalar root finding yields the temporary guidance policy at each queried state. The resulting policy has no trainable actor parameters. Appendix § 11 .4.7 details the calibration, controller solution, and derivation; Appendix § 12 reports the hyperparameter-selection protocol.
Denote ν g J \nu_g^J ν g J as the empirical distribution of behavior states weighted by w g J w_g^J w g J , introduced in § 4.3 . Starting from anchors drawn from ν g J \nu_g^J ν g J , short model rollouts under q H , g q_{H,g} q H , g generate the empirical training distribution ξ g \xi_g ξ g by retaining states at stages 0 , … , 8 0,\ldots,8 0 , … , 8 . Each retained state receives a controller query with a full T = 1000 T=1000 T = 1000 horizon, so the short rollouts select training locations without shortening the utility horizon. Using q H , g ( ⋅ ∣ s ) q_{H,g}(\cdot\mid s) q H , g ( ⋅ ∣ s ) as the fitting targets at each retained state, define the behavior-guidance gradient by
g H , g = − 2 ∇ θ μ E s ∼ ξ g [ ( arccos [ ∑ a μ θ μ ( a ∣ s ) q H , g ( a ∣ s ) ] ) 2 ] ∣ θ μ = θ g μ , (2) g_{H,g}=-2\nabla_{\theta_\mu}\mathbb{E}_{s\sim\xi_g}\left[\left(\arccos\left[\sum_a\sqrt{\mu_{\theta_\mu}(a\mid s)q_{H,g}(a\mid s)}\right]\right)^2\right]\bigg|_{\theta_\mu=\theta_g^\mu}, \tag*{(2)} g H , g = − 2 ∇ θ μ E s ∼ ξ g ( arccos [ a ∑ μ θ μ ( a ∣ s ) q H , g ( a ∣ s ) ] ) 2 θ μ = θ g μ , (2)
where q H , g q_{H,g} q H , g and ξ g \xi_g ξ g are held fixed during differentiation. Detailed in Appendix § 11 .4.9.
Delayed Heuristic–Behavior–Target Coupling
Here we explicitly construct the delayed heuristic–behavior–target coupling and certain the policies updating rules. For the n n n behavior transitions ( s i , a i , s i ′ ) (s_i,a_i,s_i') ( s i , a i , s i ′ ) in D g \mathcal{D}_g D g , we now specify the advantage face used by the coupled policy updates. Following the centering used for u g u_g u g , we evaluate each first action by its discounted return advantage: a i ( a ) = Q ( s i , a ; π g ) − ∑ b π g ( b ∣ s i ) Q ( s i , b ; π g ) a_i(a)=Q(s_i,a;\pi_g)-\sum_b\pi_g(b\mid s_i)Q(s_i,b;\pi_g) a i ( a ) = Q ( s i , a ; π g ) − ∑ b π g ( b ∣ s i ) Q ( s i , b ; π g ) , where Q ( s , a ; π g ) Q(s,a;\pi_g) Q ( s , a ; π g ) denotes the discounted model action value, helping advantage estimation, introduced in Equation (11) . The vector a ^ i \hat{\mathbf{a}}_i a ^ i records these gains at sampled state s i s_i s i compared to π g \pi_g π g , and for policy change ℓ \ell ℓ , define advantage face Λ ℓ [ ℓ ] = 1 n ∑ i = 1 n c i ℓ ( ℓ i , a ^ i ) \Lambda_\ell[\ell]=\frac{1}{n}\sum_{i=1}^n c_i^\ell(\ell_i,\hat{\mathbf{a}}_i) Λ ℓ [ ℓ ] = n 1 ∑ i = 1 n c i ℓ ( ℓ i , a ^ i ) , ( c i A , c i J , c i P ) = ( 1 , w i J , w i P ) (c_i^A,c_i^J,c_i^P)=(1,w_i^J,w_i^P) ( c i A , c i J , c i P ) = ( 1 , w i J , w i P ) . The score rewards adding weight to actions with positive estimated advantage and removing weight from actions with negative estimated advantage. Here χ ∈ { A , J , P } \chi\in\{A,J,P\} χ ∈ { A , J , P } , w i χ = w g χ ( s i ) w_i^\chi=w_g^\chi(s_i) w i χ = w g χ ( s i ) , and c i χ c_i^\chi c i χ determines how strongly each sampled state contributes. Appendix § 11 .2.3 details the estimator.
Suppressing generation indices, we hold the parent policies μ , π \mu,\pi μ , π fixed and use m , p m,p m , p to denote possible updated behavior and target policies. Both actors receive a direct advantage-driven update:
A ( m ) = τ − 1 Λ A [ m − μ ] . (3) A(m)=\tau^{-1}\Lambda_A[m-\mu]. \tag*{(3)} A ( m ) = τ − 1 Λ A [ m − μ ] . (3)
g A = τ − 1 Λ A [ μ ∇ θ μ log μ ] . (4) g_A=\tau^{-1}\Lambda_A[\mu\nabla_{\theta_\mu}\log\mu]. \tag*{(4)} g A = τ − 1 Λ A [ μ ∇ θ μ log μ ] . (4)
J ( p ) = Λ J [ p − π ] . (5) J(p)=\Lambda_J[p-\pi]. \tag*{(5)} J ( p ) = Λ J [ p − π ] . (5)
g J = τ − 1 Λ J [ π ∇ θ π log π ] . (6) g_J=\tau^{-1}\Lambda_J[\pi\nabla_{\theta_\pi}\log\pi]. \tag*{(6)} g J = τ − 1 Λ J [ π ∇ θ π log π ] . (6)
Gradients are evaluated at the frozen parent parameters, with g J = ∇ θ π ( J / τ ) g_J=\nabla_{\theta_\pi}(J/\tau) g J = ∇ θ π ( J / τ ) matching the advantage scale used by A A A and P P P . τ \tau τ is the heuristic regularization coefficient in Equation (1) . A ( m ) A(m) A ( m ) directs behavior toward actions favored by target-advantage evidence, while J ( p ) J(p) J ( p ) updates target under its own state emphasis. Together with g H g_H g H from Equation (2) , behavior combines advantage-driven learning g A g_A g A with attraction toward q H q_H q H .
Heuristic-guided behavior should also inform target action choices through an explicit policy-level signal. We define
P ( p ) = τ − 1 Λ P [ ∣ μ − π ∣ ( p / π − 1 ) ] , g P = 1 n τ ∑ i , a w i P ∣ μ i ( a ) − π i ( a ) ∣ ℓ ^ i a ∇ θ π log π i ( a ) . (7) P(p)=\tau^{-1}\Lambda_P[|\mu-\pi|(p/\pi-1)],
\qquad
g_P=\frac{1}{n\tau}\sum_{i,a}w_i^P|\mu_i(a)-\pi_i(a)|\widehat{\ell}_{ia}\nabla_{\theta_\pi}\log\pi_i(a). \tag*{(7)} P ( p ) = τ − 1 Λ P [ ∣ μ − π ∣ ( p / π − 1 )] , g P = n τ 1 i , a ∑ w i P ∣ μ i ( a ) − π i ( a ) ∣ ℓ ia ∇ θ π log π i ( a ) . (7)
where μ i ( a ) = μ ( a ∣ s i ) \mu_i(a)=\mu(a\mid s_i) μ i ( a ) = μ ( a ∣ s i ) and similarly for π \pi π . This advantage-filtered transfer weights disagreements by their magnitude and uses advantage to set their direction: supported behavioral deviations are retained, and opposed deviations are reversed in the scores. P P P can emphasize actions favored by behavior but rarely selected by target; it can increase disagreement and vanishes when μ = π \mu=\pi μ = π . Behavior’s action distribution thus guides target optimization beyond its role as a data source. Theorem 4 shows that the filtered transfer signal admits an auxiliary distribution with nonnegative gain in the frozen advantage face.
Each actor combines its two gradients in its own Fisher geometry, which relates parameter changes to local changes in policy distributions ([20 ] ). We use Fuse F ( g 1 , g 2 ) = F − 1 ( α 1 g 1 + α 2 g 2 ) \operatorname{Fuse}_F(g_1,g_2)=F^{-1}(\alpha_1g_1+\alpha_2g_2) Fuse F ( g 1 , g 2 ) = F − 1 ( α 1 g 1 + α 2 g 2 ) , with coefficients determined by the natural norms n i = g i T F − 1 g i n_i=\sqrt{g_i^{\mathsf T}F^{-1}g_i} n i = g i T F − 1 g i to balance their contributions. The updates are
θ g + 1 μ = θ g μ + t μ Fuse F μ ( g A , g H ) . (8) \theta_{g+1}^{\mu}=\theta_g^{\mu}+t_\mu\operatorname{Fuse}_{F_\mu}(g_A,g_H). \tag*{(8)} θ g + 1 μ = θ g μ + t μ Fuse F μ ( g A , g H ) . (8)
θ g + 1 π = θ g π + t π Fuse F π ( g J , g P ) . (9) \theta_{g+1}^{\pi}=\theta_g^{\pi}+t_\pi\operatorname{Fuse}_{F_\pi}(g_J,g_P). \tag*{(9)} θ g + 1 π = θ g π + t π Fuse F π ( g J , g P ) . (9)
Shared Fisher geometry makes the two gradients comparable in terms of local policy change, while norm-based fusion balances their contributions. Denote ν P \nu^P ν P for the empirical behavior-state distribution weighted by w P w^P w P , introduced in § 4.3 , step fractions t μ , t π ∈ [ 0 , 1 ] t_\mu,t_\pi\in[0,1] t μ , t π ∈ [ 0 , 1 ] then constrain endpoint movement on ( D , ξ ) (\mathcal{D},\xi) ( D , ξ ) and ( ν J , ν P ) (\nu^J,\nu^P) ( ν J , ν P ) , respectively, keeping each actor’s update within both channel budgets. Theorem 7 establishes bounded ascent of the frozen combined objective for the ideal fused direction and characterizes when both channels are first-order nondecreasing. Appendix § 11 .§ 5 specifies the fusion coefficients, Fisher operators, and endpoint checks.
Critic, State Emphasis and Initialization
The value estimates used above are constructed from the frozen critic reference v g v_g v g under P ^ g \widehat{\mathcal{P}}_g P g . Let P ^ g , z \widehat{\mathcal{P}}_{g,z} P g , z denote the model kernel at fixed risk z z z . For a model transition with reward R t R_t R t and successor observation S t + 1 S_{t+1} S t + 1 , define the one-step backup and target-policy Bellman residual by
L g , z v g ( s , a ) = E P ^ g , z [ R t + γ v g ( S t + 1 ) ∣ S t = s , A t = a ] , L_{g,z}v_g(s,a)=\mathbb{E}_{\widehat{\mathcal{P}}_{g,z}}\left[R_t+\gamma v_g(S_{t+1})\mid S_t=s,A_t=a\right], L g , z v g ( s , a ) = E P g , z [ R t + γ v g ( S t + 1 ) ∣ S t = s , A t = a ] ,
e g , z ( s ) = ∑ a ′ ∈ A π g ( a ′ ∣ s ) L g , z v g ( s , a ′ ) − v g ( s ) . (10) e_{g,z}(s)=\sum_{a'\in\mathcal{A}}\pi_g(a'\mid s)L_{g,z}v_g(s,a')-v_g(s). \tag*{(10)} e g , z ( s ) = a ′ ∈ A ∑ π g ( a ′ ∣ s ) L g , z v g ( s , a ′ ) − v g ( s ) . (10)
Appendix B.2.2 gives the expanded backups. Theorem 1 proves that, under its stated conditions, the expected discounted residual sum exactly equals the action-value error of the one-step backup under the frozen model. Adding this correction therefore recovers the model’s exact discounted action value without requiring the reference critic to have converged:
Q g π g ( s , a ) = E Z ∼ m [ L g , z v g ( s , a ) ] + E Z ∼ m , P ^ g , z , π g [ ∑ k = 1 ∞ γ k e g , z ( S k ) | S 0 = s , A 0 = a ] . (11) Q_g^{\pi_g}(s,a)=\mathbb{E}_{Z\sim m}\left[L_{g,z}v_g(s,a)\right]+\mathbb{E}_{Z\sim m,\widehat{\mathcal{P}}_{g,z},\pi_g}\left[\sum_{k=1}^{\infty}\gamma^k e_{g,z}(S_k)\,\middle|\,S_0=s,A_0=a\right]. \tag*{(11)} Q g π g ( s , a ) = E Z ∼ m [ L g , z v g ( s , a ) ] + E Z ∼ m , P g , z , π g [ k = 1 ∑ ∞ γ k e g , z ( S k ) S 0 = s , A 0 = a ] . (11)
where each rollout follows π g \pi_g π g after the first action. To update the critics, we simulate continuations starting from observations sampled from the current batch D g \mathcal{D}_g D g . The resulting training distribution ζ ~ g \widetilde{\zeta}_g ζ g describes the simulated continuation states and their retained risks (Appendix B.2.5). Each successor critic starts from its parent and is trained to minimize an expected squared TD error of the form E [ ( V θ − T V θ ′ ) 2 ] \mathbb{E}[(V_\theta-TV_{\theta'})^2] E [( V θ − T V θ ′ ) 2 ] , where the expectation is over training samples generated using ζ ~ g \widetilde{\zeta}_g ζ g , and T \mathcal{T} T denotes the model-based Bellman backup used for critic fitting under π g \pi_g π g . After a finite number of optimization steps, the updated critics are averaged to form v g + 1 v_{g+1} v g + 1 . Appendices § 11 .2.5–§ 11 .2.7 specify the sampling and update procedures.
Behavior and target policies can visit states with different frequencies, so behavior-data frequencies alone may not reflect state importance for target learning. HBPO learns two positive state M J M^J M J and M P M^P M P (raw masses) from D g D_g D g . M J M^J M J carries state importance through target-policy continuation from behavior states. For selective transfer from behavior, M P M^P M P combines confidence in heuristic improvement with existing state importance propagated along behavior-policy transitions, so state emphasis in later target updates reflects both where the heuristic appears helpful and how the behavior policy moves between states. On behavior data D g D_g D g collected by μ g \mu_g μ g , full-data normalization and shrinkage of the parent masses give positive, mean-one weights:
w g χ ( s ) = 1 + M g χ ( s ) − E D g [ M g χ ] E D g [ ( M g χ ) 2 ] , χ ∈ { J , P } . (12) w_g^\chi(s) = 1 + \frac{M_g^\chi(s) - E_{D_g}[M_g^\chi]}{\sqrt{E_{D_g}[(M_g^\chi)^2]}}, \qquad\chi\in\{J, P\}. \tag*{(12)} w g χ ( s ) = 1 + E D g [( M g χ ) 2 ] M g χ ( s ) − E D g [ M g χ ] , χ ∈ { J , P } . (12)
where E D g E_{D_g} E D g averages over all current-state occurrences in the data. These weights induce the empirical state distributions ν g J \nu_g^J ν g J and ν g P \nu_g^P ν g P used above for target-return learning and selective transfer from behavior respectively. Appendices § 11 .3.3–§ 11 .3.7 specify the mass fitting and weight construction.
To obtain the confidence used by M P M^P M P , paired simulations share random inputs to compare T T T -step undiscounted returns: q H , g q_{H,g} q H , g for nine stages then π g \pi_g π g , versus π g \pi_g π g throughout. The paired simulations return differences yield confidence scores for heuristic improvement. These scores are interpolated using fixed state features to the successor states in D g D_g D g and centered around one to form the interest signal for fitting M g + 1 P M^P_{g+1} M g + 1 P . Appendix § 11 .3.5 specifies the confidence and interest construction.
The recursion starts from cold initialization or a warm handoff. Both produce separately stored policies with μ 0 = π 0 \mu_0 = \pi_0 μ 0 = π 0 , five prefitted scalar critics, and neutral emphatic masses M 0 J = M 0 P ≡ 1 M_0^J = M_0^P \equiv1 M 0 J = M 0 P ≡ 1 , yielding w 0 J = w 0 P ≡ 1 w_0^J = w_0^P \equiv1 w 0 J = w 0 P ≡ 1 (Appendix § 11 .3.9). Cold initialization uses a near-uniform policy and fits critics on simulated states. In DSIS-RB, the model is fitted to the same first-generation real-data batch, included in the common 2M provider-transition budget. Warm handoff copies or distills a source policy and fits fresh scalar critics using centered action information from temporary Q networks trained to evaluate π 0 \pi_0 π 0 . Both routes finish with constant-offset calibration (Appendix § 11 .2.8).
Experiments
Experimental Setup
We compare HBPO with nine native baselines on DSIS and DSIS-RB: PPO, A2C, Discrete SAC, ACE, PPO-81, TRPO, GRPO, TPO, and GPS. Warm comparisons branch from each baseline’s 0.5M-interaction checkpoint into native continuation and HBPO, each using another 1.5M real interactions to assess recovery and improvement across policy sources and initializations. Cold uses 2M real interactions without native pretraining to assess learning without a source policy. Baselines use individually selected configurations; HBPO shares one configuration across all ten initializations per environment, selected on three tuning seeds to balance relative cost reductions from high-cost and non-collapsed starts among seven configuration sets. Final policies from five separate training seeds are evaluated with sampled actions; HBPO reports its target policy π \pi π . Appendix § 12 details baseline adaptations, initialization, tuning, evaluation, and separately accounted simulation costs.
Main Results
Table 1 classifies final costs as Non-collapse (c ≤ 1010 c \le1010 c ≤ 1010 ), Collapse (1010 < c ≤ 3980 1010 < c \le3980 1010 < c ≤ 3980 ), or Severe Collapse (c > 3980 c > 3980 c > 3980 ). Thresholds 1010 and 3980 represent one and four periods of backlog-cost accumulation, respectively, indicating economic severity rather than delivery delays (Appendix § 12 .§ 3 ). Appendix § 13 .§ 1 provides high-precision results and cost-grade calculations.
HBPO lowers mean final cost in 17 of 18 warm source–environment combinations, median cost in 15, and sample standard deviation across seeds in all 18. Discrete SAC on DSIS-RB is the mean-cost exception (1.45% higher), despite a lower median and standard deviation. For one seed, its native endpoint is Collapse while HBPO’s is Severe Collapse. High-cost native endpoints. Among
36 seed-matched comparisons with native final costs above 1010, HBPO achieves lower costs in 35 and costs ≤ \le ≤ 1010 in 29 (DSIS: 10/12; DSIS-RB: 19/24). Low-cost native endpoints. All 54 native endpoints with costs ≤ \le ≤ 1010 have seed-matched HBPO endpoints within this range, with 46 achieving lower costs. Cold starts. Cold avoids final cost collapse across all five seeds in both environments, with mean costs of 45.1 on DSIS and 72.1 on DSIS-RB. By median cost over all five seeds, Cold ranks seventh among Cold and the nine native baselines on DSIS and first on DSIS-RB.
Tracing Heuristic–Behavior–Target Transmission
These diagnostics characterize heuristic uptake and subsequent advantage-aligned target responses during policy recovery and stabilization. Figure 2 examines the delayed transmission chain π g → q H , g → μ g + 1 → π g + 2 \pi_g \to q_{H,g} \to\mu_{g+1} \to\pi_{g+2} π g → q H , g → μ g + 1 → π g + 2 (red in Section 1). Relative steps measure update size relative to each guidance gap ρ H μ = ∥ μ g + 1 − μ g ∥ H ∥ q H , g − μ g ∥ H \rho_{H\mu} = \frac{\lVert\mu_{g+1}-\mu_g\rVert_H}{\lVert q_{H,g}-\mu_g\rVert_H} ρ H μ = ∥ q H , g − μ g ∥ H ∥ μ g + 1 − μ g ∥ H , ρ μ π = ∥ π g + 2 − π g + 1 ∥ J ∥ μ g + 1 − π g + 1 ∥ J \rho_{\mu\pi} = \frac{\lVert\pi_{g+2}-\pi_{g+1}\rVert_J}{\lVert\mu_{g+1}-\pi_{g+1}\rVert_J} ρ μ π = ∥ μ g + 1 − π g + 1 ∥ J ∥ π g + 2 − π g + 1 ∥ J . Here, 0 denotes no update and 1 equal update and gap norms. Larger ratios need not be better: the two updates balance A A A with H H H and J J J with P P P respectively. Weighted cosines measure alignment with the same gaps: c H μ = cos H ( μ g + 1 − μ g , q H , g − μ g ) c_{H\mu} = \cos_H(\mu_{g+1}-\mu_g, q_{H,g}-\mu_g) c H μ = cos H ( μ g + 1 − μ g , q H , g − μ g ) , c μ π = cos J ( π g + 2 − π g + 1 , μ g + 1 − π g + 1 ) c_{\mu\pi} = \cos_J(\pi_{g+2}-\pi_{g+1}, \mu_{g+1}-\pi_{g+1}) c μ π = cos J ( π g + 2 − π g + 1 , μ g + 1 − π g + 1 ) . Values − 1 -1 − 1 , 0, and 1 indicate opposite, orthogonal, and fully aligned directions. Under the frozen target advantage a ^ g + 1 \hat a_{g+1} a ^ g + 1 , S ± = 1 { ( ± ( μ g + 1 − π g + 1 ) a ^ g + 1 > 0 } S_{\pm} = 1\{(\pm(\mu_{g+1}-\pi_{g+1})\hat a_{g+1} > 0\} S ± = 1 {( ± ( μ g + 1 − π g + 1 ) a ^ g + 1 > 0 } identify advantageous and disadvantageous behavioral deviations. Their opportunity magnitudes and net target responses are B ± = ⟨ [ ± ( μ g + 1 − π g + 1 ) a ^ g + 1 ] + ⟩ P B_{\pm} = \langle[\pm(\mu_{g+1}-\pi_{g+1})\hat a_{g+1}]_+ \rangle_P B ± = ⟨[ ± ( μ g + 1 − π g + 1 ) a ^ g + 1 ] + ⟩ P , G ± = ⟨ ( π g + 2 − π g + 1 ) a ^ g + 1 S ± ⟩ P G_{\pm} = \langle(\pi_{g+2}-\pi_{g+1})\hat a_{g+1}S_{\pm}\rangle_P G ± = ⟨( π g + 2 − π g + 1 ) a ^ g + 1 S ± ⟩ P . Positive G + G_+ G + and G − G_- G − indicate adoption of advantageous deviations and correction of disadvantageous ones respectively. Norms, cosines, and averages use the corresponding diagnostic weights.
Figure 2. Transmission analysis. Left: standard evaluation costs of μ \mu μ , π \pi π and a q H q_H q H reference averaging H H H -weighted immediate costs over the original 9-stages rollouts (lower is better). Middle: relative step ϱ \varrho ϱ (top) and direction cosine c c c (bottom) for q H → μ q_H \to\mu q H → μ and μ → π \mu\to\pi μ → π . Right: adoption and reflection, with net responses G ± G_{\pm} G ± , dashed opportunities B ± B_{\pm} B ± , and shaded favorable/adverse contributions above/below zero.
Together, these observations provide evidence of effective heuristic–behavior–target transmission, whose corrective dynamics help explain recovery from policy collapse, improved average performance, and stable cold-start learning. Existence. In Figure 2 , broadly corresponding trends in the short-horizon q H q_H q H reference and the behavior and target costs accompany nonzero relative steps and predominantly positive alignment along α H , g → μ g + 1 → π g + 2 \alpha_{H,g} \to\mu_{g+1} \to\pi_{g+2} α H , g → μ g + 1 → π g + 2 . Together, these indicate sustained guidance, with target improvement sometimes following behavior improvement. Corrective effectiveness. During the pronounced DSIS–PPO cost reduction, μ → π \mu\to\pi μ → π exhibits relatively large steps and closer alignment with nonzero q H → μ q_H \to\mu q H → μ steps, alongside mostly positive adoption and reflection responses. This connects continued heuristic uptake with target adjustments that, on balance, retain advantageous behavioral deviations and oppose disadvantageous ones. Positive reflection also explains how useful transfer can coexist with weak geometric following. Interpretability. In DSIS–Cold, strong early alignment and substantial selection responses give way to weaker following and smaller B B B , G G G as costs stabilize. DSIS-RB–Cold shows a complementary pattern: the target lags behind behavior when relative transfer steps are small, then catches up as these steps increase under positive alignment. Changes in transmission thus correspond to identifiable phases of cost improvement. The appendix shows related correspondences: DSIS–TPO exhibits diminishing opportunities and responses after rapid recovery, whereas DSIS–ACE combines positive adoption and reflection with weak or negative alignment and falling costs. In DSIS–TRPO, an interval of negative reflection instead accompanies subsequent cost increases, helping interpret temporary setbacks. These patterns characterize how selective transmission operates during learning. The component interventions below test its contribution to final policy performance.
Ablation Results
No-A removes target-advantage feedback from behavior learning; No-P removes explicit μ → π \mu\to\pi μ → π transfer while retaining behavior-generated data. KL-P replaces P P P with KL imitation at a matched natural-gradient norm. Near-on-policy uses the newly updated behavior policy for immediate KL transfer, then sets μ = π \mu= \pi μ = π . In Figure 3 , Full consistently outperforms all four ablations from Cold and avoids the high-cost failures under No-P and Near-on-policy in DSIS–TRPO. Together with broad degradation under No-A, these results support target-informed behavior learning and explicit transfer beyond data collection. KL-P and Near-on-policy often yield slightly lower costs from PPO and SAC starts, indicating a trade-off with protection against high-cost failures. Mean paired relative-cost changes favor Full for all four ablations.
Figure 3. Final target-policy cost changes (%) relative to matched Full HBPO; positive values favor Full. Each color represents one environment–source, with seeds 0–4 from left to right. Dark bars indicate ablation costs above 10 10 10^{10} 1 0 10 . Shared axes are linear up to 100% and logarithmic above.
No-H removes H H H and its weighting and geometry from the Fuse behavior update while retaining behavior–target transmission. Random-H generates random guidance targets around μ \mu μ , matching the statewise distance of the simulation-derived q H q_H q H from μ \mu μ . Both increase mean final cost in most tested settings. In these DSIS ablations, No-H and Random-H produce three and one collapsed endpoints, respectively, versus none for Full. Random-H occasionally outperforms Full but costs more in 32 of 40 matched runs, supporting the value of informative guidance directions.
Figure 4. Heuristic-guidance ablations, No-H and Random-H. Plotting conventions follow Figure 3.
Discussion and Limitations
First, generation-level diagnostics and component ablations together support heuristic–behavior–target transmission as a mechanism contributing to policy recovery and improvement. The delayed path follows from the frozen-parent update schedule; the experiments assess the coupled learning mechanism and its component channels. Emphatic weighting adapts the principle of propagating state interest through policy-dependent transitions to HBPO’s two learning roles, providing separate state emphasis for target-return learning and heuristic-informed transfer. The ablations in Appendix § 13 .§ 3 show mixed endpoint effects, so a consistent final-cost benefit is not established in the evaluated settings. We adopt this pathway considering its theoretical advantages, which not obviously show in experiment endpoint. Achieving sustained state-distribution correction remains an opportunity to strengthen this framework. Second, HBPO improves 35 of 36 comparisons with high-cost native endpoints. For DSIS-RB SAC seed 0, cost falls from 7.76k at handoff to 5.36k but exceeds native continuation’s 1.18k, highlighting limits to recovery across initializations. Third, the coupling rules are not inventory-specific, though HBPO uses structural priors in P ^ \hat{P} P ^ . Shared information access and comparison with model-informed GPS support the architectural comparison under common modeling assumptions. Isolating the contribution of structural priors and testing other domains remain future directions.
Conclusion
We introduced HBPO, built on an explicit delayed heuristic–behavior–target coupling mechanism. Across DSIS and DSIS-RB, HBPO recovers many high-cost policies, improves overall performance, and avoids final collapse in all tested cold starts. Transmission diagnostics trace how heuristic guidance shapes behavior learning and subsequent selective target updates, helping explain these gains. Ablations further support the contributions of informative heuristic guidance, target-informed behavior learning, and explicit behavior–target transfer to performance and robustness.
AI Use Statement
We used OpenAI Codex with GPT-5.5 Extra high, GPT-5.6 Sol Ultra and GPT-6 Astra Ultra, primarily in regular mode and occasionally in fast mode. These were the only generative AI assistants used in this work. Codex assisted formalizing the mathematics of the authors’ method, developing and checking proofs, and preparing and inspecting the appendix. It also generated the method implementation and ran and debugged experiments under the authors’ supervision. The authors polished the main section, identified and reviewed related work, and checked references, with assistance from Codex. The authors conceived the method and reinforcement-learning framework, designed the experiments, analyzed the results, and produced the figures.
The authors carefully drafted the main section, reviewed the manuscript, read the AI-assisted proofs, and manually worked through selected derivations. Their code review consisted of inspecting key implementation components. The authors take full responsibility for the final manuscript and accompanying code, including the mathematical claims, references, and experimental results.
Reproducibility Statement
Appendix A summarizes the notation. Appendix B specifies the algorithm, benchmark dynamics, information access, initialization procedures, and mathematical assumptions and derivations. Appendix C describes hyperparameter selection, interaction budgets, random seeds, and evaluation. Appendix D reports extended results, and Appendix E details the ablation interventions and diagnostic procedures. An anonymized code package is included in the supplementary material, with the implementation, experiment configurations, dependency specifications, and instructions for running training and evaluation. The code will be made publicly available upon acceptance.
Notation
Table 2 summarizes the benchmark, model, critic, and policy-update notation. Its main entries follow Appendix B; the final groups record notation used in the main text and the other appendices. Scope-qualified entries distinguish local uses of the same letter rather than identifying different mathematical objects.
Table 2. Notation and scope. The generation-g g g model, policies, reference critic, and sampling measures are frozen when constructing that generation’s learning signals.
Algorithmic Details and Mathematical Derivations
Benchmark and Fitted Simulation Model
Scope and the DSIS special case. We first specify DSIS-RB and its fitted simulation model; the critic construction follows in Section B.2. DSIS is recovered as a special case of this framework. by making regular supply permanently available and replacing the demand process with its original public i.i.d. distribution. Its previous-demand coordinate and latent-risk variable can then be omitted. Thus, recovering DSIS requires specializing both supply and demand; setting the RB risk to zero alone does not recover its original demand process.
Throughout, ρ \rho ρ denotes the actual trajectory-specific risk in the environment, whereas z z z is a realization of the synthetic risk variable Z Z Z drawn by the learning algorithm. Neither risk is an input to the policy or value networks. Model-based computations use a declared simulation distribution for Z Z Z , independently of the unobserved risk of an observed trajectory. The nonnegative integer g g g indexes learning generations; model snapshot g g g includes the real batch collected for that generation. A synthetic tape is one simulated continuation with its own random draws.
Public supply-chain dynamics and rewards.
At period t ∈ { 0 , 1 , … } t \in\{0,1,\ldots\} t ∈ { 0 , 1 , … } , let s t s_t s t denote the observed inventory configuration. Its net inventory is I t ∈ Z I_t \in\mathbb{Z} I t ∈ Z ; its regular-order pipeline is p t = ( p t , 1 , p t , 2 , p t , 3 , p t , 4 ) \mathbf{p}_t=(p_{t,1},p_{t,2},p_{t,3},p_{t,4}) p t = ( p t , 1 , p t , 2 , p t , 3 , p t , 4 ) , where p t , ℓ ∈ { 0 , … , 8 } p_{t,\ell}\in\{0,\ldots,8\} p t , ℓ ∈ { 0 , … , 8 } is the outstanding quantity at stage ℓ ∈ { 1 , 2 , 3 , 4 } \ell\in\{1,2,3,4\} ℓ ∈ { 1 , 2 , 3 , 4 } ; and its current availability is B t ∈ { 0 , 1 } B_t\in\{0,1\} B t ∈ { 0 , 1 } , with 0 meaning unavailable and 1 available. Negative I t I_t I t represents backlog. DSIS-RB additionally observes the previous demand d t − 1 d_{t-1} d t − 1 . The six- and seven-dimensional observations are
s t D S I S = ( I t , p t , B t ) , s t R B = ( I t , p t , B t , d t − 1 ) . (13) s_t^{\mathrm{DSIS}}=(I_t,\mathbf{p}_t,B_t),\qquad s_t^{\mathrm{RB}}=(I_t,\mathbf{p}_t,B_t,d_{t-1}). \tag*{(13)} s t DSIS = ( I t , p t , B t ) , s t RB = ( I t , p t , B t , d t − 1 ) . (13)
Hereafter, s t s_t s t refers to the applicable observation above. We use S t S_t S t for the corresponding random observation and D t D_t D t for random demand, with realization d t d_t d t . Let A \mathcal{A} A be the finite action set, and let q t r q_t^r q t r and q t e q_t^e q t e denote the regular and expedited order quantities:
a t = ( q t r , q t e ) ∈ A = { 0 , … , 8 } 2 . (14) a_t=(q_t^r,q_t^e)\in\mathcal{A}=\{0,\ldots,8\}^2. \tag*{(14)} a t = ( q t r , q t e ) ∈ A = { 0 , … , 8 } 2 . (14)
There are 81 actions. The implementation encodes ( q t r , q t e ) (q_t^r,q_t^e) ( q t r , q t e ) by the integer 9 q t r + q t e ∈ { 0 , … , 80 } 9q_t^r+q_t^e\in\{0,\ldots,80\} 9 q t r + q t e ∈ { 0 , … , 80 } ; the equations below use the pair.
Given realized demand d t d_t d t and next-period availability B t + 1 B_{t+1} B t + 1 , the public inventory and pipeline equations are
I t + 1 = I t + p t , 1 + q t e − d t , p t + 1 = ( p t , 2 , p t , 3 , p t , 4 , B t q t r ) . (15) \begin{aligned}
I_{t+1}&=I_t+p_{t,1}+q_t^e-d_t,\\
\mathbf{p}_{t+1}&=(p_{t,2},p_{t,3},p_{t,4},B_tq_t^r). \tag*{(15)}
\end{aligned} I t + 1 p t + 1 = I t + p t , 1 + q t e − d t , = ( p t , 2 , p t , 3 , p t , 4 , B t q t r ) . (15)
Define the deterministic RB observation-transition map F F F by
s t + 1 = F ( s t , a t , d t , B t + 1 ) = ( I t + 1 , p t , 2 , p t , 3 , p t , 4 , B t q t r , B t + 1 , d t ) . (16) s_{t+1}=F(s_t,a_t,d_t,B_{t+1})=(I_{t+1},p_{t,2},p_{t,3},p_{t,4},B_tq_t^r,B_{t+1},d_t). \tag*{(16)} s t + 1 = F ( s t , a t , d t , B t + 1 ) = ( I t + 1 , p t , 2 , p t , 3 , p t , 4 , B t q t r , B t + 1 , d t ) . (16)
DSIS uses the same map with the last coordinate omitted and B t = B t + 1 = 1 B_t=B_{t+1}=1 B t = B t + 1 = 1 . Current availability B t B_t B t determines whether the new regular order q t r q_t^r q t r is accepted. Previously accepted pipeline orders p t , 1 p_{t,1} p t , 1 still arrive during an outage. Expedited orders q t e q_t^e q t e arrive immediately, while accepted regular orders B t q t r B_tq_t^r B t q t r enter the four-period pipeline.
For a real number x x x , write [ x ] + = max { x , 0 } [x]_+=\max\{x,0\} [ x ] + = max { x , 0 } . Let c ( s , a , d ) c(s,a,d) c ( s , a , d ) and r ( s , a , d ) r(s,a,d) r ( s , a , d ) denote the period cost and normalized reward functions, respectively; their realized values are c t c_t c t and r t r_t r t . They satisfy
c t = c ( s t , a t , d t ) = 10 q t e + 5 [ I t + 1 ] + + 495 [ − I t + 1 ] + , r t = r ( s t , a t , d t ) = − c t 495 . (17) \begin{aligned}
c_t&=c(s_t,a_t,d_t)=10q_t^e+5[I_{t+1}]_++495[-I_{t+1}]_+,\\
r_t&=r(s_t,a_t,d_t)=-\frac{c_t}{495}. \tag*{(17)}
\end{aligned} c t r t = c ( s t , a t , d t ) = 10 q t e + 5 [ I t + 1 ] + + 495 [ − I t + 1 ] + , = r ( s t , a t , d t ) = − 495 c t . (17)
All values, Bellman residuals, and heuristic utilities below use these normalized reward units. The critic discount factor is γ = 0.995 \gamma=0.995 γ = 0.995 .
The true benchmark transition law.
For the RB benchmark, let D R B \mathcal{D}_{\mathrm{RB}} D RB denote the demand support:
D R B = { 0 , 1 , 2 , 3 , 8 } . (18) \mathcal{D}_{\mathrm{RB}}=\{0,1,2,3,8\}. \tag*{(18)} D RB = { 0 , 1 , 2 , 3 , 8 } . (18)
The stationary demand probability row vector ω D \omega_D ω D is
ω D = ( 0.3 , 0.2 , 0.2 , 0.2 , 0.1 ) = 1 10 ( 3 , 2 , 2 , 2 , 1 ) (19) \omega_D=(0.3,0.2,0.2,0.2,0.1)=\frac{1}{10}(3,2,2,2,1) \tag*{(19)} ω D = ( 0.3 , 0.2 , 0.2 , 0.2 , 0.1 ) = 10 1 ( 3 , 2 , 2 , 2 , 1 ) (19)
Its entries follow the displayed order in D R B \mathcal{D}_{\mathrm{RB}} D RB , and ω D ( d ) \omega_D(d) ω D ( d ) denotes the entry associated with demand value d d d . Thus ω D ( d ) = Pr ( D t = d ) \omega_D(d) = \Pr(D_t = d) ω D ( d ) = Pr ( D t = d ) under stationary initialization, where Pr \Pr Pr denotes probability. Let 1 { E } \mathbf{1}\{E\} 1 { E } equal one when statement E E E holds and zero otherwise. For x , d ∈ D R B x,d \in\mathcal{D}_{\mathrm{RB}} x , d ∈ D RB , define the demand transition matrix K D K_D K D by
Pr ( D t = d ∣ D t − 1 = x ) = K D ( x , d ) = 0.751 { d = x } + 0.25 ω D ( d ) . (20) \Pr(D_t = d \mid D_{t-1} = x) = K_D(x,d) = 0.75\mathbf{1}\{d = x\} + 0.25\omega_D(d). \tag*{(20)} Pr ( D t = d ∣ D t − 1 = x ) = K D ( x , d ) = 0.75 1 { d = x } + 0.25 ω D ( d ) . (20)
Thus, with probability 0.75 demand retains its previous value; otherwise it is independently redrawn from ω D \omega_D ω D . This construction satisfies ω D K D = ω D \omega_D K_D = \omega_D ω D K D = ω D and ∑ d ∈ D R B d ω D ( d ) = 2 \sum_{d \in\mathcal{D}_{\mathrm{RB}}} d\omega_D(d) = 2 ∑ d ∈ D RB d ω D ( d ) = 2 .
An environment trajectory receives a risk ρ ∈ [ 0 , 0.5 ] \rho\in[0,0.5] ρ ∈ [ 0 , 0.5 ] , which remains fixed throughout that trajectory. Conditional on ρ \rho ρ , let K B , ρ ( b c u r , u ) = Pr ( B t + 1 = u ∣ B t = b c u r , ρ ) K_{B,\rho}(b_{\mathrm{cur}},u) = \Pr(B_{t+1} = u \mid B_t = b_{\mathrm{cur}},\rho) K B , ρ ( b cur , u ) = Pr ( B t + 1 = u ∣ B t = b cur , ρ ) for b c u r , u ∈ { 0 , 1 } b_{\mathrm{cur}},u \in\{0,1\} b cur , u ∈ { 0 , 1 } . In row and column order ( 0 , 1 ) (0,1) ( 0 , 1 ) , this supply transition matrix is
K B , ρ = ( 3 4 1 4 ρ 4 ( 1 − ρ ) 1 − ρ 4 ( 1 − ρ ) ) . (21) K_{B,\rho} =
\begin{pmatrix}
\frac{3}{4} & \frac{1}{4} \\
\frac{\rho}{4(1-\rho)} & 1-\frac{\rho}{4(1-\rho)}
\end{pmatrix}. \tag*{(21)} K B , ρ = ( 4 3 4 ( 1 − ρ ) ρ 4 1 1 − 4 ( 1 − ρ ) ρ ) . (21)
Its stationary probability row vector is ( ρ , 1 − ρ ) (\rho,1-\rho) ( ρ , 1 − ρ ) : the flows ρ / 4 \rho/4 ρ /4 from 0 to 1 and ( 1 − ρ ) ρ / [ 4 ( 1 − ρ ) ] (1-\rho)\rho/[4(1-\rho)] ( 1 − ρ ) ρ / [ 4 ( 1 − ρ )] from 1 to 0 agree. Demand and supply innovations are independent conditional on the current observation and ρ \rho ρ . Below we suppress time indices: s = s t s=s_t s = s t , a = a t a=a_t a = a t , s ′ = s t + 1 s'=s_{t+1} s ′ = s t + 1 , and u = B t + 1 u=B_{t+1} u = B t + 1 . Write B ( s ) B(s) B ( s ) for the availability coordinate of s s s and d − ( s ) d^{-}(s) d − ( s ) for its previous-demand coordinate. For any point x x x , δ x \delta_x δ x denotes the probability measure with unit mass at x x x ; d r dr d r and d s ′ ds' d s ′ indicate the reward and next-observation coordinates of a measure. The joint reward–observation transition kernel P ρ P_\rho P ρ is
P ρ ( d r , d s ′ ∣ s , a ) = ∑ d ∈ D R B ∑ u ∈ { 0 , 1 } K D ( d − ( s ) , d ) K B , ρ ( B ( s ) , u ) δ ( r ( s , a , d ) , F ( s , a , d , u ) ) ( d r , d s ′ ) . (22) P_\rho(dr,ds' \mid s,a) =
\sum_{d \in\mathcal{D}_{\mathrm{RB}}}
\sum_{u \in\{0,1\}}
K_D(d^{-}(s),d)K_{B,\rho}(B(s),u)
\delta_{(r(s,a,d),F(s,a,d,u))}(dr,ds'). \tag*{(22)} P ρ ( d r , d s ′ ∣ s , a ) = d ∈ D RB ∑ u ∈ { 0 , 1 } ∑ K D ( d − ( s ) , d ) K B , ρ ( B ( s ) , u ) δ ( r ( s , a , d ) , F ( s , a , d , u )) ( d r , d s ′ ) . (22)
At each episode initialization, inventory and pipeline quantities are zero. Writing Bernoulli ( p ) \operatorname{Bernoulli}(p) Bernoulli ( p ) for a binary distribution with probability p p p of one, the initial demand and availability obey
D − 1 ∼ ω D , B 0 ∣ ρ ∼ Bernoulli ( 1 − ρ ) , (23) D_{-1} \sim\omega_D,\qquad B_0 \mid\rho\sim\operatorname{Bernoulli}(1-\rho), \tag*{(23)} D − 1 ∼ ω D , B 0 ∣ ρ ∼ Bernoulli ( 1 − ρ ) , (23)
with independent demand and supply initialization conditional on ρ \rho ρ .
Risk stratification and its statistical target. The registered risk grid is ρ j = 0.1 j \rho_j=0.1j ρ j = 0.1 j , j = 0 , … , 5 j=0,\ldots,5 j = 0 , … , 5 . An episode contains H e n v = 1000 H_{\mathrm{env}}=1000 H env = 1000 physical periods. Let π \pi π be a fixed policy assigning a probability π ( a ∣ s ) \pi(a\mid s) π ( a ∣ s ) to each a ∈ A a\in\mathcal{A} a ∈ A , and let E π , ρ j \mathbb{E}_{\pi,\rho_j} E π , ρ j denote expectation over its actions and the benchmark disturbances at risk ρ j \rho_j ρ j , with the initialization above. Its expected episode mean cost is
J π ( ρ j ) = E π , ρ j [ 1 H e n v ∑ t = 0 H e n v − 1 c t ] . (24) J_\pi(\rho_j) = \mathbb{E}_{\pi,\rho_j}\left[\frac{1}{H_{\mathrm{env}}}\sum_{t=0}^{H_{\mathrm{env}}-1}c_t\right]. \tag*{(24)} J π ( ρ j ) = E π , ρ j [ H env 1 t = 0 ∑ H env − 1 c t ] . (24)
Define R t r a p ( π ) \mathcal{R}_{\mathrm{trap}}(\pi) R trap ( π ) as the normalized trapezoidal risk score and ω ρ , j \omega_{\rho,j} ω ρ , j as its weight at grid point j j j ; ω ρ \omega_\rho ω ρ collects these six weights in grid order:
R t r a p ( π ) = 1 0.5 ∑ j = 0 4 0.1 2 [ J π ( ρ j ) + J π ( ρ j + 1 ) ] = ∑ j = 0 5 ω ρ , j J π ( ρ j ) , ω ρ = 1 10 ( 1 , 2 , 2 , 2 , 2 , 1 ) . (25) \begin{aligned}
\mathcal{R}_{\mathrm{trap}}(\pi)
= \frac{1}{0.5}\sum_{j=0}^{4}\frac{0.1}{2}\left[J_\pi(\rho_j)+J_\pi(\rho_{j+1})\right] \\
&= \sum_{j=0}^{5}\omega_{\rho,j}J_\pi(\rho_j),\qquad
\omega_\rho=\frac{1}{10}(1,2,2,2,2,1). \tag*{(25)}
\end{aligned} R trap ( π ) = 0.5 1 j = 0 ∑ 4 2 0.1 [ J π ( ρ j ) + J π ( ρ j + 1 ) ] = j = 0 ∑ 5 ω ρ , j J π ( ρ j ) , ω ρ = 10 1 ( 1 , 2 , 2 , 2 , 2 , 1 ) . (25)
The endpoints receive half the weight of an interior grid point because each endpoint belongs to one trapezoid, whereas each interior point belongs to two. These risk weights are distinct from the demand probabilities ω D \omega_D ω D .
The probability measure ν e n v \nu_{\mathrm{env}} ν env describing the target risk composition is
ν e n v = ∑ j = 0 5 ω ρ , j δ ρ j . (26) \nu_{\mathrm{env}}=\sum_{j=0}^{5}\omega_{\rho,j}\delta_{\rho_j}. \tag*{(26)} ν env = j = 0 ∑ 5 ω ρ , j δ ρ j . (26)
Writing ρ ∼ ν e n v \rho\sim\nu_{\mathrm{env}} ρ ∼ ν env specifies this target composition, not independent sampling of every training trajectory. Training implements proportional stratification in blocks of 25 trajectories. The desired counts are 25 ω ρ = ( 2.5 , 5 , 5 , 5 , 5 , 2.5 ) 25\omega_{\rho}=(2.5,5,5,5,5,2.5) 25 ω ρ = ( 2.5 , 5 , 5 , 5 , 5 , 2.5 ) . Let n ( 0 ) \mathbf{n}^{(0)} n ( 0 ) and n ( 1 ) \mathbf{n}^{(1)} n ( 1 ) denote the alternating integer count vectors, with coordinates ordered by the risk grid:
n ( 0 ) = ( 3 , 5 , 5 , 5 , 5 , 2 ) , \mathbf{n}^{(0)}=(3,5,5,5,5,2), n ( 0 ) = ( 3 , 5 , 5 , 5 , 5 , 2 ) ,
n ( 1 ) = ( 2 , 5 , 5 , 5 , 5 , 3 ) , \mathbf{n}^{(1)}=(2,5,5,5,5,3), n ( 1 ) = ( 2 , 5 , 5 , 5 , 5 , 3 ) ,
n ( 0 ) + n ( 1 ) = ( 5 , 10 , 10 , 10 , 10 , 5 ) = 50 ω ρ . (27) \mathbf{n}^{(0)}+\mathbf{n}^{(1)}=(5,10,10,10,10,5)=50\omega_{\rho}. \tag*{(27)} n ( 0 ) + n ( 1 ) = ( 5 , 10 , 10 , 10 , 10 , 5 ) = 50 ω ρ . (27)
Each block is privately permuted before trajectories are supplied. Alternating the two integer allocations avoids systematically rounding one endpoint upward: every complete pair of blocks has exactly the registered risk proportions, without random fluctuation in its stratum counts.
For a policy held fixed across such a pair, let Y ℓ Y_{\ell} Y ℓ be the realized mean cost of trajectory ℓ ∈ { 1 , … , 50 } \ell\in\{1,\ldots,50\} ℓ ∈ { 1 , … , 50 } , and let E \mathbb{E} E denote expectation over benchmark and policy randomness. Then
E [ 1 50 ∑ ℓ = 1 50 Y ℓ ] = ∑ j = 0 5 ω ρ , j J π ( ρ j ) = R t r a p ( π ) . (28) \mathbb{E}\left[\frac{1}{50}\sum_{\ell=1}^{50}Y_{\ell}\right]=\sum_{j=0}^{5}\omega_{\rho,j}J_{\pi}(\rho_j)=\mathcal{R}_{\mathrm{trap}}(\pi). \tag*{(28)} E [ 50 1 ℓ = 1 ∑ 50 Y ℓ ] = j = 0 ∑ 5 ω ρ , j J π ( ρ j ) = R trap ( π ) . (28)
This is unbiasedness for the registered discrete trapezoidal target. For the actual changing training policies, the applicable guarantee is exact two-block risk balance; it does not assert unbiasedness of a training update for a single fixed-policy objective. Likewise, the trapezoidal score is a quadrature approximation to a continuous-risk integral, not an exactly unbiased estimator of that integral. Formal evaluation holds the policy fixed, evaluates each stratum separately, and combines its sample mean with ω ρ \omega_{\rho} ω ρ .
Within an episode, the augmented variable X t = ( S t , ρ ) X_t=(S_t,\rho) X t = ( S t , ρ ) is a controlled Markov state for the physical dynamics. The observation S t S_t S t alone need not be Markov after a persistent hidden risk is marginalized. The displayed physical kernel excludes the scheduled reset after H e n v H_{\mathrm{env}} H env periods. The preceding probabilities specify the benchmark for reproducibility; the RB learner does not receive its true stochastic kernels, trajectory risks, risk assignments, or private random tapes.
For DSIS, the same physical and reward equations apply on the reachable set B t = B t + 1 = 1 B_t=B_{t+1}=1 B t = B t + 1 = 1 . Here Unif denotes the uniform distribution, and demand is independent and identically distributed (i.i.d.):
D t ∼ i i d Unif { 0 , 1 , 2 , 3 , 4 } . (29) D_t \mathrel{\overset{\mathrm{iid}}{\sim}} \operatorname{Unif}\{0,1,2,3,4\}. \tag*{(29)} D t ∼ iid Unif { 0 , 1 , 2 , 3 , 4 } . (29)
Its observation is the six-dimensional vector
s t = ( I t , p t , B t ) . (30) s_t=(I_t,p_t,B_t). \tag*{(30)} s t = ( I t , p t , B t ) . (30)
Constructing the fitted simulation model.
For DSIS-RB, the generation-g g g model is fitted only to counted real observations. We define its components before summarizing all structural and estimation assumptions in Table 3 .
Table 3. Declared structural and estimation priors. Both means DSIS and DSIS-RB. Baseline use refers to explicit use in the learner; all methods interact with the same physical benchmark.
Observed-data demand model. Let D ≤ g r e a l \mathcal{D}^{\mathrm{real}}_{\leq g} D ≤ g real be the multiset of counted, observed transitions available after collecting the generation-g g g batch, including any counted initialization prefix. Every observed occurrence is counted once, including repeated transitions. Set N g r e a l = ∣ D ≤ g r e a l ∣ > 0 N_g^{\mathrm{real}}=\lvert\mathcal{D}^{\mathrm{real}}_{\leq g}\rvert>0 N g real = ∣ D ≤ g real ∣ > 0 ; no fitted RB model is formed without real data. Recall that d − ( s ) d^{-}(s) d − ( s ) selects the previous-demand coordinate. Let D ^ g \widehat{\mathcal{D}}_g D g be the union of the values of d − ( s ) d^{-}(s) d − ( s ) and d − ( s ′ ) d^{-}(s') d − ( s ′ ) in this multiset. For x , y ∈ D ^ g x,y\in\widehat{\mathcal{D}}_g x , y ∈ D g , define the pair count C g ( x , y ) C_g(x,y) C g ( x , y ) , its row total C g ( x ) C_g(x) C g ( x ) , and the total count as follows:
C g ( x , y ) = ∑ ( s , a , r , s ′ ) ∈ D ≤ g r e a l 1 { d − ( s ) = x , d − ( s ′ ) = y } . (31) C_g(x,y)=\sum_{(s,a,r,s')\in\mathcal{D}^{\mathrm{real}}_{\leq g}}\mathbf{1}\{d^{-}(s)=x,\ d^{-}(s')=y\}. \tag*{(31)} C g ( x , y ) = ( s , a , r , s ′ ) ∈ D ≤ g real ∑ 1 { d − ( s ) = x , d − ( s ′ ) = y } . (31)
The row and total counts satisfy
C g ( x ) = ∑ y ∈ D ^ g C g ( x , y ) , N g r e a l = ∑ x , y ∈ D ^ g C g ( x , y ) . (32) C_g(x)=\sum_{y\in\widehat{\mathcal{D}}_g}C_g(x,y),\qquad N_g^{\mathrm{real}}=\sum_{x,y\in\widehat{\mathcal{D}}_g}C_g(x,y). \tag*{(32)} C g ( x ) = y ∈ D g ∑ C g ( x , y ) , N g real = x , y ∈ D g ∑ C g ( x , y ) . (32)
The pooled empirical next-demand probability mass function ω ^ g \widehat{\omega}_g ω g is
ω ^ g ( y ) = ∑ x ∈ D ^ g C g ( x , y ) N g r e a l . (33) \widehat{\omega}_g(y)=\frac{\sum_{x\in\widehat{\mathcal{D}}_g}C_g(x,y)}{N_g^{\mathrm{real}}}. \tag*{(33)} ω g ( y ) = N g real ∑ x ∈ D g C g ( x , y ) . (33)
With pseudocount mass α D = 1 \alpha_D = 1 α D = 1 , define the fitted demand transition matrix K ^ D , g \widehat{K}_{D,g} K D , g on D ^ g \widehat{D}_g D g by
K ^ D , g ( x , y ) = C g ( x , y ) + α D ω ^ g ( y ) C g ( x ) + α D , α D = 1. (34) \widehat{K}_{D,g}(x,y)=\frac{C_g(x,y)+\alpha_D\widehat{\omega}_g(y)}{C_g(x)+\alpha_D},\qquad\alpha_D=1. \tag*{(34)} K D , g ( x , y ) = C g ( x ) + α D C g ( x , y ) + α D ω g ( y ) , α D = 1. (34)
Each conditional row is smoothed toward the pooled empirical distribution using one total pseudocount. Its entries are nonnegative and sum to one. If C g ( x ) = 0 C_g(x)=0 C g ( x ) = 0 , the row equals ω ^ g \widehat{\omega}_g ω g ; this includes a value observed only as a successor. Queries require d − ( s ) ∈ D ^ g d^{-}(s)\in\widehat{D}_g d − ( s ) ∈ D g ; the implementation rejects an unseen conditioning value rather than inventing a row.
Supply model and persistent synthetic risk. Recall that B ( s ) B(s) B ( s ) is the availability coordinate. Let C g ↓ C_g^\downarrow C g ↓ count observed transitions starting in an outage and C g r e c C_g^{\mathrm{rec}} C g rec count those that recover:
C g ↓ = ∑ ( s , a , r , s ′ ) ∈ D ≤ g r e a l 1 { B ( s ) = 0 } , C g r e c = ∑ ( s , a , r , s ′ ) ∈ D ≤ g r e a l 1 { B ( s ) = 0 , B ( s ′ ) = 1 } . (35) \begin{aligned}
C_g^\downarrow&= \sum_{(s,a,r,s')\in\mathcal{D}^{\mathrm{real}}_{\leq g}} \mathbf{1}\{B(s)=0\},\\
C_g^{\mathrm{rec}} &= \sum_{(s,a,r,s')\in\mathcal{D}^{\mathrm{real}}_{\leq g}} \mathbf{1}\{B(s)=0,B(s')=1\}. \tag*{(35)}
\end{aligned} C g ↓ C g rec = ( s , a , r , s ′ ) ∈ D ≤ g real ∑ 1 { B ( s ) = 0 } , = ( s , a , r , s ′ ) ∈ D ≤ g real ∑ 1 { B ( s ) = 0 , B ( s ′ ) = 1 } . (35)
Let κ r e c ∈ ( 0 , 1 ) \kappa^{\mathrm{rec}}\in(0,1) κ rec ∈ ( 0 , 1 ) be the common recovery-probability parameter in the fitted supply family, and let κ ^ g r e c \widehat{\kappa}^{\mathrm{rec}}_g κ g rec be its estimate:
κ ^ g r e c = C g r e c + 1 C g ↓ + 2 . (36) \widehat{\kappa}^{\mathrm{rec}}_g=\frac{C_g^{\mathrm{rec}}+1}{C_g^\downarrow+2}. \tag*{(36)} κ g rec = C g ↓ + 2 C g rec + 1 . (36)
Under a Beta( 1 , 1 ) (1,1) ( 1 , 1 ) prior (the uniform prior on ( 0 , 1 ) (0,1) ( 0 , 1 ) ) for κ r e c \kappa^{\mathrm{rec}} κ rec , this is the posterior mean based on the outage-origin transitions. It remains defined when no outage has been observed, in which case κ ^ g r e c = 1 / 2 \widehat{\kappa}^{\mathrm{rec}}_g=1/2 κ g rec = 1/2 . Let m m m denote the synthetic-risk distribution. At the root of each new synthetic tape, the algorithm draws Z Z Z independently of the queried observation and of the other tapes:
Z ∼ m = Unif [ 0 , 0.5 ] . (37) Z\sim m=\operatorname{Unif}[0,0.5]. \tag*{(37)} Z ∼ m = Unif [ 0 , 0.5 ] . (37)
The tape retains this draw throughout its evolution. Conditional on Z = z Z=z Z = z , let K ^ B , g , z \widehat{K}_{B,g,z} K B , g , z be the modeled supply transition matrix, with rows and columns ordered as ( 0 , 1 ) (0,1) ( 0 , 1 ) :
K ^ B , g , z = ( 1 − κ ^ g r e c z 1 − z κ ^ g r e c z 1 − z κ ^ g r e c 1 − κ ^ g r e c ) . (38) \widehat{K}_{B,g,z}=
\begin{pmatrix}
1-\widehat{\kappa}^{\mathrm{rec}}_g\frac{z}{1-z} & \widehat{\kappa}^{\mathrm{rec}}_g\frac{z}{1-z}\\
\widehat{\kappa}^{\mathrm{rec}}_g & 1-\widehat{\kappa}^{\mathrm{rec}}_g
\end{pmatrix}. \tag*{(38)} K B , g , z = ( 1 − κ g rec 1 − z z κ g rec κ g rec 1 − z z 1 − κ g rec ) . (38)
Because 0 < κ ^ g r e c < 1 0<\widehat{\kappa}^{\mathrm{rec}}_g<1 0 < κ g rec < 1 and 0 ≤ z / ( 1 − z ) ≤ 1 0\leq z/(1-z)\leq1 0 ≤ z / ( 1 − z ) ≤ 1 , all matrix entries lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] and each row sums to one. The identity
( 1 − z ) K ^ B , g , z ( 1 , 0 ) = z K ^ B , g , z ( 0 , 1 ) (39) (1-z)\widehat{K}_{B,g,z}(1,0)=z\widehat{K}_{B,g,z}(0,1) \tag*{(39)} ( 1 − z ) K B , g , z ( 1 , 0 ) = z K B , g , z ( 0 , 1 ) (39)
shows that ( z , 1 − z ) (z,1-z) ( z , 1 − z ) is a stationary probability row vector, so the stationary outage probability is z z z , including the boundary z = 0 z=0 z = 0 . This parameterization is a modeling assumption; knowing only the risk interval would not imply it.
The combined model and generation-level freezing. Recall that r r r and F F F are the public reward and physical-transition maps in Equations (17) and (16) . For a query with d − ( s ) ∈ D ^ g d^{-}(s)\in\widehat{D}_g d − ( s ) ∈ D g and z ∈ [ 0 , 0.5 ] z\in[0,0.5] z ∈ [ 0 , 0.5 ] , define the conditional reward–observation kernel P ^ g , z \widehat{P}_{g,z} P g , z by
P ^ g , z ( d r , d s ′ ∣ s , a ) = ∑ d ∈ D ^ g ∑ u ∈ { 0 , 1 } K ^ D , g ( d − ( s ) , d ) K ^ B , g , z ( B ( s ) , u ) × δ ( r ( s , a , d ) , F ( s , a , d , u ) ) ( d r , d s ′ ) . (40) \begin{aligned}
\widehat{P}_{g,z}(dr,ds'\mid s,a)
&=\sum_{d\in\widehat{D}_g}\sum_{u\in\{0,1\}}
\widehat{K}_{D,g}(d^{-}(s),d)\widehat{K}_{B,g,z}(B(s),u)\\
&\qquad{}\times\delta_{(r(s,a,d),F(s,a,d,u))}(dr,ds'). \tag*{(40)}
\end{aligned} P g , z ( d r , d s ′ ∣ s , a ) = d ∈ D g ∑ u ∈ { 0 , 1 } ∑ K D , g ( d − ( s ) , d ) K B , g , z ( B ( s ) , u ) × δ ( r ( s , a , d ) , F ( s , a , d , u )) ( d r , d s ′ ) . (40)
Here δ x \delta_x δ x again denotes unit probability mass at x x x . We write P ^ g \widehat{P}_g P g for the full simulation specification: a query observation, an independent initial risk draw from m m m , and the conditional dynamics above with that risk retained. To state its Markov transition explicitly, let z ′ z' z ′ be the successor risk, let d z ′ dz' d z ′ indicate its measure coordinate, and define the augmented kernel P ~ g \widetilde{P}_g P g on observation–risk pairs by
P ~ g ( d r , d s ′ , d z ′ ∣ s , z , a ) = P ^ g , z ( d r , d s ′ ∣ s , a ) δ z ( d z ′ ) . (41) \widetilde{P}_g(dr,ds',dz'\mid s,z,a)=\widehat{P}_{g,z}(dr,ds'\mid s,a)\delta_z(dz'). \tag*{(41)} P g ( d r , d s ′ , d z ′ ∣ s , z , a ) = P g , z ( d r , d s ′ ∣ s , a ) δ z ( d z ′ ) . (41)
The model snapshot is frozen before generation-g g g model-based computations begin. For DSIS, P ^ g , z \widehat{P}_{g,z} P g , z reduces to the public nominal kernel independently of g g g and z z z ; no stochastic-law estimation or latent-risk simulation is required.
Summary of assumptions. Table 3 lists all structural and estimation assumptions; HBPO uses every applicable row. The last column reports explicit baseline use, separately from access to public information. In the table, p ^ g \hat{p}_g p ^ g denotes the conditional joint probability mass function of modeled demand d d d and next availability u u u , with its one-variable versions denoting the corresponding marginals. For a synthetic tape, k ∈ { 0 , 1 , … } k \in\{0,1,\ldots\} k ∈ { 0 , 1 , … } indexes simulated periods, S 0 S_0 S 0 is its query observation, and Z k Z_k Z k is its retained risk at period k k k ; ⊥ \perp ⊥ means independence. In rows marked Both, DSIS omits the unused previous-demand and risk arguments: its demand is uniform on { 0 , … , 4 } \{0,\ldots,4\} { 0 , … , 4 } and its next availability equals one.
Reading the baseline column. All refers to PPO, A2C, Discrete SAC, ACE, PPO-81, TRPO, GRPO, TPO, and GPS. Every method has access to the same public inventory rules and cost function, and to the complete nominal stochastic law in DSIS. The declared model-class assumptions are available to all comparison methods. An entry naming GPS identifies explicit use in its predictive model, rather than exclusive access to the corresponding information. The other baselines use their native learning rules on observed transitions; receiving a reward does not itself constitute evaluating the analytical reward map on a hypothetical transition. GPS and HBPO fit unknown RB probabilities to their respective counted data, so sharing an estimator does not imply sharing fitted parameters.
Information boundary. The functional forms in the table are specified in advance, rather than learned solely from data. In particular, the supply parameterization κ ^ g r e c z / ( 1 − z ) \widehat{\kappa}^{\mathrm{rec}}_g z/(1-z) κ g rec z / ( 1 − z ) does not follow from knowing the risk interval [ 0 , 0.5 ] [0,0.5] [ 0 , 0.5 ] alone: it also assumes a common recovery probability and a stationary outage share z z z . These assumptions do not give the learner the true RB recovery probability 0.25 0.25 0.25 , the true demand transition matrix, the actual trajectory risk ρ \rho ρ , or future random tapes. The RB demand support is discovered from observations, and unknown probabilities are estimated only from the learner’s own counted observations. The synthetic distribution of Z Z Z is neither the private benchmark risk-assignment law nor a posterior conditioned on the queried state. The DSIS fallback separately grants every method its complete public nominal law.
Use of structural priors and the information-fair comparison.
The priors in Table 3 enter HBPO through initialization, including fitted Q evaluation (FQE) on counterfactuals using observed disturbances; supply-conditioned replay; action-value backups and residual continuations; successor-critic state generation and fitting; and heuristic utility, occupancy, calibration, and confidence/interest calculations. These computations use simulated transitions and conditional expectations for learning, with enumeration over finite action and disturbance branches rather than the inventory state space. They provide no additional real-environment observations.
All methods share public information and a 2,000,000 real-interaction budget per formal learning run, with synthetic computation recorded separately. GPS uses the same model class, estimation rules, and tape-fixed uniform synthetic-risk prior as HBPO, fitted to its own observations. Neither learner receives unrestricted environment resets. Fairness therefore concerns information access and real interactions, not equal computation or identical fitted models.
The structural prior is part of HBPO. Comparison with model-informed GPS assesses the overall learning architecture beyond shared model access; it does not isolate an individual transmission component. Mechanism analyses and ablations examine guidance and transmission under common modeling and estimation rules, while their collected data and fitted probabilities may differ. Using models and inventory structure to support policy learning has established precedents ([36 , 22 , 4 ] ).
Critic Construction and Model-Based Residual Completion
The five scalar critics and the frozen value reference.
At generation g g g , HBPO maintains five separately parameterized scalar networks. For member index j ∈ { 1 , … , 5 } j \in\{1,\ldots,5\} j ∈ { 1 , … , 5 } , let θ g , j \theta_{g,j} θ g , j denote its trainable parameters and write
V g , j ( s ) = V θ g , j ( s ) . (42) V_{g,j}(s) = V_{\theta_{g,j}}(s). \tag*{(42)} V g , j ( s ) = V θ g , j ( s ) . (42)
The frozen ensemble reference is the mean of their scalar outputs,
v g ( s ) = 1 5 ∑ j = 1 5 V g , j ( s ) . (43) v_g(s) = \frac{1}{5}\sum_{j=1}^{5} V_{g,j}(s). \tag*{(43)} v g ( s ) = 5 1 j = 1 ∑ 5 V g , j ( s ) . (43)
The networks have no shared trainable layers.
Let d ϕ d_{\phi} d ϕ denote the feature dimension: d ϕ = 7 d_{\phi}=7 d ϕ = 7 in RB and d ϕ = 6 d_{\phi}=6 d ϕ = 6 in DSIS. Recall that I I I is inventory, p 1 , … , p 4 p_1,\ldots,p_4 p 1 , … , p 4 are pipeline coordinates, B B B is current availability, and d − d^{-} d − is previous demand. The fixed RB feature map, written as a column vector, is
ϕ ( s ) = ( asinh ( I / 4 ) , p 1 / 8 , p 2 / 8 , p 3 / 8 , p 4 / 8 , B , d − / 8 ) T ∈ R 7 , (44) \phi(s) = \left(\operatorname{asinh}(I/4),p_1/8,p_2/8,p_3/8,p_4/8,B,d^{-}/8\right)^{\mathsf{T}} \in\mathbb{R}^{7}, \tag*{(44)} ϕ ( s ) = ( asinh ( I /4 ) , p 1 /8 , p 2 /8 , p 3 /8 , p 4 /8 , B , d − /8 ) T ∈ R 7 , (44)
where asinh is the inverse hyperbolic sine. DSIS omits the last coordinate.
Each network has two Tanh hidden layers of width 128 and a scalar linear output. For a generic parameter vector θ = ( W 1 , b 1 , W 2 , b 2 , w 3 , b 3 ) \theta=(W_1,b_1,W_2,b_2,w_3,b_3) θ = ( W 1 , b 1 , W 2 , b 2 , w 3 , b 3 ) , the dimensions are W 1 ∈ R 128 × d ϕ W_1\in\mathbb{R}^{128\times d_{\phi}} W 1 ∈ R 128 × d ϕ , W 2 ∈ R 128 × 128 W_2\in\mathbb{R}^{128\times128} W 2 ∈ R 128 × 128 , b 1 , b 2 , w 3 ∈ R 128 b_1,b_2,w_3\in\mathbb{R}^{128} b 1 , b 2 , w 3 ∈ R 128 , and b 3 ∈ R b_3\in\mathbb{R} b 3 ∈ R ; tanh acts coordinatewise. The resulting scalar function is
V θ ( s ) = w 3 T tanh ( W 2 tanh ( W 1 ϕ ( s ) + b 1 ) + b 2 ) + b 3 . (45) V_{\theta}(s) = w_3^{\mathsf{T}}\tanh\left(W_2\tanh\left(W_1\phi(s)+b_1\right)+b_2\right)+b_3. \tag*{(45)} V θ ( s ) = w 3 T tanh ( W 2 tanh ( W 1 ϕ ( s ) + b 1 ) + b 2 ) + b 3 . (45)
During generation g g g , v g v_g v g is a frozen continuation-value reference. It need not equal the exact value under either the current model or the environment. For g ≥ 1 g\geq1 g ≥ 1 , its parameters are inherited from the preceding generation even when the newly fitted model has changed; v 0 v_0 v 0 is supplied by the initialization procedure described below.
One-step action backups and the target-policy residual.
Recall that P ^ g , z \widehat{P}_{g,z} P g , z is the frozen model conditional on synthetic risk z z z , D ^ g \widehat{D}_g D g is its finite demand support, and K ^ D , g \widehat{K}_{D,g} K D , g and K ^ B , g , z \widehat{K}_{B,g,z} K B , g , z are its demand and availability kernels. Also recall the deterministic successor map F F F , normalized reward r r r , and discount γ = 0.995 \gamma=0.995 γ = 0.995 . Here d − ( s ) d^{-}(s) d − ( s ) and B ( s ) B(s) B ( s ) denote the previous-demand and current-availability coordinates of s s s . For a real-valued function V V V with finite backups and a prescribed next availability u ∈ { 0 , 1 } u\in\{0,1\} u ∈ { 0 , 1 } , define the supply-conditioned action backup by integrating demand:
L g u V ( s , a ) = ∑ d ∈ D ^ g K ^ D , g ( d − ( s ) , d ) [ r ( s , a , d ) + γ V ( F ( s , a , d , u ) ) ] . (46) L_g^{u}V(s,a) = \sum_{d\in\widehat{D}_g}\widehat{K}_{D,g}\left(d^{-}(s),d\right)\left[r(s,a,d)+\gamma V\left(F(s,a,d,u)\right)\right]. \tag*{(46)} L g u V ( s , a ) = d ∈ D g ∑ K D , g ( d − ( s ) , d ) [ r ( s , a , d ) + γ V ( F ( s , a , d , u ) ) ] . (46)
Integrating next availability conditional on the fixed tape risk z z z gives
L g , z V ( s , a ) = ∑ u ∈ { 0 , 1 } K ^ B , g , z ( B ( s ) , u ) L g u V ( s , a ) . (47) L_{g,z}V(s,a) = \sum_{u\in\{0,1\}}\widehat{K}_{B,g,z}\left(B(s),u\right)L_g^{u}V(s,a). \tag*{(47)} L g , z V ( s , a ) = u ∈ { 0 , 1 } ∑ K B , g , z ( B ( s ) , u ) L g u V ( s , a ) . (47)
Let R R R and S ′ S^{\prime} S ′ denote the reward and successor from one transition of this model. Then
L g , z V ( s , a ) = E P ^ g , z [ R + γ V ( S ′ ) ∣ s , a ] . (48) L_{g,z}V(s,a) = \mathbb{E}_{\widehat{\mathbb{P}}_{g,z}}\left[R+\gamma V(S^{\prime})\mid s,a\right]. \tag*{(48)} L g , z V ( s , a ) = E P g , z [ R + γ V ( S ′ ) ∣ s , a ] . (48)
The frozen parent target policy π g \pi_g π g induces the discounted Bellman operator T g , z π g \mathcal{T}_{g,z}^{\pi_g} T g , z π g ,
T g , z π g V ( s ) = ∑ a ∈ A π g ( a ∣ s ) L g , z V ( s , a ) . (49) \mathcal{T}_{g,z}^{\pi_g}V(s) = \sum_{a\in\mathcal{A}}\pi_g(a\mid s)L_{g,z}V(s,a). \tag*{(49)} T g , z π g V ( s ) = a ∈ A ∑ π g ( a ∣ s ) L g , z V ( s , a ) . (49)
Its residual at the frozen reference v g v_g v g is
e g , z ( s ) = ( T g , z π g v g ) ( s ) − v g ( s ) = ∑ a ′ ∈ A π g ( a ′ ∣ s ) ∑ d ∈ D g ∑ u ∈ { 0 , 1 } K ^ D , g ( d − ( s ) , d ) K ^ B , g , z ( B ( s ) , u ) [ r ( s , a ′ , d ) + γ v g ( F ( s , a ′ , d , u ) ) − v g ( s ) ] . (50) e_{g,z}(s)=\left(\mathcal{T}_{g,z}^{\pi_g}v_g\right)(s)-v_g(s)
= \sum_{a'\in\mathcal{A}}\pi_g(a'\mid s)\sum_{d\in\mathcal{D}_g}\sum_{u\in\{0,1\}}\widehat{K}_{D,g}(d^{-}(s),d)\widehat{K}_{B,g,z}(B(s),u)
\left[r(s,a',d)+\gamma v_g(F(s,a',d,u))-v_g(s)\right]. \tag*{(50)} e g , z ( s ) = ( T g , z π g v g ) ( s ) − v g ( s ) = a ′ ∈ A ∑ π g ( a ′ ∣ s ) d ∈ D g ∑ u ∈ { 0 , 1 } ∑ K D , g ( d − ( s ) , d ) K B , g , z ( B ( s ) , u ) [ r ( s , a ′ , d ) + γ v g ( F ( s , a ′ , d , u )) − v g ( s ) ] . (50)
This residual integrates the target action, demand, and next availability conditional on z z z . It is the conditional mean of a temporal-difference error, rather than a sampled reward increment. Its sign indicates whether the model’s one-step evaluation exceeds or falls below the reference value; it does not indicate that the policy itself has improved. Theorem 1 shows why propagating this signed inconsistency recovers the remaining continuation-value error.
Recall the synthetic-risk prior m = Unif [ 0 , 1 / 2 ] m=\operatorname{Unif}[0,1/2] m = Unif [ 0 , 1/2 ] in RB. The risk-averaged one-step base is
Q g 0 ( s , a ) = E Z ∼ m [ L g , Z v g ( s , a ) ] . (51) Q_g^{0}(s,a)=\mathbb{E}_{Z\sim m}[L_{g,Z}v_g(s,a)]. \tag*{(51)} Q g 0 ( s , a ) = E Z ∼ m [ L g , Z v g ( s , a )] . (51)
Let κ ^ g r e c \widehat{\kappa}_g^{\mathrm{rec}} κ g rec denote the fitted recovery probability, as in the supply model above. Since v g v_g v g has no risk input, the RB root backup uses the analytic mean supply kernel
E Z ∼ m [ K ^ B , g , Z ( B , 0 ) ] = { 1 − κ ^ g r e c , B = 0 , κ ^ g r e c ( 2 log 2 − 1 ) , B = 1 , (52) \mathbb{E}_{Z\sim m}\left[\widehat{K}_{B,g,Z}(B,0)\right]=
\begin{cases}
1-\widehat{\kappa}_g^{\mathrm{rec}}, & B=0,\\
\widehat{\kappa}_g^{\mathrm{rec}}(2\log2-1), & B=1,
\end{cases} \tag*{(52)} E Z ∼ m [ K B , g , Z ( B , 0 ) ] = { 1 − κ g rec , κ g rec ( 2 log 2 − 1 ) , B = 0 , B = 1 , (52)
where log \log log is the natural logarithm and E Z ∼ m [ Z / ( 1 − Z ) ] = 2 ∫ 0 1 / 2 z / ( 1 − z ) d z = 2 log 2 − 1 \mathbb{E}_{Z\sim m}[Z/(1-Z)]=2\int_{0}^{1/2}z/(1-z)\,\mathrm{d}z=2\log2-1 E Z ∼ m [ Z / ( 1 − Z )] = 2 ∫ 0 1/2 z / ( 1 − z ) d z = 2 log 2 − 1 . Only the one-step root backup is averaged in this way. Multistep simulation retains the sampled z z z ; replacing its kernel at every step by the prior mean would in general define a different process.
Residual completion of the discounted action value.
Recall that m m m is the synthetic-risk law and that P ^ g , z \widehat{P}_{g,z} P g , z is the frozen reward–observation kernel at generation g g g (Equation (40) ). Let S k S_k S k , A k A_k A k , and R k R_k R k denote the observation, action, and normalized reward at simulated step k k k . Define the model action-value function queried by HBPO as
Q g π g ( s , a ) = E Z ∼ m E P ^ g , Z , π g [ ∑ k = 0 ∞ γ k R k | S 0 = s , A 0 = a ] . (53) Q_g^{\pi_g}(s,a)=\mathbb{E}_{Z\sim m}\mathbb{E}_{\widehat{P}_{g,Z},\pi_g}\left[\sum_{k=0}^{\infty}\gamma^kR_k\ \middle|\ S_0=s,A_0=a\right]. \tag*{(53)} Q g π g ( s , a ) = E Z ∼ m E P g , Z , π g [ k = 0 ∑ ∞ γ k R k S 0 = s , A 0 = a ] . (53)
where the first action is fixed to a a a , and A k ∼ π g ( ⋅ ∣ S k ) A_k\sim\pi_g(\mathord{\cdot}\mid S_k) A k ∼ π g ( ⋅ ∣ S k ) for k ≥ 1 k\geq1 k ≥ 1 . The risk is sampled independently at the query root and remains fixed thereafter. This defines a model action-value function under the declared risk prior; z z z is a simulated scenario, not an estimate of the queried trajectory’s true risk ρ \rho ρ .
For the following derivation, fix generation g g g and write E z s , a \mathbb{E}_{z}^{s,a} E z s , a for expectation under the rollout law with fixed risk z z z , initial observation s s s , and forced first action a a a . Write E z s \mathbb{E}_{z}^{s} E z s when the initial action also follows π g \pi_g π g . Define the exact conditional model value V g , z π g V_{g,z}^{\pi_g} V g , z π g and its error Δ g , z \Delta_{g,z} Δ g , z relative to v g v_g v g by
V g , z π g ( s ) = E z s [ ∑ k = 0 ∞ γ k R k ] , Δ g , z ( s ) = V g , z π g ( s ) − v g ( s ) . (54) V_{g,z}^{\pi_g}(s)=\mathbb{E}_{z}^{s}\left[\sum_{k=0}^{\infty}\gamma^kR_k\right],
\qquad
\Delta_{g,z}(s)=V_{g,z}^{\pi_g}(s)-v_g(s). \tag*{(54)} V g , z π g ( s ) = E z s [ k = 0 ∑ ∞ γ k R k ] , Δ g , z ( s ) = V g , z π g ( s ) − v g ( s ) . (54)
Here V g , z π g V_{g,z}^{\pi_g} V g , z π g is the exact conditional model value , whereas v g v_g v g is the frozen network reference. For a scalar function f f f , let the model’s one-step state-transition operators be P ^ g , z a \widehat{P}_{g,z}^{a} P g , z a (forced first action a a a ) and P ^ g , z π g \widehat{P}_{g,z}^{\pi_g} P g , z π g (first action sampled from π g \pi_g π g ):
( P ^ g , z a f ) ( s ) = E z s , a [ f ( S 1 ) ] , ( P ^ g , z π g f ) ( s ) = ∑ a ′ ∈ A π g ( a ′ ∣ s ) ( P ^ g , z a ′ f ) ( s ) . (55) (\widehat{P}_{g,z}^{a}f)(s)=\mathbb{E}_{z}^{s,a}[f(S_1)],
\qquad
(\widehat{P}_{g,z}^{\pi_g}f)(s)=\sum_{a'\in\mathcal{A}}\pi_g(a'\mid s)(\widehat{P}_{g,z}^{a'}f)(s). \tag*{(55)} ( P g , z a f ) ( s ) = E z s , a [ f ( S 1 )] , ( P g , z π g f ) ( s ) = a ′ ∈ A ∑ π g ( a ′ ∣ s ) ( P g , z a ′ f ) ( s ) . (55)
Theorem 1 (Residual completion and the omitted continuation error). Let S \mathcal{S} S be the countable observation space of the frozen inventory model, closed under its transitions. Fix the model, the observation-based policy π g \pi_g π g , a discount 0 < γ < 1 0<\gamma<1 0 < γ < 1 , and a reference v g v_g v g satisfying ∣ v g ( s ) ∣ ≤ C v < ∞ \lvert v_g(s)\rvert\leq C_v<\infty ∣ v g ( s )∣ ≤ C v < ∞ for all s ∈ S s\in\mathcal{S} s ∈ S , where C v C_v C v is a constant. Assume that, for every s ∈ S s\in\mathcal{S} s ∈ S and a ∈ A a\in\mathcal{A} a ∈ A ,
∫ E z s , a [ ∑ k = 0 ∞ γ k ∣ R k ∣ ] m ( d z ) < ∞ . (56) \int\mathbb{E}_{z}^{s,a}\left[\sum_{k=0}^{\infty}\gamma^k\lvert R_k\rvert\right]m(\mathrm{d}z)<\infty. \tag*{(56)} ∫ E z s , a [ k = 0 ∑ ∞ γ k ∣ R k ∣ ] m ( d z ) < ∞. (56)
The action set is finite, so this also implies the corresponding condition when the first action follows π g \pi_g π g . Recall that T g , z π g \mathcal{T}^{\pi_g}_{g,z} T g , z π g is the Bellman operator in Equation (49) , and that e g , z = T g , z π g v g − v g e_{g,z}=\mathcal{T}^{\pi_g}_{g,z}v_g-v_g e g , z = T g , z π g v g − v g . Write I d \mathrm{Id} Id for the identity operator on scalar functions. For an integer ℓ ≥ 0 \ell\ge0 ℓ ≥ 0 , ( P ^ g , z π g ) ℓ (\widehat{P}^{\pi_g}_{g,z})^\ell ( P g , z π g ) ℓ denotes ℓ \ell ℓ -fold operator composition, with zeroth power I d \mathrm{Id} Id . Then the Bellman residual and conditional value error satisfy
e g , z = ( I d − γ P ^ g , z π g ) Δ g , z , Δ g , z = ∑ ℓ = 0 ∞ γ ℓ ( P ^ g , z π g ) ℓ e g , z . (57) e_{g,z}=(\mathrm{Id}-\gamma\widehat{P}^{\pi_g}_{g,z})\Delta_{g,z},\qquad\Delta_{g,z}=\sum_{\ell=0}^{\infty}\gamma^\ell(\widehat{P}^{\pi_g}_{g,z})^\ell e_{g,z}. \tag*{(57)} e g , z = ( Id − γ P g , z π g ) Δ g , z , Δ g , z = ℓ = 0 ∑ ∞ γ ℓ ( P g , z π g ) ℓ e g , z . (57)
These identities hold at every s ∈ S s\in\mathcal{S} s ∈ S on a common set of risks of full m m m -measure. The series converges absolutely pointwise; no bounded-value-space inverse is assumed. In particular, the exact error of the one-step base is
Q g π g ( s , a ) − Q g 0 ( s , a ) = γ E Z ∼ m [ ( P ^ g , Z a Δ g , Z ) ( s ) ] . (58) Q^{\pi_g}_g(s,a)-Q^0_g(s,a)=\gamma\mathbb{E}_{Z\sim m}[(\widehat{P}^{a}_{g,Z}\Delta_{g,Z})(s)]. \tag*{(58)} Q g π g ( s , a ) − Q g 0 ( s , a ) = γ E Z ∼ m [( P g , Z a Δ g , Z ) ( s )] . (58)
It is recovered by the discounted residual sum:
Q g π g ( s , a ) = Q g 0 ( s , a ) + ∫ E z s , a [ ∑ k = 1 ∞ γ k e g , z ( S k ) ] m ( d z ) . (59) Q^{\pi_g}_g(s,a)=Q^0_g(s,a)+\int\mathbb{E}^{s,a}_z\left[\sum_{k=1}^{\infty}\gamma^k e_{g,z}(S_k)\right]m(dz). \tag*{(59)} Q g π g ( s , a ) = Q g 0 ( s , a ) + ∫ E z s , a [ k = 1 ∑ ∞ γ k e g , z ( S k ) ] m ( d z ) . (59)
Proof. Because S \mathcal{S} S is countable and A \mathcal{A} A is finite, Equation (56) provides a common set of risks of full m m m -measure on which absolute discounted reward integrability holds for every initial state and action. Fix a risk z z z in this set. Conditional on z z z , the observation process is Markov. Bellman consistency of the exact conditional value gives
Δ g , z = e g , z + γ P ^ g , z π g Δ g , z . (60) \Delta_{g,z}=e_{g,z}+\gamma\widehat{P}^{\pi_g}_{g,z}\Delta_{g,z}. \tag*{(60)} Δ g , z = e g , z + γ P g , z π g Δ g , z . (60)
For any integer n t r ≥ 0 n_{\mathrm{tr}}\ge0 n tr ≥ 0 , iterating this relation yields
Δ g , z ( s ) = ∑ ℓ = 0 n t r γ ℓ ( ( P ^ g , z π g ) ℓ e g , z ) ( s ) + γ n t r + 1 E z s [ Δ g , z ( S n t r + 1 ) ] . (61) \Delta_{g,z}(s)=\sum_{\ell=0}^{n_{\mathrm{tr}}}\gamma^\ell\left((\widehat{P}^{\pi_g}_{g,z})^\ell e_{g,z}\right)(s)+\gamma^{n_{\mathrm{tr}}+1}\mathbb{E}^{s}_z[\Delta_{g,z}(S_{n_{\mathrm{tr}}+1})]. \tag*{(61)} Δ g , z ( s ) = ℓ = 0 ∑ n tr γ ℓ ( ( P g , z π g ) ℓ e g , z ) ( s ) + γ n tr + 1 E z s [ Δ g , z ( S n tr + 1 )] . (61)
The absolute terminal term is bounded by
γ n t r + 1 E z s [ ∣ Δ g , z ( S n t r + 1 ) ∣ ] ≤ E z s [ ∑ k = n t r + 1 ∞ γ k ∣ R k ∣ ] + γ n t r + 1 C v ⟶ 0. (62) \gamma^{n_{\mathrm{tr}}+1}\mathbb{E}^{s}_z[|\Delta_{g,z}(S_{n_{\mathrm{tr}}+1})|]\le\mathbb{E}^{s}_z\left[\sum_{k=n_{\mathrm{tr}}+1}^{\infty}\gamma^k|R_k|\right]+\gamma^{n_{\mathrm{tr}}+1}C_v\longrightarrow0. \tag*{(62)} γ n tr + 1 E z s [ ∣ Δ g , z ( S n tr + 1 ) ∣ ] ≤ E z s [ k = n tr + 1 ∑ ∞ γ k ∣ R k ∣ ] + γ n tr + 1 C v ⟶ 0. (62)
Here the reward-tail bound follows from the Markov property and conditional Jensen’s inequality. Absolute integrability makes its expectation tend to zero. Moreover, conditional Jensen’s inequality applied to the one-step residual gives
∑ ℓ = 0 ∞ γ ℓ E z s [ ∣ e g , z ( S ℓ ) ∣ ] ≤ E z s [ ∑ ℓ = 0 ∞ γ ℓ ∣ R ℓ ∣ ] + ( 1 + γ ) C v 1 − γ < ∞ . (63) \sum_{\ell=0}^{\infty}\gamma^\ell\mathbb{E}^{s}_z[|e_{g,z}(S_\ell)|]\le\mathbb{E}^{s}_z\left[\sum_{\ell=0}^{\infty}\gamma^\ell|R_\ell|\right]+\frac{(1+\gamma)C_v}{1-\gamma}<\infty. \tag*{(63)} ℓ = 0 ∑ ∞ γ ℓ E z s [ ∣ e g , z ( S ℓ ) ∣ ] ≤ E z s [ ℓ = 0 ∑ ∞ γ ℓ ∣ R ℓ ∣ ] + 1 − γ ( 1 + γ ) C v < ∞. (63)
This proves both identities in Equation (57) , including absolute convergence of its series. The same bounds can be integrated against m ( d z ) m(dz) m ( d z ) by the stated mixture integrability assumption.
Conditioning the discounted return on its first transition yields
Q g π g ( s , a ) = ∫ { L g , z v g ( s , a ) + γ ( P ^ g , z a Δ g , z ) ( s ) } m ( d z ) . (64) Q^{\pi_g}_g(s,a)=\int\left\{L_{g,z}v_g(s,a)+\gamma(\widehat{P}^{a}_{g,z}\Delta_{g,z})(s)\right\}m(dz). \tag*{(64)} Q g π g ( s , a ) = ∫ { L g , z v g ( s , a ) + γ ( P g , z a Δ g , z ) ( s ) } m ( d z ) . (64)
Subtracting Q g 0 ( s , a ) = ∫ L g , z v g ( s , a ) m ( d z ) Q^0_g(s,a)=\int L_{g,z}v_g(s,a)m(dz) Q g 0 ( s , a ) = ∫ L g , z v g ( s , a ) m ( d z ) proves Equation (58) .
For the residual-completion identity, at every continuation step k ≥ 1 k\ge1 k ≥ 1 the tower property gives
E z s , a [ e g , z ( S k ) ] = E z s , a [ R k + γ v g ( S k + 1 ) − v g ( S k ) ] . (65) \mathbb{E}^{s,a}_z[e_{g,z}(S_k)]=\mathbb{E}^{s,a}_z[R_k+\gamma v_g(S_{k+1})-v_g(S_k)]. \tag*{(65)} E z s , a [ e g , z ( S k )] = E z s , a [ R k + γ v g ( S k + 1 ) − v g ( S k )] . (65)
The restriction k ≥ 1 k\ge1 k ≥ 1 ensures that the action follows π g \pi_g π g ; the forced root action is handled by L g , z v g ( s , a ) L_{g,z}v_g(s,a) L g , z v g ( s , a ) . Multiplying by γ k \gamma^k γ k and summing gives, for each integer n t r ≥ 0 n_{\mathrm{tr}}\ge0 n tr ≥ 0 ,
L g , z v g ( s , a ) + E z s , a [ ∑ k = 1 n t r γ k e g , z ( S k ) ] = E z s , a [ ∑ k = 0 n t r γ k R k + γ n t r + 1 v g ( S n t r + 1 ) ] . (66) L_{g,z}v_g(s,a)+\mathbb{E}^{s,a}_z\left[\sum_{k=1}^{n_{\mathrm{tr}}}\gamma^k e_{g,z}(S_k)\right]=\mathbb{E}^{s,a}_z\left[\sum_{k=0}^{n_{\mathrm{tr}}}\gamma^kR_k+\gamma^{n_{\mathrm{tr}}+1}v_g(S_{n_{\mathrm{tr}}+1})\right]. \tag*{(66)} L g , z v g ( s , a ) + E z s , a [ k = 1 ∑ n tr γ k e g , z ( S k ) ] = E z s , a [ k = 0 ∑ n tr γ k R k + γ n tr + 1 v g ( S n tr + 1 ) ] . (66)
The sum is empty when n t r = 0 n_{\mathrm{tr}}=0 n tr = 0 . After integration over m ( d z ) m(\mathrm{d}z) m ( d z ) , the remaining reference-value term satisfies
γ n t r + 1 ∫ E z s , a [ ∣ v g ( S n t r + 1 ) ∣ ] m ( d z ) ≤ γ n t r + 1 C v ⟶ 0. (67) \gamma^{n_{\mathrm{tr}}+1}\int\mathbb{E}^{s,a}_{z}\left[\lvert v_g(S_{n_{\mathrm{tr}}+1})\rvert\right]m(\mathrm{d}z)\leq\gamma^{n_{\mathrm{tr}}+1}C_v\longrightarrow0. \tag*{(67)} γ n tr + 1 ∫ E z s , a [ ∣ v g ( S n tr + 1 )∣ ] m ( d z ) ≤ γ n tr + 1 C v ⟶ 0. (67)
For the forced-action rollout, the analogue of Equation (63) starts at k = 1 k=1 k = 1 ; its right-hand side remains integrable over the root risk. Consequently the infinite residual sum and its expectation can be interchanged, and taking n t r → ∞ n_{\mathrm{tr}}\to\infty n tr → ∞ proves Equation (59) . □
The theorem identifies the correction exactly: it is the part of future policy value missing from the one-step bootstrap. The factor γ k \gamma^k γ k is imposed by the original discounted return and the telescoping cancellation; it is not a separately chosen weight for risk or optimism. At a finite deterministic continuation length n t r n_{\mathrm{tr}} n tr , the remaining error is exactly
Q g π g ( s , a ) − { Q g 0 ( s , a ) + ∫ E z s , a [ ∑ k = 1 n t r γ k e g , z ( S k ) ] m ( d z ) } = γ n t r + 1 ∫ E z s , a [ Δ g , z ( S n t r + 1 ) ] m ( d z ) . (68) Q_g^{\pi_g}(s,a)-\left\{Q_g^0(s,a)+\int\mathbb{E}^{s,a}_{z}\left[\sum_{k=1}^{n_{\mathrm{tr}}}\gamma^k e_{g,z}(S_k)\right]m(\mathrm{d}z)\right\}=\gamma^{n_{\mathrm{tr}}+1}\int\mathbb{E}^{s,a}_{z}\left[\Delta_{g,z}(S_{n_{\mathrm{tr}}+1})\right]m(\mathrm{d}z). \tag*{(68)} Q g π g ( s , a ) − { Q g 0 ( s , a ) + ∫ E z s , a [ k = 1 ∑ n tr γ k e g , z ( S k ) ] m ( d z ) } = γ n tr + 1 ∫ E z s , a [ Δ g , z ( S n tr + 1 ) ] m ( d z ) . (68)
Thus a fixed truncation retains a terminal evaluation error. The geometrically stopped estimator below implements the complete series with an almost surely finite random continuation.
Persistent risk and an observation-only critic. RB introduces a further reason for conditional evaluation error: v g ( s ) v_g(s) v g ( s ) has no z z z input, whereas V g , z π g ( s ) V_{g,z}^{\pi_g}(s) V g , z π g ( s ) can depend on the persistent risk. At any fixed state s s s , assume E Z ∼ m [ ( V g , Z π g ( s ) ) 2 ] < ∞ \mathbb{E}_{Z\sim m}[(V_{g,Z}^{\pi_g}(s))^2]<\infty E Z ∼ m [( V g , Z π g ( s ) ) 2 ] < ∞ . Writing c v a l ∈ R c_{\mathrm{val}}\in\mathbb{R} c val ∈ R for a scalar prediction and Var Z ∼ m \operatorname{Var}_{Z\sim m} Var Z ∼ m for variance over the root-risk law,
inf c v a l ∈ R E Z ∼ m [ ( V g , Z π g ( s ) − c v a l ) 2 ] = Var Z ∼ m ( V g , Z π g ( s ) ) . (69) \inf_{c_{\mathrm{val}}\in\mathbb{R}}\mathbb{E}_{Z\sim m}\left[(V_{g,Z}^{\pi_g}(s)-c_{\mathrm{val}})^2\right]=\operatorname{Var}_{Z\sim m}(V_{g,Z}^{\pi_g}(s)). \tag*{(69)} c val ∈ R inf E Z ∼ m [ ( V g , Z π g ( s ) − c val ) 2 ] = Var Z ∼ m ( V g , Z π g ( s )) . (69)
Indeed, the expected squared error equals this variance plus ( c v a l − E Z ∼ m [ V g , Z π g ( s ) ] ) 2 (c_{\mathrm{val}}-\mathbb{E}_{Z\sim m}[V_{g,Z}^{\pi_g}(s)])^2 ( c val − E Z ∼ m [ V g , Z π g ( s )] ) 2 . When the conditional values differ, even an ideal scalar reference cannot represent all of them simultaneously. Moreover, setting V ‾ g ( s ) = E Z ∼ m [ V g , Z π g ( s ) ] \overline{V}_g(s)=\mathbb{E}_{Z\sim m}[V_{g,Z}^{\pi_g}(s)] V g ( s ) = E Z ∼ m [ V g , Z π g ( s )] does not generally make E Z ∼ m [ P g , Z π g ( V g , Z π g − V ‾ g ) ] ( s ) \mathbb{E}_{Z\sim m}[\mathcal{P}_{g,Z}^{\pi_g}(V_{g,Z}^{\pi_g}-\overline{V}_g)](s) E Z ∼ m [ P g , Z π g ( V g , Z π g − V g )] ( s ) zero: the same z z z determines both the successor distribution and the continuation value. This is why HBPO conditions both the residual and the continuation on one persistent z z z , then averages their completed action values. Averaging the transition kernel anew at every step would describe a different stochastic process.
The theorem’s assumptions hold for the implemented inventory models. Let d max d_{\max} d m a x be the largest demand in the frozen model’s finite support, and let ( I , p ) (I,p) ( I , p ) be the initial inventory and pipeline. Since each accepted regular order and each expedited order is at most 8, for every simulated step k ≥ 0 k\geq0 k ≥ 0 ,
∣ I k ∣ ≤ ∣ I ∣ + ∑ ℓ = 1 4 p ℓ + ( 16 + d max ) k , ∣ R k ∣ ≤ 80 495 + ∣ I k + 1 ∣ . (70) \begin{aligned}
\lvert I_k\rvert&\leq\lvert I\rvert+\sum_{\ell=1}^{4}p_\ell+(16+d_{\max})k,\\
\lvert R_k\rvert&\leq\frac{80}{495}+\lvert I_{k+1}\rvert. \tag*{(70)}
\end{aligned} ∣ I k ∣ ∣ R k ∣ ≤ ∣ I ∣ + ℓ = 1 ∑ 4 p ℓ + ( 16 + d m a x ) k , ≤ 495 80 + ∣ I k + 1 ∣ . (70)
These bounds are uniform in z z z ; geometric discounting makes their sums finite. A frozen Tanh network with finite parameters is bounded by ∑ i ∣ w 3 , i ∣ + ∣ b 3 ∣ \sum_i\lvert w_{3,i}\rvert+\lvert b_3\rvert ∑ i ∣ w 3 , i ∣ + ∣ b 3 ∣ , with w 3 , b 3 w_3,b_3 w 3 , b 3 as in Equation (45) . Thus the theorem does not require rewards or exact values to be globally bounded across initial states.
Interpretation and scope of the guarantee. The theorem establishes the correction needed to make the one-step reference exact under the frozen model. It does not make residual simulation the only possible policy-evaluation method. Its telescoping basis is closely related to multi-step temporal-difference decompositions, including generalized advantage estimation with its trace parameter set to one [33 ] ; here the increments are model-integrated residuals and the RB latent persists throughout each continuation. The reference critic can therefore be used as a control variate without changing the population action-value target. Finite simulated tapes still introduce Monte Carlo error, and the identity alone does not guarantee a variance reduction. Model misspecification and mismatch between the simulation risk prior and the benchmark remain separate sources of error. Claims about real-environment return improvements consequently rest on the experiments, rather than on this model-evaluation identity alone.
Finite computation through geometrically stopped continuations.
We estimate the infinite residual series using a random, almost surely finite continuation length. For a fixed query ( s , a ) (s,a) ( s , a ) , independently of the entire model path and of its risk Z Z Z , draw
N ∈ { 0 , 1 , … } , Pr ( N = ℓ ) = ( 1 − γ ) γ ℓ , ℓ = 0 , 1 , … . (71) N \in\{0,1,\ldots\}, \qquad\Pr(N=\ell)=(1-\gamma)\gamma^\ell,\qquad\ell=0,1,\ldots. \tag*{(71)} N ∈ { 0 , 1 , … } , Pr ( N = ℓ ) = ( 1 − γ ) γ ℓ , ℓ = 0 , 1 , … . (71)
For any z z z in the common full-m m m -measure set of Theorem 1, all path expectations below use E z s , a \mathbb{E}^{s,a}_z E z s , a , and the outer expectation also averages over N N N . Since Pr ( N ≥ k ) = γ k \Pr(N\ge k)=\gamma^k Pr ( N ≥ k ) = γ k for integers k ≥ 0 k\ge0 k ≥ 0 , absolute residual integrability in Theorem 1 justifies
E [ ∑ k = 1 N e g , z ( S k ) ] = ∑ k = 1 ∞ Pr ( N ≥ k ) E [ e g , z ( S k ) ] = E [ ∑ k = 1 ∞ γ k e g , z ( S k ) ] . (72) \mathbb{E}\left[\sum_{k=1}^{N}e_{g,z}(S_k)\right]
=\sum_{k=1}^{\infty}\Pr(N\ge k)\mathbb{E}[e_{g,z}(S_k)]
=\mathbb{E}\left[\sum_{k=1}^{\infty}\gamma^k e_{g,z}(S_k)\right]. \tag*{(72)} E [ k = 1 ∑ N e g , z ( S k ) ] = k = 1 ∑ ∞ Pr ( N ≥ k ) E [ e g , z ( S k )] = E [ k = 1 ∑ ∞ γ k e g , z ( S k ) ] . (72)
Let M = 32 M=32 M = 32 be the number of tapes per sampled query root, and let b = 1 , … , M b=1,\ldots,M b = 1 , … , M index the tapes. Tape b b b draws an independent Z b ∼ m Z_b\sim m Z b ∼ m and a length N b N_b N b with the law above. Write S b , k a S^a_{b,k} S b , k a for its step-k k k observation after forcing first action a a a and following π g \pi_g π g thereafter. An ideal action-value estimator is
Q ^ g ( s , a ) = Q g o ( s , a ) + 1 M ∑ b = 1 M ∑ k = 1 N b e g , Z b ( S b , k a ) , M = 32. (73) \widehat{Q}_g(s,a)=Q_g^o(s,a)+\frac{1}{M}\sum_{b=1}^{M}\sum_{k=1}^{N_b}e_{g,Z_b}(S^a_{b,k}),\qquad M=32. \tag*{(73)} Q g ( s , a ) = Q g o ( s , a ) + M 1 b = 1 ∑ M k = 1 ∑ N b e g , Z b ( S b , k a ) , M = 32. (73)
Each tape samples its own Z b Z_b Z b , which is shared across the 81 alternative first-action branches and held fixed throughout their continuations. The branches also share the addressed random numbers used for geometric stopping, exogenous disturbances, and subsequent policy-action sampling. These shared numbers couple action branches without changing any one branch’s marginal rollout law.
Geometric survival already supplies the discount, so the accumulated residuals receive no additional factor γ k \gamma^k γ k . The N b = 0 N_b=0 N b = 0 sum is empty. These continuations have no fixed 1000-step cutoff or terminal bootstrap. In RB, the first modeled transition also uses P ^ g , Z b \widehat{P}_{g,Z_b} P g , Z b ; it is not conditioned on the real transition’s next availability.
Under the stated sampling laws and exact arithmetic, E [ Q ^ g ( s , a ) ] = Q g π g ( s , a ) \mathbb{E}[\widehat{Q}_g(s,a)]=Q_g^{\pi_g}(s,a) E [ Q g ( s , a )] = Q g π g ( s , a ) . A finite collection of tapes still produces Monte Carlo error. For the policy-learning interface, define the sampled centered advantage by
A ^ g ( s , a ) = Q ^ g ( s , a ) − ∑ a ′ ∈ A π g ( a ′ ∣ s ) Q ^ g ( s , a ′ ) . (74) \widehat{A}_g(s,a)=\widehat{Q}_g(s,a)-\sum_{a'\in\mathcal{A}}\pi_g(a'\mid s)\widehat{Q}_g(s,a'). \tag*{(74)} A g ( s , a ) = Q g ( s , a ) − a ′ ∈ A ∑ π g ( a ′ ∣ s ) Q g ( s , a ′ ) . (74)
The implementation stores the centered residual correction directly. When all first-action branches on a tape reach the same observation, their remaining common tail cancels under this centering and need not be simulated. Thus it implements the centered estimator without materializing every absolute Q ^ g ( s , a ) \widehat{Q}_g(s,a) Q g ( s , a ) . The selection of query roots and their use by policy updates are described in Section B.5.
The state distribution used to train successor critics.
The next critics are fitted at states reached when the current behavior and target policies use the model. Let μ g \mu_g μ g and π g \pi_g π g denote the frozen parent behavior and target policies, respectively. We first specify how starting observations are selected, then how their continuations determine the training distribution. Let S 1 , … , S n S_1,\ldots,S_n S 1 , … , S n be the observations in the generation-g g g batch, with n = 25,000 n=25{,}000 n = 25 , 000 . These are transition occurrences: repeated observations retain their multiplicity. Write δ s \delta_s δ s for the Dirac probability measure concentrated at s s s , so that δ s ( E ) = 1 { s ∈ E } \delta_s(E)=\mathbf{1}\{s\in E\} δ s ( E ) = 1 { s ∈ E } for any measurable observation set E E E .
For the training-bank construction, let D g = n − 1 ∑ i = 1 n δ S i D_g=n^{-1}\sum_{i=1}^{n}\delta_{S_i} D g = n − 1 ∑ i = 1 n δ S i be the empirical observation distribution, and let ν g J \nu_g^J ν g J and ν g P \nu_g^P ν g P be the normalized empirical distributions obtained by the frozen target-return and transmission emphatic weights. Each selects an occurrence from this same batch; their weights are nonnegative and have empirical mean one. Their exact definitions and occurrence-sampling interpretation are in Section B.3. The critic treats these three root distributions as fixed inputs.
For RB, recall the independent synthetic risk in Equation (37) . Let m = Unif [ 0 , 1 / 2 ] m=\operatorname{Unif}[0,1/2] m = Unif [ 0 , 1/2 ] denote its probability law, with density m ( d z ) = 21 [ 0 , 1 / 2 ] ( z ) d z m(dz)=2\mathbf{1}_{[0,1/2]}(z)\,dz m ( d z ) = 2 1 [ 0 , 1/2 ] ( z ) d z . The symbol ⊗ \otimes ⊗ denotes the product of probability measures: for observation and risk sets E E E and C C C , ( D g ⊗ m ) ( E × C ) = D g ( E ) m ( C ) (D_g\otimes m)(E\times C)=D_g(E)m(C) ( D g ⊗ m ) ( E × C ) = D g ( E ) m ( C ) . either weighting role χ ∈ { J , P } \chi\in\{J, P\} χ ∈ { J , P } , define
D ~ g = D g ⊗ m , ν ~ g χ = ν g χ ⊗ m , χ ∈ { J , P } , (75) \widetilde{D}_g = D_g \otimes m,\qquad\widetilde{\nu}^{\chi}_g = \nu^{\chi}_g \otimes m,\qquad\chi\in\{J,P\}, \tag*{(75)} D g = D g ⊗ m , ν g χ = ν g χ ⊗ m , χ ∈ { J , P } , (75)
These product measures correspond to sampling a starting observation from the indicated distribution and an independent Z ∼ m Z \sim m Z ∼ m .
Let P ~ g , σ \widetilde{P}_{g,\sigma} P g , σ be the transition kernel on ( S , Z ) (S,Z) ( S , Z ) that samples an action from a fixed policy σ \sigma σ , advances the observation using P ^ g , Z \widehat{P}_{g,Z} P g , Z , and preserves Z Z Z . For an augmented-state distribution λ \lambda λ , the measure λ P ~ g , σ \lambda\widetilde{P}_{g,\sigma} λ P g , σ is the distribution after one such transition. For any measurable set E E E of augmented states,
( λ P ~ g , σ ) ( E ) = ∫ P ~ g , σ ( ( s , z ) , E ) λ ( d s , d z ) . (76) (\lambda\widetilde{P}_{g,\sigma})(E) = \int\widetilde{P}_{g,\sigma}((s,z),E)\,\lambda(ds,dz). \tag*{(76)} ( λ P g , σ ) ( E ) = ∫ P g , σ (( s , z ) , E ) λ ( d s , d z ) . (76)
For a nonnegative integer ℓ \ell ℓ , the power P ~ g , σ ℓ \widetilde{P}^{\ell}_{g,\sigma} P g , σ ℓ denotes ℓ \ell ℓ successive transitions; its zeroth power is the identity kernel.
The three uses of the critic select roots and first actions from ( D g , μ g ) (D_g,\mu_g) ( D g , μ g ) , ( ν g J , π g ) (\nu^J_g,\pi_g) ( ν g J , π g ) , and ( ν g P , μ g ) (\nu^P_g,\mu_g) ( ν g P , μ g ) , respectively. Mixing these uses equally gives the distribution after the first transition,
η ~ g = 1 3 ( D ~ g P ~ g , μ g + ν ~ g J P ~ g , π g + ν ~ g P P ~ g , μ g ) , (77) \widetilde{\eta}_g = \frac{1}{3}\left(\widetilde{D}_g\widetilde{P}_{g,\mu_g}+\widetilde{\nu}^{J}_g\widetilde{P}_{g,\pi_g}+\widetilde{\nu}^{P}_g\widetilde{P}_{g,\mu_g}\right), \tag*{(77)} η g = 3 1 ( D g P g , μ g + ν g J P g , π g + ν g P P g , μ g ) , (77)
After this first transition, all three routes follow the frozen target policy π g \pi_g π g . We weight their subsequent states geometrically, giving the normalized critic-training distribution
ζ ~ g = ( 1 − γ ) ∑ ℓ = 0 ∞ γ ℓ η ~ g P ~ g , π g ℓ . (78) \widetilde{\zeta}_g = (1-\gamma)\sum_{\ell=0}^{\infty}\gamma^\ell\widetilde{\eta}_g\widetilde{P}^{\ell}_{g,\pi_g}. \tag*{(78)} ζ g = ( 1 − γ ) ℓ = 0 ∑ ∞ γ ℓ η g P g , π g ℓ . (78)
The coefficients sum to one. In particular, the term ℓ = 0 \ell=0 ℓ = 0 describes the state immediately after the first transition; it does not describe the original root observation.
Equation (78) directly specifies a sampling procedure for a finite training bank. Choose one of the three root-distribution/policy pairs above with probability 1 / 3 1/3 1/3 . Draw S 0 S_0 S 0 from that root distribution and an independent Z ∼ m Z \sim m Z ∼ m , sample the first action A 0 A_0 A 0 from the selected policy, and simulate the first successor S 1 S_1 S 1 . The pair ( S 1 , Z ) (S_1,Z) ( S 1 , Z ) then has distribution η ~ g \widetilde{\eta}_g η g . Independently draw the number of additional target-policy transitions,
G ∼ Geom 0 ( 1 − γ ) , Pr ( G = ℓ ) = ( 1 − γ ) γ ℓ , ℓ = 0 , 1 , … . (79) G \sim\operatorname{Geom}_0(1-\gamma),\qquad\Pr(G=\ell)=(1-\gamma)\gamma^\ell,\qquad\ell=0,1,\ldots. \tag*{(79)} G ∼ Geom 0 ( 1 − γ ) , Pr ( G = ℓ ) = ( 1 − γ ) γ ℓ , ℓ = 0 , 1 , … . (79)
Starting from S 1 S_1 S 1 , simulate G G G further transitions under π g \pi_g π g and retain ( S 1 + G , Z ) (S_{1+G},Z) ( S 1 + G , Z ) . Conditional on G = ℓ G=\ell G = ℓ , its distribution is η ~ g P ~ g , π g ℓ \widetilde{\eta}_g\widetilde{P}^{\ell}_{g,\pi_g} η g P g , π g ℓ ; averaging over G G G therefore gives exactly ζ ~ g \widetilde{\zeta}_g ζ g . When G = 0 G=0 G = 0 , retain S 1 S_1 S 1 without an additional transition. The same sampled Z Z Z is used for every transition on this path.
For each retained pair ( S , z ) (S,z) ( S , z ) , sample one additional next availability
U ∼ K ^ B , g , z ( B ( S ) , ⋅ ) . (80) U \sim\widehat{K}_{B,g,z}(B(S),\cdot). \tag*{(80)} U ∼ K B , g , z ( B ( S ) , ⋅ ) . (80)
This draw uses the same tape risk, rather than a newly sampled risk or the availability from the first transition. The generation contains n ζ = 25,000 n_{\zeta}=25{,}000 n ζ = 25 , 000 such training pairs ( S , U ) (S,U) ( S , U ) , shared by all five critics; n ζ n_{\zeta} n ζ denotes the training-bank size.
The augmented formulation is essential: although Z Z Z is independent of the root observation, it generally becomes statistically dependent on later observations. Replacing the augmented kernel by powers of a risk-averaged observation kernel would change ζ ~ g \widetilde{\zeta}_g ζ g . For DSIS, the latent coordinate disappears and U = 1 U=1 U = 1 .
Learning the successor critics by minimizing Bellman-consistency error.
Let V θ V_{\theta} V θ denote one live scalar critic, with trainable parameter vector θ \theta θ ; the following objective is optimized separately for each of the five members. We fit it using a model-based form of the one-step temporal-difference (TD) error. As in TD-based critic learning, this error compares the current value prediction with the immediate reward plus a discounted bootstrap value at the successor observation. Here, we average the TD error over actions under the frozen target policy π g \pi_g π g and over demand under the frozen model, conditional on the sampled next availability U U U . We then minimize the square of this conditional mean. The resulting population objective is
J V , g ( θ ) = E ( S , z ) ∼ ζ ~ g U ∼ K ^ B , g , z ( B ( S ) , ⋅ ) [ ( V θ ( S ) − ∑ a ∈ A π g ( a ∣ S ) ∑ d ∈ D ~ g K ^ D , g ( d − ( S ) , d ) ⋅ [ r ( S , a , d ) + γ V θ ( F ( S , a , d , U ) ) ] ) 2 ] . (81) J_{V,g}(\theta)=\mathbb{E}_{(S,z)\sim\widetilde{\zeta}_g \atop U\sim\widehat{K}_{B,g,z}(B(S),\cdot)}\left[\left(V_\theta(S)-\sum_{a\in\mathcal{A}}\pi_g(a\mid S)\sum_{d\in\widetilde{\mathcal{D}}_g}\widehat{K}_{D,g}(d^{-}(S),d)\cdot\left[r(S,a,d)+\gamma V_\theta(F(S,a,d,U))\right]\right)^2\right]. \tag*{(81)} J V , g ( θ ) = E U ∼ K B , g , z ( B ( S ) , ⋅ ) ( S , z ) ∼ ζ g V θ ( S ) − a ∈ A ∑ π g ( a ∣ S ) d ∈ D g ∑ K D , g ( d − ( S ) , d ) ⋅ [ r ( S , a , d ) + γ V θ ( F ( S , a , d , U )) ] 2 . (81)
The expression inside the square is the negative of the conditional expected TD error; this sign convention does not affect the squared loss. We optimize this objective with full gradients through both the current and successor value predictions, as detailed below.
For the frozen bank of n ζ n_\zeta n ζ pairs sampled above, { ( S ℓ , U ℓ ) } ℓ = 1 n ζ \{(S_\ell,U_\ell)\}_{\ell=1}^{n_\zeta} {( S ℓ , U ℓ ) } ℓ = 1 n ζ , define the empirical objective
J ^ V , g ( θ ) = 1 n ζ ∑ ℓ = 1 n ζ [ V θ ( S ℓ ) − ∑ a ∈ A π g ( a ∣ S ℓ ) ∑ d ∈ D ~ g K ^ D , g ( d − ( S ℓ ) , d ) ⋅ { r ( S ℓ , a , d ) + γ V θ ( F ( S ℓ , a , d , U ℓ ) ) } ] 2 . (82) \widehat{J}_{V,g}(\theta)=\frac{1}{n_\zeta}\sum_{\ell=1}^{n_\zeta}\left[V_\theta(S_\ell)-\sum_{a\in\mathcal{A}}\pi_g(a\mid S_\ell)\sum_{d\in\widetilde{\mathcal{D}}_g}\widehat{K}_{D,g}(d^{-}(S_\ell),d)\cdot\left\{r(S_\ell,a,d)+\gamma V_\theta(F(S_\ell,a,d,U_\ell))\right\}\right]^2. \tag*{(82)} J V , g ( θ ) = n ζ 1 ℓ = 1 ∑ n ζ V θ ( S ℓ ) − a ∈ A ∑ π g ( a ∣ S ℓ ) d ∈ D g ∑ K D , g ( d − ( S ℓ ) , d ) ⋅ { r ( S ℓ , a , d ) + γ V θ ( F ( S ℓ , a , d , U ℓ )) } 2 . (82)
Recall that L g u L_g^u L g u in Equation (46) averages the one-step reward and discounted next value over model demand, holding next availability u u u fixed. Define the supply-conditioned target-policy Bellman backup T g u T_g^u T g u and its signed error ϵ g \epsilon_g ϵ g by
T g u V θ ( s ) = ∑ a ∈ A π g ( a ∣ s ) L g u V θ ( s , a ) , (83) T_g^uV_\theta(s)=\sum_{a\in\mathcal{A}}\pi_g(a\mid s)L_g^uV_\theta(s,a), \tag*{(83)} T g u V θ ( s ) = a ∈ A ∑ π g ( a ∣ s ) L g u V θ ( s , a ) , (83)
ϵ g ( θ ; s , u ) = V θ ( s ) − T g u V θ ( s ) . (84) \epsilon_g(\theta;s,u)=V_\theta(s)-T_g^uV_\theta(s). \tag*{(84)} ϵ g ( θ ; s , u ) = V θ ( s ) − T g u V θ ( s ) . (84)
The order of operations is part of the objective: actions and demand are integrated before squaring, while next availability is sampled and retained in the training pair. Recall that T g , z T_{g,z} T g , z in Equation (49) additionally averages over next availability. For fixed ( s , z ) (s,z) ( s , z ) , let E U \mathbb{E}_U E U and Var U \operatorname{Var}_U Var U denote expectation and variance under U ∼ K ^ B , g , z ( B ( s ) , ⋅ ) U\sim\widehat{K}_{B,g,z}(B(s),\cdot) U ∼ K B , g , z ( B ( s ) , ⋅ ) . Then
E U [ ( V θ ( s ) − T g U V θ ( s ) ) 2 ] = [ V θ ( s ) − ( T g , z g V θ ) ( s ) ] 2 + Var U [ T g U V θ ( s ) ] . (85) \mathbb{E}_U\left[\left(V_\theta(s)-T_g^UV_\theta(s)\right)^2\right]=\left[V_\theta(s)-\left(T_{g,z}^gV_\theta\right)(s)\right]^2+\operatorname{Var}_U\left[T_g^UV_\theta(s)\right]. \tag*{(85)} E U [ ( V θ ( s ) − T g U V θ ( s ) ) 2 ] = [ V θ ( s ) − ( T g , z g V θ ) ( s ) ] 2 + Var U [ T g U V θ ( s ) ] . (85)
Therefore, the RB fitting objective is generally different from the square of the fully supply-integrated Bellman residual. The distinction vanishes in DSIS, where next availability is deterministic.
Both value terms belong to the same live member and remain differentiable. Write ∇ θ \nabla_\theta ∇ θ for the gradient with respect to critic parameters θ \theta θ . In particular,
∇ θ ϵ g ( θ ; s , u ) = ∇ θ V θ ( s ) − γ ∑ a ∈ A π g ( a ∣ s ) ∑ d ∈ D ~ g K ^ D , g ( d − ( s ) , d ) ∇ θ V θ ( F ( s , a , d , u ) ) , (86) \nabla_\theta\epsilon_g(\theta;s,u)=\nabla_\theta V_\theta(s)-\gamma\sum_{a\in\mathcal{A}}\pi_g(a\mid s)\sum_{d\in\widetilde{\mathcal{D}}_g}\widehat{K}_{D,g}(d^{-}(s),d)\nabla_\theta V_\theta(F(s,a,d,u)), \tag*{(86)} ∇ θ ϵ g ( θ ; s , u ) = ∇ θ V θ ( s ) − γ a ∈ A ∑ π g ( a ∣ s ) d ∈ D g ∑ K D , g ( d − ( s ) , d ) ∇ θ V θ ( F ( s , a , d , u )) , (86)
and
∇ θ J ^ V , g ( θ ) = 2 n ζ ∑ ℓ = 1 n ζ ϵ g ( θ ; S ℓ , U ℓ ) ∇ θ ϵ g ( θ ; S ℓ , U ℓ ) . (87) \nabla_\theta\widehat{J}_{V,g}(\theta)=\frac{2}{n_\zeta}\sum_{\ell=1}^{n_\zeta}\epsilon_g(\theta;S_\ell,U_\ell)\nabla_\theta\epsilon_g(\theta;S_\ell,U_\ell). \tag*{(87)} ∇ θ J V , g ( θ ) = n ζ 2 ℓ = 1 ∑ n ζ ϵ g ( θ ; S ℓ , U ℓ ) ∇ θ ϵ g ( θ ; S ℓ , U ℓ ) . (87)
The model, policy, and sampled bank are fixed during this optimization. There is no stop-gradient or slowly updated target critic on the successor-value side of this loss.
Finite update and use of the successor.
For member j ∈ { 1 , … , 5 } j\in\{1,\ldots,5\} j ∈ { 1 , … , 5 } , write θ g , j \theta_{g,j} θ g , j for its frozen parent parameter vector and θ j ( k ) \theta_j^{(k)} θ j ( k ) for its live parameters after k k k optimizer updates. It starts from θ j ( 0 ) = θ g , j \theta_j^{(0)}=\theta_{g,j} θ j ( 0 ) = θ g , j , and performs 1000 Adam updates with minibatch size 256 and the configured value learning rate. Adam uses first- and second-moment decay coefficients ( β 1 , β 2 ) = ( 0.9 , 0.999 ) (\beta_1,\beta_2)=(0.9,0.999) ( β 1 , β 2 ) = ( 0.9 , 0.999 ) , denominator stabilizer ϵ A d a m = 10 − 8 \epsilon_{\mathrm{Adam}}=10^{-8} ϵ Adam = 1 0 − 8 , zero weight decay, and fresh optimizer state; gradients are not clipped. Members share the frozen training bank but use separately addressed minibatch schedules. The successor is the finite optimization output,
V g + 1 , j = V θ j ( 1000 ) , v g + 1 = 1 5 ∑ j = 1 5 V g + 1 , j . (88) V_{g+1,j}=V_{\theta_j}^{(1000)},\qquad v_{g+1}=\frac{1}{5}\sum_{j=1}^{5}V_{g+1,j}. \tag*{(88)} V g + 1 , j = V θ j ( 1000 ) , v g + 1 = 5 1 j = 1 ∑ 5 V g + 1 , j . (88)
This definition does not presume an exact minimizer or Bellman fixed point.
The subscript g + 1 g+1 g + 1 records when the fitted critic becomes available; the policy in its fitting objective is still the frozen parent π g \pi_g π g . Throughout this fit, current-generation residual estimation, heuristic queries, and actor updates continue to use v g v_g v g . The fitted mean v g + 1 v_{g+1} v g + 1 becomes their value reference in the next generation. This separates the live network being optimized from the frozen reference used to construct the current generation’s learning signals.
Warm and cold critic initialization.
Both initialization routes produce five scalar critics V 0 , j V_{0,j} V 0 , j , j ∈ { 1 , … , 5 } j\in\{1,\ldots,5\} j ∈ { 1 , … , 5 } . The warm route first evaluates the initial categorical policy π 0 \pi_0 π 0 with temporary Q networks and transfers their centered action information; the cold route directly fits scalar critics on simulated states. Both finish with the same constant-offset calibration.
Warm initialization and temporary Q evaluation. Warm initialization uses the counted 500,000-transition source prefix. After constructing the initial categorical policy π 0 \pi_0 π 0 , five temporary action-value networks are initialized from the source critic. A scalar source value is copied across action outputs; for SAC, the five initial networks use its two online Q networks, two target Q networks, and an additional copy of the first online Q network. These copies have separate parameter storage.
Write n 0 = 500,000 n_0=500{,}000 n 0 = 500 , 000 for the number of initialization pairs. For source occurrence i ∈ { 1 , … , n 0 } i\in\{1,\ldots,n_0\} i ∈ { 1 , … , n 0 } , let s i s_i s i be its observation, d i d_i d i its realized demand, and u i u_i u i its next availability, all obtained from the recorded transition. In RB, d i d_i d i is recorded in the next observation; in DSIS, it is recovered from the public inventory balance using the recorded action and current and next observations. Public dynamics provide counterfactual successors and rewards for every action:
s i , a ′ = F ( s i , a , d i , u i ) , r i , a = r ( s i , a , d i ) . (89) s'_{i,a}=F(s_i,a,d_i,u_i),\qquad r_{i,a}=r(s_i,a,d_i). \tag*{(89)} s i , a ′ = F ( s i , a , d i , u i ) , r i , a = r ( s i , a , d i ) . (89)
No unobserved demand or supply realization is needed for this construction.
For fitted Q evaluation (FQE), let Q j ( k ) Q_j^{(k)} Q j ( k ) and Q j − , ( k ) Q_j^{-,(k)} Q j − , ( k ) denote member j j j ’s online and target action-value networks at iteration k k k , with parameter vectors ψ j ( k ) \psi_j^{(k)} ψ j ( k ) and ψ j − , ( k ) \psi_j^{-,(k)} ψ j − , ( k ) , respectively. Each target network starts as a separate copy of its online network. The initial policy π 0 \pi_0 π 0 remains fixed. For every recorded occurrence i i i and action a ∈ A a\in\mathcal{A} a ∈ A , define the detached target
y j , i , a ( k ) = r i , a + γ ∑ a ′ ∈ A π 0 ( a ′ ∣ s i , a ′ ) Q j − , ( k ) ( s i , a ′ , a ′ ) . (90) y_{j,i,a}^{(k)}=r_{i,a}+\gamma\sum_{a'\in\mathcal{A}}\pi_0(a'\mid s'_{i,a})Q_j^{-,(k)}(s'_{i,a},a'). \tag*{(90)} y j , i , a ( k ) = r i , a + γ a ′ ∈ A ∑ π 0 ( a ′ ∣ s i , a ′ ) Q j − , ( k ) ( s i , a ′ , a ′ ) . (90)
Here “detached” means that differentiation of the FQE loss does not propagate through y j , i , a ( k ) y_{j,i,a}^{(k)} y j , i , a ( k ) . At each iteration, draw an ordered minibatch I ( k ) = ( i 1 , … , i n b ) \mathcal{I}^{(k)}=(i_1,\ldots,i_{n_b}) I ( k ) = ( i 1 , … , i n b ) of n b = 256 n_b=256 n b = 256 occurrence indices uniformly with replacement from { 1 , … , n 0 } \{1,\ldots,n_0\} { 1 , … , n 0 } ; the five members share this minibatch. For each member j j j , minibatch position h h h , and action a a a , draw a fresh binary mask ξ j , h , a ( k ) \xi_{j,h,a}^{(k)} ξ j , h , a ( k ) , independently across positions, actions, members, and iterations, with probability 0.8 of being one. Thus ξ j , h , a ( k ) ∼ Bernoulli ( 0.8 ) \xi_{j,h,a}^{(k)}\sim\operatorname{Bernoulli}(0.8) ξ j , h , a ( k ) ∼ Bernoulli ( 0.8 ) . The loss is
L j F Q E , ( k ) = 1 n b ∣ A ∣ ∑ h = 1 n b ∑ a ∈ A ξ j , h , a ( k ) [ Q j ( k ) ( s i h , a ) − y j , i h , a ( k ) ] 2 . (91) \mathcal{L}_{j}^{\mathrm{FQE},(k)}=\frac{1}{n_b|\mathcal{A}|}\sum_{h=1}^{n_b}\sum_{a\in\mathcal{A}}\xi_{j,h,a}^{(k)}\left[Q_j^{(k)}(s_{i_h},a)-y_{j,i_h,a}^{(k)}\right]^2. \tag*{(91)} L j FQE , ( k ) = n b ∣ A ∣ 1 h = 1 ∑ n b a ∈ A ∑ ξ j , h , a ( k ) [ Q j ( k ) ( s i h , a ) − y j , i h , a ( k ) ] 2 . (91)
The denominator counts every minibatch-action pair, including masked pairs; it is not the number of nonzero masks. After the Adam update of the online parameters to ψ j ( k + 1 ) \psi_j^{(k+1)} ψ j ( k + 1 ) , update the target by
ψ j − , ( k + 1 ) = 0.995 ψ j − , ( k ) + 0.005 ψ j ( k + 1 ) . (92) \psi_j^{-,(k+1)}=0.995\,\psi_j^{-,(k)}+0.005\,\psi_j^{(k+1)}. \tag*{(92)} ψ j − , ( k + 1 ) = 0.995 ψ j − , ( k ) + 0.005 ψ j ( k + 1 ) . (92)
FQE performs 2000 updates per member. Its Adam learning rate is 1.2020438491997917 × 10 − 4 1.2020438491997917 \times10^{-4} 1.2020438491997917 × 1 0 − 4 for PPO, 4.418119410401287 × 10 − 5 4.418119410401287 \times10^{-5} 4.418119410401287 × 1 0 − 5 for ACE, and 5.669429427494781 × 10 − 4 5.669429427494781 \times10^{-4} 5.669429427494781 × 1 0 − 4 for the other source methods. These target Q networks belong only to temporary initialization; generation-level scalar-critic fitting uses the full gradient through both value predictions, as specified above. Write Q j = Q j ( 2000 ) Q_j = Q_j^{(2000)} Q j = Q j ( 2000 ) for the final online Q network used in the next step.
Scalar fitting for the two initialization routes. Five fresh scalar critics are initialized with separately randomized orthogonal hidden weights, zero hidden biases, and zero output weights and biases. In the warm case, let the fitted temporary Q networks define
A j i n i t ( s , a ) = Q j ( s , a ) − ∑ a ′ ∈ A π 0 ( a ′ ∣ s ) Q j ( s , a ′ ) . (93) A_j^{\mathrm{init}}(s,a) = Q_j(s,a) - \sum_{a' \in A} \pi_0(a' \mid s) Q_j(s,a'). \tag*{(93)} A j init ( s , a ) = Q j ( s , a ) − a ′ ∈ A ∑ π 0 ( a ′ ∣ s ) Q j ( s , a ′ ) . (93)
The advantage labels A j i n i t A_j^{\mathrm{init}} A j init and the policy π 0 \pi_0 π 0 are held fixed during scalar fitting. The operator L i n i t u L_{\mathrm{init}}^u L init u is the supply-conditioned backup in Equation (46) , evaluated with the frozen initialization model. The warm scalar-fitting objective is
J j w a r m ( θ ) = 1 n 0 ∑ i = 1 n 0 ∑ a ∈ A π 0 ( a ∣ s i ) ⋅ [ L i n i t u i V θ ( s i , a ) − V θ ( s i ) − A j i n i t ( s i , a ) ] 2 . (94) \mathcal{J}_j^{\mathrm{warm}}(\theta) = \frac{1}{n_0} \sum_{i=1}^{n_0} \sum_{a \in A} \pi_0(a \mid s_i) \cdot\left[L_{\mathrm{init}}^{u_i} V_\theta(s_i,a) - V_\theta(s_i) - A_j^{\mathrm{init}}(s_i,a)\right]^2 . \tag*{(94)} J j warm ( θ ) = n 0 1 i = 1 ∑ n 0 a ∈ A ∑ π 0 ( a ∣ s i ) ⋅ [ L init u i V θ ( s i , a ) − V θ ( s i ) − A j init ( s i , a ) ] 2 . (94)
Here u i u_i u i is the observed next availability in the source prefix, while L i n i t u i L_{\mathrm{init}}^{u_i} L init u i integrates demand using the frozen initialization model. Both occurrences of V θ V_\theta V θ are differentiated. This objective transfers centered action information from the temporary Q critics into the scalar value representation.
Cold initialization instead creates 500 synthetic trajectories of length 1000 under π 0 \pi_0 π 0 . Their n 0 = 500,000 n_0 = 500{,}000 n 0 = 500 , 000 current-observation/next-availability pairs are again denoted by ( s i , u i ) (s_i,u_i) ( s i , u i ) ; here both coordinates come from model simulation. Each member minimizes
J j c o l d ( θ ) = 1 n 0 ∑ i = 1 n 0 [ V θ ( s i ) − ∑ a ∈ A π 0 ( a ∣ s i ) L i n i t u i V θ ( s i , a ) ] 2 . (95) \mathcal{J}_j^{\mathrm{cold}}(\theta) = \frac{1}{n_0} \sum_{i=1}^{n_0} \left[V_\theta(s_i) - \sum_{a \in A} \pi_0(a \mid s_i)L_{\mathrm{init}}^{u_i} V_\theta(s_i,a)\right]^2 . \tag*{(95)} J j cold ( θ ) = n 0 1 i = 1 ∑ n 0 [ V θ ( s i ) − a ∈ A ∑ π 0 ( a ∣ s i ) L init u i V θ ( s i , a ) ] 2 . (95)
For RB, this simulation starts only after collecting the initial counted 25,000 transitions and fitting their model. Initial observations are sampled uniformly, with multiplicity, from the observed joint reset-state occurrences. Every synthetic trajectory receives an independent Z ∼ m = Unif [ 0 , 1 / 2 ] Z \sim m = \operatorname{Unif}[0,1/2] Z ∼ m = Unif [ 0 , 1/2 ] , fixed for its entire length. For DSIS, simulation uses the public initial state and known model. The cold bank does not reuse private environment tapes.
Calibrating the constant value offset. For either route, each scalar critic is optimized for two complete epochs with minibatch size 256 and Adam learning rate 5.669429427494781 × 10 − 4 5.669429427494781 \times10^{-4} 5.669429427494781 × 1 0 − 4 . Each epoch visits all n 0 n_0 n 0 pairs once in a member-specific permutation; the last minibatch contains 32 pairs. Adam state is initialized once for this fit and retained across its two epochs, with the same moment coefficients and denominator stabilizer as in generation-level fitting. Let V ~ j \widetilde V_j V j be the resulting network. To describe the final calibration, specialize the supply-conditioned Bellman backup in Equation (83) to the initial policy and model:
T i n i t u V ~ j ( s ) = ∑ a ∈ A π 0 ( a ∣ s ) L i n i t u V ~ j ( s , a ) . (96) T_{\mathrm{init}}^u \widetilde V_j(s) = \sum_{a \in A} \pi_0(a \mid s)L_{\mathrm{init}}^u \widetilde V_j(s,a). \tag*{(96)} T init u V j ( s ) = a ∈ A ∑ π 0 ( a ∣ s ) L init u V j ( s , a ) . (96)
This operator returns the expected immediate reward plus discounted next value, averaging over the action and model demand while holding the next availability u u u fixed. Its conditioning matches the initialization pairs ( s i , u i ) (s_i,u_i) ( s i , u i ) : u i u_i u i is observed in the warm bank and generated by the model in the cold bank. It is the conditional counterpart of T g , ε π T_{g,\varepsilon}^{\pi} T g , ε π in Equation (49) , which additionally averages over next availability.
Compute the full-bank mean backup-minus-prediction residual,
e ˉ j = 1 n 0 ∑ i = 1 n 0 [ ( T i n i t u i V ~ j ) ( s i ) − V ~ j ( s i ) ] . (97) \bar e_j = \frac{1}{n_0} \sum_{i=1}^{n_0} \left[\left(T_{\mathrm{init}}^{u_i} \widetilde V_j\right)(s_i) - \widetilde V_j(s_i)\right]. \tag*{(97)} e ˉ j = n 0 1 i = 1 ∑ n 0 [ ( T init u i V j ) ( s i ) − V j ( s i ) ] . (97)
Adding a constant c o f f c_{\mathrm{off}} c off to every value prediction adds γ c o f f \gamma c_{\mathrm{off}} γ c off to the discounted next-value term, because the action and demand probabilities each sum to one. Therefore,
T i n i t u ( V ~ j + c o f f ) ( s ) = T i n i t u V ~ j ( s ) + γ c o f f , (98) T_{\mathrm{init}}^u\left(\widetilde V_j + c_{\mathrm{off}}\right)(s) = T_{\mathrm{init}}^u \widetilde V_j(s) + \gamma c_{\mathrm{off}}, \tag*{(98)} T init u ( V j + c off ) ( s ) = T init u V j ( s ) + γ c off , (98)
and hence the residual changes by − ( 1 − γ ) c o f f -(1-\gamma)c_{\mathrm{off}} − ( 1 − γ ) c off at every pair:
T i n i t u ( V ~ j + c o f f ) ( s ) − ( V ~ j ( s ) + c o f f ) = T i n i t u V ~ j ( s ) − V ~ j ( s ) − ( 1 − γ ) c o f f . (99) \mathcal{T}_{\mathrm{init}}^{u}(\widetilde{V}_{j}+c_{\mathrm{off}})(s)-(\widetilde{V}_{j}(s)+c_{\mathrm{off}})=\mathcal{T}_{\mathrm{init}}^{u}\widetilde{V}_{j}(s)-\widetilde{V}_{j}(s)-(1-\gamma)c_{\mathrm{off}}. \tag*{(99)} T init u ( V j + c off ) ( s ) − ( V j ( s ) + c off ) = T init u V j ( s ) − V j ( s ) − ( 1 − γ ) c off . (99)
The final initialization chooses
V 0 , j ( s ) = V ~ j ( s ) + e ˉ j 1 − γ , (100) V_{0,j}(s)=\widetilde{V}_{j}(s)+\frac{\bar{e}_{j}}{1-\gamma}, \tag*{(100)} V 0 , j ( s ) = V j ( s ) + 1 − γ e ˉ j , (100)
implemented by shifting the scalar output bias (up to floating-point rounding of that bias). The new mean residual is e ˉ j − ( 1 − γ ) e ˉ j / ( 1 − γ ) = 0 \bar{e}_{j}-(1-\gamma)\bar{e}_{j}/(1-\gamma)=0 e ˉ j − ( 1 − γ ) e ˉ j / ( 1 − γ ) = 0 in exact arithmetic. This removes a constant offset in the empirical Bellman error; it does not set every statewise residual to zero. After warm initialization, the temporary Q networks are discarded.
Emphatic Weighting: Mass Learning, Heuristic Interest, and Cross-Generation Use
Meaning of emphatic weighting and the collected data.
Emphatic weighting assigns state-dependent importance to learning updates by propagating an interest signal through transitions. Here interest is an externally specified or data-derived nonnegative function that states which locations should receive additional learning emphasis. This separates how frequently an observation occurs in behavior data from how strongly it contributes to a learning objective. At learning generation g g g , HBPO uses two weighting roles: target-return learning, denoted by J J J , and behavior-to-target transmission, denoted by P P P . We call their positive state scores M g J M_{g}^{J} M g J and M g P M_{g}^{P} M g P raw masses. They are functions of the observation, not normalized state distributions. Normalization and shrinkage convert these scores into weights and then into empirical probability measures for the corresponding learning objectives.
Let D g \mathcal{D}_{g} D g be the indexed multiset of n = 25,000 n=25{,}000 n = 25 , 000 transitions collected along trajectories of the frozen behavior policy μ g \mu_{g} μ g . The index i = 1 , … , n i=1,\ldots,n i = 1 , … , n identifies an occurrence, including repeated observations and transitions:
D g = ( ( s i , a i , s i ′ ) ) i = 1 n . (101) \mathcal{D}_{g}=\left((s_{i},a_{i},s_{i}^{\prime})\right)_{i=1}^{n}. \tag*{(101)} D g = ( ( s i , a i , s i ′ ) ) i = 1 n . (101)
Recall that D g \mathcal{D}_{g} D g averages over current observations. Let D g ′ \mathcal{D}_{g}^{\prime} D g ′ average over successor observations and let Γ g \Gamma_{g} Γ g retain the paired transition occurrences. Writing δ x \delta_{x} δ x for unit point mass at x x x , these empirical measures are
D g = 1 n ∑ i = 1 n δ s i , D g ′ = 1 n ∑ i = 1 n δ s i ′ , Γ g = 1 n ∑ i = 1 n δ ( s i , a i , s i ′ ) . (102) \mathcal{D}_{g}=\frac{1}{n}\sum_{i=1}^{n}\delta_{s_{i}},\qquad\mathcal{D}_{g}^{\prime}=\frac{1}{n}\sum_{i=1}^{n}\delta_{s_{i}^{\prime}},\qquad\Gamma_{g}=\frac{1}{n}\sum_{i=1}^{n}\delta_{(s_{i},a_{i},s_{i}^{\prime})}. \tag*{(102)} D g = n 1 i = 1 ∑ n δ s i , D g ′ = n 1 i = 1 ∑ n δ s i ′ , Γ g = n 1 i = 1 ∑ n δ ( s i , a i , s i ′ ) . (102)
Repeated observations retain their occurrence multiplicities. Masses are fitted at successor observations, whereas the weights consumed in generation g g g are computed from the parent masses at current observations. The symbol χ ∈ { J , P } \chi\in\{J,P\} χ ∈ { J , P } denotes a weighting role.
Emphatic weighting in prior work.
The emphatic TD (ETD) construction of Sutton et al. [38 ] propagates user-specified interest through target-policy transitions to weight bootstrapped value updates. Its linear stability theorem concerns the expected update under the paper’s assumptions; it is not a stability theorem for nonlinear mass networks. To summarize the follow-on part of this construction, consider a finite Markov model, fixed behavior and target policies μ \mu μ and π \pi π , a constant discount 0 < γ < 1 0<\gamma<1 0 < γ < 1 , and a fixed nonnegative interest function ι \iota ι . Assume that d μ d_{\mu} d μ is a strictly positive stationary state distribution under μ \mu μ , that S 0 ∼ d μ \mathcal{S}_{0}\sim d_{\mu} S 0 ∼ d μ , and that μ ( a ∣ s ) > 0 \mu(a\mid s)>0 μ ( a ∣ s ) > 0 whenever π ( a ∣ s ) > 0 \pi(a\mid s)>0 π ( a ∣ s ) > 0 . Generate the trajectory under μ \mu μ . Let k k k index its steps, and let F k f o F_{k}^{\mathrm{fo}} F k fo denote the follow-on trace. Initialize F 0 f o = ι ( S 0 ) F_{0}^{\mathrm{fo}}=\iota(S_{0}) F 0 fo = ι ( S 0 ) ; for k ≥ 1 k\geq1 k ≥ 1 , using the target-to-behavior action ratio ρ k − 1 \rho_{k-1} ρ k − 1 , set
F k f o = ι ( S k ) + γ ρ k − 1 F k − 1 f o , ρ k − 1 = π ( A k − 1 ∣ S k − 1 ) μ ( A k − 1 ∣ S k − 1 ) . (103) F_{k}^{\mathrm{fo}}=\iota(S_{k})+\gamma\rho_{k-1}F_{k-1}^{\mathrm{fo}},\qquad\rho_{k-1}=\frac{\pi(A_{k-1}\mid S_{k-1})}{\mu(A_{k-1}\mid S_{k-1})}. \tag*{(103)} F k fo = ι ( S k ) + γ ρ k − 1 F k − 1 fo , ρ k − 1 = μ ( A k − 1 ∣ S k − 1 ) π ( A k − 1 ∣ S k − 1 ) . (103)
Let P ( s ′ ∣ s , a ) P(s^{\prime}\mid s,a) P ( s ′ ∣ s , a ) be this model’s transition probability and let P π ( s , s ′ ) = ∑ a π ( a ∣ s ) P ( s ′ ∣ s , a ) P_{\pi}(s,s^{\prime})=\sum_{a}\pi(a\mid s)P(s^{\prime}\mid s,a) P π ( s , s ′ ) = ∑ a π ( a ∣ s ) P ( s ′ ∣ s , a ) be its target-policy transition matrix. Define the expected follow-on state mass by f ι ( s ) = d μ ( s ) lim k → ∞ E [ F k f o ∣ S k = s ] f_{\iota}(s)=d_{\mu}(s)\lim_{k\to\infty}\mathbb{E}[F_{k}^{\mathrm{fo}}\mid S_{k}=s] f ι ( s ) = d μ ( s ) lim k → ∞ E [ F k fo ∣ S k = s ] . The mass satisfies
f ι = d μ ∘ ι + γ P π T f ι , f ι = ( I d − γ P π T ) − 1 ( d μ ∘ ι ) . (104) f_{\iota}=d_{\mu}\circ\iota+\gamma P_{\pi}^{\mathsf{T}}f_{\iota},\qquad f_{\iota}=(\mathrm{Id}-\gamma P_{\pi}^{\mathsf{T}})^{-1}(d_{\mu}\circ\iota). \tag*{(104)} f ι = d μ ∘ ι + γ P π T f ι , f ι = ( Id − γ P π T ) − 1 ( d μ ∘ ι ) . (104)
Thus the conditional mean trace and the state mass obtained by multiplying it by d μ d_\mu d μ are different objects. We use this follow-on component to explain the connection, rather than identifying it with the full emphasis of every ETD trace-parameter setting.
OffPAC ([7 ] ) corrects sampled actions with importance ratios but retains behavior-state weighting in its actor semi-gradient. For this literature comparison, let ϑ \vartheta ϑ be the actor parameter vector and let π ϑ \pi_\vartheta π ϑ be a differentiable target policy. For the displayed score formula, assume that both μ ( a ∣ s ) \mu(a \mid s) μ ( a ∣ s ) and π ϑ ( a ∣ s ) \pi_\vartheta(a \mid s) π ϑ ( a ∣ s ) are strictly positive for every state and action. Let Q π ϑ Q^{\pi_\vartheta} Q π ϑ and V π ϑ V^{\pi_\vartheta} V π ϑ be the exact discounted action and state values in the finite Markov model, and let ∇ ϑ \nabla_\vartheta ∇ ϑ denote differentiation with respect to ϑ \vartheta ϑ . Using exact action values to isolate the state-weighting issue, write OffPAC’s basic actor direction as g O f f P A C ( ϑ ) g_{\mathrm{OffPAC}}(\vartheta) g OffPAC ( ϑ ) :
g O f f P A C ( ϑ ) = E S ∼ d μ , A ∼ μ ( ⋅ ∣ S ) [ π ϑ ( A ∣ S ) μ ( A ∣ S ) Q π ϑ ( S , A ) ∇ ϑ log π ϑ ( A ∣ S ) ] = ∑ s d μ ( s ) ∑ a ∇ ϑ π ϑ ( a ∣ s ) Q π ϑ ( s , a ) . (105) \begin{aligned}
g_{\mathrm{OffPAC}}(\vartheta) &= \mathbb{E}_{S \sim d_\mu, A \sim\mu(\cdot\mid S)}\left[\frac{\pi_\vartheta(A \mid S)}{\mu(A \mid S)} Q^{\pi_\vartheta}(S,A)\nabla_\vartheta\log\pi_\vartheta(A \mid S)\right] \\
&= \sum_s d_\mu(s)\sum_a \nabla_\vartheta\pi_\vartheta(a \mid s)Q^{\pi_\vartheta}(s,a). \tag*{(105)}
\end{aligned} g OffPAC ( ϑ ) = E S ∼ d μ , A ∼ μ ( ⋅ ∣ S ) [ μ ( A ∣ S ) π ϑ ( A ∣ S ) Q π ϑ ( S , A ) ∇ ϑ log π ϑ ( A ∣ S ) ] = s ∑ d μ ( s ) a ∑ ∇ ϑ π ϑ ( a ∣ s ) Q π ϑ ( s , a ) . (105)
Action importance sampling changes the action average to the target policy, while the outer state average remains weighted by d μ d_\mu d μ .
The policy-gradient theorem underlying Actor–Critic with Emphatic weightings (ACE; Imani et al. [16 ] ) supplies the emphatic state weighting for the exact gradient of the interest-weighted excursion objective J ι J_\iota J ι :
J ι ( ϑ ) = ∑ s d μ ( s ) ι ( s ) V π ϑ ( s ) . (106) J_\iota(\vartheta)=\sum_s d_\mu(s)\iota(s)V^{\pi_\vartheta}(s). \tag*{(106)} J ι ( ϑ ) = s ∑ d μ ( s ) ι ( s ) V π ϑ ( s ) . (106)
Holding μ \mu μ , d μ d_\mu d μ , and ι \iota ι fixed when differentiating, its policy-gradient theorem gives
∇ ϑ J ι ( ϑ ) = ∑ s f ι ( s ) ∑ a ∇ ϑ π ϑ ( a ∣ s ) Q π ϑ ( s , a ) , (107) \nabla_\vartheta J_\iota(\vartheta)=\sum_s f_\iota(s)\sum_a\nabla_\vartheta\pi_\vartheta(a \mid s)Q^{\pi_\vartheta}(s,a), \tag*{(107)} ∇ ϑ J ι ( ϑ ) = s ∑ f ι ( s ) a ∑ ∇ ϑ π ϑ ( a ∣ s ) Q π ϑ ( s , a ) , (107)
where f ι f_\iota f ι is the follow-on state mass in Equation (104) , evaluated at π = π ϑ \pi=\pi_\vartheta π = π ϑ . For unit interest, comparison with Equation (105) isolates the missing follow-on state weighting in the semi-gradient. This equality uses exact values and state weights; finite trace and value estimates introduce additional approximation. Graves et al. [13 ] also study directly learning conditional emphatic weights from predecessor transitions. This motivates the regression viewpoint below. HBPO specifies its own labels, anchored mass functions, shrinkage, and generation schedule; the cited results do not certify these modifications.
Why learn positive masses, and what the regression represents.
The purpose of the mass networks is to store a state-dependent importance field that can be evaluated on a later generation’s observations. A recorded transition provides a predecessor carrying existing mass and a successor at which that mass should contribute to learning. Fitting a shared function at successor observations pools this incoming evidence and generalizes it across the fixed observation features. The two roles use different labels, specified below, but share the following parameterization and fitting principle.
Recall that χ ∈ { J , P } \chi\in\{J,P\} χ ∈ { J , P } indexes the two raw mass functions. For each role, let f ω χ χ f^{\chi}_{\omega^\chi} f ω χ χ be a separate scalar network with parameter vector ω χ \omega^\chi ω χ , two width-128 Tanh hidden layers, and a scalar output. Recall the fixed observation features ϕ \phi ϕ from Equation (44) . Let s ∘ s_\circ s ∘ be the reference observation with zero inventory and pipeline and available regular supply; its previous-demand coordinate is also zero when present. Let β g χ > 0 \beta^\chi_g>0 β g χ > 0 be the prescribed mass at this reference. Define
M g χ ( s ) = β g χ exp [ f ω χ χ ( ϕ ( s ) ) − f ω χ χ ( ϕ ( s ∘ ) ) ] , β g χ > 0. (108) M^\chi_g(s)=\beta^\chi_g\exp\left[f^\chi_{\omega^\chi}(\phi(s))-f^\chi_{\omega^\chi}(\phi(s_\circ))\right],\qquad\beta^\chi_g>0. \tag*{(108)} M g χ ( s ) = β g χ exp [ f ω χ χ ( ϕ ( s )) − f ω χ χ ( ϕ ( s ∘ )) ] , β g χ > 0. (108)
Thus M g χ ( s ) > 0 M^\chi_g(s)>0 M g χ ( s ) > 0 and M g χ ( s ∘ ) = β g χ M^\chi_g(s_\circ)=\beta^\chi_g M g χ ( s ∘ ) = β g χ . The reference value is imposed by the parameterization. If s ∘ s_\circ s ∘ occurs among the observed successors, this value need not equal the unrestricted regression target there.
For a fixed role χ \chi χ , let Y i χ > 0 Y^\chi_i>0 Y i χ > 0 be the regression label attached to transition occurrence i i i ; the two roles construct these labels below. For a candidate positive mass function M M M , define the full-data loss L g X ( M ) \mathcal{L}^{X}_{g}(M) L g X ( M ) by
L g X ( M ) = 1 n ∑ i = 1 n [ M ( s i ′ ) − Y i X log M ( s i ′ ) ] . (109) \mathcal{L}^{X}_{g}(M)=\frac{1}{n}\sum_{i=1}^{n}\left[M(s'_i)-Y^X_i\log M(s'_i)\right]. \tag*{(109)} L g X ( M ) = n 1 i = 1 ∑ n [ M ( s i ′ ) − Y i X log M ( s i ′ ) ] . (109)
This is the Poisson-form loss (a quasi-likelihood objective for positive real labels), with label-only terms omitted. It does not assume that these labels are integer-valued Poisson observations. Recall that Γ g \Gamma_g Γ g in Equation (102) samples recorded transition occurrences uniformly. More explicitly, draw an occurrence index I I I uniformly from { 1 , … , n } \{1,\ldots,n\} { 1 , … , n } and set ( S , A , S ′ , Y X ) = ( s I , a I , s I ′ , Y I X ) (S,A,S',Y^X)=(s_I,a_I,s'_I,Y^X_I) ( S , A , S ′ , Y X ) = ( s I , a I , s I ′ , Y I X ) . This defines the joint empirical law including the label; expectations subscripted by Γ g \Gamma_g Γ g below use this attached-label law when needed. Its current and successor marginals remain D g D_g D g and D g ′ D'_g D g ′ .
For an observed successor s ′ s' s ′ , write x log = log M ( s ′ ) x_{\log}=\log M(s') x l o g = log M ( s ′ ) , with log \log log denoting the natural logarithm. Let ℓ ( x log ; s ′ ) \ell(x_{\log};s') ℓ ( x l o g ; s ′ ) be the conditional loss. Its value and first two derivatives are
ℓ ( x log ; s ′ ) = e x log − E Γ g [ Y X ∣ S ′ = s ′ ] x log , \ell(x_{\log};s')=e^{x_{\log}}-\mathbb{E}_{\Gamma_g}[Y^X\mid S'=s']x_{\log}, ℓ ( x l o g ; s ′ ) = e x l o g − E Γ g [ Y X ∣ S ′ = s ′ ] x l o g ,
∂ ℓ ∂ x log = e x log − E Γ g [ Y X ∣ S ′ = s ′ ] , ∂ 2 ℓ ∂ x log 2 = e x log > 0. (110) \frac{\partial\ell}{\partial x_{\log}}=e^{x_{\log}}-\mathbb{E}_{\Gamma_g}[Y^X\mid S'=s'],\qquad\frac{\partial^2\ell}{\partial x_{\log}^2}=e^{x_{\log}}>0. \tag*{(110)} ∂ x l o g ∂ ℓ = e x l o g − E Γ g [ Y X ∣ S ′ = s ′ ] , ∂ x l o g 2 ∂ 2 ℓ = e x l o g > 0. (110)
Let M ⋆ ( s ′ ) M^\star(s') M ⋆ ( s ′ ) denote the unrestricted pointwise minimizer at an observed successor. The positive conditional mean and strict convexity give
M ⋆ ( s ′ ) = E Γ g [ Y X ∣ S ′ = s ′ ] . (111) M^\star(s')=\mathbb{E}_{\Gamma_g}[Y^X\mid S'=s']. \tag*{(111)} M ⋆ ( s ′ ) = E Γ g [ Y X ∣ S ′ = s ′ ] . (111)
A successor receives the average label carried into it by its observed predecessors. This calculation explains the meaning of the labels and the choice of loss. The actual learner approximates this regression with the anchored network and finite optimization; it does not set each prediction equal to the unrestricted minimizer.
Target-return mass M J M^J M J .
The J J J mass addresses the difference between the states represented by behavior data and those reached when the target policy continues from that behavior-distributed source. Action importance ratios alone change the action distribution at a given observation; they do not change how often that observation contributes to the update. Propagating mass through target-policy transitions provides a state-level adjustment for the target-return channel. Its ideal reference is specified below, so this motivation does not presume a stationary target-occupancy ratio.
Recall that π g \pi_g π g and μ g \mu_g μ g are the frozen parent target and behavior policies. Define their action-ratio function ϱ g \varrho_g ϱ g and its value at recorded occurrence i i i by
ϱ g ( s , a ) = π g ( a ∣ s ) μ g ( a ∣ s ) , ϱ g , i = ϱ g ( s i , a i ) . (112) \varrho_g(s,a)=\frac{\pi_g(a\mid s)}{\mu_g(a\mid s)},\qquad\varrho_{g,i}=\varrho_g(s_i,a_i). \tag*{(112)} ϱ g ( s , a ) = μ g ( a ∣ s ) π g ( a ∣ s ) , ϱ g , i = ϱ g ( s i , a i ) . (112)
The implementation evaluates this ratio from parent-policy log probabilities without clipping; the categorical policies have common positive action support. The J J J learner has an online parameter vector ω g , k J \omega^J_{g,k} ω g , k J and a private slow copy ω ‾ g , k J \overline{\omega}^J_{g,k} ω g , k J at inner iteration k k k , initialized by ω g , 0 J = ω ‾ g , 0 J = ω g J \omega^J_{g,0}=\overline{\omega}^J_{g,0}=\omega^J_g ω g , 0 J = ω g , 0 J = ω g J . For k = 0 , … , 999 k=0,\ldots,999 k = 0 , … , 999 , define the slow mass M ‾ g , k J \overline{M}^J_{g,k} M g , k J and the detached regression label Y g , i , k J Y^J_{g,i,k} Y g , i , k J as
M ‾ g , k J ( s ) = β g J exp [ f ω ‾ g , k J ( ϕ ( s ) ) − f ω ‾ g , k J ( ϕ ( s ∘ ) ) ] , (113) \overline{M}^J_{g,k}(s)=\beta^J_g\exp\left[f_{\overline{\omega}^J_{g,k}}(\phi(s))-f_{\overline{\omega}^J_{g,k}}(\phi(s_\circ))\right], \tag*{(113)} M g , k J ( s ) = β g J exp [ f ω g , k J ( ϕ ( s )) − f ω g , k J ( ϕ ( s ∘ )) ] , (113)
Y g , i , k J = ( 1 − γ ) + γ ϱ g , i M ‾ g , k J ( s i ) . (114) Y^J_{g,i,k}=(1-\gamma)+\gamma\varrho_{g,i}\overline{M}^J_{g,k}(s_i). \tag*{(114)} Y g , i , k J = ( 1 − γ ) + γ ϱ g , i M g , k J ( s i ) . (114)
The source term 1 − γ 1-\gamma 1 − γ injects positive mass; the continuation term carries predecessor mass through the recorded transition with an action correction toward π g \pi_g π g . Recall that β g J \beta^J_g β g J is the parent reference mass. At iteration k k k , the online prediction uses the parameterization in Equation (108) with parameters ω g , k J \omega^J_{g,k} ω g , k J and successor reference value β g + 1 J = 1 − γ \beta^J_{g+1}=1-\gamma β g + 1 J = 1 − γ . It is fitted at s i ′ s'_i s i ′ using Equation (109) with Y i X = Y g , i , k J Y^X_i=Y^J_{g,i,k} Y i X = Y g , i , k J , holding the labels fixed during that optimization step. After the step, the slow parameters are updated by
ω ‾ g , k + 1 J = 0.995 ω ‾ g , k J + 0.005 ω g , k + 1 J . (115) \overline{\omega}^J_{g,k+1}=0.995\overline{\omega}^J_{g,k}+0.005\omega^J_{g,k+1}. \tag*{(115)} ω g , k + 1 J = 0.995 ω g , k J + 0.005 ω g , k + 1 J . (115)
The slow mass retains β g J \beta^J_g β g J throughout the generation, while the online mass uses β g + 1 J \beta^J_{g+1} β g + 1 J . Only the final online network is committed as M g + 1 J M^J_{g+1} M g + 1 J .
For a scalar state function h h h , define the empirical reverse-transition operator B g J B^J_g B g J at each observed successor by
( B g J h ) ( s ′ ) = E Γ g [ ϱ g ( S , A ) h ( S ) ∣ S ′ = s ′ ] . (116) (B^J_g h)(s') = \mathbb{E}_{\Gamma_g}\left[\varrho_g(S,A)h(S)\mid S'=s'\right]. \tag*{(116)} ( B g J h ) ( s ′ ) = E Γ g [ ϱ g ( S , A ) h ( S ) ∣ S ′ = s ′ ] . (116)
For a fixed inner iteration k k k , let M g , k J , ⋆ ( s ′ ) M^{J,\star}_{g,k}(s') M g , k J , ⋆ ( s ′ ) be the unrestricted pointwise minimizer with the slow mass M ‾ g , k J \overline{M}^J_{g,k} M g , k J held fixed. Recalling the regression identity in Equation (111), we obtain
M g , k J , ⋆ ( s ′ ) = ( 1 − γ ) + γ ( B g J M ‾ g , k J ) ( s ′ ) . (117) M^{J,\star}_{g,k}(s') = (1-\gamma) + \gamma(B^J_g\overline{M}^J_{g,k})(s'). \tag*{(117)} M g , k J , ⋆ ( s ′ ) = ( 1 − γ ) + γ ( B g J M g , k J ) ( s ′ ) . (117)
This empirical identity holds on the successor-data support and does not require D g = D g ′ D_g = D'_g D g = D g ′ .
For an ideal population reference, now replace the empirical transition law Γ g \Gamma_g Γ g by the stationary transition law of a finite Markov model. Fix policies μ \mu μ and π \pi π , assume a strictly positive stationary behavior distribution d μ d_\mu d μ and target-action support covered by μ \mu μ , and write P ( s ′ ∣ s , a ) P(s'\mid s,a) P ( s ′ ∣ s , a ) for the model’s transition probability. Define the population joint law Γ μ \Gamma_\mu Γ μ and target transition matrix P π P_\pi P π by
Γ μ ( s , a , s ′ ) = d μ ( s ) μ ( a ∣ s ) P ( s ′ ∣ s , a ) , P π ( s , s ′ ) = ∑ a π ( a ∣ s ) P ( s ′ ∣ s , a ) . (118) \Gamma_\mu(s,a,s') = d_\mu(s)\mu(a\mid s)P(s'\mid s,a), \qquad P_\pi(s,s') = \sum_a \pi(a\mid s)P(s'\mid s,a). \tag*{(118)} Γ μ ( s , a , s ′ ) = d μ ( s ) μ ( a ∣ s ) P ( s ′ ∣ s , a ) , P π ( s , s ′ ) = a ∑ π ( a ∣ s ) P ( s ′ ∣ s , a ) . (118)
Both state marginals of Γ μ \Gamma_\mu Γ μ are d μ d_\mu d μ . This replacement is a population idealization, not a consequence of equality of empirical marginals. Remove the neural parameterization and its reference-anchor constraint for this idealization. Suppress the generation and inner-iteration indices, and let M J M^J M J be the unrestricted positive fixed point of the population conditional equation
M J ( s ′ ) = ( 1 − γ ) + γ E Γ μ [ π ( A ∣ S ) μ ( A ∣ S ) M J ( S ) | S ′ = s ′ ] . (119) M^J(s') = (1-\gamma) + \gamma\mathbb{E}_{\Gamma_\mu}\left[\frac{\pi(A\mid S)}{\mu(A\mid S)}M^J(S)\,\middle|\,S'=s'\right]. \tag*{(119)} M J ( s ′ ) = ( 1 − γ ) + γ E Γ μ [ μ ( A ∣ S ) π ( A ∣ S ) M J ( S ) S ′ = s ′ ] . (119)
Define its induced state mass e J e^J e J by
e J ( s ) = d μ ( s ) M J ( s ) , or e J = d μ ⊙ M J . (120) e^J(s) = d_\mu(s)M^J(s), \qquad\text{or} \qquad e^J = d_\mu\odot M^J. \tag*{(120)} e J ( s ) = d μ ( s ) M J ( s ) , or e J = d μ ⊙ M J . (120)
This multiplication converts a per-occurrence importance multiplier into a state mass: behavior visits s s s with frequency d μ ( s ) d_\mu(s) d μ ( s ) , and each such occurrence contributes mass M J ( s ) M^J(s) M J ( s ) . For any scalar test function h h h , it follows that E d μ [ M J ( S ) h ( S ) ] = ∑ s e J ( s ) h ( s ) \mathbb{E}_{d_\mu}[M^J(S)h(S)] = \sum_s e^J(s)h(s) E d μ [ M J ( S ) h ( S )] = ∑ s e J ( s ) h ( s ) . Multiplying the population conditional fixed-point equation by d μ ( s ′ ) d_\mu(s') d μ ( s ′ ) gives
e J ( s ′ ) = ( 1 − γ ) d μ ( s ′ ) + γ ∑ s ∑ a : μ ( a ∣ s ) > 0 d μ ( s ) μ ( a ∣ s ) P ( s ′ ∣ s , a ) π ( a ∣ s ) μ ( a ∣ s ) M J ( s ) = ( 1 − γ ) d μ ( s ′ ) + γ ∑ s P π ( s , s ′ ) e J ( s ) . (121) \begin{aligned}
e^J(s') &= (1-\gamma)d_\mu(s') + \gamma\sum_s\sum_{a:\mu(a\mid s)>0}d_\mu(s)\mu(a\mid s)P(s'\mid s,a)\frac{\pi(a\mid s)}{\mu(a\mid s)}M^J(s) \\
&= (1-\gamma)d_\mu(s') + \gamma\sum_s P_\pi(s,s')e^J(s). \tag*{(121)}
\end{aligned} e J ( s ′ ) = ( 1 − γ ) d μ ( s ′ ) + γ s ∑ a : μ ( a ∣ s ) > 0 ∑ d μ ( s ) μ ( a ∣ s ) P ( s ′ ∣ s , a ) μ ( a ∣ s ) π ( a ∣ s ) M J ( s ) = ( 1 − γ ) d μ ( s ′ ) + γ s ∑ P π ( s , s ′ ) e J ( s ) . (121)
Here P P P is the population transition probability just defined. Summing over s ′ s' s ′ gives ∑ s e J ( s ) = 1 \sum_s e^J(s)=1 ∑ s e J ( s ) = 1 . Recall that state masses are column vectors, ⊤ \top ⊤ denotes transpose, and I d \mathrm{Id} Id is the identity matrix. Expanding the inverse in powers indexed by the nonnegative integer ℓ \ell ℓ yields
e J = ( 1 − γ ) ( I d − γ P π ⊤ ) − 1 d μ = ( 1 − γ ) ∑ ℓ = 0 ∞ γ ℓ ( P π ⊤ ) ℓ d μ . (122) e^J = (1-\gamma)(\mathrm{Id}-\gamma P_\pi^\top)^{-1}d_\mu= (1-\gamma)\sum_{\ell=0}^{\infty}\gamma^\ell(P_\pi^\top)^\ell d_\mu. \tag*{(122)} e J = ( 1 − γ ) ( Id − γ P π ⊤ ) − 1 d μ = ( 1 − γ ) ℓ = 0 ∑ ∞ γ ℓ ( P π ⊤ ) ℓ d μ . (122)
The ideal reference is the discounted visitation distribution of target-policy excursions beginning from d μ d_\mu d μ . The multiplier M J ( s ) = e J ( s ) / d μ ( s ) M^J(s)=e^J(s)/d_\mu(s) M J ( s ) = e J ( s ) / d μ ( s ) adjusts behavior-state frequencies toward this reference. The reference distribution generally differs from a stationary target distribution d π d_\pi d π satisfying P π ⊤ d π = d π P_\pi^\top d_\pi=d_\pi P π ⊤ d π = d π , and from a finite evaluation trajectory’s occupancy. Uniqueness of d π d_\pi d π is not needed for this reference construction.
From heuristic improvement to confidence and interest.
The P P P role receives interest from the heuristic’s estimated improvement over the parent target policy. Recall that w i J w^J_i w i J is the positive, mean-one weight of current observation occurrence i i i , obtained from the parent mass M g J M^J_g M g J by Equation (138) . Draw 16 anchor occurrences with replacement, selecting occurrence i i i with probability w i J / n w^J_i/n w i J / n ; denote them by s ˉ g , f \bar{s}_{g,f} s ˉ g , f for f = 1 , … , 16 f = 1,\ldots,16 f = 1 , … , 16 . At each anchor, run 32 paired simulations, indexed by b = 1 , … , 32 b = 1,\ldots,32 b = 1 , … , 32 , with horizon H = 1000 H = 1000 H = 1000 and period index k = 0 , … , H − 1 k = 0,\ldots,H-1 k = 0 , … , H − 1 . Recall the heuristic controller q H , g q_{H,g} q H , g defined in Equation (1) . Superscripts H H H and π \pi π denote the intervention and baseline arms, while S g , f , b , k H , S g , f , b , k π S^H_{g,f,b,k}, S^\pi_{g,f,b,k} S g , f , b , k H , S g , f , b , k π and A g , f , b , k H , A g , f , b , k π A^H_{g,f,b,k}, A^\pi_{g,f,b,k} A g , f , b , k H , A g , f , b , k π denote their observations and actions. Both arms start at S g , f , b , 0 H = S g , f , b , 0 π = s ˉ g , f S^H_{g,f,b,0}=S^\pi_{g,f,b,0}=\bar{s}_{g,f} S g , f , b , 0 H = S g , f , b , 0 π = s ˉ g , f , with action laws
A g , f , b , k H ∼ { q H , g ( ⋅ ∣ S g , f , b , k H ) , 0 ≤ k < 9 , π g ( ⋅ ∣ S g , f , b , k H ) , 9 ≤ k < H , A g , f , b , k π ∼ π g ( ⋅ ∣ S g , f , b , k π ) , 0 ≤ k < H . (123) A^H_{g,f,b,k} \sim\begin{cases} q_{H,g}(\cdot\mid S^H_{g,f,b,k}), & 0 \le k < 9, \\ \pi_g(\cdot\mid S^H_{g,f,b,k}), & 9 \le k < H, \end{cases}
\qquad
A^\pi_{g,f,b,k} \sim\pi_g(\cdot\mid S^\pi_{g,f,b,k}), \qquad0 \le k < H. \tag*{(123)} A g , f , b , k H ∼ { q H , g ( ⋅ ∣ S g , f , b , k H ) , π g ( ⋅ ∣ S g , f , b , k H ) , 0 ≤ k < 9 , 9 ≤ k < H , A g , f , b , k π ∼ π g ( ⋅ ∣ S g , f , b , k π ) , 0 ≤ k < H . (123)
Each arm advances from its own observation using the frozen fitted reward–observation kernel in Equation (40) . Within each pair, the arms use common random inputs, including the root risk draw in RB (Equation (37) ); that risk is retained throughout the pair as in Equation (41) .
Let C g , f , b , k H C^H_{g,f,b,k} C g , f , b , k H and C g , f , b , k π C^\pi_{g,f,b,k} C g , f , b , k π be the realized period costs of the two arms, and let R g , f , b , k H R^H_{g,f,b,k} R g , f , b , k H and R g , f , b , k π R^\pi_{g,f,b,k} R g , f , b , k π be their normalized rewards. Using R = − C / 495 R=-C/495 R = − C /495 from Equation (17) , define the complete paired improvement Δ g , f , b \Delta_{g,f,b} Δ g , f , b by
Δ g , f , b = ∑ k = 0 H − 1 ( C g , f , b , k π − C g , f , b , k H ) = 495 ∑ k = 0 H − 1 ( R g , f , b , k H − R g , f , b , k π ) . (124) \Delta_{g,f,b}=\sum_{k=0}^{H-1}\left(C^\pi_{g,f,b,k}-C^H_{g,f,b,k}\right)=495\sum_{k=0}^{H-1}\left(R^H_{g,f,b,k}-R^\pi_{g,f,b,k}\right). \tag*{(124)} Δ g , f , b = k = 0 ∑ H − 1 ( C g , f , b , k π − C g , f , b , k H ) = 495 k = 0 ∑ H − 1 ( R g , f , b , k H − R g , f , b , k π ) . (124)
Positive Δ g , f , b \Delta_{g,f,b} Δ g , f , b means that the finite heuristic intervention reduced total cost over the H H H simulated periods. These are complete undiscounted H H H -period returns. The common factor 495 cancels from the standardized statistic. Let Δ ‾ g , f \overline{\Delta}_{g,f} Δ g , f and σ ^ g , f 2 \widehat{\sigma}^2_{g,f} σ g , f 2 be the paired sample mean and sample variance:
Δ ‾ g , f = 1 32 ∑ b = 1 32 Δ g , f , b , σ ^ g , f 2 = 1 31 ∑ b = 1 32 ( Δ g , f , b − Δ ‾ g , f ) 2 . (125) \overline{\Delta}_{g,f}=\frac{1}{32}\sum_{b=1}^{32}\Delta_{g,f,b}, \qquad\widehat{\sigma}^2_{g,f}=\frac{1}{31}\sum_{b=1}^{32}\left(\Delta_{g,f,b}-\overline{\Delta}_{g,f}\right)^2. \tag*{(125)} Δ g , f = 32 1 b = 1 ∑ 32 Δ g , f , b , σ g , f 2 = 31 1 b = 1 ∑ 32 ( Δ g , f , b − Δ g , f ) 2 . (125)
Let σ ^ g , f \widehat{\sigma}_{g,f} σ g , f be the nonnegative square root of this sample variance and let CDF t 31 \operatorname{CDF}_{t_{31}} CDF t 31 denote the cumulative distribution function of the standard Student-t t t distribution with 31 degrees of freedom. For σ ^ g , f > 0 \widehat{\sigma}_{g,f}>0 σ g , f > 0 , define the confidence score c g , f c_{g,f} c g , f by
c g , f = CDF t 31 ( 32 Δ ‾ g , f σ ^ g , f ) . (126) c_{g,f}=\operatorname{CDF}_{t_{31}}\left(\frac{\sqrt{32}\,\overline{\Delta}_{g,f}}{\widehat{\sigma}_{g,f}}\right). \tag*{(126)} c g , f = CDF t 31 ( σ g , f 32 Δ g , f ) . (126)
At zero sample standard deviation, set c g , f c_{g,f} c g , f to 1, 1/2, or 0 according as the mean improvement is positive, zero, or negative. This Student-t t t transformation supplies a bounded directional score; no calibrated posterior-probability interpretation is required.
Recall the fixed feature vector ϕ \phi ϕ . Define the normalized Gaussian interpolation coefficients a g , f ( s ) a_{g,f}(s) a g , f ( s ) and the resulting confidence field c ^ g ( s ) \widehat{c}_g(s) c g ( s ) below, where ∥ ⋅ ∥ 2 \lVert\cdot\rVert_2 ∥ ⋅ ∥ 2 is the Euclidean norm and h h h in the denominator is a dummy anchor index:
a g , f ( s ) = exp [ − ∥ ϕ ( s ) − ϕ ( s ˉ g , f ) ∥ 2 2 / 2 ] ∑ h = 1 16 exp [ − ∥ ϕ ( s ) − ϕ ( s ˉ g , h ) ∥ 2 2 / 2 ] , c ^ g ( s ) = ∑ f = 1 16 a g , f ( s ) c g , f . (127) a_{g,f}(s)=\frac{\exp\left[-\lVert\phi(s)-\phi(\bar{s}_{g,f})\rVert_2^2/2\right]}{\sum_{h=1}^{16}\exp\left[-\lVert\phi(s)-\phi(\bar{s}_{g,h})\rVert_2^2/2\right]},
\qquad
\widehat{c}_g(s)=\sum_{f=1}^{16}a_{g,f}(s)c_{g,f}. \tag*{(127)} a g , f ( s ) = ∑ h = 1 16 exp [ − ∥ ϕ ( s ) − ϕ ( s ˉ g , h ) ∥ 2 2 /2 ] exp [ − ∥ ϕ ( s ) − ϕ ( s ˉ g , f ) ∥ 2 2 /2 ] , c g ( s ) = f = 1 ∑ 16 a g , f ( s ) c g , f . (127)
Let i g P i^P_g i g P be the transmission interest. Recall that D g ′ D'_g D g ′ is the empirical measure of all successor observations s i ′ s'_i s i ′ . Centering the confidence field over this complete successor data gives
i g P ( s ) = 1 + c ^ g ( s ) − E D g ′ [ c ^ g ] , E D g ′ [ c ^ g ] = 1 n ∑ i = 1 n c ^ g ( s i ′ ) . (128) i^P_g(s)=1+\widehat{c}_g(s)-\mathbb{E}_{D'_g}[\widehat{c}_g], \qquad\mathbb{E}_{D'_g}[\widehat{c}_g]=\frac{1}{n}\sum_{i=1}^{n}\widehat{c}_g(s'_i). \tag*{(128)} i g P ( s ) = 1 + c g ( s ) − E D g ′ [ c g ] , E D g ′ [ c g ] = n 1 i = 1 ∑ n c g ( s i ′ ) . (128)
Hence E D g ′ [ i g P ] = 1 \mathbb{E}_{D'_g}[i^P_g]=1 E D g ′ [ i g P ] = 1 . For finite features, all interpolation coefficients are strictly positive. If the anchor scores differ, each interpolated score lies strictly between their minimum and maximum; if all scores coincide, the field is constant. Together with c g , f ∈ [ 0 , 1 ] c_{g,f}\in[0,1] c g , f ∈ [ 0 , 1 ] , this proves i g P ( s ) > 0 i^P_g(s)>0 i g P ( s ) > 0 in exact arithmetic. The interest emphasizes locations with confidence above the successor-data average. Constant confidence, including uniformly high confidence, gives i g P = 1 i^P_g=1 i g P = 1 .
Transmission mass M P M^P M P .
The transmission channel requires a different notion of state importance. Behavior visitation describes where experience was collected, but it does not indicate where a finite heuristic intervention has stronger evidence of improving on the parent target. The P P P mass converts that spatial variation in confidence into persistent state emphasis for transmission. Its comparison is between behavior visitation and a confidence-weighted reference propagated under the behavior policy. It does not estimate visitation under q H , g q_{H,g} q H , g or directly measure behavior–target action disagreement; the latter enters the separate transmission objective.
Recall that M g P M_g^P M g P is the frozen parent transmission mass and that D g D_g D g averages over current observations s i s_i s i . Define its normalized multiplier R g P R_g^P R g P by
R g P ( s ) = M g P ( s ) E D g [ M g P ] , E D g [ R g P ] = 1. (129) R_g^P(s)=\frac{M_g^P(s)}{\mathbb{E}_{D_g}[M_g^P]},\qquad\mathbb{E}_{D_g}[R_g^P]=1. \tag*{(129)} R g P ( s ) = E D g [ M g P ] M g P ( s ) , E D g [ R g P ] = 1. (129)
Before optimization, construct an unnormalized label Y ~ g , i P \widetilde{Y}_{g,i}^P Y g , i P and its full-data mean-one normalization Y g , i P Y_{g,i}^P Y g , i P for every transition occurrence i i i . Freeze this complete label vector throughout the fit:
Y ~ g , i P = i g P ( s i ′ ) + γ R g P ( s i ) , Y g , i P = Y ~ g , i P n − 1 ∑ j = 1 n Y ~ g , j P . (130) \widetilde{Y}_{g,i}^P=i_g^P(s_i')+\gamma R_g^P(s_i),\qquad Y_{g,i}^P=\frac{\widetilde{Y}_{g,i}^P}{n^{-1}\sum_{j=1}^{n}\widetilde{Y}_{g,j}^P}. \tag*{(130)} Y g , i P = i g P ( s i ′ ) + γ R g P ( s i ) , Y g , i P = n − 1 ∑ j = 1 n Y g , j P Y g , i P . (130)
Because interest and normalized parent mass have unit mean on their respective data marginals, exact arithmetic gives
Y g , i P = i g P ( s i ′ ) + γ R g P ( s i ) 1 + γ . (131) Y_{g,i}^P=\frac{i_g^P(s_i')+\gamma R_g^P(s_i)}{1+\gamma}. \tag*{(131)} Y g , i P = 1 + γ i g P ( s i ′ ) + γ R g P ( s i ) . (131)
The implementation computes the full-data denominator explicitly. This label contains no action importance ratio and no evolving slow network: it combines successor interest with normalized predecessor mass carried along behavior transitions. For a scalar state function h h h , define the empirical reverse-transition operator B g P \mathcal{B}_g^P B g P at each observed successor by
( B g P h ) ( s ′ ) = E Γ g [ h ( S ) ∣ S ′ = s ′ ] . (132) (\mathcal{B}_g^P h)(s')=\mathbb{E}_{\Gamma_g}[h(S)\mid S'=s']. \tag*{(132)} ( B g P h ) ( s ′ ) = E Γ g [ h ( S ) ∣ S ′ = s ′ ] . (132)
Let M g P , ⋆ M_g^{P,\star} M g P , ⋆ be the unrestricted conditional regression target. By Equation (111) , on the successor-data support it is
M g P , ⋆ ( s ′ ) = i g P ( s ′ ) + γ ( B g P R g P ) ( s ′ ) 1 + γ . (133) M_g^{P,\star}(s')=\frac{i_g^P(s')+\gamma(\mathcal{B}_g^P R_g^P)(s')}{1+\gamma}. \tag*{(133)} M g P , ⋆ ( s ′ ) = 1 + γ i g P ( s ′ ) + γ ( B g P R g P ) ( s ′ ) . (133)
The online network starts from the parent parameters and is fitted with Equation (109) using the fixed labels Y i X = Y g , i P Y_i^X=Y_{g,i}^P Y i X = Y g , i P . Recall the reference observation s ∘ s_\circ s ∘ in Equation (108) ; the successor reference value is set to β g + 1 P = i g P ( s ∘ ) \beta_{g+1}^P=i_g^P(s_\circ) β g + 1 P = i g P ( s ∘ ) . This is an imposed anchor. If s ∘ s_\circ s ∘ occurs among the observed successors, it need not equal the unrestricted conditional target there.
For the ideal population reference, recall the stationary law Γ μ \Gamma_\mu Γ μ and positive behavior distribution d μ d_\mu d μ in Equation (118) . Define the behavior-policy transition matrix P μ ( s , s ′ ) = ∑ a μ ( a ∣ s ) P ( s ′ ∣ s , a ) P_\mu(s,s')=\sum_a\mu(a\mid s)P(s'\mid s,a) P μ ( s , s ′ ) = ∑ a μ ( a ∣ s ) P ( s ′ ∣ s , a ) . Freeze a positive interest function i P i^P i P normalized by E d μ [ i P ] = 1 \mathbb{E}_{d_\mu}[i^P]=1 E d μ [ i P ] = 1 , and replace the empirical conditional expectations by expectations under Γ μ \Gamma_\mu Γ μ . For any normalized predecessor multiplier, the resulting unrestricted conditional target has mean ( 1 + γ ) / ( 1 + γ ) = 1 (1+\gamma)/(1+\gamma)=1 ( 1 + γ ) / ( 1 + γ ) = 1 under d μ d_\mu d μ . At an exact fixed point, therefore, the raw output equals its normalized population target, which we denote by e P e^P e P . Define the associated state mass e P ( s ) = d μ ( s ) R P ( s ) e^P(s)=d_\mu(s)R^P(s) e P ( s ) = d μ ( s ) R P ( s ) . Multiplication of the population conditional fixed-point equation by d μ ( s ′ ) d_\mu(s') d μ ( s ′ ) gives
e P = d μ ⊙ i P + γ P μ T e P 1 + γ . (134) e^P=\frac{d_\mu\odot i^P+\gamma P_\mu^{\mathsf T}e^P}{1+\gamma}. \tag*{(134)} e P = 1 + γ d μ ⊙ i P + γ P μ T e P . (134)
Define the effective propagation factor λ P = γ / ( 1 + γ ) \lambda_P=\gamma/(1+\gamma) λ P = γ / ( 1 + γ ) , so 1 − λ P = 1 / ( 1 + γ ) 1-\lambda_P=1/(1+\gamma) 1 − λ P = 1/ ( 1 + γ ) . This is the factor induced by the P P P -label normalization, not an additional eligibility-trace parameter. Solving with the identity matrix I d \mathrm{Id} Id gives
e P = ( 1 − λ P ) ( I d − λ P P μ T ) − 1 ( d μ ⊙ i P ) . (135) e^P=(1-\lambda_P)(\mathrm{Id}-\lambda_P P_\mu^{\mathsf T})^{-1}(d_\mu\odot i^P). \tag*{(135)} e P = ( 1 − λ P ) ( Id − λ P P μ T ) − 1 ( d μ ⊙ i P ) . (135)
This reference starts from interest-weighted behavior states and propagates under μ \mu μ . In this idealization, R P = e P / d μ R^{P}=e^{P}/d_{\mu} R P = e P / d μ describes the corresponding change in state importance. Unit interest has the neutral fixed point R P ≡ 1 R^{P}\equiv1 R P ≡ 1 , irrespective of behavior–target policy disagreement.
Both mass learners use γ = 0.995 \gamma=0.995 γ = 0.995 and 1000 Adam updates with minibatches of 256 uniformly sampled transition occurrences, with replacement. Each role has fresh optimizer state and the configured emphatic learning rate; the Adam coefficients are those specified in Section B.2.7. All labels are detached during differentiation. The J J J labels evolve through its private slow network, while the P P P labels remain fixed throughout its fit.
Deriving consumer weights from the masses.
Recall the frozen parent raw masses M g χ M_g^{\chi} M g χ for χ ∈ { J , P } \chi\in\{J,P\} χ ∈ { J , P } and the current-observation empirical measure D g D_g D g . Let m ‾ g χ \overline{m}_g^{\chi} m g χ be the full-data mean mass, R g χ R_g^{\chi} R g χ the normalized mass multiplier (agreeing with Equation (129) for P P P ), and v M , g χ v_{M,g}^{\chi} v M , g χ its mean squared deviation from one:
m ‾ g χ = E D g [ M g χ ] , R g χ ( s ) = M g χ ( s ) m ‾ g χ , v M , g χ = E D g [ ( R g χ − 1 ) 2 ] . (136) \overline{m}_g^{\chi}=\mathbb{E}_{D_g}[M_g^{\chi}],\qquad R_g^{\chi}(s)=\frac{M_g^{\chi}(s)}{\overline{m}_g^{\chi}},\qquad v_{M,g}^{\chi}=\mathbb{E}_{D_g}[(R_g^{\chi}-1)^2]. \tag*{(136)} m g χ = E D g [ M g χ ] , R g χ ( s ) = m g χ M g χ ( s ) , v M , g χ = E D g [( R g χ − 1 ) 2 ] . (136)
Since E D g [ R g χ ] = 1 \mathbb{E}_{D_g}[R_g^{\chi}]=1 E D g [ R g χ ] = 1 , its second moment equals 1 + v M , g χ 1+v_{M,g}^{\chi} 1 + v M , g χ . Define the shrinkage coefficient α g χ \alpha_g^{\chi} α g χ and the consumer weight function w g χ w_g^{\chi} w g χ by
α g χ = ( 1 + v M , g χ ) − 1 / 2 , w g χ ( s ) = 1 + α g χ ( R g χ ( s ) − 1 ) . (137) \alpha_g^{\chi}=(1+v_{M,g}^{\chi})^{-1/2},\qquad w_g^{\chi}(s)=1+\alpha_g^{\chi}(R_g^{\chi}(s)-1). \tag*{(137)} α g χ = ( 1 + v M , g χ ) − 1/2 , w g χ ( s ) = 1 + α g χ ( R g χ ( s ) − 1 ) . (137)
Substitution gives the main-text expression (Equation (12) ):
w g χ ( s ) = 1 + M g χ ( s ) − E D g [ M g χ ] E D g [ ( M g χ ) 2 ] , χ ∈ { J , P } . (138) w_g^{\chi}(s)=1+\frac{M_g^{\chi}(s)-\mathbb{E}_{D_g}[M_g^{\chi}]}{\sqrt{\mathbb{E}_{D_g}[(M_g^{\chi})^2]}},\qquad\chi\in\{J,P\}. \tag*{(138)} w g χ ( s ) = 1 + E D g [( M g χ ) 2 ] M g χ ( s ) − E D g [ M g χ ] , χ ∈ { J , P } . (138)
The moments are computed once over all current-state occurrences and remain fixed during subsequent minibatch computations.
For finite positive masses, 0 < α g χ ≤ 1 0<\alpha_g^{\chi}\leq1 0 < α g χ ≤ 1 , so
w g χ = ( 1 − α g χ ) + α g χ R g χ > 0 , E D g [ w g χ ] = 1. (139) w_g^{\chi}=(1-\alpha_g^{\chi})+\alpha_g^{\chi}R_g^{\chi}>0,\qquad\mathbb{E}_{D_g}[w_g^{\chi}]=1. \tag*{(139)} w g χ = ( 1 − α g χ ) + α g χ R g χ > 0 , E D g [ w g χ ] = 1. (139)
Multiplying M g χ M_g^{\chi} M g χ by a positive constant leaves R g χ R_g^{\chi} R g χ and w g χ w_g^{\chi} w g χ unchanged. Moreover,
E D g [ ( w g χ − 1 ) 2 ] = v M , g χ 1 + v M , g χ < 1. (140) \mathbb{E}_{D_g}[(w_g^{\chi}-1)^2]=\frac{v_{M,g}^{\chi}}{1+v_{M,g}^{\chi}}<1. \tag*{(140)} E D g [( w g χ − 1 ) 2 ] = 1 + v M , g χ v M , g χ < 1. (140)
Let w i χ = w g χ ( s i ) w_i^{\chi}=w_g^{\chi}(s_i) w i χ = w g χ ( s i ) and let w χ = ( w 1 χ , … , w n χ ) \mathbf{w}^{\chi}=(w_1^{\chi},\ldots,w_n^{\chi}) w χ = ( w 1 χ , … , w n χ ) be the vector of occurrence weights. Its algebraic effective sample size, denoted by ESS ( w χ ) \operatorname{ESS}(\mathbf{w}^{\chi}) ESS ( w χ ) , is
ESS ( w χ ) = ( ∑ i w i χ ) 2 ∑ i ( w i χ ) 2 = n 1 + v M , g χ 1 + 2 v M , g χ > n 2 . (141) \operatorname{ESS}(\mathbf{w}^{\chi})=\frac{\left(\sum_i w_i^{\chi}\right)^2}{\sum_i(w_i^{\chi})^2}=n\frac{1+v_{M,g}^{\chi}}{1+2v_{M,g}^{\chi}}>\frac{n}{2}. \tag*{(141)} ESS ( w χ ) = ∑ i ( w i χ ) 2 ( ∑ i w i χ ) 2 = n 1 + 2 v M , g χ 1 + v M , g χ > 2 n . (141)
These identities establish positivity, normalization, and controlled dispersion of the consumer weights. Scale invariance here concerns the consumer map; the J J J fitting rule is not scale invariant because its additive source term 1 − γ 1-\gamma 1 − γ remains fixed when raw mass is rescaled.
Induced measures and their use in updating the critic.
Recall that w i χ = w g χ ( s i ) w_i^{\chi}=w_g^{\chi}(s_i) w i χ = w g χ ( s i ) is the frozen weight of current occurrence i i i . Let ν g χ \nu_g^{\chi} ν g χ be the probability measure induced by these weights on the observed roots. Together with the unweighted empirical measure D g D_g D g , it is
D g = 1 n ∑ i = 1 n δ s i , ν g χ = 1 n ∑ i = 1 n w i χ δ s i , χ ∈ { J , P } . (142) D_g=\frac{1}{n}\sum_{i=1}^{n}\delta_{s_i},\qquad\nu_g^{\chi}=\frac{1}{n}\sum_{i=1}^{n}w_i^{\chi}\delta_{s_i},\qquad\chi\in\{J,P\}. \tag*{(142)} D g = n 1 i = 1 ∑ n δ s i , ν g χ = n 1 i = 1 ∑ n w i χ δ s i , χ ∈ { J , P } . (142)
Sampling from ν g X \nu_g^X ν g X means selecting occurrence i i i with probability w i X / n w_i^X/n w i X / n . For any real-valued function f f f that is finite on the observed roots,
E S ∼ ν g X [ f ( S ) ] = 1 n ∑ i = 1 n w i X f ( s i ) . (143) \mathbb{E}_{S\sim\nu_g^X}[f(S)] = \frac{1}{n}\sum_{i=1}^{n}w_i^X f(s_i). \tag*{(143)} E S ∼ ν g X [ f ( S )] = n 1 i = 1 ∑ n w i X f ( s i ) . (143)
Recall the normalized mass multiplier R g X R_g^X R g X and shrinkage coefficient α g X \alpha_g^X α g X in Equations (136) and (137) . Define the normalized-mass probability measure ν ˉ g X = n − 1 ∑ i R g X ( s i ) δ s i \bar{\nu}_g^X = n^{-1}\sum_i R_g^X(s_i)\delta_{s_i} ν ˉ g X = n − 1 ∑ i R g X ( s i ) δ s i . The consumer measure then satisfies
ν g X = ( 1 − α g X ) D g + α g X ν ˉ g X . (144) \nu_g^X = (1-\alpha_g^X)D_g+\alpha_g^X\bar{\nu}_g^X. \tag*{(144)} ν g X = ( 1 − α g X ) D g + α g X ν ˉ g X . (144)
The induced measure is an explicit mixture of the empirical behavior measure and the normalized-mass measure.
Emphasis enters the successor-critic update through the distribution of its training states. Recall from Section B.2.5 that a root-policy pair is chosen uniformly from ( D g , μ g ) (D_g,\mu_g) ( D g , μ g ) , ( ν g J , π g ) (\nu_g^J,\pi_g) ( ν g J , π g ) , and ( ν g P , μ g ) (\nu_g^P,\mu_g) ( ν g P , μ g ) . The augmented root measures add an independent root risk in RB (Equation (75) ). Recall that η ~ g \tilde{\eta}_g η ~ g is the augmented state law after the first consumer transition, and ζ ~ g \tilde{\zeta}_g ζ ~ g is the law after an additional geometric number of target-policy transitions (Equations (77) and (78) ). The resulting training bank contains n ζ = 25,000 n_\zeta=25{,}000 n ζ = 25 , 000 pairs ( S ℓ , U ℓ ) (S_\ell,U_\ell) ( S ℓ , U ℓ ) , indexed by ℓ = 1 , … , n ζ \ell=1,\ldots,n_\zeta ℓ = 1 , … , n ζ , where S ℓ S_\ell S ℓ is the retained observation and U ℓ U_\ell U ℓ is the additional next-availability draw in Equation (80) . Recall that V θ V_\theta V θ is a live scalar critic with parameter vector θ \theta θ , and that T g u T_g^u T g u denotes its target-policy Bellman backup conditional on next availability u u u . The prediction-minus-backup fitting error is ε g ( θ ; s , u ) = V θ ( s ) − ( T g u V θ ) ( s ) \varepsilon_g(\theta;s,u)=V_\theta(s)-(T_g^uV_\theta)(s) ε g ( θ ; s , u ) = V θ ( s ) − ( T g u V θ ) ( s ) , as defined in Equation (84) . The empirical fitting objective J ^ V , g ( θ ) \widehat{J}_{V,g}(\theta) J V , g ( θ ) is
J ^ V , g ( θ ) = 1 n ζ ∑ ℓ = 1 n ζ ε g ( θ ; S ℓ , U ℓ ) 2 , (145) \widehat{J}_{V,g}(\theta)=\frac{1}{n_\zeta}\sum_{\ell=1}^{n_\zeta}\varepsilon_g(\theta;S_\ell,U_\ell)^2, \tag*{(145)} J V , g ( θ ) = n ζ 1 ℓ = 1 ∑ n ζ ε g ( θ ; S ℓ , U ℓ ) 2 , (145)
which is Equation (82) . Running the prescribed finite optimization produces each successor member V g + 1 , j V_{g+1,j} V g + 1 , j , for j = 1 , … , 5 j=1,\ldots,5 j = 1 , … , 5 , and their output mean v g + 1 v_{g+1} v g + 1 (Equation (88) ). Thus emphatic weights affect where successor critics are fitted. Their effect is incorporated through root selection and continuation sampling; the training-bank loss is not multiplied by w i X w_i^X w i X a second time.
The same frozen weights enter the target-return and transmission update channels. The induced measures ν g J \nu_g^J ν g J and ν g P \nu_g^P ν g P also enter the target actor’s shared Fisher geometry and its two channelwise endpoint constraints (§ 4.2 ). Recall from Equation (184) that ϖ i = ( 1 + w i J + w i P ) / ( 3 n ) \varpi_i=(1+w_i^J+w_i^P)/(3n) ϖ i = ( 1 + w i J + w i P ) / ( 3 n ) is the occurrence probability for their shared residual-root proposal; the corresponding importance factors are given in Equation (186) . Finally, ν g J \nu_g^J ν g J selects the heuristic anchors used above, linking current target-return emphasis to future confidence-based transmission emphasis.
Neutral mass and weight initialization.
Both initialization routes start from neutral masses, M 0 J = M 0 P = 1 M_0^J=M_0^P=1 M 0 J = M 0 P = 1 , so w 0 J = w 0 P = 1 w_0^J=w_0^P=1 w 0 J = w 0 P = 1 and ν 0 J = ν 0 P = D 0 \nu_0^J=\nu_0^P=D_0 ν 0 J = ν 0 P = D 0 .
Cross-generation evolution and two-generation transmission.
During generation g g g , parent masses are evaluated on the newly collected data D g D_g D g . Their consumer weights remain frozen while successor actors, critics, and masses are constructed; actor differentiation treats the weights as constants. The confidence-to-transmission-mass pathway is
q H , g ⟶ ( c g , f ) f = 1 16 ⟶ i g P ⟶ M g + 1 P ⟶ ( w g + 1 P , ν g + 1 P ) on D g + 1 ⟶ π g + 2 . (146) q_{H,g}\longrightarrow(c_{g,f})_{f=1}^{16}\longrightarrow i_g^P\longrightarrow M_{g+1}^P\longrightarrow(w_{g+1}^P,\nu_{g+1}^P)\text{ on }D_{g+1}\longrightarrow\pi_{g+2}. \tag*{(146)} q H , g ⟶ ( c g , f ) f = 1 16 ⟶ i g P ⟶ M g + 1 P ⟶ ( w g + 1 P , ν g + 1 P ) on D g + 1 ⟶ π g + 2 . (146)
These arrows indicate computational dependence, including the paired comparison with π g \pi_g π g used to obtain c g , f c_{g,f} c g , f . Interest i g P i_g^P i g P helps fit M g + 1 P M_{g+1}^P M g + 1 P on D g D_g D g , while the update producing π g + 1 \pi_{g+1} π g + 1 uses M g P M_g^P M g P . After the joint successor commit, M g + 1 M_{g+1} M g + 1 is evaluated and normalized on D g + 1 D_{g+1} D g + 1 ; its weights can first affect the target update producing π g + 2 \pi_{g+2} π g + 2 .
Weights change both because mass functions are learned and because the behavior data used to compute their moments change. Even an unchanged mass can induce different weights on a new data set. Generation-g g g training labels and normalization constants are therefore not reused as generation-( g + 1 ) (g+1) ( g + 1 ) consumer weights. The J J J mass follows the same parent-consumption/successor-fitting schedule, although its fitting labels evolve within a generation through the private slow network.
Design interpretation and scope of the claims.
The two roles allocate learning effort to different state references: target-policy continuation from behavior-distributed states for J J J , and confidence-weighted behavior continuation for P P P . In the implemented algorithm, these references motivate the labels rather than identify the fitted networks with exact state-distribution ratios. The resulting weights enter the target-update objectives, shared Fisher geometry, and endpoint constraints; they also determine critic-training root selection and select heuristic-confidence anchors. Equations (139) –(141) prove positivity, unit mean, and bounded occurrence-weight concentration. Equation (146) specifies the earliest target update reachable through newly learned confidence-based mass.
The stationary references in Equations (122) and (135) are population fixed points under Γ μ \Gamma_\mu Γ μ in Equation (118) . They require fixed policies, sufficient action support, a sufficient Markov state, a positive stationary behavior distribution, and unrestricted exact regression; the P P P reference also requires fixed normalized interest. Equality of two empirical state marginals alone does not imply these population equations. Finite trajectory data, anchored neural approximation, finite optimization, and evolving generations need not realize these references. The empirical conditional regression identities remain valid on the observed successor support for their declared frozen labels, without implying that the fitted networks attain those targets.
These results establish controlled reweighting and the stated computational dependencies. They do not by themselves establish exact occupancy correction, monotone reduction of visitation mismatch, or improved return in every generation. The ESS bound concerns algebraic weight concentration and does not remove temporal dependence in the data. The confidence score summarizes standardized simulated improvement under the frozen fitted kernel in Equation (40) ; its calibration and transfer to actual returns require separate evidence. ETD’s linear expected-update stability result and ACE’s exact-gradient theorem do not automatically extend to HBPO’s fitted, shrunken, and delayed weights.
Heuristic Guidance: Construction, Estimation, and Behavior Fitting
Role of heuristic guidance and frozen inputs
At generation g g g , heuristic guidance converts long-horizon action comparisons under the frozen simulation model P ~ g \widetilde{P}_g P g into a temporary stochastic controller q H , g q_{H,g} q H , g . The parent target policy π g \pi_g π g supplies both the continuation policy used to evaluate actions and the reference distribution used to regularize the controller. Recall the five frozen scalar critic members V g , j V_{g,j} V g , j , j ∈ { 1 , … , 5 } j \in\{1,\ldots,5\} j ∈ { 1 , … , 5 } , and their output mean v g ( s ) = 1 5 ∑ j = 1 5 V g , j ( s ) v_g(s)=\frac{1}{5}\sum_{j=1}^{5}V_{g,j}(s) v g ( s ) = 5 1 ∑ j = 1 5 V g , j ( s ) , which supplies a control variate. The emphatic probability measure ν g J \nu_g^J ν g J , recalled below, selects the starting observations. The model, parent policies, reference critic, and emphatic weights remain fixed while the controller is constructed.
Heuristic guidance acts on the behavior policy μ \mu μ , which collects the next generation of interaction data. We therefore define a behavior-fitting objective H g H_g H g that attracts μ \mu μ toward q H , g q_{H,g} q H , g . The target policy π g \pi_g π g determines the controller’s preferences; the parameters differentiated in H g H_g H g are those of θ H \theta^H θ H .
Throughout this subsection, T = 1000 T=1000 T = 1000 is the heuristic evaluation horizon, denoted by H H H in the notation table’s heuristic-utility entries. In DSIS-RB, abbreviated RB below, each simulated continuation retains one synthetic supply–risk draw. DSIS uses the corresponding construction without a latent-risk coordinate.
The resulting controller provides action-distribution targets for the behavior policy. It also supplies evidence for the finite-intervention confidence used to fit the next generation’s transmission emphatic mass M g + 1 P M^P_{g+1} M g + 1 P . Here, paired returns compare two T T T -step simulations starting from the same anchor observation (a starting state selected as described below) and sharing their exogenous and action-sampling random inputs, including the root risk in RB: one follows q H , g q_{H,g} q H , g at stages 0 , … , 8 0,\ldots,8 0 , … , 8 and π g \pi_g π g thereafter, whereas the other follows π g \pi_g π g throughout. Their return difference measures the simulated gain from this finite intervention. The confidence and interest construction is specified in Appendix B.3.5. Here we develop the action preferences and the behavior-guidance gradient. The controller has no separately trained actor parameters.
Selecting anchor states
Let n n n be the number of collected observation occurrences and s i s_i s i the observation at occurrence i ∈ { 1 , … , n } i \in\{1,\ldots,n\} i ∈ { 1 , … , n } . Write δ x \delta_x δ x for unit point mass at observation x x x . Recall the empirical measure D g = n − 1 ∑ i = 1 n δ s i D_g = n^{-1}\sum_{i=1}^n \delta_{s_i} D g = n − 1 ∑ i = 1 n δ s i and the positive consumer weights w i J w_i^J w i J for the target-return role, normalized so that n − 1 ∑ i w i J = 1 n^{-1}\sum_i w_i^J = 1 n − 1 ∑ i w i J = 1 . The heuristic uses the already defined measure
ν g J = 1 n ∑ i = 1 n w i J δ s i . (147) \nu_g^J = \frac{1}{n}\sum_{i=1}^n w_i^J \delta_{s_i}. \tag*{(147)} ν g J = n 1 i = 1 ∑ n w i J δ s i . (147)
At each generation, we sample 16 observation occurrences independently with replacement. Let f ∈ { 1 , … , 16 } f \in\{1,\ldots,16\} f ∈ { 1 , … , 16 } index the selections, I g , f ∈ { 1 , … , n } I_{g,f} \in\{1,\ldots,n\} I g , f ∈ { 1 , … , n } be the selected occurrence index, and s ˉ g , f \bar{s}_{g,f} s ˉ g , f its anchor observation:
Pr ( I g , f = i ) = w i J n , s ˉ g , f = s I g , f , f = 1 , … , 16. (148) \Pr(I_{g,f}=i)=\frac{w_i^J}{n},\qquad\bar{s}_{g,f}=s_{I_{g,f}},\qquad f=1,\ldots,16. \tag*{(148)} Pr ( I g , f = i ) = n w i J , s ˉ g , f = s I g , f , f = 1 , … , 16. (148)
Anchors may occur anywhere in the collected trajectories; they are not restricted to episode starts. Repeated observations and repeated selections retain their multiplicity. These are the same anchor occurrences used by the confidence construction.
Every controller query, whether at an anchor or at a subsequently simulated state, starts a new local evaluation clock at k = 0 k=0 k = 0 . Its utility horizon is always T = 1000 T=1000 T = 1000 , independently of the queried observation’s position in its originating trajectory.
The return, centered utility, and simulation tapes
We first expand the expectation E H \mathbb{E}_H E H used in § 4.1 . For a fixed synthetic risk z ∈ [ 0 , 1 / 2 ] z \in[0,1/2] z ∈ [ 0 , 1/2 ] , recall that P ^ g , z \widehat{P}_{g,z} P g , z is the frozen joint reward–observation kernel conditional on that risk. Let S k S_k S k , A k A_k A k , and R k R_k R k denote the observation, action, and normalized reward at local stage k k k , and let E z s , a \mathbb{E}_z^{s,a} E z s , a denote expectation under the rollout law
S 0 = s , A 0 = a , ( R k , S k + 1 ) ∼ P ^ g , z ( ⋅ ∣ S k , A k ) , 0 ≤ k < T , A k ∼ π g ( ⋅ ∣ S k ) , 1 ≤ k < T . (149) \begin{aligned}
S_0 &= s,\qquad A_0=a,\\
(R_k,S_{k+1}) &\sim\widehat{P}_{g,z}(\cdot\mid S_k,A_k),\qquad0\leq k<T,\\
A_k &\sim\pi_g(\cdot\mid S_k),\qquad1\leq k<T. \tag*{(149)}
\end{aligned} S 0 ( R k , S k + 1 ) A k = s , A 0 = a , ∼ P g , z ( ⋅ ∣ S k , A k ) , 0 ≤ k < T , ∼ π g ( ⋅ ∣ S k ) , 1 ≤ k < T . (149)
Let m = Unif [ 0 , 1 / 2 ] m=\operatorname{Unif}[0,1/2] m = Unif [ 0 , 1/2 ] be the synthetic-risk prior. For any integrable rollout functional F \mathcal{F} F , the RB expectation is
E H [ F ∣ s , a ] = ∫ [ 0 , 1 / 2 ] E z s , a [ F ] m ( d z ) . (150) \mathbb{E}_H[\mathcal{F}\mid s,a]=\int_{[0,1/2]}\mathbb{E}_z^{s,a}[\mathcal{F}]\,m(\mathrm{d}z). \tag*{(150)} E H [ F ∣ s , a ] = ∫ [ 0 , 1/2 ] E z s , a [ F ] m ( d z ) . (150)
The root risk is drawn independently of the query observation and retained for the entire continuation; it is not inferred from that observation or redrawn at each stage. The kernel and its risk convention are specified in Appendix B.1.3. For DSIS, omit z z z and the integral over m m m , and use its frozen simulation kernel. The finite return in the main text is therefore
G T , g ( s , a ) = E H [ ∑ k = 0 T − 1 R k ∣ s , a ] . (151) G_{T,g}(s,a)=\mathbb{E}_H\left[\sum_{k=0}^{T-1}R_k\mid s,a\right]. \tag*{(151)} G T , g ( s , a ) = E H [ k = 0 ∑ T − 1 R k ∣ s , a ] . (151)
For RB, this includes the risk integral in Equation (150); for DSIS it uses the risk-free rollout law. Write c k c_k c k for the period cost on the simulated trajectory. Rewards retain the task’s normalization R k = − c k / 495 R_k=-c_k/495 R k = − c k /495 , and the return is undiscounted.
Recall that A = { 0 , … , 8 } 2 \mathcal{A}=\{0,\ldots,8\}^2 A = { 0 , … , 8 } 2 is the set of 81 regular/expedited order pairs. Let G ^ T , g ( s , a ) \widehat{G}_{T,g}(s,a) G T , g ( s , a ) denote the sample estimate of this return constructed below. The centered estimated utility, denoted by u g u_g u g in the main text and h g h_g h g in the notation table’s heuristic-utility entries, is
u g ( s , a ) = h g ( s , a ) = G ^ T , g ( s , a ) − ∑ a ′ ∈ A π g ( a ′ ∣ s ) G ^ T , g ( s , a ′ ) . (152) u_g(s,a)=h_g(s,a)=\widehat{G}_{T,g}(s,a)-\sum_{a'\in\mathcal{A}}\pi_g(a'\mid s)\widehat{G}_{T,g}(s,a'). \tag*{(152)} u g ( s , a ) = h g ( s , a ) = G T , g ( s , a ) − a ′ ∈ A ∑ π g ( a ′ ∣ s ) G T , g ( s , a ′ ) . (152)
Thus ∑ a ∈ A π g ( a ∣ s ) u g ( s , a ) = 0 \sum_{a\in\mathcal{A}}\pi_g(a\mid s)u_g(s,a)=0 ∑ a ∈ A π g ( a ∣ s ) u g ( s , a ) = 0 . We next construct the coupled simulations and the critic-based estimator that supplies G ^ T , g \widehat{G}_{T,g} G T , g .
A simulation tape records the random inputs used to generate a continuation. A trajectory records the resulting observations, actions, and rewards. A trajectory thus depends on the starting state, policy, and model as well as the tape. We use tapes so that alternative first actions can be compared using common random inputs, rather than generating an independent collection of inputs for every alternative. This couples the simulation noise in the comparisons and permits identical continuation work to be shared.
Under the ideal sampling law, each fixed query observation uses 32 independent tapes, indexed by b ∈ { 1 , … , 32 } b \in\{1,\ldots,32\} b ∈ { 1 , … , 32 } . In RB, let Z b ∼ m Z_b \sim m Z b ∼ m be tape b b b ’s root risk, and let U b , k D , U b , k B ∼ Unif ( 0 , 1 ) U^{D}_{b,k}, U^{B}_{b,k} \sim\operatorname{Unif}(0,1) U b , k D , U b , k B ∼ Unif ( 0 , 1 ) be its demand and next-availability inputs. For each action a ′ ∈ A a' \in\mathcal{A} a ′ ∈ A , let ϵ b , k , a ′ \epsilon_{b,k,a'} ϵ b , k , a ′ be a standard Gumbel variable, with cumulative distribution function x ↦ exp ( − e − x ) x \mapsto\exp(-e^{-x}) x ↦ exp ( − e − x ) , and write ϵ b , k = ( ϵ b , k , a ′ ) a ′ ∈ A \epsilon_{b,k}=(\epsilon_{b,k,a'})_{a'\in\mathcal{A}} ϵ b , k = ( ϵ b , k , a ′ ) a ′ ∈ A . These primitive inputs are mutually independent across tapes, stages, action coordinates, and input types, including independence from the risks. A utility-evaluation tape t b t_b t b is
t b = ( Z b , { U b , k D , U b , k B , ϵ b , k } k = 0 T − 1 ) . (153) t_b = \left(Z_b,\{U^{D}_{b,k},U^{B}_{b,k},\epsilon_{b,k}\}_{k=0}^{T-1}\right). \tag*{(153)} t b = ( Z b , { U b , k D , U b , k B , ϵ b , k } k = 0 T − 1 ) . (153)
The action noise at k = 0 k=0 k = 0 is unused because the first action is forced; the final action and transition need not be sampled when evaluating the control estimator below. DSIS omits the risk and stochastic-availability inputs. At each distinct query state, we enumerate all 81 first actions in A \mathcal{A} A and evaluate them using these 32 tapes in total. Each tape supports 81 coupled trajectories, one for each forced first action. Write S b , k a S^{a}_{b,k} S b , k a and A b , k a A^{a}_{b,k} A b , k a for the stage-k k k observation and action on tape b b b ’s branch whose forced first action is a a a . Thus S b , 0 a = s S^{a}_{b,0}=s S b , 0 a = s and A b , 0 a = a A^{a}_{b,0}=a A b , 0 a = a . The finite-logit categorical target policy has π g ( a ′ ∣ s ′ ) > 0 \pi_g(a' \mid s')>0 π g ( a ′ ∣ s ′ ) > 0 for every action and queried observation. For 1 ≤ k < T 1 \leq k < T 1 ≤ k < T , branch a a a samples its action as
A b , k a = arg max a ′ ∈ A { log π g ( a ′ ∣ S b , k a ) + ϵ b , k , a ′ } . (154) A^{a}_{b,k}=\underset{a'\in\mathcal{A}}{\arg\max}\left\{\log\pi_g(a'\mid S^{a}_{b,k})+\epsilon_{b,k,a'}\right\}. \tag*{(154)} A b , k a = a ′ ∈ A arg max { log π g ( a ′ ∣ S b , k a ) + ϵ b , k , a ′ } . (154)
Demand and next availability are obtained by applying the model’s inverse cumulative distribution functions to U b , k D U^{D}_{b,k} U b , k D and U b , k B U^{B}_{b,k} U b , k B , respectively, using the branch’s current observation and, in RB, its fixed risk Z b Z_b Z b . The branches share the tape’s risk and random inputs, but their realized actions and successors can differ because their conditional distributions can differ. Sharing a tape preserves the prescribed marginal rollout law for every first action.
Estimating the return with the frozen critic
Let R R R and S ′ S' S ′ denote a one-step reward and successor observation under P ^ g , z \widehat{P}_{g,z} P g , z . Define Q g , z , k v , H Q^{v,H}_{g,z,k} Q g , z , k v , H as the undiscouted control backup at local stage k k k , where the superscripts indicate the frozen reference v g v_g v g and the heuristic evaluation:
Q g , z , k v , H ( s , a ) = { E P ^ g , z [ R + v g ( S ′ ) ∣ s , a ] , 0 ≤ k ≤ T − 2 , E P ^ g , z [ R ∣ s , a ] , k = T − 1. (155) Q^{v,H}_{g,z,k}(s,a)=
\begin{cases}
\mathbb{E}_{\widehat{P}_{g,z}}[R+v_g(S')\mid s,a], & 0\leq k\leq T-2,\\
\mathbb{E}_{\widehat{P}_{g,z}}[R\mid s,a], & k=T-1.
\end{cases} \tag*{(155)} Q g , z , k v , H ( s , a ) = { E P g , z [ R + v g ( S ′ ) ∣ s , a ] , E P g , z [ R ∣ s , a ] , 0 ≤ k ≤ T − 2 , k = T − 1. (155)
This is exactly the evaluated backup L g , z , k H L^{H}_{g,z,k} L g , z , k H listed in the notation table, with T = H T=H T = H : it uses v g v_g v g at nonterminal successor stages and zero control at the horizon. This quantity is distinct from the completed, discounted action value Q g π g Q^{\pi_g}_g Q g π g in Equation (11) . The one-step expectations integrate the model’s demand support and, in RB, both next-availability outcomes.
For branch a a a on tape b b b , let G ~ T , g , z b ( b ) ( s , a ) \widetilde{G}^{(b)}_{T,g,z_b}(s,a) G T , g , z b ( b ) ( s , a ) denote the complete one-tape control-variate estimator, defined by
G ~ T , g , z b ( b ) ( s , a ) = Q g , z , 0 v , H ( s , a ) + ∑ k = 1 T − 1 [ ∑ a ′ ∈ A π g ( a ′ ∣ S b , k a ) Q g , z b , k v , H ( S b , k a , a ′ ) − v g ( S b , k a ) ] . (156) \widetilde{G}^{(b)}_{T,g,z_b}(s,a)
=
Q^{v,H}_{g,z,0}(s,a)
+
\sum_{k=1}^{T-1}
\left[
\sum_{a'\in\mathcal{A}}
\pi_g(a'\mid S^{a}_{b,k})Q^{v,H}_{g,z_b,k}(S^{a}_{b,k},a')
-
v_g(S^{a}_{b,k})
\right]. \tag*{(156)} G T , g , z b ( b ) ( s , a ) = Q g , z , 0 v , H ( s , a ) + k = 1 ∑ T − 1 [ a ′ ∈ A ∑ π g ( a ′ ∣ S b , k a ) Q g , z b , k v , H ( S b , k a , a ′ ) − v g ( S b , k a ) ] . (156)
The first term evaluates the forced first action. Each later term is the target-policy average of the one-step backup, minus the reference value at the visited observation. The sampled continuation determines where these control residuals are evaluated.
The same conditional-expectation and telescoping argument used in the proof of Theorem 1 yields the finite-horizon identity below. The theorem itself concerns an infinite discounted return; here the return is undiscounted and the terminal control contribution is zero. Fix s , a , z s,a,z s , a , z , and write G ~ T , g , z ( s , a ) \widetilde{G}_{T,g,z}(s,a) G T , g , z ( s , a ) for one generic complete tape estimator conditional on risk z z z , suppressing the tape and branch indices on both the estimator and the trajectory. The tower property gives
E z s , a [ G ~ T , g , z ( s , a ) ] = E z s , a [ R 0 + v g ( S 1 ) + ∑ k = 1 T − 2 { R k + v g ( S k + 1 ) − v g ( S k ) } + R T − 1 − v g ( S T − 1 ) ] = E z s , a [ ∑ k = 0 T − 1 R k ] . (157) \begin{aligned}
\mathbb{E}^{s,a}_{z}\left[\widetilde{G}_{T,g,z}(s,a)\right]
=
\mathbb{E}^{s,a}_{z}\left[
\begin{aligned}
&R_0+v_g(S_1)\\
&+\sum_{k=1}^{T-2}\{R_k+v_g(S_{k+1})-v_g(S_k)\}+R_{T-1}-v_g(S_{T-1})
\end{aligned}
\right] \\
&=\mathbb{E}^{s,a}_{z}\left[\sum_{k=0}^{T-1}R_k\right]. \tag*{(157)}
\end{aligned} E z s , a [ G T , g , z ( s , a ) ] = E z s , a R 0 + v g ( S 1 ) + k = 1 ∑ T − 2 { R k + v g ( S k + 1 ) − v g ( S k )} + R T − 1 − v g ( S T − 1 ) = E z s , a [ k = 0 ∑ T − 1 R k ] . (157)
All reference values cancel. For each fixed starting observation, bounded actions and finite demand support give finitely many reachable observations over this finite horizon. Period costs and the finite-parameter frozen networks are finite on that set, uniformly over z ∈ [ 0 , 1 / 2 ] z\in[0,1/2] z ∈ [ 0 , 1/2 ] . Integrating this identity against m ( d z ) m(\mathrm{d}z) m ( d z ) therefore recovers G T , g ( s , a ) G_{T,g}(s,a) G T , g ( s , a ) .
The critic need not equal the finite-horizon value function: its role is to supply terms whose expectations cancel. This is an identity in expectation, so the estimate need not equal the realized reward sum on an individual tape. Rewards already enter through Q g , z , k V , H Q^{V,H}_{g,z,k} Q g , z , k V , H ; an additional sampled reward sum would count them again.
There are two changes from the discounted critic backup. Recall that γ = 0.995 \gamma=0.995 γ = 0.995 is the critic discount factor. Before the last period, the successor value has coefficient 1 1 1 instead of γ \gamma γ , because every reward in this return receives equal weight. At the last period, no successor value is included, because rewards after period T − 1 T-1 T − 1 are outside the objective. Recall the discounted backup L g , z v g L_{g,z}v_g L g , z v g and the state-transition operator P ^ g , z a \widehat{P}^{a}_{g,z} P g , z a , evaluated on v g v_g v g :
L g , z v g ( s , a ) = E P ^ g , z a [ R + γ v g ( S ′ ) ∣ s , a ] , ( P ^ g , z a v g ) ( s ) = E P ^ g , z a [ v g ( S ′ ) ∣ s , a ] . (158) L_{g,z}v_g(s,a)=\mathbb{E}_{\widehat{P}_{g,z}^{a}}[R+\gamma v_g(S')\mid s,a],
\qquad
(\widehat{P}^{a}_{g,z}v_g)(s)=\mathbb{E}_{\widehat{P}_{g,z}^{a}}[v_g(S')\mid s,a]. \tag*{(158)} L g , z v g ( s , a ) = E P g , z a [ R + γ v g ( S ′ ) ∣ s , a ] , ( P g , z a v g ) ( s ) = E P g , z a [ v g ( S ′ ) ∣ s , a ] . (158)
Writing 1 { k < T − 1 } \mathbf{1}_{\{k<T-1\}} 1 { k < T − 1 } for the indicator of k < T − 1 k<T-1 k < T − 1 , the required adjustment is
Q g , z , k V , H ( s , a ) = L g , z v g ( s , a ) + ( 1 { k < T − 1 } − γ ) ( P ^ g , z a v g ) ( s ) . (159) Q^{V,H}_{g,z,k}(s,a)=L_{g,z}v_g(s,a)+\left(\mathbf{1}_{\{k<T-1\}}-\gamma\right)(\widehat{P}^{a}_{g,z}v_g)(s). \tag*{(159)} Q g , z , k V , H ( s , a ) = L g , z v g ( s , a ) + ( 1 { k < T − 1 } − γ ) ( P g , z a v g ) ( s ) . (159)
Keeping the discounted backup unchanged would break the cancellation. Keeping a terminal value would add E z s , a [ v g ( S T ) ] \mathbb{E}^{s,a}_{z}[v_g(S_T)] E z s , a [ v g ( S T )] to the expected estimate.
Averaging and computing action differences
Average the 32 complete tape estimates to obtain the return estimate used in Equation (152) :
G ^ T , g ( s , a ) = 1 32 ∑ b = 1 32 G ~ T , g , Z b ( b ) ( s , a ) . (160) \widehat{G}_{T,g}(s,a)=\frac{1}{32}\sum_{b=1}^{32}\widetilde{G}^{(b)}_{T,g,Z_b}(s,a). \tag*{(160)} G T , g ( s , a ) = 32 1 b = 1 ∑ 32 G T , g , Z b ( b ) ( s , a ) . (160)
Recall that u g = h g u_g=h_g u g = h g is this estimate centered under π g ( ⋅ ∣ s ) \pi_g(\cdot\mid s) π g ( ⋅ ∣ s ) , rather than under a uniform action law. For a fixed query observation independent of its utility tapes, hold the model, policy, and reference critic fixed. In the next display, E \mathbb{E} E averages only over the 32 utility tapes, including their risks in RB. The preceding expectation identity gives
E [ G ^ T , g ( s , a ) ] = G T , g ( s , a ) , E [ u g ( s , a ) ] = G T , g ( s , a ) − ∑ a ′ π g ( a ′ ∣ s ) G T , g ( s , a ′ ) . (161) \mathbb{E}[\widehat{G}_{T,g}(s,a)]=G_{T,g}(s,a),
\qquad
\mathbb{E}[u_g(s,a)]=G_{T,g}(s,a)-\sum_{a'}\pi_g(a'\mid s)G_{T,g}(s,a'). \tag*{(161)} E [ G T , g ( s , a )] = G T , g ( s , a ) , E [ u g ( s , a )] = G T , g ( s , a ) − a ′ ∑ π g ( a ′ ∣ s ) G T , g ( s , a ′ ) . (161)
These statements concern the complete fixed-state return and utility estimators in exact arithmetic. They do not assert conditional unbiasedness after selecting a state using the same cached utility estimates. Nonlinear transformations of their estimates, including the controller below, do not inherit unbiasedness automatically. Utilities remain in cumulative normalized-reward units, without division by T T T .
Only action differences are needed to construct u g u_g u g . For any fixed reference action a ⋆ ∈ A a_\star\in\mathcal{A} a ⋆ ∈ A ,
u g ( s , a ) = [ G ^ T , g ( s , a ) − G ^ T , g ( s , a ⋆ ) ] − ∑ a ′ π g ( a ′ ∣ s ) [ G ^ T , g ( s , a ′ ) − G ^ T , g ( s , a ⋆ ) ] . (162) u_g(s,a)=\left[\widehat{G}_{T,g}(s,a)-\widehat{G}_{T,g}(s,a_\star)\right]-\sum_{a'}\pi_g(a'\mid s)\left[\widehat{G}_{T,g}(s,a')-\widehat{G}_{T,g}(s,a_\star)\right]. \tag*{(162)} u g ( s , a ) = [ G T , g ( s , a ) − G T , g ( s , a ⋆ ) ] − a ′ ∑ π g ( a ′ ∣ s ) [ G T , g ( s , a ′ ) − G T , g ( s , a ⋆ ) ] . (162)
The implementation either accumulates tape-level differences against the reference order pair a ⋆ = ( 0 , 0 ) a_\star= (0,0) a ⋆ = ( 0 , 0 ) , or accumulates coupled branch estimates and centers them afterwards. On the same tape, branches reaching the same observation at the same stage have identical future contributions under their shared inputs. Their subsequent work can be shared, and a tail common to all branches can be omitted. These operations preserve the centered utilities in exact arithmetic; an accumulator with an omitted common tail need not represent an absolute return.
The identities above use exact arithmetic and the same evaluated frozen policy and control function throughout. On RB graphics processors, the heuristic service evaluates bfloat16 copies of the frozen actor and critics, with 32-bit floating-point accumulation. Discounted residual evaluation uses 32-bit critic inference and 64-bit residual algebra. Equation (159) therefore compares backups of the same mathematical functions; it does not assert bitwise equality between these numerical services.
The regularization coefficient
The coefficient τ > 0 \tau> 0 τ > 0 controls how strongly the controller is penalized for departing from π g \pi_g π g : larger values give more weight to staying near the target prior. The shared hyperparameter-selection protocol tunes a positive, dimensionless multiplier κ \kappa κ , distinct from the fitted supply-recovery parameter κ ^ g r e c \widehat{\kappa}^{\mathrm{rec}}_g κ g rec . At initialization, let s 0 , f c a l s^{\mathrm{cal}}_{0,f} s 0 , f cal , f = 1 , … , 16 f = 1,\ldots,16 f = 1 , … , 16 , denote calibration anchors sampled with replacement from the initial emphatic measure ν 0 0 \nu^0_0 ν 0 0 of Equation (147) , using a separate random-number domain. Let u 0 0 u^0_0 u 0 0 be their centered utility estimates, computed with the same 81 forced first actions, 32 tapes, and T = 1000 T = 1000 T = 1000 horizon. Define the calibration scale S c a l S_{\mathrm{cal}} S cal and set
S c a l = median f = 1 , … , 16 [ max a ∈ A u 0 c a l ( s 0 , f c a l , a ) − min a ∈ A u 0 c a l ( s 0 , f c a l , a ) ] , τ = κ S c a l . (163) S_{\mathrm{cal}} = \operatorname{median}_{f=1,\ldots,16}\left[\max_{a\in A}u^{\mathrm{cal}}_0(s^{\mathrm{cal}}_{0,f},a)-\min_{a\in A}u^{\mathrm{cal}}_0(s^{\mathrm{cal}}_{0,f},a)\right],
\qquad
\tau= \kappa S_{\mathrm{cal}}. \tag*{(163)} S cal = median f = 1 , … , 16 [ a ∈ A max u 0 cal ( s 0 , f cal , a ) − a ∈ A min u 0 cal ( s 0 , f cal , a ) ] , τ = κ S cal . (163)
For these 16 ranges, the median is the mean of the eighth and ninth ordered values. The construction requires 0 < S c a l < ∞ 0 < S_{\mathrm{cal}} < \infty 0 < S cal < ∞ , hence 0 < τ < ∞ 0 < \tau< \infty 0 < τ < ∞ . The implementation rejects a zero or nonfinite calibration scale; no positive floor is substituted. Thus S c a l S_{\mathrm{cal}} S cal supplies a one-time scale conversion; it is not an additional tuned parameter. The resulting τ \tau τ stays fixed throughout the run, with no separate normalization of later states’ utilities.
The Hellinger-regularized controller
Recall that u g u_g u g is the centered utility in Equation (152) , and that the positive scale τ = κ S c a l \tau= \kappa S_{\mathrm{cal}} τ = κ S cal is fixed by Equation (163) . At a fixed query observation, write p = ( p a ) a ∈ A p = (p_a)_{a\in A} p = ( p a ) a ∈ A for the target prior with p a = π g ( a ∣ s ) > 0 p_a = \pi_g(a\mid s) > 0 p a = π g ( a ∣ s ) > 0 , and u a = u g ( s , a ) u_a = u_g(s,a) u a = u g ( s , a ) . Let q = ( q a ) a ∈ A q = (q_a)_{a\in A} q = ( q a ) a ∈ A be a candidate distribution for the controller’s action at s s s , where q a q_a q a is the probability assigned to action a a a . The optimization ranges over the entire simplex Δ ( A ) = { q : q a ≥ 0 , ∑ a ∈ A q a = 1 } \Delta(A) = \{q:q_a \ge0,\ \sum_{a\in A}q_a = 1\} Δ ( A ) = { q : q a ≥ 0 , ∑ a ∈ A q a = 1 } , the set of all probability distributions on the 81 actions. The prior p p p supplies the regularization reference; no initial candidate q q q is required, because the maximizing distribution is computed by the scalar root solve derived below. The controller is defined by
q H , g ( ⋅ ∣ s ) = arg max q ∈ Δ ( A ) { ∑ a q a u a − 2 τ ∑ a ( q a − p a ) 2 } . (164) q_{H,g}(\cdot\mid s)=\underset{q\in\Delta(A)}{\operatorname{arg\,max}}\left\{\sum_a q_a u_a-2\tau\sum_a\left(\sqrt{q_a}-\sqrt{p_a}\right)^2\right\}. \tag*{(164)} q H , g ( ⋅ ∣ s ) = q ∈ Δ ( A ) arg max { a ∑ q a u a − 2 τ a ∑ ( q a − p a ) 2 } . (164)
The displayed penalty uses the unnormalized squared Hellinger discrepancy ∑ a ( q a − p a ) 2 \sum_a(\sqrt{q_a}-\sqrt{p_a})^2 ∑ a ( q a − p a ) 2 ; the convention with a factor 1 / 2 1/2 1/2 gives half this value.
Since p p p and q q q have unit mass, denote the same objective by Φ ( q ) \Phi(q) Φ ( q ) and expand it as
Φ ( q ) = ∑ a q a u a + 4 τ ∑ a p a q a − 4 τ . (165) \Phi(q)=\sum_a q_a u_a+4\tau\sum_a\sqrt{p_aq_a}-4\tau. \tag*{(165)} Φ ( q ) = a ∑ q a u a + 4 τ a ∑ p a q a − 4 τ . (165)
At an interior point q a > 0 q_a > 0 q a > 0 for all a a a , its Hessian entries are
∂ 2 Φ ∂ q a 2 = − τ p a q a − 3 / 2 < 0 , ∂ 2 Φ ∂ q a ∂ q a ′ = 0 ( a ≠ a ′ ) . (166) \frac{\partial^2\Phi}{\partial q_a^2}=-\tau\sqrt{p_a}\,q_a^{-3/2}<0,
\qquad
\frac{\partial^2\Phi}{\partial q_a\partial q_{a'}}=0\quad(a\ne a'). \tag*{(166)} ∂ q a 2 ∂ 2 Φ = − τ p a q a − 3/2 < 0 , ∂ q a ∂ q a ′ ∂ 2 Φ = 0 ( a = a ′ ) . (166)
The objective is strictly concave. Its maximizer has full support: if q a = 0 q_a = 0 q a = 0 , transfer a small mass ε > 0 \varepsilon> 0 ε > 0 from any positive coordinate to action a a a . Because p a > 0 p_a > 0 p a > 0 , the gain in the square-root term is of order ε \sqrt{\varepsilon} ε , whereas the utility change and the loss at the donating coordinate are of order ε \varepsilon ε . Compactness gives existence, and strict concavity gives uniqueness.
Introducing a multiplier λ g ( s ) \lambda_g(s) λ g ( s ) for the unit-mass constraint yields
u a + 2 τ p a q a − λ g ( s ) = 0. (167) u_a + 2\tau\sqrt{\frac{p_a}{q_a}}-\lambda_g(s)=0. \tag*{(167)} u a + 2 τ q a p a − λ g ( s ) = 0. (167)
Therefore,
q H , g ( a ∣ s ) = π g ( a ∣ s ) ( 2 τ λ g ( s ) − u g ( s , a ) ) 2 , λ g ( s ) > max a u g ( s , a ) , (168) q_{H,g}(a\mid s)=\pi_g(a\mid s)\left(\frac{2\tau}{\lambda_g(s)-u_g(s,a)}\right)^2,\qquad\lambda_g(s)>\max_a u_g(s,a), \tag*{(168)} q H , g ( a ∣ s ) = π g ( a ∣ s ) ( λ g ( s ) − u g ( s , a ) 2 τ ) 2 , λ g ( s ) > a max u g ( s , a ) , (168)
where λ g ( s ) \lambda_g(s) λ g ( s ) is determined by
∑ a π g ( a ∣ s ) ( 2 τ λ g ( s ) − u g ( s , a ) ) 2 = 1. (169) \sum_a \pi_g(a\mid s)\left(\frac{2\tau}{\lambda_g(s)-u_g(s,a)}\right)^2=1. \tag*{(169)} a ∑ π g ( a ∣ s ) ( λ g ( s ) − u g ( s , a ) 2 τ ) 2 = 1. (169)
The left-hand side is continuous and strictly decreasing in λ g ( s ) \lambda_g(s) λ g ( s ) , tends to infinity at the lower boundary, and tends to zero as λ g ( s ) → ∞ \lambda_g(s)\to\infty λ g ( s ) → ∞ . Hence a unique scalar root determines the entire action distribution.
For a bounded numerical bracket, let u max u_{\max} u m a x be the largest utility, p max p_{\max} p m a x the total prior mass of all maximizing actions (not the largest prior probability), and δ \delta δ the positive multiplier gap:
u max = max a u a , p max = ∑ a : u a = u max p a , δ = λ g ( s ) − u max . (170) u_{\max}=\max_a u_a,\qquad p_{\max}=\sum_{a:u_a=u_{\max}}p_a,\qquad\delta=\lambda_g(s)-u_{\max}. \tag*{(170)} u m a x = a max u a , p m a x = a : u a = u m a x ∑ p a , δ = λ g ( s ) − u m a x . (170)
At the normalization root, the maximizing actions contribute ( 2 τ / δ ) 2 p max (2\tau/\delta)^2p_{\max} ( 2 τ / δ ) 2 p m a x , and every denominator is at least δ \delta δ . Hence ( 2 τ / δ ) 2 p max ≤ 1 ≤ ( 2 τ / δ ) 2 (2\tau/\delta)^2p_{\max}\le1\le(2\tau/\delta)^2 ( 2 τ / δ ) 2 p m a x ≤ 1 ≤ ( 2 τ / δ ) 2 , giving
2 τ p max ≤ δ ≤ 2 τ . (171) 2\tau\sqrt{p_{\max}}\le\delta\le2\tau. \tag*{(171)} 2 τ p m a x ≤ δ ≤ 2 τ . (171)
The implementation uses 64 bisection iterations in log δ \log\delta log δ , retaining log prior probabilities. This solves for one scalar normalizer per query; the resulting probabilities satisfy the controller formula up to floating-point and root-solve error, without adding a probability floor.
Since q = p q=p q = p is feasible and ∑ a p a u a = 0 \sum_a p_a u_a=0 ∑ a p a u a = 0 , optimality of the exact controller gives
∑ a ∈ A q H , g ( a ∣ s ) u a ≥ 2 τ ∑ a ∈ A ( q H , g ( a ∣ s ) − p a ) 2 ≥ 0. (172) \sum_{a\in\mathcal{A}}q_{H,g}(a\mid s)u_a\ge2\tau\sum_{a\in\mathcal{A}}\left(\sqrt{q_{H,g}(a\mid s)}-\sqrt{p_a}\right)^2\ge0. \tag*{(172)} a ∈ A ∑ q H , g ( a ∣ s ) u a ≥ 2 τ a ∈ A ∑ ( q H , g ( a ∣ s ) − p a ) 2 ≥ 0. (172)
This is a property of the estimated first-action replacement objective under the frozen model. It neither guarantees improvement in the exact model return after Monte Carlo action selection nor establishes improvement from repeatedly executing the controller or in the real environment.
Selecting fitting locations through short controller rollouts
Recall the 16 anchor occurrences s ˉ g , f \bar{s}_{g,f} s ˉ g , f from Equation (148) . From each, generate 8 outer model trajectories under the controller. The superscript O labels these outer trajectories: let S g , f , b , k O S^{O}_{g,f,b,k} S g , f , b , k O and A g , f , b , k O A^{O}_{g,f,b,k} A g , f , b , k O denote their stage-k k k observations and actions, with b ∈ { 1 , … , 8 } b\in\{1,\ldots,8\} b ∈ { 1 , … , 8 } indexing outer paths and S g , f , b , 0 O = s ˉ g , f S^{O}_{g,f,b,0}=\bar{s}_{g,f} S g , f , b , 0 O = s ˉ g , f . In RB, each path draws a risk Z g , f , b O Z^{O}_{g,f,b} Z g , f , b O independently from m = Unif [ 0 , 1 / 2 ] m=\operatorname{Unif}[0,1/2] m = Unif [ 0 , 1/2 ] , the prior recalled in Equation (150) , and retains it throughout. At stages k = 0 , … , 8 k=0,\ldots,8 k = 0 , … , 8 , sample
A g , f , b , k O ∼ q H , g ( ⋅ ∣ S g , f , b , k O ) (173) A^{O}_{g,f,b,k}\sim q_{H,g}(\cdot\mid S^{O}_{g,f,b,k}) \tag*{(173)} A g , f , b , k O ∼ q H , g ( ⋅ ∣ S g , f , b , k O ) (173)
and then sample the reward and successor from P ^ g , Z g , f , b O ( ⋅ ∣ S g , f , b , k O , A g , f , b , k O ) \widehat{P}_{g,Z^{O}_{g,f,b}}(\cdot\mid S^{O}_{g,f,b,k},A^{O}_{g,f,b,k}) P g , Z g , f , b O ( ⋅ ∣ S g , f , b , k O , A g , f , b , k O ) in RB, or from the risk-free model in DSIS. Let ξ g \xi_g ξ g be the empirical fitting measure of the pre-action observations at these nine stages. Recalling that δ x \delta_x δ x denotes unit point mass at x x x , set
ξ g = 1 16 ⋅ 8 ⋅ 9 ∑ f = 1 16 ∑ b = 1 8 ∑ k = 0 8 δ S g , f , b , k O . (174) \xi_g=\frac{1}{16\cdot8\cdot9}\sum_{f=1}^{16}\sum_{b=1}^{8}\sum_{k=0}^{8}\delta_{S^{O}_{g,f,b,k}}. \tag*{(174)} ξ g = 16 ⋅ 8 ⋅ 9 1 f = 1 ∑ 16 b = 1 ∑ 8 k = 0 ∑ 8 δ S g , f , b , k O . (174)
This panel contains exactly 1152 observation occurrences, including repetitions. Each occurrence receives equal fitting weight; the emphatic weights have already entered through anchor selection.
At every retained observation, the fitting target is the full distribution q H , g ( ⋅ ∣ s ) q_{H,g}(\cdot\mid s) q H , g ( ⋅ ∣ s ) , obtained from a utility query with a new local horizon of 1000 periods. The nine-stage rollout selects fitting locations without reducing the utility horizon at later stages.
Utility estimation and outer trajectory generation use separate random-number domains. In RB, an inner controller query samples its own utility-tape risks and does not read the outer trajectory’s latent risk, so the controller remains a function of the visible observation. Repeated queries at the same observation reuse the generation’s cached estimate. Separate outer paths use separately addressed random inputs, including when their anchors coincide. Thus 1152 counts fitting occurrences, not necessarily distinct utility queries; observations along a path are also temporally dependent.
Deriving the behavior-guidance gradient
Recall that ξ g \xi_g ξ g is the equal-weight empirical measure of the 1152 pre-action observations S g , f , b , k O S^{O}_{g,f,b,k} S g , f , b , k O in (174) ; O denotes the outer controller trajectories. Let θ μ \theta^\mu θ μ parameterize the candidate behavior policy μ θ μ \mu_{\theta^\mu} μ θ μ , and recall the parent policy μ g = μ θ g μ \mu_g=\mu_{\theta_g^\mu} μ g = μ θ g μ at parameters θ g μ \theta_g^\mu θ g μ . Assume that the candidate probabilities are strictly positive and differentiable in a neighborhood of those parent parameters, as for the finite-logit categorical actor. For two action distributions p = ( p a ) p=(p_a) p = ( p a ) and q = ( q a ) q=(q_a) q = ( q a ) , define the categorical Fisher–Rao distance d F R d_{\mathrm{FR}} d FR by
d F R ( p , q ) = 2 arccos ( ∑ a ∈ A p a q a ) . (175) d_{\mathrm{FR}}(p,q)=2\arccos\left(\sum_{a\in\mathcal{A}}\sqrt{p_aq_a}\right). \tag*{(175)} d FR ( p , q ) = 2 arccos ( a ∈ A ∑ p a q a ) . (175)
With ξ g \xi_g ξ g and all controller targets fixed, define the behavior-fitting objective
H g ( μ θ μ ) = − 1 2 E s ∼ ξ g [ d F R 2 ( μ θ μ ( ⋅ ∣ s ) , q H , g ( ⋅ ∣ s ) ) ] . (176) H_g(\mu_{\theta^\mu})=-\frac{1}{2}\mathbb{E}_{s\sim\xi_g}\left[d_{\mathrm{FR}}^2\left(\mu_{\theta^\mu}(\cdot\mid s),q_{H,g}(\cdot\mid s)\right)\right]. \tag*{(176)} H g ( μ θ μ ) = − 2 1 E s ∼ ξ g [ d FR 2 ( μ θ μ ( ⋅ ∣ s ) , q H , g ( ⋅ ∣ s ) ) ] . (176)
Maximizing H g H_g H g reduces the distance between the behavior policy and the controller at the retained observations. Define its behavior-guidance gradient g H , g g_{H,g} g H , g at the parent behavior parameters by
g H , g = ∇ θ μ H g ( μ θ μ ) ∣ θ μ = θ g μ = − 2 ∇ θ μ E s ∼ ξ g [ arccos 2 ( ∑ a μ θ μ ( a ∣ s ) q H , g ( a ∣ s ) ) ] ∣ θ μ = θ g μ , (177) \begin{aligned}
g_{H,g}&=\left.\nabla_{\theta^\mu}H_g(\mu_{\theta^\mu})\right|_{\theta^\mu=\theta_g^\mu}\\
&=-2\left.\nabla_{\theta^\mu}\mathbb{E}_{s\sim\xi_g}\left[\arccos^2\left(\sum_a\sqrt{\mu_{\theta^\mu}(a\mid s)q_{H,g}(a\mid s)}\right)\right]\right|_{\theta^\mu=\theta_g^\mu}, \tag*{(177)}
\end{aligned} g H , g = ∇ θ μ H g ( μ θ μ ) ∣ θ μ = θ g μ = − 2 ∇ θ μ E s ∼ ξ g [ arccos 2 ( a ∑ μ θ μ ( a ∣ s ) q H , g ( a ∣ s ) ) ] θ μ = θ g μ , (177)
recovering (2) .
For an explicit expression, define the affinity B g ( s ) B_g(s) B g ( s ) , angle φ g ( s ) \varphi_g(s) φ g ( s ) , and behavior-policy score vector s c g μ ( s , a ) \mathrm{sc}_g^\mu(s,a) sc g μ ( s , a ) at the parent parameters:
B g ( s ) = ∑ a μ g ( a ∣ s ) q H , g ( a ∣ s ) , φ g ( s ) = arccos B g ( s ) , s c g μ ( s , a ) = ∇ θ μ log μ θ μ ( a ∣ s ) ∣ θ μ = θ g μ . (178) \begin{aligned}
B_g(s)&=\sum_a\sqrt{\mu_g(a\mid s)q_{H,g}(a\mid s)},\qquad\varphi_g(s)=\arccos B_g(s),\\
\mathrm{sc}_g^\mu(s,a)&=\left.\nabla_{\theta^\mu}\log\mu_{\theta^\mu}(a\mid s)\right|_{\theta^\mu=\theta_g^\mu}. \tag*{(178)}
\end{aligned} B g ( s ) sc g μ ( s , a ) = a ∑ μ g ( a ∣ s ) q H , g ( a ∣ s ) , φ g ( s ) = arccos B g ( s ) , = ∇ θ μ log μ θ μ ( a ∣ s ) ∣ θ μ = θ g μ . (178)
Let B θ μ ( s ) B_{\theta^\mu}(s) B θ μ ( s ) denote the same affinity with μ g \mu_g μ g replaced by μ θ μ \mu_{\theta^\mu} μ θ μ . Differentiation gives
∇ θ μ B θ μ ( s ) ∣ θ μ = θ g μ = 1 2 ∑ a μ g ( a ∣ s ) q H , g ( a ∣ s ) s c g μ ( s , a ) . (179) \left.\nabla_{\theta^\mu}B_{\theta^\mu}(s)\right|_{\theta^\mu=\theta_g^\mu}
=\frac{1}{2}\sum_a\sqrt{\mu_g(a\mid s)q_{H,g}(a\mid s)}\,\mathrm{sc}_g^\mu(s,a). \tag*{(179)} ∇ θ μ B θ μ ( s ) ∣ θ μ = θ g μ = 2 1 a ∑ μ g ( a ∣ s ) q H , g ( a ∣ s ) sc g μ ( s , a ) . (179)
Apply the chain rule to − 2 arccos 2 B θ μ ( s ) -2\arccos^2 B_{\theta^\mu}(s) − 2 arccos 2 B θ μ ( s ) and use the score identity ∑ a μ g ( a ∣ s ) s c g μ ( s , a ) = 0 \sum_a\mu_g(a\mid s)\mathrm{sc}_g^\mu(s,a)=0 ∑ a μ g ( a ∣ s ) sc g μ ( s , a ) = 0 , which follows by differentiating ∑ a μ θ μ ( a ∣ s ) = 1 \sum_a\mu_{\theta^\mu}(a\mid s)=1 ∑ a μ θ μ ( a ∣ s ) = 1 . This gives
g H , g = E s ∼ ξ g [ 2 φ g ( s ) sin φ g ( s ) ∑ a ( μ g ( a ∣ s ) q H , g ( a ∣ s ) − B g ( s ) μ g ( a ∣ s ) ) s c g μ ( s , a ) ] . (180) g_{H,g}=\mathbb{E}_{s\sim\xi_g}\left[\frac{2\varphi_g(s)}{\sin\varphi_g(s)}\sum_a\left(\sqrt{\mu_g(a\mid s)q_{H,g}(a\mid s)}-B_g(s)\mu_g(a\mid s)\right)\mathrm{sc}_g^\mu(s,a)\right]. \tag*{(180)} g H , g = E s ∼ ξ g [ sin φ g ( s ) 2 φ g ( s ) a ∑ ( μ g ( a ∣ s ) q H , g ( a ∣ s ) − B g ( s ) μ g ( a ∣ s ) ) sc g μ ( s , a ) ] . (180)
The expectation is the arithmetic average over the 1152 retained occurrences, and the action sum includes all 81 actions. The subtracted term B g ( s ) μ g ( a ∣ s ) B_g(s)\mu_g(a\mid s) B g ( s ) μ g ( a ∣ s ) contributes zero after the score sum.
a tangent-space centering of the same Fisher–Rao gradient; there is no additional action-importance ratio or state-weight correction in this fitting objective.
At μ g ( ⋅ ∣ s ) = q H , g ( ⋅ ∣ s ) \mu_g(\cdot\mid s)=q_{H,g}(\cdot\mid s) μ g ( ⋅ ∣ s ) = q H , g ( ⋅ ∣ s ) , use lim φ → 0 φ / sin φ = 1 \lim_{\varphi\to0}\varphi/\sin\varphi=1 lim φ → 0 φ / sin φ = 1 ; the centered action coefficients are zero, so the contribution is zero. For stable evaluation, let r g ( s ) \mathbf{r}_g(s) r g ( s ) be the residual between the square-root controller vector and its projection onto the square-root behavior vector. Square roots below act coordinatewise, ∥ ⋅ ∥ 2 \|\cdot\|_2 ∥ ⋅ ∥ 2 is the Euclidean norm, and atan2 ( y , x ) \operatorname{atan2}(y,x) atan2 ( y , x ) is the two-argument angle function:
r g ( s ) = q H , g ( ⋅ ∣ s ) − B g ( s ) μ g ( ⋅ ∣ s ) , \mathbf{r}_g(s)=\sqrt{\mathbf{q}_{H,g}(\cdot\mid s)}-B_g(s)\sqrt{\boldsymbol{\mu}_g(\cdot\mid s)}, r g ( s ) = q H , g ( ⋅ ∣ s ) − B g ( s ) μ g ( ⋅ ∣ s ) ,
sin φ g ( s ) = ∥ r g ( s ) ∥ 2 , \sin\varphi_g(s)=\|\mathbf{r}_g(s)\|_2, sin φ g ( s ) = ∥ r g ( s ) ∥ 2 ,
φ g ( s ) = atan2 ( ∥ r g ( s ) ∥ 2 , B g ( s ) ) . (181) \varphi_g(s)=\operatorname{atan2}\left(\|\mathbf{r}_g(s)\|_2,B_g(s)\right). \tag*{(181)} φ g ( s ) = atan2 ( ∥ r g ( s ) ∥ 2 , B g ( s ) ) . (181)
This evaluates the same gradient without directly differentiating an inverse cosine at coincident distributions.
During differentiation, the sampled observations, controller targets, utility estimates, and parent simulation components remain fixed. No gradient passes through action selection, model trajectories, the critic, or the controller’s scalar root. Recall that τ = κ S c a l \tau=\kappa S_{\mathrm{cal}} τ = κ S cal , with tuned multiplier κ \kappa κ and initialization scale S c a l S_{\mathrm{cal}} S cal , is fixed by Equation (163) . It affects g H , g g_{H,g} g H , g through the controller and its induced fitting locations, and adds no multiplicative factor to this gradient. The resulting g H , g g_{H,g} g H , g enters the behavior update through the coupled-update rule defined in the main text.
Delayed Heuristic–Behavior–Target Coupling and Policy Updates
This subsection assembles the shared advantage evidence, the direct behavior and target objectives, the transmission objective, and the fused finite actor updates, in that order. All updates use the same frozen generation-g g g parents: behavior policy μ g \mu_g μ g , target policy π g \pi_g π g , reference critic v g v_g v g , simulation model, and normalized consumer weights w i J , w i P w_i^J,w_i^P w i J , w i P defined below. The simulation model is the public kernel in DSIS and the fitted kernel in DSIS-RB. Once generated, residual samples, controller targets, and heuristic fitting locations are also held fixed during actor differentiation. In particular, the target successor uses μ g \mu_g μ g , not the simultaneously constructed μ g + 1 \mu_{g+1} μ g + 1 : guidance that changes behavior in generation g g g can first enter the discrepancy used to construct π g + 2 \pi_{g+2} π g + 2 . The corresponding delay through the transmission mass is specified in Appendix B.3.10.
The behavior actor combines the advantage objective A A A with the heuristic objective H g H_g H g ; the target actor combines its advantage objective J J J with the transmission objective P P P . Recall that H g H_g H g is the negative half squared Fisher–Rao distance to the frozen heuristic controller, averaged over its fitting observations, and write g H = g H , g g_H=g_{H,g} g H = g H , g for its parent gradient from Equations (176) and (180) . We differentiate the candidate behavior policy in H g H_g H g , holding the controller and its fitting measure fixed. The results below concern these frozen policy-update objectives and their induced policy changes.
Constructing the shared advantage face
We first specify the exact, base, and sampled advantages used by all three advantage channels. Recall that the generation batch contains n = 25,000 n=25{,}000 n = 25 , 000 observation occurrences s i s_i s i , indexed by i = 1 , … , n i=1,\ldots,n i = 1 , … , n , and that D g = n − 1 ∑ i δ s i D_g=n^{-1}\sum_i\delta_{s_i} D g = n − 1 ∑ i δ s i is their empirical measure, with δ s i \delta_{s_i} δ s i the unit point mass. The frozen weights satisfy w i J , w i P > 0 w_i^J,w_i^P>0 w i J , w i P > 0 and n − 1 ∑ i w i J = n − 1 ∑ i w i P = 1 n^{-1}\sum_i w_i^J=n^{-1}\sum_i w_i^P=1 n − 1 ∑ i w i J = n − 1 ∑ i w i P = 1 ; their weighted measures are ν g χ = n − 1 ∑ i w i χ δ s i \nu_g^\chi=n^{-1}\sum_i w_i^\chi\delta_{s_i} ν g χ = n − 1 ∑ i w i χ δ s i for χ ∈ { J , P } \chi\in\{J,P\} χ ∈ { J , P } . All action sums below range over the finite action set A \mathcal{A} A . Write π i ( a ) = π g ( a ∣ s i ) \pi_i(a)=\pi_g(a\mid s_i) π i ( a ) = π g ( a ∣ s i ) and μ i ( a ) = μ g ( a ∣ s i ) \mu_i(a)=\mu_g(a\mid s_i) μ i ( a ) = μ g ( a ∣ s i ) . For action-indexed arrays x , y x,y x , y , use ⟨ x , y ⟩ = ∑ a ∈ A x ( a ) y ( a ) \langle x,y\rangle=\sum_{a\in\mathcal{A}}x(a)y(a) ⟨ x , y ⟩ = ∑ a ∈ A x ( a ) y ( a ) , a componentwise dot product. Recall the exact discounted model action value Q g π g Q_g^{\pi_g} Q g π g and its one-step base Q g 0 Q_g^0 Q g 0 from Equations (53) and (51) . Their centered advantages are
a i ( a ) = A g a d v ( s i , a ) = Q g π g ( s i , a ) − ∑ b π i ( b ) Q g π g ( s i , b ) , a_i(a)=A_g^{\mathrm{adv}}(s_i,a)=Q_g^{\pi_g}(s_i,a)-\sum_b\pi_i(b)Q_g^{\pi_g}(s_i,b), a i ( a ) = A g adv ( s i , a ) = Q g π g ( s i , a ) − b ∑ π i ( b ) Q g π g ( s i , b ) ,
a i 0 ( a ) = A g b a s e ( s i , a ) = Q g 0 ( s i , a ) − ∑ b π i ( b ) Q g 0 ( s i , b ) . (182) a_i^0(a)=A_g^{\mathrm{base}}(s_i,a)=Q_g^0(s_i,a)-\sum_b\pi_i(b)Q_g^0(s_i,b). \tag*{(182)} a i 0 ( a ) = A g base ( s i , a ) = Q g 0 ( s i , a ) − b ∑ π i ( b ) Q g 0 ( s i , b ) . (182)
All three channels use target-policy centering. The completed evidence evaluates a first action followed by π g \pi_g π g , while the one-step base approximates that continuation through v g v_g v g , including when the evidence is used to update behavior.
Centering does not generally remove the critic’s continuation error. Recall that γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) is the critic discount, Z ∼ m Z \sim m Z ∼ m is a synthetic risk with prior m = Unif [ 0 , 1 / 2 ] m = \operatorname{Unif}[0,1/2] m = Unif [ 0 , 1/2 ] , drawn independently at the query root and retained along the continuation, and Δ g , z = V g , z Γ − v g \Delta_{g,z} = V^{\Gamma}_{g,z} - v_g Δ g , z = V g , z Γ − v g is the conditional model-value error. Here V g , z Γ V^{\Gamma}_{g,z} V g , z Γ is the exact discounted value at fixed risk z z z , and P ^ g , z a , P ^ g , z π g \widehat{P}^{a}_{g,z}, \widehat{P}^{\pi_g}_{g,z} P g , z a , P g , z π g are the one-step observation-transition operators at that risk, with the action fixed to a a a or averaged under π g \pi_g π g , respectively. In DSIS the risk variable is omitted. Theorem 1, followed by subtraction of the π i \pi_i π i -weighted action mean, gives
a i ( a ) − a i 0 ( a ) = γ E Z ∼ m [ ( P ^ g , Z a − P ^ g , Z π g ) Δ g , Z ] ( s i ) . (183) a_i(a) - a_i^0(a) = \gamma\mathbb{E}_{Z\sim m}\left[\left(\widehat{P}^{a}_{g,Z} - \widehat{P}^{\pi_g}_{g,Z}\right)\Delta_{g,Z}\right](s_i). \tag*{(183)} a i ( a ) − a i 0 ( a ) = γ E Z ∼ m [ ( P g , Z a − P g , Z π g ) Δ g , Z ] ( s i ) . (183)
Here the operators act on the function Δ g , z \Delta_{g,z} Δ g , z before evaluation at s i s_i s i . Consequently, the base recovers the entire centered action vector exactly if and only if the omitted continuation Q g π ( s i , a ) − Q g 0 ( s i , a ) Q^\pi_g(s_i,a)-Q^0_g(s_i,a) Q g π ( s i , a ) − Q g 0 ( s i , a ) is constant across actions. On-policy sampling does not impose this condition. The shared advantage face therefore includes the centered residual correction derived earlier.
The base is evaluated at every batch occurrence. For residual completion, the implementation draws K r e s = 128 K_{\mathrm{res}} = 128 K res = 128 occurrences with replacement. Let ϖ i \varpi_i ϖ i denote the probability of occurrence i i i and I k I_k I k the occurrence index in draw k k k ; the draws are independent and identically distributed (iid) conditional on the frozen inputs:
ϖ i = 1 + w i J + w i P 3 n , I k ∼ i i d Categorical ( ϖ 1 , … , ϖ n ) , k = 1 , … , K r e s . (184) \varpi_i = \frac{1+w_i^J+w_i^P}{3n}, \qquad I_k \mathrel{\overset{\mathrm{iid}}{\sim}} \operatorname{Categorical}(\varpi_1,\ldots,\varpi_n), \qquad k=1,\ldots,K_{\mathrm{res}}. \tag*{(184)} ϖ i = 3 n 1 + w i J + w i P , I k ∼ iid Categorical ( ϖ 1 , … , ϖ n ) , k = 1 , … , K res . (184)
Recall that A ^ g ( k ) \widehat{A}^{(k)}_g A g ( k ) denotes the sampled discounted target-policy advantage from (74) , using the independent bank of 32 continuation tapes assigned to draw k k k . It is centered under π g \pi_g π g at the selected occurrence. Repeated selections of the same occurrence receive distinct banks. Define the centered residual correction by
Δ a ^ ( k ) ( a ) = A ^ g ( k ) ( s I k , a ) − a I k 0 ( a ) . (185) \Delta\widehat{\mathbf{a}}^{(k)}(a)=\widehat{A}^{(k)}_g(s_{I_k},a)-a^0_{I_k}(a). \tag*{(185)} Δ a ( k ) ( a ) = A g ( k ) ( s I k , a ) − a I k 0 ( a ) . (185)
Let χ ∈ { 0 , J , P } \chi\in\{0,J,P\} χ ∈ { 0 , J , P } index the behavior-advantage, target-advantage, and transmission roles, respectively, and set w i 0 = 1 w_i^0=1 w i 0 = 1 . For finite actionwise coefficients ℓ = ( ℓ i ( a ) ) i , a \ell=(\ell_i(a))_{i,a} ℓ = ( ℓ i ( a ) ) i , a , define the sampled linear evidence functional
Λ ^ g χ [ ℓ ] = 1 n ∑ i = 1 n w i χ ⟨ ℓ i , a i 0 ⟩ + 1 K r e s ∑ k = 1 K r e s w I k χ n ϖ I k ⟨ ℓ I k , Δ a ^ ( k ) ⟩ . (186) \widehat{\Lambda}^{\chi}_g[\ell]=\frac{1}{n}\sum_{i=1}^{n}w_i^\chi\langle\ell_i,\mathbf{a}_i^0\rangle+\frac{1}{K_{\mathrm{res}}}\sum_{k=1}^{K_{\mathrm{res}}}\frac{w_{I_k}^\chi}{n\varpi_{I_k}}\langle\ell_{I_k},\Delta\widehat{\mathbf{a}}^{(k)}\rangle. \tag*{(186)} Λ g χ [ ℓ ] = n 1 i = 1 ∑ n w i χ ⟨ ℓ i , a i 0 ⟩ + K res 1 k = 1 ∑ K res n ϖ I k w I k χ ⟨ ℓ I k , Δ a ( k ) ⟩ . (186)
All three channels share the same root draws and continuation banks. Since each weight vector has empirical mean one, ∑ i ϖ i = 1 \sum_i\varpi_i=1 ∑ i ϖ i = 1 and 0 < w i χ / ( n ϖ i ) ≤ 3 0<w_i^\chi/(n\varpi_i)\leq3 0 < w i χ / ( n ϖ i ) ≤ 3 in exact arithmetic. In floating-point arithmetic, the implementation normalizes 1 + w i J + w i P 1+w_i^J+w_i^P 1 + w i J + w i P by its computed sum and reuses that same probability vector for root sampling and inverse-proposal factors. For vector-valued coefficients, the functional acts componentwise.
Define the occurrence-level estimated advantage vector a ^ i \widehat{\mathbf{a}}_i a i by the following algebraic representation of the main-text evidence:
a ^ i = a i 0 + 1 K r e s ϖ i ∑ k : I k = i Δ a ^ ( k ) . (187) \widehat{\mathbf{a}}_i=\mathbf{a}_i^0+\frac{1}{K_{\mathrm{res}}\varpi_i}\sum_{k:I_k=i}\Delta\widehat{\mathbf{a}}^{(k)}. \tag*{(187)} a i = a i 0 + K res ϖ i 1 k : I k = i ∑ Δ a ( k ) . (187)
An empty sum is zero, so an occurrence not selected for residual completion retains its base advantage. Therefore,
Λ ^ g χ [ ℓ ] = 1 n ∑ i w i χ ⟨ ℓ i , a ^ i ⟩ , ⟨ π i , a ^ i ⟩ = 0. (188) \widehat{\Lambda}^{\chi}_g[\ell]=\frac{1}{n}\sum_i w_i^\chi\langle\ell_i,\widehat{\mathbf{a}}_i\rangle,\qquad\langle\pi_i,\widehat{\mathbf{a}}_i\rangle=0. \tag*{(188)} Λ g χ [ ℓ ] = n 1 i ∑ w i χ ⟨ ℓ i , a i ⟩ , ⟨ π i , a i ⟩ = 0. (188)
This identity does not require materializing a completed action-value table at all roots: the implementation retains the dense base and sampled corrections separately. Distinct occurrences of the same observation may have different realized corrections. The main-text abbreviations are Λ A = Λ ^ g 0 \Lambda_A=\widehat{\Lambda}_g^0 Λ A = Λ g 0 , Λ J = Λ ^ g J \Lambda_J=\widehat{\Lambda}_g^J Λ J = Λ g J , and Λ P = Λ ^ g P \Lambda_P=\widehat{\Lambda}_g^P Λ P = Λ g P .
Theorem 2 (Conditional mean of the shared advantage evidence). Let G g \mathcal{G}_g G g be the sigma-field generated by the frozen generation inputs before residual-root and continuation-bank sampling. Expectations below are taken under the residual-sampling law conditional on these inputs. Assume the integrability conditions of Theorem 1. Conditionally on G g \mathcal{G}_g G g , draw roots with Equation (184) and use the unbiased centered continuation estimators from Equations (72) and (74) . In exact arithmetic, for every finite G g \mathcal{G}_g G g -measurable coefficient array ℓ \ell ℓ and every role χ ∈ { 0 , J , P } \chi\in\{0, J, \mathrm{P}\} χ ∈ { 0 , J , P } ,
E [ a ^ i ∣ G g ] = a i , E [ Λ ^ g χ [ ℓ ] ∣ G g ] = 1 n ∑ i w i χ ⟨ ℓ i , a i ⟩ . (189) \mathbb{E}[\widehat{\mathbf{a}}_i \mid\mathcal{G}_g] = \mathbf{a}_i,\qquad
\mathbb{E}[\widehat{\Lambda}^{\chi}_{g}[\ell] \mid\mathcal{G}_g] = \frac{1}{n}\sum_i w_i^{\chi}\langle\ell_i,\mathbf{a}_i\rangle. \tag*{(189)} E [ a i ∣ G g ] = a i , E [ Λ g χ [ ℓ ] ∣ G g ] = n 1 i ∑ w i χ ⟨ ℓ i , a i ⟩ . (189) Proof. For a root draw k k k , let 1 { I k = i } \mathbf{1}_{\{I_k=i\}} 1 { I k = i } be the indicator of selecting occurrence i i i . Conditional unbiasedness of the continuation bank gives
E [ 1 { I k = i } Δ a ^ ( k ) ∣ G g ] = ϖ i ( a i − a i 0 ) , \mathbb{E}\left[\mathbf{1}_{\{I_k=i\}}\Delta\widehat{\mathbf{a}}^{(k)} \mid\mathcal{G}_g\right] = \varpi_i(\mathbf{a}_i-\mathbf{a}_i^0), E [ 1 { I k = i } Δ a ( k ) ∣ G g ] = ϖ i ( a i − a i 0 ) , Substitution in Equation (187) cancels the proposal factor and proves the first identity. The second follows from Equation (188) , conditional linearity, and measurability of the frozen weights and coefficients.
The result applies componentwise to frozen parent-policy score coefficients. Nonlinear transformations, gradient norms, fusion coefficients, and adaptively selected candidates do not inherit this unbiasedness statement.
Further consequences of critic evaluation error. The following consequence also applies to fixed query observations outside the empirical batch. For such observations, A g a d v ( s , ⋅ ) A_g^{\mathrm{adv}}(s,\cdot) A g adv ( s , ⋅ ) and A g b a s e ( s , ⋅ ) A_g^{\mathrm{base}}(s,\cdot) A g base ( s , ⋅ ) denote the π g ( ⋅ ∣ s ) \pi_g(\cdot\mid s) π g ( ⋅ ∣ s ) -centered versions of Q g π g ( s , ⋅ ) Q_g^{\pi_g}(s,\cdot) Q g π g ( s , ⋅ ) and Q g 0 ( s , ⋅ ) Q_g^0(s,\cdot) Q g 0 ( s , ⋅ ) , respectively, as in Equation (182) .
Corollary 1 (Action-dependent evaluation error survives centering). Under Theorem 1, fix an observation s s s satisfying its assumptions. The one-step base recovers all centered action values at s s s exactly if and only if Q g π g ( s , a ) − Q g 0 ( s , a ) Q_g^{\pi_g}(s,a)-Q_g^0(s,a) Q g π g ( s , a ) − Q g 0 ( s , a ) is constant across actions. At batch occurrences, Equation (183) gives the corresponding centered correction.
Proof. Subtract the π g \pi_g π g -weighted mean over actions from (58) . Writing Δ Q g ( s , a ) = Q g π g ( s , a ) − Q g 0 ( s , a ) \Delta Q_g(s,a)=Q_g^{\pi_g}(s,a)-Q_g^0(s,a) Δ Q g ( s , a ) = Q g π g ( s , a ) − Q g 0 ( s , a ) for the actionwise error, equality of the centered vectors means Δ Q g ( s , a ) = ∑ a ′ ∈ A π g ( a ′ ∣ s ) Δ Q g ( s , a ′ ) \Delta Q_g(s,a)=\sum_{a'\in\mathcal{A}}\pi_g(a'\mid s)\Delta Q_g(s,a') Δ Q g ( s , a ) = ∑ a ′ ∈ A π g ( a ′ ∣ s ) Δ Q g ( s , a ′ ) for every action a a a . The right-hand side is independent of a a a , which proves necessity; a constant vector also satisfies this equality, proving sufficiency.
On-policy sampling does not impose this constancy condition or make Δ g , z \Delta_{g,z} Δ g , z vanish. To express its consequence for a policy update, let ν \nu ν be any fixed state-sampling distribution, including one obtained on-policy. Let ϑ \vartheta ϑ denote actor parameters, distinct from critic parameters θ \theta θ , and let ϑ g \vartheta_g ϑ g be the parent actor parameter, so that π g = π ϑ g \pi_g=\pi_{\vartheta_g} π g = π ϑ g . For a frozen scalar action-evaluation function f f f , define the local actor surrogate J g a c t o r \mathcal{J}_g^{\mathrm{actor}} J g actor and the parent-policy score vector s c g \mathrm{sc}_g sc g by
J g a c t o r ( ϑ ; f , ν ) = E S ∼ ν [ ∑ a π ϑ ( a ∣ S ) f ( S , a ) ] , s c g ( s , a ) = ∇ ϑ log π ϑ ( a ∣ s ) ∣ ϑ = ϑ g . (190) \mathcal{J}_g^{\mathrm{actor}}(\vartheta;f,\nu)=\mathbb{E}_{S\sim\nu}\left[\sum_a\pi_{\vartheta}(a\mid S)f(S,a)\right],\qquad
\mathrm{sc}_g(s,a)=\left.\nabla_{\vartheta}\log\pi_{\vartheta}(a\mid s)\right|_{\vartheta=\vartheta_g}. \tag*{(190)} J g actor ( ϑ ; f , ν ) = E S ∼ ν [ a ∑ π ϑ ( a ∣ S ) f ( S , a ) ] , sc g ( s , a ) = ∇ ϑ log π ϑ ( a ∣ s ) ∣ ϑ = ϑ g . (190)
In the expectation below, S ∼ ν S\sim\nu S ∼ ν and A ∣ S ∼ π g ( ⋅ ∣ S ) A\mid S\sim\pi_g(\cdot\mid S) A ∣ S ∼ π g ( ⋅ ∣ S ) ; the shorthand Q Q Q in J g a c t o r \mathcal{J}_g^{\mathrm{actor}} J g actor denotes the frozen function Q g π g Q_g^{\pi_g} Q g π g . Assume the policy is differentiable with positive action probabilities, the surrogate expectations are finite, and differentiation can be interchanged with the state expectation. Also assume absolute score integrability, where ∥ ⋅ ∥ 2 \lVert\cdot\rVert_2 ∥ ⋅ ∥ 2 denotes the Euclidean norm:
E ν , π g [ ∥ s c g ( S , A ) ∥ 2 max a ′ ∈ A ( ∣ Q g π g ( S , a ′ ) ∣ + ∣ Q g 0 ( S , a ′ ) ∣ ) ] < ∞ . (191) \mathbb{E}_{\nu,\pi_g}\left[\lVert\mathrm{sc}_g(S,A)\rVert_2\max_{a'\in\mathcal{A}}\left(|Q_g^{\pi_g}(S,a')|+|Q_g^0(S,a')|\right)\right]<\infty. \tag*{(191)} E ν , π g [ ∥ sc g ( S , A ) ∥ 2 a ′ ∈ A max ( ∣ Q g π g ( S , a ′ ) ∣ + ∣ Q g 0 ( S , a ′ ) ∣ ) ] < ∞. (191)
For a finite empirical state measure these integrability and interchange conditions hold whenever the pointwise scores and action values are finite. Then
∇ ϑ J g a c t o r ( ϑ ; Q , ν ) ∣ ϑ g − ∇ ϑ J g a c t o r ( ϑ ; Q g 0 , ν ) ∣ ϑ g = E ν , π g [ s c g ( S , A ) ( A g a d v ( S , A ) − A g b a s e ( S , A ) ) ] . (192) \left.\nabla_{\vartheta}\mathcal{J}_g^{\mathrm{actor}}(\vartheta;Q,\nu)\right|_{\vartheta_g}
-
\left.\nabla_{\vartheta}\mathcal{J}_g^{\mathrm{actor}}(\vartheta;Q_g^0,\nu)\right|_{\vartheta_g}
=
\mathbb{E}_{\nu,\pi_g}\left[\mathrm{sc}_g(S,A)\left(A_g^{\mathrm{adv}}(S,A)-A_g^{\mathrm{base}}(S,A)\right)\right]. \tag*{(192)} ∇ ϑ J g actor ( ϑ ; Q , ν ) ϑ g − ∇ ϑ J g actor ( ϑ ; Q g 0 , ν ) ϑ g = E ν , π g [ sc g ( S , A ) ( A g adv ( S , A ) − A g base ( S , A ) ) ] . (192)
This follows from ∇ ϑ π ϑ = π ϑ ∇ ϑ log π ϑ \nabla_{\vartheta}\pi_{\vartheta}=\pi_{\vartheta}\nabla_{\vartheta}\log\pi_{\vartheta} ∇ ϑ π ϑ = π ϑ ∇ ϑ log π ϑ and ∑ a ∇ ϑ π ϑ ( a ∣ s ) = 0 \sum_a\nabla_{\vartheta}\pi_{\vartheta}(a\mid s)=0 ∑ a ∇ ϑ π ϑ ( a ∣ s ) = 0 , the same baseline cancellation used in policy-gradient methods [37 ] . Residual completion supplies the missing population score term without requiring the scalar critics to have converged. This is a statement about the frozen, fixed-state surrogate. Interpreting it as the gradient of a trajectory-return objective additionally requires that objective’s correct state and risk occupancy measure.
Why the supply-chain structure makes the correction relevant. A regular order changes the pipeline immediately, but its inventory and cost consequences occur after the four-period lead time. For example, with v g ≡ 0 v_g\equiv0 v g ≡ 0 and fixed expedited quantity,
Q g 0 ( s , ( q r , q e ) ) = ∫ E z s , ( q r , q e ) [ R 0 ] m ( d z ) is independent of q r . (193) Q_g^0\left(s,\left(q^r,q^e\right)\right)=\int\mathbb{E}_{z}^{s,(q^r,q^e)}[R_0]m(\mathrm{d}z)\ \text{ is independent of }q^r. \tag*{(193)} Q g 0 ( s , ( q r , q e ) ) = ∫ E z s , ( q r , q e ) [ R 0 ] m ( d z ) is independent of q r . (193)
Immediate cost contains no regular-order charge, and the current regular quantity does not change end-of-period inventory. When supply is available, however, different regular quantities enter different successor pipelines and can produce different future holding and backlog costs. The residual continuation recovers precisely the long-term action differences omitted by this base. An approximate nonzero critic can already capture part of those differences; the same identity completes the part it misses. This issue exists in DSIS as well as DSIS-RB, regardless of whether data collection is on-policy.
Direct advantage learning for behavior and target
Suppress generation indices on the parent distributions and write m θ μ ( a ∣ s ) = μ θ μ ( a ∣ s ) m_{\theta^\mu}(a\mid s)=\mu_{\theta^\mu}(a\mid s) m θ μ ( a ∣ s ) = μ θ μ ( a ∣ s ) and p θ π ( a ∣ s ) = π θ π ( a ∣ s ) p_{\theta^\pi}(a\mid s)=\pi_{\theta^\pi}(a\mid s) p θ π ( a ∣ s ) = π θ π ( a ∣ s ) for the two parameterized candidate policies. Let θ g μ , θ g π \theta^\mu_g,\theta^\pi_g θ g μ , θ g π be their frozen parent parameter vectors, so m θ g μ = μ m_{\theta^\mu_g}=\mu m θ g μ = μ and p θ g π = π p_{\theta^\pi_g}=\pi p θ g π = π . The symbols m , p m,p m , p in the local objectives denote candidate distributions; this use of m m m is distinct from the synthetic-risk prior above. Write m i ( a ) = m θ μ ( a ∣ s i ) m_i(a)=m_{\theta^\mu}(a\mid s_i) m i ( a ) = m θ μ ( a ∣ s i ) and p i ( a ) = p θ π ( a ∣ s i ) p_i(a)=p_{\theta^\pi}(a\mid s_i) p i ( a ) = p θ π ( a ∣ s i ) for their occurrence-level rows. Assume positive, differentiable candidate probabilities near the parents and finite values and derivatives at the retained observations. Recall that τ = κ S c a l > 0 \tau=\kappa S_{\mathrm{cal}}>0 τ = κ S cal > 0 is the fixed run-level reward scale from Equation (163) ; κ > 0 \kappa>0 κ > 0 is the tuned dimensionless multiplier and S c a l > 0 S_{\mathrm{cal}}>0 S cal > 0 is the initialization utility-range scale.
The score function of a policy is its log-probability gradient. The parent behavior and target scores are
sc g μ ( s , a ) = ∇ θ μ log m θ μ ( a ∣ s ) ∣ θ g μ , \operatorname{sc}^{\mu}_{g}(s,a)=\left.\nabla_{\theta^\mu}\log m_{\theta^\mu}(a\mid s)\right|_{\theta^\mu_g}, sc g μ ( s , a ) = ∇ θ μ log m θ μ ( a ∣ s ) ∣ θ g μ ,
sc g ( s , a ) = ∇ θ π log p θ π ( a ∣ s ) ∣ θ g π . (194) \operatorname{sc}_{g}(s,a)=\left.\nabla_{\theta^\pi}\log p_{\theta^\pi}(a\mid s)\right|_{\theta^\pi_g}. \tag*{(194)} sc g ( s , a ) = ∇ θ π log p θ π ( a ∣ s ) ∣ θ g π . (194)
The target score is the sc g \operatorname{sc}_g sc g of the notation table, with ϑ g = θ g π \vartheta_g=\theta^\pi_g ϑ g = θ g π . Write sc i μ ( a ) = sc g μ ( s i , a ) \operatorname{sc}^{\mu}_i(a)=\operatorname{sc}^{\mu}_g(s_i,a) sc i μ ( a ) = sc g μ ( s i , a ) and sc i ( a ) = sc g ( s i , a ) \operatorname{sc}_i(a)=\operatorname{sc}_g(s_i,a) sc i ( a ) = sc g ( s i , a ) .
Recall that Λ ^ g x \widehat{\Lambda}^{x}_g Λ g x is the sampled linear advantage functional in Equation (186) ; role 0 uses unit weights and role J J J uses w i J w_i^J w i J . The two direct surfaces are
A ( m ) = 1 τ Λ ^ g 0 [ m − μ ] , J ( p ) = Λ ^ g J [ p − π ] . (195) A(m)=\frac{1}{\tau}\widehat{\Lambda}^{0}_g[m-\mu],\qquad J(p)=\widehat{\Lambda}^{J}_g[p-\pi]. \tag*{(195)} A ( m ) = τ 1 Λ g 0 [ m − μ ] , J ( p ) = Λ g J [ p − π ] . (195)
During differentiation, the subtracted parents μ , π \mu,\pi μ , π and the weights, base advantages, sampled root indices, and continuation corrections defining Λ ^ g x \widehat{\Lambda}^{x}_g Λ g x remain fixed. In particular, the appearance of π g \pi_g π g in the centering and continuation estimator does not create an additional derivative. Only the candidate distribution depends on the differentiated actor parameters.
Theorem 3 (Parent gradients of the direct advantage surfaces). Under the positivity and differentiability assumptions above, define g A = ∇ θ μ A ( m θ μ ) ∣ θ g μ g_A=\left.\nabla_{\theta^\mu}A(m_{\theta^\mu})\right|_{\theta^\mu_g} g A = ∇ θ μ A ( m θ μ ) ∣ θ g μ and g J = ∇ θ π ( J ( p θ π ) / τ ) ∣ θ g π g_J=\left.\nabla_{\theta^\pi}(J(p_{\theta^\pi})/\tau)\right|_{\theta^\pi_g} g J = ∇ θ π ( J ( p θ π ) / τ ) ∣ θ g π . Inside the functionals, the arrays sc μ , sc \operatorname{sc}^{\mu},\operatorname{sc} sc μ , sc have rows sc i μ , sc i \operatorname{sc}^{\mu}_i,\operatorname{sc}_i sc i μ , sc i ; multiplication by a policy probability is action-wise. With all evidence and parents frozen,
g A = 1 τ Λ ^ g 0 [ μ sc μ ] = 1 n τ ∑ i , a μ i ( a ) a ^ i ( a ) sc i μ ( a ) , (196) g_A=\frac{1}{\tau}\widehat{\Lambda}^{0}_g[\mu\,\operatorname{sc}^{\mu}]
=\frac{1}{n\tau}\sum_{i,a}\mu_i(a)\widehat{a}_i(a)\operatorname{sc}^{\mu}_i(a), \tag*{(196)} g A = τ 1 Λ g 0 [ μ sc μ ] = n τ 1 i , a ∑ μ i ( a ) a i ( a ) sc i μ ( a ) , (196) and
g J = 1 τ Λ ^ g J [ π sc ] = 1 n τ ∑ i , a w i J π i ( a ) a ^ i ( a ) sc i ( a ) . (197) g_J=\frac{1}{\tau}\widehat{\Lambda}^{J}_g[\pi\,\operatorname{sc}]
=\frac{1}{n\tau}\sum_{i,a}w_i^J\pi_i(a)\widehat{a}_i(a)\operatorname{sc}_i(a). \tag*{(197)} g J = τ 1 Λ g J [ π sc ] = n τ 1 i , a ∑ w i J π i ( a ) a i ( a ) sc i ( a ) . (197)
Proof. For behavior, linearity and the chain rule give
∇ θ μ A ( m θ μ ) = 1 n τ ∑ i , a a ^ i ( a ) ∇ θ μ m θ μ ( a ∣ s i ) = 1 n τ ∑ i , a a ^ i ( a ) m θ μ ( a ∣ s i ) ∇ θ μ log m θ μ ( a ∣ s i ) . (198) \begin{aligned}
\nabla_{\theta^\mu} A(m_{\theta^\mu}) &= \frac{1}{n\tau}\sum_{i,a}\hat{a}_i(a)\nabla_{\theta^\mu}m_{\theta^\mu}(a\mid s_i) \\
&= \frac{1}{n\tau}\sum_{i,a}\hat{a}_i(a)m_{\theta^\mu}(a\mid s_i)\nabla_{\theta^\mu}\log m_{\theta^\mu}(a\mid s_i). \tag*{(198)}
\end{aligned} ∇ θ μ A ( m θ μ ) = n τ 1 i , a ∑ a ^ i ( a ) ∇ θ μ m θ μ ( a ∣ s i ) = n τ 1 i , a ∑ a ^ i ( a ) m θ μ ( a ∣ s i ) ∇ θ μ log m θ μ ( a ∣ s i ) . (198)
Evaluation at θ g μ \theta_g^\mu θ g μ proves (196) . For target, J J J is expressed in unscaled advantage units, so
∇ θ π J ( p θ π ) τ = 1 n τ ∑ i , a w i J a ^ i ( a ) ∇ θ π p θ π ( a ∣ s i ) = 1 n τ ∑ i , a w i J a ^ i ( a ) p θ π ( a ∣ s i ) ∇ θ π log p θ π ( a ∣ s i ) . (199) \begin{aligned}
\nabla_{\theta^\pi}\frac{J(p_{\theta^\pi})}{\tau} &= \frac{1}{n\tau}\sum_{i,a}w_i^J\hat{a}_i(a)\nabla_{\theta^\pi}p_{\theta^\pi}(a\mid s_i) \\
&= \frac{1}{n\tau}\sum_{i,a}w_i^J\hat{a}_i(a)p_{\theta^\pi}(a\mid s_i)\nabla_{\theta^\pi}\log p_{\theta^\pi}(a\mid s_i). \tag*{(199)}
\end{aligned} ∇ θ π τ J ( p θ π ) = n τ 1 i , a ∑ w i J a ^ i ( a ) ∇ θ π p θ π ( a ∣ s i ) = n τ 1 i , a ∑ w i J a ^ i ( a ) p θ π ( a ∣ s i ) ∇ θ π log p θ π ( a ∣ s i ) . (199)
Evaluation at θ g π \theta_g^\pi θ g π proves (197) . □
Behavior therefore learns whether its first-action changes are supported by target-continuation evidence; no separate Q μ g Q^{\mu_g} Q μ g is estimated. Target uses the same action evidence under ν g J \nu_g^J ν g J . For both surfaces, the coefficient rows m i − μ i m_i-\mu_i m i − μ i and p i − π i p_i-\pi_i p i − π i sum to zero. For any positive differentiable policy q q q , differentiating its probability normalization gives ∑ a q ( a ∣ s ) ∇ log q ( a ∣ s ) = 0 \sum_a q(a\mid s)\nabla\log q(a\mid s)=0 ∑ a q ( a ∣ s ) ∇ log q ( a ∣ s ) = 0 , where the gradient is taken with respect to that policy’s parameters. Thus action-independent baselines cancel in these two channels. The transmission coefficients below do not generally have this cancellation property.
Advantage-filtered behavior-to-target transmission
Recall that μ i ( a ) \mu_i(a) μ i ( a ) , π i ( a ) \pi_i(a) π i ( a ) are the frozen parent probabilities, a ^ i \hat{a}_i a ^ i is the sampled discounted target-centered advantage, and Λ ^ g P \hat{\Lambda}_g^P Λ ^ g P is their linear functional with transmission weights w i P w_i^P w i P in (186) . Write δ i ( a ) = μ i ( a ) − π i ( a ) \delta_i(a)=\mu_i(a)-\pi_i(a) δ i ( a ) = μ i ( a ) − π i ( a ) for the parent probability discrepancy, distinct from the point mass δ s i \delta_{s_i} δ s i and the earlier controller multiplier gap. Write δ = ( δ i ( a ) ) i , a \delta=(\delta_i(a))_{i,a} δ = ( δ i ( a ) ) i , a for the full array; products, absolute values, and ratios inside the functional below act actionwise. The transmission surface is
P ( p ) = 1 τ Λ ^ g P [ ∣ δ ∣ ( p π − 1 ) ] . (200) P(p)=\frac{1}{\tau}\hat{\Lambda}_g^P\left[|\delta|\left(\frac{p}{\pi}-1\right)\right]. \tag*{(200)} P ( p ) = τ 1 Λ ^ g P [ ∣ δ ∣ ( π p − 1 ) ] . (200)
For an arbitrary candidate p θ π p_{\theta^\pi} p θ π ,
∇ θ π P ( p θ π ) = 1 τ Λ ^ g P [ ∣ δ ∣ π ∇ θ π p θ π ] = 1 τ Λ ^ g P [ ∣ δ ∣ p θ π π ∇ θ π log p θ π ] . (201) \nabla_{\theta^\pi}P(p_{\theta^\pi})=\frac{1}{\tau}\hat{\Lambda}_g^P\left[\frac{|\delta|}{\pi}\nabla_{\theta^\pi}p_{\theta^\pi}\right]=\frac{1}{\tau}\hat{\Lambda}_g^P\left[|\delta|\frac{p_{\theta^\pi}}{\pi}\nabla_{\theta^\pi}\log p_{\theta^\pi}\right]. \tag*{(201)} ∇ θ π P ( p θ π ) = τ 1 Λ ^ g P [ π ∣ δ ∣ ∇ θ π p θ π ] = τ 1 Λ ^ g P [ ∣ δ ∣ π p θ π ∇ θ π log p θ π ] . (201)
The parent denominator π \pi π , the discrepancy δ \delta δ , and the evidence are constants here. Evaluating at p θ g π = π p_{\theta_g^\pi}=\pi p θ g π = π cancels the probability ratio. Define g P = ∇ θ π P ( p θ π ) ∣ θ g π g_P=\left.\nabla_{\theta^\pi}P(p_{\theta^\pi})\right|_{\theta_g^\pi} g P = ∇ θ π P ( p θ π ) ∣ θ g π ; then
g P = 1 τ Λ ^ g P [ ∣ δ ∣ s c ] = 1 n τ ∑ i , a w i P ∣ δ i ( a ) ∣ a ^ i ( a ) s c i ( a ) . (202) g_P=\frac{1}{\tau}\hat{\Lambda}_g^P[|\delta|\mathop{\mathrm{sc}}]=\frac{1}{n\tau}\sum_{i,a}w_i^P|\delta_i(a)|\hat{a}_i(a)\mathop{\mathrm{sc}}_i(a). \tag*{(202)} g P = τ 1 Λ ^ g P [ ∣ δ ∣ sc ] = n τ 1 i , a ∑ w i P ∣ δ i ( a ) ∣ a ^ i ( a ) sc i ( a ) . (202)
Disagreement supplies the magnitude and advantage supplies the sign. Unlike g J g_J g J , this score coefficient is not multiplied by π i ( a ) \pi_i(a) π i ( a ) ; an action rarely selected by target can still receive substantial transmission when behavior assigns it appreciably more mass.
Centered evidence is essential because, in general, ∑ a ∣ δ i ( a ) ∣ ( p i ( a ) / π i ( a ) − 1 ) ≠ 0 \sum_a|\delta_i(a)|(p_i(a)/\pi_i(a)-1)\ne0 ∑ a ∣ δ i ( a ) ∣ ( p i ( a ) / π i ( a ) − 1 ) = 0 . Replacing advantages by uncentered action values changes P P P . Both the dense base and sampled correction are therefore centered before entering this channel.
Theorem 4 (Selective transmission in the advantage face). Fix an occurrence on the finite action set A \mathcal{A} A . Write π = ( π a ) a \boldsymbol{\pi}=(\pi_a)_a π = ( π a ) a for a strictly positive target distribution, μ = ( μ a ) a \boldsymbol{\mu}=(\mu_a)_a μ = ( μ a ) a for a behavior distribution, and e = ( e a ) a \boldsymbol{e}=(e_a)_a e = ( e a ) a for finite centered evidence with ∑ a π a e a = 0 \sum_a \pi_a e_a=0 ∑ a π a e a = 0 . The evidence can be the exact vector a i \boldsymbol{a}_i a i or the realized sampled vector a b i \boldsymbol{a}_{b_i} a b i . Let τ > 0 \tau>0 τ > 0 . Define the discrepancy δ a \delta_a δ a , its relative magnitude h a h_a h a , and the signed score coefficient b a b_a b a by
δ a = μ a − π a , h a = ∣ δ a ∣ π a , b a = ∣ δ a ∣ e a τ . (203) \delta_a=\mu_a-\pi_a,\qquad h_a=\frac{|\delta_a|}{\pi_a},\qquad b_a=\frac{|\delta_a|e_a}{\tau}. \tag*{(203)} δ a = μ a − π a , h a = π a ∣ δ a ∣ , b a = τ ∣ δ a ∣ e a . (203) Write sign ( x ) = 1 \operatorname{sign}(x)=1 sign ( x ) = 1 for x > 0 x>0 x > 0 and − 1 -1 − 1 for x < 0 x<0 x < 0 . For δ a e a ≠ 0 \delta_a e_a\ne0 δ a e a = 0 ,
b a = δ a ∣ e a ∣ τ sign ( δ a e a ) . (204) b_a=\frac{\delta_a|e_a|}{\tau}\operatorname{sign}(\delta_a e_a). \tag*{(204)} b a = τ δ a ∣ e a ∣ sign ( δ a e a ) . (204) Thus the sign of the discrepancy is retained in the coefficient b a b_a b a when δ a e a > 0 \delta_a e_a>0 δ a e a > 0 and reversed when δ a e a < 0 \delta_a e_a<0 δ a e a < 0 . For a nonnegative auxiliary step η \eta η , define the distribution q η R q_{\eta}^{R} q η R (with R R R labeling reflection) and its normalizing constant Z η Z_\eta Z η by
q η , a R = π a exp ( η h a e a / τ ) Z η , Z η = ∑ b π b exp ( η h b e b / τ ) . (205) q_{\eta,a}^{R}=\frac{\pi_a\exp(\eta h_a e_a/\tau)}{Z_\eta},\qquad Z_\eta=\sum_b\pi_b\exp(\eta h_b e_b/\tau). \tag*{(205)} q η , a R = Z η π a exp ( η h a e a / τ ) , Z η = b ∑ π b exp ( η h b e b / τ ) . (205) It satisfies
⟨ q η R − π , e ⟩ = 1 Z η ∑ a π a e a [ exp ( η h a e a / τ ) − 1 ] ≥ 0. (206) \langle q_\eta^R-\pi,e\rangle=\frac{1}{Z_\eta}\sum_a\pi_a e_a\left[\exp(\eta h_a e_a/\tau)-1\right]\ge0. \tag*{(206)} ⟨ q η R − π , e ⟩ = Z η 1 a ∑ π a e a [ exp ( η h a e a / τ ) − 1 ] ≥ 0. (206) For η > 0 \eta>0 η > 0 , the inequality is strict if and only if ∣ δ a ∣ ∣ e a ∣ > 0 |\delta_a||e_a|>0 ∣ δ a ∣∣ e a ∣ > 0 for at least one action. In addition,
d d η ⟨ q η R − π , e ⟩ ∣ η = 0 = 1 τ ∑ a ∣ δ a ∣ e a 2 . (207) \left.\frac{d}{d\eta}\langle q_\eta^R-\pi,e\rangle\right|_{\eta=0}=\frac{1}{\tau}\sum_a|\delta_a|e_a^2. \tag*{(207)} d η d ⟨ q η R − π , e ⟩ η = 0 = τ 1 a ∑ ∣ δ a ∣ e a 2 . (207) Proof. For one occurrence, the transmission contribution of action a a a to a candidate p p p is
T a ( p ) = ∣ δ a ∣ τ π a ( p a − π a ) e a . (208) T_a(p)=\frac{|\delta_a|}{\tau\pi_a}(p_a-\pi_a)e_a. \tag*{(208)} T a ( p ) = τ π a ∣ δ a ∣ ( p a − π a ) e a . (208) The omitted occurrence weight is positive and does not change signs. Whenever δ a ≠ 0 \delta_a\ne0 δ a = 0 , T a ( p ) T_a(p) T a ( p ) is positive precisely when the candidate’s coordinate change is supported by the evidence. We examine the four nonzero sign combinations separately.
Positive advantage and increased behavioral mass: e a > 0 e_a>0 e a > 0 , δ a > 0 \delta_a>0 δ a > 0 . Behavior assigns more probability to a favorable action. Here b a = δ a e a / τ > 0 b_a=\delta_a e_a/\tau>0 b a = δ a e a / τ > 0 , so increasing p a p_a p a contributes positively to P P P . The behavioral deviation is retained.
Negative advantage and decreased behavioral mass: e a < 0 e_a<0 e a < 0 , δ a < 0 \delta_a<0 δ a < 0 . Behavior assigns less probability to an unfavorable action. Here b a = ( − δ a ) e a / τ < 0 b_a=(-\delta_a)e_a/\tau<0 b a = ( − δ a ) e a / τ < 0 , and decreasing p a p_a p a makes both ( p a − π a ) (p_a-\pi_a) ( p a − π a ) and e a e_a e a negative, so T a ( p ) > 0 T_a(p)>0 T a ( p ) > 0 . This beneficial reduction is also retained.
Positive advantage and decreased behavioral mass: e a > 0 e_a>0 e a > 0 , δ a < 0 \delta_a<0 δ a < 0 . Behavior underweights a favorable action. Following that decrease would make T a ( p ) < 0 T_a(p)<0 T a ( p ) < 0 . Instead, b a = ( − δ a ) e a / τ > 0 b_a=(-\delta_a)e_a/\tau>0 b a = ( − δ a ) e a / τ > 0 rewards an increase, reflecting the behavioral deviation.
Negative advantage and increased behavioral mass: e a < 0 e_a<0 e a < 0 , δ a > 0 \delta_a>0 δ a > 0 . Behavior overweights an unfavorable action. Following that increase would again make T a ( p ) < 0 T_a(p)<0 T a ( p ) < 0 . Here b a = δ a e a / τ < 0 b_a=\delta_a e_a/\tau<0 b a = δ a e a / τ < 0 rewards a decrease, reflecting this deviation.
These cases establish Equation (204) . If e a = 0 e_a=0 e a = 0 or δ a = 0 \delta_a=0 δ a = 0 , both b a b_a b a and the coordinate contribution are zero. Coordinate statements describe contributions to the surface; feasible distributions must also obey the shared unit-mass constraint.
We next show that the auxiliary distribution respects this constraint while preserving a nonnegative total advantage. Positivity of π \pi π and finiteness of the evidence give 0 < Z η < ∞ 0<Z_\eta<\infty 0 < Z η < ∞ . Consequently q η R q_\eta^R q η R
is a positive probability distribution. Centering gives
⟨ q η R − π , e ⟩ = 1 Z η ∑ a π a e a exp ( η h a e a / τ ) = 1 Z η ∑ a π a e a [ exp ( η h a e a / τ ) − 1 ] . \langle q_{\eta}^{R}-\pi,e\rangle= \frac{1}{Z_{\eta}}\sum_a \pi_a e_a \exp(\eta h_a e_a/\tau)=\frac{1}{Z_{\eta}}\sum_a \pi_a e_a[\exp(\eta h_a e_a/\tau)-1]. ⟨ q η R − π , e ⟩ = Z η 1 a ∑ π a e a exp ( η h a e a / τ ) = Z η 1 a ∑ π a e a [ exp ( η h a e a / τ ) − 1 ] .
If e a > 0 e_a>0 e a > 0 , then h a ≥ 0 h_a\ge0 h a ≥ 0 makes the bracket nonnegative. If e a < 0 e_a<0 e a < 0 , the bracket is nonpositive, so its product with e a e_a e a is again nonnegative. If e a = 0 e_a=0 e a = 0 or h a = 0 h_a=0 h a = 0 , the product is zero. For η > 0 \eta>0 η > 0 , it is strictly positive exactly when h a ∣ e a ∣ > 0 h_a|e_a|>0 h a ∣ e a ∣ > 0 . Since all π a > 0 \pi_a>0 π a > 0 , summation proves the inequality and its strictness condition.
Finally, let N η N_\eta N η be the numerator on the right-hand side of Equation (206) . At η = 0 \eta=0 η = 0 , N 0 = 0 N_0=0 N 0 = 0 and Z 0 = 1 Z_0=1 Z 0 = 1 . The quotient rule therefore gives
d d η N η Z η ∣ 0 = N 0 ′ = 1 τ ∑ a π a h a e a 2 = 1 τ ∑ a ∣ δ a ∣ e a 2 . \left.\frac{d}{d\eta}\frac{N_\eta}{Z_\eta}\right|_0=N'_0=\frac{1}{\tau}\sum_a \pi_a h_a e_a^2=\frac{1}{\tau}\sum_a|\delta_a|e_a^2. d η d Z η N η 0 = N 0 ′ = τ 1 a ∑ π a h a e a 2 = τ 1 a ∑ ∣ δ a ∣ e a 2 .
which proves Equation (207) .
Here favorable and unfavorable mean positive and negative centered evidence relative to π \pi π . The theorem does not require behavior to be better than target in aggregate: beneficial deviations in the first two cases are retained even when behavior has harmful deviations elsewhere. The latter two cases use the disagreement magnitude while reversing its unsupported direction.
The auxiliary distribution also explains the surface itself. For any candidate action distribution p p p at the same occurrence, setting η = 1 \eta=1 η = 1 and using probability normalization gives
∑ a ( p a − π a ) log q η , a R π a = ∑ a ( p a − π a ) ( h a e a τ − log Z 1 ) = 1 τ ∑ a ∣ μ a − π a ∣ ( p a π a − 1 ) e a . (209) \sum_a(p_a-\pi_a)\log\frac{q_{\eta,a}^{R}}{\pi_a}
=\sum_a(p_a-\pi_a)\left(\frac{h_a e_a}{\tau}-\log Z_1\right)
=\frac{1}{\tau}\sum_a|\mu_a-\pi_a|\left(\frac{p_a}{\pi_a}-1\right)e_a. \tag*{(209)} a ∑ ( p a − π a ) log π a q η , a R = a ∑ ( p a − π a ) ( τ h a e a − log Z 1 ) = τ 1 a ∑ ∣ μ a − π a ∣ ( π a p a − 1 ) e a . (209)
The normalizer cancels because ∑ a ( p a − π a ) = 0 \sum_a(p_a-\pi_a)=0 ∑ a ( p a − π a ) = 0 . The implementation evaluates the right-hand side directly; q η R q_{\eta}^{R} q η R is a proof device, not an additional actor or a policy-mixing step.
With exact evidence e = a i e=a_i e = a i , the theorem concerns the frozen model’s first-action evaluation. With e = a ^ i e=\hat{a}_i e = a ^ i , it is a pathwise statement about the realized occurrence-level evidence; repeated states need not define a single common auxiliary policy. The specific auxiliary path has nonnegative advantage gain, but an arbitrary increase of P P P need not increase the unweighted advantage face. Nor does the theorem assign signs to every coordinate of a parameterized actor update. For independent categorical logits at this occurrence, the actual parent logit gradient for this occurrence is the positive occurrence weight times the following vector, where b = ( b a ) a b=(b_a)_a b = ( b a ) a recalls the coefficients in Equation (203) :
b − π ∑ a b a (210) b-\pi\sum_a b_a \tag*{(210)} b − π a ∑ b a (210)
because differentiating the softmax couples all action probabilities. Shared network parameters further couple states. The channel can increase behavior–target disagreement. It vanishes identically when μ = π \mu=\pi μ = π ; a zero parameter gradient alone does not imply a constant finite surface.
Why the advantage channels use the scale 1/τ \tau τ
Recall that g H = g H , g g_H=g_{H,g} g H = g H , g is the parent gradient of the heuristic objective H g H_g H g in Equation (180) . The heuristic objective is a dimensionless squared statistical distance; its utility dependence enters through the regularized controller. The unscaled advantage surfaces instead carry reward units. Since τ = κ S c a l \tau=\kappa S_{\mathrm{cal}} τ = κ S cal in Equation (163) has the same units as the heuristic utility, division by τ \tau τ expresses the three advantage channels in compatible units. The next result also quantifies the utility-induced scale of guidance while accounting for any behavior–target mismatch already present at the parent.
Theorem 5 (Scale of heuristic guidance). *Recall the frozen centered heuristic utility u g u_g u g , the Hellinger-regularized controller q H , g q_{H,g} q H , g from Equation (168) , and its finite empirical fitting measure ξ g \xi_g ξ g from Equation (174) . The utility is the undiscounted finite-horizon quantity, distinct from the discounted advantages a ^ i \hat{a}_i a ^ i . Recall also d F R ( p , q ) = 2 arccos ∑ a p a q a \mathrm{d}_{\mathrm{FR}}(p,q)=2\arccos\sum_a\sqrt{p_aq_a} d FR ( p , q ) = 2 arccos ∑ a p a q a for action distributions
p p p , q q q . Assume finite utilities and actor scores on this panel, positive parent probabilities, and τ > 0 \tau> 0 τ > 0 . Define the utility range R g ( s ) R_g(s) R g ( s ) and the root-mean-square parent distance D μ , π D_{\mu,\pi} D μ , π by
R g ( s ) = max a u g ( s , a ) − min a u g ( s , a ) , D μ , π = ( E ξ g d F R 2 ( μ g , π g ) ) 1 / 2 . (211) R_g(s) = \max_a u_g(s,a) - \min_a u_g(s,a), \qquad D_{\mu,\pi} = \left(\mathbb{E}_{\xi_g} d_{\mathrm{FR}}^2(\mu_g,\pi_g)\right)^{1/2}. \tag*{(211)} R g ( s ) = a max u g ( s , a ) − a min u g ( s , a ) , D μ , π = ( E ξ g d FR 2 ( μ g , π g ) ) 1/2 . (211)
For symmetric matrices, write X ⪰ Y X \succeq Y X ⪰ Y when X − Y X-Y X − Y is positive semidefinite and X ≻ 0 X \succ0 X ≻ 0 when X X X is positive definite; a superscript ⊤ \top ⊤ denotes transpose. Let the behavior-actor matrix F μ ≻ 0 F_\mu\succ0 F μ ≻ 0 satisfy
F μ ⪰ 1 2 E s ∼ ξ g ∑ a μ g ( a ∣ s ) sc g μ ( s , a ) sc g μ ( s , a ) ⊤ . (212) F_\mu\succeq\frac{1}{2}\mathbb{E}_{s\sim\xi_g}\sum_a \mu_g(a\mid s)\operatorname{sc}_g^\mu(s,a)\operatorname{sc}_g^\mu(s,a)^\top. \tag*{(212)} F μ ⪰ 2 1 E s ∼ ξ g a ∑ μ g ( a ∣ s ) sc g μ ( s , a ) sc g μ ( s , a ) ⊤ . (212)
The actor matrix defined below has this property. Use ∥ x ∥ G = ( x ⊤ G x ) 1 / 2 \lVert x\rVert_G = (x^\top Gx)^{1/2} ∥ x ∥ G = ( x ⊤ G x ) 1/2 for a positive-definite G G G . The numerical factor π \pi π in the following bounds is the circle constant. Then
d F R ( π g ( ⋅ ∣ s ) , q H , g ( ⋅ ∣ s ) ) ≤ π R g ( s ) 4 τ , (213) d_{\mathrm{FR}}(\pi_g(\cdot\mid s),q_{H,g}(\cdot\mid s)) \le\frac{\pi R_g(s)}{4\tau}, \tag*{(213)} d FR ( π g ( ⋅ ∣ s ) , q H , g ( ⋅ ∣ s )) ≤ 4 τ π R g ( s ) , (213)
and
∥ g H ∥ F μ − 1 ≤ 2 [ D μ , π + π 4 τ ( E ξ g R g ( S ) 2 ) 1 / 2 ] . (214) \lVert g_H\rVert_{F_\mu^{-1}} \le\sqrt{2}\left[D_{\mu,\pi}+\frac{\pi}{4\tau}\left(\mathbb{E}_{\xi_g}R_g(S)^2\right)^{1/2}\right]. \tag*{(214)} ∥ g H ∥ F μ − 1 ≤ 2 [ D μ , π + 4 τ π ( E ξ g R g ( S ) 2 ) 1/2 ] . (214)
In particular, at matched parents μ g = π g \mu_g=\pi_g μ g = π g , the natural norm is O ( τ − 1 ) O(\tau^{-1}) O ( τ − 1 ) as τ → ∞ \tau\to\infty τ → ∞ for fixed fitting locations, utility evidence, parent policies, and score vectors.
Proof. At one fitting location, abbreviate p = π g ( ⋅ ∣ s ) p=\pi_g(\cdot\mid s) p = π g ( ⋅ ∣ s ) , q = q H , g ( ⋅ ∣ s ) q=q_{H,g}(\cdot\mid s) q = q H , g ( ⋅ ∣ s ) , u a = u g ( s , a ) u_a=u_g(s,a) u a = u g ( s , a ) , and λ = λ g ( s ) \lambda=\lambda_g(s) λ = λ g ( s ) , the controller’s unit-mass multiplier. Set B π , H = ∑ a p a q a B_{\pi,H}=\sum_a\sqrt{p_aq_a} B π , H = ∑ a p a q a . Here E q \mathbb{E}_q E q and Var q \operatorname{Var}_q Var q denote expectation and variance over actions drawn from q q q . This affinity is between target and controller, distinct from the behavior–controller affinity in Equation (178) . The stationarity equation (167) gives λ − u a = 2 τ p a / q a \lambda-u_a=2\tau\sqrt{p_a/q_a} λ − u a = 2 τ p a / q a . Its first two moments under q q q are
E q ( λ − u ) = 2 τ B π , H , E q [ ( λ − u ) 2 ] = 4 τ 2 . \mathbb{E}_q(\lambda-u)=2\tau B_{\pi,H}, \qquad\mathbb{E}_q[(\lambda-u)^2]=4\tau^2. E q ( λ − u ) = 2 τ B π , H , E q [( λ − u ) 2 ] = 4 τ 2 .
Thus Var q ( u ) = 4 τ 2 ( 1 − B π , H 2 ) \operatorname{Var}_q(u)=4\tau^2(1-B_{\pi,H}^2) Var q ( u ) = 4 τ 2 ( 1 − B π , H 2 ) . Set ψ = arccos B π , H ∈ [ 0 , π / 2 ] \psi=\arccos B_{\pi,H}\in[0,\pi/2] ψ = arccos B π , H ∈ [ 0 , π /2 ] . Since 2 ψ ≤ π sin ψ 2\psi\le\pi\sin\psi 2 ψ ≤ π sin ψ and a variable of range R g ( s ) R_g(s) R g ( s ) has variance at most R g ( s ) 2 / 4 R_g(s)^2/4 R g ( s ) 2 /4 ,
d F R ( p , q ) = 2 ψ ≤ π 2 τ Var q ( u ) ≤ π R g ( s ) 4 τ . d_{\mathrm{FR}}(p,q)=2\psi\le\frac{\pi}{2\tau}\sqrt{\operatorname{Var}_q(u)}\le\frac{\pi R_g(s)}{4\tau}. d FR ( p , q ) = 2 ψ ≤ 2 τ π Var q ( u ) ≤ 4 τ π R g ( s ) .
For the guidance gradient, apply Cauchy–Schwarz to the centered formula in Equation (180) . Recall the behavior–controller affinity B g ( s ) = ∑ a μ g ( a ∣ s ) q H , g ( a ∣ s ) B_g(s)=\sum_a\sqrt{\mu_g(a\mid s)q_{H,g}(a\mid s)} B g ( s ) = ∑ a μ g ( a ∣ s ) q H , g ( a ∣ s ) and angle φ g ( s ) = arccos B g ( s ) \varphi_g(s)=\arccos B_g(s) φ g ( s ) = arccos B g ( s ) . At each state, the centered factor q H , g / μ g − B g \sqrt{q_{H,g}/\mu_g}-B_g q H , g / μ g − B g has second moment 1 − B g 2 = sin 2 φ g 1-B_g^2=\sin^2\varphi_g 1 − B g 2 = sin 2 φ g under μ g \mu_g μ g ; this cancels the denominator sin φ g \sin\varphi_g sin φ g in that formula. The coincident case follows by continuity. For any parameter direction v v v , a second Cauchy–Schwarz inequality over the fitting locations gives
∣ g H ⊤ v ∣ ≤ ( E ξ g d F R 2 ( μ g , q H , g ) ) 1 / 2 ( E ξ g ∑ a μ g ( a ∣ S ) [ sc g μ ( S , a ) ⊤ v ] 2 ) 1 / 2 . (215) \lvert g_H^\top v\rvert\le\left(\mathbb{E}_{\xi_g}d_{\mathrm{FR}}^2(\mu_g,q_{H,g})\right)^{1/2}\left(\mathbb{E}_{\xi_g}\sum_a\mu_g(a\mid S)[\operatorname{sc}_g^\mu(S,a)^\top v]^2\right)^{1/2}. \tag*{(215)} ∣ g H ⊤ v ∣ ≤ ( E ξ g d FR 2 ( μ g , q H , g ) ) 1/2 ( E ξ g a ∑ μ g ( a ∣ S ) [ sc g μ ( S , a ) ⊤ v ] 2 ) 1/2 . (215)
Equation (212) bounds the last factor by 2 ∥ v ∥ F μ \sqrt{2}\lVert v\rVert_{F_\mu} 2 ∥ v ∥ F μ . Taking the dual norm, then using the metric triangle inequality and Minkowski’s inequality, proves Equation (214) .
For a more direct interpretation, hold the finite fitting locations, utility values, parent policies, and score vectors fixed and let τ → ∞ \tau\to\infty τ → ∞ . The remainders below are uniform over this finite panel and action set; the vector remainder is measured in the Euclidean parameter norm. At each state, put ϵ τ = 1 / ( 2 τ ) \epsilon_\tau=1/(2\tau) ϵ τ = 1/ ( 2 τ ) and y τ = λ g ( s ) / ( 2 τ ) y_\tau=\lambda_g(s)/(2\tau) y τ = λ g ( s ) / ( 2 τ ) . The controller normalization is ∑ a π g ( a ∣ s ) ( y τ − ϵ τ u g ( s , a ) ) = 1 \sum_a\pi_g(a\mid s)(y_\tau-\epsilon_\tau u_g(s,a))=1 ∑ a π g ( a ∣ s ) ( y τ − ϵ τ u g ( s , a )) = 1 . At ( y τ , ϵ τ ) = ( 1 , 0 ) (y_\tau,\epsilon_\tau)=(1,0) ( y τ , ϵ τ ) = ( 1 , 0 ) , its derivative with respect to y τ y_\tau y τ is 1 1 1 , so the implicit function theorem and target centering give y τ = 1 + O ( ϵ τ 2 ) y_\tau=1+O(\epsilon_\tau^2) y τ = 1 + O ( ϵ τ 2 ) . Expanding the inverse square therefore yields
q H , g ( a ∣ s ) = π g ( a ∣ s ) [ 1 + u g ( s , a ) τ + O ( τ − 2 ) ] . (216) q_{H,g}(a\mid s)=\pi_g(a\mid s)\left[1+\frac{u_g(s,a)}{\tau}+O(\tau^{-2})\right]. \tag*{(216)} q H , g ( a ∣ s ) = π g ( a ∣ s ) [ 1 + τ u g ( s , a ) + O ( τ − 2 ) ] . (216)
At μ g = π g \mu_g=\pi_g μ g = π g , this implies, pointwise on the fixed panel, π g q H , g = π g [ 1 + u g / ( 2 τ ) + O ( τ − 2 ) ] \sqrt{\pi_gq_{H,g}}=\pi_g[1+u_g/(2\tau)+O(\tau^{-2})] π g q H , g = π g [ 1 + u g / ( 2 τ ) + O ( τ − 2 )] and B g = 1 + O ( τ − 2 ) B_g=1+O(\tau^{-2}) B g = 1 + O ( τ − 2 ) . Also 2 φ g / sin φ g = 2 + O ( τ − 2 ) 2\varphi_g/\sin\varphi_g=2+O(\tau^{-2}) 2 φ g / sin φ g = 2 + O ( τ − 2 ) , with the continuous value 2 2 2 at φ g = 0 \varphi_g=0 φ g = 0 . Substitution in the centered guidance gradient of Equation (180) yields
g H = 1 τ E ξ g ∑ a π g ( a ∣ S ) u g ( S , a ) sc g μ ( S , a ) + O ( τ − 2 ) . (217) g_H=\frac{1}{\tau}\mathbb{E}_{\xi_g}\sum_a\pi_g(a\mid S)u_g(S,a)\operatorname{sc}^{\mu}_g(S,a)+O(\tau^{-2}). \tag*{(217)} g H = τ 1 E ξ g a ∑ π g ( a ∣ S ) u g ( S , a ) sc g μ ( S , a ) + O ( τ − 2 ) . (217)
This fixed-panel expansion does not differentiate through the controller rollouts that originally selected ξ g \xi_g ξ g . In general, the whole gradient need not decay as 1 / τ 1/\tau 1/ τ : if u g ≡ 0 u_g\equiv0 u g ≡ 0 , then q H , g = π g q_{H,g}=\pi_g q H , g = π g for every τ \tau τ , while guidance can remain nonzero when μ g ≠ π g \mu_g\ne\pi_g μ g = π g . The mismatch term in Theorem 5 accounts for this component.
There is also an exact unit-conversion interpretation. Multiplying all frozen return evidence and calibration utilities by c > 0 c>0 c > 0 multiplies u g u_g u g , a ^ i \hat{a}_i a ^ i , and τ \tau τ by c c c . The controller, H g H_g H g , and the scaled surfaces A A A , J / τ J/\tau J / τ , P P P are unchanged. This is an invariance of the frozen update interface, not a claim that arbitrary critic optimization is invariant to a change of reward units. It does not make the four gradient norms equal; their relative contributions are handled by the fusion below.
Fisher geometry and the Fuse construction
The score-based Fisher metric and inverse-Fisher preconditioning follow the natural-policy-gradient construction of Kakade [20 ] . Here we specify the frozen consumer measures and the fusion rule. Recall the four empirical state measures: D g D_g D g is the unweighted generation batch, ν g J \nu_g^J ν g J and ν g P \nu_g^P ν g P use the positive mean-one weights w i J w_i^J w i J and w i P w_i^P w i P , and ξ g \xi_g ξ g is the heuristic fitting measure from Equation (174) . For a parent actor q q q and a frozen probability measure ν \nu ν on observations, let sc q ( s , a ) \operatorname{sc}_q(s,a) sc q ( s , a ) be its log-probability gradient at its own parent parameters. Define its empirical Fisher matrix F ( q , ν ) F(q,\nu) F ( q , ν ) by
F ( q , ν ) = E s ∼ ν ∑ a q ( a ∣ s ) sc q ( s , a ) sc q ( s , a ) ⊤ , (218) F(q,\nu)=\mathbb{E}_{s\sim\nu}\sum_a q(a\mid s)\operatorname{sc}_q(s,a)\operatorname{sc}_q(s,a)^\top, \tag*{(218)} F ( q , ν ) = E s ∼ ν a ∑ q ( a ∣ s ) sc q ( s , a ) sc q ( s , a ) ⊤ , (218)
Let I I I denote the identity matrix on the relevant actor parameter space and λ F > 0 \lambda_F>0 λ F > 0 the damping coefficient. The common matrix for each actor’s two channels is
F μ = 1 2 [ F ( μ g , D g ) + F ( μ g , ξ g ) ] + λ F I , (219) F_\mu=\frac{1}{2}\left[F(\mu_g,D_g)+F(\mu_g,\xi_g)\right]+\lambda_F I, \tag*{(219)} F μ = 2 1 [ F ( μ g , D g ) + F ( μ g , ξ g ) ] + λ F I , (219)
F π = 1 2 [ F ( π g , ν g J ) + F ( π g , ν g P ) ] + λ F I . F_\pi=\frac{1}{2}\left[F(\pi_g,\nu_g^J)+F(\pi_g,\nu_g^P)\right]+\lambda_F I. F π = 2 1 [ F ( π g , ν g J ) + F ( π g , ν g P ) ] + λ F I .
Thus the common matrix contains a single ridge term λ F I \lambda_F I λ F I . The matrices are positive definite and actor-specific; the two policies do not share a parameter-space update.
For either ( g 1 , g 2 ) = ( g A , g H ) (g_1,g_2)=(g_A,g_H) ( g 1 , g 2 ) = ( g A , g H ) or ( g 1 , g 2 ) = ( g J , g P ) (g_1,g_2)=(g_J,g_P) ( g 1 , g 2 ) = ( g J , g P ) , use the corresponding fixed matrix F = F μ F=F_\mu F = F μ or F = F π F=F_\pi F = F π . In this subsection, i ∈ { 1 , 2 } i\in\{1,2\} i ∈ { 1 , 2 } indexes channels, rather than data occurrences. Define their natural norms n i n_i n i , their maximum n max n_{\max} n m a x , and the scalar denominator Δ F \Delta_F Δ F by
n i = g i ⊤ F − 1 g i , n max = max ( n 1 , n 2 ) , Δ F = n 1 2 + 6 n 1 n 2 + n 2 2 . (220) n_i=\sqrt{g_i^\top F^{-1}g_i},\qquad n_{\max}=\max(n_1,n_2),\qquad\Delta_F=n_1^2+6n_1n_2+n_2^2. \tag*{(220)} n i = g i ⊤ F − 1 g i , n m a x = max ( n 1 , n 2 ) , Δ F = n 1 2 + 6 n 1 n 2 + n 2 2 . (220)
For n max > 0 n_{\max}>0 n m a x > 0 , define the fusion coefficients α 1 , α 2 \alpha_1,\alpha_2 α 1 , α 2 and the ideal parameter direction d d d by
α 1 = n max ( n 1 + 3 n 2 ) Δ F , α 2 = n max ( n 2 + 3 n 1 ) Δ F , (221) \alpha_1=\frac{n_{\max}(n_1+3n_2)}{\Delta_F},\qquad\alpha_2=\frac{n_{\max}(n_2+3n_1)}{\Delta_F}, \tag*{(221)} α 1 = Δ F n m a x ( n 1 + 3 n 2 ) , α 2 = Δ F n m a x ( n 2 + 3 n 1 ) , (221)
d = Fuse F ( g 1 , g 2 ) = F − 1 ( α 1 g 1 + α 2 g 2 ) . d=\operatorname{Fuse}_F(g_1,g_2)=F^{-1}(\alpha_1g_1+\alpha_2g_2). d = Fuse F ( g 1 , g 2 ) = F − 1 ( α 1 g 1 + α 2 g 2 ) .
If both norms are zero, set both coefficients and d d d to zero. For n max > 0 n_{\max}>0 n m a x > 0 , the normalized contributions
ω 1 = n 1 ( n 1 + 3 n 2 ) Δ F , ω 2 = n 2 ( n 2 + 3 n 1 ) Δ F (222) \omega_1=\frac{n_1(n_1+3n_2)}{\Delta_F},\qquad\omega_2=\frac{n_2(n_2+3n_1)}{\Delta_F} \tag*{(222)} ω 1 = Δ F n 1 ( n 1 + 3 n 2 ) , ω 2 = Δ F n 2 ( n 2 + 3 n 1 ) (222)
satisfy ω 1 + ω 2 = 1 \omega_1+\omega_2=1 ω 1 + ω 2 = 1 and α i n i = n max ω i \alpha_i n_i=n_{\max}\omega_i α i n i = n m a x ω i . The α i \alpha_i α i themselves need not sum to one.
Theorem 6 (Variational characterization of the fusion weights). Let n 1 , n 2 > 0 n_1,n_2>0 n 1 , n 2 > 0 , and define the two-coordinate probability simplex Δ 2 = { ω ∈ R 2 : ω i ≥ 0 , ω 1 + ω 2 = 1 } \Delta_2=\{\omega\in\mathbb{R}^2:\omega_i\ge0,\ \omega_1+\omega_2=1\} Δ 2 = { ω ∈ R 2 : ω i ≥ 0 , ω 1 + ω 2 = 1 } . For probability vectors x , y x,y x , y with y i > 0 y_i>0 y i > 0 , define the Pearson divergence χ 2 ( x ∥ y ) = ∑ i ( x i − y i ) 2 / y i \chi^2(x\mathbin{\|}y)=\sum_i(x_i-y_i)^2/y_i χ 2 ( x ∥ y ) = ∑ i ( x i − y i ) 2 / y i . The weights in Equation (222) are the unique solution of
min ω ∈ Δ 2 { χ 2 ( ω ∥ ( n 1 , n 2 ) n 1 + n 2 ) + χ 2 ( ω ∥ ( 1 / 2 , 1 / 2 ) ) } . (223) \min_{\omega\in\Delta_2}\left\{\chi^2\left(\omega\mathbin{\Big\|}\frac{(n_1,n_2)}{n_1+n_2}\right)+\chi^2\left(\omega\mathbin{\Big\|}(1/2,1/2)\right)\right\}. \tag*{(223)} ω ∈ Δ 2 min { χ 2 ( ω n 1 + n 2 ( n 1 , n 2 ) ) + χ 2 ( ω ( 1/2 , 1/2 ) ) } . (223)
Proof. Write r i = n i / ( n 1 + n 2 ) r_i = n_i/(n_1+n_2) r i = n i / ( n 1 + n 2 ) for the normalized natural strengths. The objective equals ∑ i ω i 2 ( 1 / r i + 2 ) − 2 \sum_i \omega_i^2(1/r_i+2)-2 ∑ i ω i 2 ( 1/ r i + 2 ) − 2 . Its constrained minimizer obeys ω i ∝ ( 1 / r i + 2 ) − 1 \omega_i \propto(1/r_i+2)^{-1} ω i ∝ ( 1/ r i + 2 ) − 1 , yielding Equation (222) . The positive diagonal quadratic coefficients make the objective strictly convex, proving uniqueness.
For one zero norm, use the extended convention: a summand with y i = 0 y_i=0 y i = 0 is zero if x i = 0 x_i=0 x i = 0 and + ∞ +\infty + ∞ if x i > 0 x_i>0 x i > 0 . The only finite feasible weight then assigns all mass to the nonzero channel, agreeing with Equation (222) . The double-zero case uses the separately defined zero direction and requires no normalized votes. The two terms balance relative natural-gradient strength and equal channel contributions.
Equal positive norms give α 1 = α 2 = 1 / 2 \alpha_1=\alpha_2=1/2 α 1 = α 2 = 1/2 . If n 1 > 0 , n 2 = 0 n_1>0,n_2=0 n 1 > 0 , n 2 = 0 , the coefficients are ( 1 , 3 ) (1,3) ( 1 , 3 ) and d = F − 1 g 1 d=F^{-1}g_1 d = F − 1 g 1 . The second gradient contributes zero, but its frozen finite objective need not be constant away from the parent. A vanishing channel has a continuously vanishing contribution, and the fused direction extends continuously to the double-zero case without a threshold or denominator floor.
Theorem 7. (Bounded fusion and channel compatibility). Let θ ⋆ \theta_\star θ ⋆ denote the parent parameter vector of the actor under consideration, and let U 1 , U 2 U_1,U_2 U 1 , U 2 be two differentiable channel objectives on a neighborhood of θ ⋆ \theta_\star θ ⋆ . Let F ≻ 0 F\succ0 F ≻ 0 be fixed and g i = ∇ U i ( θ ⋆ ) g_i=\nabla U_i(\theta_\star) g i = ∇ U i ( θ ⋆ ) for i ∈ { 1 , 2 } i\in\{1,2\} i ∈ { 1 , 2 } . Construct n i , α i , d n_i,\alpha_i,d n i , α i , d as in Equations (220) –(221) ; the votes ω i \omega_i ω i in Equation (222) are used only when n max > 0 n_{\max}>0 n m a x > 0 . Recall ∥ x ∥ F = ( x ⊤ F x ) 1 / 2 \lVert x\rVert_F=(x^\top F x)^{1/2} ∥ x ∥ F = ( x ⊤ F x ) 1/2 . When n 1 n 2 > 0 n_1n_2>0 n 1 n 2 > 0 , define the unit natural directions v i v_i v i and their Fisher cosine c F c_F c F by
v i = F − 1 g i n i , c F = g 1 ⊤ F − 1 g 2 n 1 n 2 ∈ [ − 1 , 1 ] . (224) v_i=\frac{F^{-1}g_i}{n_i},\qquad c_F=\frac{g_1^\top F^{-1}g_2}{n_1n_2}\in[-1,1]. \tag*{(224)} v i = n i F − 1 g i , c F = n 1 n 2 g 1 ⊤ F − 1 g 2 ∈ [ − 1 , 1 ] . (224) For each i ∈ { 1 , 2 } i\in\{1,2\} i ∈ { 1 , 2 } , let j = 3 − i j=3-i j = 3 − i denote the other channel. Then
d = n max ( ω 1 v 1 + ω 2 v 2 ) , d=n_{\max}(\omega_1v_1+\omega_2v_2), d = n m a x ( ω 1 v 1 + ω 2 v 2 ) , n max ∣ ω 1 − ω 2 ∣ ≤ ∥ d ∥ F ≤ n max , n_{\max}|\omega_1-\omega_2|\le\lVert d\rVert_F\le n_{\max}, n m a x ∣ ω 1 − ω 2 ∣ ≤ ∥ d ∥ F ≤ n m a x , g i ⊤ d = n max n i ( ω i + c F ω j ) , j ≠ i . (225) g_i^\top d=n_{\max}n_i(\omega_i+c_F\omega_j),\qquad j\ne i. \tag*{(225)} g i ⊤ d = n m a x n i ( ω i + c F ω j ) , j = i . (225) Both directional derivatives are nonnegative if and only if
c F ≥ − min { ω 1 ω 2 , ω 2 ω 1 } . (226) c_F\ge-\min\left\{\frac{\omega_1}{\omega_2},\frac{\omega_2}{\omega_1}\right\}. \tag*{(226)} c F ≥ − min { ω 2 ω 1 , ω 1 ω 2 } . (226) The channel with the larger natural norm always has a nonnegative directional derivative. If n 1 ≥ n 2 n_1\ge n_2 n 1 ≥ n 2 , the weaker channel satisfies
g 2 ⊤ d ≥ − n max n 2 n 1 2 − n 2 2 Δ F . (227) g_2^\top d\ge-n_{\max}n_2\frac{n_1^2-n_2^2}{\Delta_F}. \tag*{(227)} g 2 ⊤ d ≥ − n m a x n 2 Δ F n 1 2 − n 2 2 . (227) For any norms, hold α 1 , α 2 \alpha_1,\alpha_2 α 1 , α 2 fixed and set U = α 1 U 1 + α 2 U 2 U=\alpha_1U_1+\alpha_2U_2 U = α 1 U 1 + α 2 U 2 . Then ∇ U ( θ ⋆ ) ⊤ d = ∥ d ∥ F 2 \nabla U(\theta_\star)^\top d=\lVert d\rVert_F^2 ∇ U ( θ ⋆ ) ⊤ d = ∥ d ∥ F 2 . If U U U is twice continuously differentiable along the update segment and ∇ 2 U ⪰ − L F \nabla^2U\succeq-LF ∇ 2 U ⪰ − L F there for a nonnegative curvature constant L L L , then for a parameter-step length t ≥ 0 t\ge0 t ≥ 0 ,
U ( θ ⋆ + t d ) − U ( θ ⋆ ) ≥ t ( 1 − L t 2 ) ∥ d ∥ F 2 , t ≥ 0. (228) U(\theta_\star+td)-U(\theta_\star)\ge t\left(1-\frac{Lt}{2}\right)\lVert d\rVert_F^2,\qquad t\ge0. \tag*{(228)} U ( θ ⋆ + t d ) − U ( θ ⋆ ) ≥ t ( 1 − 2 L t ) ∥ d ∥ F 2 , t ≥ 0. (228) The bound applies whenever the entire segment is in the stated domain.
Proof. The identity α i n i = n max ω i \alpha_i n_i=n_{\max}\omega_i α i n i = n m a x ω i gives the representation of d d d . Each v i v_i v i has unit F F F -norm, so the triangle and reverse-triangle inequalities give the two norm bounds. Since g i ⊤ v i = n i g_i^\top v_i=n_i g i ⊤ v i = n i and g i ⊤ v j = n i c F g_i^\top v_j=n_i c_F g i ⊤ v j = n i c F , taking the inner product with g i g_i g i gives its slope. Both slopes are nonnegative exactly when c F ≥ − ω 1 / ω 2 c_F\ge-\omega_1/\omega_2 c F ≥ − ω 1 / ω 2 and c F ≥ − ω 2 / ω 1 c_F\ge-\omega_2/\omega_1 c F ≥ − ω 2 / ω 1 , which is Equation (226) .
Furthermore, ω 1 − ω 2 = ( n 1 2 − n 2 2 ) / Δ F \omega_1-\omega_2=(n_1^2-n_2^2)/\Delta_F ω 1 − ω 2 = ( n 1 2 − n 2 2 ) / Δ F . If n 1 ≥ n 2 n_1\ge n_2 n 1 ≥ n 2 , then ω 1 ≥ ω 2 \omega_1\ge\omega_2 ω 1 ≥ ω 2 and c F ≥ − 1 c_F\ge-1 c F ≥ − 1 gives g 1 ⊤ d ≥ n max n 1 ( ω 1 − ω 2 ) ≥ 0 g_1^\top d\ge n_{\max}n_1(\omega_1-\omega_2)\ge0 g 1 ⊤ d ≥ n m a x n 1 ( ω 1 − ω 2 ) ≥ 0 and the stated lower bound for g 2 ⊤ d g_2^\top d g 2 ⊤ d . The zero-norm cases follow directly from positive definiteness of F F F and the definition of d d d .
Finally, with the coefficients frozen, ∇ U ( θ ⋆ ) = α 1 g 1 + α 2 g 2 = F d \nabla U(\theta_\star)=\alpha_1g_1+\alpha_2g_2=Fd ∇ U ( θ ⋆ ) = α 1 g 1 + α 2 g 2 = F d . Taylor’s formula with integral remainder yields
U ( θ ⋆ + t d ) − U ( θ ⋆ ) = t ∥ d ∥ F 2 + ∫ 0 t ( t − s ) d ⊤ ∇ 2 U ( θ ⋆ + s d ) d d s . U(\theta_\star+td)-U(\theta_\star)=t\lVert d\rVert_F^2+\int_0^t(t-s)d^\top\nabla^2U(\theta_\star+sd)d\,ds. U ( θ ⋆ + t d ) − U ( θ ⋆ ) = t ∥ d ∥ F 2 + ∫ 0 t ( t − s ) d ⊤ ∇ 2 U ( θ ⋆ + s d ) d d s .
The Hessian lower bound makes the integral at least − L t 2 ∥ d ∥ F 2 / 2 -Lt^2\lVert d\rVert_F^2/2 − L t 2 ∥ d ∥ F 2 /2 , proving the result.
Likewise, suppose each U i U_i U i is twice continuously differentiable on the segment and ∇ 2 U i ⪰ − L i F \nabla^2 U_i \succeq-L_i F ∇ 2 U i ⪰ − L i F there for constants L i ≥ 0 L_i \ge0 L i ≥ 0 . For the same nonnegative step length t t t ,
U i ( θ ⋆ + t d ) − U i ( θ ⋆ ) ≥ t g i T d − L i t 2 2 ∥ d ∥ F 2 . (229) U_i(\theta_\star+ td) - U_i(\theta_\star) \ge tg_i^\mathsf{T}d - \frac{L_i t^2}{2}\lVert d\rVert_F^2. \tag*{(229)} U i ( θ ⋆ + t d ) − U i ( θ ⋆ ) ≥ t g i T d − 2 L i t 2 ∥ d ∥ F 2 . (229)
When both norms are positive, the slope is g i T d = n max n i ( ω i + c F ω j ) g_i^\mathsf{T}d = n_{\max}n_i(\omega_i + c_F\omega_j) g i T d = n m a x n i ( ω i + c F ω j ) ; the displayed form also covers zero-norm channels. Strictly positive initial slopes therefore give simultaneous improvement for sufficiently small positive steps. Behavior uses ( U 1 , U 2 ) = ( A , H g ) (U_1,U_2)=(A,H_g) ( U 1 , U 2 ) = ( A , H g ) and target uses ( U 1 , U 2 ) = ( J / τ , P ) (U_1,U_2)=(J/\tau,P) ( U 1 , U 2 ) = ( J / τ , P ) . No derivative passes through the parent Fisher matrices, norms, or fusion coefficients.
These are conditional compatibility guarantees, not unconditional common ascent. For example, F = 1 F=1 F = 1 , g 1 = 2 g_1=2 g 1 = 2 , and g 2 = − 1 g_2=-1 g 2 = − 1 give ( α 1 , α 2 ) = ( 10 / 17 , 14 / 17 ) (\alpha_1,\alpha_2)=(10/17,14/17) ( α 1 , α 2 ) = ( 10/17 , 14/17 ) and d = 6 / 17 d=6/17 d = 6/17 , so the second channel decreases to first order. A larger natural norm describes gradient strength in this geometry, not statistical confidence.
Numerical updates and endpoint feasibility control
Fisher damping and its selection. The ridge λ F \lambda_F λ F is a fixed positive hyperparameter that stabilizes the linear systems and makes the common Fisher matrices invertible. We use
λ F = 0.0067202476327024195 (230) \lambda_F = 0.0067202476327024195 \tag*{(230)} λ F = 0.0067202476327024195 (230)
for both actors, all initialization sources, and all generations. This value was inherited from an earlier cold-start study of the precursor implementation that jointly varied the critic learning rate, emphatic learning rate, and Fisher damping. The study evaluated 12 fixed Sobol configurations, mapping damping logarithmically over [ 10 − 4 , 10 − 2 ] [10^{-4},10^{-2}] [ 1 0 − 4 , 1 0 − 2 ] , with three training seeds per configuration. Each run used 60 generations of 25,000 transitions. Configurations were ranked by the mean stochastic target-policy cost across the three seeds on common tuning tapes, with ties resolved by configuration index. The selected damping was then fixed and inherited by the present implementation. The current seven hyperparameter configurations vary the critic learning rate, emphatic learning rate, and heuristic multiplier κ \kappa κ ; they do not retune λ F \lambda_F λ F . It is not adapted from within-run returns or gradient norms.
Numerical natural-gradient directions. Two conjugate-gradient (CG) solves estimate the channel natural norms. Let α ~ 1 , α ~ 2 \tilde{\alpha}_1,\tilde{\alpha}_2 α ~ 1 , α ~ 2 be the coefficients obtained by applying (221) to those estimated norms. After freezing these coefficients, a separate CG solve uses the right-hand side α ~ 1 g 1 + α ~ 2 g 2 \tilde{\alpha}_1g_1+\tilde{\alpha}_2g_2 α ~ 1 g 1 + α ~ 2 g 2 . The final direction is not formed by adding the two approximate channel solutions. Fisher-vector products avoid constructing a dense parameter-space matrix. The J14 r3 implementation uses a base limit of 256 iterations and Euclidean relative residual tolerance 10 − 4 10^{-4} 1 0 − 4 . The retained J12 implementation uses the same base limit and tolerance 5 × 10 − 5 5\times10^{-5} 5 × 1 0 − 5 . Numerical recovery retains the same frozen equations and data rather than resampling evidence.
Theorem 8 (Residual bound for a numerical fused solve). Let F ≻ 0 F\succ0 F ≻ 0 and let α ~ 1 , α ~ 2 \tilde{\alpha}_1,\tilde{\alpha}_2 α ~ 1 , α ~ 2 be the fixed coefficients actually passed to the fused CG solve, possibly computed from approximate channel norms. Define its exact solution d α ~ = F − 1 ( α ~ 1 g 1 + α ~ 2 g 2 ) d_{\tilde{\alpha}}=F^{-1}(\tilde{\alpha}_1g_1+\tilde{\alpha}_2g_2) d α ~ = F − 1 ( α ~ 1 g 1 + α ~ 2 g 2 ) . For a computed direction d ~ \tilde{d} d ~ , define the linear-system residual r C G r_{\mathrm{CG}} r CG and its dual norm ε C G \varepsilon_{\mathrm{CG}} ε CG by
r C G = ( α ~ 1 g 1 + α ~ 2 g 2 ) − F d ~ , ε C G = ∥ r C G ∥ F − 1 . (231) r_{\mathrm{CG}}=(\tilde{\alpha}_1g_1+\tilde{\alpha}_2g_2)-F\tilde{d},\qquad\varepsilon_{\mathrm{CG}}=\lVert r_{\mathrm{CG}}\rVert_{F^{-1}}. \tag*{(231)} r CG = ( α ~ 1 g 1 + α ~ 2 g 2 ) − F d ~ , ε CG = ∥ r CG ∥ F − 1 . (231) With the exact channel norms n i = ∥ g i ∥ F − 1 n_i=\lVert g_i\rVert_{F^{-1}} n i = ∥ g i ∥ F − 1 ,
∥ d ~ − d α ~ ∥ F = ε C G , ∣ g i T ( d ~ − d α ~ ) ∣ ≤ n i ε C G . (232) \lVert\tilde{d}-d_{\tilde{\alpha}}\rVert_F=\varepsilon_{\mathrm{CG}},\qquad\left|g_i^\mathsf{T}(\tilde{d}-d_{\tilde{\alpha}})\right|\le n_i\varepsilon_{\mathrm{CG}}. \tag*{(232)} ∥ d ~ − d α ~ ∥ F = ε CG , g i T ( d ~ − d α ~ ) ≤ n i ε CG . (232) If d d d and α i \alpha_i α i denote the ideal exact-norm fusion from (221) , then also
∥ d ~ − d ∥ F ≤ ε C G + ∑ i = 1 2 ∣ α ~ i − α i ∣ n i . (233) \lVert\tilde{d}-d\rVert_F\le\varepsilon_{\mathrm{CG}}+\sum_{i=1}^{2}|\tilde{\alpha}_i-\alpha_i|n_i. \tag*{(233)} ∥ d ~ − d ∥ F ≤ ε CG + i = 1 ∑ 2 ∣ α ~ i − α i ∣ n i . (233) Proof. The residual identity gives d α ~ − d ~ = F − 1 r C G d_{\tilde{\alpha}}-\tilde{d}=F^{-1}r_{\mathrm{CG}} d α ~ − d ~ = F − 1 r CG . Taking the F F F -norm proves the equality, and dual Cauchy–Schwarz proves the slope bound. Finally, d α ~ − d = ∑ i ( α ~ i − α i ) F − 1 g i d_{\tilde{\alpha}}-d=\sum_i(\tilde{\alpha}_i-\alpha_i)F^{-1}g_i d α ~ − d = ∑ i ( α ~ i − α i ) F − 1 g i ; the triangle inequality and ∥ F − 1 g i ∥ F = n i \lVert F^{-1}g_i\rVert_F=n_i ∥ F − 1 g i ∥ F = n i prove the last bound. □ \square □
Here ε C G \varepsilon_{\mathrm{CG}} ε CG is an analytical residual norm; the runtime stopping tolerance uses a Euclidean relative residual and does not directly equal this quantity. The coefficient-error term must also be controlled to transfer the exact-norm guarantees of Theorem 7 to a numerical direction. Endpoint control below evaluates the numerical candidate directly.
Finite updates and the two channel budgets. Recall that the Fuse directions in Equation (221) use exact solves at the frozen parents. Let t μ , t π ∈ [ 0 , 1 ] t_\mu,t_\pi\in[0,1] t μ , t π ∈ [ 0 , 1 ] denote behavior and target step fractions. Because all channels are maximization objectives, the ideal updates use the ascent sign:
θ g + 1 μ = θ g μ + t μ Fuse F μ ( g A , g H ) , θ g + 1 π = θ g π + t π Fuse F π ( g J , g P ) . (234) \begin{aligned}
\theta_{g+1}^{\mu} &= \theta_g^{\mu} + t_\mu\operatorname{Fuse}_{F_\mu}(g_A,g_H),\\
\theta_{g+1}^{\pi} &= \theta_g^{\pi} + t_\pi\operatorname{Fuse}_{F_\pi}(g_J,g_P). \tag*{(234)}
\end{aligned} θ g + 1 μ θ g + 1 π = θ g μ + t μ Fuse F μ ( g A , g H ) , = θ g π + t π Fuse F π ( g J , g P ) . (234)
For either actor, let θ ⋆ \theta_\star θ ⋆ be its parent parameters and d ~ \tilde{d} d ~ its computed direction. Endpoint feasibility control (EFC) chooses a step fraction t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] . Let FP32 denote rounding to the stored single-precision floating-point parameter representation. EFC evaluates the actual candidate θ ( t ) = FP32 ( θ ⋆ + t d ~ ) \theta(t)=\operatorname{FP32}(\theta_\star+t\tilde{d}) θ ( t ) = FP32 ( θ ⋆ + t d ~ ) . Let q q q be the parent policy, q t q_t q t the policy represented by θ ( t ) \theta(t) θ ( t ) , and ν \nu ν one of the actor’s channel probability measures. Define the averaged Pearson discrepancy D 2 , ν D_{2,\nu} D 2 , ν by
D 2 , ν ( q t ∥ q ) = E s ∼ ν ∑ a ( q t ( a ∣ s ) − q ( a ∣ s ) ) 2 q ( a ∣ s ) = E s ∼ ν ∑ a q t ( a ∣ s ) 2 q ( a ∣ s ) − 1. (235) \begin{aligned}
D_{2,\nu}(q_t\mathbin{\|}q)
&= \mathbb{E}_{s\sim\nu}\sum_a\frac{(q_t(a\mid s)-q(a\mid s))^2}{q(a\mid s)}\\
&= \mathbb{E}_{s\sim\nu}\sum_a\frac{q_t(a\mid s)^2}{q(a\mid s)}-1. \tag*{(235)}
\end{aligned} D 2 , ν ( q t ∥ q ) = E s ∼ ν a ∑ q ( a ∣ s ) ( q t ( a ∣ s ) − q ( a ∣ s ) ) 2 = E s ∼ ν a ∑ q ( a ∣ s ) q t ( a ∣ s ) 2 − 1. (235)
Let ε E F C = 0.01 \varepsilon_{\mathrm{EFC}}=0.01 ε EFC = 0.01 be the per-channel endpoint budget. The numerical constraint D 2 , ν ( q t ∥ q ) ≤ ε E F C D_{2,\nu}(q_t\mathbin{\|}q)\le\varepsilon_{\mathrm{EFC}} D 2 , ν ( q t ∥ q ) ≤ ε EFC is checked separately for ν = D g , ξ ~ g \nu=D_g,\tilde{\xi}_g ν = D g , ξ ~ g for behavior and ν = ν g J , ν g P \nu=\nu_g^J,\nu_g^P ν = ν g J , ν g P for target. Checking both measures separately prevents one channel’s small movement from concealing a large movement under the other.
EFC first tests t = 1 t=1 t = 1 . If infeasible, it initializes the feasible lower endpoint at the parent t = 0 t=0 t = 0 , the upper endpoint at 1 1 1 , and performs 24 midpoint probes. A feasible probe replaces the lower endpoint and its cached parameter vector; an infeasible probe replaces the upper endpoint. The algorithm commits that exact cached feasible vector, without reconstructing it from the returned scalar. Thus the committed parameters are exactly those that passed the implemented numerical checks. This search does not require global monotonicity of feasibility and need not return the largest feasible step. The mathematical bounds below assume the stated discrepancy inequality for that endpoint; finite-precision checks alone are not interval-arithmetic certificates of the exact inequality.
Corollary 2 (Finite policy-change bounds under each channel measure). Let ν \nu ν be a probability measure on observations, let q q q be positive on the finite action set, and let q t q_t q t be a candidate policy. Recall that t t t labels the endpoint step fraction. Define, at each observation, the total variation and forward Kullback–Leibler divergence by TV ( q t , q ) = 1 2 ∑ a ∣ q t ( a ) − q ( a ) ∣ \operatorname{TV}(q_t,q)=\frac{1}{2}\sum_a\lvert q_t(a)-q(a)\rvert TV ( q t , q ) = 2 1 ∑ a ∣ q t ( a ) − q ( a )∣ and KL ( q t ∥ q ) = ∑ a q t ( a ) log ( q t ( a ) / q ( a ) ) \operatorname{KL}(q_t\mathbin{\|}q)=\sum_a q_t(a)\log(q_t(a)/q(a)) KL ( q t ∥ q ) = ∑ a q t ( a ) log ( q t ( a ) / q ( a )) , using natural logarithms and the convention 0 log 0 = 0 0\log0=0 0 log 0 = 0 . If D 2 , ν ( q t ∥ q ) ≤ ε E F C D_{2,\nu}(q_t\mathbin{\|}q)\le\varepsilon_{\mathrm{EFC}} D 2 , ν ( q t ∥ q ) ≤ ε EFC , then
E ν TV ( q t , q ) ≤ ε E F C 2 , E ν KL ( q t ∥ q ) ≤ log ( 1 + ε E F C ) . (236) \mathbb{E}_{\nu}\operatorname{TV}(q_t,q)\le\frac{\sqrt{\varepsilon_{\mathrm{EFC}}}}{2},\qquad
\mathbb{E}_{\nu}\operatorname{KL}(q_t\mathbin{\|}q)\le\log(1+\varepsilon_{\mathrm{EFC}}). \tag*{(236)} E ν TV ( q t , q ) ≤ 2 ε EFC , E ν KL ( q t ∥ q ) ≤ log ( 1 + ε EFC ) . (236)
For any frozen real action payoff f ( s , a ) f(s,a) f ( s , a ) with finite integrated conditional variance E S ∼ ν Var a ∼ q ( ⋅ ∣ S ) f ( S , a ) < ∞ \mathbb{E}_{S\sim\nu}\operatorname{Var}_{a\sim q(\cdot\mid S)}f(S,a)<\infty E S ∼ ν Var a ∼ q ( ⋅ ∣ S ) f ( S , a ) < ∞ ,
∣ E ν ∑ a ( q t ( a ∣ S ) − q ( a ∣ S ) ) f ( S , a ) ∣ ≤ ε E F C [ E ν Var a ∼ q ( ⋅ ∣ S ) f ( S , a ) ] 1 / 2 . (237) \left\lvert\mathbb{E}_{\nu}\sum_a(q_t(a\mid S)-q(a\mid S))f(S,a)\right\rvert
\le\sqrt{\varepsilon_{\mathrm{EFC}}\left[\mathbb{E}_{\nu}\operatorname{Var}_{a\sim q(\cdot\mid S)}f(S,a)\right]^{1/2}}. \tag*{(237)} E ν a ∑ ( q t ( a ∣ S ) − q ( a ∣ S )) f ( S , a ) ≤ ε EFC [ E ν Var a ∼ q ( ⋅ ∣ S ) f ( S , a ) ] 1/2 . (237)
Proof. Set f ˉ ( s ) = ∑ a q ( a ∣ s ) f ( s , a ) \bar{f}(s)=\sum_a q(a\mid s)f(s,a) f ˉ ( s ) = ∑ a q ( a ∣ s ) f ( s , a ) . Probability normalization allows replacing f f f by f − f ˉ f-\bar{f} f − f ˉ in the policy difference. Applying Cauchy–Schwarz under the joint measure ν ( d s ) q ( a ∣ s ) \nu(ds)q(a\mid s) ν ( d s ) q ( a ∣ s ) gives the following bound, where E ν , q \mathbb{E}_{\nu,q} E ν , q denotes expectation under that measure:
∣ E ν , q [ ( q t q − 1 ) ( f − f ˉ ) ] ∣ ≤ D 2 , ν ( q t ∥ q ) E ν Var q ( f ) . \left\lvert\mathbb{E}_{\nu,q}\left[\left(\frac{q_t}{q}-1\right)(f-\bar{f})\right]\right\rvert
\le\sqrt{D_{2,\nu}(q_t\mathbin{\|}q)}\sqrt{\mathbb{E}_{\nu}\operatorname{Var}_{q}(f)}. E ν , q [ ( q q t − 1 ) ( f − f ˉ ) ] ≤ D 2 , ν ( q t ∥ q ) E ν Var q ( f ) .
Similarly, TV ( q t , q ) = 1 2 E q ∣ q t / q − 1 ∣ \operatorname{TV}(q_t,q)=\frac{1}{2}\mathbb{E}_{q}\lvert q_t/q-1\rvert TV ( q t , q ) = 2 1 E q ∣ q t / q − 1 ∣ gives the total-variation bound. Recall the pointwise Pearson divergence χ 2 ( q t ∥ q ) = ∑ a ( q t ( a ) − q ( a ) ) 2 / q ( a ) \chi^2(q_t\Vert q)=\sum_a(q_t(a)-q(a))^2/q(a) χ 2 ( q t ∥ q ) = ∑ a ( q t ( a ) − q ( a ) ) 2 / q ( a ) , whose ν \nu ν -average is D 2 , ν ( q t ∥ q ) D_{2,\nu}(q_t\Vert q) D 2 , ν ( q t ∥ q ) . At each state, Jensen’s inequality under q t q_t q t yields
KL ( q t ∥ q ) = E q t log ( q t / q ) ≤ log E q t ( q t / q ) = log ( 1 + χ 2 ( q t ∥ q ) ) . \operatorname{KL}(q_t\Vert q)=\mathbb{E}_{q_t}\log(q_t/q)\leq\log\mathbb{E}_{q_t}(q_t/q)=\log\left(1+\chi^2(q_t\Vert q)\right). KL ( q t ∥ q ) = E q t log ( q t / q ) ≤ log E q t ( q t / q ) = log ( 1 + χ 2 ( q t ∥ q ) ) .
Averaging the displayed inequality over ν \nu ν and applying Jensen again proves the KL bound. □
Under the corollary’s discrepancy premise with the implemented budget, the average total-variation change is at most 0.05 under each channel measure. These constraints are empirical averages, not uniform bounds at every state. EFC controls distribution movement and does not check objective improvement or the curvature-dependent step conditions in Theorem 7 . The fusion result characterizes the ideal proposed direction; the residual theorem describes numerical direction error, and the endpoint corollary bounds finite policy change whenever its discrepancy premise holds.
Experiments Protocol
We compare HBPO with nine native methods: PPO (Schulman et al. [34 ] ), A2C (Mnih et al. [27 ] ), Discrete SAC (Haarnoja et al. [15 ] ; Christodoulou [6 ] ), ACE (Imani et al. [16 ] ), PPO-81 (Schulman et al. [34 ] ), TRPO (Schulman et al. [32 ] ), GRPO (Shao et al. [35 ] ), TPO (Kaddour [18 ] ), and Guided Policy Search (GPS) (Levine & Koltun [22 ] ; Levine et al. [23 ] ). Each method provides a warm initialization; Cold supplies a tenth HBPO initialization.
Native baseline selection
Each method is tuned independently in DSIS and DSIS-RB using twelve joint configurations from an eight-dimensional scrambled Sobol sequence (seed 1413164337). Table 4 gives the search ranges. Learning rates use logarithmic sampling on [ 10 − 5 , 10 − 3 ] [10^{-5},10^{-3}] [ 1 0 − 5 , 1 0 − 3 ] unless specified otherwise; clipping and GAE intervals use linear sampling, and discrete choices occupy equal intervals.
Table 4. Native search spaces. The DSIS-RB collection multiplier applies only to GRPO and TPO.
Each candidate trains for 500,000 real interactions. The eight non-GPS methods use tuning seed 0; GPS uses seeds 5, 6, 7. Tuning and formal training use separate random streams even when seed labels coincide. Candidates are evaluated with sampled actions on 100 trajectories of 1,000 periods per risk level, using the cost measure defined below. Within each method and tuning seed, candidates share evaluation tapes and action-randomness streams. We select the lowest validation cost, averaged over the three training seeds for GPS, and break ties by trial identifier. Table 5 and Table 6 give the selected parameters, rounded to six significant figures. These settings remain fixed for all formal seeds and their paired native/HBPO comparisons.
Table 5. Selected native configurations for DSIS.
Table 6. Selected native configurations for DSIS-RB.
HBPO selection
In each environment, we evaluate the seven configurations in Table 7 on all ten initializations (nine warm sources and Cold), using tuning seeds 5, 6, 7. Candidates share the initial learner state, first data batch, and calibration scale within each initialization and seed. Each candidate runs for 15 generations of 25,000 real interactions, including the first batch, and is validated with sampled target actions on 100 trajectories of 1,000 periods per risk level. The grid varies value learning rate, emphatic learning rate, and the heuristic scale multiplier κ \kappa κ , with other settings fixed. The calibration of κ \kappa κ is specified in Appendix § 11 .4.6.
Table 7. The complete HBPO candidate set. H5 is used in the final experiments.
For each tuning seed, split initializations into initial-cost groups C 0 ≤ 1010 C_0 \le1010 C 0 ≤ 1010 and C 0 > 1010 C_0 > 1010 C 0 > 1010 . For each candidate, compute percentage improvement 100 ( 1 − C 15 / C 0 ) 100(1-C_{15}/C_0) 100 ( 1 − C 15 / C 0 ) , average within each group, and then average these group means across seeds with nonempty groups. Denote the two scores by P P P and R R R , respectively. Rank by each score, breaking ties by the other score and then candidate ID. Select the highest-P P P candidate in the intersection of the two top-three lists; if the intersection is empty, select the highest-R R R candidate among the P P P top three. If one group is empty for every seed, use the other ranking. This rule selects H5 in both environments: value learning rate 5.66943 × 10 − 4 5.66943 \times10^{-4} 5.66943 × 1 0 − 4 , emphatic learning rate 1.56958 × 10 − 5 1.56958 \times10^{-5} 1.56958 × 1 0 − 5 , and κ = 1 / 32 \kappa= 1/32 κ = 1/32 . One configuration is shared across all initializations and formal seeds within each environment; ablations use it without retuning.
Evaluation protocol
Budgets and seeds. Each formal run uses 2M real interactions and one of five training seeds, 0, . . . , 4. Warm comparisons branch from the same native 0.5M checkpoint into native continuation and HBPO, each receiving another 1.5M interactions. Cold receives 2M without a native prefix. Tuning and formal-evaluation data are separate, and formal results do not influence hyperparameter or checkpoint selection. Simulation and evaluation are accounted for separately from real training interactions.
Information access. All methods have the same public-information permissions and visible observations. DSIS’ stochastic law is public; DSIS-RB’s true stochastic kernels, episode risks, and future random tapes are hidden. In DSIS-RB, HBPO and GPS fit the same model class to their own budgeted observations. Modeling assumptions are given in Appendix § 11 .1.4.
Test cost and reporting. We freeze the final policy and sample its actions; HBPO always reports the target policy π \pi π . Each policy is tested on 500 independent trajectories of 1,000 periods, including all periods from the initial state. DSIS reports mean undiscounted raw cost per period. DSIS-RB uses 500 trajectories at each ρ ∈ { 0 , .1 , .2 , .3 , .4 , .5 } \rho\in\{0,.1,.2,.3,.4,.5\} ρ ∈ { 0 , .1 , .2 , .3 , .4 , .5 } and reports C s R B = ∑ k = 0 5 ω k C s ( 0.1 k ) C^{\mathrm{RB}}_s = \sum_{k=0}^{5}\omega_k C_s(0.1k) C s RB = ∑ k = 0 5 ω k C s ( 0.1 k ) , where ω = ( .1 , .2 , .2 , .2 , .2 , .1 ) \omega= (.1,.2,.2,.2,.2,.1) ω = ( .1 , .2 , .2 , .2 , .2 , .1 ) and C s ( ρ ) C_s(\rho) C s ( ρ ) is the mean cost for training seed s s s at risk ρ \rho ρ . This is normalized trapezoidal risk-AUC. We report the mean, median, and sample standard deviation across the five training seeds, using unrounded costs for comparisons. Ablations are paired with their corresponding Full run by source, seed, and evaluation bank, as detailed in Appendix § 14 .
Cost-collapse thresholds. Mean demand is two units per period. Expedited replacement costs 2 × 10 = 20 2 \times10 = 20 2 × 10 = 20 , and k k k periods of mean demand in backlog incur 2 k × 495 = 990 k 2k \times495 = 990k 2 k × 495 = 990 k per period. The reference cost 20 + 990 k 20 + 990k 20 + 990 k gives 1010 for one period and 3980 for four periods. We classify costs as Non-collapse (C ≤ 1010 C \le1010 C ≤ 1010 ), Collapse (1010 < C ≤ 3980 1010 < C \le3980 1010 < C ≤ 3980 ), or Severe Collapse (C > 3980 C > 3980 C > 3980 ). These are economic severity thresholds, not estimates of actual delivery delay. Endpoint recovery means a native 2M cost above 10 10 10^{10} 1 0 10 paired with an HBPO 2M cost at or below 10 10 10^{10} 1 0 10 .
Extended Results
Detailed Main Results
Table 8 and Table 9 expand Table 1 using the same 190 final-policy costs: 90 native endpoints, 90 warm HBPO endpoints, and 10 Cold endpoints. Each policy is evaluated after a total budget of 2M environment interactions; warm HBPO includes the 0.5M native prefix and reports the final target policy π \pi π . H denotes warm HBPO, and Cold has no native prefix or paired native endpoint. All summaries retain all five training seeds. Evaluation protocols and cost grades follow Appendix § 12 .§ 3 .
Table 1. Final policy costs at a matched budget of 2M environment interactions across five training seeds. H denotes warm-start HBPO; Cold denotes HBPO without a native training prefix. Seed costs are black for Non-collapse, yellow for Collapse, and red for Severe Collapse. Std is the sample standard deviation across seeds; bold marks the lower paired mean or median, computed from unrounded values. Underlining marks HBPO endpoints with costs ≤ 10 10 \le10^{10} ≤ 1 0 10 whose seed-matched native endpoints exceed 10 10 10^{10} 1 0 10 . k/M denote 10 3 / 10 6 10^3/10^6 1 0 3 /1 0 6 .
Table 8: Detailed final-policy costs on DSIS (mean cost per period; lower is better), expanding Table 1. Panel (a) retains five decimal places for every seed; panel (b) reports exact means and medians, with sample standard deviations displayed to nine decimal places. Formatting follows the conventions in Appendix D.1.
Table 8.
Table 9: Detailed final-policy costs on DSIS-RB (normalized risk-AUC cost; lower is better), expanding Table 1. Panel (a) retains six decimal places for every seed; panel (b) reports exact means and medians, with sample standard deviations displayed to nine decimal places. Formatting follows the conventions in Appendix D.1.
Table 9.
Precision and table conventions. Seed costs retain their full effective decimal precision: five places for DSIS and six for DSIS-RB, without k/M abbreviations. Only binary floating-point representation tails are removed. Means and medians are exact at this precision; sample standard deviations use the n − 1 = 4 n - 1 = 4 n − 1 = 4 denominator and are displayed to nine decimal places. Black, ochre, and red seed entries indicate c ≤ 10 10 c \le10^{10} c ≤ 1 0 10 , 10 10 < c ≤ 3980 10^{10} < c \le3980 1 0 10 < c ≤ 3980 , and c > 3980 c > 3980 c > 3980 , respectively. Bold marks the lower paired mean or median; underlining marks HBPO costs ≤ 10 10 \le10^{10} ≤ 1 0 10 whose seed-matched native endpoints exceed 10 10 10^{10} 1 0 10 . Comparisons use the full-precision values, not the abbreviated main-table entries.
DSIS. HBPO lowers mean cost in all nine warm-source comparisons, median cost in eight (TRPO is the exception), and sample standard deviation in all nine. It lowers cost in 40 of 45 seed-matched comparisons. All 12 native endpoints above 10 10 10^{10} 1 0 10 have lower-cost HBPO counterparts, and 10 cross to costs ≤ 10 10 \le10^{10} ≤ 1 0 10 . All 33 native endpoints at or below 10 10 10^{10} 1 0 10 remain within this range under HBPO, with 28 achieving lower costs. The two HBPO endpoints still above 10 10 10^{10} 1 0 10 are A2C seed 2 (6638.13434) and ACE seed 0 (1207.57231).
DSIS-RB. HBPO lowers mean cost in eight of nine warm-source comparisons, median cost in seven (PPO and A2C are the exceptions), and sample standard deviation in all nine. It lowers cost in 41 of 45 seed-matched comparisons. Among the 24 native endpoints above 10 10 10^{10} 1 0 10 , 23 have lower-cost HBPO counterparts and 19 cross to costs ≤ 10 10 \le10^{10} ≤ 1 0 10 . All 21 native endpoints at or below 10 10 10^{10} 1 0 10 remain within this range under HBPO, with 18 achieving lower costs. Discrete SAC is the mean-cost exception: 2479.0853802 increases to 2514.9923076 (1.4484%), although its median and standard deviation decrease. Its seed 0 worsens from 1183 to 5359.05708, moving from Collapse to Severe Collapse. The five HBPO endpoints still above 10 10 10^{10} 1 0 10 are PPO seed 2, A2C seed 2, Discrete SAC seeds 0 and 1, and ACE seed 0.
Cold starts and scope of the comparisons. All five Cold endpoints are at or below 10 10 10^{10} 1 0 10 in each environment. DSIS has mean 45.103804 and range [42.67906, 47.51533]; DSIS-RB has mean 72.1150812 and range [69.675982, 74.916383]. Warm comparisons pair native and HBPO endpoints by environment, source, training seed, and interaction budget; they do not assume identical evaluation random banks across campaigns. Crossing the threshold relative to a native 2M endpoint is distinct from recovery relative to the 0.5M initialization. These descriptive results characterize the evaluated seeds and do not establish statistical significance or a task-independent stability guarantee.
Additional Transmission Analyses
Figures 5–8 complete the transmission analysis in Section 5.3. They show the remaining 16 source–environment combinations: A2C, Discrete SAC, ACE, PPO-81, TRPO, GRPO, TPO, and GPS on DSIS and DSIS-RB. Together with the four Cold and PPO combinations in Figure 2, these cover all ten initializations in both environments. Each combination contains five training seeds; HBPO-SAC denotes the Discrete SAC initialization.
Reading the panels. Within each combination, the left panel shows standard evaluation costs for μ \mu μ (blue) and π \pi π (ochre), together with the green dashed q H q_H q H reference. The latter averages H H H -weighted immediate costs over the original nine-step rollouts and is not a comparable long-horizon policy evaluation. The middle panels show relative step ϱ \varrho ϱ and direction cosine c c c , with purple for q H → μ q_H \to\mu q H → μ and rose for μ → π \mu\to\pi μ → π . The right panels show adoption and reflection: solid teal curves are net responses G ± G_{\pm} G ± , gray dashed curves are opportunities B ± B_{\pm} B ± , and pale green/pink backgrounds are mean favorable and adverse contributions above/below zero. Positive G + G_{+} G + and G − G_{-} G − denote favorable local responses in the adoption and reflection regions, respectively.
Aggregation and timing. Curves are arithmetic means across seeds; cost panels also retain individual seed traces, while mechanism bands show the 25th–75th percentiles. The signed contribution backgrounds are not uncertainty bands. Statistics are computed before axis transformations, without smoothing or subsampling. Costs use logarithmic axes; relative steps and selection responses use inverse-hyperbolic-sine axes with linear scales 10 − 2 10^{-2} 1 0 − 2 and 10 − 6 10^{-6} 1 0 − 6 , respectively. Cost-axis limits are shared within each environment; mechanism limits are shared across all combinations. These limits include the extrema from all 20 combinations. Costs retain checkpoint time t t t ; mechanism panels pair the q H → μ q_H \to\mu q H → μ update at g g g with the μ → π \mu\to\pi μ → π update and selection responses at g + 1 g + 1 g + 1 , following π g → q H , g → μ g + 1 → π g + 2 \pi_g \to q_{H,g} \to\mu_{g+1} \to\pi_{g+2} π g → q H , g → μ g + 1 → π g + 2 . All panels here are warm starts and display generation plus 20. These are descriptive diagnostics of the recorded trajectories, rather than additional intervention experiments.
Figure 5. Additional transmission analysis for A2C (left) and Discrete SAC (right). Top: DSIS; bottom: DSIS-RB. Conventions follow Appendix § 13 .§ 2 .
Figure 6. Additional transmission analysis for ACE (left) and PPO-81 (right). Top: DSIS; bottom: DSIS-RB. Conventions follow Appendix § 13 .§ 2 .
Figure 7. Additional transmission analysis for TRPO (left) and GRPO (right). Top: DSIS; bottom: DSIS-RB. Conventions follow Appendix D.2.
Figure 8. Additional transmission analysis for TPO (left) and GPS (right). Top: DSIS; bottom: DSIS-RB. Conventions follow Appendix D.2.
Emphatic Weighting and Final Cost
Figure 9 shows that emphatic weighting does not provide a clear, consistent improvement in final target-policy cost after the full 2M-interaction training budget. We compare Full HBPO with No-E, J-only E, and P-only E across Cold, PPO, Discrete SAC, and TRPO initializations in both environments, retaining all five training seeds. The interventions change the emphatic weighting branches while retaining the J and P policy objectives (Appendix § 14 .§ 4 ). Against No-E, Full achieves a lower final cost in only 19 of 40 matched runs and a lower mean in four of the eight environment–initialization settings. The DSIS Cold mean favors Full, but this difference is driven mainly by one seed; J-only E and P-only E likewise show mixed endpoint effects. Thus, these results do not support a general long-run final-cost advantage from emphatic weighting in the evaluated settings. This is a descriptive comparison, not a formal significance or equivalence test, and it does not determine whether weighting affects earlier learning.
Figure 9. Emphatic-weight ablations at the final target-policy endpoint. No-E disables both emphatic branches; J-only E and P-only E retain only the indicated weighting branch. Each bar gives the percentage cost change relative to matched Full HBPO; positive values favor Full. Each environment–initialization block contains seeds 0–4 from left to right; darker bars mark ablation costs above 10 10 10^{10} 1 0 10 . All panels share the group order shown below and the H/P ablation scale, linear up to 100% and logarithmic above. All emphatic-ablation bars lie in the linear region.
Ablation Details and Additional Diagnostics
This appendix specifies the nine implemented interventions and the additional diagnostics underlying the component and transmission analyses. The interventions measure changes in the complete learning process at a fixed setting; the observational diagnostics describe actual updates.
Matched ablation panel and reporting
The ablation panel comprises PPO, Discrete SAC, TRPO, and Cold initializations in each environment, with training seeds 0–4. Each intervention therefore has 40 matched runs. We evaluate two heuristic interventions, four coupling interventions, and three emphatic interventions, giving 360 intervention endpoints and 40 shared Full references. These are separate changes to Full, not a factorial combination of multiple intervention families. Every variant uses the selected H5 learning rates and κ = 1 / 32 \kappa= 1/32 κ = 1/32 without retuning. Warm runs use the same source checkpoint after 0.5M interactions and another 60 generations of 25,000 real transitions; Cold uses 80 generations. Thus the final comparison is at 2M real interactions. The source policy, initialization procedure, scientific seed, and initial real-data batch are matched to the corresponding Full run. Subsequent policies, collected data, fitted models, and simulation work can differ as a consequence of the intervention.
The final target policy is evaluated by sampled actions under the formal protocol of Appendix C: 500 episodes of 1,000 periods in DSIS, and the same number at each of the six risk levels in DSIS-RB, followed by normalized trapezoidal risk aggregation. These measurements do not feed back into training. The endpoint record contains all five seeds in every cell. It combines qualified historical DSIS runs for SAC, TRPO, and Cold with the subsequent DSIS PPO and DSIS-RB runs. Each variant is compared with the Full reference evaluated on its own matched bank. In particular, later Full reruns or reevaluations used for the main comparison and generation diagnostics do not replace these ablation denominators. This preserves within-comparison pairing when the historical and subsequent evaluation banks differ.
For an environment–initialization–seed coordinate i i i , the plotted change for variant v v v is
Δ v , i = 100 ( c v , i c F u l l , i − 1 ) . (238) \Delta_{v,i} = 100\left(\frac{c_{v,i}}{c_{\mathrm{Full},i}} - 1\right). \tag*{(238)} Δ v , i = 100 ( c Full , i c v , i − 1 ) . (238)
Positive values favor Full. An average paired percentage is the arithmetic mean of these ratios; it is distinct from the percentage change between two group means. Dark bars in the main ablation figures indicate that the variant ’s final cost exceeds 1010. Across-seed dispersion uses the sample standard deviation of the five training-seed costs, rather than the standard error over evaluation episodes. The ablation outcome evidence is an endpoint comparison; it does not establish learning-curve smoothness, faster convergence, or a training-curve area under the curve. Real interaction budgets are matched, while model simulation and numerical work are accounted for separately and need not be equal after a component is removed.
Heuristic-guidance interventions
No-H. This intervention deletes the entire finite heuristic objective, so that H ( θ ) ≡ 0 H(\theta) \equiv0 H ( θ ) ≡ 0 and g H = 0 g_H = 0 g H = 0 , and removes the associated H-occupancy construction and confidence calibration. The H state channel in the behavior Fisher and endpoint constraints is replaced by the complete real-data panel D g D_g D g with unit weights. Consequently, the two behavior channels use the same data measure and their averaged Fisher reduces to the A-channel Fisher. The guidance-derived P interest is set to one at the root and success states. The A, J, and P objectives, complete residual corrections, and remaining emphatic learning are retained; in particular, setting P interest to one does not disable the P emphatic learner. The initial utility-scale calibration for τ \tau τ can still require model queries. No-H therefore tests the total contribution of the guidance module, including its state weighting and geometry, rather than only setting a single gradient vector to zero.
Random-H. At each queried public state, we construct a random target around this variant’s own frozen parent behavior policy. Let q H , g s h a d o q^{\mathrm{shado}}_{H,g} q H , g shado be the simulation-derived controller at that same state under the same parent policies and model. Draw a Gaussian action vector z g ( s , ⋅ ) z_g(s,\cdot) z g ( s , ⋅ ) and choose β g ( s ) ≥ 0 \beta_g(s) \ge0 β g ( s ) ≥ 0 so that
q R , g ( a ∣ s ) = μ g ( a ∣ s ) exp ( β g ( s ) z g ( s , a ) ) ∑ b μ g ( b ∣ s ) exp ( β g ( s ) z g ( s , b ) ) (239) q_{R,g}(a \mid s) = \frac{\mu_g(a \mid s)\exp(\beta_g(s)z_g(s,a))}{\sum_b \mu_g(b \mid s)\exp(\beta_g(s)z_g(s,b))} \tag*{(239)} q R , g ( a ∣ s ) = ∑ b μ g ( b ∣ s ) exp ( β g ( s ) z g ( s , b )) μ g ( a ∣ s ) exp ( β g ( s ) z g ( s , a )) (239)
d F R ( μ g , q R , g ) = d F R ( μ g , q H , g s h a d o w ) . (240) d_{\mathrm{FR}}(\mu_g,q_{R,g}) = d_{\mathrm{FR}}(\mu_g,q^{\mathrm{shadow}}_{H,g}). \tag*{(240)} d FR ( μ g , q R , g ) = d FR ( μ g , q H , g shadow ) . (240)
For zero radius, the target is μ g \mu_g μ g . For a positive radius, directions are rejected only if their limiting maximum-action vertex cannot reach that radius; returns are not used to select a random direction. A bracketed scalar solve finds the radius-matching β g ( s ) \beta_g(s) β g ( s ) . The random field is addressed by environment, initialization, training seed, generation, and public state, so repeated visits to the same state within a generation use the same direction. Hidden risk does not enter the address.
The random controller replaces the informative controller in both occupancy rollouts and the paired calibration rollouts. The original confidence-to-interest construction is retained. Because the shadow controller still sets the local radius, Random-H retains model and return information about the magnitude of guidance. The matching is local to this arm’s parent policy and visited states; it does not force the Random-H and Full trajectories to visit the same states or to have identical future exploration. This comparison tests the value of the informative guidance direction within the resulting learning process.
Behavior–target coupling interventions
No-A. We remove the complete A channel from behavior learning, including both the dense base term and sampled residual correction, and set its finite objective to zero. The remaining H channel is passed through the same fusion rule. The original behavior Fisher, both state-channel endpoint constraints, target update, and critic construction are retained. The removed signal is target-continuation advantage consumed by the behavior actor.
No-P. We set the explicit P receiver gradient and finite P objective to zero and recompute the target fusion with the unchanged J channel. The original J and P state measures still define the target Fisher and endpoint constraints. Behavior continues to generate the real training data and can influence the target through data, critics, state weights, and geometry. Thus No-P removes the explicit transfer objective; it does not remove every effect of behavior on target learning.
KL-P. We replace P by forward-KL imitation of the parent behavior policy. Up to a constant independent of the candidate target parameters, the maximization objective is
K g ( θ ) = 1 ∣ D g ∣ ∑ s ∈ D g w g P ( s ) ∑ a μ g ( a ∣ s ) log π θ ( a ∣ s ) . (241) K_g(\theta) = \frac{1}{|D_g|}\sum_{s \in D_g} w_g^P(s)\sum_a \mu_g(a \mid s)\log\pi_\theta(a \mid s). \tag*{(241)} K g ( θ ) = ∣ D g ∣ 1 s ∈ D g ∑ w g P ( s ) a ∑ μ g ( a ∣ s ) log π θ ( a ∣ s ) . (241)
This replacement has no advantage filter. Write g K = ∇ K g g_K = \nabla K_g g K = ∇ K g , let g P g_P g P be the complete original P gradient including the residual correction, and retain the original target Fisher F π F_\pi F π . For positive norms, the receiver is
g ~ K = g P T F π − 1 g P g K T F π − 1 g K g K . (242) \widetilde{g}_K = \frac{\sqrt{g_P^{\mathsf T}F_\pi^{-1}g_P}}{\sqrt{g_K^{\mathsf T}F_\pi^{-1}g_K}}g_K. \tag*{(242)} g K = g K T F π − 1 g K g P T F π − 1 g P g K . (242)
The same frozen scale multiplies the finite KL objective. If the P reference norm is zero, the replacement is zero; a zero KL gradient with a nonzero reference norm is recorded as an unmatchable direction and also yields zero. The scaled receiver and J are fused using the usual target geometry and endpoint constraints. This matches the parent natural-gradient norm, not the finite policy displacement.
Near-on-policy. This arm maintains μ g = π g \mu_g = \pi_g μ g = π g at committed generation boundaries. It first computes the accepted AH behavior successor as a temporary guide μ ~ g + 1 \tilde{\mu}_{g+1} μ ~ g + 1 , then updates the target using J and norm-matched KL imitation of this fresh guide. For this matching, the reference P objective also uses μ ~ g + 1 − π g \tilde{\mu}_{g+1} - \pi_g μ ~ g + 1 − π g as its teacher gap. The real-data panel, critic evidence, and residual corrections remain those of the frozen parent generation; no new real batch is collected between the two actor updates. After the target update, an independent copy of π g + 1 \pi_{g+1} π g + 1 is committed as μ g + 1 \mu_{g+1} μ g + 1 . This intervention jointly changes the receiver objective, transmission timing, and persistent separation of the policies. It is therefore an alternative coupling architecture rather than a pure delay-only intervention, and it is not an implementation of GPS.
Emphatic-weight interventions
The three emphatic interventions retain the H, A, J, and P objectives but change which learned state-weight branches are consumed:
Table E.4.
A disabled branch uses a neutral mass network with zero output head and unit root mass, forces both its raw mass and consumed weight to one, and skips that branch’s learner update. These unit weights are applied before the residual-root sampling, heuristic construction, critic evidence, actor objectives, and Fisher readers consume them. Disabling P emphatic additionally sets its root and successor interest to one. J-only E and P-only E refer to weighting branches; both J and P policy objectives remain present in either case. The endpoint effects include all subsequent changes in sampling and optimization, so they do not independently identify a distribution correction mechanism.
Transmission diagnostics and their timing
The transmission analysis follows the actual committed policy sequence, π g → q H , g → μ g + 1 → π g + 2 \pi_g \to q_{H,g} \to\mu_{g+1} \to\pi_{g+2} π g → q H , g → μ g + 1 → π g + 2 . Both actors in an ordinary generation read frozen parents: the target update that produces π g + 1 \pi_{g+1} π g + 1 uses μ g \mu_g μ g , not the simultaneously produced μ g + 1 \mu_{g+1} μ g + 1 . We therefore pair the heuristic–behavior measurements from update g → g + 1 g \to g+1 g → g + 1 with the behavior–target measurements from update g + 1 → g + 2 g+1 \to g+2 g + 1 → g + 2 . The latter use the residual evidence, weights, and advantage of generation g + 1 g+1 g + 1 . Parent and child checkpoint identities are checked before joining the two updates; no link crosses runs or training seeds.
Evidence population. The complete diagnostic population contains both environments, all nine warm sources and Cold, and five seeds. Warm runs have 60 updates and Cold runs have 80, giving 6,200 generation packets, 6,300 checkpoints, and 6,100 complete two-generation links. Each generation retains the original 1,152 heuristic fitting occurrences, the 128 residual-root draws, all 81 actions, and all 32 residual tapes per draw. Offline contraction of these arrays introduces no further environment interactions or model rollouts.
Occurrence measures. The two links are measured on different panels. Write { s h H , ω h H } h = 1 1152 \{s_h^H,\omega_h^H\}_{h=1}^{1152} { s h H , ω h H } h = 1 1152 for the original heuristic fitting occurrences and their training weights. For residual draw k k k , let i k i_k i k be its sampled batch index and ϖ i k = ( 1 + w i k J + w i k P ) / ( 3 n ) \varpi_{i_k}=(1+w_{i_k}^J+w_{i_k}^P)/(3n) ϖ i k = ( 1 + w i k J + w i k P ) / ( 3 n ) its proposal probability. The retained importance factors are ι k J = w i k J / ( n ϖ i k ) \iota_k^J=w_{i_k}^J/(n\varpi_{i_k}) ι k J = w i k J / ( n ϖ i k ) and ι k P = w i k P / ( n ϖ i k ) \iota_k^P=w_{i_k}^P/(n\varpi_{i_k}) ι k P = w i k P / ( n ϖ i k ) . Define the diagnostic contractions
⟨ x , y ⟩ H = 1 1152 ∑ h ω h H ∑ a x h ( a ) y h ( a ) . (243) \langle x,y\rangle_H=\frac{1}{1152}\sum_h \omega_h^H\sum_a x_h(a)y_h(a). \tag*{(243)} ⟨ x , y ⟩ H = 1152 1 h ∑ ω h H a ∑ x h ( a ) y h ( a ) . (243)
⟨ x , y ⟩ J = 1 128 ∑ k ι k J ∑ a x k ( a ) y k ( a ) , ⟨ z ⟩ P = 1 128 ∑ k ι k P ∑ a z k ( a ) . (244) \begin{aligned}
\langle x,y\rangle_J&=\frac{1}{128}\sum_k \iota_k^J\sum_a x_k(a)y_k(a),&
\langle z\rangle_P&=\frac{1}{128}\sum_k \iota_k^P\sum_a z_k(a). \tag*{(244)}
\end{aligned} ⟨ x , y ⟩ J = 128 1 k ∑ ι k J a ∑ x k ( a ) y k ( a ) , ⟨ z ⟩ P = 128 1 k ∑ ι k P a ∑ z k ( a ) . (244)
The weights are those supplied to the corresponding training packet. In particular, the finite residual sample is not self-normalized by its realized sum of importance weights. These diagnostics are occurrence-based Euclidean contractions in action-probability space, not the parameter-space Fisher norm used by the actor optimizer. Repeated states and repeated root draws retain their original multiplicities.
Step size and direction. On the heuristic panel of generation g g g , set d H = q H , g − μ g d_H = q_{H,g} - \mu_g d H = q H , g − μ g , u H = μ g + 1 − μ g u_H = \mu_{g+1} - \mu_g u H = μ g + 1 − μ g . On the residual panel of generation g + 1 g + 1 g + 1 , set d J = μ g + 1 − π g + 1 d_J = \mu_{g+1} - \pi_{g+1} d J = μ g + 1 − π g + 1 , u J = π g + 2 − π g + 1 u_J = \pi_{g+2} - \pi_{g+1} u J = π g + 2 − π g + 1 . For either link L ∈ { H , J } L \in\{H, J\} L ∈ { H , J } , compute
ϱ L = ⟨ u L , u L ⟩ L ⟨ d L , d L ⟩ L , c L = ⟨ d L , u L ⟩ L ⟨ d L , d L ⟩ L ⟨ u L , u L ⟩ L . (245) \varrho_L = \sqrt{\frac{\langle u_L,u_L\rangle_L}{\langle d_L,d_L\rangle_L}}, \qquad c_L = \frac{\langle d_L,u_L\rangle_L}{\sqrt{\langle d_L,d_L\rangle_L\langle u_L,u_L\rangle_L}}. \tag*{(245)} ϱ L = ⟨ d L , d L ⟩ L ⟨ u L , u L ⟩ L , c L = ⟨ d L , d L ⟩ L ⟨ u L , u L ⟩ L ⟨ d L , u L ⟩ L . (245)
The original projection statistic is ⟨ d L , u L ⟩ L / ⟨ d L , d L ⟩ L = ϱ L c L \langle d_L,u_L\rangle_L/\langle d_L,d_L\rangle_L = \varrho_L c_L ⟨ d L , u L ⟩ L / ⟨ d L , d L ⟩ L = ϱ L c L , whenever both factors are defined. We show the factors separately to distinguish update magnitude from directional agreement. A zero guidance gap makes both displayed statistics undefined; a zero update with a nonzero gap gives ϱ L = 0 \varrho_L = 0 ϱ L = 0 and an undefined cosine. No small-gap threshold or zero-filling is introduced. Stable, scaled contractions preserve very small nonzero gaps. Larger relative steps and positive cosines describe geometric following, not necessarily better policy value.
Adoption and reflection. At each residual draw of generation g + 1 g + 1 g + 1 , reconstruct the full frozen sampled advantage from the dense base and every saved residual tape:
a ^ k ( a ) = a k 0 ( a ) + 1 32 ∑ b = 1 32 Δ a k b ( a ) . (246) \widehat{a}_k(a) = a_k^0(a) + \frac{1}{32}\sum_{b=1}^{32}\Delta a_{kb}(a). \tag*{(246)} a k ( a ) = a k 0 ( a ) + 32 1 b = 1 ∑ 32 Δ a k b ( a ) . (246)
Here Δ a k b \Delta a_{kb} Δ a k b is the stored action-centered residual sum over a geometrically stopped continuation; discounting is represented by that stopping law. No tape selection, refitting, clipping, or additional centering is applied during postprocessing. With d = d J d = d_J d = d J , u = u J u = u_J u = u J , define actionwise masks S + = 1 { d ^ a > 0 } S_+ = 1\{\widehat{d}a > 0\} S + = 1 { d a > 0 } and S − = 1 { d ^ a < 0 } S_- = 1\{\widehat{d}a < 0\} S − = 1 { d a < 0 } . Opportunities and responses are
B + = ⟨ [ d ^ a ] + ⟩ P , B − = ⟨ [ − d ^ a ] + ⟩ P , (247) B_+ = \langle[\widehat{d}a]_+\rangle_P,\qquad B_- = \langle[-\widehat{d}a]_+\rangle_P, \tag*{(247)} B + = ⟨[ d a ] + ⟩ P , B − = ⟨[ − d a ] + ⟩ P , (247)
G + = ⟨ u a S + ⟩ P , G − = ⟨ u a S − ⟩ P . (248) G_+ = \langle uaS_+\rangle_P,\qquad G_- = \langle uaS_-\rangle_P. \tag*{(248)} G + = ⟨ u a S + ⟩ P , G − = ⟨ u a S − ⟩ P . (248)
Positive G + G_+ G + denotes favorable target movement on advantageous behavioral deviations; positive G − G_- G − denotes favorable correction of adverse deviations. Negative responses are retained. In each region, the plotted favorable and adverse components are
F ± = ⟨ [ u a S ± ] + ⟩ P , L ± = ⟨ [ − u a S ± ] + ⟩ P , G ± = F ± − L ± . (249) F_\pm= \langle[uaS_\pm]_+\rangle_P,\qquad L_\pm= \langle[-uaS_\pm]_+\rangle_P,\qquad G_\pm= F_\pm- L_\pm. \tag*{(249)} F ± = ⟨[ u a S ± ] + ⟩ P , L ± = ⟨[ − u a S ± ] + ⟩ P , G ± = F ± − L ± . (249)
The shaded areas show F ± F_\pm F ± above zero and − L ± -L_\pm − L ± below zero. The neutral region d ^ a = 0 \widehat{d}a = 0 d a = 0 is retained in the audit, so its response together with G + + G − G_+ + G_- G + + G − recovers the total sampled target advantage change. When B ± = 0 B_\pm= 0 B ± = 0 , that opportunity is inactive; no response-to-opportunity ratio is imputed. These are signed sums under a frozen sampled advantage, without division by τ \tau τ . They are neither raw-cost improvements nor estimates of a separately identified causal mediation effect.
Costs and presentation. The μ \mu μ and π \pi π curves use the standard stochastic 500-tape, 1,000-period evaluator, at every saved checkpoint. In contrast,
C H ( g ) = 1 1152 ∑ h ω h H c h (250) C_H(g) = \frac{1}{1152}\sum_h \omega_h^H c_h \tag*{(250)} C H ( g ) = 1152 1 h ∑ ω h H c h (250)
is the realized immediate raw cost on the original controller occupancy panel: 16 anchors, eight paths per anchor, and nine stages per path. For DSIS-RB these paths use the generation-frozen fitted model; for DSIS they use the public model. Thus C H C_H C H is a short-rollout reference from training, not a standard-reset, long-run evaluation of a third policy. It uses neither a new rollout nor an additional bootstrap value, and has no fabricated terminal-generation point.
Costs are indexed by their own checkpoint generations. Link geometry and responses are indexed by the origin g g g of the complete delayed chain; the last heuristic update has no subsequent target update and is omitted from these linked curves. Warm axes add 20 generation equivalents for the 500,000-interaction native prefix; Cold axes begin at zero. All statistics are computed per seed before aggregation. The main figure shows arithmetic means over all five seeds, pointwise 25th–75th percentile bands for the link and response curves (linear interpolation), and individual seed cost curves. These bands describe seed dispersion, not confidence intervals. Undefined seed-level values leave a gap in the five-seed mean; they do not reduce its denominator. There is no smoothing or temporal subsampling. Logarithmic cost axes and inverse-hyperbolic-sine step/response axes alter only display coordinates, not the underlying means or signed components.