Introduction
Mean–variance portfolio selection provides an intuitive way to balance the expected terminal wealth against risk, which has been widely used in practice since the seminal work of Markowitz [34 ] and its dynamic extensions in Li and Ng [33 ] for multi-period model and in Zhou and Li [51 ] for the continuous-time model. The endogenous linear-quadratic structure brings the favorable tractability in the dynamic control problem, however, relying crucially on the precise information of model parameters. In practice, the asset drift is unobservable and particularly difficult to estimate using the history data [35 ] , leaving possibly large errors in implementing the theoretical optimal solution using the inaccurate drift [4 ] . A common response in the literature is to adopt the identify-then-optimize procedure used in adaptive control [5 , 31 , 18 ] . Once the resulting estimates are treated as the true parameters, the remaining parameter uncertainty is hidden. Consequently, when the estimated model is substituted directly into the complete-information feedback policy, estimation errors might be further amplified [4 , 30 ] .
These limitations motivate the development of reinforcement learning (RL) approach that allows the agent to learn the optimal decision via the trial-and-error interactions with unknown environment. For continuous-time diffusion models, Wang et al. (2020) propose the exploratory formulation of stochastic policies via the entropy regularization. Within this framework, Wang and Zhou [45 ] study the precommitted mean–variance problem and show that the optimal stochastic policy is Gaussian, which leads to a structured and implementable learning method. Along this direction, there is a fast growth in the development of entropy regularized reinforcement learning for continuous-time stochastic control problems and financial applications, see among Jia and Zhou [26 ] , Jia and Zhou [27 ] , Guo et al. [22 ] , Jia and Zhou [28 ] , Dai et al. [11 ] , Szpruch et al. [43 ] , Wu and Li [49 ] , Gierich et al. (2024), Wei et al. [48 ] , Bo et al. (2025), Wei and Yu [47 ] , Dai et al. [12 ] , Frikha et al. [19 ] , Jia et al. [29 ] , Huang et al. [23 ] , Gao et al. [20 ] , Bo et al. (2026), Huang et al. [24 ] , and references therein.
In all aforementioned studies, although the entropy regularization provides a tractable way for the policy randomization, its induced exogenous noise in the randomized policy cannot reflect the agent’s improving uncertainty on unknown model parameters along the learning. Intuitively, when the agent continues to learn, he would gain more information of the unknown environment and employ less randomness for the purpose of exploration. In addition, how to choose the temperature parameter in the standard entropy regularization to balance the tradeoff between exploration and exploitation in RL algorithms is still an open problem. On the other hand, Bayesian filtering quantifies that reduced unknownness through a posterior updated from observed prices, providing a statistical basis for calibrating exploration. The use of posterior beliefs to inform investment decisions is well established in portfolio optimization with partial information [14 , 50 ] . Lakner [32 ] derives optimal investment and consumption decisions when only prices are observed, while Sass and Haussmann [41 ] use a filter for a hidden Markov-chain drift to characterize the terminal-wealth optimizer. Bismuth et al. [6 ] extend Bayesian learning to portfolio-choice, liquidation, and transition problems with unknown expected returns. In these models, the filter carries information of both the estimated parameter and its remaining uncertainty along the investment horizon. The economic relevance of retaining that uncertainty is illustrated by Rogers [39 ] , which reveals that the welfare cost can exceed the cost of infrequent portfolio review.
Some recent studies simply combine this approach with the standard entropy-regularization in continuous-time RL. In a regime-switching market, Wu and Li [49 ] incorporate the filtered regime belief into an actor–critic algorithm in the classical exploratory formulation. For an unknown constant drift, Au [1 ] combines Bayesian filtering with action-space entropy regularization and shows that the mean allocation remains the deterministic Bayesian feedback, while entropy changes the allocation variance. In these formulations, the randomness in the stochastic policy still comes exogenously from the entropy regularization. The filtered belief can affect the optimal allocation variance, but a positive entropy weight can sustain the randomization in the stochastic policy even when parameter uncertainty vanishes, as illustrated by the known-drift limit in Au [1 ] . For a small investor whose trades do not affect market prices, randomized policy does not change the return observations available to the filter and therefore does not provide additional information to improve the knowledge of the unknown drift. Such randomization can generate different wealth outcomes for policy evaluation, but it also incurs the cost of trying unnecessary suboptimal policies [43 ] .
This work aims to provide a new perspective to address the issue of over-randomization in standard entropy regularization as well as a proper choice of temperature parameter during the learning procedure when the agent gets more information on the unknown parameters by filtering. We contribute a novel continuous-time RL formulation that uses the posterior to guide the exploration search over candidate Sharpe ratios based on the classical mean–variance feedback form. At each time, the agent selects a sampling density q t q_t q t , and each sampled ratio generates a portfolio allocation through that feedback rule. The Bayesian filter supplies the current Gaussian posterior p t p_t p t as a reference distribution. A Kullback–Leibler (KL) penalty on deviations of the sampling density from the posterior connects the parameter search to the information accumulated from returns. As the posterior variance decreases, it provides a shrinking effect for exploration. This construction is related to posterior sampling in Bayesian RL [42 ] , with the sampling law also allowed to respond to investment performance. Following the principle of reward-biased maximum likelihood estimation [36 ] , reward feedback can shift probability toward candidate parameters whose allocations yield better outcomes.
Our new formulation is built on the balance between the weighted posterior-relative KL penalty and the current policy value. This value scaling yields a multiplicative effect and preserves the quadratic structure inherited from the mean–variance objective, facilitating the explicit characterization of the optimal stochastic policy. For the algorithm aspect, the randomized allocations generate wealth outcomes for evaluating and improving portfolio policies. The Bayesian posterior continues to be updated from exogenous market returns independently of the investor’s position, so randomization does not accelerate posterior contraction. In deriving the exploratory dynamics, we retain the common return innovation shared by wealth and the posterior mean. Figure 1 summarizes the flow chart of how the posterior, parameter search, portfolio allocation, and reward feedback are implemented.
Figure 1. Posterior-guided reward-biased exploration.
The main contributions of the present paper are summarized as follows:
First, we formulate a posterior-guided and reward-biased exploratory control framework, new to the literature, to learn the optimal policy in the context of mean-variance portfolio optimization. Its main novelty is a multiplicative criterion in which the posterior-relative KL divergence is weighted by both the exploration coefficient and the current policy value. This value scaling preserves the quadratic homogeneity inherited from the mean–variance objective.
Second, we solve this exploratory control problem analytically. Notably, one state variable is an evolving probability density and the associated equation is nonlinear. However, by strategically choosing the temperature parameter in a proper form, we can still exploit the mean–variance structure and Gaussian posterior to reduce the optimizer to a unique Gaussian law and derive closed-form expressions for the value function and the stochastic policy, including the exceptional case in which the general solution is not directly applicable. The solution explicitly separates the role of posterior uncertainty in controlling the exploration scale from that of reward feedback, and shows that the degree of exploration declines automatically as the filtering information accumulates.
Third, we turn the analytical result into an implementable learning method and establish some learning performance guarantees. To overcome the unknown oracle term in the closed-form policy and the instability caused by vanishing exploration variance, we develop a structure-preserving episodic actor–critic algorithm with specially normalized and centered martingale updates. We rigorously prove convergence of the filter, actor, sampling policy as well as Lagrange multiplier without requiring the critic proxy itself to identify the true Sharpe ratio. We further show that first-order policy errors cancel at the limiting optimum, leading to a nearly inverse-linear last-iterate performance rate in the stationary market.
Fourth, to illustrate the learning performance of the proposed algorithm, we conduct both simulation experiments and empirical study by comparing benchmarks from the rolling maximum likelihood and entropy-regularized Bayesian-filtering. The results complement the theoretical findings and showcase the improved risk-adjusted performance and greater stability of learning.
The remainder of the paper is organized as follows. Section 2 introduces the classical benchmark, the Bayesian filter, and the exploratory state dynamics. Section 3 formulates the posterior-guided KL objective and derives the explicit characterization of optimal Gaussian stochastic policy. Section 4 presents the actor–critic model-based RL algorithm and establishes its convergence analysis. Sections 5 and 6 report the simulation experiments and empirical studies of the RL algorithm, respectively. The concluding section summarizes the findings, and the Appendix collects the proofs and auxiliary results.
Bayesian Oriented Exploratory Formulation
In this section, we first identify the dependence of the mean-variance feedback control on the market Sharpe ratio, and then introduce a filtering state and derive exploratory dynamics that preserve the common return innovation between wealth and beliefs.
Classical Benchmark
Let T > 0 T > 0 T > 0 be the finite horizon, and let { W t , 0 ≤ t ≤ T } \{W_t, 0 \le t \le T\} { W t , 0 ≤ t ≤ T } be a standard one-dimensional Brownian motion defined on a filtered probability space ( Ω , F , { F t } 0 ≤ t ≤ T , P ) (\Omega,\mathcal{F},\{\mathcal{F}_t\}_{0 \le t \le T},\mathbb{P}) ( Ω , F , { F t } 0 ≤ t ≤ T , P ) that satisfies the usual conditions. The price process of the risky asset is a geometric Brownian motion governed by
d S t = S t ( μ d t + σ d W t ) , 0 ≤ t ≤ T , (1) \mathrm{d}S_t = S_t \left(\mu\mathrm{d}t + \sigma\mathrm{d}W_t\right), \qquad0 \le t \le T, \tag*{(1)} d S t = S t ( μ d t + σ d W t ) , 0 ≤ t ≤ T , (1)
with S 0 = s 0 > 0 S_0 = s_0 > 0 S 0 = s 0 > 0 being the initial price, and μ ∈ R \mu\in\mathbb{R} μ ∈ R , σ > 0 \sigma> 0 σ > 0 being the mean and volatility parameters, respectively. The riskless asset has a constant interest rate r > 0 r > 0 r > 0 . The Sharpe ratio of the risky asset is defined by
ρ = μ − r σ . (2) \rho= \frac{\mu-r}{\sigma}. \tag*{(2)} ρ = σ μ − r . (2)
Denote by { X t u , 0 ≤ t ≤ T } \{X_t^u, 0 \le t \le T\} { X t u , 0 ≤ t ≤ T } the discounted wealth process under the portfolio strategy u = { u t , 0 ≤ t ≤ T } u=\{u_t, 0 \le t \le T\} u = { u t , 0 ≤ t ≤ T } . Here, u t u_t u t is the discounted dollar value allocated in the risky asset at time t t t , which satisfies the standard self-financing assumption and admissibility conditions that will be given later. It follows from (1) that the discounted wealth process satisfies
d X t u = σ u t ( ρ d t + d W t ) , 0 ≤ t ≤ T , (3) dX_t^u=\sigma u_t(\rho dt+dW_t),\qquad0\le t\le T, \tag*{(3)} d X t u = σ u t ( ρ d t + d W t ) , 0 ≤ t ≤ T , (3)
with initial wealth X 0 u = x 0 ∈ R X_0^u=x_0\in\mathbb{R} X 0 u = x 0 ∈ R .
When the parameter θ ∗ : = ( μ ∗ , σ ∗ ) \theta^*:=(\mu^*,\sigma^*) θ ∗ := ( μ ∗ , σ ∗ ) is known, the classical continuous-time Mean-Variance (MV) model aims to solve the optimization problem [45 ] :
V c l ( s , y ; w ) : = min u ∈ A c l ( s , y ) E [ ( X T u − w ) 2 ∣ X s u = y ] − ( w − z ) 2 , (4) V^{\mathrm{cl}}(s,y;w):=\min_{u\in\mathcal{A}^{\mathrm{cl}}(s,y)}\mathbb{E}\left[(X_T^u-w)^2\mid X_s^u=y\right]-(w-z)^2, \tag*{(4)} V cl ( s , y ; w ) := u ∈ A cl ( s , y ) min E [ ( X T u − w ) 2 ∣ X s u = y ] − ( w − z ) 2 , (4)
for ( s , y ) ∈ [ 0 , T ) × R (s,y)\in[0,T)\times\mathbb{R} ( s , y ) ∈ [ 0 , T ) × R , where w ∈ R w\in\mathbb{R} w ∈ R is a fixed Lagrange multiplier. Once this problem is solved, w w w can be determined by the constraint E [ X T ∗ ] = z \mathbb{E}[X_T^*]=z E [ X T ∗ ] = z , with { X t ∗ } t ∈ [ 0 , T ] \{X_t^*\}_{t\in[0,T]} { X t ∗ } t ∈ [ 0 , T ] being the optimal wealth process under optimal portfolio u ∗ u^* u ∗ .
In this case, it is well known that the optimal control u ∗ u^* u ∗ of (4) is given in a feedback form by
u ∗ = ψ ( t , X t ∗ , w ; θ ∗ ) = − ρ ∗ σ ∗ ( X t ∗ − w ) , (5) u^*=\psi(t,X_t^*,w;\theta^*)=-\frac{\rho^*}{\sigma^*}(X_t^*-w), \tag*{(5)} u ∗ = ψ ( t , X t ∗ , w ; θ ∗ ) = − σ ∗ ρ ∗ ( X t ∗ − w ) , (5)
and the resulting optimal wealth process is the unique strong solution to the SDE
d X t ∗ = − ( ρ ∗ ) 2 ( X t ∗ − w ) d t − ρ ∗ ( X t ∗ − w ) d W t , X 0 ∗ = x 0 . (6) dX_t^*=-(\rho^*)^2(X_t^*-w)dt-\rho^*(X_t^*-w)dW_t,\qquad X_0^*=x_0. \tag*{(6)} d X t ∗ = − ( ρ ∗ ) 2 ( X t ∗ − w ) d t − ρ ∗ ( X t ∗ − w ) d W t , X 0 ∗ = x 0 . (6)
If the parameter θ ∗ \theta^* θ ∗ is unknown to the agent, the agent searches for the optimal control to solve the MV model while learning the system, i.e., the parameter θ ∗ \theta^* θ ∗ . However, if the agent is a small investor, the current portfolio selection problem has a distinct feature: the instantaneous returns of the risky and risk-free assets can be observed directly from the market, therefore, the information used for learning is unaffected by the portfolio strategies the agent applies. That said, the agent is always able to use an estimate θ ^ \hat{\theta} θ ^ (by observing market’s time evolution) to solve the optimization problem while disregarding the modeling error, and use the resulting controller ψ ( θ ) \psi(\theta) ψ ( θ ) on the true system with θ ∗ \theta^* θ ∗ . That is, at time t > 0 t>0 t > 0 , by observing { S s } s ∈ [ 0 , t ] \{S_s\}_{s\in[0,t]} { S s } s ∈ [ 0 , t ] , a maximum likelihood estimator (MLE) can be derived from maximizing log-likelihood: θ ^ t \hat{\theta}_t θ ^ t . Then, a feedback control ψ ( ⋅ ; w , θ ^ t ) \psi(\cdot;w,\hat{\theta}_t) ψ ( ⋅ ; w , θ ^ t ) is exercised, and the state process evolves according to the dynamics (3) controlled by policy ψ ( ⋅ ; w , θ ^ t ) \psi(\cdot;w,\hat{\theta}_t) ψ ( ⋅ ; w , θ ^ t ) :
d X t = σ ∗ ψ ( t , X t ; w , θ ^ t ) ( ρ ∗ d t + d W t ) , 0 ≤ t ≤ T , (7) dX_t=\sigma^*\psi(t,X_t;w,\hat{\theta}_t)(\rho^*dt+dW_t),\qquad0\le t\le T, \tag*{(7)} d X t = σ ∗ ψ ( t , X t ; w , θ ^ t ) ( ρ ∗ d t + d W t ) , 0 ≤ t ≤ T , (7)
The agent then inherits the sampling error of θ ^ t \hat{\theta}_t θ ^ t . Because these parameters are estimated from random samples, they are themselves random and introduce modeling errors that may propagate into undesirable behaviors over long time horizons. To address this, the next subsection retains the estimation uncertainty as an explicit state variable rather than treating the estimate as exact.
Filtering and Exploratory Dynamics
When σ ∗ \sigma^* σ ∗ is known, the stochastic filtering theory characterizes the conditional distribution process { ρ t : t ≥ 0 } \{\rho_t:t\ge0\} { ρ t : t ≥ 0 } of ρ ∗ \rho^* ρ ∗ given S t S_t S t , where S t S_t S t is the (completed) σ \sigma σ -algebra generated by the observation (stock price process { S s } s ∈ [ 0 , t ] \{S_s\}_{s\in[0,t]} { S s } s ∈ [ 0 , t ] ) up to time t t t . We recall that in this setting, the conditional distribution ρ t \rho_t ρ t of ρ ∗ \rho^* ρ ∗ given S t S_t S t is a normal distribution, and hence completely characterized by its mean m t = E [ ρ ∗ ∣ S t ] m_t=\mathbb{E}[\rho^*|S_t] m t = E [ ρ ∗ ∣ S t ] and the variance R t = E [ ( ρ ∗ − m t ) 2 ∣ S t ] R_t=\mathbb{E}[(\rho^*-m_t)^2|S_t] R t = E [( ρ ∗ − m t ) 2 ∣ S t ] , which satisfy the following equation
d m t = R t σ ∗ ( d S t S t − ρ d t − σ ∗ m t d t ) = R t ( ( ρ ∗ − m t ) d t + d W t ) , (8) dm_t=\frac{R_t}{\sigma^*}\left(\frac{dS_t}{S_t}-\rho dt-\sigma^*m_tdt\right)=R_t((\rho^*-m_t)dt+dW_t), \tag*{(8)} d m t = σ ∗ R t ( S t d S t − ρ d t − σ ∗ m t d t ) = R t (( ρ ∗ − m t ) d t + d W t ) , (8)
d R t = − R t 2 d t , (9) dR_t=-R_t^2dt, \tag*{(9)} d R t = − R t 2 d t , (9)
with the prior distribution of ρ ∗ \rho^* ρ ∗ given by N ( m 0 , R 0 ) \mathcal{N}(m_0,R_0) N ( m 0 , R 0 ) at time 0. It follows that posterior variance R t R_t R t is deterministic and has closed-form: R t = R 0 / ( 1 + R 0 t ) R_t=R_0/(1+R_0t) R t = R 0 / ( 1 + R 0 t ) , which means the uncertainty decays at rate O ( 1 / t ) O(1/t) O ( 1/ t ) regardless of the observations or policy. In addition, m t m_t m t is a S t \mathcal{S}_t S t -martingale, converges to ρ ∗ \rho^* ρ ∗ a.s. When σ ∗ \sigma^* σ ∗ is unknown, we estimate it from observed returns and substitute the estimate into the filtering equations, obtaining a plug-in filtering state ( m t , R t ) (m_t, R_t) ( m t , R t ) . Although this procedure provides an estimate of the unknown Sharpe ratio, uncertainty about that parameter remains and carries over to the portfolio allocation determined by the classical feedback rule. We therefore use the filtering state to guide a stochastic portfolio policy, denoted by π t \pi_t π t , retaining parameter uncertainty in the policy construction rather than relying solely on a point estimate.
Recall that the classical optimal feedback control, u ∗ = ψ ( ⋅ ; θ ) u^* = \psi(\cdot;\theta) u ∗ = ψ ( ⋅ ; θ ) as given by (5), is explicitly determined by the true parameter. To construct the exploratory policy, we treat ρ \rho ρ as a random variable drawn from a sampling distribution q t ( ρ ∣ m t , R t ) q_t(\rho\mid m_t, R_t) q t ( ρ ∣ m t , R t ) . This mapping naturally transforms the classical feedback control into a stochastic continuous-time policy π t ( u ∣ m t , R t ) \pi_t(u \mid m_t, R_t) π t ( u ∣ m t , R t ) . Crucially, q t q_t q t need not be the exact Bayesian posterior p t p_t p t . While this approach serves as a continuous-time analog of Thompson Sampling [40 ] , relaxing the strict Bayesian assumption leaves mathematical room to integrate robust, generalized learning frameworks, such as the Reward-Biased Maximum Likelihood Estimation (RBMLE) approach which will be introduced in § 3 .
To rigorously analyze the macroscopic behavior of the state process under this randomized policy, we adopt the exploratory stochastic differential equation (SDE) framework introduced by Wang and Zhou (2020). We consider N N N independent identical agents, each experiencing an independent realization path of the physical Brownian motion W i W^i W i , i = 1 , ⋯ , N i = 1, \cdots, N i = 1 , ⋯ , N , and updates their Bayesian posterior ( m t i , R t ) (m^i_t, R_t) ( m t i , R t ) , i = 1 , ⋯ , N i = 1, \cdots, N i = 1 , ⋯ , N identically. They sample independently their own control u t i u^i_t u t i , i = 1 , ⋯ , N i = 1, \cdots, N i = 1 , ⋯ , N , from random policy π t \pi_t π t , resulting in X t i X^i_t X t i , i = 1 , ⋯ , N i = 1, \cdots, N i = 1 , ⋯ , N being copies of state process. Then the increments of these state process copies are, for i = 1 , ⋯ , N i = 1, \cdots, N i = 1 , ⋯ , N ,
Δ X t i = σ ∗ ρ ∗ u t i Δ t + σ ∗ u t i ( W t + Δ t i − W t i ) , Δ m t i = R t ( ( ρ ∗ − m t i ) Δ t + ( W t + Δ t i − W t i ) ) , t ≥ 0. (10) \Delta X^i_t = \sigma^* \rho^* u^i_t \Delta t + \sigma^* u^i_t(W^i_{t+\Delta t} - W^i_t), \qquad\Delta m^i_t = R_t\left((\rho^* - m^i_t)\Delta t + (W^i_{t+\Delta t} - W^i_t)\right), \qquad t \ge0. \tag*{(10)} Δ X t i = σ ∗ ρ ∗ u t i Δ t + σ ∗ u t i ( W t + Δ t i − W t i ) , Δ m t i = R t ( ( ρ ∗ − m t i ) Δ t + ( W t + Δ t i − W t i ) ) , t ≥ 0. (10)
Because the randomized policy π t \pi_t π t is parameterized by the current Bayesian posterior ( m t , R t ) (m_t, R_t) ( m t , R t ) , it is inherently path-dependent. However, the randomization of the action u t i u^i_t u t i is conditionally independent of the physical noise Δ W t \Delta W_t Δ W t . Therefore, by the law of large numbers, as N → ∞ N \to\infty N → ∞ , the empirical averages of the state increments converge almost surely to expectation taken over action space, while leaving the physical filtration of W t W_t W t intact:
1 N ∑ i = 1 N Δ X t i → a . s . E u ∼ π t [ Δ X t i ] , 1 N ∑ i = 1 N ( Δ X t i ) 2 → a . s . E u ∼ π t [ ( Δ X t i ) 2 ] ; \frac{1}{N}\sum_{i=1}^{N}\Delta X^i_t \xrightarrow{\mathrm{a.s.}} \mathbb{E}_{u\sim\pi_t}[\Delta X^i_t], \qquad\frac{1}{N}\sum_{i=1}^{N}(\Delta X^i_t)^2 \xrightarrow{\mathrm{a.s.}} \mathbb{E}_{u\sim\pi_t}[(\Delta X^i_t)^2]; N 1 i = 1 ∑ N Δ X t i a.s. E u ∼ π t [ Δ X t i ] , N 1 i = 1 ∑ N ( Δ X t i ) 2 a.s. E u ∼ π t [( Δ X t i ) 2 ] ;
1 N ∑ i = 1 N ( Δ X t i Δ m t i ) → a . s . E u ∼ π t [ Δ X t i Δ m t i ] . \frac{1}{N}\sum_{i=1}^{N}(\Delta X^i_t\Delta m^i_t) \xrightarrow{\mathrm{a.s.}} \mathbb{E}_{u\sim\pi_t}[\Delta X^i_t\Delta m^i_t]. N 1 i = 1 ∑ N ( Δ X t i Δ m t i ) a.s. E u ∼ π t [ Δ X t i Δ m t i ] .
This heuristic convergence motivates the definition of exploratory state dynamics , denoted by X t π X^\pi_t X t π . Each pair ( X i , m i ) (X^i, m^i) ( X i , m i ) , i = 1 , ⋯ , N i = 1, \cdots, N i = 1 , ⋯ , N , can be viewed as an independent sample from the structural exploratory state dynamics ( X π , m ) (X^\pi, m) ( X π , m ) . That is, each ( X i , m i ) (X^i, m^i) ( X i , m i ) is generated according to the classical dynamic (7) and (8), with the corresponding u u u sampled independently under policy π \pi π . Under this framework, the effective drift, squared volatility, and cross-variations of X t π X^\pi_t X t π are obtained by integrating the classical physical coefficients over the control distribution π t \pi_t π t .
In our structural expectation method, the control is mapped from a parameterized distribution over the standardized drift, ρ ∼ q t ( ⋅ ∣ m t , R t ) \rho\sim q_t(\cdot\mid m_t, R_t) ρ ∼ q t ( ⋅ ∣ m t , R t ) , with mean M t M_t M t and variance V t V_t V t . Substituting the action mapping u t = − ρ ( X t π − w ) / σ ∗ u_t = -\rho(X^\pi_t - w)/\sigma^* u t = − ρ ( X t π − w ) / σ ∗ , we explicitly evaluate these exploratory coefficients:
1 Δ t E u ∼ π t [ Δ X t i ] → Δ t ↓ 0 E ρ ∼ q t [ σ ∗ ρ ∗ u t ] = σ ∗ ρ ∗ E ρ ∼ q t [ − ρ σ ∗ ( X t π − w ) ] = − ρ ∗ ( X t π − w ) M t . \frac{1}{\Delta t}\mathbb{E}_{u\sim\pi_t}[\Delta X^i_t] \xrightarrow{\Delta t\downarrow0} \mathbb{E}_{\rho\sim q_t}[\sigma^*\rho^*u_t] = \sigma^*\rho^*\mathbb{E}_{\rho\sim q_t}\left[-\frac{\rho}{\sigma^*}(X^\pi_t-w)\right] = -\rho^*(X^\pi_t-w)M_t. Δ t 1 E u ∼ π t [ Δ X t i ] Δ t ↓ 0 E ρ ∼ q t [ σ ∗ ρ ∗ u t ] = σ ∗ ρ ∗ E ρ ∼ q t [ − σ ∗ ρ ( X t π − w ) ] = − ρ ∗ ( X t π − w ) M t .
1 Δ t E u ∼ π t [ ( Δ X t i ) 2 ] → Δ t ↓ 0 E ρ ∼ q t [ ( σ ∗ u t ) 2 ] = ( σ ∗ ) 2 E ρ ∼ q t [ ρ 2 ( σ ∗ ) 2 ( X t π − w ) 2 ] = ( X t π − w ) 2 ( M t 2 + V t ) . \frac{1}{\Delta t}\mathbb{E}_{u\sim\pi_t}[(\Delta X^i_t)^2] \xrightarrow{\Delta t\downarrow0} \mathbb{E}_{\rho\sim q_t}[(\sigma^*u_t)^2] = (\sigma^*)^2\mathbb{E}_{\rho\sim q_t}\left[\frac{\rho^2}{(\sigma^*)^2}(X^\pi_t-w)^2\right] = (X^\pi_t-w)^2(M_t^2+V_t). Δ t 1 E u ∼ π t [( Δ X t i ) 2 ] Δ t ↓ 0 E ρ ∼ q t [( σ ∗ u t ) 2 ] = ( σ ∗ ) 2 E ρ ∼ q t [ ( σ ∗ ) 2 ρ 2 ( X t π − w ) 2 ] = ( X t π − w ) 2 ( M t 2 + V t ) .
1 Δ t E u ∼ π t [ Δ X t i Δ m t i ] → Δ t ↓ 0 E ρ ∼ q t [ σ ∗ u t R t ] = σ ∗ R t E ρ ∼ q t [ − ρ σ ∗ ( X t π − w ) ] = − M t R t ( X t π − w ) . \frac{1}{\Delta t}\mathbb{E}_{u\sim\pi_t}[\Delta X^i_t\Delta m^i_t] \xrightarrow{\Delta t\downarrow0} \mathbb{E}_{\rho\sim q_t}[\sigma^*u_tR_t] = \sigma^*R_t\mathbb{E}_{\rho\sim q_t}\left[-\frac{\rho}{\sigma^*}(X^\pi_t-w)\right] = -M_tR_t(X^\pi_t-w). Δ t 1 E u ∼ π t [ Δ X t i Δ m t i ] Δ t ↓ 0 E ρ ∼ q t [ σ ∗ u t R t ] = σ ∗ R t E ρ ∼ q t [ − σ ∗ ρ ( X t π − w ) ] = − M t R t ( X t π − w ) .
Synthesizing these limits, we arrive at the complete macroscopic SDE system. Let B B B be a standard Brownian motion independent of W W W . The exploratory state dynamics X t π X^\pi_t X t π and the path-dependent learning process m t m_t m t are governed by the following coupled system:
d X t π = − ρ ∗ ( X t π − w ) M t d t − ( X t π − w ) ( M t d W t + V t d B t ) , d m t = R t ( ( ρ ∗ − m t ) d t + d W t ) , d R t = − R t 2 d t , (11) \begin{aligned}
dX^\pi_t &= -\rho^*(X^\pi_t-w)M_tdt-(X^\pi_t-w)(M_tdW_t+\sqrt{V_t}dB_t),\\
dm_t &= R_t((\rho^*-m_t)dt+dW_t), \qquad dR_t=-R_t^2dt, \tag*{(11)}
\end{aligned} d X t π d m t = − ρ ∗ ( X t π − w ) M t d t − ( X t π − w ) ( M t d W t + V t d B t ) , = R t (( ρ ∗ − m t ) d t + d W t ) , d R t = − R t 2 d t , (11)
with the explicit cross-variation d ⟨ X π , m ⟩ t = − M t R t ( X t π − w ) d t d\langle X^\pi,m\rangle_t=-M_tR_t(X^\pi_t-w)dt d ⟨ X π , m ⟩ t = − M t R t ( X t π − w ) d t .
RBMLE-based Reinforcement Learning Objective
Having established the exploratory state dynamics, we next formulate the objective of learning. In classical adaptive control, certainty-equivalent policies rely solely on an MLE estimator or purely Bayesian estimates m t m_t m t . The statistical estimator is inherently biased in favor of P θ ∗ \mathbb{P}^{\theta^*} P θ ∗ ’s lower optimal rewards (or higher optimal cost) than under P θ \mathbb{P}^{\theta} P θ . Therefore to extricate oneself from this bind, one must necessarily tilt the balance toward exploring parameters with higher optimal rewards (or lower optimal cost). Motivated by this, Reward Biased MLE (RBMLE) is proposed to counteract this statistical trap by introducing an optimistic bias, tilting the parameter estimation toward regions that promise higher optimal rewards (or lower optimal cost) [36 ] .
We employ this RBMLE principle into our continuous-time structural exploration framework. Rather than acting on a biased point estimate, the agent controls the trajectory of the system by selecting a dynamic distribution q t ( ρ ) q_t(\rho) q t ( ρ ) over the Sharpe ratio ρ \rho ρ . Mathematically, the agent seeks to minimize the expected terminal tracking cost E [ ( X T π − w ) 2 ] \mathbb{E}[(X_T^\pi-w)^2] E [( X T π − w ) 2 ] regularized by the cumulative Kullback-Leibler (KL) divergence between the chosen exploratory policy q t q_t q t and the true Bayesian posterior p t = N ( m t , R t ) p_t=\mathcal{N}(m_t,R_t) p t = N ( m t , R t ) :
J ( t , x , m ) = inf q ∈ Q ( t , x , m ) E [ ( X T π − w ) 2 + ∫ t T λ s η ( s , X s π , m s ) D K L ( q s ∥ N ( m s , R s ) ) d s | X t π = x , m t = m ] . J(t,x,m)=\inf_{q\in\mathcal{Q}(t,x,m)}\mathbb{E}\left[(X_T^\pi-w)^2+\int_t^T\lambda_s\eta(s,X_s^\pi,m_s)D_{\mathrm{KL}}\left(q_s\middle\|\mathcal{N}(m_s,R_s)\right)\,ds\middle|X_t^\pi=x,m_t=m\right]. J ( t , x , m ) = q ∈ Q ( t , x , m ) inf E [ ( X T π − w ) 2 + ∫ t T λ s η ( s , X s π , m s ) D KL ( q s ∥ N ( m s , R s ) ) d s X t π = x , m t = m ] .
subject to system (11) . In this formulation, the state-dependent weight η ( s , X s π , m s ) \eta(s,X_s^\pi,m_s) η ( s , X s π , m s ) is chosen as J ( s , X s π , m s ) J(s,X_s^\pi,m_s) J ( s , X s π , m s ) to align with the magnitude of optimal objective function, and λ s > 0 \lambda_s>0 λ s > 0 controls the scale-invariant exploration-exploitation trade-off.
It is worth noting the connection between our RBMLE objective and classical entropy-regularized reinforcement learning. Specifically, if we were to remove the Bayesian learning mechanism—ignoring the continuous-time filtration of observed data encapsulated by ( m t , R t ) (m_t,R_t) ( m t , R t ) —and instead replace the reference distribution N ( m s , R s ) \mathcal{N}(m_s,R_s) N ( m s , R s ) with a static, uninformative prior, the KL divergence penalty would mathematically degenerate into a standard differential entropy regularizer H ( q t ) = − ∫ R q t ( ρ ) log q t ( ρ ) d ρ \mathcal{H}(q_t)=-\int_{\mathbb{R}}q_t(\rho)\log q_t(\rho)d\rho H ( q t ) = − ∫ R q t ( ρ ) log q t ( ρ ) d ρ . In this light, our formulation dynamically incorporates the agent’s learning from observed data. This differs from Au [1 ] , where entropy regularizes the allocation distribution directly and the mean allocation remains the deterministic Bayesian feedback.
The optimization problem above cannot be posed over arbitrary density-valued processes. Progressive measurability alone does not guarantee that the policy moments are finite, that the controlled system (11) is well posed, or that the terminal loss and cumulative KL penalty have finite expectation. These properties are also needed when applying Itô’s formula, taking conditional expectations, and invoking dynamic programming. Hence, Definition 1 introduces the admissible distributional controls Q ( t , x , m ) \mathcal{Q}(t,x,m) Q ( t , x , m ) and thereby specifies the effective domain of the value function.
Definition 1. Let F = ( F s ) s ∈ [ t , T ] \mathbb{F}=(\mathcal{F}_s)_{s\in[t,T]} F = ( F s ) s ∈ [ t , T ] be the usual augmentation of the filtration generated by ( W , B ) (W,B) ( W , B ) . A distributional control q = { q s } s ∈ [ t , T ] q=\{q_s\}_{s\in[t,T]} q = { q s } s ∈ [ t , T ] belongs to Q ( t , x , m ) \mathcal{Q}(t,x,m) Q ( t , x , m ) if:
(i) q s ( ⋅ ) q_s(\cdot) q s ( ⋅ ) is a probability density on R \mathbb{R} R almost surely for almost every s s s ;
(ii) for every A ∈ B ( R ) A\in\mathcal{B}(\mathbb{R}) A ∈ B ( R ) , the process { ∫ A q s ( ρ ) d ρ } s ∈ [ t , T ] \left\{\int_Aq_s(\rho)d\rho\right\}_{s\in[t,T]} { ∫ A q s ( ρ ) d ρ } s ∈ [ t , T ] is F \mathbb{F} F -progressively measurable;
(iii) its moments M s = ∫ R ρ q s ( ρ ) d ρ M_s=\int_{\mathbb{R}}\rho q_s(\rho)d\rho M s = ∫ R ρ q s ( ρ ) d ρ , V s = ∫ R ( ρ − M s ) 2 q s ( ρ ) d ρ V_s=\int_{\mathbb{R}}(\rho-M_s)^2q_s(\rho)d\rho V s = ∫ R ( ρ − M s ) 2 q s ( ρ ) d ρ satisfy
E [ ∫ t T ( M s 2 + V s ) d s ] < ∞ ; \mathbb{E}\left[\int_t^T(M_s^2+V_s)ds\right]<\infty; E [ ∫ t T ( M s 2 + V s ) d s ] < ∞ ; (iv) the system (11) has a unique strong solution satisfying
E [ sup t ≤ s ≤ T ( ∣ X s π ∣ 2 + ∣ m s ∣ 2 ) ] < ∞ ; \mathbb{E}\left[\sup_{t\leq s\leq T}\left(|X_s^\pi|^2+|m_s|^2\right)\right]<\infty; E [ t ≤ s ≤ T sup ( ∣ X s π ∣ 2 + ∣ m s ∣ 2 ) ] < ∞ ; (v) Define Λ s q : = exp { ∫ t s λ v D K L ( q v ∥ N ( m v , R v ) ) d v } \Lambda_s^q:=\exp\left\{\int_t^s\lambda_vD_{\mathrm{KL}}\left(q_v\middle\|\mathcal{N}(m_v,R_v)\right)dv\right\} Λ s q := exp { ∫ t s λ v D KL ( q v ∥ N ( m v , R v ) ) d v } , t ≤ s ≤ T t\leq s\leq T t ≤ s ≤ T . The terminal integrating factor and the corresponding weighted terminal loss satisfy
Λ T q < ∞ almost surely , E [ ∣ Λ T q ( X T π − w ) 2 ∣ ] < ∞ ; \Lambda_T^q<\infty\quad\text{almost surely},\qquad\mathbb{E}\left[\left|\Lambda_T^q(X_T^\pi-w)^2\right|\right]<\infty; Λ T q < ∞ almost surely , E [ Λ T q ( X T π − w ) 2 ] < ∞ ;
(vi) For the positive classical HJB candidate v v v used in Theorem 1, the family
{ exp ( ∫ t τ λ s D K L ( q s ∥ N ( m s , R s ) ) d s ) v ( τ , X τ π , m τ ) : τ is an [ t , T ] -valued stopping time } \left\{\exp\left(\int_t^\tau\lambda_sD_{\mathrm{KL}}(q_s\Vert N(m_s,R_s))\,\mathrm{d}s\right)v(\tau,X_\tau^\pi,m_\tau):\tau\text{ is an }[t,T]\text{-valued stopping time}\right\} { exp ( ∫ t τ λ s D KL ( q s ∥ N ( m s , R s )) d s ) v ( τ , X τ π , m τ ) : τ is an [ t , T ] -valued stopping time }
is uniformly integrable.
Following the linear BSDE formulation of recursive utility in Dai et al. [13 ] , the policy value satisfies
d J s q = − λ s D K L ( q s ∥ N ( m s , R s ) ) J s q d s + Z s q ⋅ d ( W s , B s ) ⊤ , J T q = ( X T π − w ) 2 . (12) \mathrm{d}J_s^q=-\lambda_sD_{\mathrm{KL}}(q_s\Vert N(m_s,R_s))J_s^q\,\mathrm{d}s+Z_s^q\cdot\mathrm{d}(W_s,B_s)^\top,\qquad J_T^q=(X_T^\pi-w)^2. \tag*{(12)} d J s q = − λ s D KL ( q s ∥ N ( m s , R s )) J s q d s + Z s q ⋅ d ( W s , B s ) ⊤ , J T q = ( X T π − w ) 2 . (12)
Under condition (v), its unique nonnegative solution for which { Λ τ q J τ q : τ ≤ T } \{\Lambda_\tau^qJ_\tau^q:\tau\le T\} { Λ τ q J τ q : τ ≤ T } is uniformly integrable is
J s q = 1 Λ s q E [ Λ T q ( X T π − w ) 2 ∣ F s ] . (13) J_s^q=\frac{1}{\Lambda_s^q}\mathbb{E}\left[\Lambda_T^q(X_T^\pi-w)^2\mid\mathcal{F}_s\right]. \tag*{(13)} J s q = Λ s q 1 E [ Λ T q ( X T π − w ) 2 ∣ F s ] . (13)
Indeed, Λ q J q \Lambda^qJ^q Λ q J q is the uniformly integrable martingale with terminal value Λ T q ( X T π − w ) 2 \Lambda_T^q(X_T^\pi-w)^2 Λ T q ( X T π − w ) 2 : condition (v) makes this terminal value integrable, so multiplying (13) by Λ s q \Lambda_s^q Λ s q identifies Λ s q J s q \Lambda_s^qJ_s^q Λ s q J s q with a uniformly integrable martingale. The Brownian martingale representation theorem and the product rule then yield (12). Conversely, if another solution satisfies the stated uniform-integrability condition, the product rule makes Λ q J q \Lambda^qJ^q Λ q J q a local martingale, while uniform integrability upgrades it to a martingale. Its terminal value therefore forces the same conditional-expectation representation. This L 1 L^1 L 1 solution concept and the role of uniform integrability in uniqueness are standard; see Briand et al. [9 ] . Tonelli’s theorem and the tower property show that (13) is equivalent to the recursive relation (14). Condition (vi) is the single additional condition needed to pass from localized Itô identities to the terminal time in the verification argument.
A feedback distributional policy is a Borel kernel q ( ρ ∣ s , x , m ) q(\rho\mid s,x,m) q ( ρ ∣ s , x , m ) whose induced process q s ( ρ ) = q ( ρ ∣ s , X s π , m s ) q_s(\rho)=q(\rho\mid s,X_s^\pi,m_s) q s ( ρ ) = q ( ρ ∣ s , X s π , m s ) belongs to this class for every initial state. The process { q s } \{q_s\} { q s } is the corresponding open-loop control, whereas the deterministic kernel is the feedback (closed-loop) control. For an admissible feedback distributional policy q q q , let { q s } \{q_s\} { q s } be its induced open-loop control and let X π X^\pi X π be the corresponding wealth process. The policy-specific value function is defined recursively by
J q ( t , x , m ) : = E [ ( X T π − w ) 2 + ∫ t T λ s J q ( s , X s π , m s ) D K L ( q s ∥ N ( m s , R s ) ) d s ∣ X t π = x , m t = m ] . (14) J^q(t,x,m):=\mathbb{E}\left[(X_T^\pi-w)^2+\int_t^T\lambda_sJ^q(s,X_s^\pi,m_s)D_{\mathrm{KL}}(q_s\Vert N(m_s,R_s))\,\mathrm{d}s\mid X_t^\pi=x,m_t=m\right]. \tag*{(14)} J q ( t , x , m ) := E [ ( X T π − w ) 2 + ∫ t T λ s J q ( s , X s π , m s ) D KL ( q s ∥ N ( m s , R s )) d s ∣ X t π = x , m t = m ] . (14)
Consequently, the optimal value function satisfies J ( t , x , m ) = inf q ∈ Q ( t , x , m ) J q ( t , x , m ) J(t,x,m)=\inf_{q\in\mathcal{Q}(t,x,m)}J^q(t,x,m) J ( t , x , m ) = inf q ∈ Q ( t , x , m ) J q ( t , x , m ) .
For the Gaussian policies used below, M s M_s M s and V s V_s V s depend on ( s , m s ) (s,m_s) ( s , m s ) but not on wealth. Since the filtering equation has a unique strong solution, the wealth equation is then linear and, whenever the displayed integrals are finite, has the unique solution
X s π − w = ( x − w ) exp { − ∫ t s ( ρ ∗ M v + 1 2 ( M v 2 + V v ) ) d v − ∫ t s M v d W v − ∫ t s V v d B v } . (15) X_s^\pi-w=(x-w)\exp\left\{-\int_t^s\left(\rho^*M_v+\frac{1}{2}(M_v^2+V_v)\right)\,\mathrm{d}v-\int_t^sM_v\,\mathrm{d}W_v-\int_t^s\sqrt{V_v}\,\mathrm{d}B_v\right\}. \tag*{(15)} X s π − w = ( x − w ) exp { − ∫ t s ( ρ ∗ M v + 2 1 ( M v 2 + V v ) ) d v − ∫ t s M v d W v − ∫ t s V v d B v } . (15)
Thus, existence and pathwise uniqueness follow directly for the policy class considered in this paper. Meanwhile, the moment and cost bounds remain substantive admissibility conditions.
Exploratory HJB Equation and Gaussian Optimal Policy
Having specified a well-defined learning objective for the agent, we now proceed to solve for the optimal control. Since R s R_s R s is deterministic for a fixed initial posterior variance, the controlled Markov state is ( X s π , m s ) (X_s^\pi,m_s) ( X s π , m s ) , with R s R_s R s suppressed from the arguments of J J J . Motivated by the dynamic programming principle for relaxed controls (El Karoui et al., [17 ] ), we consider the following Bellman relation
J ( t , x , m ) = inf q ∈ Q ( t , x , m ) E [ J ( s , X s π , m s ) + ∫ t s λ v J ( v , X v π , m v ) D K L ( q v ∥ N ( m v , R v ) ) d v ∣ X t π = x , m t = m ] . J(t,x,m)=\inf_{q\in\mathcal{Q}(t,x,m)}\mathbb{E}\left[J(s,X_s^\pi,m_s)+\int_t^s\lambda_vJ(v,X_v^\pi,m_v)D_{\mathrm{KL}}(q_v\Vert N(m_v,R_v))\,\mathrm{d}v\mid X_t^\pi=x,m_t=m\right]. J ( t , x , m ) = q ∈ Q ( t , x , m ) inf E [ J ( s , X s π , m s ) + ∫ t s λ v J ( v , X v π , m v ) D KL ( q v ∥ N ( m v , R v )) d v ∣ X t π = x , m t = m ] .
For fixed ( t , x , m ) (t,x,m) ( t , x , m ) and fixed derivatives of J J J , the part of the controlled generator that depends on the sampled Sharpe ratio ρ \rho ρ is
g t ( ρ ) : = − ρ ∗ ρ ( x − w ) J x + 1 2 ρ 2 ( x − w ) 2 J x x − ρ R t ( x − w ) J x m , g_t(\rho):=-\rho^*\rho(x-w)J_x+\frac{1}{2}\rho^2(x-w)^2J_{xx}-\rho R_t(x-w)J_{xm}, g t ( ρ ) := − ρ ∗ ρ ( x − w ) J x + 2 1 ρ 2 ( x − w ) 2 J xx − ρ R t ( x − w ) J x m ,
which is quadratic in the sampled Sharpe ratio ρ \rho ρ . Thus the pointwise optimization over densities is
inf q { ∫ R g t ( ρ ) q ( ρ ) d ρ + λ t J D K L ( q ∥ p t ) } . (16) \inf_q \left\{ \int_{\mathbb{R}} g_t(\rho)q(\rho)d\rho+ \lambda_t J D_{\mathrm{KL}}(q\|p_t) \right\}. \tag*{(16)} q inf { ∫ R g t ( ρ ) q ( ρ ) d ρ + λ t J D KL ( q ∥ p t ) } . (16)
For x ≠ w x \ne w x = w , let
N t : = ∫ R exp { − g t ( ρ ) λ t J ( t , x , m ) } p t ( ρ ) d ρ . N_t := \int_{\mathbb{R}} \exp\left\{-\frac{g_t(\rho)}{\lambda_t J(t,x,m)}\right\}p_t(\rho)d\rho. N t := ∫ R exp { − λ t J ( t , x , m ) g t ( ρ ) } p t ( ρ ) d ρ .
By the Donsker–Varadhan variational representation of relative entropy [16 Lemma 2.1] , or directly by the following KL decomposition, for every admissible density q q q ,
∫ R g t ( ρ ) q ( ρ ) d ρ + λ t J ( t , x , m ) D K L ( q ∥ p t ) = − λ t J ( t , x , m ) log N t + λ t J ( t , x , m ) D K L ( q ∥ q t ∗ ) . (17) \int_{\mathbb{R}} g_t(\rho)q(\rho)d\rho+ \lambda_t J(t,x,m)D_{\mathrm{KL}}(q\|p_t) = -\lambda_t J(t,x,m)\log N_t + \lambda_t J(t,x,m)D_{\mathrm{KL}}(q\|q_t^*). \tag*{(17)} ∫ R g t ( ρ ) q ( ρ ) d ρ + λ t J ( t , x , m ) D KL ( q ∥ p t ) = − λ t J ( t , x , m ) log N t + λ t J ( t , x , m ) D KL ( q ∥ q t ∗ ) . (17)
where
q t ∗ ( ρ ) = p t ( ρ ) exp { − g t ( ρ ) / ( λ t J ( t , x , m ) ) } / N t . q_t^*(\rho) = p_t(\rho)\exp\left\{-g_t(\rho)/(\lambda_t J(t,x,m))\right\}/N_t. q t ∗ ( ρ ) = p t ( ρ ) exp { − g t ( ρ ) / ( λ t J ( t , x , m )) } / N t .
As the relative entropy is nonnegative, (17) shows that q t ∗ q_t^* q t ∗ uniquely minimizes (16), provided N t < ∞ N_t < \infty N t < ∞ . Because p t p_t p t is Gaussian and g t g_t g t is quadratic, the normalized Gibbs tilt q t ∗ q_t^* q t ∗ is again Gaussian. Hence the infinite-dimensional optimization may be written, without loss, in terms of its mean M M M and variance V > 0 V > 0 V > 0 . Proceeding with standard arguments, we deduce that J J J satisfies the Hamilton-Jacobi-Bellman (HJB) equation
J t + inf M , V { − ρ ∗ M ( x − w ) J x + 1 2 ( x − w ) 2 ( M 2 + V ) J x x + R t ( ρ ∗ − m ) J m + 1 2 R t 2 J m m − M R t ( x − w ) J x m + λ t J × D K L ( N ( M , V ) ∥ N ( m , R t ) ) } = 0. (18) J_t+\inf_{M,V}\left\{-\rho^*M(x-w)J_x+\frac{1}{2}(x-w)^2(M^2+V)J_{xx}+R_t(\rho^*-m)J_m+\frac{1}{2}R_t^2J_{mm}-MR_t(x-w)J_{xm}+\lambda_tJ\times D_{\mathrm{KL}}(N(M,V)\|N(m,R_t))\right\}=0. \tag*{(18)} J t + M , V inf { − ρ ∗ M ( x − w ) J x + 2 1 ( x − w ) 2 ( M 2 + V ) J xx + R t ( ρ ∗ − m ) J m + 2 1 R t 2 J mm − M R t ( x − w ) J x m + λ t J × D KL ( N ( M , V ) ∥ N ( m , R t )) } = 0. (18)
subject to J ( T , x , m ) = ( x − w ) 2 J(T,x,m)=(x-w)^2 J ( T , x , m ) = ( x − w ) 2 . In (18), the KL divergence between two Gaussians with respective means M M M , V V V , m m m , and R R R can be written as
D K L ( q ∥ p ) = 1 2 ( log R − log V − 1 + V + ( M − m ) 2 R ) . D_{\mathrm{KL}}(q\|p)=\frac{1}{2}\left(\log R-\log V-1+\frac{V+(M-m)^2}{R}\right). D KL ( q ∥ p ) = 2 1 ( log R − log V − 1 + R V + ( M − m ) 2 ) .
The verification argument below uses the linear-BSDE representation of recursive utility.
Theorem 1. Let v v v be continuous on [ 0 , T ] × R 2 [0,T]\times\mathbb{R}^2 [ 0 , T ] × R 2 , belong to C 1 , 2 , 2 C^{1,2,2} C 1 , 2 , 2 on [ 0 , T ) × ( R ∖ { w } ) × R [0,T)\times(\mathbb{R}\setminus\{w\})\times\mathbb{R} [ 0 , T ) × ( R ∖ { w }) × R , be positive for x ≠ w x\ne w x = w , and satisfy (18) and v ( T , x , m ) = ( x − w ) 2 v(T,x,m)=(x-w)^2 v ( T , x , m ) = ( x − w ) 2 . Suppose that its measurable Hamiltonian minimizer q ∗ q^* q ∗ is admissible in the sense of Definition 1. Then
v ( t , x , m ) ≤ J q ( t , x , m ) v(t,x,m)\le J^q(t,x,m) v ( t , x , m ) ≤ J q ( t , x , m ) for every q ∈ Q ( t , x , m ) q\in\mathcal{Q}(t,x,m) q ∈ Q ( t , x , m ) ,
with equality for q = q ∗ q=q^* q = q ∗ . Consequently, v = J v=J v = J and q ∗ q^* q ∗ is optimal.
The next subsection constructs the closed-form candidate. Its candidate-specific admissibility, moment bounds and the uniform-integrability condition in Definition 1 are verified after the explicit solution is obtained.
Closed-Form Solution
The wealth dynamics are multiplicative in X π − w X^\pi-w X π − w and the terminal loss is quadratic, so it is natural to seek a solution of J J J in the form J ( t , x , m ) = A ( t , m ) ( x − w ) 2 J(t,x,m)=A(t,m)(x-w)^2 J ( t , x , m ) = A ( t , m ) ( x − w ) 2 , subject to A ( T , m ) = 1 A(T,m)=1 A ( T , m ) = 1 . Substituting this ansatz into (18), we have
A t ( x − w ) 2 + inf M , V { − 2 A ρ ∗ M ( x − w ) 2 + A ( M 2 + V ) ( x − w ) 2 + R t ( ρ ∗ − m ) A m ( x − w ) 2 + 1 2 R t 2 A m m ( x − w ) 2 − 2 A m R t M ( x − w ) 2 + λ t A 2 ( x − w ) 2 ( log R t V − 1 + V + ( M − m ) 2 R t ) } = 0. A_t(x-w)^2+\inf_{M,V}\left\{-2A\rho^*M(x-w)^2+A(M^2+V)(x-w)^2+R_t(\rho^*-m)A_m(x-w)^2+\frac{1}{2}R_t^2A_{mm}(x-w)^2-2A_mR_tM(x-w)^2+\frac{\lambda_t A}{2}(x-w)^2\left(\log\frac{R_t}{V}-1+\frac{V+(M-m)^2}{R_t}\right)\right\}=0. A t ( x − w ) 2 + M , V inf { − 2 A ρ ∗ M ( x − w ) 2 + A ( M 2 + V ) ( x − w ) 2 + R t ( ρ ∗ − m ) A m ( x − w ) 2 + 2 1 R t 2 A mm ( x − w ) 2 − 2 A m R t M ( x − w ) 2 + 2 λ t A ( x − w ) 2 ( log V R t − 1 + R t V + ( M − m ) 2 ) } = 0.
The above equation is homogeneous in ( x − w ) 2 (x-w)^2 ( x − w ) 2 , therefore the infimum over M M M and V V V completely decouples from x x x , the first order condition gives the optimal pair ( M ∗ , V ∗ ) (M^*,V^*) ( M ∗ , V ∗ ) as follows:
A + λ t A 2 ( − 1 V ∗ + 1 R t ) = 0 ⇒ 1 V ∗ = 1 R t + 2 λ t A+\frac{\lambda_t A}{2}\left(-\frac{1}{V^*}+\frac{1}{R_t}\right)=0\Rightarrow\frac{1}{V^*}=\frac{1}{R_t}+\frac{2}{\lambda_t} A + 2 λ t A ( − V ∗ 1 + R t 1 ) = 0 ⇒ V ∗ 1 = R t 1 + λ t 2
2 A M ∗ − 2 A ρ ∗ − 2 A m R t + λ t A M ∗ − m R t = 0 ⇒ M ∗ V ∗ = m R t + 2 λ t ( ρ ∗ + A m ( t , m ) A ( t , m ) R t ) . 2AM^*-2A\rho^*-2A_mR_t+\lambda_t A\frac{M^*-m}{R_t}=0\Rightarrow\frac{M^*}{V^*}=\frac{m}{R_t}+\frac{2}{\lambda_t}\left(\rho^*+\frac{A_m(t,m)}{A(t,m)}R_t\right). 2 A M ∗ − 2 A ρ ∗ − 2 A m R t + λ t A R t M ∗ − m = 0 ⇒ V ∗ M ∗ = R t m + λ t 2 ( ρ ∗ + A ( t , m ) A m ( t , m ) R t ) .
By substituting the optimal structural controls M ∗ M^* M ∗ , V ∗ V^* V ∗ back into the HJB (18) , we obtain the PDE that governs A ( t , m ) A(t,m) A ( t , m ) as follows,
A t + 1 2 R t 2 A m m − Γ ( t ) A m 2 A + ( α A ( t ) m + β A ( t ) ) A m + ( κ A ( t ) m 2 + γ A ( t ) m + δ A ( t ) ) A = 0 , A ( T , m ) = 1 , A_t+\frac{1}{2}R_t^2A_{mm}-\Gamma(t)\frac{A_m^2}{A}+(\alpha_A(t)m+\beta_A(t))A_m+(\kappa_A(t)m^2+\gamma_A(t)m+\delta_A(t))A=0,\quad A(T,m)=1, A t + 2 1 R t 2 A mm − Γ ( t ) A A m 2 + ( α A ( t ) m + β A ( t )) A m + ( κ A ( t ) m 2 + γ A ( t ) m + δ A ( t )) A = 0 , A ( T , m ) = 1 ,
with time-dependent coefficients given by
Γ ( t ) = 2 V ∗ R t 2 λ t , α A ( t ) = − R t − 2 V ∗ , \Gamma(t)=\frac{2V^*R_t^2}{\lambda_t},\quad\alpha_A(t)=-R_t-2V^*, Γ ( t ) = λ t 2 V ∗ R t 2 , α A ( t ) = − R t − 2 V ∗ ,
β A ( t ) = R t ρ ∗ ( 1 − 4 V ∗ λ t ) , κ A ( t ) = λ t 2 R t − V ∗ λ t 2 R t 2 = V ∗ R t (after simplification) , \beta_A(t)=R_t\rho^*\left(1-\frac{4V^*}{\lambda_t}\right),\quad\kappa_A(t)=\frac{\lambda_t}{2R_t}-\frac{V^*\lambda_t}{2R_t^2}=\frac{V^*}{R_t}\ \text{(after simplification)}, β A ( t ) = R t ρ ∗ ( 1 − λ t 4 V ∗ ) , κ A ( t ) = 2 R t λ t − 2 R t 2 V ∗ λ t = R t V ∗ (after simplification) ,
γ A ( t ) = − 2 V ∗ ρ ∗ R t , δ A ( t ) = λ t 2 log ( R t V ∗ ) − 2 V ∗ ( ρ ∗ ) 2 λ t . \gamma_A(t)=-\frac{2V^*\rho^*}{R_t},\quad\delta_A(t)=\frac{\lambda_t}{2}\log\left(\frac{R_t}{V^*}\right)-\frac{2V^*(\rho^*)^2}{\lambda_t}. γ A ( t ) = − R t 2 V ∗ ρ ∗ , δ A ( t ) = 2 λ t log ( V ∗ R t ) − λ t 2 V ∗ ( ρ ∗ ) 2 .
The PDE is semilinear because of the term ( A m ) 2 / A (A_m)^2/A ( A m ) 2 / A . We use a logarithmic transform to rewrite the PDE in terms of relative derivatives of A A A . Let
ℓ ( t , m ) = log A ( t , m ) ⟹ A ( t , m ) = e ℓ ( t , m ) . \ell(t,m)=\log A(t,m)\ \Longrightarrow\ A(t,m)=e^{\ell(t,m)}. ℓ ( t , m ) = log A ( t , m ) ⟹ A ( t , m ) = e ℓ ( t , m ) .
The direct substitution gives
ℓ t + 1 2 R t 2 ℓ m m + ( 1 2 R t 2 − Γ ( t ) ) ℓ m 2 + ( α A ( t ) m + β A ( t ) ) ℓ m + κ A ( t ) m 2 + γ A ( t ) m + δ A ( t ) = 0 , (19) \ell_t+\frac{1}{2}R_t^2\ell_{mm}+\left(\frac{1}{2}R_t^2-\Gamma(t)\right)\ell_m^2+(\alpha_A(t)m+\beta_A(t))\ell_m+\kappa_A(t)m^2+\gamma_A(t)m+\delta_A(t)=0, \tag*{(19)} ℓ t + 2 1 R t 2 ℓ mm + ( 2 1 R t 2 − Γ ( t ) ) ℓ m 2 + ( α A ( t ) m + β A ( t )) ℓ m + κ A ( t ) m 2 + γ A ( t ) m + δ A ( t ) = 0 , (19)
with terminal condition ℓ ( T , m ) = 0 \ell(T,m)=0 ℓ ( T , m ) = 0 .
We strategically choose the temperature parameter as λ t = c R t \lambda_t=cR_t λ t = c R t , c > 0 c>0 c > 0 , to ensure that the exploration cost scales proportionally with the current level of parameter uncertainty, thereby maintaining a consistent exploration-exploitation balance across the entire investment horizon. In this case, Equation (19) becomes
ℓ t + 1 2 R t 2 ℓ m m + c − 2 2 ( c + 2 ) R t 2 ℓ m 2 + ( α A ( t ) m + β A ( t ) ) ℓ m + κ A ( t ) m 2 + γ A ( t ) m + δ A ( t ) = 0 , ℓ ( T , m ) = 0. \ell_t+\frac{1}{2}R_t^2\ell_{mm}+\frac{c-2}{2(c+2)}R_t^2\ell_m^2+(\alpha_A(t)m+\beta_A(t))\ell_m+\kappa_A(t)m^2+\gamma_A(t)m+\delta_A(t)=0,\quad\ell(T,m)=0. ℓ t + 2 1 R t 2 ℓ mm + 2 ( c + 2 ) c − 2 R t 2 ℓ m 2 + ( α A ( t ) m + β A ( t )) ℓ m + κ A ( t ) m 2 + γ A ( t ) m + δ A ( t ) = 0 , ℓ ( T , m ) = 0.
Because all remaining dependence on m m m is quadratic, we postulate the ansatz
ℓ ( t , m ) = 1 2 P ( t ) m 2 + Q ( t ) m + K ( t ) . \ell(t,m)=\frac{1}{2}P(t)m^2+Q(t)m+K(t). ℓ ( t , m ) = 2 1 P ( t ) m 2 + Q ( t ) m + K ( t ) .
When c ≠ 2 c\ne2 c = 2 , the coefficient of the quadratic term in the Riccati system is nonzero. Solving the resulting system yields
1 2 P ˙ + c − 2 2 ( c + 2 ) R t 2 P 2 + α A ( t ) P + κ A ( t ) = 0 , P ( T ) = 0 ; \frac{1}{2}\dot{P}+\frac{c-2}{2(c+2)}R_t^2P^2+\alpha_A(t)P+\kappa_A(t)=0,\quad P(T)=0; 2 1 P ˙ + 2 ( c + 2 ) c − 2 R t 2 P 2 + α A ( t ) P + κ A ( t ) = 0 , P ( T ) = 0 ;
Q ˙ + c − 2 c + 2 R t 2 P Q + α A ( t ) Q + β A ( t ) P + γ A ( t ) = 0 , Q ( T ) = 0 ; \dot{Q}+\frac{c-2}{c+2}R_t^2PQ+\alpha_A(t)Q+\beta_A(t)P+\gamma_A(t)=0,\quad Q(T)=0; Q ˙ + c + 2 c − 2 R t 2 P Q + α A ( t ) Q + β A ( t ) P + γ A ( t ) = 0 , Q ( T ) = 0 ;
K ˙ + 1 2 R t 2 P + c − 2 2 ( c + 2 ) R t 2 Q 2 + β A ( t ) Q + δ A ( t ) = 0 , K ( T ) = 0. \dot{K}+\frac{1}{2}R_t^2P+\frac{c-2}{2(c+2)}R_t^2Q^2+\beta_A(t)Q+\delta_A(t)=0,\quad K(T)=0. K ˙ + 2 1 R t 2 P + 2 ( c + 2 ) c − 2 R t 2 Q 2 + β A ( t ) Q + δ A ( t ) = 0 , K ( T ) = 0.
The above ODE system admits the explicit solution:
P ( t ) = 1 R t r + r − ( 1 − η t ) r + − r − η t , P(t)=\frac{1}{R_t}\frac{r_+r_-(1-\eta_t)}{r_+-r_-\eta_t}, P ( t ) = R t 1 r + − r − η t r + r − ( 1 − η t ) ,
Q ( t ) = ρ ∗ R t [ r + − r − r + − r − η t ( R t R T ) c + 2 − Δ c 2 ( c + 2 ) − 1 ] , Q(t)=\frac{\rho^*}{R_t}\left[\frac{r_+-r_-}{r_+-r_-\eta_t}\left(\frac{R_t}{R_T}\right)^{\frac{c+2-\Delta_c}{2(c+2)}}-1\right], Q ( t ) = R t ρ ∗ [ r + − r − η t r + − r − ( R T R t ) 2 ( c + 2 ) c + 2 − Δ c − 1 ] ,
K ( t ) = ∫ t T [ 1 2 R s 2 P ( s ) + c − 2 2 ( c + 2 ) R s 2 Q 2 ( s ) + ρ ∗ c − 2 c + 2 R s Q ( s ) + c R s 2 log c + 2 c − 2 ( ρ ∗ ) 2 c + 2 ] d s , K(t)=\int_t^T\left[\frac{1}{2}R_s^2P(s)+\frac{c-2}{2(c+2)}R_s^2Q^2(s)+\rho^*\frac{c-2}{c+2}R_sQ(s)+\frac{cR_s}{2}\log\frac{c+2}{c}-\frac{2(\rho^*)^2}{c+2}\right]\,ds, K ( t ) = ∫ t T [ 2 1 R s 2 P ( s ) + 2 ( c + 2 ) c − 2 R s 2 Q 2 ( s ) + ρ ∗ c + 2 c − 2 R s Q ( s ) + 2 c R s log c c + 2 − c + 2 2 ( ρ ∗ ) 2 ] d s ,
where Δ c = 17 c 2 + 36 c + 4 \Delta_c=\sqrt{17c^2+36c+4} Δ c = 17 c 2 + 36 c + 4 , r ± = ( 5 c + 2 ± Δ c ) / ( 2 c − 4 ) r_\pm=(5c+2\pm\Delta_c)/(2c-4) r ± = ( 5 c + 2 ± Δ c ) / ( 2 c − 4 ) and η t = ( R t / R T ) − Δ c / ( c + 2 ) \eta_t=(R_t/R_T)^{-\Delta_c/(c+2)} η t = ( R t / R T ) − Δ c / ( c + 2 ) . The resulting optimal value function is
J ( t , x , m ) = exp ( 1 2 P ( t ) m 2 + Q ( t ) m + K ( t ) ) ( x − w ) 2 . (20) J(t,x,m)=\exp\left(\frac{1}{2}P(t)m^2+Q(t)m+K(t)\right)(x-w)^2. \tag*{(20)} J ( t , x , m ) = exp ( 2 1 P ( t ) m 2 + Q ( t ) m + K ( t ) ) ( x − w ) 2 . (20)
Accordingly, the optimal pair ( M ∗ , V ∗ ) (M^*, V^*) ( M ∗ , V ∗ ) is given by
V t ∗ = c c + 2 R t , M t ∗ = c + 2 R t P ( t ) c + 2 m + 2 ρ ∗ + 2 R t Q ( t ) c + 2 . (21) V_t^* = \frac{c}{c+2}R_t,\qquad M_t^* = \frac{c+2R_tP(t)}{c+2}m+\frac{2\rho^*+2R_tQ(t)}{c+2}. \tag*{(21)} V t ∗ = c + 2 c R t , M t ∗ = c + 2 c + 2 R t P ( t ) m + c + 2 2 ρ ∗ + 2 R t Q ( t ) . (21)
The case c = 2 c=2 c = 2 is degenerate as the coefficient of the nonlinear term vanishes. As a result, the preceding closed-form representation containing factors of ( c − 2 ) − 1 (c-2)^{-1} ( c − 2 ) − 1 cannot be evaluated at c = 2 c=2 c = 2 directly. Setting c = 2 c=2 c = 2 in the ODE system gives
1 2 P ˙ − 2 R t P + 1 2 = 0 , P ( T ) = 0 ; \frac{1}{2}\dot{P}-2R_tP+\frac{1}{2}=0,\qquad P(T)=0; 2 1 P ˙ − 2 R t P + 2 1 = 0 , P ( T ) = 0 ;
Q ˙ − 2 R t Q − ρ ∗ = 0 , Q ( T ) = 0 ; \dot{Q}-2R_tQ-\rho^*=0,\qquad Q(T)=0; Q ˙ − 2 R t Q − ρ ∗ = 0 , Q ( T ) = 0 ;
K ˙ + 1 2 R t 2 P + R t log 2 − ( ρ ∗ ) 2 2 = 0 , K ( T ) = 0. \dot{K}+\frac{1}{2}R_t^2P+R_t\log2-\frac{(\rho^*)^2}{2}=0,\qquad K(T)=0. K ˙ + 2 1 R t 2 P + R t log 2 − 2 ( ρ ∗ ) 2 = 0 , K ( T ) = 0.
Solving this degenerate linear system gives
P ( t ) = 1 3 R t [ 1 − ( R T R t ) 3 ] , Q ( t ) = − ρ ∗ R t ( 1 − R T R t ) , P(t)=\frac{1}{3R_t}\left[1-\left(\frac{R_T}{R_t}\right)^3\right],\qquad Q(t)=-\frac{\rho^*}{R_t}\left(1-\frac{R_T}{R_t}\right), P ( t ) = 3 R t 1 [ 1 − ( R t R T ) 3 ] , Q ( t ) = − R t ρ ∗ ( 1 − R t R T ) ,
K ( t ) = ( 1 6 + log 2 ) log R t R T + 1 18 ( R T 3 R t 3 − 1 ) + ( ρ ∗ ) 2 2 ( t − T ) . K(t)=\left(\frac{1}{6}+\log2\right)\log\frac{R_t}{R_T}+\frac{1}{18}\left(\frac{R_T^3}{R_t^3}-1\right)+\frac{(\rho^*)^2}{2}(t-T). K ( t ) = ( 6 1 + log 2 ) log R T R t + 18 1 ( R t 3 R T 3 − 1 ) + 2 ( ρ ∗ ) 2 ( t − T ) .
These expressions coincide with the limit of the solution when c ≠ 2 c\ne2 c = 2 as c → 2 c\to2 c → 2 , so c = 2 c=2 c = 2 can be regarded as the degenerate limiting case. Accordingly, the optimal pair ( M ∗ , V ∗ ) (M^*, V^*) ( M ∗ , V ∗ ) simplifies to
V t ∗ = R t 2 , M t ∗ = ( 2 3 − 1 6 R T 3 R t 3 ) m + 1 2 R T R t ρ ∗ . (22) V_t^*=\frac{R_t}{2},\qquad M_t^*=\left(\frac{2}{3}-\frac{1}{6}\frac{R_T^3}{R_t^3}\right)m+\frac{1}{2}\frac{R_T}{R_t}\rho^*. \tag*{(22)} V t ∗ = 2 R t , M t ∗ = ( 3 2 − 6 1 R t 3 R T 3 ) m + 2 1 R t R T ρ ∗ . (22)
Finally, it is useful to separate the dependence on the true Sharpe ratio ρ ∗ \rho^* ρ ∗ . For every c > 0 c>0 c > 0 , including the degenerate case c = 2 c=2 c = 2 , the value function can be written as
J ( t , x , m ) = exp { 1 2 P ( t ) m 2 + Q ‾ ( t ) ρ ∗ m + 1 2 K 1 ( t ) ( ρ ∗ ) 2 + K 2 ( t ) } ( x − w ) 2 . (23) J(t,x,m)=\exp\left\{\frac{1}{2}P(t)m^2+\overline{Q}(t)\rho^*m+\frac{1}{2}K_1(t)(\rho^*)^2+K_2(t)\right\}(x-w)^2. \tag*{(23)} J ( t , x , m ) = exp { 2 1 P ( t ) m 2 + Q ( t ) ρ ∗ m + 2 1 K 1 ( t ) ( ρ ∗ ) 2 + K 2 ( t ) } ( x − w ) 2 . (23)
Here Q ( t ) = ρ ∗ Q ‾ ( t ) Q(t)=\rho^*\overline{Q}(t) Q ( t ) = ρ ∗ Q ( t ) and K ( t ) = K 1 ( t ) ( ρ ∗ ) 2 / 2 + K 2 ( t ) K(t)=K_1(t)(\rho^*)^2/2+K_2(t) K ( t ) = K 1 ( t ) ( ρ ∗ ) 2 /2 + K 2 ( t ) . The explicit formulas obtained above for c ≠ 2 c\ne2 c = 2 and for c = 2 c=2 c = 2 are precisely the closed-form solutions corresponding to this decomposition.
We next verify the candidate is admissible. For fixed ( t , x , m ) (t,x,m) ( t , x , m ) with t < T t<T t < T , write
a s : = c + 2 R s P ( s ) c + 2 , b s : = 2 + 2 R s Q ‾ ( s ) c + 2 . a_s:=\frac{c+2R_sP(s)}{c+2},\qquad b_s:=\frac{2+2R_s\overline{Q}(s)}{c+2}. a s := c + 2 c + 2 R s P ( s ) , b s := c + 2 2 + 2 R s Q ( s ) .
Thus M s ∗ = a s m s + b s ρ ∗ M_s^*=a_sm_s+b_s\rho^* M s ∗ = a s m s + b s ρ ∗ , consistently with the actor parametrization used later.
Proposition 2. For every c > 0 c>0 c > 0 , every finite horizon T T T , and every initial state ( t , x , m ) (t,x,m) ( t , x , m ) with 0 ≤ t < T 0\le t<T 0 ≤ t < T , the Gaussian policy q s ∗ = N ( M s ∗ , V s ∗ ) q_s^*=\mathcal{N}(M_s^*,V_s^*) q s ∗ = N ( M s ∗ , V s ∗ ) in (21) , including the specialization (22) , belongs to Q ( t , x , m ) Q(t,x,m) Q ( t , x , m ) . Consequently, q ∗ q^* q ∗ is optimal and the function in (20) is the optimal value function.
The complete proof is given in Appendix § 9 , where all six admissibility conditions are verified directly on every finite horizon. Consequently, Proposition 2 gives q ∗ ∈ Q ( t , x , m ) q^*\in Q(t,x,m) q ∗ ∈ Q ( t , x , m ) , and Theorem 1 then shows that q ∗ q^* q ∗ is optimal and that the closed-form candidate in (20) is the optimal value function.
Actor-Critic Learning Algorithm and Convergence Analysis
Actor-Critic Algorithm
The filter tracks within-episode uncertainty, but the closed-form actor and value function still contain the fixed unknown ρ ∗ \rho^* ρ ∗ . We parameterize these occurrences and learn them from episodic trajectories: the critic evaluates the current policy, the actor updates the policy center, and the multiplier enforces the terminal mean constraint.
By Bellman consistency applied to (14) , for a fixed sampling policy q q q , with X X X denoting the macroscopic wealth under (11) , we have
J q ( t , x , m ) = E [ J q ( s , X s , m s ) + ∫ t s λ v J q ( v , X v , m v ) D K L ( q v ∥ N ( m v , R v ) ) d v | X t = x , m t = m ] . J^q(t,x,m)=\mathbb{E}\left[J^q(s,X_s,m_s)+\int_t^s\lambda_vJ^q(v,X_v,m_v)D_{\mathrm{KL}}\bigl(q_v\Vert\mathcal{N}(m_v,R_v)\bigr)\,\mathrm{d}v\middle|X_t=x,\ m_t=m\right]. J q ( t , x , m ) = E [ J q ( s , X s , m s ) + ∫ t s λ v J q ( v , X v , m v ) D KL ( q v ∥ N ( m v , R v ) ) d v X t = x , m t = m ] .
For the policy evaluation, we adopt the method based on martingale condition. More precisely, if J q J^q J q is the true value function associated with q q q , then the cost-adjusted process
M t q = J q ( t , X t , m t ) + ∫ 0 t λ s J q ( s , X s , m s ) D K L ( q s ∥ N ( m s , R s ) ) d s , \mathcal{M}^{q}_t = J^q(t,X_t,m_t) + \int_0^t \lambda_s J^q(s,X_s,m_s)D_{\mathrm{KL}}\left(q_s \mathbin{\|} N(m_s,R_s)\right)\,ds, M t q = J q ( t , X t , m t ) + ∫ 0 t λ s J q ( s , X s , m s ) D KL ( q s ∥ N ( m s , R s ) ) d s ,
is a martingale. Hence, for any predictable test function ξ t \xi_t ξ t satisfying E ∫ 0 T ξ t 2 d ⟨ M q ⟩ t < ∞ \mathbb{E}\int_0^T \xi_t^2\,d\langle\mathcal{M}^q\rangle_t < \infty E ∫ 0 T ξ t 2 d ⟨ M q ⟩ t < ∞ [26 ] ,
E [ ∫ 0 T ξ t d M t q ] = 0. \mathbb{E}\left[\int_0^T \xi_t\,d\mathcal{M}^q_t\right]=0. E [ ∫ 0 T ξ t d M t q ] = 0.
For policy improvement, Jia and Zhou [26 ] give the following policy-gradient representation, specialized here to an undiscounted additive KL criterion:
∂ ϕ J ( 0 , x , m ; q ϕ ) = E [ ∫ 0 T ∂ ϕ log q t ϕ ( ρ t ∣ m t , R t ) d M t ϕ + ∫ 0 T λ t ∂ ϕ D K L ( q t ϕ ∥ N ( m t , R t ) ) d t ] . \partial_\phi J(0,x,m;q^\phi)=\mathbb{E}\left[\int_0^T \partial_\phi\log q^\phi_t(\rho_t\mid m_t,R_t)\,d\mathcal{M}^\phi_t+\int_0^T\lambda_t\partial_\phi D_{\mathrm{KL}}\left(q^\phi_t\mathbin{\|}N(m_t,R_t)\right)\,dt\right]. ∂ ϕ J ( 0 , x , m ; q ϕ ) = E [ ∫ 0 T ∂ ϕ log q t ϕ ( ρ t ∣ m t , R t ) d M t ϕ + ∫ 0 T λ t ∂ ϕ D KL ( q t ϕ ∥ N ( m t , R t ) ) d t ] .
Here M t ϕ \mathcal{M}^\phi_t M t ϕ is the policy value plus accumulated running cost along the sampled trajectory in the additive model. We next adapt these principles to normalized actor–critic updates.
For the sampled-action updates, fix one episode and write X t X_t X t for the microscopic wealth generated by the realized draw ρ t \rho_t ρ t . Denote by J θ J^\theta J θ and q ϕ q^\phi q ϕ , respectively, the parameterized value function and sampling distribution. Along this trajectory, define the semimartingale residual
d R t θ , ϕ : = d J θ ( t , X t , m t ) + λ t J θ ( t , X t , m t ) D K L ( q t ϕ ∥ N ( m t , R t ) ) d t . (24) d\mathcal{R}^{\theta,\phi}_t := dJ^\theta(t,X_t,m_t)+\lambda_tJ^\theta(t,X_t,m_t)D_{\mathrm{KL}}\left(q^\phi_t\mathbin{\|}N(m_t,R_t)\right)\,dt. \tag*{(24)} d R t θ , ϕ := d J θ ( t , X t , m t ) + λ t J θ ( t , X t , m t ) D KL ( q t ϕ ∥ N ( m t , R t ) ) d t . (24)
This differential is the primitive object used in both the critic and actor updates.
To approximate the residual in an implementable algorithm, we discretize [ 0 , T ] [0,T] [ 0 , T ] into equal-length intervals 0 = t 0 < t 1 < ⋯ < t K = T 0=t_0<t_1<\cdots<t_K=T 0 = t 0 < t 1 < ⋯ < t K = T , with t i = i Δ t t_i=i\Delta t t i = i Δ t and Δ t = T / K \Delta t=T/K Δ t = T / K . Along one episode, we collect samples D = { ( t i , x i , m i , R i ) } i = 0 K D=\{(t_i,x_i,m_i,R_i)\}_{i=0}^K D = {( t i , x i , m i , R i ) } i = 0 K . At time t i t_i t i , a Sharpe-ratio sample ρ i \rho_i ρ i is drawn from q t i ϕ ( ⋅ ∣ m i , R i ) q^\phi_{t_i}(\cdot\mid m_i,R_i) q t i ϕ ( ⋅ ∣ m i , R i ) , and the corresponding risky allocation is u i = − ρ i ( x i − w ) / σ ^ u_i=-\rho_i(x_i-w)/\widehat{\sigma} u i = − ρ i ( x i − w ) / σ . The market simulator then generates the next wealth x i + 1 x_{i+1} x i + 1 , while the posterior state is updated using the observed return increment. Define the one-step residual increment
Δ R i θ , ϕ = J θ ( t i + 1 , x i + 1 , m i + 1 ) − J θ ( t i , x i , m i ) + λ t i J θ ( t i , x i , m i ) D K L ( q t i ϕ ∥ N ( m i , R i ) ) Δ t . \Delta\mathcal{R}^{\theta,\phi}_i=J^\theta(t_{i+1},x_{i+1},m_{i+1})-J^\theta(t_i,x_i,m_i)+\lambda_{t_i}J^\theta(t_i,x_i,m_i)D_{\mathrm{KL}}\left(q^\phi_{t_i}\mathbin{\|}N(m_i,R_i)\right)\Delta t. Δ R i θ , ϕ = J θ ( t i + 1 , x i + 1 , m i + 1 ) − J θ ( t i , x i , m i ) + λ t i J θ ( t i , x i , m i ) D KL ( q t i ϕ ∥ N ( m i , R i ) ) Δ t .
This is the finite-grid approximation of the increment of (24) over [ t i , t i + 1 ] [t_i,t_{i+1}] [ t i , t i + 1 ] .
We now specify the structured function approximators. In order to facilitate faster learning and convergence, we follow the structural form suggested by the closed-form solution in the previous section. According to Equation (21) , we focus on a Gaussian sampling distribution whose mean is affine in the posterior mean m t m_t m t and in the learnable actor parameter ϕ \phi ϕ . Specifically, define
q t ϕ ( ρ ∣ m t , R t ) = N ( ρ ∣ M ϕ ( t , m t , R t ) , V t , q^\phi_t(\rho\mid m_t,R_t)=N(\rho\mathbin{|}M^\phi(t,m_t,R_t),V_t, q t ϕ ( ρ ∣ m t , R t ) = N ( ρ ∣ M ϕ ( t , m t , R t ) , V t ,
where
M ϕ ( t , m t , R t ) = c + 2 R t P ( t ) c + 2 m t + 2 + 2 R t Q ‾ ( t ) c + 2 ϕ , V t = c R t c + 2 . (25) M^\phi(t,m_t,R_t)=\frac{c+2R_tP(t)}{c+2}m_t+\frac{2+2R_t\overline{Q}(t)}{c+2}\phi,\qquad V_t=\frac{cR_t}{c+2}. \tag*{(25)} M ϕ ( t , m t , R t ) = c + 2 c + 2 R t P ( t ) m t + c + 2 2 + 2 R t Q ( t ) ϕ , V t = c + 2 c R t . (25)
Here ϕ \phi ϕ is a learnable proxy for the unknown oracle Sharpe ratio ρ ∗ \rho^* ρ ∗ in the actor. For later use, write M t ϕ : = M ϕ ( t , m t , R t ) M^\phi_t:=M^\phi(t,m_t,R_t) M t ϕ := M ϕ ( t , m t , R t ) . The parameter ϕ \phi ϕ is not assumed to equal ρ ∗ \rho^* ρ ∗ during learning.
Similarly, motivated by Equation (23) , we parameterize the critic by
J θ ( t , x , m ) = exp { 1 2 P ( t ) m 2 + Q ‾ ( t ) θ m + 1 2 K 1 ( t ) θ 2 + K 2 ( t ) } ( x − w ) 2 . J^\theta(t,x,m)=\exp\left\{\frac{1}{2}P(t)m^2+\overline{Q}(t)\theta m+\frac{1}{2}K_1(t)\theta^2+K_2(t)\right\}(x-w)^2. J θ ( t , x , m ) = exp { 2 1 P ( t ) m 2 + Q ( t ) θ m + 2 1 K 1 ( t ) θ 2 + K 2 ( t ) } ( x − w ) 2 .
The critic parameter θ \theta θ is another learnable proxy for the unknown Sharpe ratio in the value-function approximation. The parameter θ \theta θ is also not required to equal ρ ∗ \rho^* ρ ∗ during learning.
We next construct the actor–critic estimating equations, placing both updates on the same normalized-residual scale. Although the martingale-orthogonality framework of Jia and Zhou [26 ] permits various predictable test functions and a common gradient-type choice for the critic is ∂ θ J θ \partial_\theta J^\theta ∂ θ J θ , we instead adopt a model-specific normalized test function
χ θ ( t , x , m ) : = 1 J θ ( t , x , m ) 2 ∂ J θ ∂ θ ( t , x , m ) = 1 J θ ( t , x , m ) ( Q ‾ ( t ) m + K 1 ( t ) θ ) . \chi^\theta(t,x,m) := \frac{1}{J^\theta(t,x,m)^2}\frac{\partial J^\theta}{\partial\theta}(t,x,m) = \frac{1}{J^\theta(t,x,m)}\left(\overline{Q}(t)m+K_1(t)\theta\right). χ θ ( t , x , m ) := J θ ( t , x , m ) 2 1 ∂ θ ∂ J θ ( t , x , m ) = J θ ( t , x , m ) 1 ( Q ( t ) m + K 1 ( t ) θ ) .
Indeed, as ∂ θ J θ = J θ ∂ θ log J θ \partial_\theta J^\theta= J^\theta\partial_\theta\log J^\theta ∂ θ J θ = J θ ∂ θ log J θ , its estimating increment satisfies
χ θ ( t , X t , m t ) d R t θ , ϕ = ∂ θ log J θ ( t , X t , m t ) d R t θ , ϕ J θ ( t , X t , m t ) . \chi^\theta(t,X_t,m_t)dR_t^{\theta,\phi}=\partial_\theta\log J^\theta(t,X_t,m_t)\frac{dR_t^{\theta,\phi}}{J^\theta(t,X_t,m_t)}. χ θ ( t , X t , m t ) d R t θ , ϕ = ∂ θ log J θ ( t , X t , m t ) J θ ( t , X t , m t ) d R t θ , ϕ .
To see its effect, according to (86) in Appendix § 11 , write the normalized residual decomposition schematically as
d R t θ , ϕ = J θ ( t , X t , m t ) ( G t ρ ( n ) , θ , ϕ d t + Γ t ρ ( n ) , θ , ϕ d W t ) . dR_t^{\theta,\phi}=J^\theta(t,X_t,m_t)\left(G_t^{\rho(n),\theta,\phi}dt+\Gamma_t^{\rho(n),\theta,\phi}dW_t\right). d R t θ , ϕ = J θ ( t , X t , m t ) ( G t ρ ( n ) , θ , ϕ d t + Γ t ρ ( n ) , θ , ϕ d W t ) .
As ∂ θ J θ = J θ ∂ θ log J θ \partial_\theta J^\theta=J^\theta\partial_\theta\log J^\theta ∂ θ J θ = J θ ∂ θ log J θ , the unnormalized choice ∂ θ J θ \partial_\theta J^\theta ∂ θ J θ would produce
∂ θ J θ d R t θ , ϕ = ( J θ ) 2 ∂ θ log J θ ( G t ρ ( n ) , θ , ϕ d t + Γ t ρ ( n ) , θ , ϕ d W t ) . \partial_\theta J^\theta dR_t^{\theta,\phi}=(J^\theta)^2\partial_\theta\log J^\theta\left(G_t^{\rho(n),\theta,\phi}dt+\Gamma_t^{\rho(n),\theta,\phi}dW_t\right). ∂ θ J θ d R t θ , ϕ = ( J θ ) 2 ∂ θ log J θ ( G t ρ ( n ) , θ , ϕ d t + Γ t ρ ( n ) , θ , ϕ d W t ) .
Because ( J θ ) 2 = ( A θ ) 2 ( X t − w ) 4 (J^\theta)^2=(A^\theta)^2(X_t-w)^4 ( J θ ) 2 = ( A θ ) 2 ( X t − w ) 4 , this moment heavily weights trajectories with large wealth deviations and nearly suppresses critic learning when X t X_t X t is close to w w w . By contrast, the proposed test function gives
∂ θ J θ ( J θ ) 2 d R t θ , ϕ = ∂ θ log J θ ( G t ρ ( n ) , θ , ϕ d t + Γ t ρ ( n ) , θ , ϕ d W t ) , \frac{\partial_\theta J^\theta}{(J^\theta)^2}dR_t^{\theta,\phi}=\partial_\theta\log J^\theta\left(G_t^{\rho(n),\theta,\phi}dt+\Gamma_t^{\rho(n),\theta,\phi}dW_t\right), ( J θ ) 2 ∂ θ J θ d R t θ , ϕ = ∂ θ log J θ ( G t ρ ( n ) , θ , ϕ d t + Γ t ρ ( n ) , θ , ϕ d W t ) ,
so the homogeneous wealth factor cancels. The critic direction is consequently invariant under a common multiplicative rescaling of the value function and is less sensitive to the absolute wealth scale. The actor normalization by ( J θ ) − 1 (J^\theta)^{-1} ( J θ ) − 1 performs the analogous cancellation, placing both updates on the common relative-residual scale d R t θ , ϕ / J θ dR_t^{\theta,\phi}/J^\theta d R t θ , ϕ / J θ . These normalizations are therefore designed to improve scale stability in the present homogeneous mean–variance model.
In continuous time, the corresponding normalized critic estimating direction is
H ^ ( θ , ϕ ) : = ∫ 0 T χ θ ( t , X t , m t ) d R t θ , ϕ = ∫ 0 T ∂ θ J θ ( t , X t , m t ) [ J θ ( t , X t , m t ) ] 2 d R t θ , ϕ . (26) \widehat{H}(\theta,\phi):=\int_0^T\chi^\theta(t,X_t,m_t)dR_t^{\theta,\phi}=\int_0^T\frac{\partial_\theta J^\theta(t,X_t,m_t)}{[J^\theta(t,X_t,m_t)]^2}dR_t^{\theta,\phi}. \tag*{(26)} H ( θ , ϕ ) := ∫ 0 T χ θ ( t , X t , m t ) d R t θ , ϕ = ∫ 0 T [ J θ ( t , X t , m t ) ] 2 ∂ θ J θ ( t , X t , m t ) d R t θ , ϕ . (26)
The finite-grid approximation of (26) gives the policy-evaluation update
θ ← θ + α θ ( n ) ∑ i χ θ ( t i , x i , m i ) Δ R i θ , ϕ . (27) \theta\leftarrow\theta+\alpha_\theta(n)\sum_i\chi^\theta(t_i,x_i,m_i)\Delta R_i^{\theta,\phi}. \tag*{(27)} θ ← θ + α θ ( n ) i ∑ χ θ ( t i , x i , m i ) Δ R i θ , ϕ . (27)
where α θ ( n ) \alpha_\theta(n) α θ ( n ) is the learning rate for the parameter θ \theta θ at the n n n -th iteration.
For the actor, the offline policy-gradient representation motivates weighting the same normalized residual by the policy score. Since q t ϕ q_t^\phi q t ϕ is Gaussian with mean M t ϕ M_t^\phi M t ϕ and variance V t V_t V t , its score is
∂ log q t ϕ ∂ ϕ ( ρ ∣ m t , R t ) = ρ − M t ϕ V t ⋅ 2 + 2 R t Q ‾ ( t ) c + 2 . \frac{\partial\log q_t^\phi}{\partial\phi}(\rho\mid m_t,R_t)=\frac{\rho-M_t^\phi}{V_t}\cdot\frac{2+2R_t\overline{Q}(t)}{c+2}. ∂ ϕ ∂ log q t ϕ ( ρ ∣ m t , R t ) = V t ρ − M t ϕ ⋅ c + 2 2 + 2 R t Q ( t ) .
The derivative of the KL-divergence term is
∂ ∂ ϕ D K L ( q t ϕ ∥ N ( m t , R t ) ) = M t ϕ − m t R t ⋅ 2 + 2 R t Q ‾ ( t ) c + 2 . \frac{\partial}{\partial\phi}D_{\mathrm{KL}}\left(q_t^\phi\mathrel{\|}N(m_t,R_t)\right)=\frac{M_t^\phi-m_t}{R_t}\cdot\frac{2+2R_t\overline{Q}(t)}{c+2}. ∂ ϕ ∂ D KL ( q t ϕ ∥ N ( m t , R t ) ) = R t M t ϕ − m t ⋅ c + 2 2 + 2 R t Q ( t ) .
In continuous time, the corresponding normalized actor estimating direction is
F ^ ( ϕ , θ ) = ∫ 0 T ∂ log q t ϕ ∂ ϕ ( ρ t ∣ m t , R t ) d R t θ , ϕ J θ ( t , X t , m t ) + ∫ 0 T λ t ∂ ∂ ϕ D K L ( q t ϕ ∥ N ( m t , R t ) ) d t . \widehat{F}(\phi,\theta)=\int_0^T\frac{\partial\log q_t^\phi}{\partial\phi}(\rho_t\mid m_t,R_t)\frac{dR_t^{\theta,\phi}}{J^\theta(t,X_t,m_t)}+\int_0^T\lambda_t\frac{\partial}{\partial\phi}D_{\mathrm{KL}}\left(q_t^\phi\mathrel{\|}N(m_t,R_t)\right)dt. F ( ϕ , θ ) = ∫ 0 T ∂ ϕ ∂ log q t ϕ ( ρ t ∣ m t , R t ) J θ ( t , X t , m t ) d R t θ , ϕ + ∫ 0 T λ t ∂ ϕ ∂ D KL ( q t ϕ ∥ N ( m t , R t ) ) d t .
The two normalizations are well-defined under the continuous-time recursion used in the theory. For x 0 ≠ w x_0\ne w x 0 = w , the sampled-action wealth deviation satisfies
X t − w = ( x 0 − w ) exp { − ∫ 0 t ( ρ s ∗ ρ s + 1 2 ρ s 2 ) d s − ∫ 0 t ρ s d W s } . X_t-w=(x_0-w)\exp\left\{-\int_0^t\left(\rho_s^*\rho_s+\frac{1}{2}\rho_s^2\right)ds-\int_0^t\rho_s\,dW_s\right\}. X t − w = ( x 0 − w ) exp { − ∫ 0 t ( ρ s ∗ ρ s + 2 1 ρ s 2 ) d s − ∫ 0 t ρ s d W s } .
and therefore never reaches zero. Since J θ = A θ ( t , m ) ( X t − w ) 2 J^\theta=A^\theta(t,m)(X_t-w)^2 J θ = A θ ( t , m ) ( X t − w ) 2 with A θ > 0 A^\theta>0 A θ > 0 , we have J θ ( t , X t , m t ) > 0 J^\theta(t,X_t,m_t)>0 J θ ( t , X t , m t ) > 0 along every theoretical trajectory. The multiplier projections in Appendix B impose w > x 0 w>x_0 w > x 0 , while the moment assumptions stated above and the polynomial bounds proved in the appendix ensure the required integrability after this cancellation.
We also center the normalized residual with respect to the current action before discretization. This centering step is necessary for exploration-scale stability. Write ℓ θ ( t , m ) : = log A θ ( t , m ) \ell^\theta(t,m):=\log A^\theta(t,m) ℓ θ ( t , m ) := log A θ ( t , m ) and let β c v \beta_{\mathrm{cv}} β cv be a predictable scalar. At a sampling time, the action-independent baseline
( G t M ϕ , θ , ϕ + β c v V t ) d t + Γ t M ϕ , θ , ϕ d W t \left(G_t^{M^\phi,\theta,\phi}+\beta_{\mathrm{cv}}V_t\right)dt+\Gamma_t^{M^\phi,\theta,\phi}dW_t ( G t M ϕ , θ , ϕ + β cv V t ) d t + Γ t M ϕ , θ , ϕ d W t
has zero score-weighted conditional drift, because the conditional mean of the score is zero. The centered normalized residual becomes
d C t θ , ϕ = [ ( ρ t − M t ϕ ) ( ρ t + M t ϕ − 2 m t − 2 R t ∂ m ℓ θ ( t , m t ) ) − β c v V t ] d t − 2 ( ρ t − M t ϕ ) R t d m t . (28) dC_t^{\theta,\phi}=\left[(\rho_t-M_t^\phi)\left(\rho_t+M_t^\phi-2m_t-2R_t\partial_m\ell^\theta(t,m_t)\right)-\beta_{\mathrm{cv}}V_t\right]dt-\frac{2(\rho_t-M_t^\phi)}{R_t}dm_t. \tag*{(28)} d C t θ , ϕ = [ ( ρ t − M t ϕ ) ( ρ t + M t ϕ − 2 m t − 2 R t ∂ m ℓ θ ( t , m t ) ) − β cv V t ] d t − R t 2 ( ρ t − M t ϕ ) d m t . (28)
The corresponding continuous-time centered expression is
F ^ c ( ϕ , θ ) = ∫ 0 T ∂ log q t ϕ ∂ ϕ ( ρ t ∣ m t , R t ) d C t θ , ϕ + ∫ 0 T λ t ∂ ∂ ϕ D K L ( q t ϕ ∥ N ( m t , R t ) ) d t . (29) \widehat F^{\mathrm{c}}(\phi,\theta)=\int_0^T\frac{\partial\log q_t^\phi}{\partial\phi}(\rho_t\mid m_t,R_t)dC_t^{\theta,\phi}+\int_0^T\lambda_t\frac{\partial}{\partial\phi}D_{\mathrm{KL}}\left(q_t^\phi\middle\|N(m_t,R_t)\right)dt. \tag*{(29)} F c ( ϕ , θ ) = ∫ 0 T ∂ ϕ ∂ log q t ϕ ( ρ t ∣ m t , R t ) d C t θ , ϕ + ∫ 0 T λ t ∂ ϕ ∂ D KL ( q t ϕ N ( m t , R t ) ) d t . (29)
We take β c v = 1 \beta_{\mathrm{cv}}=1 β cv = 1 for literal conditional centering; the Gaussian score-weighted minimum-variance choice is β c v = 3 \beta_{\mathrm{cv}}=3 β cv = 3 , which leaves the population direction unchanged. Importantly, centering is performed before discretization. Centering a finite-difference TD residual afterward leaves quadratic-variation remainders, such as ( Δ W i ) 2 − Δ t (\Delta W_i)^2-\Delta t ( Δ W i ) 2 − Δ t , which are action independent but are amplified by the score as V i ↓ 0 V_i\downarrow0 V i ↓ 0 . On the grid, define
Δ C i θ , ϕ : = [ ( ρ i − M i ϕ ) ( ρ i + M i ϕ − 2 m i − 2 R i ℓ m , i θ ) − β c v V i ] Δ t − 2 ( ρ i − M i ϕ ) R i ( m i + 1 − m i ) . (30) \Delta C_i^{\theta,\phi}:=\left[(\rho_i-M_i^\phi)\left(\rho_i+M_i^\phi-2m_i-2R_i\ell_{m,i}^\theta\right)-\beta_{\mathrm{cv}}V_i\right]\Delta t-\frac{2(\rho_i-M_i^\phi)}{R_i}(m_{i+1}-m_i). \tag*{(30)} Δ C i θ , ϕ := [ ( ρ i − M i ϕ ) ( ρ i + M i ϕ − 2 m i − 2 R i ℓ m , i θ ) − β cv V i ] Δ t − R i 2 ( ρ i − M i ϕ ) ( m i + 1 − m i ) . (30)
The resulting finite-grid actor update is
ϕ ← ϕ − α ϕ ( n ) ∑ i [ ∂ log q t i ϕ ∂ ϕ ( ρ i ∣ m i , R i ) Δ C i θ , ϕ + λ t i ∂ ∂ ϕ D K L ( q t i ϕ ∥ N ( m i , R i ) ) Δ t ] . (31) \phi\leftarrow\phi-\alpha_\phi(n)\sum_i\left[\frac{\partial\log q_{t_i}^\phi}{\partial\phi}(\rho_i\mid m_i,R_i)\Delta C_i^{\theta,\phi}+\lambda_{t_i}\frac{\partial}{\partial\phi}D_{\mathrm{KL}}\left(q_{t_i}^\phi\middle\|N(m_i,R_i)\right)\Delta t\right]. \tag*{(31)} ϕ ← ϕ − α ϕ ( n ) i ∑ [ ∂ ϕ ∂ log q t i ϕ ( ρ i ∣ m i , R i ) Δ C i θ , ϕ + λ t i ∂ ϕ ∂ D KL ( q t i ϕ N ( m i , R i ) ) Δ t ] . (31)
Here α ϕ ( n ) \alpha_\phi(n) α ϕ ( n ) is the actor learning rate. The action-independent KL drift cancels in the centered residual, while the explicit KL derivative is retained.
Our construction thus adopts the martingale-orthogonality and score-weighted policy-gradient principles of Jia and Zhou [26 ] , but does not use their estimating directions verbatim. The factor ( J θ ) − 2 (J^\theta)^{-2} ( J θ ) − 2 in the critic, together with the factor ( J θ ) − 1 (J^\theta)^{-1} ( J θ ) − 1 and the centering procedure in the actor, alters both the conditional means of the critic and actor estimating directions and the associated noise terms. Consequently, the convergence analysis cannot be obtained by directly transplanting an existing actor–critic proof. Instead, Appendix B rederives the conditional mean actor direction F n ( ϕ , θ ) F_n(\phi,\theta) F n ( ϕ , θ ) defined in (92) and establishes conditional second-moment bounds for the sampled actor estimating direction, while explicitly retaining the cross-variation between the wealth process and the filter.
Finally, we provide a scheme for learning the Lagrange multiplier w w w . The terminal mean constraint E [ X T ∗ ] = z \mathbb{E}[X_T^*]=z E [ X T ∗ ] = z suggests the stochastic approximation update
w r + 1 = w r − α w ( r ) ( X ‾ T , r + 1 − z ) , w_{r+1}=w_r-\alpha_w(r)(\overline{X}_{T,r+1}-z), w r + 1 = w r − α w ( r ) ( X T , r + 1 − z ) ,
where r r r indexes multiplier-update batches, α w ( r ) > 0 \alpha_w(r)>0 α w ( r ) > 0 is the corresponding learning rate, and X ‾ T , r + 1 \overline{X}_{T,r+1} X T , r + 1 is the average of the N w N_w N w terminal wealth values in batch r + 1 r+1 r + 1 . This update is statistically self-correcting. If the average terminal wealth is above the target z z z , the update decreases w w w , which reduces the subsequent aggressive allocation. If the average terminal wealth is below the target, the update increases w w w .
Algorithm 1 presents the complete discrete-time algorithm. The episodic learning scheme carries the filtering mean forward across rounds and sets the initial filtering variance by a prescribed annealing schedule:
m 0 ( n + 1 ) = m T ( n ) , R 0 ( n ) = R 0 ( n + 1 ) α , R 0 > 0 , α > 0 , n ≥ 0. (32) m_0^{(n+1)}=m_T^{(n)},\qquad R_0^{(n)}=\frac{R_0}{(n+1)^\alpha},\qquad R_0>0,\quad\alpha>0,\quad n\geq0. \tag*{(32)} m 0 ( n + 1 ) = m T ( n ) , R 0 ( n ) = ( n + 1 ) α R 0 , R 0 > 0 , α > 0 , n ≥ 0. (32)
Thus, within the n n n th episode, R t ( n ) R_t^{(n)} R t ( n ) evolves according to the filtering Riccati equation. When entering the next round, the initial variance R 0 ( n ) R_0^{(n)} R 0 ( n ) is reset according to (32). This reset separates the update of the belief mean from the schedule for the exploration intensity: the posterior mean is passed across episodes as the learned estimate, while the initial variance is specified by a deterministic annealing rule to control the exploration scale and the stochastic-approximation error.
Algorithm 1 Offline-Episodic Actor–Critic Mean-Variance Algorithm
Inputs: initial wealth x 0 x_0 x 0 , initial posterior pair ( m 0 , R 0 ) (m_0,R_0) ( m 0 , R 0 ) , horizon T T T , time step Δ t \Delta t Δ t , number of episodes N N N , number of time steps K = ⌊ T / Δ t ⌋ K=\lfloor T/\Delta t\rfloor K = ⌊ T /Δ t ⌋ , target payoff z z z , risk-free rate r r r , multiplier batch size N w N_w N w , initial learning rates α σ \alpha_\sigma α σ , α θ \alpha_\theta α θ , α ϕ \alpha_\phi α ϕ , α w \alpha_w α w and temperature parameter λ t \lambda_t λ t .
Required program: a market simulator x ′ = Market Δ t ( t , x , m , a ) x'=\operatorname{Market}_{\Delta t}(t,x,m,a) x ′ = Market Δ t ( t , x , m , a ) that takes the current time-state pair ( t , x , m ) (t,x,m) ( t , x , m ) and action a a a as inputs and generates the next wealth state x ′ x' x ′ at time t + Δ t t+\Delta t t + Δ t . The posterior state is updated from the observed return using the filtering update.
Learning procedure:
Initialize σ ^ \hat{\sigma} σ ^ , θ \theta θ , ϕ \phi ϕ , and w w w .
for episode n = 1 n=1 n = 1 to N N N do
Initialize k = 0 k=0 k = 0 . Observe the initial state x 0 ← x 0 x_0\leftarrow x_0 x 0 ← x 0 , m 0 ← m 0 m_0\leftarrow m_0 m 0 ← m 0 , R 0 ← R 0 R_0\leftarrow R_0 R 0 ← R 0 and store ( t 0 , x 0 , m 0 ) (t_0,x_0,m_0) ( t 0 , x 0 , m 0 ) .
while k < K k<K k < K do
Compute and store the test function χ θ ( t k , x k , m k ) = 1 J θ ( t k , x k , m k ) ∂ J θ ∂ θ ( t k , x k , m k ) \chi^\theta(t_k,x_k,m_k)=\frac{1}{J^\theta(t_k,x_k,m_k)}\frac{\partial J^\theta}{\partial\theta}(t_k,x_k,m_k) χ θ ( t k , x k , m k ) = J θ ( t k , x k , m k ) 1 ∂ θ ∂ J θ ( t k , x k , m k ) .
Generate a Sharpe ratio sample ρ k ∼ q t k ϕ ( ⋅ ∣ m k , R k ) \rho_k\sim q^{\phi}_{t_k}(\cdot\mid m_k,R_k) ρ k ∼ q t k ϕ ( ⋅ ∣ m k , R k ) .
Apply u k = − ρ k σ ^ ( x k − w ) u_k=-\frac{\rho_k}{\hat{\sigma}}(x_k-w) u k = − σ ^ ρ k ( x k − w ) to the market simulator and observe the new wealth state x k + 1 x_{k+1} x k + 1 .
Compute the plug-in whitened excess-return increment Δ S ~ k = 1 σ ^ ( S t k + 1 − S t k S t k − r Δ t ) \Delta\widetilde{S}_k=\frac{1}{\hat{\sigma}}\left(\frac{S_{t_{k+1}}-S_{t_k}}{S_{t_k}}-r\Delta t\right) Δ S k = σ ^ 1 ( S t k S t k + 1 − S t k − r Δ t ) .
Update the posterior state m k + 1 = m k + R k ( Δ S ~ k − m k Δ t ) m_{k+1}=m_k+R_k(\Delta\widetilde{S}_k-m_k\Delta t) m k + 1 = m k + R k ( Δ S k − m k Δ t ) , R k + 1 = R k / ( 1 + R k Δ t ) R_{k+1}=R_k/(1+R_k\Delta t) R k + 1 = R k / ( 1 + R k Δ t ) .
Store transition ( t k + 1 , x k + 1 , m k + 1 , R k + 1 , ρ k ) (t_{k+1},x_{k+1},m_{k+1},R_{k+1},\rho_k) ( t k + 1 , x k + 1 , m k + 1 , R k + 1 , ρ k ) .
Update k ← k + 1 k\leftarrow k+1 k ← k + 1 .
end while
Compute accumulated gradients:
Δ θ = ∑ i = 0 K − 1 χ θ ( t i , x i , m i ) Δ R i θ , ϕ , \Delta\theta=\sum_{i=0}^{K-1}\chi^\theta(t_i,x_i,m_i)\Delta R_i^{\theta,\phi}, Δ θ = i = 0 ∑ K − 1 χ θ ( t i , x i , m i ) Δ R i θ , ϕ ,
Δ ϕ = ∑ i = 0 K − 1 [ ∂ log q t i ϕ ∂ ϕ ( ρ i ∣ m i , R i ) Δ C i ϕ + λ t i ∂ ∂ ϕ D K L ( q t i ϕ ∥ N ( m i , R i ) ) Δ t ] . \Delta\phi=\sum_{i=0}^{K-1}\left[\frac{\partial\log q_{t_i}^{\phi}}{\partial\phi}(\rho_i\mid m_i,R_i)\Delta C_i^\phi+\lambda_{t_i}\frac{\partial}{\partial\phi}D_{\mathrm{KL}}\left(q_{t_i}^{\phi}\|N(m_i,R_i)\right)\Delta t\right]. Δ ϕ = i = 0 ∑ K − 1 [ ∂ ϕ ∂ log q t i ϕ ( ρ i ∣ m i , R i ) Δ C i ϕ + λ t i ∂ ϕ ∂ D KL ( q t i ϕ ∥ N ( m i , R i ) ) Δ t ] .
Update θ \theta θ by θ ← θ + α θ ( n − 1 ) Δ θ \theta\leftarrow\theta+\alpha_\theta(n-1)\Delta\theta θ ← θ + α θ ( n − 1 ) Δ θ .
Update ϕ \phi ϕ by ϕ ← ϕ − α ϕ ( n − 1 ) Δ ϕ \phi\leftarrow\phi-\alpha_\phi(n-1)\Delta\phi ϕ ← ϕ − α ϕ ( n − 1 ) Δ ϕ .
Store the terminal wealth X T ( n ) ← x K X_T^{(n)}\leftarrow x_K X T ( n ) ← x K .
Update w w w and σ ^ \hat{\sigma} σ ^ every N w N_w N w episodes:
if n ≡ 0 ( m o d N w ) n\equiv0\pmod{N_w} n ≡ 0 ( mod N w ) then
Update the Lagrange multiplier w ← w − α w ( n / N w − 1 ) ( 1 N w ∑ ℓ = n − N w + 1 n X T ( ℓ ) − z ) w\leftarrow w-\alpha_w(n/N_w-1)\left(\frac{1}{N_w}\sum_{\ell=n-N_w+1}^{n}X_T^{(\ell)}-z\right) w ← w − α w ( n / N w − 1 ) ( N w 1 ∑ ℓ = n − N w + 1 n X T ( ℓ ) − z ) .
Update the plug-in volatility estimate: σ ^ ← ( 1 − α σ ) σ ^ + α σ σ ^ M L E \hat{\sigma}\leftarrow(1-\alpha_\sigma)\hat{\sigma}+\alpha_\sigma\hat{\sigma}_{\mathrm{MLE}} σ ^ ← ( 1 − α σ ) σ ^ + α σ σ ^ MLE , where σ ^ M L E \hat{\sigma}_{\mathrm{MLE}} σ ^ MLE is estimated from the most recent observed returns.
end if
end for
Convergence Analysis and Sharpe Ratio
We next establish their convergence and portfolio implications. The results proceed from filtering consistency to actor and policy-center convergence, multiplier convergence, and the resulting Sharpe-ratio bound. There is an important distinction between the theoretical filter and its implementation. Equations (8) –(9) are exact when σ ∗ \sigma^* σ ∗ is known. When volatility is unknown, the standardized return uses the current estimate σ ^ n \hat{\sigma}_n σ ^ n , so ( m t ( n ) , R t ( n ) ) (m_t^{(n)},R_t^{(n)}) ( m t ( n ) , R t ( n ) ) is a plug-in filtering state. In the ideal continuous-observation model, quadratic variation identifies volatility exactly through T − 1 ⟨ log S ⟩ T = ( σ ∗ ) 2 T^{-1}\langle\log S\rangle_T=(\sigma^*)^2 T − 1 ⟨ log S ⟩ T = ( σ ∗ ) 2 . With discrete observations and no market microstructure noise, realized quadratic variation converges at rate O p ( Δ 1 / 2 ) O_p(\Delta^{1/2}) O p ( Δ 1/2 ) as the grid width Δ → 0 \Delta\to0 Δ → 0 [3 Theorem 1] . Thus the continuous-time analysis below isolates the learning mechanism from a discretization error that vanishes under infill sampling. The feasibility of this approach, along with all other proofs, is provided in the Appendix B.
We first make explicit the theoretical sampled-action dynamics corresponding to Algorithm 1. In the continuous-observation analysis, the sampled-action wealth X ~ t π \widetilde{X}_t^{\pi} X t π satisfies
d X ~ t π = − ρ t ∗ ρ t ( X ~ t π − w ) d t − ρ t ( X ~ t π − w ) d W t . (33) d\widetilde{X}_t^{\pi}=-\rho_t^*\rho_t(\widetilde{X}_t^{\pi}-w)dt-\rho_t(\widetilde{X}_t^{\pi}-w)dW_t. \tag*{(33)} d X t π = − ρ t ∗ ρ t ( X t π − w ) d t − ρ t ( X t π − w ) d W t . (33)
Since the filtering mean in (8) has diffusion coefficient R t R_t R t with respect to the same Brownian motion W W W , the sampled wealth and the filter have the explicit cross-variation
d ⟨ X ~ t π , m ⟩ t = − ρ t R t ( X ~ t π − w ) d t . d\langle\widetilde{X}^{\pi}_t,m\rangle_t=-\rho_tR_t(\widetilde{X}^{\pi}_t-w)dt. d ⟨ X t π , m ⟩ t = − ρ t R t ( X t π − w ) d t .
Two state dynamics play distinct roles in the analysis. The policy value, the HJB equation, and the portfolio moments are defined under the macroscopic relaxed-control dynamics (11) , whose coefficients are averaged over the sampling distribution. By contrast, the score-function actor statistic must retain the realized sampled action and is therefore constructed from the sampled-action microscopic dynamics.
Let F n \mathcal{F}_n F n contain the information available before that episode. Let H ^ n + 1 \widehat{H}_{n+1} H n + 1 and F ^ n + 1 c \widehat{F}^{c}_{n+1} F n + 1 c denote the episode-indexed versions of the continuous-time estimating directions (26) and (29) (with β c v = 1 \beta_{cv}=1 β c v = 1 ), respectively, evaluated along ( X t ( n ) , m t ( n ) , R t ( n ) , ρ t ( n ) ) 0 ≤ t ≤ T (X^{(n)}_t,m^{(n)}_t,R^{(n)}_t,\rho_t^{(n)})_{0\leq t\leq T} ( X t ( n ) , m t ( n ) , R t ( n ) , ρ t ( n ) ) 0 ≤ t ≤ T with parameters ( θ n , ϕ n ) (\theta_n,\phi_n) ( θ n , ϕ n ) . The projected recursions are
θ n + 1 = Π Θ [ θ n + α θ ( n ) H ^ n + 1 ( θ n , ϕ n ) ] , ϕ n + 1 = Π Φ n + 1 [ ϕ n − α ϕ ( n ) F ^ n + 1 c ( ϕ n , θ n ) ] , w r + 1 = Π W r + 1 [ w r − α w ( r ) ( X ‾ T , r + 1 − z ) ] . (34) \begin{aligned}
\theta_{n+1}&=\Pi_{\Theta}\left[\theta_n+\alpha_\theta(n)\widehat{H}_{n+1}(\theta_n,\phi_n)\right],\\
\phi_{n+1}&=\Pi_{\Phi_{n+1}}\left[\phi_n-\alpha_\phi(n)\widehat{F}^{c}_{n+1}(\phi_n,\theta_n)\right],\\
w_{r+1}&=\Pi_{\mathcal{W}_{r+1}}\left[w_r-\alpha_w(r)(\overline{X}_{T,r+1}-z)\right]. \tag*{(34)}
\end{aligned} θ n + 1 ϕ n + 1 w r + 1 = Π Θ [ θ n + α θ ( n ) H n + 1 ( θ n , ϕ n ) ] , = Π Φ n + 1 [ ϕ n − α ϕ ( n ) F n + 1 c ( ϕ n , θ n ) ] , = Π W r + 1 [ w r − α w ( r ) ( X T , r + 1 − z ) ] . (34)
Here Θ = [ − c θ , c θ ] \Theta=[-c_\theta,c_\theta] Θ = [ − c θ , c θ ] is a fixed compact interval with c θ > ∣ ρ ∗ ∣ c_\theta>|\rho^*| c θ > ∣ ρ ∗ ∣ . To make the expanding projections explicit, set
c n : = 1 ∨ [ log log ( n + e e ) ] 1 / 4 , Φ n : = [ − c n , c n ] , (35) c_n:=1\vee[\log\log(n+e^{e})]^{1/4},\qquad\Phi_n:=[-c_n,c_n], \tag*{(35)} c n := 1 ∨ [ log log ( n + e e ) ] 1/4 , Φ n := [ − c n , c n ] , (35)
and choose w ∈ ( x 0 , w ∗ ) w\in(x_0,w^*) w ∈ ( x 0 , w ∗ ) . For the batch counter r r r , set
c w , r : = w + ( 1 ∨ [ log log ( r + e e ) ] 1 / 8 ) , W r : = [ w , c w , r ] . (36) c_{w,r}:=w+\left(1\vee[\log\log(r+e^{e})]^{1/8}\right),\qquad\mathcal{W}_r:=[w,c_{w,r}]. \tag*{(36)} c w , r := w + ( 1 ∨ [ log log ( r + e e ) ] 1/8 ) , W r := [ w , c w , r ] . (36)
Thus, n n n always indexes episodes, whereas r r r indexes multiplier updates. Finally, define τ t ( n ) : = log ( R t ( n ) / R 0 ( n ) ) \tau_t^{(n)}:=\log(R_t^{(n)}/R_0^{(n)}) τ t ( n ) := log ( R t ( n ) / R 0 ( n ) ) . When the coefficient functions from § 3.2 are evaluated along R ( n ) R^{(n)} R ( n ) , denote them by P n , Q n , K 1 , n , K 2 , n P_n,Q_n,K_{1,n},K_{2,n} P n , Q n , K 1 , n , K 2 , n . Write J n θ J^\theta_n J n θ for the corresponding critic. For simplicity, we define and subsequently use L n : = 1 + log log ( n + e e ) L_n:=1+\log\log(n+e^e) L n := 1 + log log ( n + e e ) .
Assumption 3. Take constants γ 1 , γ 2 , γ 3 \gamma_1,\gamma_2,\gamma_3 γ 1 , γ 2 , γ 3 satisfying 1 / 2 < γ 1 , γ 2 , γ 3 ≤ 1 1/2<\gamma_1,\gamma_2,\gamma_3\leq1 1/2 < γ 1 , γ 2 , γ 3 ≤ 1 , and take the filtering variance decay exponent α \alpha α satisfying 0 < α ≤ 1 0<\alpha\leq1 0 < α ≤ 1 , with R 0 T > 1 / 2 R_0T>1/2 R 0 T > 1/2 when α = 1 \alpha=1 α = 1 . Take constants α θ , α ϕ , α w , β > 0 \alpha_\theta,\alpha_\phi,\alpha_w,\beta>0 α θ , α ϕ , α w , β > 0 , and let
α θ ( n ) = α θ ( n + β ) γ 1 , α ϕ ( n ) = α ϕ ( n + β ) γ 2 , α w ( r ) = α w ( ( r + 1 ) N w + β ) γ 3 . \alpha_\theta(n)=\frac{\alpha_\theta}{(n+\beta)^{\gamma_1}},\qquad\alpha_\phi(n)=\frac{\alpha_\phi}{(n+\beta)^{\gamma_2}},\qquad\alpha_w(r)=\frac{\alpha_w}{((r+1)N_w+\beta)^{\gamma_3}}. α θ ( n ) = ( n + β ) γ 1 α θ , α ϕ ( n ) = ( n + β ) γ 2 α ϕ , α w ( r ) = (( r + 1 ) N w + β ) γ 3 α w .
By the above choices of γ 1 , γ 2 , γ 3 \gamma_1,\gamma_2,\gamma_3 γ 1 , γ 2 , γ 3 and α \alpha α , we have
∑ n α θ ( n ) = ∑ n α ϕ ( n ) = ∞ , ∑ r α w ( r ) = ∞ , \sum_n\alpha_\theta(n)=\sum_n\alpha_\phi(n)=\infty,\qquad\sum_r\alpha_w(r)=\infty, n ∑ α θ ( n ) = n ∑ α ϕ ( n ) = ∞ , r ∑ α w ( r ) = ∞ ,
∑ n ( ( α θ ( n ) ) 2 + ( α ϕ ( n ) ) 2 ) < ∞ , ∑ r ( α w ( r ) ) 2 < ∞ , \sum_n\left((\alpha_\theta(n))^2+(\alpha_\phi(n))^2\right)<\infty,\qquad\sum_r(\alpha_w(r))^2<\infty, n ∑ ( ( α θ ( n ) ) 2 + ( α ϕ ( n ) ) 2 ) < ∞ , r ∑ ( α w ( r ) ) 2 < ∞ ,
∑ n ( α ϕ ( n ) ) 2 c n 4 < ∞ , ∑ r ( α w ( r ) ) 2 ( 1 + c w , r 2 ) < ∞ . (37) \sum_n(\alpha_\phi(n))^2c_n^4<\infty,\qquad\sum_r(\alpha_w(r))^2(1+c_{w,r}^2)<\infty. \tag*{(37)} n ∑ ( α ϕ ( n ) ) 2 c n 4 < ∞ , r ∑ ( α w ( r ) ) 2 ( 1 + c w , r 2 ) < ∞. (37)
Assumption 3 coordinates three distinct stochastic-approximation mechanisms. The nonsummability conditions allow the critic, actor, and multiplier to keep learning, whereas square summability controls their accumulated martingale noise. The remaining restriction 0 < α ≤ 1 0<\alpha\leq1 0 < α ≤ 1 guarantees consistency of the episodic filter under the variance-reset schedule.
Next, Lemma 1 shows that passing the posterior mean across episodes while annealing the initial variance does not create an error floor.
Lemma 1. Let e t ( n ) : = m t ( n ) − ρ ∗ e_t^{(n)}:=m_t^{(n)}-\rho^* e t ( n ) := m t ( n ) − ρ ∗ and p n : = 1 / R 0 ( n ) p_n:=1/R_0^{(n)} p n := 1/ R 0 ( n ) . Then
e 0 ( n + 1 ) = p n p n + T e 0 ( n ) + 1 p n + T W T ( n ) . (38) e_0^{(n+1)}=\frac{p_n}{p_n+T}e_0^{(n)}+\frac{1}{p_n+T}W_T^{(n)}. \tag*{(38)} e 0 ( n + 1 ) = p n + T p n e 0 ( n ) + p n + T 1 W T ( n ) . (38) Each e 0 ( n ) e_0^{(n)} e 0 ( n ) is Gaussian, sup n E ∣ e 0 ( n ) ∣ q < ∞ \sup_n\mathbb{E}|e_0^{(n)}|^q<\infty sup n E ∣ e 0 ( n ) ∣ q < ∞ for every finite q ≥ 1 q\geq1 q ≥ 1 , and constants C , c α > 0 C,c_\alpha>0 C , c α > 0 exist such that
∣ E [ e 0 ( n ) ] ∣ ≤ C ( n + 1 ) − α / 2 , E [ ( e 0 ( n ) ) 2 ] ≤ C ( n + 1 ) − α , e 0 ( n ) → 0 a.s. |\mathbb{E}[e_0^{(n)}]|\leq C(n+1)^{-\alpha/2},\qquad\mathbb{E}[(e_0^{(n)})^2]\leq C(n+1)^{-\alpha},\qquad e_0^{(n)}\to0\quad\text{a.s.} ∣ E [ e 0 ( n ) ] ∣ ≤ C ( n + 1 ) − α /2 , E [( e 0 ( n ) ) 2 ] ≤ C ( n + 1 ) − α , e 0 ( n ) → 0 a.s.
Lemma 1 shows that both the filtering bias and mean-square error vanish under the variance-reset schedule. The unified bounds cover the inverse-linear endpoint α = 1 \alpha= 1 α = 1 as well as 0 < α < 1 0 < \alpha< 1 0 < α < 1 , and the mean-square rate is O ( ( n + 1 ) − α ) O((n+1)^{-\alpha}) O (( n + 1 ) − α ) . Thus the posterior component of the policy center converges to ρ ∗ \rho^* ρ ∗ with uniformly controlled moments.
The conditional mean actor direction and noise calculations are given in Appendix B. They establish a uniformly positive stabilizing actor-drift coefficient, denote its lower bound by κ c , R 0 > 0 \kappa_{c,R_0} > 0 κ c , R 0 > 0 . In particular, after the homogeneous wealth factor is cancelled by self-normalization, the conditional mean actor direction is independent of the absolute wealth scale and of the current predictable multiplier. Based on the stability analysis of the sampled actions, we obtain the following convergence result.
Theorem 4. The actor recursion ϕ n \phi_n ϕ n satisfies
ϕ n ⟶ ρ ∗ a.s. \phi_n \longrightarrow\rho^* \qquad\text{a.s.} ϕ n ⟶ ρ ∗ a.s. If β = 1 \beta= 1 β = 1 and κ c , R 0 α ϕ > max { α , 1 / 2 } \kappa_{c,R_0}\alpha_\phi> \max\{\alpha, 1/2\} κ c , R 0 α ϕ > max { α , 1/2 } , then
E ∣ ϕ n − ρ ∗ ∣ 2 ≤ C L n [ ( n + 1 ) − 2 α + ( n + 1 ) − γ 2 ] . (39) \mathbb{E}|\phi_n-\rho^*|^2 \le CL_n\left[(n+1)^{-2\alpha}+(n+1)^{-\gamma_2}\right]. \tag*{(39)} E ∣ ϕ n − ρ ∗ ∣ 2 ≤ C L n [ ( n + 1 ) − 2 α + ( n + 1 ) − γ 2 ] . (39)
Theorem 4 gives both consistency and a quantitative decomposition of the actor error. The term ( n + 1 ) − 2 α (n+1)^{-2\alpha} ( n + 1 ) − 2 α is the squared contribution of the vanishing critic-induced perturbation, whereas ( n + 1 ) − γ 2 (n+1)^{-\gamma_2} ( n + 1 ) − γ 2 is the stochastic-approximation error generated by the actor recursion. Hence, up to the slowly varying factor L n L_n L n , the mean-square convergence is governed by the slower of the two exponents 2 α 2\alpha 2 α and γ 2 \gamma_2 γ 2 . The result also shows that convergence of the actor does not require the critic parameter itself to converge to ρ ∗ \rho^* ρ ∗ : the fixed projection keeps the critic bounded, while its influence on the actor field vanishes at rate ( n + 1 ) − α (n+1)^{-\alpha} ( n + 1 ) − α .
However, actor convergence alone does not yet identify the policy actually sampled, because its mean combines the posterior center and the actor parameter. Let S ( m , ϕ ; R ) S(m,\phi;R) S ( m , ϕ ; R ) denote the theoretical Sharpe ratio generated by prior mean m m m , actor parameter ϕ \phi ϕ , and filtering covariance R R R . Define
S R n : = S ( m 0 ( n ) , ϕ n ; R ( n ) ) . SR_n := S\left(m_0^{(n)},\phi_n;R^{(n)}\right). S R n := S ( m 0 ( n ) , ϕ n ; R ( n ) ) .
Denote the two weights in the zero filtering-variance limit by a c : = c c + 2 a_c := \frac{c}{c+2} a c := c + 2 c , b c : = 2 c + 2 b_c := \frac{2}{c+2} b c := c + 2 2 , and a c + b c = 1 a_c+b_c=1 a c + b c = 1 . In a finite episode, the sum of the two weights in the actual policy mean M t ϕ n = a t ( n ) m t ( n ) + b t ( n ) ϕ n M_t^{\phi_n}=a_t^{(n)}m_t^{(n)}+b_t^{(n)}\phi_n M t ϕ n = a t ( n ) m t ( n ) + b t ( n ) ϕ n in (42) is generally not equal to one, but under the decaying variance rule these two weights converge respectively to a c = c / ( c + 2 ) a_c=c/(c+2) a c = c / ( c + 2 ) and b c = 2 / ( c + 2 ) b_c=2/(c+2) b c = 2/ ( c + 2 ) , and their sum converges to one. The next result closes this gap.
Theorem 5. Under the conditions of Theorem 4, uniformly over t ∈ [ 0 , T ] t\in[0,T] t ∈ [ 0 , T ] ,
∣ a t ( n ) − a c ∣ + ∣ b t ( n ) − b c ∣ = O ( ( n + 1 ) − α ) , (40) |a_t^{(n)}-a_c|+|b_t^{(n)}-b_c|=O((n+1)^{-\alpha}), \tag*{(40)} ∣ a t ( n ) − a c ∣ + ∣ b t ( n ) − b c ∣ = O (( n + 1 ) − α ) , (40) and
E ∣ M t ϕ n − ρ ∗ ∣ 2 ≤ C L n [ ( n + 1 ) − α + ( n + 1 ) − γ 2 ] . (41) \mathbb{E}|M_t^{\phi_n}-\rho^*|^2 \le CL_n\left[(n+1)^{-\alpha}+(n+1)^{-\gamma_2}\right]. \tag*{(41)} E ∣ M t ϕ n − ρ ∗ ∣ 2 ≤ C L n [ ( n + 1 ) − α + ( n + 1 ) − γ 2 ] . (41) Hence, E [ M t ϕ n ] → ρ ∗ \mathbb{E}[M_t^{\phi_n}]\to\rho^* E [ M t ϕ n ] → ρ ∗ and Var ( M t ϕ n ) → 0 \operatorname{Var}(M_t^{\phi_n})\to0 Var ( M t ϕ n ) → 0 .
The result implies concentration of the entire Gaussian sampling policy around the oracle Sharpe ratio. Once the sampling policy concentrates at the oracle target, it remains to enforce the terminal mean constraint. The multiplier recursion operates on the slower batch counter r r r and uses the terminal-wealth average to solve the corresponding scalar moment equation. The next theorem verifies that this additional constraint-learning step is compatible with the actor–filter limit.
Theorem 6. The multiplier recursion w r w_r w r satisfies
w r ⟶ w ∗ = z e ( ρ ∗ ) 2 T − x 0 e ( ρ ∗ ) 2 T − 1 a.s. w_r \longrightarrow w^*=\frac{ze^{(\rho^*)^2T}-x_0}{e^{(\rho^*)^2T}-1}
\qquad\text{a.s.} w r ⟶ w ∗ = e ( ρ ∗ ) 2 T − 1 z e ( ρ ∗ ) 2 T − x 0 a.s.
Theorem 6 shows that the batch recursion converges almost surely to the unique complete-information multiplier w ∗ w^* w ∗ . Consequently, the limiting policy not only has the oracle risky-asset direction but also satisfies the prescribed terminal mean target E [ X T ] = z \mathbb{E}[X_T]=z E [ X T ] = z .
The preceding results establish convergence of every component needed to implement the portfolio, but parameter convergence alone does not quantify economic performance. We therefore pass from the learned filter, actor, and policy center to the induced terminal-wealth moments and Sharpe ratio.
For this calculation, suppress the episode superscript and consider a fixed actor parameter ϕ \phi ϕ and a deterministic covariance path R = { R t : 0 ≤ t ≤ T } R = \{R_t : 0 \le t \le T\} R = { R t : 0 ≤ t ≤ T } . The mean and variance of the sampling distribution are
M t ϕ : = a t m t + b t ϕ , V t = c R t c + 2 . (42) M_t^\phi:= a_t m_t + b_t\phi,\qquad V_t = \frac{cR_t}{c+2}. \tag*{(42)} M t ϕ := a t m t + b t ϕ , V t = c + 2 c R t . (42)
The actor parameter ϕ \phi ϕ is kept fixed throughout this subsection. Let Y t π = X t π − w Y_t^\pi= X_t^\pi- w Y t π = X t π − w . Then (11) becomes
d Y t π = − ρ ∗ M t ϕ Y t π d t − M t ϕ Y t π d W t − V t Y t π d B t . dY_t^\pi= -\rho^*M_t^\phi Y_t^\pi\,dt - M_t^\phi Y_t^\pi\,dW_t - \sqrt{V_t}Y_t^\pi\,dB_t. d Y t π = − ρ ∗ M t ϕ Y t π d t − M t ϕ Y t π d W t − V t Y t π d B t .
Assume x 0 ≠ w x_0 \ne w x 0 = w . Define G ( t , m ) : = E [ Y T π / Y t π ∣ m t = m ] G(t,m) := \mathbb{E}\left[Y_T^\pi/Y_t^\pi\mid m_t=m\right] G ( t , m ) := E [ Y T π / Y t π ∣ m t = m ] . Then
E [ Y T π ∣ Y 0 π = x 0 − w , m 0 ] = ( x 0 − w ) G ( 0 , m 0 ) . \mathbb{E}\left[Y_T^\pi\mid Y_0^\pi=x_0-w,m_0\right]=(x_0-w)G(0,m_0). E [ Y T π ∣ Y 0 π = x 0 − w , m 0 ] = ( x 0 − w ) G ( 0 , m 0 ) .
By the Feynman–Kac formula, G ( t , m ) G(t,m) G ( t , m ) satisfies
G t + 1 2 R t 2 G m m + R t ( ρ ∗ − m − M t ϕ ) G m − ρ ∗ M t ϕ G = 0 , G ( T , m ) = 1. G_t+\frac{1}{2}R_t^2G_{mm}+R_t(\rho^*-m-M_t^\phi)G_m-\rho^*M_t^\phi G=0,\qquad G(T,m)=1. G t + 2 1 R t 2 G mm + R t ( ρ ∗ − m − M t ϕ ) G m − ρ ∗ M t ϕ G = 0 , G ( T , m ) = 1.
As M t ϕ = a t m + b t ϕ M_t^\phi=a_tm+b_t\phi M t ϕ = a t m + b t ϕ when the state variable is m m m , this PDE can be written as
G t + 1 2 R t 2 G m m + R t [ ρ ∗ − ( 1 + a t ) m − b t ϕ ] G m − ρ ∗ ( a t m + b t ϕ ) G = 0 , G ( T , m ) = 1. G_t+\frac{1}{2}R_t^2G_{mm}+R_t\left[\rho^*-(1+a_t)m-b_t\phi\right]G_m-\rho^*(a_tm+b_t\phi)G=0,\qquad G(T,m)=1. G t + 2 1 R t 2 G mm + R t [ ρ ∗ − ( 1 + a t ) m − b t ϕ ] G m − ρ ∗ ( a t m + b t ϕ ) G = 0 , G ( T , m ) = 1.
We use the exponential-affine ansatz
G ( t , m ) = exp { p G ( t ) m + q G ( t ) } . G(t,m)=\exp\{p_G(t)m+q_G(t)\}. G ( t , m ) = exp { p G ( t ) m + q G ( t )} .
Substituting this ansatz into the PDE and matching the coefficients of m m m and the constant terms gives
p G ′ ( t ) − R t ( 1 + a t ) p G ( t ) − ρ ∗ a t = 0 , p G ( T ) = 0 , p_G'(t)-R_t(1+a_t)p_G(t)-\rho^*a_t=0,\qquad p_G(T)=0, p G ′ ( t ) − R t ( 1 + a t ) p G ( t ) − ρ ∗ a t = 0 , p G ( T ) = 0 ,
q G ′ ( t ) + 1 2 R t 2 p G 2 ( t ) + R t ( ρ ∗ − b t ϕ ) p G ( t ) − ρ ∗ b t ϕ = 0 , q G ( T ) = 0. q_G'(t)+\frac{1}{2}R_t^2p_G^2(t)+R_t(\rho^*-b_t\phi)p_G(t)-\rho^*b_t\phi=0,\qquad q_G(T)=0. q G ′ ( t ) + 2 1 R t 2 p G 2 ( t ) + R t ( ρ ∗ − b t ϕ ) p G ( t ) − ρ ∗ b t ϕ = 0 , q G ( T ) = 0.
Solving the above linear equations backward yields
p G ( t ) = − ∫ t T exp { − ∫ t s R u ( 1 + a u ) d u } ρ ∗ a s d s , p_G(t)=-\int_t^T\exp\left\{-\int_t^sR_u(1+a_u)\,du\right\}\rho^*a_s\,ds, p G ( t ) = − ∫ t T exp { − ∫ t s R u ( 1 + a u ) d u } ρ ∗ a s d s ,
q G ( t ) = ∫ t T [ 1 2 R s 2 p G 2 ( s ) + R s ( ρ ∗ − b s ϕ ) p G ( s ) − ρ ∗ b s ϕ ] d s . q_G(t)=\int_t^T\left[\frac{1}{2}R_s^2p_G^2(s)+R_s(\rho^*-b_s\phi)p_G(s)-\rho^*b_s\phi\right]\,ds. q G ( t ) = ∫ t T [ 2 1 R s 2 p G 2 ( s ) + R s ( ρ ∗ − b s ϕ ) p G ( s ) − ρ ∗ b s ϕ ] d s .
Therefore, G ( 0 , m 0 ) = exp { p G ( 0 ) m 0 + q G ( 0 ) } G(0,m_0)=\exp\{p_G(0)m_0+q_G(0)\} G ( 0 , m 0 ) = exp { p G ( 0 ) m 0 + q G ( 0 )} , and hence
E [ X T π ] = w + ( x 0 − w ) exp { p G ( 0 ) m 0 + q G ( 0 ) } . \mathbb{E}[X_T^\pi]=w+(x_0-w)\exp\{p_G(0)m_0+q_G(0)\}. E [ X T π ] = w + ( x 0 − w ) exp { p G ( 0 ) m 0 + q G ( 0 )} .
We next compute the second moment of Y T π Y_T^\pi Y T π . Define H ( t , m ) : = E [ ( Y T π ) 2 / ( Y t π ) 2 ∣ m t = m ] H(t,m):=\mathbb{E}\left[(Y_T^\pi)^2/(Y_t^\pi)^2\mid m_t=m\right] H ( t , m ) := E [ ( Y T π ) 2 / ( Y t π ) 2 ∣ m t = m ] . Then
E [ ( Y T π ) 2 ∣ Y 0 π = x 0 − w , m 0 ] = ( x 0 − w ) 2 H ( 0 , m 0 ) . \mathbb{E}\left[(Y_T^\pi)^2\mid Y_0^\pi=x_0-w,m_0\right]=(x_0-w)^2H(0,m_0). E [ ( Y T π ) 2 ∣ Y 0 π = x 0 − w , m 0 ] = ( x 0 − w ) 2 H ( 0 , m 0 ) .
By the Feynman–Kac formula, H ( t , m ) H(t,m) H ( t , m ) satisfies
H t + 1 2 R t 2 H m m + R t [ ρ ∗ − ( 1 + 2 a t ) m − 2 b t ϕ ] H m + [ ( a t m + b t ϕ ) 2 + V t − 2 ρ ∗ ( a t m + b t ϕ ) ] H = 0 , H ( T , m ) = 1. H_t+\frac{1}{2}R_t^2H_{mm}+R_t[\rho^*-(1+2a_t)m-2b_t\phi]H_m+\left[(a_tm+b_t\phi)^2+V_t-2\rho^*(a_tm+b_t\phi)\right]H=0,\qquad H(T,m)=1. H t + 2 1 R t 2 H mm + R t [ ρ ∗ − ( 1 + 2 a t ) m − 2 b t ϕ ] H m + [ ( a t m + b t ϕ ) 2 + V t − 2 ρ ∗ ( a t m + b t ϕ ) ] H = 0 , H ( T , m ) = 1.
We use the exponential-quadratic ansatz
H ( t , m ) = exp { 1 2 A H ( t ) m 2 + B H ( t ) m + C H ( t ) } . H(t,m)=\exp\left\{\frac{1}{2}A_H(t)m^2+B_H(t)m+C_H(t)\right\}. H ( t , m ) = exp { 2 1 A H ( t ) m 2 + B H ( t ) m + C H ( t ) } .
Substituting this ansatz into the PDE and matching the coefficients of m 2 m^2 m 2 , m m m , and the constant terms gives
A H ′ ( t ) + R t 2 A H 2 ( t ) − 2 R t ( 1 + 2 a t ) A H ( t ) + 2 a t 2 = 0 , A H ( T ) = 0 , A_H'(t)+R_t^2A_H^2(t)-2R_t(1+2a_t)A_H(t)+2a_t^2=0,\qquad A_H(T)=0, A H ′ ( t ) + R t 2 A H 2 ( t ) − 2 R t ( 1 + 2 a t ) A H ( t ) + 2 a t 2 = 0 , A H ( T ) = 0 ,
B H ′ ( t ) + R t 2 A H ( t ) B H ( t ) + R t ( ρ ∗ − 2 b t ϕ ) A H ( t ) − R t ( 1 + 2 a t ) B H ( t ) + 2 a t b t ϕ − 2 ρ ∗ a t = 0 , B H ( T ) = 0 , B_H'(t)+R_t^2A_H(t)B_H(t)+R_t(\rho^*-2b_t\phi)A_H(t)-R_t(1+2a_t)B_H(t)+2a_tb_t\phi-2\rho^*a_t=0,\qquad B_H(T)=0, B H ′ ( t ) + R t 2 A H ( t ) B H ( t ) + R t ( ρ ∗ − 2 b t ϕ ) A H ( t ) − R t ( 1 + 2 a t ) B H ( t ) + 2 a t b t ϕ − 2 ρ ∗ a t = 0 , B H ( T ) = 0 ,
C H ′ ( t ) + 1 2 R t 2 ( A H ( t ) + B H 2 ( t ) ) + R t ( ρ ∗ − 2 b t ϕ ) B H ( t ) + b t 2 ϕ 2 + V t − 2 ρ ∗ b t ϕ = 0 , C H ( T ) = 0. C_H'(t)+\frac{1}{2}R_t^2\left(A_H(t)+B_H^2(t)\right)+R_t(\rho^*-2b_t\phi)B_H(t)+b_t^2\phi^2+V_t-2\rho^*b_t\phi=0,\qquad C_H(T)=0. C H ′ ( t ) + 2 1 R t 2 ( A H ( t ) + B H 2 ( t ) ) + R t ( ρ ∗ − 2 b t ϕ ) B H ( t ) + b t 2 ϕ 2 + V t − 2 ρ ∗ b t ϕ = 0 , C H ( T ) = 0.
The first equation is a Riccati equation. Equivalently, by introducing an auxiliary function U ( t ) U(t) U ( t ) such that
A H ( t ) = U ′ ( t ) R t 2 U ( t ) (43) A_H(t)=\frac{U'(t)}{R_t^2U(t)} \tag*{(43)} A H ( t ) = R t 2 U ( t ) U ′ ( t ) (43)
we obtain the second-order linear terminal value problem
U ′ ′ ( t ) − 4 a t R t U ′ ( t ) + 2 a t 2 R t 2 U ( t ) = 0 , U ( T ) = 1 , U ′ ( T ) = 0. U''(t)-4a_tR_tU'(t)+2a_t^2R_t^2U(t)=0,\qquad U(T)=1,\qquad U'(T)=0. U ′′ ( t ) − 4 a t R t U ′ ( t ) + 2 a t 2 R t 2 U ( t ) = 0 , U ( T ) = 1 , U ′ ( T ) = 0.
Thus, A H ( t ) A_H(t) A H ( t ) is characterized by (43). Given A H ( t ) A_H(t) A H ( t ) , the remaining two equations are linear and admit the following backward integral representations:
B H ( t ) = ∫ t T exp { ∫ t s [ R u 2 A H ( u ) − R u ( 1 + 2 a u ) ] d u } ( R s ( ρ ∗ − 2 b s ϕ ) A H ( s ) + 2 a s b s ϕ − 2 ρ ∗ a s ) d s , B_H(t)=\int_t^T\exp\left\{\int_t^s\left[R_u^2A_H(u)-R_u(1+2a_u)\right]\,du\right\}\left(R_s(\rho^*-2b_s\phi)A_H(s)+2a_sb_s\phi-2\rho^*a_s\right)\,ds, B H ( t ) = ∫ t T exp { ∫ t s [ R u 2 A H ( u ) − R u ( 1 + 2 a u ) ] d u } ( R s ( ρ ∗ − 2 b s ϕ ) A H ( s ) + 2 a s b s ϕ − 2 ρ ∗ a s ) d s ,
C H ( t ) = ∫ t T [ 1 2 R s 2 ( A H ( s ) + B H 2 ( s ) ) + R s ( ρ ∗ − 2 b s ϕ ) B H ( s ) + b s 2 ϕ 2 + V s − 2 ρ ∗ b s ϕ ] d s . C_H(t)=\int_t^T\left[\frac{1}{2}R_s^2\left(A_H(s)+B_H^2(s)\right)+R_s(\rho^*-2b_s\phi)B_H(s)+b_s^2\phi^2+V_s-2\rho^*b_s\phi\right]\,ds. C H ( t ) = ∫ t T [ 2 1 R s 2 ( A H ( s ) + B H 2 ( s ) ) + R s ( ρ ∗ − 2 b s ϕ ) B H ( s ) + b s 2 ϕ 2 + V s − 2 ρ ∗ b s ϕ ] d s .
It follows that the second moment around the target level w w w is
E [ ( X T π − w ) 2 ] = ( x 0 − w ) 2 exp { 1 2 A H ( 0 ) m 0 2 + B H ( 0 ) m 0 + C H ( 0 ) } . \mathbb{E}\left[(X_T^\pi-w)^2\right]=(x_0-w)^2\exp\left\{\frac{1}{2}A_H(0)m_0^2+B_H(0)m_0+C_H(0)\right\}. E [ ( X T π − w ) 2 ] = ( x 0 − w ) 2 exp { 2 1 A H ( 0 ) m 0 2 + B H ( 0 ) m 0 + C H ( 0 ) } .
Noting Var ( X T π ) = Var ( Y T π ) = E [ ( Y T π ) 2 ] − E [ Y T π ] 2 \operatorname{Var}(X_T^\pi)=\operatorname{Var}(Y_T^\pi)=\mathbb{E}[(Y_T^\pi)^2]-\mathbb{E}[Y_T^\pi]^2 Var ( X T π ) = Var ( Y T π ) = E [( Y T π ) 2 ] − E [ Y T π ] 2 , we obtain
Var ( X T π ) = ( x 0 − w ) 2 [ exp { 1 2 A H ( 0 ) m 0 2 + B H ( 0 ) m 0 + C H ( 0 ) } − exp { 2 p G ( 0 ) m 0 + 2 q G ( 0 ) } ] . \operatorname{Var}(X_T^\pi)=(x_0-w)^2\left[\exp\left\{\frac{1}{2}A_H(0)m_0^2+B_H(0)m_0+C_H(0)\right\}-\exp\{2p_G(0)m_0+2q_G(0)\}\right]. Var ( X T π ) = ( x 0 − w ) 2 [ exp { 2 1 A H ( 0 ) m 0 2 + B H ( 0 ) m 0 + C H ( 0 ) } − exp { 2 p G ( 0 ) m 0 + 2 q G ( 0 )} ] .
By definition, S R = E [ X T π ] − x 0 Var ( X T π ) SR=\frac{\mathbb{E}[X_T^\pi]-x_0}{\sqrt{\operatorname{Var}(X_T^\pi)}} S R = Var ( X T π ) E [ X T π ] − x 0 , which can be expressed as
S R = ( w − x 0 ) + ( x 0 − w ) exp { p G ( 0 ) m 0 + q G ( 0 ) } ( x 0 − w ) 2 [ exp { 1 2 A H ( 0 ) m 0 2 + B H ( 0 ) m 0 + C H ( 0 ) } − exp { 2 p G ( 0 ) m 0 + 2 q G ( 0 ) } ] = sgn ( x 0 − w ) exp { p G ( 0 ) m 0 + q G ( 0 ) } − 1 exp { 1 2 A H ( 0 ) m 0 2 + B H ( 0 ) m 0 + C H ( 0 ) } − exp { 2 p G ( 0 ) m 0 + 2 q G ( 0 ) } . \begin{aligned}
SR=\frac{(w-x_0)+(x_0-w)\exp\{p_G(0)m_0+q_G(0)\}}{\sqrt{(x_0-w)^2\left[\exp\left\{\frac{1}{2}A_H(0)m_0^2+B_H(0)m_0+C_H(0)\right\}-\exp\{2p_G(0)m_0+2q_G(0)\}\right]}} \\
&=\operatorname{sgn}(x_0-w)\frac{\exp\{p_G(0)m_0+q_G(0)\}-1}{\sqrt{\exp\left\{\frac{1}{2}A_H(0)m_0^2+B_H(0)m_0+C_H(0)\right\}-\exp\{2p_G(0)m_0+2q_G(0)\}}}.
\end{aligned} S R = ( x 0 − w ) 2 [ exp { 2 1 A H ( 0 ) m 0 2 + B H ( 0 ) m 0 + C H ( 0 ) } − exp { 2 p G ( 0 ) m 0 + 2 q G ( 0 )} ] ( w − x 0 ) + ( x 0 − w ) exp { p G ( 0 ) m 0 + q G ( 0 )} = sgn ( x 0 − w ) exp { 2 1 A H ( 0 ) m 0 2 + B H ( 0 ) m 0 + C H ( 0 ) } − exp { 2 p G ( 0 ) m 0 + 2 q G ( 0 )} exp { p G ( 0 ) m 0 + q G ( 0 )} − 1 .
The preceding expression depends on the entire filtering path through the coefficient ODEs. The oracle Sharpe ratio is S R o r : = e ( ρ ∗ ) 2 T − 1 SR_{\mathrm{or}}:=\sqrt{e^{(\rho^*)^2T}-1} S R or := e ( ρ ∗ ) 2 T − 1 . When filtering variance is zero, the effective policy mean becomes the constant M ‾ c ( m , ϕ ) = ( c m + 2 ϕ ) / ( c + 2 ) \overline{M}_c(m,\phi)=(cm+2\phi)/(c+2) M c ( m , ϕ ) = ( c m + 2 ϕ ) / ( c + 2 ) , and the corresponding Sharpe ratio is
S R ∞ ( M ) = e ρ ∗ M T − 1 e M 2 T − 1 . SR_\infty(M)=\frac{e^{\rho^*MT}-1}{\sqrt{e^{M^2T}-1}}. S R ∞ ( M ) = e M 2 T − 1 e ρ ∗ M T − 1 .
This limiting expression separates parameter-learning error from the finite-R R R perturbation of the coefficient ODEs.
Lemma 2. Assume ρ ∗ ≠ 0 \rho^*\ne0 ρ ∗ = 0 and w > x 0 w>x_0 w > x 0 . It holds that
S R ∞ ( ρ ∗ ) = S R o r , S R ∞ ′ ( ρ ∗ ) = 0 , SR_\infty(\rho^*)=SR_{\mathrm{or}},\qquad SR_\infty'(\rho^*)=0, S R ∞ ( ρ ∗ ) = S R or , S R ∞ ′ ( ρ ∗ ) = 0 , and
S R ∞ ′ ′ ( ρ ∗ ) = − T e ( ρ ∗ ) 2 T ( e ( ρ ∗ ) 2 T − 1 − ( ρ ∗ ) 2 T ) ( e ( ρ ∗ ) 2 T − 1 ) 3 / 2 < 0. SR_\infty''(\rho^*)=-T\frac{e^{(\rho^*)^2T}\left(e^{(\rho^*)^2T}-1-(\rho^*)^2T\right)}{\left(e^{(\rho^*)^2T}-1\right)^{3/2}}<0. S R ∞ ′′ ( ρ ∗ ) = − T ( e ( ρ ∗ ) 2 T − 1 ) 3/2 e ( ρ ∗ ) 2 T ( e ( ρ ∗ ) 2 T − 1 − ( ρ ∗ ) 2 T ) < 0. Consequently, for ∣ M − ρ ∗ ∣ ≤ ∣ ρ ∗ ∣ / 4 |M-\rho^*|\le|\rho^*|/4 ∣ M − ρ ∗ ∣ ≤ ∣ ρ ∗ ∣/4 ,
∣ S R ∞ ( M ) − S R o r ∣ ≤ C ∣ M − ρ ∗ ∣ 2 . |SR_\infty(M)-SR_{\mathrm{or}}|\le C|M-\rho^*|^2. ∣ S R ∞ ( M ) − S R or ∣ ≤ C ∣ M − ρ ∗ ∣ 2 . Moreover, there exists δ O D E > 0 \delta_{\mathrm{ODE}}>0 δ ODE > 0 such that, on ∣ m − ρ ∗ ∣ ∨ ∣ ϕ − ρ ∗ ∣ ≤ δ O D E |m-\rho^*|\vee|\phi-\rho^*|\le\delta_{\mathrm{ODE}} ∣ m − ρ ∗ ∣ ∨ ∣ ϕ − ρ ∗ ∣ ≤ δ ODE , the limiting normalized variance factor H ( ∞ ) ( m , ϕ ) − G ( ∞ ) ( m , ϕ ) 2 H^{(\infty)}(m,\phi)-G^{(\infty)}(m,\phi)^2 H ( ∞ ) ( m , ϕ ) − G ( ∞ ) ( m , ϕ ) 2 has a positive lower bound. For this δ O D E \delta_{\mathrm{ODE}} δ ODE , there exist C < ∞ C<\infty C < ∞ and an integer n S R n_{\mathrm{SR}} n SR such that, for n ≥ n S R n\ge n_{\mathrm{SR}} n ≥ n SR ,
∣ S ( m , ϕ ; R ( n ) ) − S R ∞ ( M ‾ c ( m , ϕ ) ) ∣ ≤ C ( n + 1 ) α . (44) \left|\mathcal{S}(m,\phi;R^{(n)})-SR_\infty\left(\overline{M}_c(m,\phi)\right)\right|\le\frac{C}{(n+1)^\alpha}. \tag*{(44)} S ( m , ϕ ; R ( n ) ) − S R ∞ ( M c ( m , ϕ ) ) ≤ ( n + 1 ) α C . (44)
Lemma 2 justifies the performance-robustness mechanism. Because S R ∞ ′ ( ρ ∗ ) = 0 SR_{\infty}'(\rho^*) = 0 S R ∞ ′ ( ρ ∗ ) = 0 , the first-order policy-center error cancels and the zero-covariance Sharpe loss is locally quadratic. Hence filtering and actor estimation errors affect the limiting Sharpe ratio only through their squared magnitude. The remaining O ( n − α ) O(n^{-\alpha}) O ( n − α ) term is the first-order perturbation caused by the nonzero filtering covariance and exploration variance. Combining these two effects with Lemma 1 and Theorem 4 yields the portfolio-level guarantee below.
Theorem 7. Under conditions of Theorem 5 and Lemma 2, it holds that
∣ S R n − S R o r ∣ = O p ( L n ( n + 1 ) α + L n ( n + 1 ) γ 2 ) . (45) |SR_n-SR_{\mathrm{or}}| = O_p\left(\frac{L_n}{(n+1)^\alpha}+\frac{L_n}{(n+1)^{\gamma_2}}\right). \tag*{(45)} ∣ S R n − S R or ∣ = O p ( ( n + 1 ) α L n + ( n + 1 ) γ 2 L n ) . (45)
Theorem 7 underscores the parameter-convergence in the portfolio-performance guarantee. The term L n ( n + 1 ) − α L_n(n+1)^{-\alpha} L n ( n + 1 ) − α collects the effects of filtering uncertainty and the finite-covariance perturbation of the moment ODEs, whereas L n ( n + 1 ) − γ 2 L_n(n+1)^{-\gamma_2} L n ( n + 1 ) − γ 2 is inherited from the actor mean-square error. Consequently, the slower of α \alpha α and γ 2 \gamma_2 γ 2 determines the asymptotic Sharpe-ratio rate. Lemma 2 explains why these componentwise errors transfer in this form: the oracle is a stationary point of the limiting Sharpe ratio, so policy-center errors enter locally through their squared magnitude, while the remaining finite-covariance perturbation contributes at first order.
In a stationary market, observations from successive episodes concern the same fixed market parameter, and the natural variance-reset schedule is usually
α = 1 , R 0 ( n ) = R 0 n + 1 . \alpha= 1,\qquad R_0^{(n)}=\frac{R_0}{n+1}. α = 1 , R 0 ( n ) = n + 1 R 0 .
Take also the harmonic actor step size α ϕ ( n ) = α ϕ / ( n + 1 ) \alpha_\phi(n)=\alpha_\phi/(n+1) α ϕ ( n ) = α ϕ / ( n + 1 ) , so that γ 2 = 1 \gamma_2=1 γ 2 = 1 . Under the α = 1 \alpha=1 α = 1 case in Lemma 1 (which assumes R 0 T > 1 / 2 R_0T>1/2 R 0 T > 1/2 and gives the required O ( ( n + 1 ) − 1 ) O((n+1)^{-1}) O (( n + 1 ) − 1 ) filtering mean-square rate), and provided κ c , R 0 α ϕ > 1 \kappa_{c,R_0}\alpha_\phi>1 κ c , R 0 α ϕ > 1 , the rate in (45) becomes
∣ S R n − S R o r ∣ = O p ( log log n n ) . |SR_n-SR_{\mathrm{or}}| = O_p\left(\frac{\log\log n}{n}\right). ∣ S R n − S R or ∣ = O p ( n log log n ) .
The critic and multiplier exponents γ 1 , γ 3 ∈ ( 1 / 2 , 1 ] \gamma_1,\gamma_3\in(1/2,1] γ 1 , γ 3 ∈ ( 1/2 , 1 ] ensure their respective stochastic-approximation conditions but do not enter the displayed Sharpe-rate bound directly.
Simulation Experiments
In this section, we conduct numerical simulations to compare the performance of the proposed posterior-guided actor–critic method with several benchmark methods for the continuous-time mean–variance problem. We compare three methods: Rolling MLE, Entropy-Regularized Bayesian Filtering and the proposed Actor–Critic method. Rolling MLE represents the classical plug-in/adaptive-control benchmark. Entropy-Regularized Bayesian Filtering refers to the entropy-regularized Bayesian drift-uncertainty method, in which the posterior state ( m t , R t ) (m_t,R_t) ( m t , R t ) is updated by filtering and the portfolio allocation is sampled directly from the entropy-regularized Gaussian policy. Finally, the Actor–Critic method learns the sampling distribution through the martingale actor–critic updates developed in § 4 . Throughout the simulations, we set the weight parameter in the KL-regularized formulation to c = 2 c=2 c = 2 .
Maximum likelihood estimation
Maximum-likelihood estimation (MLE) is a standard method for estimating the parameters μ \mu μ and σ \sigma σ in the geometric Brownian motion model (1). We refer the interested reader to Campbell et al. [10 ] , Subsection 9.3.2 for a detailed description of this method. At each decision-making time t i t_i t i , the MLE estimators for μ \mu μ and σ \sigma σ are calculated based on the most recent 100 data points of the price. One can then substitute the MLE estimators into the optimal allocation (5) and the expression of the Lagrange multiplier
w = z e ρ 2 T − z 0 e ρ 2 T − 1 w=\frac{ze^{\rho^2T}-z_0}{e^{\rho^2T}-1} w = e ρ 2 T − 1 z e ρ 2 T − z 0
to compute the allocation u i ∈ R u_i\in\mathbb{R} u i ∈ R in the risky asset. Such a two-phase procedure is commonly used in adaptive control, where the first phase is identification and the second one is optimization (see, e.g., Bielecki et al. [5 ] ; Kumar and Varaiya [31 ] ). The real-time estimation procedure also allows the MLE to be applicable in nonstationary markets with time-varying μ \mu μ and σ \sigma σ .
Bayesian filtering
The Bayesian Filtering benchmark in this section refers to the entropy-regularized Bayesian drift-uncertainty method [1 ] . It combines Kalman–Bucy filtering for the unknown Sharpe ratio with entropy-regularized relaxed control. This is different from simple posterior sampling: the benchmark samples the risky allocation directly from the entropy-regularized optimal Gaussian policy.
As the posterior learning mechanism is the same as that adopted in our framework, we use the posterior mean m i m_i m i and posterior variance R i R_i R i generated by the filtering equations. At each decision-making time t i t_i t i , given the current wealth x i x_i x i and posterior state ( m i , R i ) (m_i, R_i) ( m i , R i ) , the entropy-regularized Bayesian policy is
π t i E n t = N ( u ˉ t i E n t , ( ς t i E n t ) 2 ) , (46) \pi_{t_i}^{\mathrm{Ent}} = N\left(\bar{u}_{t_i}^{\mathrm{Ent}},\left(\varsigma_{t_i}^{\mathrm{Ent}}\right)^2\right), \tag*{(46)} π t i Ent = N ( u ˉ t i Ent , ( ς t i Ent ) 2 ) , (46)
where
u ˉ t i E n t = − m i [ 1 + 2 R i α E n t ( t i ) ] σ ( x i − w ) , α E n t ( t ) = − T − t 1 + 2 R t ( T − t ) ; \bar{u}_{t_i}^{\mathrm{Ent}}=-\frac{m_i\left[1+2R_i\alpha^{\mathrm{Ent}}(t_i)\right]}{\sigma}(x_i-w),\quad\alpha^{\mathrm{Ent}}(t)=-\frac{T-t}{1+2R_t(T-t)}; u ˉ t i Ent = − σ m i [ 1 + 2 R i α Ent ( t i ) ] ( x i − w ) , α Ent ( t ) = − 1 + 2 R t ( T − t ) T − t ;
( ς t i E n t ) 2 = ε E n t 2 σ 2 A E n t ( t i , m i ) , A E n t ( t , m ) = exp { ( t − T ) m 2 1 + 2 R t ( T − t ) + 1 2 log ( 1 + R 0 t ) { 1 + R 0 ( 2 T − t ) } ( 1 + R 0 T ) 2 } . \left(\varsigma_{t_i}^{\mathrm{Ent}}\right)^2=\frac{\varepsilon_{\mathrm{Ent}}}{2\sigma^2 A^{\mathrm{Ent}}(t_i,m_i)},\quad A^{\mathrm{Ent}}(t,m)=\exp\left\{\frac{(t-T)m^2}{1+2R_t(T-t)}+\frac{1}{2}\log\frac{(1+R_0t)\{1+R_0(2T-t)\}}{(1+R_0T)^2}\right\}. ( ς t i Ent ) 2 = 2 σ 2 A Ent ( t i , m i ) ε Ent , A Ent ( t , m ) = exp { 1 + 2 R t ( T − t ) ( t − T ) m 2 + 2 1 log ( 1 + R 0 T ) 2 ( 1 + R 0 t ) { 1 + R 0 ( 2 T − t )} } .
Here ε E n t > 0 \varepsilon_{\mathrm{Ent}}>0 ε Ent > 0 is the entropy-regularization parameter. In the theoretical entropy-regularized Bayesian model, σ \sigma σ is assumed known. In our implementation, when volatility is estimated, we use the current plug-in estimate σ ^ n \hat{\sigma}_n σ ^ n in the allocation formula, held fixed within episode n n n .
At each time t i t_i t i , we sample u i E n t u_i^{\mathrm{Ent}} u i Ent and apply this sampled dollar allocation to the wealth dynamics. The posterior state is then updated from the observed return. The entropy-regularized Bayesian formulation identifies w w w as the Lagrange multiplier associated with the terminal mean constraint E [ X T ∗ ] = z \mathbb{E}[X_T^*]=z E [ X T ∗ ] = z , but it does not provide a separate numerical updating scheme for w w w . To ensure a fair comparison, we therefore update w w w using the same batch-based stochastic approximation procedure employed in our proposed method. This benchmark isolates the effect of entropy regularization under Bayesian drift uncertainty. In particular, its mean control coincides with the deterministic Bayesian Markowitz feedback, while the entropy term affects the policy variance. Therefore, it should not be described as simple Bayesian posterior sampling.
The stationary market case
We first perform numerical simulations in a stationary market environment, where the price process is simulated according to the geometric Brownian motion (1) with constant μ \mu μ and σ \sigma σ . We take T = 1 T=1 T = 1 and Δ t = 1 / 252 \Delta t=1/252 Δ t = 1/252 , indicating that the MV problem is considered over a 1-year period, with daily rebalancing. For the simulations, we select plausible values of the annualized return and volatility from the sets μ ∈ { − 0.5 , − 0.3 , − 0.1 , 0.0 , 0.1 , 0.3 , 0.5 } \mu\in\{-0.5,-0.3,-0.1,0.0,0.1,0.3,0.5\} μ ∈ { − 0.5 , − 0.3 , − 0.1 , 0.0 , 0.1 , 0.3 , 0.5 } and σ ∈ { 0.1 , 0.2 , 0.3 , 0.4 } \sigma\in\{0.1,0.2,0.3,0.4\} σ ∈ { 0.1 , 0.2 , 0.3 , 0.4 } , respectively. These values are commonly used to model a typical stock in simulation studies [25 ] . The annualized interest rate is taken to be r = 0.02 r=0.02 r = 0.02 .
We consider the problem with a 40% annualized target return on the terminal wealth starting from a normalized initial wealth x 0 = 1 x_0 = 1 x 0 = 1 , hence, z = 1.4 z = 1.4 z = 1.4 . These model parameters will be fixed for all the simulations considered in this section. For the algorithms, we take the total training episodes N = 20000 N = 20000 N = 20000 , and take the sample size N w = 10 N_w = 10 N w = 10 for learning the Lagrange multiplier w w w . Across all the simulations in this section, the learning rates are fixed at α θ = α ϕ = 0.1 \alpha_\theta= \alpha_\phi= 0.1 α θ = α ϕ = 0.1 , α σ = 0.1 \alpha_\sigma= 0.1 α σ = 0.1 and α w = 0.5 \alpha_w = 0.5 α w = 0.5 , with learning-rate schedules α = γ 1 = γ 2 = γ 3 = 0.51 \alpha= \gamma_1 = \gamma_2 = \gamma_3 = 0.51 α = γ 1 = γ 2 = γ 3 = 0.51 . The comparative results under the setting are reported in Table 2 and Figure 2 . To assess numerically whether the observed errors are consistent with the rates established in Theorems 4 and 7, we adopt the rate-normalization plotting approach used in Banholzer et al. [2 ] . Figure 3 is constructed from 100 independent repetitions of our algorithm. In the left panel, the squared actor error ∣ ϕ n − ϕ ∗ ∣ 2 \lvert\phi_n - \phi^*\rvert^2 ∣ ϕ n − ϕ ∗ ∣ 2 is first averaged across the repetitions and then divided by the theoretical actor rate L n [ ( n + 1 ) − 2 α + ( n + 1 ) − γ 2 ] L_n[(n + 1)^{-2\alpha} + (n + 1)^{-\gamma_2}] L n [( n + 1 ) − 2 α + ( n + 1 ) − γ 2 ] . In the right panel, the absolute Sharpe-ratio error ∣ S R n − S R o r ∣ \lvert SR_n - SR_{\mathrm{or}}\rvert ∣ S R n − S R or ∣ is averaged in the same way and divided by the theoretical Sharpe-ratio rate L n [ ( n + 1 ) − α + ( n + 1 ) − γ 2 ] L_n[(n + 1)^{-\alpha} + (n + 1)^{-\gamma_2}] L n [( n + 1 ) − α + ( n + 1 ) − γ 2 ] .
Figure 2. The learned parameters and average reward along one sample trajectory ( μ = 0.3 , σ = 0.1 ) (\mu=0.3,\ \sigma=0.1) ( μ = 0.3 , σ = 0.1 ) .
Figure 3. Rate-normalized simulation errors based on 100 independent repetitions (μ = 0.3 \mu= 0.3 μ = 0.3 , σ = 0.1 \sigma= 0.1 σ = 0.1 ). The red dashed lines indicate the tail averages over the stable region n ≥ 10000 n \ge10000 n ≥ 10000 .
Table 2. Comparison of mean terminal wealth, terminal-wealth variance, and Sharpe ratio under Actor–Critic, Bayesian filtering, and Rolling MLE over 100 independent repetitions. Each method is evaluated using 1,000 independent test paths. Entries report the across-repetition mean followed by the standard deviation in parentheses.
As shown in Figure 2 , the learned actor parameter ϕ \phi ϕ and the mean terminal wealth rapidly approach the true value and the target level, respectively. This indicates that actor learning and multiplier adjustment jointly achieve the desired terminal-wealth constraint. Table 2 shows that Actor–Critic records the highest or tied-highest Sharpe ratio in most parameter settings. Bayesian Filtering typically exhibits higher terminal-wealth variance and a lower Sharpe ratio, while Rolling MLE is markedly less stable, producing very large variances and near-zero Sharpe ratios in many cases. In Figure 3 , the relatively large values at the beginning reflect initialization and the finite-sample transient before the algorithm enters its asymptotic regime. After this initial period, both normalized errors fluctuate around finite levels and show no systematic upward trend. This tail behavior is consistent with the theoretical convergence bounds.
Empirical Analysis
In this section, we conduct some empirical analysis using real market data. The raw panel was downloaded from Yahoo Finance and contains daily adjusted closing prices for 499 firms in the contemporaneous S&P 500 constituent list, from January 2, 1990 to December 31, 2024. We retain only stocks with no missing adjusted closing price over the entire 1990–2024 experimental span, leaving 235 eligible stocks.
The split is chronological. The period January 1, 1990–December 31, 2009 is used exclusively for initialization and training. Initial volatility and Sharpe-ratio estimates are computed from all daily basket excess returns in this period. A common training pool is then formed by drawing 20,000 random starting positions with replacement and constructing one-year windows of 252 return increments (253 price observations). Each learned method resamples 20,000 episodes with replacement from this pool. Consequently, training episodes may overlap, but none contains a post-2009 observation. The holdout period January 1, 2010–December 31, 2024 is divided chronologically into 15 non-overlapping complete 252-increment episodes. No holdout return is used to initialize or bootstrap-train a method.
For each of six experiments, ten stocks are sampled without replacement within the basket; the same stock may appear in different baskets. Dividing each stock price by its first observation removes differences in price level, after which the one-dimensional basket required by the model is obtained by taking the equally weighted average of the ten normalized price series. The basket path is rebased to one at the beginning of every episode. Daily simple returns are then computed from consecutive basket values, and the daily portion of the 2% annual risk-free rate, based on 252 trading days per year, is subtracted to obtain excess returns. These excess returns are the market observations supplied to all learning rules.
The empirical target is z = 1.15 z = 1.15 z = 1.15 , corresponding to 15% one-year wealth growth. Transaction costs are omitted. Since the 20,000 training episodes are not completely mutually independent, the filtering learning rate is set to α = 0.2 \alpha= 0.2 α = 0.2 . All other parameter settings and stochastic methods follow the implementation conventions in § 5 . For each method, we record compounded terminal wealth, the mean and standard deviation of one-year episode returns, their ratio, and whether any episode has non-finite or non-positive wealth. Table 1 reports cross-experiment means. It shows that Actor–Critic achieves the highest Sharpe-like ratio, 1.376, together with the lowest episode-return standard deviation, 0.122, among all reported methods. Although buy-and-hold produces the largest compounded terminal wealth, the Actor–Critic result indicates a more favorable risk-adjusted trade-off and no bankruptcy events. Rolling MLE attains the largest mean episode return but combines it with a standard deviation of 4.250, zero terminal wealth, and a 100% bankruptcy rate, revealing substantial tail instability relative to the Bayesian and Actor–Critic methods.
Table 1. Aggregate out-of-sample performance across six ten-stock S&P 500 baskets. Notes: Entries are means across six experiments; the workbook does not report standard errors. Bold marks the best entry under each criterion.
Conclusions
This paper develops a novel posterior-guided reinforcement-learning framework for continuous-time mean–variance portfolio optimization. Instead of randomizing the portfolio policies with exogenous noise under standard entropy regularization, the method samples a Sharpe ratio from a Gaussian distribution anchored to the Bayesian filter by a KL penalty and maps that draw into the feedback portfolio control. The resulting formulation preserves the common return innovation shared by wealth and beliefs, leading to a Gaussian stochastic policy with declining degree of exploration guided by the accumulated filtering information of the unknown drift.
For implementation, normalized martingale estimating equations update the critic and actor, while a separate batch stochastic-approximation recursion enforces the terminal mean constraint. The convergence analysis establishes consistency of the episodic filter, actor, policy center, and Lagrange multiplier. It further yields a last-iterate Sharpe-ratio error of O p ( L n ( n + 1 ) − α + L n ( n + 1 ) − γ ) O_p\left(L_n(n+1)^{-\alpha} + L_n(n+1)^{-\gamma}\right) O p ( L n ( n + 1 ) − α + L n ( n + 1 ) − γ ) ; in the stationary-market specialization with harmonic filtering and actor schedules, this becomes O p ( log log n / n ) O_p(\log\log n/n) O p ( log log n / n ) . These results explain how the filtering and actor learning jointly determine the data-driven portfolio decision with dynamically adjusted exploration.
Acknowledgements: Xiang Yu is supported by the Hong Kong RGC General Research Fund (GRF) under grants no. 15211524 and no. 15214125.
References
Verification and Admissibility Proofs
Proof of Theorem 1
Proof. Fix ( t , x , m ) (t,x,m) ( t , x , m ) with x ≠ w x \ne w x = w and an arbitrary q ∈ Q ( t , x , m ) q \in\mathcal{Q}(t,x,m) q ∈ Q ( t , x , m ) . For brevity, all the derivatives of v v v below are evaluated at ( s , X s π , m s ) (s,X_s^\pi,m_s) ( s , X s π , m s ) . Under (11) , the controlled generator applied to v v v is
L q v = − ρ ∗ M s ( X s π − w ) v x + 1 2 ( X s π − w ) 2 ( M s 2 + V s ) v x x + R s ( ρ ∗ − m s ) v m + 1 2 R s 2 v m m − M s R s ( X s π − w ) v x m . (47) \mathcal{L}^q v=-\rho^*M_s(X_s^\pi-w)v_x+\frac{1}{2}(X_s^\pi-w)^2(M_s^2+V_s)v_{xx}+R_s(\rho^*-m_s)v_m+\frac{1}{2}R_s^2v_{mm}-M_sR_s(X_s^\pi-w)v_{xm}. \tag*{(47)} L q v = − ρ ∗ M s ( X s π − w ) v x + 2 1 ( X s π − w ) 2 ( M s 2 + V s ) v xx + R s ( ρ ∗ − m s ) v m + 2 1 R s 2 v mm − M s R s ( X s π − w ) v x m . (47)
Since the infimum of the Hamiltonian is bounded above by its value at the specific density q s q_s q s , the HJB equation implies
( v t + L q v + λ s D K L ( q s ∥ N ( m s , R s ) ) v ) ( s , X s π , m s ) ≥ 0. (48) \left(v_t+\mathcal{L}^qv+\lambda_sD_{\mathrm{KL}}(q_s\|N(m_s,R_s))v\right)(s,X_s^\pi,m_s)\ge0. \tag*{(48)} ( v t + L q v + λ s D KL ( q s ∥ N ( m s , R s )) v ) ( s , X s π , m s ) ≥ 0. (48)
Recall the integrating factor Λ s q \Lambda_s^q Λ s q defined in Definition 1. Then
d Λ s q = λ s D K L ( q s ∥ N ( m s , R s ) ) Λ s q d s , Λ t q = 1. d\Lambda_s^q=\lambda_sD_{\mathrm{KL}}(q_s\|N(m_s,R_s))\Lambda_s^qds,\qquad\Lambda_t^q=1. d Λ s q = λ s D KL ( q s ∥ N ( m s , R s )) Λ s q d s , Λ t q = 1.
Itô’s formula yields
d v ( s , X s π , m s ) = ( v t + L q v ) ( s , X s π , m s ) d s + [ R s v m − M s ( X s π − w ) v x ] ( s , X s π , m s ) d W s − V s ( X s π − w ) v x ( s , X s π , m s ) d B s . (49) dv(s,X_s^\pi,m_s)=(v_t+\mathcal{L}^qv)(s,X_s^\pi,m_s)ds+\left[R_sv_m-M_s(X_s^\pi-w)v_x\right](s,X_s^\pi,m_s)dW_s-\sqrt{V_s}(X_s^\pi-w)v_x(s,X_s^\pi,m_s)dB_s. \tag*{(49)} d v ( s , X s π , m s ) = ( v t + L q v ) ( s , X s π , m s ) d s + [ R s v m − M s ( X s π − w ) v x ] ( s , X s π , m s ) d W s − V s ( X s π − w ) v x ( s , X s π , m s ) d B s . (49)
Because this integrating factor d Λ s q d\Lambda_s^q d Λ s q has finite variation, its quadratic covariation with v ( s , X s π , m s ) v(s,X_s^\pi,m_s) v ( s , X s π , m s ) is zero. The product rule gives
d [ Λ s q v ( s , X s π , m s ) ] = Λ s q ( v t + L q v + λ s D K L ( q s ∥ N ( m s , R s ) ) v ) ( s , X s π , m s ) d s + Λ s q [ R s v m − M s ( X s π − w ) v x ] ( s , X s π , m s ) d W s − Λ s q V s ( X s π − w ) v x ( s , X s π , m s ) d B s . (50) \begin{aligned}
d\left[\Lambda_s^qv(s,X_s^\pi,m_s)\right]={}&\Lambda_s^q\left(v_t+\mathcal{L}^qv+\lambda_sD_{\mathrm{KL}}(q_s\|N(m_s,R_s))v\right)(s,X_s^\pi,m_s)ds\\
&+\Lambda_s^q\left[R_sv_m-M_s(X_s^\pi-w)v_x\right](s,X_s^\pi,m_s)dW_s\\
&-\Lambda_s^q\sqrt{V_s}(X_s^\pi-w)v_x(s,X_s^\pi,m_s)dB_s. \tag*{(50)}
\end{aligned} d [ Λ s q v ( s , X s π , m s ) ] = Λ s q ( v t + L q v + λ s D KL ( q s ∥ N ( m s , R s )) v ) ( s , X s π , m s ) d s + Λ s q [ R s v m − M s ( X s π − w ) v x ] ( s , X s π , m s ) d W s − Λ s q V s ( X s π − w ) v x ( s , X s π , m s ) d B s . (50)
Choose the explicit localization sequence
τ n : = ( T − T − t n ) ∧ inf { s ≥ t : ∫ t s ( Λ v q ) 2 ( [ R v v m ( v , X v π , m v ) − M v ( X v π − w ) v x ( v , X v π , m v ) ] 2 + V v ( X v π − w ) 2 v x 2 ( v , X v π , m v ) ) d v ≥ n } , (51) \tau_n:=\left(T-\frac{T-t}{n}\right)\wedge\inf\left\{s\ge t:\int_t^s(\Lambda_v^q)^2\left(\left[R_vv_m(v,X_v^\pi,m_v)-M_v(X_v^\pi-w)v_x(v,X_v^\pi,m_v)\right]^2+V_v(X_v^\pi-w)^2v_x^2(v,X_v^\pi,m_v)\right)dv\ge n\right\}, \tag*{(51)} τ n := ( T − n T − t ) ∧ inf { s ≥ t : ∫ t s ( Λ v q ) 2 ( [ R v v m ( v , X v π , m v ) − M v ( X v π − w ) v x ( v , X v π , m v ) ] 2 + V v ( X v π − w ) 2 v x 2 ( v , X v π , m v ) ) d v ≥ n } , (51)
with the convention inf ∅ = ∞ \inf\varnothing=\infty inf ∅ = ∞ . The stochastic integrals in (50) , stopped at τ n \tau_n τ n , are square-integrable martingales, because their joint quadratic variation is bounded by n n n . Moreover, τ n ↑ T \tau_n\uparrow T τ n ↑ T almost surely. Indeed, the stochastic-integral coefficients are locally square-integrable on every compact subinterval of [ t , T ) [t,T) [ t , T ) . Also, by (15) , X s π − w X_s^\pi-w X s π − w does not reach zero when x ≠ w x\ne w x = w . Integrating (50) from t t t to τ n \tau_n τ n and taking E \mathbb{E} E , we get
E [ Λ τ n q v ( τ n , X τ n π , m τ n ) ] − v ( t , x , m ) = E ∫ t τ n Λ s q ( v t + L q v + λ s D K L ( q s ∥ N ( m s , R s ) ) v ) ( s , X s π , m s ) d s ≥ 0 , (52) \mathbb{E}\left[\Lambda_{\tau_n}^qv(\tau_n,X_{\tau_n}^\pi,m_{\tau_n})\right]-v(t,x,m)=\mathbb{E}\int_t^{\tau_n}\Lambda_s^q\left(v_t+\mathcal{L}^qv+\lambda_sD_{\mathrm{KL}}(q_s\|N(m_s,R_s))v\right)(s,X_s^\pi,m_s)ds\ge0, \tag*{(52)} E [ Λ τ n q v ( τ n , X τ n π , m τ n ) ] − v ( t , x , m ) = E ∫ t τ n Λ s q ( v t + L q v + λ s D KL ( q s ∥ N ( m s , R s )) v ) ( s , X s π , m s ) d s ≥ 0 , (52)
where the last inequality follows from (48) . Hence, for every n n n ,
v ( t , x , m ) ≤ E [ Λ τ n q v ( τ n , X τ n π , m τ n ) ] . (53) v(t,x,m)\le\mathbb{E}\left[\Lambda_{\tau_n}^qv(\tau_n,X_{\tau_n}^\pi,m_{\tau_n})\right]. \tag*{(53)} v ( t , x , m ) ≤ E [ Λ τ n q v ( τ n , X τ n π , m τ n ) ] . (53)
As τ n ↑ T \tau_n\uparrow T τ n ↑ T , continuity of v v v and the terminal condition imply
Λ τ n q v ( τ n , X τ n π , m τ n ) ⟶ Λ T q v ( T , X T π , m T ) = Λ T q ( X T π − w ) 2 \Lambda_{\tau_n}^qv(\tau_n,X_{\tau_n}^\pi,m_{\tau_n})\longrightarrow\Lambda_T^qv(T,X_T^\pi,m_T)=\Lambda_T^q(X_T^\pi-w)^2 Λ τ n q v ( τ n , X τ n π , m τ n ) ⟶ Λ T q v ( T , X T π , m T ) = Λ T q ( X T π − w ) 2
almost surely.
Condition (vi) of Definition 1 asserts precisely that the random variables on the left, as τ n \tau_n τ n ranges over stopping times, form a uniformly integrable family. Therefore, by Vitali’s convergence theorem, the almost-sure convergence above and condition (vi) imply L 1 L^1 L 1 convergence. Hence,
lim n → ∞ E [ Λ τ n q v ( τ n , X τ n π , m τ n ) ] = E [ Λ T q ( X T π − w ) 2 ] . (54) \lim_{n\to\infty}\mathbb{E}\left[\Lambda_{\tau_n}^qv(\tau_n,X_{\tau_n}^\pi,m_{\tau_n})\right]=\mathbb{E}\left[\Lambda_T^q(X_T^\pi-w)^2\right]. \tag*{(54)} n → ∞ lim E [ Λ τ n q v ( τ n , X τ n π , m τ n ) ] = E [ Λ T q ( X T π − w ) 2 ] . (54)
It follows from (53) and (54) that
v ( t , x , m ) ≤ E [ Λ T q ( X T π − w ) 2 ] . (55) v(t,x,m) \le\mathbb{E}\left[\Lambda_T^q(X_T^\pi-w)^2\right]. \tag*{(55)} v ( t , x , m ) ≤ E [ Λ T q ( X T π − w ) 2 ] . (55)
It remains to identify the expectation in (55) with the recursively defined policy value. Condition (v) gives directly
E [ Λ T q ( X T π − w ) 2 ] < ∞ . \mathbb{E}\left[\Lambda_T^q(X_T^\pi-w)^2\right]<\infty. E [ Λ T q ( X T π − w ) 2 ] < ∞.
The L 1 L^1 L 1 representation (13) then shows that Λ q J q \Lambda^qJ^q Λ q J q is the uniformly integrable martingale with this terminal value. Consequently,
Λ s q J s q = E [ Λ T q ( X T π − w ) 2 ∣ F s ] , J t q = E [ Λ T q ( X T π − w ) 2 ] . (56) \Lambda_s^qJ_s^q=\mathbb{E}\left[\Lambda_T^q(X_T^\pi-w)^2\mid\mathcal{F}_s\right],\qquad J_t^q=\mathbb{E}\left[\Lambda_T^q(X_T^\pi-w)^2\right]. \tag*{(56)} Λ s q J s q = E [ Λ T q ( X T π − w ) 2 ∣ F s ] , J t q = E [ Λ T q ( X T π − w ) 2 ] . (56)
It holds by (55) and (56) that
v ( t , x , m ) ≤ J q ( t , x , m ) v(t,x,m)\leq J^q(t,x,m) v ( t , x , m ) ≤ J q ( t , x , m )
for every q ∈ Q ( t , x , m ) q\in\mathcal{Q}(t,x,m) q ∈ Q ( t , x , m ) .
For q = q ∗ q=q^* q = q ∗ , the drift in (50) is identically zero. Therefore (52) is an equality for every n n n , and the same uniformly integrable limit gives v ( t , x , m ) = J q ∗ ( t , x , m ) v(t,x,m)=J^{q^*}(t,x,m) v ( t , x , m ) = J q ∗ ( t , x , m ) . It follows that
v ( t , x , m ) ≤ inf q ∈ Q ( t , x , m ) J q ( t , x , m ) ≤ J q ∗ ( t , x , m ) = v ( t , x , m ) , v(t,x,m)\leq\inf_{q\in\mathcal{Q}(t,x,m)}J^q(t,x,m)\leq J^{q^*}(t,x,m)=v(t,x,m), v ( t , x , m ) ≤ q ∈ Q ( t , x , m ) inf J q ( t , x , m ) ≤ J q ∗ ( t , x , m ) = v ( t , x , m ) ,
which proves v = J v=J v = J and the optimality of q ∗ q^* q ∗ .
Finally, if x = w x=w x = w , then (11) implies X s π − w = 0 X_s^\pi-w=0 X s π − w = 0 for all s ∈ [ t , T ] s\in[t,T] s ∈ [ t , T ] . The terminal variable in (13) is therefore zero, and that representation gives J q ( t , w , m ) = 0 J^q(t,w,m)=0 J q ( t , w , m ) = 0 for every admissible q q q . By the continuous extension stipulated in the theorem, v ( t , w , m ) = 0 v(t,w,m)=0 v ( t , w , m ) = 0 as well. Hence the same identity holds. □ \square □
Proof of Proposition 2
Proof. Fix ( t , x , m ) (t,x,m) ( t , x , m ) with t < T t<T t < T . We verify the six items in Definition 1 directly on the given finite horizon.
Step 1: the elementary admissibility conditions. Because V s ∗ = c R s / ( c + 2 ) > 0 V_s^*=cR_s/(c+2)>0 V s ∗ = c R s / ( c + 2 ) > 0 on [ t , T ] [t,T] [ t , T ] , q s ∗ = N ( M s ∗ , V s ∗ ) q_s^*=N(M_s^*,V_s^*) q s ∗ = N ( M s ∗ , V s ∗ ) is a probability density. To make the measurability and moment claims explicit, divide the filtering equation by R s R_s R s and use d ( R s − 1 ) = d s d(R_s^{-1})=ds d ( R s − 1 ) = d s :
d ( m s − ρ ∗ R s ) = d W s . d\left(\frac{m_s-\rho^*}{R_s}\right)=dW_s. d ( R s m s − ρ ∗ ) = d W s .
Hence
m s = ρ ∗ + R s R t ( m − ρ ∗ ) + R s ( W s − W t ) , (57) m_s=\rho^*+\frac{R_s}{R_t}(m-\rho^*)+R_s(W_s-W_t), \tag*{(57)} m s = ρ ∗ + R t R s ( m − ρ ∗ ) + R s ( W s − W t ) , (57)
which is adapted and Gaussian. In view that M s ∗ = a s m s + b s ρ ∗ M_s^*=a_sm_s+b_s\rho^* M s ∗ = a s m s + b s ρ ∗ and a s , b s , V s ∗ a_s,b_s,V_s^* a s , b s , V s ∗ are bounded deterministic functions on [ t , T ] [t,T] [ t , T ] , it holds that q ∗ q^* q ∗ is progressively measurable and E ∫ t T ( ( M s ∗ ) 2 + V s ∗ ) d s < ∞ \mathbb{E}\int_t^T((M_s^*)^2+V_s^*)\,ds<\infty E ∫ t T (( M s ∗ ) 2 + V s ∗ ) d s < ∞ . This verifies conditions (i)–(iii). The same integrability ensures that each term in (15) is well defined, and the existence and pathwise uniqueness of the linear wealth SDE also follow.
Step 2: a global auxiliary estimate for the wealth moments. We use the time scale already employed in the learning analysis τ s : = log ( R s / R T ) \tau_s:=\log(R_s/R_T) τ s := log ( R s / R T ) and write p c ( τ s ) : = R s P ( s ) p_c(\tau_s):=R_sP(s) p c ( τ s ) := R s P ( s ) . Since d τ s / d s = − R s d\tau_s/ds=-R_s d τ s / d s = − R s , the Riccati equation for P P P gives
p c ′ = ( c − 2 ) p c 2 − ( 5 c + 2 ) p c + 2 c c + 2 , p c ( 0 ) = 0. (58) p_c'=\frac{(c-2)p_c^2-(5c+2)p_c+2c}{c+2},\qquad p_c(0)=0. \tag*{(58)} p c ′ = c + 2 ( c − 2 ) p c 2 − ( 5 c + 2 ) p c + 2 c , p c ( 0 ) = 0. (58)
For c ≠ 2 c\ne2 c = 2 , the right-hand side of (58) is positive at zero and equals − 2 -2 − 2 at one. Its first positive root is r − r_- r − , so phase-line analysis gives 0 ≤ p c ( τ ) < r − < 1 0\leq p_c(\tau)<r_-<1 0 ≤ p c ( τ ) < r − < 1 and p c ′ ( τ ) ≥ 0 p_c'(\tau)\geq0 p c ′ ( τ ) ≥ 0 . For c = 2 c=2 c = 2 , the same facts follow from p 2 ( τ ) = ( 1 − e − 3 τ ) / 3 p_2(\tau)=(1-e^{-3\tau})/3 p 2 ( τ ) = ( 1 − e − 3 τ ) /3 . Consequently,
c c + 2 ≤ a s = c + 2 p c ( τ s ) c + 2 < 1 , d a s d τ s ≥ 0. (59) \frac{c}{c+2}\leq a_s=\frac{c+2p_c(\tau_s)}{c+2}<1,\qquad\frac{da_s}{d\tau_s}\geq0. \tag*{(59)} c + 2 c ≤ a s = c + 2 c + 2 p c ( τ s ) < 1 , d τ s d a s ≥ 0. (59)
For later use, let G X : = G ( T ) G_X:=G^{(T)} G X := G ( T ) be the maximal backward solution of
1 2 G ˙ X − R s ( 1 + 4 a s ) G X + 1 2 R s 2 G X 2 + 6 a s 2 = 0 , G X ( T ) = 0. (60) \frac{1}{2}\dot{G}_X-R_s(1+4a_s)G_X+\frac{1}{2}R_s^2G_X^2+6a_s^2=0,\qquad G_X(T)=0. \tag*{(60)} 2 1 G ˙ X − R s ( 1 + 4 a s ) G X + 2 1 R s 2 G X 2 + 6 a s 2 = 0 , G X ( T ) = 0. (60)
Writing G ^ X ( τ s ) : = R s G X ( s ) \widehat{G}_X(\tau_s) := R_sG_X(s) G X ( τ s ) := R s G X ( s ) transforms this equation into
G ^ X ′ = G ^ X 2 − ( 1 + 8 a s ) G ^ X + 12 a s 2 , G ^ X ( 0 ) = 0. (61) \widehat{G}_X' = \widehat{G}_X^2 - (1 + 8a_s)\widehat{G}_X + 12a_s^2,\qquad\widehat{G}_X(0) = 0. \tag*{(61)} G X ′ = G X 2 − ( 1 + 8 a s ) G X + 12 a s 2 , G X ( 0 ) = 0. (61)
The nonnegative half-line is invariant because the right-hand side of (61) equals 12 a s 2 ≥ 0 12a_s^2 \ge0 12 a s 2 ≥ 0 at zero. Viewed as a quadratic polynomial in G ^ X \widehat{G}_X G X , its smaller positive zero is h s : = 1 + 8 a s − 1 + 16 a s + 16 a s 2 2 h_s := \frac{1+8a_s-\sqrt{1+16a_s+16a_s^2}}{2} h s := 2 1 + 8 a s − 1 + 16 a s + 16 a s 2 . This expression is increasing in a s a_s a s as 1 + 16 a s + 16 a s 2 > 1 + 2 a s \sqrt{1+16a_s+16a_s^2}>1+2a_s 1 + 16 a s + 16 a s 2 > 1 + 2 a s , and hence is nondecreasing in τ s \tau_s τ s by (59). Its defining identity gives
d h s d τ s ≥ h s 2 − ( 1 + 8 a s ) h s + 12 a s 2 = 0. \frac{d h_s}{d\tau_s} \ge h_s^2-(1+8a_s)h_s+12a_s^2=0. d τ s d h s ≥ h s 2 − ( 1 + 8 a s ) h s + 12 a s 2 = 0.
Thus h s h_s h s is a supersolution of (61) . As G ^ X ( 0 ) = 0 < h T \widehat{G}_X(0)=0<h_T G X ( 0 ) = 0 < h T , scalar comparison yields
0 ≤ G ^ X ( τ s ) ≤ h s = 1 + 8 a s − 1 + 16 a s + 16 a s 2 2 < ∞ 0\le\widehat{G}_X(\tau_s)\le h_s=\frac{1+8a_s-\sqrt{1+16a_s+16a_s^2}}{2}<\infty 0 ≤ G X ( τ s ) ≤ h s = 2 1 + 8 a s − 1 + 16 a s + 16 a s 2 < ∞
on every finite interval. As a result, G X G_X G X is finite on [ t , T ] [t,T] [ t , T ] for every c > 0 c>0 c > 0 and every finite horizon, without any restriction on log ( R t / R T ) \log(R_t/R_T) log ( R t / R T ) .
Step 3: the wealth moments. Set Y s ∗ = X s π ∗ − w Y_s^*=X_s^{\pi^*}-w Y s ∗ = X s π ∗ − w . Under the candidate policy,
d Y s ∗ = − ρ ∗ M s ∗ Y s ∗ d s − M s ∗ Y s ∗ d W s − V s ∗ Y s ∗ d B s , d m s = R s ( ρ ∗ − m s ) d s + R s d W s . (62) \begin{aligned}
dY_s^*&=-\rho^*M_s^*Y_s^*\,ds-M_s^*Y_s^*\,dW_s-\sqrt{V_s^*}Y_s^*\,dB_s,\\
dm_s&=R_s(\rho^*-m_s)\,ds+R_s\,dW_s. \tag*{(62)}
\end{aligned} d Y s ∗ d m s = − ρ ∗ M s ∗ Y s ∗ d s − M s ∗ Y s ∗ d W s − V s ∗ Y s ∗ d B s , = R s ( ρ ∗ − m s ) d s + R s d W s . (62)
The shared Brownian motion gives d ⟨ Y ∗ , m ⟩ s = − M s ∗ R s Y s ∗ d s d\langle Y^*,m\rangle_s=-M_s^*R_sY_s^*\,ds d ⟨ Y ∗ , m ⟩ s = − M s ∗ R s Y s ∗ d s . For each fixed u ∈ [ t , T ] u\in[t,T] u ∈ [ t , T ] , consider the conditional fourth-moment function
( s , y , z ) ⟼ E [ ∣ Y u ∣ 4 ∣ Y s ∗ = y , m s = z ] , t ≤ s ≤ u . (s,y,z)\longmapsto\mathbb{E}\left[|Y_u|^4\mid Y_s^*=y,\ m_s=z\right],\qquad t\le s\le u. ( s , y , z ) ⟼ E [ ∣ Y u ∣ 4 ∣ Y s ∗ = y , m s = z ] , t ≤ s ≤ u .
Its exponential-quadratic candidate is
U ( u ) ( s , y , z ) = y 4 exp { 1 2 G ( u ) ( s ) z 2 + H ( u ) ( s ) z + I ( u ) ( s ) } , U ( u ) ( u , y , z ) = y 4 . U^{(u)}(s,y,z)=y^4\exp\left\{\frac{1}{2}G^{(u)}(s)z^2+H^{(u)}(s)z+I^{(u)}(s)\right\},\qquad U^{(u)}(u,y,z)=y^4. U ( u ) ( s , y , z ) = y 4 exp { 2 1 G ( u ) ( s ) z 2 + H ( u ) ( s ) z + I ( u ) ( s ) } , U ( u ) ( u , y , z ) = y 4 .
The multiplicative wealth dynamics make the fourth moment homogeneous of degree four in y y y , while the affine dependence of M s ∗ M_s^* M s ∗ on the Gaussian filter gives the exponential-quadratic dependence on z z z . The superscript records the terminal date. Before the required integrability is established, U ( u ) U^{(u)} U ( u ) is used only as a positive auxiliary function for the stopped argument below rather than being assumed a priori to equal the displayed conditional expectation. The drift in Itô’s formula is obtained from
U s ( u ) − ρ ∗ M s ∗ y U y ( u ) + R s ( ρ ∗ − z ) U z ( u ) + 1 2 y 2 ( ( M s ∗ ) 2 + V s ∗ ) U y y ( u ) + 1 2 R s 2 U z z ( u ) − M s ∗ R s y U y z ( u ) . U_s^{(u)}-\rho^*M_s^*yU_y^{(u)}+R_s(\rho^*-z)U_z^{(u)}+\frac{1}{2}y^2\left((M_s^*)^2+V_s^*\right)U_{yy}^{(u)}+\frac{1}{2}R_s^2U_{zz}^{(u)}-M_s^*R_syU_{yz}^{(u)}. U s ( u ) − ρ ∗ M s ∗ y U y ( u ) + R s ( ρ ∗ − z ) U z ( u ) + 2 1 y 2 ( ( M s ∗ ) 2 + V s ∗ ) U y y ( u ) + 2 1 R s 2 U z z ( u ) − M s ∗ R s y U y z ( u ) .
Using M s ∗ = a s z + b s ρ ∗ M_s^*=a_sz+b_s\rho^* M s ∗ = a s z + b s ρ ∗ , the coefficient of z 2 U ( u ) z^2U^{(u)} z 2 U ( u ) vanishes when
1 2 G ˙ ( u ) − R s ( 1 + 4 a s ) G ( u ) + 1 2 R s 2 ( G ( u ) ) 2 + 6 a s 2 = 0 , G ( u ) ( u ) = 0. \frac{1}{2}\dot{G}^{(u)}-R_s(1+4a_s)G^{(u)}+\frac{1}{2}R_s^2\left(G^{(u)}\right)^2+6a_s^2=0,\qquad G^{(u)}(u)=0. 2 1 G ˙ ( u ) − R s ( 1 + 4 a s ) G ( u ) + 2 1 R s 2 ( G ( u ) ) 2 + 6 a s 2 = 0 , G ( u ) ( u ) = 0.
For u = T u=T u = T , this is (60), so G ( T ) = G X G^{(T)}=G_X G ( T ) = G X . For an earlier terminal date, scalar comparison after the time change in Step 2 shows that
0 ≤ G ( u ) ( s ) ≤ G X ( s ) , t ≤ s ≤ u . 0\le G^{(u)}(s)\le G_X(s),\qquad t\le s\le u. 0 ≤ G ( u ) ( s ) ≤ G X ( s ) , t ≤ s ≤ u .
Thus G ( u ) G^{(u)} G ( u ) is finite, uniformly over u ∈ [ t , T ] u\in[t,T] u ∈ [ t , T ] . Equating the coefficients of z U ( u ) zU^{(u)} z U ( u ) and U ( u ) U^{(u)} U ( u ) to zero gives, respectively,
H ˙ ( u ) + [ R s 2 G ( u ) − R s ( 1 + 4 a s ) ] H ( u ) + ρ ∗ [ − 4 a s + 12 a s b s + R s G ( u ) ( 1 − 4 b s ) ] = 0 , H ( u ) ( u ) = 0 , \dot{H}^{(u)}+\left[R_s^2G^{(u)}-R_s(1+4a_s)\right]H^{(u)}+\rho^*\left[-4a_s+12a_sb_s+R_sG^{(u)}(1-4b_s)\right]=0,\qquad H^{(u)}(u)=0, H ˙ ( u ) + [ R s 2 G ( u ) − R s ( 1 + 4 a s ) ] H ( u ) + ρ ∗ [ − 4 a s + 12 a s b s + R s G ( u ) ( 1 − 4 b s ) ] = 0 , H ( u ) ( u ) = 0 ,
I ˙ ( u ) + 6 V s ∗ + 1 2 R s 2 ( G ( u ) + ( H ( u ) ) 2 ) + R s ρ ∗ ( 1 − 4 b s ) H ( u ) + ( ρ ∗ ) 2 ( 6 b s 2 − 4 b s ) = 0 , I ( u ) ( u ) = 0. \dot{I}^{(u)}+6V_s^*+\frac{1}{2}R_s^2\left(G^{(u)}+\left(H^{(u)}\right)^2\right)+R_s\rho^*(1-4b_s)H^{(u)}+\left(\rho^*\right)^2(6b_s^2-4b_s)=0,\qquad I^{(u)}(u)=0. I ˙ ( u ) + 6 V s ∗ + 2 1 R s 2 ( G ( u ) + ( H ( u ) ) 2 ) + R s ρ ∗ ( 1 − 4 b s ) H ( u ) + ( ρ ∗ ) 2 ( 6 b s 2 − 4 b s ) = 0 , I ( u ) ( u ) = 0.
Once G ( u ) G^{(u)} G ( u ) is fixed, the first equation is linear in H ( u ) H^{(u)} H ( u ) , and the second determines I ( u ) I^{(u)} I ( u ) after H ( u ) H^{(u)} H ( u ) is known. The bound by G X G_X G X and the boundedness of the deterministic coefficients therefore imply, by the standard linear-ODE estimates, that H ( u ) H^{(u)} H ( u ) and I ( u ) I^{(u)} I ( u ) are also uniformly bounded over u ∈ [ t , T ] u\in[t,T] u ∈ [ t , T ] . Consequently, U ( u ) U^{(u)} U ( u ) solves the backward Kolmogorov equation for the joint process ( Y ∗ , m ) (Y^*,m) ( Y ∗ , m ) with terminal value y 4 y^4 y 4 .
For completeness, stop the joint state at σ n ( u ) : = inf { s ≥ t : ∣ Y s ∗ ∣ + ∣ m s ∣ ≥ n } ∧ u \sigma_n^{(u)} := \inf\{s \ge t : |Y_s^*| + |m_s| \ge n\} \wedge u σ n ( u ) := inf { s ≥ t : ∣ Y s ∗ ∣ + ∣ m s ∣ ≥ n } ∧ u . With the three coefficient equations just described, the stopped process U ( u ) ( s ∧ σ n ( u ) , Y s ∧ σ n ( u ) ∗ , m s ∧ σ n ( u ) ) U^{(u)}(s \wedge\sigma_n^{(u)},Y_{s\wedge\sigma_n^{(u)}}^*,m_{s\wedge\sigma_n^{(u)}}) U ( u ) ( s ∧ σ n ( u ) , Y s ∧ σ n ( u ) ∗ , m s ∧ σ n ( u ) ) is a true martingale because all its stochastic-integral coefficients are bounded. Consequently,
U ( u ) ( t , x − w , m ) ≥ E [ 1 { σ n ( u ) = u } ∣ Y u ∗ ∣ 4 ] . U^{(u)}(t,x-w,m) \ge\mathbb{E}\left[\mathbf{1}_{\{\sigma_n^{(u)}=u\}}|Y_u^*|^4\right]. U ( u ) ( t , x − w , m ) ≥ E [ 1 { σ n ( u ) = u } ∣ Y u ∗ ∣ 4 ] .
Continuity of ( Y ∗ , m ) (Y^*,m) ( Y ∗ , m ) gives σ n ( u ) ↑ u \sigma_n^{(u)} \uparrow u σ n ( u ) ↑ u almost surely, and Fatou’s lemma yields E ∣ Y u ∗ ∣ 4 < ∞ \mathbb{E}|Y_u^*|^4 < \infty E ∣ Y u ∗ ∣ 4 < ∞ . The uniform bounds on G ( u ) G^{(u)} G ( u ) , H ( u ) H^{(u)} H ( u ) , I ( u ) I^{(u)} I ( u ) further give sup t ≤ s ≤ T E ∣ Y s ∗ ∣ 4 < ∞ \sup_{t\le s\le T}\mathbb{E}|Y_s^*|^4 < \infty sup t ≤ s ≤ T E ∣ Y s ∗ ∣ 4 < ∞ . Moreover, (57) implies E [ sup t ≤ s ≤ T ∣ m s ∣ 2 ] < ∞ \mathbb{E}[\sup_{t\le s\le T}|m_s|^2] < \infty E [ sup t ≤ s ≤ T ∣ m s ∣ 2 ] < ∞ and sup s E ∣ M s ∗ ∣ 4 < ∞ \sup_s\mathbb{E}|M_s^*|^4 < \infty sup s E ∣ M s ∗ ∣ 4 < ∞ . Applying the BDG inequality to (62), followed by Cauchy–Schwarz, gives
E sup t ≤ s ≤ T ∣ Y s ∗ ∣ 2 ≤ C ( ∣ x − w ∣ 2 + ∫ t T ( E ∣ M s ∗ ∣ 4 ) 1 / 2 ( E ∣ Y s ∗ ∣ 4 ) 1 / 2 d s + ∫ t T V s ∗ E ∣ Y s ∗ ∣ 2 d s ) < ∞ . \mathbb{E}\sup_{t\le s\le T}|Y_s^*|^2 \le C\left(|x-w|^2+\int_t^T(\mathbb{E}|M_s^*|^4)^{1/2}(\mathbb{E}|Y_s^*|^4)^{1/2}\,ds+\int_t^T V_s^*\mathbb{E}|Y_s^*|^2\,ds\right)<\infty. E t ≤ s ≤ T sup ∣ Y s ∗ ∣ 2 ≤ C ( ∣ x − w ∣ 2 + ∫ t T ( E ∣ M s ∗ ∣ 4 ) 1/2 ( E ∣ Y s ∗ ∣ 4 ) 1/2 d s + ∫ t T V s ∗ E ∣ Y s ∗ ∣ 2 d s ) < ∞.
In view of X s π ∗ = Y s ∗ + w X_s^{\pi^*}=Y_s^*+w X s π ∗ = Y s ∗ + w , these estimates verify the condition (iv).
Step 4: the weighted terminal loss and uniform integrability. The Gaussian KL formula, the affine representation M s ∗ = a s m s + b s ρ ∗ M_s^*=a_sm_s+b_s\rho^* M s ∗ = a s m s + b s ρ ∗ , and continuity of m m m show first that
∫ t T λ s D K L ( q s ∗ ∥ N ( m s , R s ) ) d s < ∞ almost surely . \int_t^T\lambda_sD_{\mathrm{KL}}(q_s^*\|N(m_s,R_s))\,ds<\infty\quad\text{almost surely}. ∫ t T λ s D KL ( q s ∗ ∥ N ( m s , R s )) d s < ∞ almost surely .
Let v v v be the closed-form candidate in (23). For x ≠ w x\ne w x = w , normalize the process used in the verification theorem as
Z s : = exp { ∫ t s λ v D K L ( q v ∗ ∥ N ( m v , R v ) ) d v } v ( s , X s π ∗ , m s ) v ( t , x , m ) . Z_s:=\frac{\exp\left\{\int_t^s\lambda_vD_{\mathrm{KL}}(q_v^*\|N(m_v,R_v))\,dv\right\}v(s,X_s^{\pi^*},m_s)}{v(t,x,m)}. Z s := v ( t , x , m ) exp { ∫ t s λ v D KL ( q v ∗ ∥ N ( m v , R v )) d v } v ( s , X s π ∗ , m s ) .
Indeed, the closed-form expression for v v v gives, along the state process,
v x v = 2 Y s ∗ , v m v = P ( s ) m s + ρ ∗ Q ‾ ( s ) . \frac{v_x}{v}=\frac{2}{Y_s^*},\qquad\frac{v_m}{v}=P(s)m_s+\rho^*\overline{Q}(s). v v x = Y s ∗ 2 , v v m = P ( s ) m s + ρ ∗ Q ( s ) .
Because q ∗ q^* q ∗ attains the HJB minimum, the HJB equation implies that the drift of v ( s , X s π ∗ , m s ) v(s,X_s^{\pi^*},m_s) v ( s , X s π ∗ , m s ) is − λ s D K L ( q s ∗ ∥ N ( m s , R s ) ) v d s -\lambda_sD_{\mathrm{KL}}(q_s^*\|N(m_s,R_s))v\,ds − λ s D KL ( q s ∗ ∥ N ( m s , R s )) v d s . Its d W s dW_s d W s and d B s dB_s d B s coefficients are, respectively, − M s ∗ Y s ∗ v x + R s v m -M_s^*Y_s^*v_x+R_sv_m − M s ∗ Y s ∗ v x + R s v m and − V s ∗ Y s ∗ v x -\sqrt{V_s^*}Y_s^*v_x − V s ∗ Y s ∗ v x . Hence the product rule shows that the exponential integrating factor in Z Z Z cancels the drift. Using M s ∗ = a s m s + b s ρ ∗ M_s^*=a_sm_s+b_s\rho^* M s ∗ = a s m s + b s ρ ∗ then gives
d Z s Z s = [ ( R s P ( s ) − 2 a s ) m s + ρ ∗ ( R s Q ‾ ( s ) − 2 b s ) ] d W s − 2 V s ∗ d B s . (63) \frac{dZ_s}{Z_s}=\left[(R_sP(s)-2a_s)m_s+\rho^*(R_s\overline{Q}(s)-2b_s)\right]dW_s-2\sqrt{V_s^*}\,dB_s. \tag*{(63)} Z s d Z s = [ ( R s P ( s ) − 2 a s ) m s + ρ ∗ ( R s Q ( s ) − 2 b s ) ] d W s − 2 V s ∗ d B s . (63)
The coefficients are locally square integrable, so this equation makes Z Z Z a continuous local martingale. Moreover, the multiplicative equation for Y ∗ Y^* Y ∗ preserves Y s ∗ ≠ 0 Y_s^*\ne0 Y s ∗ = 0 when x ≠ w x\ne w x = w . Hence, v > 0 v>0 v > 0 and Z Z Z is strictly positive. All deterministic coefficients in (63) are bounded on [ t , T ] [t,T] [ t , T ] . Hence its squared diffusion coefficient is bounded by C ( 1 + m s 2 ) C(1+m_s^2) C ( 1 + m s 2 ) .
We now prove that Z Z Z is a true martingale. Set σ n : = inf { s ≥ t : ∣ m s ∣ ≥ n } ∧ T \sigma_n:=\inf\{s\ge t:|m_s|\ge n\}\wedge T σ n := inf { s ≥ t : ∣ m s ∣ ≥ n } ∧ T . The stopped coefficients in (63) are bounded, so Z s ∧ σ n Z_{s\wedge\sigma_n} Z s ∧ σ n is a true martingale and E [ Z σ n ] = 1 \mathbb{E}[Z_{\sigma_n}]=1 E [ Z σ n ] = 1 . Hence d Q n / d P ∣ F T = Z σ n dQ_n/dP|_{\mathcal{F}_T}=Z_{\sigma_n} d Q n / d P ∣ F T = Z σ n defines a probability measure. The two-dimensional Girsanov theorem states that subtracting the time integral of each stopped diffusion coefficient in (63) gives Brownian motions under Q n Q_n Q n . In particular,
W s − ∫ t s ∧ σ n [ ( R v P ( v ) − 2 a v ) m v + ρ ∗ ( R v Q ‾ ( v ) − 2 b v ) ] d v W_s-\int_t^{s\wedge\sigma_n}\left[(R_vP(v)-2a_v)m_v+\rho^*(R_v\overline{Q}(v)-2b_v)\right]\,dv W s − ∫ t s ∧ σ n [ ( R v P ( v ) − 2 a v ) m v + ρ ∗ ( R v Q ( v ) − 2 b v ) ] d v
is a Brownian motion under Q n Q_n Q n . Substituting the corresponding decomposition of d W s dW_s d W s into the filtering equation shows that, under Q n Q_n Q n , the drift of m s m_s m s is
R s [ ρ ∗ − m s + 1 { s ≤ σ n } ( ( R s P ( s ) − 2 a s ) m s + ρ ∗ ( R s Q ‾ ( s ) − 2 b s ) ) ] . R_s\left[\rho^*-m_s+\mathbf{1}_{\{s\le\sigma_n\}}\left((R_sP(s)-2a_s)m_s+\rho^*(R_s\overline{Q}(s)-2b_s)\right)\right]. R s [ ρ ∗ − m s + 1 { s ≤ σ n } ( ( R s P ( s ) − 2 a s ) m s + ρ ∗ ( R s Q ( s ) − 2 b s ) ) ] .
This change of measure is used below through E [ Z σ n log Z σ n ] = E Q n [ log Z σ n ] \mathbb{E}[Z_{\sigma_n}\log Z_{\sigma_n}]=\mathbb{E}^{Q_n}[\log Z_{\sigma_n}] E [ Z σ n log Z σ n ] = E Q n [ log Z σ n ] . Denote the displayed Q n Q_n Q n -Brownian motion by W Q n W^{Q_n} W Q n . The boundedness of R , P , Q ‾ , a , b R,P,\overline{Q},a,b R , P , Q , a , b gives, with C C C independent of n n n ,
∣ R s [ ρ ∗ − m s + 1 { s ≤ σ n } ( ( R s P ( s ) − 2 a s ) m s + ρ ∗ ( R s Q ‾ ( s ) − 2 b s ) ) ] ∣ ≤ C ( 1 + ∣ m s ∣ ) , ∣ R s ∣ ≤ C . \left|R_s\left[\rho^*-m_s+\mathbf{1}_{\{s\le\sigma_n\}}\left((R_sP(s)-2a_s)m_s+\rho^*(R_s\overline{Q}(s)-2b_s)\right)\right]\right|\le C(1+|m_s|),\qquad|R_s|\le C. R s [ ρ ∗ − m s + 1 { s ≤ σ n } ( ( R s P ( s ) − 2 a s ) m s + ρ ∗ ( R s Q ( s ) − 2 b s ) ) ] ≤ C ( 1 + ∣ m s ∣ ) , ∣ R s ∣ ≤ C .
For s ∈ [ t , T ] s \in[t,T] s ∈ [ t , T ] , Itô’s formula applied to ∣ m s ∣ 2 |m_s|^2 ∣ m s ∣ 2 gives
∣ m s ∣ 2 = ∣ m ∣ 2 + 2 ∫ t s m v R v [ ρ ∗ − m v + 1 { v ≤ σ n } ( ( R v P ( v ) − 2 a v ) m v + ρ ∗ ( R v Q ‾ ( v ) − 2 b v ) ) ] d v + ∫ t s R v 2 d v + 2 ∫ t s R v m v d W v Q n . |m_s|^2 = |m|^2 + 2\int_t^s m_v R_v\left[\rho^* - m_v + 1_{\{v\leq\sigma_n\}}\left((R_vP(v)-2a_v)m_v+\rho^*(R_v\overline{Q}(v)-2b_v)\right)\right]\,dv + \int_t^s R_v^2\,dv + 2\int_t^s R_vm_v\,dW_v^{\mathbb{Q}^n}. ∣ m s ∣ 2 = ∣ m ∣ 2 + 2 ∫ t s m v R v [ ρ ∗ − m v + 1 { v ≤ σ n } ( ( R v P ( v ) − 2 a v ) m v + ρ ∗ ( R v Q ( v ) − 2 b v ) ) ] d v + ∫ t s R v 2 d v + 2 ∫ t s R v m v d W v Q n .
Taking the supremum and expectation and using 2 ∣ m v ∣ ( 1 + ∣ m v ∣ ) ≤ C ( 1 + ∣ m v ∣ 2 ) 2|m_v|(1+|m_v|)\leq C(1+|m_v|^2) 2∣ m v ∣ ( 1 + ∣ m v ∣ ) ≤ C ( 1 + ∣ m v ∣ 2 ) to control the drift gives
E Q n sup t ≤ u ≤ s ∣ m u ∣ 2 ≤ ∣ m ∣ 2 + C ∫ t s ( 1 + E Q n sup t ≤ u ≤ v ∣ m u ∣ 2 ) d v + 2 E Q n sup t ≤ u ≤ s ∣ ∫ t u R v m v d W v Q n ∣ . \mathbb{E}^{\mathbb{Q}^n}\sup_{t\leq u\leq s}|m_u|^2 \leq|m|^2+C\int_t^s\left(1+\mathbb{E}^{\mathbb{Q}^n}\sup_{t\leq u\leq v}|m_u|^2\right)\,dv+2\mathbb{E}^{\mathbb{Q}^n}\sup_{t\leq u\leq s}\left|\int_t^u R_vm_v\,dW_v^{\mathbb{Q}^n}\right|. E Q n t ≤ u ≤ s sup ∣ m u ∣ 2 ≤ ∣ m ∣ 2 + C ∫ t s ( 1 + E Q n t ≤ u ≤ v sup ∣ m u ∣ 2 ) d v + 2 E Q n t ≤ u ≤ s sup ∫ t u R v m v d W v Q n .
For the last term, the BDG inequality {(citep)}(Revuz and Yor, 1999, Theorem 4.1) first replaces the martingale maximum by its quadratic variation; boundedness of R R R and Cauchy–Schwarz give the second inequality below, and Young’s inequality implies the last:
E Q n sup t ≤ u ≤ s ∣ ∫ t u R v m v d W v Q n ∣ ≤ C E Q n ( ∫ t s R v 2 ∣ m v ∣ 2 d v ) 1 / 2 ≤ C ( E Q n sup t ≤ u ≤ s ∣ m u ∣ 2 ) 1 / 2 ≤ 1 2 E Q n sup t ≤ u ≤ s ∣ m u ∣ 2 + C . \mathbb{E}^{\mathbb{Q}^n}\sup_{t\leq u\leq s}\left|\int_t^u R_vm_v\,dW_v^{\mathbb{Q}^n}\right|
\leq C\mathbb{E}^{\mathbb{Q}^n}\left(\int_t^s R_v^2|m_v|^2\,dv\right)^{1/2}
\leq C\left(\mathbb{E}^{\mathbb{Q}^n}\sup_{t\leq u\leq s}|m_u|^2\right)^{1/2}
\leq\frac{1}{2}\mathbb{E}^{\mathbb{Q}^n}\sup_{t\leq u\leq s}|m_u|^2+C. E Q n t ≤ u ≤ s sup ∫ t u R v m v d W v Q n ≤ C E Q n ( ∫ t s R v 2 ∣ m v ∣ 2 d v ) 1/2 ≤ C ( E Q n t ≤ u ≤ s sup ∣ m u ∣ 2 ) 1/2 ≤ 2 1 E Q n t ≤ u ≤ s sup ∣ m u ∣ 2 + C .
After absorbing the last expectation into the left-hand side, we obtain
E Q n sup t ≤ u ≤ s ∣ m u ∣ 2 ≤ C ( 1 + ∣ m ∣ 2 ) + C ∫ t s E Q n sup t ≤ u ≤ v ∣ m u ∣ 2 d v . \mathbb{E}^{\mathbb{Q}^n}\sup_{t\leq u\leq s}|m_u|^2\leq C(1+|m|^2)+C\int_t^s\mathbb{E}^{\mathbb{Q}^n}\sup_{t\leq u\leq v}|m_u|^2\,dv. E Q n t ≤ u ≤ s sup ∣ m u ∣ 2 ≤ C ( 1 + ∣ m ∣ 2 ) + C ∫ t s E Q n t ≤ u ≤ v sup ∣ m u ∣ 2 d v .
Gronwall’s inequality then yields, with a constant independent of n n n ,
sup n E Q n [ sup t ≤ s ≤ T ∣ m s ∣ 2 ] ≤ C ( 1 + ∣ m ∣ 2 ) e C ( T − t ) < ∞ . \sup_n\mathbb{E}^{\mathbb{Q}^n}\left[\sup_{t\leq s\leq T}|m_s|^2\right]\leq C(1+|m|^2)e^{C(T-t)}<\infty. n sup E Q n [ t ≤ s ≤ T sup ∣ m s ∣ 2 ] ≤ C ( 1 + ∣ m ∣ 2 ) e C ( T − t ) < ∞.
Using (63) and Girsanov’s theorem once more, we get that
E [ Z σ n log Z σ n ] = 1 2 E Q n ∫ t σ n ( [ ( R v P ( v ) − 2 a v ) m v + ρ ∗ ( R v Q ‾ ( v ) − 2 b v ) ] 2 + 4 V v ∗ ) d v ≤ C ( 1 + E Q n ∫ t T ∣ m v ∣ 2 d v ) ≤ C . (64) \mathbb{E}[Z_{\sigma_n}\log Z_{\sigma_n}]
=\frac{1}{2}\mathbb{E}^{\mathbb{Q}^n}\int_t^{\sigma_n}\left(\left[(R_vP(v)-2a_v)m_v+\rho^*(R_v\overline{Q}(v)-2b_v)\right]^2+4V_v^*\right)\,dv
\leq C\left(1+\mathbb{E}^{\mathbb{Q}^n}\int_t^T|m_v|^2\,dv\right)\leq C. \tag*{(64)} E [ Z σ n log Z σ n ] = 2 1 E Q n ∫ t σ n ( [ ( R v P ( v ) − 2 a v ) m v + ρ ∗ ( R v Q ( v ) − 2 b v ) ] 2 + 4 V v ∗ ) d v ≤ C ( 1 + E Q n ∫ t T ∣ m v ∣ 2 d v ) ≤ C . (64)
As x log x ≥ − e − 1 x\log x\geq-e^{-1} x log x ≥ − e − 1 for x ≥ 0 x\geq0 x ≥ 0 , it follows from (64) that, with log + x : = max { log x , 0 } \log^+x:=\max\{\log x,0\} log + x := max { log x , 0 } ,
sup n E [ Z σ n log + Z σ n ] ≤ sup n E [ Z σ n log Z σ n ] + e − 1 < ∞ . \sup_n\mathbb{E}[Z_{\sigma_n}\log^+Z_{\sigma_n}]
\leq\sup_n\mathbb{E}[Z_{\sigma_n}\log Z_{\sigma_n}]+e^{-1}<\infty. n sup E [ Z σ n log + Z σ n ] ≤ n sup E [ Z σ n log Z σ n ] + e − 1 < ∞.
As x log + x x\log^+x x log + x is increasing, convex, and grows faster than linearly, the de la Vallée–Poussin criterion {(citep)}(De La Vallée Poussin, 1915) makes { Z σ n } n ≥ 1 \{Z_{\sigma_n}\}_{n\geq1} { Z σ n } n ≥ 1 uniformly integrable. Continuity ensures σ n ↑ T \sigma_n\uparrow T σ n ↑ T and Z σ n → Z T Z_{\sigma_n}\to Z_T Z σ n → Z T almost surely; uniform integrability then gives convergence in L 1 L^1 L 1 and E [ Z T ] = 1 \mathbb{E}[Z_T]=1 E [ Z T ] = 1 . Hence the positive local martingale Z Z Z is a true martingale on [ t , T ] [t,T] [ t , T ] . Fatou’s lemma and conditional Jensen’s inequality further give, for every [ t , T ] [t,T] [ t , T ] -valued stopping time τ \tau τ ,
E [ Z τ log + Z τ ] ≤ E [ Z T log + Z T ] ≤ lim inf n E [ Z σ n log + Z σ n ] < ∞ . \mathbb{E}[Z_\tau\log^+Z_\tau]\leq\mathbb{E}[Z_T\log^+Z_T]\leq\liminf_n\mathbb{E}[Z_{\sigma_n}\log^+Z_{\sigma_n}]<\infty. E [ Z τ log + Z τ ] ≤ E [ Z T log + Z T ] ≤ n lim inf E [ Z σ n log + Z σ n ] < ∞.
The same criterion then shows that { Z τ : τ ≤ T } \{Z_\tau:\tau\leq T\} { Z τ : τ ≤ T } is uniformly integrable.
By virtue of v ( T , X T π ∗ , m T ) = ( X T π ∗ − w ) 2 v(T,X_T^{\pi^*},m_T)=(X_T^{\pi^*}-w)^2 v ( T , X T π ∗ , m T ) = ( X T π ∗ − w ) 2 , we also obtain
E [ exp { ∫ t T λ s D K L ( q s ∗ ∥ N ( m s , R s ) ) d s } ∣ X T π ∗ − w ∣ 2 ] = v ( t , x , m ) < ∞ . \mathbb{E}\left[\exp\left\{\int_t^T\lambda_sD_{\mathrm{KL}}(q_s^*\|N(m_s,R_s))\,ds\right\}|X_T^{\pi^*}-w|^2\right]=v(t,x,m)<\infty. E [ exp { ∫ t T λ s D KL ( q s ∗ ∥ N ( m s , R s )) d s } ∣ X T π ∗ − w ∣ 2 ] = v ( t , x , m ) < ∞.
This proves condition (v). Moreover, by the definition of Z Z Z , for every [ t , T ] [t,T] [ t , T ] -valued stopping time τ \tau τ ,
exp { ∫ t τ λ s D K L ( q s ∗ ∥ N ( m s , R s ) ) d s } v ( τ , X τ π ∗ , m τ ) = v ( t , x , m ) Z τ . \exp\left\{\int_t^\tau\lambda_sD_{\mathrm{KL}}(q_s^*\|N(m_s,R_s))\,ds\right\}v(\tau,X_\tau^{\pi^*},m_\tau)=v(t,x,m)Z_\tau. exp { ∫ t τ λ s D KL ( q s ∗ ∥ N ( m s , R s )) d s } v ( τ , X τ π ∗ , m τ ) = v ( t , x , m ) Z τ .
As v ( t , x , m ) < ∞ v(t,x,m)<\infty v ( t , x , m ) < ∞ is deterministic, multiplication by this constant preserves uniform integrability. Thus the uniform integrability of { Z τ : τ ≤ T } \{Z_\tau:\tau\le T\} { Z τ : τ ≤ T } proves condition (vi). If x = w x=w x = w , the wealth deviation and the weighted candidate process are identically zero, and the same conclusion follows directly.
We have verified conditions (i)–(vi) on every finite horizon, and hence q ∗ ∈ Q ( t , x , m ) q^*\in\mathcal{Q}(t,x,m) q ∗ ∈ Q ( t , x , m ) without an additional restriction. Theorem 1 then shows that q ∗ q^* q ∗ is optimal and that the closed-form candidate is the value function.
Convergence Proofs and Auxiliary Estimates
Basic Model Setup
In the conditional derivation of the conditional mean actor direction, the Lagrange multiplier w w w is first treated as a given constant. Its convergence is proved after the actor analysis. The policy value, filtering, and Sharpe ratio use the macroscopic relaxed-control dynamics in Equation (11) . To remain consistent with the discrete actor update (33) , the actor–critic convergence part uses the sampled-action microscopic dynamics before averaging over actions.
For the n n n th episode, let F n \mathcal{F}_n F n denote the information available before the start of that episode. The original market excess-return process is
d S t ( n ) S t ( n ) = σ ∗ ( ρ ∗ d t + d W t ( n ) ) . \frac{dS_t^{(n)}}{S_t^{(n)}}=\sigma^*\left(\rho^*dt+dW_t^{(n)}\right). S t ( n ) d S t ( n ) = σ ∗ ( ρ ∗ d t + d W t ( n ) ) .
When σ ∗ \sigma^* σ ∗ is unknown, we first use the historical data available before the episode to construct an F n \mathcal{F}_n F n -measurable volatility estimator σ ^ n \widehat{\sigma}_n σ n . For a nonempty closed convex set C ⊂ R C\subset\mathbb{R} C ⊂ R , denote the Euclidean projection by
Π C ( z ) : = arg min y ∈ C ∣ z − y ∣ . \Pi_C(z):=\underset{y\in C}{\arg\min}\,|z-y|. Π C ( z ) := y ∈ C arg min ∣ z − y ∣.
Correspondingly, define the distance from a point z z z to C C C by
dist ( z , C ) : = inf y ∈ C ∣ z − y ∣ = ∣ z − Π C ( z ) ∣ . \operatorname{dist}(z,C):=\underset{y\in C}{\inf}\,|z-y|=|z-\Pi_C(z)|. dist ( z , C ) := y ∈ C inf ∣ z − y ∣ = ∣ z − Π C ( z ) ∣.
Take constants 0 < σ ‾ < σ ∗ < σ ‾ < ∞ 0<\underline{\sigma}<\sigma^*<\overline{\sigma}<\infty 0 < σ < σ ∗ < σ < ∞ , and let the volatility projection set be Σ : = [ σ ‾ , σ ‾ ] \Sigma:=[\underline{\sigma},\overline{\sigma}] Σ := [ σ , σ ] . After the j j j th episode ends, define the nonnegative volatility statistic using the time-averaged continuous quadratic variation
U j : = 1 T ∫ 0 T d ⟨ ln S ( j ) ⟩ t . U_j:=\frac{1}{T}\int_0^T d\langle\ln S^{(j)}\rangle_t. U j := T 1 ∫ 0 T d ⟨ ln S ( j ) ⟩ t .
The corresponding projected stochastic-approximation recursion is written as
σ ^ n + 1 = Π Σ [ ( 1 n + 1 ∑ j = 0 n U j ) 1 / 2 ] . (65) \widehat{\sigma}_{n+1}=\Pi_\Sigma\left[\left(\frac{1}{n+1}\sum_{j=0}^{n}U_j\right)^{1/2}\right]. \tag*{(65)} σ n + 1 = Π Σ ( n + 1 1 j = 0 ∑ n U j ) 1/2 . (65)
For the initial episode, set χ 0 = 1 \chi_0=1 χ 0 = 1 by convention. For n ≥ 1 n\ge1 n ≥ 1 , let χ n = σ ∗ / σ ^ n \chi_n=\sigma^*/\widehat{\sigma}_n χ n = σ ∗ / σ n denote the ratio between the true and estimated standard deviations. Therefore, the standardized market excess-return observation satisfies
d S ~ t ( n ) = χ n ( ρ ∗ d t + d W t ( n ) ) , d\widetilde{S}_t^{(n)}=\chi_n\left(\rho^*dt+dW_t^{(n)}\right), d S t ( n ) = χ n ( ρ ∗ d t + d W t ( n ) ) ,
d m t ( n ) = R t ( n ) [ d S ~ t ( n ) − m t ( n ) d t ] , dm_t^{(n)}=R_t^{(n)}\left[d\widetilde{S}_t^{(n)}-m_t^{(n)}dt\right], d m t ( n ) = R t ( n ) [ d S t ( n ) − m t ( n ) d t ] ,
d R t ( n ) = − ( R t ( n ) ) 2 d t , 0 ≤ t ≤ T . (66) dR_t^{(n)}=-\left(R_t^{(n)}\right)^2dt,\qquad0\le t\le T. \tag*{(66)} d R t ( n ) = − ( R t ( n ) ) 2 d t , 0 ≤ t ≤ T . (66)
The Brownian increments generated in different episodes are mutually independent conditioned on past information. Across episodes, the posterior mean and filtering variance are initialized according to (32) .
For the continuous-time sampled-action model, writing Y t ( n ) : = X ~ t π , n − w Y_t^{(n)}:=\widetilde{X}_t^{\pi,n}-w Y t ( n ) := X t π , n − w , the wealth dynamics is
d Y t ( n ) = − χ n ρ t π ∗ ( n ) Y t ( n ) d t − χ n ρ t π ∗ ( n ) Y t ( n ) d W t ( n ) . (67) dY_t^{(n)}=-\chi_n\rho_t^{\pi^*(n)}Y_t^{(n)}dt-\chi_n\rho_t^{\pi^*(n)}Y_t^{(n)}dW_t^{(n)}. \tag*{(67)} d Y t ( n ) = − χ n ρ t π ∗ ( n ) Y t ( n ) d t − χ n ρ t π ∗ ( n ) Y t ( n ) d W t ( n ) . (67)
Thus, the volatility estimator in fact enters filtering and wealth separately through two channels: the standardized observation and the physical allocation. In the continuous-observation model, the quadratic-variation identity holds directly, namely U j = ( σ ∗ ) 2 U_j=(\sigma^*)^2 U j = ( σ ∗ ) 2 . Therefore, in this setting the volatility estimator is equivalent to the oracle volatility, and χ n \chi_n χ n is identically equal to 1, so that the filtering, wealth, actor–critic, Sharpe-ratio, and multiplier dynamics used below all reduce exactly to their forms under oracle volatility.
In addition, in an infill regime without market microstructure noise, discrete realized quadratic variation converges to continuous quadratic variation at the rate O p ( Δ 1 / 2 ) O_p(\Delta^{1/2}) O p ( Δ 1/2 ) as the grid width Δ → 0 \Delta\to0 Δ → 0 ([3 Theorem 1] ). Let σ ^ n , Δ \widehat{\sigma}_{n,\Delta} σ n , Δ denote the estimator obtained from discrete realized quadratic variation, and let χ n , Δ : = σ ∗ / σ ^ n , Δ \chi_{n,\Delta}:=\sigma^*/\widehat{\sigma}_{n,\Delta} χ n , Δ := σ ∗ / σ n , Δ . By the consistency above, finite averaging, and continuity of the projection, for each fixed episode n ≥ 1 n\ge1 n ≥ 1 ,
σ ^ n , Δ ⟶ p σ ∗ , χ n , Δ ⟶ p 1. \widehat{\sigma}_{n,\Delta}\mathrel{\mathop{\longrightarrow}^{p}}\sigma^*,\qquad\chi_{n,\Delta}\mathrel{\mathop{\longrightarrow}^{p}}1. σ n , Δ ⟶ p σ ∗ , χ n , Δ ⟶ p 1.
Let m ~ ( n , Δ ) , Y ~ ( n , Δ ) \widetilde{m}^{(n,\Delta)},\widetilde{Y}^{(n,\Delta)} m ( n , Δ ) , Y ( n , Δ ) denote the system using χ n , Δ \chi_{n,\Delta} χ n , Δ , coupled with the same Brownian motion, sampled action, and initial values. By the explicit solution of the scalar linear filtering equation,
m ~ t ( n , Δ ) − m t ( n ) = R t ( n ) ( χ n , Δ − 1 ) ( ρ t ∗ + W t ( n ) ) , \widetilde{m}^{(n,\Delta)}_t-m^{(n)}_t=R^{(n)}_t(\chi_{n,\Delta}-1)(\rho^*_t+W^{(n)}_t), m t ( n , Δ ) − m t ( n ) = R t ( n ) ( χ n , Δ − 1 ) ( ρ t ∗ + W t ( n ) ) ,
so this filter perturbation converges to zero uniformly in probability on [ 0 , T ] [0,T] [ 0 , T ] . For Y 0 ( n ) ≠ 0 Y^{(n)}_0\ne0 Y 0 ( n ) = 0 , the explicit wealth solution gives
log ∣ Y ~ t ( n , Δ ) Y t ( n ) ∣ = − ( χ n , Δ − 1 ) ρ ∗ ∫ 0 t ρ s ( n ) d s − χ n , Δ 2 − 1 2 ∫ 0 t ( ρ s ( n ) ) 2 d s − ( χ n , Δ − 1 ) ∫ 0 t ρ s ( n ) d W s ( n ) . \log\left|\frac{\widetilde{Y}^{(n,\Delta)}_t}{Y^{(n)}_t}\right|=-(\chi_{n,\Delta}-1)\rho^*\int_0^t\rho^{(n)}_s\,ds-\frac{\chi_{n,\Delta}^2-1}{2}\int_0^t(\rho^{(n)}_s)^2\,ds-(\chi_{n,\Delta}-1)\int_0^t\rho^{(n)}_s\,dW^{(n)}_s. log Y t ( n ) Y t ( n , Δ ) = − ( χ n , Δ − 1 ) ρ ∗ ∫ 0 t ρ s ( n ) d s − 2 χ n , Δ 2 − 1 ∫ 0 t ( ρ s ( n ) ) 2 d s − ( χ n , Δ − 1 ) ∫ 0 t ρ s ( n ) d W s ( n ) .
For fixed n n n , the coefficients a s ( n ) a^{(n)}_s a s ( n ) , b s ( n ) b^{(n)}_s b s ( n ) and V s ( n ) V^{(n)}_s V s ( n ) are bounded on [ 0 , T ] [0,T] [ 0 , T ] . Moreover, ∣ ϕ n ∣ ≤ c n < ∞ |\phi_n|\le c_n<\infty ∣ ϕ n ∣ ≤ c n < ∞ , and (72) gives sup s ≤ T E ∣ m s ( n ) ∣ 2 < ∞ \sup_{s\le T}\mathbb{E}|m^{(n)}_s|^2<\infty sup s ≤ T E ∣ m s ( n ) ∣ 2 < ∞ . Thus the conditional Gaussian law (74) and the tower property yield
E ∣ ρ s ( n ) ∣ 2 = E [ ∣ M s ϕ n ∣ 2 + V s ( n ) ] ≤ 2 ∣ a s ( n ) ∣ 2 E ∣ m s ( n ) ∣ 2 + 2 ∣ b s ( n ) ∣ 2 c n 2 + V s ( n ) ≤ C n , 0 ≤ s ≤ T , \mathbb{E}|\rho^{(n)}_s|^2=\mathbb{E}\left[|M^{\phi_n}_s|^2+V^{(n)}_s\right]\le2|a^{(n)}_s|^2\mathbb{E}|m^{(n)}_s|^2+2|b^{(n)}_s|^2c_n^2+V^{(n)}_s\le C_n,\qquad0\le s\le T, E ∣ ρ s ( n ) ∣ 2 = E [ ∣ M s ϕ n ∣ 2 + V s ( n ) ] ≤ 2∣ a s ( n ) ∣ 2 E ∣ m s ( n ) ∣ 2 + 2∣ b s ( n ) ∣ 2 c n 2 + V s ( n ) ≤ C n , 0 ≤ s ≤ T ,
where C n C_n C n is a deterministic constant for this fixed n n n . By joint measurability and Tonelli’s theorem,
E [ ∫ 0 T ∣ ρ s ( n ) ∣ 2 d s ] = ∫ 0 T E ∣ ρ s ( n ) ∣ 2 d s ≤ T C n < ∞ . \mathbb{E}\left[\int_0^T|\rho^{(n)}_s|^2\,ds\right]=\int_0^T\mathbb{E}|\rho^{(n)}_s|^2\,ds\le TC_n<\infty. E [ ∫ 0 T ∣ ρ s ( n ) ∣ 2 d s ] = ∫ 0 T E ∣ ρ s ( n ) ∣ 2 d s ≤ T C n < ∞.
Since a nonnegative random variable with finite expectation is finite almost surely, ∫ 0 T ∣ ρ s ( n ) ∣ 2 d s < ∞ \int_0^T|\rho^{(n)}_s|^2\,ds<\infty ∫ 0 T ∣ ρ s ( n ) ∣ 2 d s < ∞ almost surely. Hence these integrals and sup t ≤ T ∣ Y t ( n ) ∣ \sup_{t\le T}|Y^{(n)}_t| sup t ≤ T ∣ Y t ( n ) ∣ are finite almost surely. Since χ n , Δ → 1 \chi_{n,\Delta}\to1 χ n , Δ → 1 in probability,
sup 0 ≤ t ≤ T ∣ Y ~ t ( n , Δ ) − Y t ( n ) ∣ ⟶ p 0. \sup_{0\le t\le T}\left|\widetilde{Y}^{(n,\Delta)}_t-Y^{(n)}_t\right|\mathrel{\mathop{\longrightarrow}^{p}}0. 0 ≤ t ≤ T sup Y t ( n , Δ ) − Y t ( n ) ⟶ p 0.
For Y 0 ( n ) = 0 Y^{(n)}_0=0 Y 0 ( n ) = 0 , both wealth deviations vanish identically.
For the approximation of the continuous-time representation by the discrete sampled dynamics and the corresponding policy-gradient estimator, see Jia et al. [29 ] . Therefore, the subsequent continuous-time convergence proofs do not track the discrete execution error again.
To write the policy coefficients for general c > 0 c>0 c > 0 , let τ t ( n ) : = log ( R t ( n ) / R T ( n ) ) \tau^{(n)}_t:=\log(R^{(n)}_t/R^{(n)}_T) τ t ( n ) := log ( R t ( n ) / R T ( n ) ) . Define p c , h c p_c,h_c p c , h c as the solutions on τ ≥ 0 \tau\ge0 τ ≥ 0 of the following initial-value problems:
d p c d τ = ( c − 2 ) p c 2 − ( 5 c + 2 ) p c + 2 c c + 2 , p c ( 0 ) = 0 , (68) \frac{dp_c}{d\tau}=\frac{(c-2)p_c^2-(5c+2)p_c+2c}{c+2},\qquad p_c(0)=0, \tag*{(68)} d τ d p c = c + 2 ( c − 2 ) p c 2 − ( 5 c + 2 ) p c + 2 c , p c ( 0 ) = 0 , (68)
d h c d τ = − 2 c − ( c − 2 ) p c c + 2 h c , h c ( 0 ) = 1. (69) \frac{dh_c}{d\tau}=-\frac{2c-(c-2)p_c}{c+2}h_c,\qquad h_c(0)=1. \tag*{(69)} d τ d h c = − c + 2 2 c − ( c − 2 ) p c h c , h c ( 0 ) = 1. (69)
They come from the HJB coefficient equations for general c c c . Indeed, if p c = R t P ( t ) p_c=R_tP(t) p c = R t P ( t ) and h c = 1 + R t Q ‾ ( t ) h_c=1+R_t\overline{Q}(t) h c = 1 + R t Q ( t ) , then using R ˙ t = − R t 2 \dot{R}_t=-R_t^2 R ˙ t = − R t 2 and d τ t / d t = − R t d\tau_t/dt=-R_t d τ t / d t = − R t , the equations for P ‾ , Q ‾ \overline{P},\overline{Q} P , Q
0 = P ˙ − 2 R t P + R t 2 P 2 + c − ( c + 2 R t P ) 2 c + 2 , P ( T ) = 0 , 0=\dot{P}-2R_tP+R_t^2P^2+c-\frac{(c+2R_tP)^2}{c+2},\qquad P(T)=0, 0 = P ˙ − 2 R t P + R t 2 P 2 + c − c + 2 ( c + 2 R t P ) 2 , P ( T ) = 0 ,
0 = Q ‾ ˙ + R t ( P − Q ‾ ) + R t 2 P Q ‾ − 2 ( 1 + R t Q ‾ ) ( c + 2 R t P ) c + 2 , Q ‾ ( T ) = 0 , 0=\dot{\overline{Q}}+R_t(P-\overline{Q})+R_t^2P\overline{Q}-\frac{2(1+R_t\overline{Q})(c+2R_tP)}{c+2},\qquad\overline{Q}(T)=0, 0 = Q ˙ + R t ( P − Q ) + R t 2 P Q − c + 2 2 ( 1 + R t Q ) ( c + 2 R t P ) , Q ( T ) = 0 ,
reduce exactly to (68)–(69). Therefore, the actor sampling distribution in the n n n th episode is
ϕ t ϕ = N ( M t ϕ , V t ( n ) ) , M t ϕ = a t ( n ) m t ( n ) + b t ( n ) ϕ , V t ( n ) = c c + 2 R t ( n ) , \phi^{\phi}_t=N\left(M^{\phi}_t,V^{(n)}_t\right),\qquad M^{\phi}_t=a^{(n)}_tm^{(n)}_t+b^{(n)}_t\phi,\qquad V^{(n)}_t=\frac{c}{c+2}R^{(n)}_t, ϕ t ϕ = N ( M t ϕ , V t ( n ) ) , M t ϕ = a t ( n ) m t ( n ) + b t ( n ) ϕ , V t ( n ) = c + 2 c R t ( n ) ,
where
a t ( n ) = c + 2 p c ( τ t ( n ) ) c + 2 , b t ( n ) = 2 h c ( τ t ( n ) ) c + 2 . (70) a^{(n)}_t=\frac{c+2p_c(\tau^{(n)}_t)}{c+2},\qquad b^{(n)}_t=\frac{2h_c(\tau^{(n)}_t)}{c+2}. \tag*{(70)} a t ( n ) = c + 2 c + 2 p c ( τ t ( n ) ) , b t ( n ) = c + 2 2 h c ( τ t ( n ) ) . (70)
Proof of Lemma 1
Because the update of the filtering state depends only on the market observation process, after giving the basic setup above, we first analyze the evolution of the filtering state.
Proof. In addition to notations in Lemma 1, set g n : = 1 p n + T g_n := \frac{1}{p_n+T} g n := p n + T 1 and a n : = p n p n + T a_n := \frac{p_n}{p_n+T} a n := p n + T p n . Then
R t ( n ) = 1 p n + t = R 0 ( n + 1 ) α + R 0 t , 0 ≤ t ≤ T , (71) R_t^{(n)} = \frac{1}{p_n+t} = \frac{R_0}{(n+1)^\alpha+R_0t}, \qquad0 \le t \le T, \tag*{(71)} R t ( n ) = p n + t 1 = ( n + 1 ) α + R 0 t R 0 , 0 ≤ t ≤ T , (71)
Combining this with (66) gives
d e t ( n ) = − e t ( n ) p n + t d t + 1 p n + t d W t ( n ) . de_t^{(n)} = -\frac{e_t^{(n)}}{p_n+t}\,dt+\frac{1}{p_n+t}\,dW_t^{(n)}. d e t ( n ) = − p n + t e t ( n ) d t + p n + t 1 d W t ( n ) .
As p n + t p_n+t p n + t is a deterministic C 1 C^1 C 1 finite-variation process and e t ( n ) e_t^{(n)} e t ( n ) is a continuous Itô process, both are continuous semimartingales. Applying the Itô product rule to ( p n + t ) e t ( n ) (p_n+t)e_t^{(n)} ( p n + t ) e t ( n ) gives
d [ ( p n + t ) e t ( n ) ] = ( p n + t ) d e t ( n ) + e t ( n ) d ( p n + t ) + d ⟨ p n + t , e ( n ) ⟩ t = d W t ( n ) . d\left[(p_n+t)e_t^{(n)}\right]=(p_n+t)\,de_t^{(n)}+e_t^{(n)}\,d(p_n+t)+d\langle p_n+t,e^{(n)}\rangle_t=dW_t^{(n)}. d [ ( p n + t ) e t ( n ) ] = ( p n + t ) d e t ( n ) + e t ( n ) d ( p n + t ) + d ⟨ p n + t , e ( n ) ⟩ t = d W t ( n ) .
Integrating the above identity and using the initial error e 0 ( n ) e_0^{(n)} e 0 ( n ) gives
( p n + t ) e t ( n ) = p n e 0 ( n ) + W t ( n ) , i.e. m t ( n ) − ρ ∗ = p n e 0 ( n ) + W t ( n ) p n + t . (72) (p_n+t)e_t^{(n)}=p_ne_0^{(n)}+W_t^{(n)},\quad\text{i.e.}\quad m_t^{(n)}-\rho^*=\frac{p_ne_0^{(n)}+W_t^{(n)}}{p_n+t}. \tag*{(72)} ( p n + t ) e t ( n ) = p n e 0 ( n ) + W t ( n ) , i.e. m t ( n ) − ρ ∗ = p n + t p n e 0 ( n ) + W t ( n ) . (72)
Setting t = T t=T t = T and using m 0 ( n + 1 ) = m T ( n ) m_0^{(n+1)}=m_T^{(n)} m 0 ( n + 1 ) = m T ( n ) gives (38). It follows from iteration that
e 0 ( n ) = A 0 , n e 0 ( 0 ) + ∑ j = 0 n − 1 A j + 1 , n g j W T ( j ) , A k , n : = ∏ ℓ = k n − 1 a ℓ , A n , n : = 1. e_0^{(n)}=A_{0,n}e_0^{(0)}+\sum_{j=0}^{n-1}A_{j+1,n}g_jW_T^{(j)},\quad A_{k,n}:=\prod_{\ell=k}^{n-1}a_\ell,\quad A_{n,n}:=1. e 0 ( n ) = A 0 , n e 0 ( 0 ) + j = 0 ∑ n − 1 A j + 1 , n g j W T ( j ) , A k , n := ℓ = k ∏ n − 1 a ℓ , A n , n := 1.
Consequently, e 0 ( n ) e_0^{(n)} e 0 ( n ) is Gaussian. Write μ n : = E [ e 0 ( n ) ] \mu_n:=\mathbb{E}[e_0^{(n)}] μ n := E [ e 0 ( n ) ] and v n : = Var ( e 0 ( n ) ) v_n:=\operatorname{Var}(e_0^{(n)}) v n := Var ( e 0 ( n ) ) . Then
μ n + 1 = a n μ n , v n + 1 = a n 2 v n + T g n 2 . (73) \mu_{n+1}=a_n\mu_n,\quad v_{n+1}=a_n^2v_n+Tg_n^2. \tag*{(73)} μ n + 1 = a n μ n , v n + 1 = a n 2 v n + T g n 2 . (73)
As 0 < g n ≤ 1 / T 0<g_n\le1/T 0 < g n ≤ 1/ T ,
v n + 1 − v n = − T g n ( 2 − T g n ) v n + T g n 2 . v_{n+1}-v_n=-Tg_n(2-Tg_n)v_n+Tg_n^2. v n + 1 − v n = − T g n ( 2 − T g n ) v n + T g n 2 .
If v n ≥ T − 1 v_n\ge T^{-1} v n ≥ T − 1 , then
v n + 1 − v n ≤ − T g n ( 2 − T g n ) T − 1 + T g n 2 = − 2 g n + 2 T g n 2 ≤ 0. v_{n+1}-v_n\le-Tg_n(2-Tg_n)T^{-1}+Tg_n^2=-2g_n+2Tg_n^2\le0. v n + 1 − v n ≤ − T g n ( 2 − T g n ) T − 1 + T g n 2 = − 2 g n + 2 T g n 2 ≤ 0.
If v n < T − 1 v_n<T^{-1} v n < T − 1 , then by (73) and a n = 1 − T g n a_n=1-Tg_n a n = 1 − T g n ,
v n + 1 = ( 1 − T g n ) 2 v n + T g n 2 ≤ ( 1 − T g n ) 2 T − 1 + T g n 2 = T − 1 − 2 g n + 2 T g n 2 ≤ T − 1 . v_{n+1}=(1-Tg_n)^2v_n+Tg_n^2\le(1-Tg_n)^2T^{-1}+Tg_n^2=T^{-1}-2g_n+2Tg_n^2\le T^{-1}. v n + 1 = ( 1 − T g n ) 2 v n + T g n 2 ≤ ( 1 − T g n ) 2 T − 1 + T g n 2 = T − 1 − 2 g n + 2 T g n 2 ≤ T − 1 .
Therefore, induction gives sup n v n ≤ max { v 0 , T − 1 } \sup_n v_n\le\max\{v_0,T^{-1}\} sup n v n ≤ max { v 0 , T − 1 } . Hence the Gaussian moment formula gives the uniform moment bound in Lemma 1.
We next prove the rates for the mean and the second moment. Since g n ≍ ( n + 1 ) − α g_n\asymp(n+1)^{-\alpha} g n ≍ ( n + 1 ) − α and 0 < α < 1 0<\alpha<1 0 < α < 1 ,
∑ j = 0 n − 1 g j ≥ c α n 1 − α . \sum_{j=0}^{n-1}g_j\ge c_\alpha n^{1-\alpha}. j = 0 ∑ n − 1 g j ≥ c α n 1 − α .
Therefore
∣ μ n ∣ = ∣ e 0 ( 0 ) ∣ ∏ j = 0 n − 1 ( 1 − T g j ) ≤ ∣ e 0 ( 0 ) ∣ exp { − T ∑ j = 0 n − 1 g j } ≤ C e − c α n 1 − α . |\mu_n|=|e_0^{(0)}|\prod_{j=0}^{n-1}(1-Tg_j)\le|e_0^{(0)}|\exp\left\{-T\sum_{j=0}^{n-1}g_j\right\}\le Ce^{-c_\alpha n^{1-\alpha}}. ∣ μ n ∣ = ∣ e 0 ( 0 ) ∣ j = 0 ∏ n − 1 ( 1 − T g j ) ≤ ∣ e 0 ( 0 ) ∣ exp { − T j = 0 ∑ n − 1 g j } ≤ C e − c α n 1 − α .
Also, since
g n − g n + 1 = O ( ( n + 1 ) − α − 1 ) = o ( g n 2 ) , g_n-g_{n+1}=O((n+1)^{-\alpha-1})=o(g_n^2), g n − g n + 1 = O (( n + 1 ) − α − 1 ) = o ( g n 2 ) ,
we can take N N N sufficiently large so that g n − g n + 1 ≤ T g n 2 g_n-g_{n+1}\leq Tg_n^2 g n − g n + 1 ≤ T g n 2 and T g n ≤ 1 / 2 Tg_n\leq1/2 T g n ≤ 1/2 hold for n ≥ N n\geq N n ≥ N . Since v N < ∞ v_N<\infty v N < ∞ and g N > 0 g_N>0 g N > 0 , choose a finite constant C ≥ max { 2 , v N / g N } C\geq\max\{2,v_N/g_N\} C ≥ max { 2 , v N / g N } . Then v N ≤ C g N v_N\leq Cg_N v N ≤ C g N . If v n ≤ C g n v_n\leq Cg_n v n ≤ C g n for some n ≥ N n\geq N n ≥ N , then (73) gives
v n + 1 = ( 1 − T g n ) 2 v n + T g n 2 ≤ C g n − ( 2 T C − T ) g n 2 + C T 2 g n 3 . v_{n+1}=(1-Tg_n)^2v_n+Tg_n^2\leq Cg_n-(2TC-T)g_n^2+CT^2g_n^3. v n + 1 = ( 1 − T g n ) 2 v n + T g n 2 ≤ C g n − ( 2 T C − T ) g n 2 + C T 2 g n 3 .
Because 0 < g n ≤ 1 / T 0<g_n\leq1/T 0 < g n ≤ 1/ T and C ≥ 2 C\geq2 C ≥ 2 ,
v n + 1 ≤ C g n − C T g n 2 ≤ C g n − C ( g n − g n + 1 ) = C g n + 1 . v_{n+1}\leq Cg_n-CTg_n^2\leq Cg_n-C(g_n-g_{n+1})=Cg_{n+1}. v n + 1 ≤ C g n − C T g n 2 ≤ C g n − C ( g n − g n + 1 ) = C g n + 1 .
Induction yields v n ≤ C g n v_n\leq Cg_n v n ≤ C g n for all n ≥ N n\geq N n ≥ N . Hence
v n = O ( g n ) = O ( ( n + 1 ) − α ) . v_n=O(g_n)=O((n+1)^{-\alpha}). v n = O ( g n ) = O (( n + 1 ) − α ) .
Combining this with the exponential bound for the mean gives
E [ ( e 0 ( n ) ) 2 ] = v n + μ n 2 ≤ C ( n + 1 ) − α , \mathbb{E}[(e_0^{(n)})^2]=v_n+\mu_n^2\leq C(n+1)^{-\alpha}, E [( e 0 ( n ) ) 2 ] = v n + μ n 2 ≤ C ( n + 1 ) − α ,
This proves the bounds on the mean and the second moment when 0 < α < 1 0<\alpha<1 0 < α < 1 . Now suppose α = 1 \alpha=1 α = 1 , so that R 0 T > 1 / 2 R_0T>1/2 R 0 T > 1/2 . Then
g n = R 0 n + 1 + R 0 T , a n = n + 1 n + 1 + R 0 T . g_n=\frac{R_0}{n+1+R_0T},\qquad a_n=\frac{n+1}{n+1+R_0T}. g n = n + 1 + R 0 T R 0 , a n = n + 1 + R 0 T n + 1 .
The same product estimate for μ n \mu_n μ n , together with T ∑ j = 0 n − 1 g j ≥ R 0 T log ( n + 1 ) − C T\sum_{j=0}^{n-1}g_j\geq R_0T\log(n+1)-C T ∑ j = 0 n − 1 g j ≥ R 0 T log ( n + 1 ) − C , yields
∣ μ n ∣ ≤ C ( n + 1 ) − R 0 T ≤ C ( n + 1 ) − 1 / 2 . |\mu_n|\leq C(n+1)^{-R_0T}\leq C(n+1)^{-1/2}. ∣ μ n ∣ ≤ C ( n + 1 ) − R 0 T ≤ C ( n + 1 ) − 1/2 .
Also, choose C ≥ v 0 C\geq v_0 C ≥ v 0 sufficiently large. If v n ≤ C / ( n + 1 ) v_n\leq C/(n+1) v n ≤ C / ( n + 1 ) , then
v n + 1 ≤ C ( n + 1 ) + T R 0 2 ( n + 1 + R 0 T ) 2 ≤ C n + 2 , v_{n+1}\leq\frac{C(n+1)+TR_0^2}{(n+1+R_0T)^2}\leq\frac{C}{n+2}, v n + 1 ≤ ( n + 1 + R 0 T ) 2 C ( n + 1 ) + T R 0 2 ≤ n + 2 C ,
where the last inequality holds after choosing C C C sufficiently large, because 2 R 0 T − 1 > 0 2R_0T-1>0 2 R 0 T − 1 > 0 . Induction therefore gives v n = O ( ( n + 1 ) − 1 ) v_n=O((n+1)^{-1}) v n = O (( n + 1 ) − 1 ) . In both cases, E [ ( e 0 ( n ) ) 2 ] = v n + μ n 2 ≤ C ( n + 1 ) − α \mathbb{E}[(e_0^{(n)})^2]=v_n+\mu_n^2\leq C(n+1)^{-\alpha} E [( e 0 ( n ) ) 2 ] = v n + μ n 2 ≤ C ( n + 1 ) − α , which proves the two rate estimates in Lemma 1.
Finally, we prove a.s. convergence. For any ε > 0 \varepsilon>0 ε > 0 , when n n n is sufficiently large, ∣ μ n ∣ ≤ ε / 2 |\mu_n|\leq\varepsilon/2 ∣ μ n ∣ ≤ ε /2 . By the Gaussian tail bound,
P ( ∣ e 0 ( n ) ∣ > ε ) ≤ 2 exp { − c ε ( n + 1 ) α } . \mathbb{P}(|e_0^{(n)}|>\varepsilon)\leq2\exp\{-c\varepsilon(n+1)^\alpha\}. P ( ∣ e 0 ( n ) ∣ > ε ) ≤ 2 exp { − c ε ( n + 1 ) α } .
The right-hand side is summable, so the Borel–Cantelli lemma gives e 0 ( n ) → 0 e_0^{(n)}\to0 e 0 ( n ) → 0 a.s. □ \square □
Furthermore, (72) directly implies the conditional moment bound needed below. For $n\geq 0,
E [ ∣ m t ( n ) ∣ 2 ∣ F n ] ≤ C R 0 , T ( 1 + ∣ e 0 ( n ) ∣ 2 ) , 0 ≤ t ≤ T , \mathbb{E}\left[|m_t^{(n)}|^2\mid\mathcal{F}_n\right]\leq C_{R_0,T}\left(1+|e_0^{(n)}|^2\right),\qquad0\leq t\leq T, E [ ∣ m t ( n ) ∣ 2 ∣ F n ] ≤ C R 0 , T ( 1 + ∣ e 0 ( n ) ∣ 2 ) , 0 ≤ t ≤ T ,
where C R 0 , T < ∞ C_{R_0,T}<\infty C R 0 , T < ∞ is independent of n n n , e 0 ( n ) e_0^{(n)} e 0 ( n ) , ϕ n \phi_n ϕ n .
Conditional Second-Moment Bound for Actor Noise
Having given the convergence analysis for filtering, we next give the convergence analysis for the parameters in the actor–critic iteration. First, we derive the actor-noise bound. Let G t ( n ) \mathcal{G}_t^{(n)} G t ( n ) denote the information before the current action is sampled. Then ϕ n \phi_n ϕ n , m t ( n ) m_t^{(n)} m t ( n ) , R t ( n ) R_t^{(n)} R t ( n ) , M t ϕ n M_t^{\phi_n} M t ϕ n , and V t ( n ) V_t^{(n)} V t ( n ) are all G t ( n ) \mathcal{G}_t^{(n)} G t ( n ) -measurable, and write
ρ t ( n ) − M t ϕ n ∣ G t ( n ) ∼ N ( 0 , V t ( n ) ) . (74) \rho_t^{(n)}-M_t^{\phi_n}\mid\mathcal{G}_t^{(n)}\sim N\left(0,V_t^{(n)}\right). \tag*{(74)} ρ t ( n ) − M t ϕ n ∣ G t ( n ) ∼ N ( 0 , V t ( n ) ) . (74)
The Gaussian score is
s t ( n ) : = ∂ ϕ log q t ϕ n ( ρ t ( n ) ) = b t ( n ) ρ t ( n ) − M t ϕ n V t ( n ) . (75) s_t^{(n)}:=\partial_\phi\log q_t^{\phi_n}\left(\rho_t^{(n)}\right)=b_t^{(n)}\frac{\rho_t^{(n)}-M_t^{\phi_n}}{V_t^{(n)}}. \tag*{(75)} s t ( n ) := ∂ ϕ log q t ϕ n ( ρ t ( n ) ) = b t ( n ) V t ( n ) ρ t ( n ) − M t ϕ n . (75)
By the conditional Gaussian moment formula,
E [ ∣ s t ( n ) ∣ 2 ∣ G t ( n ) ] = ∣ b t ( n ) ∣ 2 V t ( n ) . \mathbb{E}\left[|s_t^{(n)}|^2\mid\mathcal{G}_t^{(n)}\right]=\frac{|b_t^{(n)}|^2}{V_t^{(n)}}. E [ ∣ s t ( n ) ∣ 2 ∣ G t ( n ) ] = V t ( n ) ∣ b t ( n ) ∣ 2 .
For all n ≥ 0 n \ge0 n ≥ 0 ,
τ t ( n ) = log p n + T p n + t ∈ [ 0 , τ ‾ 0 ] , τ ‾ 0 : = log ( 1 + R 0 T ) . \tau_t^{(n)}=\log\frac{p_n+T}{p_n+t}\in[0,\overline{\tau}_0],\qquad\overline{\tau}_0:=\log(1+R_0T). τ t ( n ) = log p n + t p n + T ∈ [ 0 , τ 0 ] , τ 0 := log ( 1 + R 0 T ) .
Let
f c ( p ) : = ( c − 2 ) p 2 − ( 5 c + 2 ) p + 2 c c + 2 . f_c(p):=\frac{(c-2)p^2-(5c+2)p+2c}{c+2}. f c ( p ) := c + 2 ( c − 2 ) p 2 − ( 5 c + 2 ) p + 2 c .
When c ≠ 2 c\ne2 c = 2 , it holds that
f c ( 0 ) > 0 , f c ( p ) > 0 ( 0 ≤ p < r − ) , f c ( r − ) = 0. f_c(0)>0,\qquad f_c(p)>0\quad(0\le p<r_-),\qquad f_c(r_-)=0. f c ( 0 ) > 0 , f c ( p ) > 0 ( 0 ≤ p < r − ) , f c ( r − ) = 0.
Thanks to the uniqueness of the solution and phase-line analysis for the one-dimensional autonomous ODE [44 Lemma 1.1] , we get
0 ≤ p c ( τ ) < r − , p c ( τ ) ↑ r − . 0\le p_c(\tau)<r_-,\qquad p_c(\tau)\uparrow r_-. 0 ≤ p c ( τ ) < r − , p c ( τ ) ↑ r − .
When c = 2 c=2 c = 2 ,
p 2 ( τ ) = 1 3 ( 1 − e − 3 τ ) ↑ 1 3 . p_2(\tau)=\frac{1}{3}(1-e^{-3\tau})\uparrow\frac{1}{3}. p 2 ( τ ) = 3 1 ( 1 − e − 3 τ ) ↑ 3 1 .
Moreover,
2 c − ( c − 2 ) p c ( τ ) > 0 , 2c-(c-2)p_c(\tau)>0, 2 c − ( c − 2 ) p c ( τ ) > 0 ,
because when 0 < c ≤ 2 0<c\le2 0 < c ≤ 2 its left-hand side is no smaller than 2 c 2c 2 c . When c > 2 c>2 c > 2 , it follows that
f c ( 2 c c − 2 ) = − 4 c c − 2 < 0 ⟹ p c ( τ ) < r − < 2 c c − 2 . f_c\left(\frac{2c}{c-2}\right)=-\frac{4c}{c-2}<0\quad\Longrightarrow\quad p_c(\tau)<r_-<\frac{2c}{c-2}. f c ( c − 2 2 c ) = − c − 2 4 c < 0 ⟹ p c ( τ ) < r − < c − 2 2 c .
Consequently,
h c ( τ ) = exp { − ∫ 0 τ 2 c − ( c − 2 ) p c ( s ) c + 2 d s } ∈ ( 0 , 1 ] . h_c(\tau)=\exp\left\{-\int_0^\tau\frac{2c-(c-2)p_c(s)}{c+2}\,ds\right\}\in(0,1]. h c ( τ ) = exp { − ∫ 0 τ c + 2 2 c − ( c − 2 ) p c ( s ) d s } ∈ ( 0 , 1 ] .
Then 0 ≤ p c ( τ ) ≤ p c ( τ ‾ 0 ) 0\le p_c(\tau)\le p_c(\overline{\tau}_0) 0 ≤ p c ( τ ) ≤ p c ( τ 0 ) , h c ( τ ‾ 0 ) ≤ h c ( τ ) ≤ 1 h_c(\overline{\tau}_0)\le h_c(\tau)\le1 h c ( τ 0 ) ≤ h c ( τ ) ≤ 1 . By (70) , we obtain
c c + 2 ≤ a t ( n ) ≤ c + 2 p c ( τ ‾ 0 ) c + 2 , 2 h c ( τ ‾ 0 ) c + 2 ≤ b t ( n ) ≤ 2 c + 2 . (76) \frac{c}{c+2}\le a_t^{(n)}\le\frac{c+2p_c(\overline{\tau}_0)}{c+2},\qquad\frac{2h_c(\overline{\tau}_0)}{c+2}\le b_t^{(n)}\le\frac{2}{c+2}. \tag*{(76)} c + 2 c ≤ a t ( n ) ≤ c + 2 c + 2 p c ( τ 0 ) , c + 2 2 h c ( τ 0 ) ≤ b t ( n ) ≤ c + 2 2 . (76)
Moreover,
c R 0 ( c + 2 ) ( ( n + 1 ) α + R 0 T ) ≤ V t ( n ) ≤ c R 0 ( c + 2 ) ( n + 1 ) α . (77) \frac{cR_0}{(c+2)((n+1)^\alpha+R_0T)}\le V_t^{(n)}\le\frac{cR_0}{(c+2)(n+1)^\alpha}. \tag*{(77)} ( c + 2 ) (( n + 1 ) α + R 0 T ) c R 0 ≤ V t ( n ) ≤ ( c + 2 ) ( n + 1 ) α c R 0 . (77)
Therefore the conditional second moment of the score is controlled as follows:
E [ ∣ s t ( n ) ∣ 2 ∣ G t ( n ) ] ≤ C c , R 0 ( n + 1 ) α , \mathbb{E}\left[\left|s_t^{(n)}\right|^2\mid\mathcal{G}_t^{(n)}\right]\le C_{c,R_0}(n+1)^\alpha, E [ s t ( n ) 2 ∣ G t ( n ) ] ≤ C c , R 0 ( n + 1 ) α ,
where C c , R 0 < ∞ C_{c,R_0}<\infty C c , R 0 < ∞ is a constant. Moreover, uniformly for t ∈ [ 0 , T ] t\in[0,T] t ∈ [ 0 , T ] ,
τ t ( n ) = ∫ t T R 0 ( n ) 1 + s R 0 ( n ) d s = ( T − t ) R 0 ( n ) + O ( ( R 0 ( n ) ) 2 ) , 0 ≤ τ t ( n ) ≤ T R 0 ( n ) . \tau_t^{(n)}=\int_t^T\frac{R_0^{(n)}}{1+sR_0^{(n)}}\,ds=(T-t)R_0^{(n)}+O\left((R_0^{(n)})^2\right),\qquad0\le\tau_t^{(n)}\le TR_0^{(n)}. τ t ( n ) = ∫ t T 1 + s R 0 ( n ) R 0 ( n ) d s = ( T − t ) R 0 ( n ) + O ( ( R 0 ( n ) ) 2 ) , 0 ≤ τ t ( n ) ≤ T R 0 ( n ) .
Since R 0 ( n ) = R 0 ( n + 1 ) − α R_0^{(n)}=R_0(n+1)^{-\alpha} R 0 ( n ) = R 0 ( n + 1 ) − α and the ODE solution is smooth near τ = 0 \tau=0 τ = 0 ,
p c ( τ t ( n ) ) = O ( ( n + 1 ) − α ) , 1 − h c ( τ t ( n ) ) = O ( ( n + 1 ) − α ) . (78) p_c(\tau_t^{(n)})=O((n+1)^{-\alpha}),\qquad1-h_c(\tau_t^{(n)})=O((n+1)^{-\alpha}). \tag*{(78)} p c ( τ t ( n ) ) = O (( n + 1 ) − α ) , 1 − h c ( τ t ( n ) ) = O (( n + 1 ) − α ) . (78)
For any fixed c > 0 c>0 c > 0 , define
J n θ ( t , x , m ) = A n θ ( t , m ) ( x − w ) 2 , A n θ ( t , m ) > 0 , J_n^\theta(t,x,m)=A_n^\theta(t,m)(x-w)^2,\qquad A_n^\theta(t,m)>0, J n θ ( t , x , m ) = A n θ ( t , m ) ( x − w ) 2 , A n θ ( t , m ) > 0 ,
where
log A n θ ( t , m ) = 1 2 P n ( t ) m 2 + Q ‾ n ( t ) θ m + 1 2 K 1 , n ( t ) θ 2 + K 2 , n ( t ) . (79) \log A_n^\theta(t,m)=\frac{1}{2}P_n(t)m^2+\overline{Q}_n(t)\theta m+\frac{1}{2}K_{1,n}(t)\theta^2+K_{2,n}(t). \tag*{(79)} log A n θ ( t , m ) = 2 1 P n ( t ) m 2 + Q n ( t ) θ m + 2 1 K 1 , n ( t ) θ 2 + K 2 , n ( t ) . (79)
P n ( t ) = p c ( τ t ( n ) ) R t ( n ) , Q ‾ n ( t ) = h c ( τ t ( n ) ) − 1 R t ( n ) . (80) P_n(t)=\frac{p_c(\tau_t^{(n)})}{R_t^{(n)}},\qquad\overline{Q}_n(t)=\frac{h_c(\tau_t^{(n)})-1}{R_t^{(n)}}. \tag*{(80)} P n ( t ) = R t ( n ) p c ( τ t ( n ) ) , Q n ( t ) = R t ( n ) h c ( τ t ( n ) ) − 1 . (80)
K ˙ 1 , n ( t ) = 1 − c − 2 c + 2 h c ( τ t ( n ) ) 2 , K 1 , n ( T ) = 0. (81) \dot{K}_{1,n}(t)=1-\frac{c-2}{c+2}h_c(\tau_t^{(n)})^2,\qquad K_{1,n}(T)=0. \tag*{(81)} K ˙ 1 , n ( t ) = 1 − c + 2 c − 2 h c ( τ t ( n ) ) 2 , K 1 , n ( T ) = 0. (81)
K ˙ 2 , n ( t ) = − 1 2 R t ( n ) p c ( τ t ( n ) ) − c R t ( n ) 2 log c + 2 c , K 2 , n ( T ) = 0. (82) \dot{K}_{2,n}(t)=-\frac{1}{2}R_t^{(n)}p_c(\tau_t^{(n)})-\frac{cR_t^{(n)}}{2}\log\frac{c+2}{c},\qquad K_{2,n}(T)=0. \tag*{(82)} K ˙ 2 , n ( t ) = − 2 1 R t ( n ) p c ( τ t ( n ) ) − 2 c R t ( n ) log c c + 2 , K 2 , n ( T ) = 0. (82)
(78) shows that p c ( τ t ( n ) ) / R t ( n ) p_c(\tau_t^{(n)})/R_t^{(n)} p c ( τ t ( n ) ) / R t ( n ) and ( h c ( τ t ( n ) ) − 1 ) / R t ( n ) (h_c(\tau_t^{(n)})-1)/R_t^{(n)} ( h c ( τ t ( n ) ) − 1 ) / R t ( n ) are uniformly bounded. By (80)–(82) and the smoothness of the ODE, all coefficients and their time derivatives are bounded on [ 0 , T ] [0,T] [ 0 , T ] . Therefore there exists C c , R 0 , T < ∞ C_{c,R_0,T}<\infty C c , R 0 , T < ∞ such that
sup n ≥ 0 sup 0 ≤ t ≤ T ( ∣ P n ( t ) ∣ + ∣ Q ‾ n ( t ) ∣ + ∣ K 1 , n ( t ) ∣ + ∣ K 2 , n ( t ) ∣ + ∣ P ˙ n ( t ) ∣ + ∣ Q ‾ ˙ n ( t ) ∣ + ∣ K ˙ 1 , n ( t ) ∣ + ∣ K ˙ 2 , n ( t ) ∣ ) ≤ C c , R 0 , T . (83) \sup_{n\geq0}\sup_{0\leq t\leq T}\left(\left|P_n(t)\right|+\left|\overline{Q}_n(t)\right|+\left|K_{1,n}(t)\right|+\left|K_{2,n}(t)\right|+\left|\dot{P}_n(t)\right|+\left|\dot{\overline{Q}}_n(t)\right|+\left|\dot{K}_{1,n}(t)\right|+\left|\dot{K}_{2,n}(t)\right|\right)\leq C_{c,R_0,T}. \tag*{(83)} n ≥ 0 sup 0 ≤ t ≤ T sup ( ∣ P n ( t ) ∣ + Q n ( t ) + ∣ K 1 , n ( t ) ∣ + ∣ K 2 , n ( t ) ∣ + P ˙ n ( t ) + Q ˙ n ( t ) + K ˙ 1 , n ( t ) + K ˙ 2 , n ( t ) ) ≤ C c , R 0 , T . (83)
Conditional Mean Actor Direction under a Fixed Critic Projection
To remain consistent with the discrete update, after the sampled action ρ t ( n ) \rho_t^{(n)} ρ t ( n ) is given, the wealth process used in the proof of actor–critic convergence is based on the microscopic dynamics (67) . Therefore, the explicit cross-variation is d ⟨ Y ( n ) , m ( n ) ⟩ t = − ρ t ( n ) R t ( n ) Y t ( n ) d t d\langle Y^{(n)},m^{(n)}\rangle_t=-\rho_t^{(n)}R_t^{(n)}Y_t^{(n)}\,dt d ⟨ Y ( n ) , m ( n ) ⟩ t = − ρ t ( n ) R t ( n ) Y t ( n ) d t . Let ℓ n θ : = log A n θ \ell_n^\theta:=\log A_n^\theta ℓ n θ := log A n θ . Applying Itô formula to J n θ = A n θ ( Y ( n ) ) J_n^\theta=A_n^\theta(Y^{(n)}) J n θ = A n θ ( Y ( n ) ) , we may write
d J n θ = J n θ ( D t ρ ( n ) , θ , n d t + Γ t ρ ( n ) , θ , n d W t ( n ) ) . (84) dJ_n^\theta=J_n^\theta\left(\mathcal{D}_t^{\rho^{(n)},\theta,n}\,dt+\Gamma_t^{\rho^{(n)},\theta,n}\,dW_t^{(n)}\right). \tag*{(84)} d J n θ = J n θ ( D t ρ ( n ) , θ , n d t + Γ t ρ ( n ) , θ , n d W t ( n ) ) . (84)
where
D t ρ , θ , n : = ∂ t ℓ n θ + R t ( n ) ( ρ ∗ − m t ( n ) ) ∂ m ℓ n θ + 1 2 ( R t ( n ) ) 2 ( ∂ m m ℓ n θ + ( ∂ m ℓ n θ ) 2 ) + ρ 2 − 2 ρ ( ρ ∗ + R t ( n ) ∂ m ℓ n θ ) , \mathcal{D}_t^{\rho,\theta,n}:=\partial_t\ell_n^\theta+R_t^{(n)}(\rho^*-m_t^{(n)})\partial_m\ell_n^\theta+\frac{1}{2}(R_t^{(n)})^2\left(\partial_{mm}\ell_n^\theta+(\partial_m\ell_n^\theta)^2\right)+\rho^2-2\rho\left(\rho^*+R_t^{(n)}\partial_m\ell_n^\theta\right), D t ρ , θ , n := ∂ t ℓ n θ + R t ( n ) ( ρ ∗ − m t ( n ) ) ∂ m ℓ n θ + 2 1 ( R t ( n ) ) 2 ( ∂ mm ℓ n θ + ( ∂ m ℓ n θ ) 2 ) + ρ 2 − 2 ρ ( ρ ∗ + R t ( n ) ∂ m ℓ n θ ) ,
Γ t ρ , θ , n : = − 2 ρ + R t ( n ) ∂ m ℓ n θ . \Gamma_t^{\rho,\theta,n}:=-2\rho+R_t^{(n)}\partial_m\ell_n^\theta. Γ t ρ , θ , n := − 2 ρ + R t ( n ) ∂ m ℓ n θ .
After adding the KL drift, define the sampled-action critic residual
d R t θ , n ( ϕ ) : = d J n θ + λ t J n θ D K L ( q t ϕ ∥ N ( m t ( n ) , R t ( n ) ) ) d t . d\mathcal{R}_t^{\theta,n}(\phi):=dJ_n^\theta+\lambda_tJ_n^\theta D_{\mathrm{KL}}\left(q_t^\phi\Vert\mathcal{N}(m_t^{(n)},R_t^{(n)})\right)\,dt. d R t θ , n ( ϕ ) := d J n θ + λ t J n θ D KL ( q t ϕ ∥ N ( m t ( n ) , R t ( n ) ) ) d t .
The corresponding normalized residual drift is defined by
G t ρ , θ , n ( ϕ ) = D t ρ , θ , n + λ t D K L ( q t ϕ ∥ N ( m t ( n ) , R t ( n ) ) ) . (85) G_t^{\rho,\theta,n}(\phi)=\mathcal{D}_t^{\rho,\theta,n}+\lambda_tD_{\mathrm{KL}}\left(q_t^\phi\Vert\mathcal{N}(m_t^{(n)},R_t^{(n)})\right). \tag*{(85)} G t ρ , θ , n ( ϕ ) = D t ρ , θ , n + λ t D KL ( q t ϕ ∥ N ( m t ( n ) , R t ( n ) ) ) . (85)
By (84) and (85) ,
d R t θ , n ( ϕ ) J n θ = G t ρ ( n ) , θ , n ( ϕ ) d t + Γ t ρ ( n ) , θ , n d W t ( n ) . (86) \frac{d\mathcal{R}_t^{\theta,n}(\phi)}{J_n^\theta}=G_t^{\rho^{(n)},\theta,n}(\phi)\,dt+\Gamma_t^{\rho^{(n)},\theta,n}\,dW_t^{(n)}. \tag*{(86)} J n θ d R t θ , n ( ϕ ) = G t ρ ( n ) , θ , n ( ϕ ) d t + Γ t ρ ( n ) , θ , n d W t ( n ) . (86)
For the actor, using d m t ( n ) = R t ( n ) ( ρ ∗ − m t ( n ) ) d t + R t ( n ) d W t ( n ) dm_t^{(n)}=R_t^{(n)}(\rho^*-m_t^{(n)})dt+R_t^{(n)}dW_t^{(n)} d m t ( n ) = R t ( n ) ( ρ ∗ − m t ( n ) ) d t + R t ( n ) d W t ( n ) in (28) , with β c v = 1 \beta_{\mathrm{cv}}=1 β cv = 1 , gives the episode-wise centered decomposition
d C t θ n , ϕ n = G ‾ t θ n , ϕ n , n d t + Γ ‾ t ϕ n , n d W t ( n ) , (87) dC_t^{\theta_n,\phi_n}=\overline{G}_t^{\theta_n,\phi_n,n}\,dt+\overline{\Gamma}_t^{\phi_n,n}\,dW_t^{(n)}, \tag*{(87)} d C t θ n , ϕ n = G t θ n , ϕ n , n d t + Γ t ϕ n , n d W t ( n ) , (87)
where
G ‾ t θ , ϕ , n : = ( ρ t − M t ϕ ) ( ρ t + M t ϕ − 2 ρ ∗ − 2 R t ( n ) ∂ m ℓ n θ ( t , m t ( n ) ) ) − V t ( n ) , (88) \overline{G}_t^{\theta,\phi,n}:=(\rho_t-M_t^\phi)(\rho_t+M_t^\phi-2\rho^*-2R_t^{(n)}\partial_m\ell_n^\theta(t,m_t^{(n)}))-V_t^{(n)}, \tag*{(88)} G t θ , ϕ , n := ( ρ t − M t ϕ ) ( ρ t + M t ϕ − 2 ρ ∗ − 2 R t ( n ) ∂ m ℓ n θ ( t , m t ( n ) )) − V t ( n ) , (88)
Γ ‾ t ϕ , n : = − 2 ( ρ t − M t ϕ ) . (89) \overline{\Gamma}_t^{\phi,n}:=-2(\rho_t-M_t^\phi). \tag*{(89)} Γ t ϕ , n := − 2 ( ρ t − M t ϕ ) . (89)
Thus the continuous-time actor increment corresponding to the adjusted update is
F ^ n + 1 c ( ϕ n , θ n ) = ∫ 0 T s t ( n ) G ‾ t θ n , ϕ n , n d t + ∫ 0 T s t ( n ) Γ ‾ t ϕ n , n d W t ( n ) + ∫ 0 T λ t ∂ ϕ D K L ( q t ϕ n ∥ N ( m t ( n ) , R t ( n ) ) ) d t . (90) \widehat{F}_{n+1}^{c}(\phi_n,\theta_n)=\int_0^T s_t^{(n)}\overline{G}_t^{\theta_n,\phi_n,n}\,dt+\int_0^T s_t^{(n)}\overline{\Gamma}_t^{\phi_n,n}\,dW_t^{(n)}+\int_0^T\lambda_t\partial_\phi D_{\mathrm{KL}}\left(q_t^{\phi_n}\Vert\mathcal{N}(m_t^{(n)},R_t^{(n)})\right)\,dt. \tag*{(90)} F n + 1 c ( ϕ n , θ n ) = ∫ 0 T s t ( n ) G t θ n , ϕ n , n d t + ∫ 0 T s t ( n ) Γ t ϕ n , n d W t ( n ) + ∫ 0 T λ t ∂ ϕ D KL ( q t ϕ n ∥ N ( m t ( n ) , R t ( n ) ) ) d t . (90)
The second integral is the centered corresponding martingale-noise term. For the third integral, the Gaussian KL formula gives
λ t ∂ ϕ D K L ( q t ϕ ∥ N ( m t ( n ) , R t ( n ) ) ) = c b t ( n ) ( M t ϕ − m t ( n ) ) . (91) \lambda_t\partial_\phi D_{\mathrm{KL}}\left(q_t^\phi\Vert\mathcal{N}(m_t^{(n)},R_t^{(n)})\right)=c b_t^{(n)}(M_t^\phi-m_t^{(n)}). \tag*{(91)} λ t ∂ ϕ D KL ( q t ϕ ∥ N ( m t ( n ) , R t ( n ) ) ) = c b t ( n ) ( M t ϕ − m t ( n ) ) . (91)
Its stochastic-approximation decomposition is
F ^ n + 1 c ( ϕ n , θ n ) = F n ( ϕ n , θ n ) + ξ n + 1 , F n ( ϕ , θ ) : = E [ F ^ n + 1 c ( ϕ , θ ) ∣ F n ] . (92) \widehat{F}_{n+1}^{c}(\phi_n,\theta_n)=F_n(\phi_n,\theta_n)+\xi_{n+1},\qquad F_n(\phi,\theta):=\mathbb{E}\left[\widehat{F}_{n+1}^{c}(\phi,\theta)\mid\mathcal{F}_n\right]. \tag*{(92)} F n + 1 c ( ϕ n , θ n ) = F n ( ϕ n , θ n ) + ξ n + 1 , F n ( ϕ , θ ) := E [ F n + 1 c ( ϕ , θ ) ∣ F n ] . (92)
where ξ ^ n + 1 : = F ^ n + 1 c ( ϕ n , θ n ) − F n ( ϕ n , θ n ) \widehat{\xi}_{n+1}:=\widehat{F}^{c}_{n+1}(\phi_n,\theta_n)-F_n(\phi_n,\theta_n) ξ n + 1 := F n + 1 c ( ϕ n , θ n ) − F n ( ϕ n , θ n ) and E [ ξ ^ n + 1 ∣ F n ] = 0 \mathbb{E}[\widehat{\xi}_{n+1}\mid\mathcal{F}_n]=0 E [ ξ n + 1 ∣ F n ] = 0 . Note that F n F_n F n depends on the episode-initial state ( m 0 ( n ) , R 0 ( n ) ) (m_0^{(n)},R_0^{(n)}) ( m 0 ( n ) , R 0 ( n ) ) . The sampled-action estimating statistic corresponding to critic update (26) is defined by
H ^ n + 1 ( θ n , ϕ n ) : = ∫ 0 T ∂ θ J n θ n ( J n θ n ) 2 d R t θ n , n ( ϕ n ; ρ t θ n , n ) . \widehat{H}_{n+1}(\theta_n,\phi_n):=\int_0^T\frac{\partial_{\theta}J_n^{\theta_n}}{(J_n^{\theta_n})^2}\,d\mathcal{R}_t^{\theta_n,n}(\phi_n;\rho_t^{\theta_n,n}). H n + 1 ( θ n , ϕ n ) := ∫ 0 T ( J n θ n ) 2 ∂ θ J n θ n d R t θ n , n ( ϕ n ; ρ t θ n , n ) .
If the terminal wealth of the n n n th episode is denoted by X T ( n ) X_T^{(n)} X T ( n ) , then a multiplier update is triggered at each time n ≡ 0 ( m o d N w ) n\equiv0\pmod{N_w} n ≡ 0 ( mod N w ) , and the single sample X T X_T X T is replaced by the sample mean of the most recent N w N_w N w terminal wealth values. That is, the ( r + 1 ) (r+1) ( r + 1 ) st multiplier update occurs at episode ( r + 1 ) N w (r+1)N_w ( r + 1 ) N w and
I r + 1 : = { r N w + 1 , … , ( r + 1 ) N w } , \mathcal{I}_{r+1}:=\{rN_w+1,\ldots,(r+1)N_w\}, I r + 1 := { r N w + 1 , … , ( r + 1 ) N w } ,
where N w ∈ Z + N_w\in\mathbb{Z}^+ N w ∈ Z + . The batch terminal-wealth statistic used for the update at this time is
X ‾ T , r + 1 : = 1 ∣ I r + 1 ∣ ∑ ℓ ∈ I r + 1 X T ( ℓ ) = 1 N w ∑ ℓ = r N w + 1 ( r + 1 ) N w X T ( ℓ ) . (93) \overline{X}_{T,r+1}:=\frac{1}{|\mathcal{I}_{r+1}|}\sum_{\ell\in\mathcal{I}_{r+1}}X_T^{(\ell)}=\frac{1}{N_w}\sum_{\ell=rN_w+1}^{(r+1)N_w}X_T^{(\ell)}. \tag*{(93)} X T , r + 1 := ∣ I r + 1 ∣ 1 ℓ ∈ I r + 1 ∑ X T ( ℓ ) = N w 1 ℓ = r N w + 1 ∑ ( r + 1 ) N w X T ( ℓ ) . (93)
Lemma 3. For all n ≥ 0 n\ge0 n ≥ 0 , ϕ ∈ R \phi\in\mathbb{R} ϕ ∈ R , and θ ∈ Θ \theta\in\Theta θ ∈ Θ ,
F n ( ϕ , θ ) : = E [ F ^ n + 1 c ( ϕ , θ ) ∣ F n ] = K n ( ϕ − ρ ∗ ) + B n ( θ − ρ ∗ ) (94) F_n(\phi,\theta):=\mathbb{E}[\widehat{F}^{c}_{n+1}(\phi,\theta)\mid\mathcal{F}_n]=K_n(\phi-\rho^*)+B_n(\theta-\rho^*) \tag*{(94)} F n ( ϕ , θ ) := E [ F n + 1 c ( ϕ , θ ) ∣ F n ] = K n ( ϕ − ρ ∗ ) + B n ( θ − ρ ∗ ) (94)
where
K n : = 4 c + 2 ∫ 0 T h c ( τ t ( n ) ) 2 d t , B n : = 4 c + 2 ∫ 0 T h c ( τ t ( n ) ) ( 1 − h c ( τ t ( n ) ) ) d t . (95) K_n:=\frac{4}{c+2}\int_0^T h_c(\tau_t^{(n)})^2\,dt,\qquad B_n:=\frac{4}{c+2}\int_0^T h_c(\tau_t^{(n)})(1-h_c(\tau_t^{(n)}))\,dt. \tag*{(95)} K n := c + 2 4 ∫ 0 T h c ( τ t ( n ) ) 2 d t , B n := c + 2 4 ∫ 0 T h c ( τ t ( n ) ) ( 1 − h c ( τ t ( n ) )) d t . (95)
and
K n ≥ κ c , R 0 : = 4 T h c 2 ( τ ‾ 0 ) c + 2 > 0 , 0 ≤ B n ≤ C c , R 0 , T ( n + 1 ) − α . (96) K_n\ge\kappa_{c,R_0}:=\frac{4Th_c^2(\overline{\tau}_0)}{c+2}>0,\qquad0\le B_n\le C_{c,R_0,T}(n+1)^{-\alpha}. \tag*{(96)} K n ≥ κ c , R 0 := c + 2 4 T h c 2 ( τ 0 ) > 0 , 0 ≤ B n ≤ C c , R 0 , T ( n + 1 ) − α . (96)
Proof. Fix t t t and condition on G t ( n ) \mathcal{G}_t^{(n)} G t ( n ) . Write M : = M t ϕ M:=M_t^\phi M := M t ϕ , V : = V t ( n ) V:=V_t^{(n)} V := V t ( n ) , b : = b t ( n ) b:=b_t^{(n)} b := b t ( n ) , Z : = ρ t ( n ) − M Z:=\rho_t^{(n)}-M Z := ρ t ( n ) − M . By (74) and (75), these abbreviations satisfy Z ∣ G t ( n ) ∼ N ( 0 , V ) Z\mid\mathcal{G}_t^{(n)}\sim N(0,V) Z ∣ G t ( n ) ∼ N ( 0 , V ) and s t ( n ) = b Z / V s_t^{(n)}=bZ/V s t ( n ) = b Z / V . By (88)–(89),
s t ( n ) G ‾ t θ , ϕ , n = b [ 2 Z 2 V ( M − ρ ∗ − R t ( n ) ∂ m ℓ t θ ( t , m t ( n ) ) ) + Z 3 V − Z ] , (97) s_t^{(n)}\overline{G}_t^{\theta,\phi,n}=b\left[\frac{2Z^2}{V}\left(M-\rho^*-R_t^{(n)}\partial_m\ell_t^\theta(t,m_t^{(n)})\right)+\frac{Z^3}{V}-Z\right], \tag*{(97)} s t ( n ) G t θ , ϕ , n = b [ V 2 Z 2 ( M − ρ ∗ − R t ( n ) ∂ m ℓ t θ ( t , m t ( n ) ) ) + V Z 3 − Z ] , (97)
s t ( n ) Γ ‾ t ϕ , n = − 2 b Z 2 V . (98) s_t^{(n)}\overline{\Gamma}_t^{\phi,n}=-2b\frac{Z^2}{V}. \tag*{(98)} s t ( n ) Γ t ϕ , n = − 2 b V Z 2 . (98)
The Gaussian identities E [ Z ] = E [ Z 3 ] = 0 \mathbb{E}[Z]=\mathbb{E}[Z^3]=0 E [ Z ] = E [ Z 3 ] = 0 , E [ Z 2 ] = V \mathbb{E}[Z^2]=V E [ Z 2 ] = V , E [ Z 4 ] = 3 V 2 \mathbb{E}[Z^4]=3V^2 E [ Z 4 ] = 3 V 2 , E [ Z 6 ] = 15 V 3 \mathbb{E}[Z^6]=15V^3 E [ Z 6 ] = 15 V 3 imply
E [ s t ( n ) G ‾ t θ , ϕ , n ∣ G t ( n ) ] = 2 b t ( n ) [ M t ϕ − ρ ∗ − R t ( n ) ∂ m ℓ n θ ( t , m t ( n ) ) ] . (99) \mathbb{E}\left[s_t^{(n)}\overline{G}_t^{\theta,\phi,n}\mid\mathcal{G}_t^{(n)}\right]=2b_t^{(n)}\left[M_t^\phi-\rho^*-R_t^{(n)}\partial_m\ell_n^\theta(t,m_t^{(n)})\right]. \tag*{(99)} E [ s t ( n ) G t θ , ϕ , n ∣ G t ( n ) ] = 2 b t ( n ) [ M t ϕ − ρ ∗ − R t ( n ) ∂ m ℓ n θ ( t , m t ( n ) ) ] . (99)
For the conditional second-moment estimates, use the policy formula preceding (70) and differentiate (79) with respect to m m m . The uniform bounds on a t ( n ) a_t^{(n)} a t ( n ) , b t ( n ) b_t^{(n)} b t ( n ) , R t ( n ) P n ( t ) R_t^{(n)}P_n(t) R t ( n ) P n ( t ) , and R t ( n ) Q ‾ n ( t ) R_t^{(n)}\overline{Q}_n(t) R t ( n ) Q n ( t ) , together with the fixed projection θ ∈ Θ \theta\in\Theta θ ∈ Θ , give
∣ M t ϕ − ρ ∗ − R t ( n ) ∂ m ℓ n θ ( t , m t ( n ) ) ∣ ≤ C c , R 0 , T , Θ ( 1 + ∣ m t ( n ) ∣ + ∣ ϕ ∣ ) . \left|M_t^\phi-\rho^*-R_t^{(n)}\partial_m\ell_n^\theta(t,m_t^{(n)})\right|\le C_{c,R_0,T,\Theta}\left(1+|m_t^{(n)}|+|\phi|\right). M t ϕ − ρ ∗ − R t ( n ) ∂ m ℓ n θ ( t , m t ( n ) ) ≤ C c , R 0 , T , Θ ( 1 + ∣ m t ( n ) ∣ + ∣ ϕ ∣ ) .
Also, (77) gives V ≤ C c , R 0 V\le C_{c,R_0} V ≤ C c , R 0 . Applying ∣ x + y + z ∣ 2 ≤ 3 ( ∣ x ∣ 2 + ∣ y ∣ 2 + ∣ z ∣ 2 ) |x+y+z|^2\le3(|x|^2+|y|^2+|z|^2) ∣ x + y + z ∣ 2 ≤ 3 ( ∣ x ∣ 2 + ∣ y ∣ 2 + ∣ z ∣ 2 ) to (97) and using the Gaussian fourth and sixth moments yields
E [ ∣ s t ( n ) G ‾ t θ , ϕ , n ∣ 2 ∣ G t ( n ) ] ≤ 3 b t ( n ) 2 [ E [ Z 4 ∣ G t ( n ) ] V 2 ∣ M t ϕ − ρ ∗ − R t ( n ) ∂ m ℓ n θ ( t , m t ( n ) ) ∣ 2 + E [ Z 6 ∣ G t ( n ) ] V 2 + E [ Z 2 ∣ G t ( n ) ] ] = 3 b t ( n ) 2 [ 12 ∣ M t ϕ − ρ ∗ − R t ( n ) ∂ m ℓ n θ ( t , m t ( n ) ) ∣ 2 + 16 V ] ≤ C c , R 0 , T , Θ ( 1 + ∣ m t ( n ) ∣ 2 + ∣ ϕ ∣ 2 ) . (100) \begin{aligned}
\mathbb{E}\left[\left|s_t^{(n)}\overline{G}_t^{\theta,\phi,n}\right|^2\mid\mathcal{G}_t^{(n)}\right]
&\le3b_t^{(n)2}\left[\frac{\mathbb{E}[Z^4\mid\mathcal{G}_t^{(n)}]}{V^2}\left|M_t^\phi-\rho^*-R_t^{(n)}\partial_m\ell_n^\theta(t,m_t^{(n)})\right|^2\right.\\
&\qquad\left.+\frac{\mathbb{E}[Z^6\mid\mathcal{G}_t^{(n)}]}{V^2}+\mathbb{E}[Z^2\mid\mathcal{G}_t^{(n)}]\right]\\
&=3b_t^{(n)2}\left[12\left|M_t^\phi-\rho^*-R_t^{(n)}\partial_m\ell_n^\theta(t,m_t^{(n)})\right|^2+16V\right]\\
&\le C_{c,R_0,T,\Theta}\left(1+|m_t^{(n)}|^2+|\phi|^2\right). \tag*{(100)}
\end{aligned} E [ s t ( n ) G t θ , ϕ , n 2 ∣ G t ( n ) ] ≤ 3 b t ( n ) 2 [ V 2 E [ Z 4 ∣ G t ( n ) ] M t ϕ − ρ ∗ − R t ( n ) ∂ m ℓ n θ ( t , m t ( n ) ) 2 + V 2 E [ Z 6 ∣ G t ( n ) ] + E [ Z 2 ∣ G t ( n ) ] ] = 3 b t ( n ) 2 [ 12 M t ϕ − ρ ∗ − R t ( n ) ∂ m ℓ n θ ( t , m t ( n ) ) 2 + 16 V ] ≤ C c , R 0 , T , Θ ( 1 + ∣ m t ( n ) ∣ 2 + ∣ ϕ ∣ 2 ) . (100)
Similarly, (98) and the Gaussian fourth-moment identity above give
E [ ∣ s t ( n ) Γ ‾ t ϕ , n ∣ 2 ∣ G t ( n ) ] = 4 b t 2 E [ Z 4 ∣ G t ( n ) ] V 2 = 12 b 2 ≤ C c , R 0 , T . (101) \mathbb{E}\left[\left|s_t^{(n)}\overline{\Gamma}_t^{\phi,n}\right|^2\mid\mathcal{G}_t^{(n)}\right]=4b_t^2\frac{\mathbb{E}[Z^4\mid\mathcal{G}_t^{(n)}]}{V^2}=12b^2\le C_{c,R_0,T}. \tag*{(101)} E [ s t ( n ) Γ t ϕ , n 2 ∣ G t ( n ) ] = 4 b t 2 V 2 E [ Z 4 ∣ G t ( n ) ] = 12 b 2 ≤ C c , R 0 , T . (101)
To verify the conditional integrability of the drift integral, use the conditional second-moment bound following Lemma 1. That bound, the conditional Tonelli theorem and the tower property, then give
\mathbb{E}\left[\int_{0}^{T}\left|s_{t}^{(n)}G_{t}^{\theta,\phi,n}\right|^{2}\,dt\middle|\mathcal{F}_{n}\right]=\int_{0}^{T}\mathbb{E}\left[\left|s_{t}^{(n)}G_{t}^{\theta,\phi,n}\right|^{2}\middle|\mathcal{G}_{t}^{(n)}\right]\mathord{\,}dt\middle|\mathcal{F}_{n}\leq C_{c,R_{0},T,\Theta}\left(1+\left|e_{0}^{(n)}\right|^{2}+|\phi|^{2}\right)<\infty.
Thus the integral Cauchy–Schwarz inequality makes the drift integral in (90) conditionally integrable.
For the stochastic integral, the conditional Tonelli theorem, the tower property and (101) give
\mathbb{E}\left[\int_{0}^{T}\left|s_{t}^{(n)}\overline{T}_{t}^{\phi,n}\right|^{2}\,dt\middle|\mathcal{F}_{n}\right]=\int_{0}^{T}\mathbb{E}\left[\left|s_{t}^{(n)}\overline{T}_{t}^{\phi,n}\right|^{2}\middle|\mathcal{G}_{t}^{(n)}\right]\mathord{\,}dt\middle|\mathcal{F}_{n}\leq C_{c,R_{0},T}T<\infty.
Thus the stochastic integral is square integrable and has conditional mean zero given F n \mathcal{F}_{n} F n .
For the KL-gradient integral, (91) , the policy formula preceding (70) , and the uniform bound on b t ( n ) b_{t}^{(n)} b t ( n ) imply that
∣ λ t ∂ ϕ D K L ( q t ϕ ∥ N ( m t ( n ) , R t ( n ) ) ) ∣ 2 ≤ C c , R 0 , T ( ∣ m t ( n ) ∣ 2 + ∣ ϕ ∣ 2 ) . \left|\lambda_{t}\partial_{\phi}D_{\mathrm{KL}}\left(q_{t}^{\phi}\middle\|N\left(m_{t}^{(n)},R_{t}^{(n)}\right)\right)\right|^{2}\leq C_{c,R_{0},T}\left(\left|m_{t}^{(n)}\right|^{2}+|\phi|^{2}\right). λ t ∂ ϕ D KL ( q t ϕ N ( m t ( n ) , R t ( n ) ) ) 2 ≤ C c , R 0 , T ( m t ( n ) 2 + ∣ ϕ ∣ 2 ) .
Therefore, the conditional second-moment bound following Lemma 1, the conditional Tonelli theorem and the tower property imply
E [ ∫ 0 T ∣ λ t ∂ ϕ D K L ( q t ϕ ∥ N ( m t ( n ) , R t ( n ) ) ) ∣ 2 d t | F n ] ≤ C c , R 0 , T ( 1 + ∣ e 0 ( n ) ∣ 2 + ∣ ϕ ∣ 2 ) < ∞ . \mathbb{E}\left[\int_{0}^{T}\left|\lambda_{t}\partial_{\phi}D_{\mathrm{KL}}\left(q_{t}^{\phi}\middle\|N\left(m_{t}^{(n)},R_{t}^{(n)}\right)\right)\right|^{2}\,dt\middle|\mathcal{F}_{n}\right]\leq C_{c,R_{0},T}\left(1+\left|e_{0}^{(n)}\right|^{2}+|\phi|^{2}\right)<\infty. E [ ∫ 0 T λ t ∂ ϕ D KL ( q t ϕ N ( m t ( n ) , R t ( n ) ) ) 2 d t F n ] ≤ C c , R 0 , T ( 1 + e 0 ( n ) 2 + ∣ ϕ ∣ 2 ) < ∞.
The integral Cauchy–Schwarz inequality then gives
E [ ∫ 0 T ∣ λ t ∂ ϕ D K L ( q t ϕ ∥ N ( m t ( n ) , R t ( n ) ) ) ∣ d t | F n ] < ∞ . \mathbb{E}\left[\int_{0}^{T}\left|\lambda_{t}\partial_{\phi}D_{\mathrm{KL}}\left(q_{t}^{\phi}\middle\|N\left(m_{t}^{(n)},R_{t}^{(n)}\right)\right)\right|\mathord{\,}dt\middle|\mathcal{F}_{n}\right]<\infty. E [ ∫ 0 T λ t ∂ ϕ D KL ( q t ϕ N ( m t ( n ) , R t ( n ) ) ) d t F n ] < ∞.
Adding the explicit KL derivative c b t ( n ) ( M t ϕ − m t ( n ) ) c b_{t}^{(n)}\left(M_{t}^{\phi}-m_{t}^{(n)}\right) c b t ( n ) ( M t ϕ − m t ( n ) ) from (91) to (99) gives the conditional mean actor direction per unit time
b t ( n ) [ ( c + 2 ) M t ϕ − c m t ( n ) − 2 ρ ∗ − 2 R t ( n ) ∂ m log A n θ ( t , m t ( n ) ) ] . (102) b_{t}^{(n)}\left[(c+2)M_{t}^{\phi}-cm_{t}^{(n)}-2\rho^{*}-2R_{t}^{(n)}\partial_{m}\log A_{n}^{\theta}\left(t,m_{t}^{(n)}\right)\right]. \tag*{(102)} b t ( n ) [ ( c + 2 ) M t ϕ − c m t ( n ) − 2 ρ ∗ − 2 R t ( n ) ∂ m log A n θ ( t , m t ( n ) ) ] . (102)
By (79) , ∂ m log A n θ = P n ( t ) m + Q ‾ n ( t ) θ \partial_{m}\log A_{n}^{\theta}=P_{n}(t)m+\overline{Q}_{n}(t)\theta ∂ m log A n θ = P n ( t ) m + Q n ( t ) θ . Use the coefficient identities in (80) and (70) . Consequently, (102) equals
4 c + 2 [ h c ( τ t ( n ) ) 2 ( ϕ − ρ ∗ ) + h c ( τ t ( n ) ) ( 1 − h c ( τ t ( n ) ) ) ( θ − ρ ∗ ) ] . \frac{4}{c+2}\left[h_{c}(\tau_{t}^{(n)})^{2}(\phi-\rho^{*})+h_{c}(\tau_{t}^{(n)})(1-h_{c}(\tau_{t}^{(n)}))(\theta-\rho^{*})\right]. c + 2 4 [ h c ( τ t ( n ) ) 2 ( ϕ − ρ ∗ ) + h c ( τ t ( n ) ) ( 1 − h c ( τ t ( n ) )) ( θ − ρ ∗ ) ] .
Integration over [ 0 , T ] [0,T] [ 0 , T ] gives (94) –(95) . Finally, h c ( τ ‾ 0 ) ≤ h c ( τ t ( n ) ) ≤ 1 h_{c}(\overline{\tau}_{0})\leq h_{c}(\tau_{t}^{(n)})\leq1 h c ( τ 0 ) ≤ h c ( τ t ( n ) ) ≤ 1 and (78) yield (96) .
Remark 1. The derivation of the conditional mean actor direction above first treats the multiplier w w w as a given constant for the following reason: the actor field uses the product of the normalized residual coefficient and the score s t ( n ) s_{t}^{(n)} s t ( n ) . After normalization, the common quadratic factor ( X t − w ) 2 (X_{t}-w)^{2} ( X t − w ) 2 has already canceled before taking the conditional expectation. Therefore (102) gives a conditional mean actor direction that does not depend on w w w . Similarly, the moment bound for the actor noise does not need to absorb w w w into the constants. Thus, Theorem 4 may be understood as holding for any episode-wise predictable multiplier sequence.
Lemma 4. Under the fixed projection (34) , let ϕ n , θ n \phi_{n},\theta_{n} ϕ n , θ n be F n \mathcal{F}_{n} F n -measurable. Then there exists C < ∞ C<\infty C < ∞ such that, for all n ≥ 0 n\geq0 n ≥ 0 ,
E [ ∣ F ^ n + 1 c ( ϕ n , θ n ) ∣ 2 | F n ] ≤ C ( 1 + ∣ e 0 ( n ) ∣ 4 + ∣ ϕ n ∣ 4 ) , (103) \mathbb{E}\left[\left|\widehat{F}_{n+1}^{c}(\phi_{n},\theta_{n})\right|^{2}\middle|\mathcal{F}_{n}\right]\leq C\left(1+\left|e_{0}^{(n)}\right|^{4}+|\phi_{n}|^{4}\right), \tag*{(103)} E [ F n + 1 c ( ϕ n , θ n ) 2 F n ] ≤ C ( 1 + e 0 ( n ) 4 + ∣ ϕ n ∣ 4 ) , (103) E [ ∣ ξ n + 1 ∣ 4 | F n ] ≤ C ( 1 + ∣ e 0 ( n ) ∣ 4 + ∣ ϕ n ∣ 4 ) . (104) \mathbb{E}\left[|\xi_{n+1}|^{4}\middle|\mathcal{F}_{n}\right]\leq C\left(1+\left|e_{0}^{(n)}\right|^{4}+|\phi_{n}|^{4}\right). \tag*{(104)} E [ ∣ ξ n + 1 ∣ 4 F n ] ≤ C ( 1 + e 0 ( n ) 4 + ∣ ϕ n ∣ 4 ) . (104)
Proof. Denote the three terms in (90) by
I 1 , n : = ∫ 0 T s t ( n ) G ‾ t θ n , ϕ n d t , I 2 , n : = ∫ 0 T s t ( n ) T ‾ t ϕ n , n d W t ( n ) , I_{1,n}:=\int_{0}^{T}s_{t}^{(n)}\overline{G}_{t}^{\theta_{n},\phi_{n}}\,dt,\qquad I_{2,n}:=\int_{0}^{T}s_{t}^{(n)}\overline{T}_{t}^{\phi_{n},n}\,dW_{t}^{(n)}, I 1 , n := ∫ 0 T s t ( n ) G t θ n , ϕ n d t , I 2 , n := ∫ 0 T s t ( n ) T t ϕ n , n d W t ( n ) ,
and
I 3 , n : = ∫ 0 T K t ( n ) d t , K t ( n ) : = λ t ∂ ϕ D K L ( q t ϕ n ∥ N ( m t ( n ) , R t ( n ) ) ) . I_{3,n}:=\int_{0}^{T}K_{t}^{(n)}\,\mathrm{d}t,\qquad K_{t}^{(n)}:=\lambda_{t}\partial_{\phi}D_{\mathrm{KL}}\left(q_{t}^{\phi_{n}}\middle\|N\left(m_{t}^{(n)},R_{t}^{(n)}\right)\right). I 3 , n := ∫ 0 T K t ( n ) d t , K t ( n ) := λ t ∂ ϕ D KL ( q t ϕ n N ( m t ( n ) , R t ( n ) ) ) .
Thus F ^ n + 1 c = I 1 , n + I 2 , n + I 3 , n \widehat{F}_{n+1}^{c}=I_{1,n}+I_{2,n}+I_{3,n} F n + 1 c = I 1 , n + I 2 , n + I 3 , n . For I 1 , n I_{1,n} I 1 , n , the integral Cauchy–Schwarz inequality first gives the pathwise bound
∣ I 1 , n ∣ 2 ≤ T ∫ 0 T ∣ S t ( n ) G t θ n , ϕ n ∣ 2 d t . |I_{1,n}|^{2}\leq T\int_{0}^{T}\left|S_{t}^{(n)}G_{t}^{\theta_{n},\phi_{n}}\right|^{2}\,\mathrm{d}t. ∣ I 1 , n ∣ 2 ≤ T ∫ 0 T S t ( n ) G t θ n , ϕ n 2 d t .
Taking conditional expectation, applying the conditional Tonelli theorem and the tower property, and then using (100) and the conditional second-moment bound following Lemma 1, we obtain
E [ ∣ I 1 , n ∣ 2 ∣ F n ] ≤ T ∫ 0 T E [ E [ ∣ S t ( n ) G t θ n , ϕ n ∣ 2 | G t ( n ) ] | F n ] d t ≤ C ∫ 0 T ( 1 + E [ ∣ m t ( n ) ∣ 2 ∣ F n ] + ∣ ϕ n ∣ 2 ) d t ≤ C ( 1 + ∣ e 0 ( n ) ∣ 4 + ∣ ϕ n ∣ 4 ) . \begin{aligned}
\mathbb{E}\left[|I_{1,n}|^{2}\mid\mathcal{F}_{n}\right]\leq T\int_{0}^{T}\mathbb{E}\left[\mathbb{E}\left[\left|S_{t}^{(n)}G_{t}^{\theta_{n},\phi_{n}}\right|^{2}\middle|\mathcal{G}_{t}^{(n)}\right]\middle|\mathcal{F}_{n}\right]\mathrm{d}t \\
&\leq C\int_{0}^{T}\left(1+\mathbb{E}\left[|m_{t}^{(n)}|^{2}\mid\mathcal{F}_{n}\right]+|\phi_{n}|^{2}\right)\mathrm{d}t \\
&\leq C\left(1+|e_{0}^{(n)}|^{4}+|\phi_{n}|^{4}\right).
\end{aligned} E [ ∣ I 1 , n ∣ 2 ∣ F n ] ≤ T ∫ 0 T E [ E [ S t ( n ) G t θ n , ϕ n 2 G t ( n ) ] F n ] d t ≤ C ∫ 0 T ( 1 + E [ ∣ m t ( n ) ∣ 2 ∣ F n ] + ∣ ϕ n ∣ 2 ) d t ≤ C ( 1 + ∣ e 0 ( n ) ∣ 4 + ∣ ϕ n ∣ 4 ) .
For I 2 , n I_{2,n} I 2 , n , by the predictable-version convention and the square-integrability estimate established in Lemma 3, the conditional Itô isometry gives
E [ ∣ I 2 , n ∣ 2 ∣ F n ] = E [ ∫ 0 T ∣ S t ( n ) Φ ‾ t ϕ n , n ∣ 2 d t ∣ F n ] ≤ C . \mathbb{E}\left[|I_{2,n}|^{2}\mid\mathcal{F}_{n}\right]=\mathbb{E}\left[\left.\int_{0}^{T}\left|S_{t}^{(n)}\overline{\Phi}_{t}^{\phi_{n},n}\right|^{2}\mathrm{d}t\right|\mathcal{F}_{n}\right]\leq C. E [ ∣ I 2 , n ∣ 2 ∣ F n ] = E [ ∫ 0 T S t ( n ) Φ t ϕ n , n 2 d t F n ] ≤ C .
For I 3 , n I_{3,n} I 3 , n , the integral Cauchy–Schwarz inequality implies ∣ I 3 , n ∣ 2 ≤ T ∫ 0 T ∣ K t ( n ) ∣ 2 d t |I_{3,n}|^{2}\leq T\int_{0}^{T}|K_{t}^{(n)}|^{2}\mathrm{d}t ∣ I 3 , n ∣ 2 ≤ T ∫ 0 T ∣ K t ( n ) ∣ 2 d t . The KL-gradient square-integrability estimate established in the proof of Lemma 3, applied with ϕ = ϕ n \phi=\phi_{n} ϕ = ϕ n , therefore yields
E [ ∣ I 3 , n ∣ 2 ∣ F n ] ≤ C T E [ ∫ 0 T ∣ K t ( n ) ∣ 2 d t ∣ F n ] ≤ C ( 1 + ∣ e 0 ( n ) ∣ 2 + ∣ ϕ n ∣ 2 ) . \mathbb{E}\left[|I_{3,n}|^{2}\mid\mathcal{F}_{n}\right]\leq CT\mathbb{E}\left[\left.\int_{0}^{T}|K_{t}^{(n)}|^{2}\mathrm{d}t\right|\mathcal{F}_{n}\right]\leq C\left(1+|e_{0}^{(n)}|^{2}+|\phi_{n}|^{2}\right). E [ ∣ I 3 , n ∣ 2 ∣ F n ] ≤ C T E [ ∫ 0 T ∣ K t ( n ) ∣ 2 d t F n ] ≤ C ( 1 + ∣ e 0 ( n ) ∣ 2 + ∣ ϕ n ∣ 2 ) .
Combining the three estimates with ∣ x + y + z ∣ 2 ≤ 3 ( ∣ x ∣ 2 + ∣ y ∣ 2 + ∣ z ∣ 2 ) |x+y+z|^{2}\leq3(|x|^{2}+|y|^{2}+|z|^{2}) ∣ x + y + z ∣ 2 ≤ 3 ( ∣ x ∣ 2 + ∣ y ∣ 2 + ∣ z ∣ 2 ) gives
E [ ∣ F ^ n + 1 c ( ϕ n , θ n ) ∣ 2 ∣ F n ] ≤ 3 ∑ j = 1 3 E [ ∣ I j , n ∣ 2 ∣ F n ] ≤ C ( 1 + ∣ e 0 ( n ) ∣ 4 + ∣ ϕ n ∣ 4 ) . \mathbb{E}\left[\left|\widehat{F}_{n+1}^{c}(\phi_{n},\theta_{n})\right|^{2}\mid\mathcal{F}_{n}\right]\leq3\sum_{j=1}^{3}\mathbb{E}\left[|I_{j,n}|^{2}\mid\mathcal{F}_{n}\right]\leq C\left(1+|e_{0}^{(n)}|^{4}+|\phi_{n}|^{4}\right). E [ F n + 1 c ( ϕ n , θ n ) 2 ∣ F n ] ≤ 3 j = 1 ∑ 3 E [ ∣ I j , n ∣ 2 ∣ F n ] ≤ C ( 1 + ∣ e 0 ( n ) ∣ 4 + ∣ ϕ n ∣ 4 ) .
which proves (103). Finally, since ξ n + 1 = F ^ n + 1 c − E [ F ^ n + 1 c ∣ F n ] \xi_{n+1}=\widehat{F}_{n+1}^{c}-\mathbb{E}[\widehat{F}_{n+1}^{c}\mid\mathcal{F}_{n}] ξ n + 1 = F n + 1 c − E [ F n + 1 c ∣ F n ] , the conditional variance identity gives
E [ ∣ ξ n + 1 ∣ 2 ∣ F n ] = E [ ∣ F ^ n + 1 c ∣ 2 ∣ F n ] − ∣ E [ F ^ n + 1 c ∣ F n ] ∣ 2 ≤ E [ ∣ F ^ n + 1 c ∣ 2 ∣ F n ] , \begin{aligned}
\mathbb{E}\left[|\xi_{n+1}|^{2}\mid\mathcal{F}_{n}\right]=\mathbb{E}\left[\left|\widehat{F}_{n+1}^{c}\right|^{2}\mid\mathcal{F}_{n}\right]-\left|\mathbb{E}\left[\widehat{F}_{n+1}^{c}\mid\mathcal{F}_{n}\right]\right|^{2} \\
&\leq\mathbb{E}\left[\left|\widehat{F}_{n+1}^{c}\right|^{2}\mid\mathcal{F}_{n}\right],
\end{aligned} E [ ∣ ξ n + 1 ∣ 2 ∣ F n ] = E [ F n + 1 c 2 ∣ F n ] − E [ F n + 1 c ∣ F n ] 2 ≤ E [ F n + 1 c 2 ∣ F n ] ,
and (104) follows from (103).
Proof of Theorem 4
Proof. Let x n : = ϕ n − ρ ∗ x_{n}:=\phi_{n}-\rho^{*} x n := ϕ n − ρ ∗ and d n : = θ n − ρ ∗ d_{n}:=\theta_{n}-\rho^{*} d n := θ n − ρ ∗ . We first prove almost sure convergence. As c n c_{n} c n is nondecreasing and tends to infinity, ρ ∗ ∈ Φ n ⊆ Φ n + 1 \rho^{*}\in\Phi_{n}\subseteq\Phi_{n+1} ρ ∗ ∈ Φ n ⊆ Φ n + 1 for all sufficiently large n n n . For a closed convex set C ⊂ R C\subset\mathbb{R} C ⊂ R , the Euclidean projection satisfies ∥ Π C ( z ) − y ∥ ≤ ∣ z − y ∣ \|\Pi_{C}(z)-y\|\leq|z-y| ∥ Π C ( z ) − y ∥ ≤ ∣ z − y ∣ , y ∈ C y\in C y ∈ C . Therefore, we obtain that
∣ x n + 1 ∣ 2 ≤ ∣ x n − α ϕ ( n ) F ^ n + 1 c ( ϕ n , θ n ) ∣ 2 = ∣ x n ∣ 2 − 2 α ϕ ( n ) x n F ^ n + 1 c ( ϕ n , θ n ) + ( α ϕ ( n ) ) 2 ∣ F ^ n + 1 c ( ϕ n , θ n ) ∣ 2 . \begin{aligned}
|x_{n+1}|^{2}\leq\left|x_{n}-\alpha_{\phi}(n)\widehat{F}_{n+1}^{c}(\phi_{n},\theta_{n})\right|^{2} \\
&=|x_{n}|^{2}-2\alpha_{\phi}(n)x_{n}\widehat{F}_{n+1}^{c}(\phi_{n},\theta_{n})+\left(\alpha_{\phi}(n)\right)^{2}\left|\widehat{F}_{n+1}^{c}(\phi_{n},\theta_{n})\right|^{2}.
\end{aligned} ∣ x n + 1 ∣ 2 ≤ x n − α ϕ ( n ) F n + 1 c ( ϕ n , θ n ) 2 = ∣ x n ∣ 2 − 2 α ϕ ( n ) x n F n + 1 c ( ϕ n , θ n ) + ( α ϕ ( n ) ) 2 F n + 1 c ( ϕ n , θ n ) 2 .
As x n , θ n x_{n},\theta_{n} x n , θ n are F n \mathcal{F}_{n} F n -measurable, taking conditional expectation of the above inequality with respect to F n \mathcal{F}_{n} F n and using (92) gives
E [ ∣ x n + 1 ∣ 2 ∣ F n ] ≤ ∣ x n ∣ 2 − 2 α ϕ ( n ) x n F n ( ϕ n , θ n ) + ( α ϕ ( n ) ) 2 E [ ∣ F ^ n + 1 c ( ϕ n , θ n ) ∣ 2 ∣ F n ] . \mathbb{E}\left[|x_{n+1}|^{2}\mid\mathcal{F}_{n}\right]\leq|x_{n}|^{2}-2\alpha_{\phi}(n)x_{n}F_{n}(\phi_{n},\theta_{n})+\left(\alpha_{\phi}(n)\right)^{2}\mathbb{E}\left[\left|\widehat{F}_{n+1}^{c}(\phi_{n},\theta_{n})\right|^{2}\mid\mathcal{F}_{n}\right]. E [ ∣ x n + 1 ∣ 2 ∣ F n ] ≤ ∣ x n ∣ 2 − 2 α ϕ ( n ) x n F n ( ϕ n , θ n ) + ( α ϕ ( n ) ) 2 E [ F n + 1 c ( ϕ n , θ n ) 2 ∣ F n ] .
For any fixed ε > 0 \varepsilon> 0 ε > 0 , let D ε : = [ − ε , ε ] D_{\varepsilon}:=[-\varepsilon,\varepsilon] D ε := [ − ε , ε ] , V n ( ε ) : = ( ∣ x n ∣ − ε ) + 2 V_n^{(\varepsilon)} := (|x_n|-\varepsilon)_+^2 V n ( ε ) := ( ∣ x n ∣ − ε ) + 2 .
If the closed intervals D ⊆ C D \subseteq C D ⊆ C , then
dist ( Π C ( z ) , D ) ≤ dist ( z , D ) . (105) \operatorname{dist}(\Pi_C(z),D)\le\operatorname{dist}(z,D). \tag*{(105)} dist ( Π C ( z ) , D ) ≤ dist ( z , D ) . (105)
Moreover, since Π D ε ( x ) ∈ D ε \Pi_{D_{\varepsilon}}(x)\in D_{\varepsilon} Π D ε ( x ) ∈ D ε , the definition of distance gives
dist ( x + h , D ε ) 2 ≤ ∣ x + h − Π D ε ( x ) ∣ 2 = dist ( x , D ε ) 2 + 2 ( x − Π D ε ( x ) ) h + h 2 . (106) \operatorname{dist}(x+h,D_{\varepsilon})^2\le|x+h-\Pi_{D_{\varepsilon}}(x)|^2
= \operatorname{dist}(x,D_{\varepsilon})^2+2(x-\Pi_{D_{\varepsilon}}(x))h+h^2. \tag*{(106)} dist ( x + h , D ε ) 2 ≤ ∣ x + h − Π D ε ( x ) ∣ 2 = dist ( x , D ε ) 2 + 2 ( x − Π D ε ( x )) h + h 2 . (106)
For fixed ε \varepsilon ε , when n n n is sufficiently large, D ε ⊆ Φ n + 1 − ρ ∗ D_{\varepsilon}\subseteq\Phi_{n+1}-\rho^* D ε ⊆ Φ n + 1 − ρ ∗ . Applying (105) –(106) to the actor update and taking conditional expectation given F n \mathcal{F}_n F n yields
E [ V n + 1 ( ε ) ∣ F n ] ≤ V n ( ε ) − 2 α ϕ ( n ) ( x n − Π D ε ( x n ) ) ( K n x n + B n d n ) + ( α ϕ ( n ) ) 2 E [ ∣ F ^ n + 1 c ( ϕ n , θ n ) ∣ 2 ∣ F n ] . (107) \mathbb{E}[V_{n+1}^{(\varepsilon)}\mid\mathcal{F}_n]\le V_n^{(\varepsilon)}-2\alpha_{\phi}(n)(x_n-\Pi_{D_{\varepsilon}}(x_n))(K_nx_n+B_nd_n)
+(\alpha_{\phi}(n))^2\mathbb{E}\left[|\widehat{F}^{c}_{n+1}(\phi_n,\theta_n)|^2\mid\mathcal{F}_n\right]. \tag*{(107)} E [ V n + 1 ( ε ) ∣ F n ] ≤ V n ( ε ) − 2 α ϕ ( n ) ( x n − Π D ε ( x n )) ( K n x n + B n d n ) + ( α ϕ ( n ) ) 2 E [ ∣ F n + 1 c ( ϕ n , θ n ) ∣ 2 ∣ F n ] . (107)
In view of θ n ∈ Θ \theta_n\in\Theta θ n ∈ Θ and (96), ∣ B n d n ∣ → 0 |B_nd_n|\to0 ∣ B n d n ∣ → 0 . Thus one may choose N ε N_{\varepsilon} N ε such that, for all n ≥ N ε n\ge N_{\varepsilon} n ≥ N ε ,
∣ B n d n ∣ ≤ κ c , R 0 ε 2 . |B_nd_n|\le\frac{\kappa_{c,R_0}\varepsilon}{2}. ∣ B n d n ∣ ≤ 2 κ c , R 0 ε .
When ∣ x n ∣ > ε |x_n|>\varepsilon ∣ x n ∣ > ε , x n − Π D ε ( x n ) x_n-\Pi_{D_{\varepsilon}}(x_n) x n − Π D ε ( x n ) has the same sign as x n x_n x n , and ∣ x n ∣ = ∣ x n − Π D ε ( x n ) ∣ + ε |x_n|=|x_n-\Pi_{D_{\varepsilon}}(x_n)|+\varepsilon ∣ x n ∣ = ∣ x n − Π D ε ( x n ) ∣ + ε . Write Δ n ( ε ) : = x n − Π D ε ( x n ) \Delta_n^{(\varepsilon)}:=x_n-\Pi_{D_{\varepsilon}}(x_n) Δ n ( ε ) := x n − Π D ε ( x n ) . It then holds that
Δ n ( ε ) ( K n x n + B n d n ) ≥ κ c , R 0 ∣ Δ n ( ε ) ∣ ( ∣ Δ n ( ε ) ∣ + ε ) − ∣ Δ n ( ε ) ∣ ∣ B n d n ∣ ≥ κ c , R 0 ∣ Δ n ( ε ) ∣ 2 . \Delta_n^{(\varepsilon)}(K_nx_n+B_nd_n)\ge\kappa_{c,R_0}|\Delta_n^{(\varepsilon)}|\left(|\Delta_n^{(\varepsilon)}|+\varepsilon\right)-|\Delta_n^{(\varepsilon)}||B_nd_n|\ge\kappa_{c,R_0}|\Delta_n^{(\varepsilon)}|^2. Δ n ( ε ) ( K n x n + B n d n ) ≥ κ c , R 0 ∣ Δ n ( ε ) ∣ ( ∣ Δ n ( ε ) ∣ + ε ) − ∣ Δ n ( ε ) ∣∣ B n d n ∣ ≥ κ c , R 0 ∣ Δ n ( ε ) ∣ 2 .
When ∣ x n ∣ ≤ ε |x_n|\le\varepsilon ∣ x n ∣ ≤ ε , both sides are zero, so this lower bound holds for all x n x_n x n . On the other hand, by Lemma 4 and ∣ ϕ n ∣ ≤ c n |\phi_n|\le c_n ∣ ϕ n ∣ ≤ c n ,
E [ ∣ F ^ n + 1 c ( ϕ n , θ n ) ∣ 2 ∣ F n ] ≤ C ( 1 + ∣ e 0 ( n ) ∣ 4 + c n 4 ) . (108) \mathbb{E}\left[|\widehat{F}^{c}_{n+1}(\phi_n,\theta_n)|^2\mid\mathcal{F}_n\right]\le C\left(1+|e_0^{(n)}|^4+c_n^4\right). \tag*{(108)} E [ ∣ F n + 1 c ( ϕ n , θ n ) ∣ 2 ∣ F n ] ≤ C ( 1 + ∣ e 0 ( n ) ∣ 4 + c n 4 ) . (108)
By the Gaussian fourth-moment formula, Lemma 1, and (37),
E [ ∣ e 0 ( n ) ∣ 4 ] = μ n 4 + 6 μ n 2 v n + 3 v n 2 ≤ C ( n + 1 ) − 2 α , \mathbb{E}[|e_0^{(n)}|^4]=\mu_n^4+6\mu_n^2v_n+3v_n^2\le C(n+1)^{-2\alpha}, E [ ∣ e 0 ( n ) ∣ 4 ] = μ n 4 + 6 μ n 2 v n + 3 v n 2 ≤ C ( n + 1 ) − 2 α ,
and hence
E [ ∑ n ( α ϕ ( n ) ) 2 ∣ e 0 ( n ) ∣ 4 ] ≤ C ∑ n ( α ϕ ( n ) ) 2 ( n + 1 ) − 2 α < ∞ . \mathbb{E}\left[\sum_n(\alpha_{\phi}(n))^2|e_0^{(n)}|^4\right]\le C\sum_n(\alpha_{\phi}(n))^2(n+1)^{-2\alpha}<\infty. E [ n ∑ ( α ϕ ( n ) ) 2 ∣ e 0 ( n ) ∣ 4 ] ≤ C n ∑ ( α ϕ ( n ) ) 2 ( n + 1 ) − 2 α < ∞.
The deterministic part is controlled by ∑ n ( α ϕ ( n ) ) 2 c n 4 < ∞ \sum_n(\alpha_{\phi}(n))^2c_n^4<\infty ∑ n ( α ϕ ( n ) ) 2 c n 4 < ∞ . Therefore
∑ n C ( α ϕ ( n ) ) 2 ( 1 + ∣ e 0 ( n ) ∣ 4 + c n 4 ) < ∞ a.s. \sum_n C(\alpha_{\phi}(n))^2\left(1+|e_0^{(n)}|^4+c_n^4\right)<\infty\quad\text{a.s.} n ∑ C ( α ϕ ( n ) ) 2 ( 1 + ∣ e 0 ( n ) ∣ 4 + c n 4 ) < ∞ a.s.
Combining Lemma 4 with (107) gives the final recursion
E [ V n + 1 ( ε ) ∣ F n ] ≤ V n ( ε ) − 2 κ c , R 0 α ϕ ( n ) V n ( ε ) + C ( α ϕ ( n ) ) 2 ( 1 + ∣ e 0 ( n ) ∣ 4 + c n 4 ) . \mathbb{E}[V_{n+1}^{(\varepsilon)}\mid\mathcal{F}_n]\le V_n^{(\varepsilon)}-2\kappa_{c,R_0}\alpha_{\phi}(n)V_n^{(\varepsilon)}+C(\alpha_{\phi}(n))^2\left(1+|e_0^{(n)}|^4+c_n^4\right). E [ V n + 1 ( ε ) ∣ F n ] ≤ V n ( ε ) − 2 κ c , R 0 α ϕ ( n ) V n ( ε ) + C ( α ϕ ( n ) ) 2 ( 1 + ∣ e 0 ( n ) ∣ 4 + c n 4 ) .
The Robbins–Siegmund theorem (Robbins and Siegmund, [38 ] ) shows that V n ( ε ) V_n^{(\varepsilon)} V n ( ε ) converges a.s. and ∑ n α ϕ ( n ) V n ( ε ) < ∞ \sum_n\alpha_{\phi}(n)V_n^{(\varepsilon)}<\infty ∑ n α ϕ ( n ) V n ( ε ) < ∞ a.s. As ∑ n α ϕ ( n ) = ∞ \sum_n\alpha_{\phi}(n)=\infty ∑ n α ϕ ( n ) = ∞ , it holds that V n ( ε ) → 0 V_n^{(\varepsilon)}\to0 V n ( ε ) → 0 a.s. Taking the probability-one event simultaneously over the countable sequence ε m = 1 / m \varepsilon_m=1/m ε m = 1/ m gives
lim sup n → ∞ ∣ x n ∣ ≤ 1 m for all m ≥ 1 , \limsup_{n\to\infty}|x_n|\le\frac{1}{m}\quad\text{for all }m\ge1, n → ∞ lim sup ∣ x n ∣ ≤ m 1 for all m ≥ 1 ,
so x n → 0 x_n\to0 x n → 0 a.s., that is, ϕ n → ρ ∗ \phi_n\to\rho^* ϕ n → ρ ∗ a.s.
We next prove the mean-square rate. Fix δ ∈ ( 0 , 2 ) \delta\in(0,2) δ ∈ ( 0 , 2 ) . When γ 2 = 1 \gamma_2=1 γ 2 = 1 , choose δ > 0 \delta>0 δ > 0 sufficiently small that ( 2 − δ ) κ c , R 0 α ϕ > max { 2 α , 1 } (2-\delta)\kappa_{c,R_0}\alpha_{\phi}>\max\{2\alpha,1\} ( 2 − δ ) κ c , R 0 α ϕ > max { 2 α , 1 } . Such a choice is possible because κ c , R 0 α ϕ > max { α , 1 / 2 } \kappa_{c,R_0}\alpha_{\phi}>\max\{\alpha,1/2\} κ c , R 0 α ϕ > max { α , 1/2 } . When 1 / 2 < γ 2 < 1 1/2 < \gamma_2 < 1 1/2 < γ 2 < 1 , any fixed δ ∈ ( 0 , 2 ) \delta\in(0,2) δ ∈ ( 0 , 2 ) may be used. Applying the weighted Young inequality to the critic-bias term gives
2 B n ∣ x n ∣ ∣ d n ∣ ≤ δ K n ∣ x n ∣ 2 + B n 2 δ K n ∣ d n ∣ 2 . 2B_n|x_n|\,|d_n| \le\delta K_n|x_n|^2 + \frac{B_n^2}{\delta K_n}|d_n|^2. 2 B n ∣ x n ∣ ∣ d n ∣ ≤ δ K n ∣ x n ∣ 2 + δ K n B n 2 ∣ d n ∣ 2 .
Therefore,
− 2 x n F n ( ϕ n , θ n ) ≤ − ( 2 − δ ) K n ∣ x n ∣ 2 + B n 2 δ K n ∣ d n ∣ 2 ≤ − ( 2 − δ ) κ c , R 0 ∣ x n ∣ 2 + C ( n + 1 ) − 2 α . -2x_nF_n(\phi_n,\theta_n) \le-(2-\delta)K_n|x_n|^2+\frac{B_n^2}{\delta K_n}|d_n|^2 \le-(2-\delta)\kappa_{c,R_0}|x_n|^2+C(n+1)^{-2\alpha}. − 2 x n F n ( ϕ n , θ n ) ≤ − ( 2 − δ ) K n ∣ x n ∣ 2 + δ K n B n 2 ∣ d n ∣ 2 ≤ − ( 2 − δ ) κ c , R 0 ∣ x n ∣ 2 + C ( n + 1 ) − 2 α .
Substituting (108) and the preceding estimate into the conditional recursion gives
E [ ∣ x n + 1 ∣ 2 ∣ F n ] ≤ ( 1 − ( 2 − δ ) κ c , R 0 α ϕ ( n ) ) ∣ x n ∣ 2 + C α ϕ ( n ) ( n + 1 ) − 2 α + C ( α ϕ ( n ) ) 2 ( 1 + ∣ e 0 ( n ) ∣ 4 + c n 4 ) . (109) \mathbb{E}[|x_{n+1}|^2\mid\mathcal{F}_n]\leq\left(1-(2-\delta)\kappa_{c,R_0}\alpha\phi(n)\right)|x_n|^2+C\alpha\phi(n)(n+1)^{-2\alpha}
+C(\alpha\phi(n))^2\left(1+|e_0^{(n)}|^4+c_n^4\right). \tag*{(109)} E [ ∣ x n + 1 ∣ 2 ∣ F n ] ≤ ( 1 − ( 2 − δ ) κ c , R 0 α ϕ ( n ) ) ∣ x n ∣ 2 + C α ϕ ( n ) ( n + 1 ) − 2 α + C ( α ϕ ( n ) ) 2 ( 1 + ∣ e 0 ( n ) ∣ 4 + c n 4 ) . (109)
Taking full expectation in (109) gives
E [ ∣ x n + 1 ∣ 2 ] ≤ ( 1 − ( 2 − δ ) κ c , R 0 α ϕ ( n ) ) E [ ∣ x n ∣ 2 ] + C α ϕ ( n ) ( n + 1 ) − 2 α + C ( α ϕ ( n ) ) 2 L n . \mathbb{E}[|x_{n+1}|^2]\leq\left(1-(2-\delta)\kappa_{c,R_0}\alpha\phi(n)\right)\mathbb{E}[|x_n|^2]+C\alpha\phi(n)(n+1)^{-2\alpha}+C(\alpha\phi(n))^2L_n. E [ ∣ x n + 1 ∣ 2 ] ≤ ( 1 − ( 2 − δ ) κ c , R 0 α ϕ ( n ) ) E [ ∣ x n ∣ 2 ] + C α ϕ ( n ) ( n + 1 ) − 2 α + C ( α ϕ ( n ) ) 2 L n .
Take N 0 N_0 N 0 sufficiently large so that 0 < ( 2 − δ ) κ c , R 0 α ϕ ( n ) < 1 0<(2-\delta)\kappa_{c,R_0}\alpha\phi(n)<1 0 < ( 2 − δ ) κ c , R 0 α ϕ ( n ) < 1 for all n ≥ N 0 n\geq N_0 n ≥ N 0 . Iterating the above display from N 0 N_0 N 0 to n − 1 n-1 n − 1 gives
E [ ∣ x n ∣ 2 ] ≤ ∏ j = N 0 n − 1 ( 1 − ( 2 − δ ) κ c , R 0 α ϕ ( j ) ) E [ ∣ x N 0 ∣ 2 ] + ∑ k = N 0 n − 1 ∏ j = k + 1 n − 1 ( 1 − ( 2 − δ ) κ c , R 0 α ϕ ( j ) ) [ C α ϕ ( k ) ( k + 1 ) − 2 α + C ( α ϕ ( k ) ) 2 L k ] . \mathbb{E}[|x_n|^2]\leq\prod_{j=N_0}^{n-1}\left(1-(2-\delta)\kappa_{c,R_0}\alpha\phi(j)\right)\mathbb{E}[|x_{N_0}|^2]
+\sum_{k=N_0}^{n-1}\prod_{j=k+1}^{n-1}\left(1-(2-\delta)\kappa_{c,R_0}\alpha\phi(j)\right)\left[C\alpha\phi(k)(k+1)^{-2\alpha}+C(\alpha\phi(k))^2L_k\right]. E [ ∣ x n ∣ 2 ] ≤ j = N 0 ∏ n − 1 ( 1 − ( 2 − δ ) κ c , R 0 α ϕ ( j ) ) E [ ∣ x N 0 ∣ 2 ] + k = N 0 ∑ n − 1 j = k + 1 ∏ n − 1 ( 1 − ( 2 − δ ) κ c , R 0 α ϕ ( j ) ) [ C α ϕ ( k ) ( k + 1 ) − 2 α + C ( α ϕ ( k ) ) 2 L k ] .
First consider 1 / 2 < γ 2 < 1 1/2<\gamma_2<1 1/2 < γ 2 < 1 . Since β = 1 \beta=1 β = 1 and α ϕ ( j ) = α ϕ ( j + 1 ) γ 2 \alpha\phi(j)=\frac{\alpha\phi}{(j+1)^{\gamma_2}} α ϕ ( j ) = ( j + 1 ) γ 2 α ϕ , and since
∏ j = k + 1 n − 1 ( 1 − ( 2 − δ ) κ c , R 0 α ϕ ( j + 1 ) − γ 2 ) ≤ exp { − c ∑ j = k + 1 n − 1 ( j + 1 ) − γ 2 } , \prod_{j=k+1}^{n-1}\left(1-(2-\delta)\kappa_{c,R_0}\alpha\phi(j+1)^{-\gamma_2}\right)\leq\exp\left\{-c\sum_{j=k+1}^{n-1}(j+1)^{-\gamma_2}\right\}, j = k + 1 ∏ n − 1 ( 1 − ( 2 − δ ) κ c , R 0 α ϕ ( j + 1 ) − γ 2 ) ≤ exp ⎩ ⎨ ⎧ − c j = k + 1 ∑ n − 1 ( j + 1 ) − γ 2 ⎭ ⎬ ⎫ ,
since ϕ N 0 ∈ Φ N 0 = [ − c N 0 , c N 0 ] \phi_{N_0}\in\Phi_{N_0}=[-cN_0,cN_0] ϕ N 0 ∈ Φ N 0 = [ − c N 0 , c N 0 ] , E [ ∣ x N 0 ∣ 2 ] ≤ ( c N 0 + ∣ ρ ∗ ∣ ) 2 < ∞ \mathbb{E}[|x_{N_0}|^2]\leq(cN_0+|\rho^*|)^2<\infty E [ ∣ x N 0 ∣ 2 ] ≤ ( c N 0 + ∣ ρ ∗ ∣ ) 2 < ∞ . The initial term is controlled by C exp { − c n 1 − γ 2 } C\exp\{-cn^{1-\gamma_2}\} C exp { − c n 1 − γ 2 } . For the convolution term, if N 0 ≤ k ≤ ⌊ n / 2 ⌋ N_0\leq k\leq\lfloor n/2\rfloor N 0 ≤ k ≤ ⌊ n /2 ⌋ , then likewise ∑ j = k + 1 n − 1 ( j + 1 ) − γ 2 ≥ c n 1 − γ 2 \sum_{j=k+1}^{n-1}(j+1)^{-\gamma_2}\geq cn^{1-\gamma_2} ∑ j = k + 1 n − 1 ( j + 1 ) − γ 2 ≥ c n 1 − γ 2 , so this part is also controlled by C exp { − c n 1 − γ 2 } C\exp\{-cn^{1-\gamma_2}\} C exp { − c n 1 − γ 2 } . If ⌊ n / 2 ⌋ < k ≤ n − 1 \lfloor n/2\rfloor<k\leq n-1 ⌊ n /2 ⌋ < k ≤ n − 1 , then L k ≤ L n L_k\leq L_n L k ≤ L n , and the local discrete kernel satisfies
∑ k = ⌊ n / 2 ⌋ n − 1 exp { − c ∑ j = k + 1 n − 1 ( j + 1 ) − γ 2 } ( k + 1 ) − γ 2 − 2 α ≤ C ( n + 1 ) − 2 α , \sum_{k=\lfloor n/2\rfloor}^{n-1}\exp\left\{-c\sum_{j=k+1}^{n-1}(j+1)^{-\gamma_2}\right\}(k+1)^{-\gamma_2-2\alpha}\leq C(n+1)^{-2\alpha}, k = ⌊ n /2 ⌋ ∑ n − 1 exp ⎩ ⎨ ⎧ − c j = k + 1 ∑ n − 1 ( j + 1 ) − γ 2 ⎭ ⎬ ⎫ ( k + 1 ) − γ 2 − 2 α ≤ C ( n + 1 ) − 2 α ,
∑ k = ⌊ n / 2 ⌋ n − 1 exp { − c ∑ j = k + 1 n − 1 ( j + 1 ) − γ 2 } L k ( k + 1 ) − 2 γ 2 ≤ C L n ( n + 1 ) − γ 2 . \sum_{k=\lfloor n/2\rfloor}^{n-1}\exp\left\{-c\sum_{j=k+1}^{n-1}(j+1)^{-\gamma_2}\right\}L_k(k+1)^{-2\gamma_2}\leq CL_n(n+1)^{-\gamma_2}. k = ⌊ n /2 ⌋ ∑ n − 1 exp ⎩ ⎨ ⎧ − c j = k + 1 ∑ n − 1 ( j + 1 ) − γ 2 ⎭ ⎬ ⎫ L k ( k + 1 ) − 2 γ 2 ≤ C L n ( n + 1 ) − γ 2 .
The two estimates here come from the same local scale: when k k k is close to n n n , the effective window length of ∑ j = k + 1 n − 1 ( j + 1 ) − γ 2 \sum_{j=k+1}^{n-1}(j+1)^{-\gamma_2} ∑ j = k + 1 n − 1 ( j + 1 ) − γ 2 is ( n + 1 ) γ 2 (n+1)^{\gamma_2} ( n + 1 ) γ 2 . Combining the terms above and absorbing the stretched exponential term into an arbitrarily high-order polynomial decay term, we obtain
E [ ∣ x n ∣ 2 ] ≤ C L n [ ( n + 1 ) − 2 α + ( n + 1 ) − γ 2 ] . \mathbb{E}[|x_n|^2]\leq CL_n\left[(n+1)^{-2\alpha}+(n+1)^{-\gamma_2}\right]. E [ ∣ x n ∣ 2 ] ≤ C L n [ ( n + 1 ) − 2 α + ( n + 1 ) − γ 2 ] .
When γ 2 = 1 \gamma_2=1 γ 2 = 1 , the same expansion degenerates to the harmonic case. By the choice of δ \delta δ above, ( 2 − δ ) κ c , R 0 α ϕ > max { 2 α , 1 } (2-\delta)\kappa_{c,R_0}\alpha\phi>\max\{2\alpha,1\} ( 2 − δ ) κ c , R 0 α ϕ > max { 2 α , 1 } , and
∏ j = k + 1 n − 1 ( 1 − ( 2 − δ ) κ c , R 0 α ϕ ( j + 1 ) − 1 ) ≤ C ( k + 2 n + 1 ) ( 2 − δ ) κ c , R 0 α ϕ , \prod_{j=k+1}^{n-1}\left(1-(2-\delta)\kappa_{c,R_0}\alpha\phi(j+1)^{-1}\right)\leq C\left(\frac{k+2}{n+1}\right)^{(2-\delta)\kappa_{c,R_0}\alpha\phi}, j = k + 1 ∏ n − 1 ( 1 − ( 2 − δ ) κ c , R 0 α ϕ ( j + 1 ) − 1 ) ≤ C ( n + 1 k + 2 ) ( 2 − δ ) κ c , R 0 α ϕ ,
therefore the initial term is controlled by C ( n + 1 ) − ( 2 − δ ) κ c , R 0 α ϕ C(n+1)^{-(2-\delta)\kappa_{c,R_0}\alpha\phi} C ( n + 1 ) − ( 2 − δ ) κ c , R 0 α ϕ , and
∑ k = N 0 n − 1 ( k + 2 n + 1 ) ( 2 − δ ) κ c , R 0 α ϕ ( k + 1 ) − 1 − 2 α ≤ C ( n + 1 ) − 2 α , \sum_{k=N_0}^{n-1}\left(\frac{k+2}{n+1}\right)^{(2-\delta)\kappa_{c,R_0}\alpha\phi}(k+1)^{-1-2\alpha}\leq C(n+1)^{-2\alpha}, k = N 0 ∑ n − 1 ( n + 1 k + 2 ) ( 2 − δ ) κ c , R 0 α ϕ ( k + 1 ) − 1 − 2 α ≤ C ( n + 1 ) − 2 α ,
∑ k = N 0 n − 1 ( k + 2 n + 1 ) ( 2 − δ ) κ c , R 0 α ϕ L k ( k + 1 ) − 2 ≤ C L n ( n + 1 ) − 1 . \sum_{k=N_0}^{n-1}\left(\frac{k+2}{n+1}\right)^{(2-\delta)\kappa_{c,R_0}\alpha\phi}L_k(k+1)^{-2}\leq CL_n(n+1)^{-1}. k = N 0 ∑ n − 1 ( n + 1 k + 2 ) ( 2 − δ ) κ c , R 0 α ϕ L k ( k + 1 ) − 2 ≤ C L n ( n + 1 ) − 1 .
Here the first term uses ( 2 − δ ) κ c , R 0 α ϕ > 2 α (2-\delta)\kappa_{c,R_0}\alpha_{\phi}>2\alpha ( 2 − δ ) κ c , R 0 α ϕ > 2 α , and the second term uses ( 2 − δ ) κ c , R 0 α ϕ > 1 (2-\delta)\kappa_{c,R_0}\alpha_{\phi}>1 ( 2 − δ ) κ c , R 0 α ϕ > 1 . Hence, when γ 2 = 1 \gamma_2=1 γ 2 = 1 , we likewise have
E [ ∣ x n ∣ 2 ] ≤ C L n [ ( n + 1 ) − 2 α + ( n + 1 ) − γ 2 ] . \mathbb{E}[|x_n|^2]\le CL_n\left[(n+1)^{-2\alpha}+(n+1)^{-\gamma_2}\right]. E [ ∣ x n ∣ 2 ] ≤ C L n [ ( n + 1 ) − 2 α + ( n + 1 ) − γ 2 ] .
Furthermore, let r n : = L n [ ( n + 1 ) − α + ( n + 1 ) − γ 2 / 2 ] r_n:=\sqrt{L_n\left[(n+1)^{-\alpha}+(n+1)^{-\gamma_2/2}\right]} r n := L n [ ( n + 1 ) − α + ( n + 1 ) − γ 2 /2 ] . For any M > 0 M>0 M > 0 , Markov’s inequality gives
P ( ∣ ϕ n − ρ ∗ ∣ > M r n ) ≤ E [ ∣ ϕ n − ρ ∗ ∣ 2 ] M 2 r n 2 ≤ C M 2 . \mathbb{P}\left(|\phi_n-\rho^*|>Mr_n\right)\le\frac{\mathbb{E}[|\phi_n-\rho^*|^2]}{M^2r_n^2}\le\frac{C}{M^2}. P ( ∣ ϕ n − ρ ∗ ∣ > M r n ) ≤ M 2 r n 2 E [ ∣ ϕ n − ρ ∗ ∣ 2 ] ≤ M 2 C .
Letting M → ∞ M\to\infty M → ∞ gives
ϕ n − ρ ∗ = O p ( L n [ ( n + 1 ) − α + ( n + 1 ) − γ 2 / 2 ] ) . (110) \phi_n-\rho^*=O_p\left(\sqrt{L_n}\left[(n+1)^{-\alpha}+(n+1)^{-\gamma_2/2}\right]\right). \tag*{(110)} ϕ n − ρ ∗ = O p ( L n [ ( n + 1 ) − α + ( n + 1 ) − γ 2 /2 ] ) . (110)
Proof of Theorem 5
Proof. First prove the coefficient estimates. By (78) , both required estimates hold uniformly for 0 ≤ t ≤ T 0\le t\le T 0 ≤ t ≤ T . Hence
a t ( n ) − a c = 2 p c ( τ t ( n ) ) c + 2 = O ( ( n + 1 ) − α ) , a_t^{(n)}-a_c=\frac{2p_c(\tau_t^{(n)})}{c+2}=O((n+1)^{-\alpha}), a t ( n ) − a c = c + 2 2 p c ( τ t ( n ) ) = O (( n + 1 ) − α ) ,
and
b t ( n ) − b c = 2 ( h c ( τ t ( n ) ) − 1 ) c + 2 = O ( ( n + 1 ) − α ) . b_t^{(n)}-b_c=\frac{2(h_c(\tau_t^{(n)})-1)}{c+2}=O((n+1)^{-\alpha}). b t ( n ) − b c = c + 2 2 ( h c ( τ t ( n ) ) − 1 ) = O (( n + 1 ) − α ) .
We next estimate M t ϕ n − ρ ∗ M_t^{\phi_n}-\rho^* M t ϕ n − ρ ∗ . By (72) ,
M t ϕ n − ρ ∗ = ( a t ( n ) + b t ( n ) − 1 ) ρ ∗ + a t ( n ) p n p n + t ϵ 0 ( n ) + a t ( n ) p n + t W t ( n ) + b t ( n ) ( ϕ n − ρ ∗ ) . M_t^{\phi_n}-\rho^*=(a_t^{(n)}+b_t^{(n)}-1)\rho^*+\frac{a_t^{(n)}p_n}{p_n+t}\epsilon_0^{(n)}+\frac{a_t^{(n)}}{p_n+t}W_t^{(n)}+b_t^{(n)}(\phi_n-\rho^*). M t ϕ n − ρ ∗ = ( a t ( n ) + b t ( n ) − 1 ) ρ ∗ + p n + t a t ( n ) p n ϵ 0 ( n ) + p n + t a t ( n ) W t ( n ) + b t ( n ) ( ϕ n − ρ ∗ ) .
By the two coefficient estimates above and a c + b c = 1 a_c+b_c=1 a c + b c = 1 ,
∣ a t ( n ) + b t ( n ) − 1 ∣ ≤ ∣ a t ( n ) − a c ∣ + ∣ b t ( n ) − b c ∣ ≤ C ( n + 1 ) − α . |a_t^{(n)}+b_t^{(n)}-1|\le|a_t^{(n)}-a_c|+|b_t^{(n)}-b_c|\le C(n+1)^{-\alpha}. ∣ a t ( n ) + b t ( n ) − 1∣ ≤ ∣ a t ( n ) − a c ∣ + ∣ b t ( n ) − b c ∣ ≤ C ( n + 1 ) − α .
Then, by ( z 1 + ⋯ + z 4 ) 2 ≤ 4 ∑ i z i 2 (z_1+\cdots+z_4)^2\le4\sum_i z_i^2 ( z 1 + ⋯ + z 4 ) 2 ≤ 4 ∑ i z i 2 and the uniform boundedness of a t ( n ) a_t^{(n)} a t ( n ) , b t ( n ) b_t^{(n)} b t ( n ) , and p n / ( p n + t ) p_n/(p_n+t) p n / ( p n + t ) , we obtain
E ∣ M t ϕ n − ρ ∗ ∣ 2 ≤ C ( n + 1 ) − 2 α + C E [ ( ϵ 0 ( n ) ) 2 ] + C E [ ( W t ( n ) ) 2 ( p n + t ) 2 ] + C E ∣ ϕ n − ρ ∗ ∣ 2 ≤ C ( n + 1 ) − 2 α + C E [ ( ϵ 0 ( n ) ) 2 ] + C ( R 0 ( n ) ) 2 E [ ( W t ( n ) ) 2 ] + C E ∣ ϕ n − ρ ∗ ∣ 2 . \begin{aligned}
\mathbb{E}|M_t^{\phi_n}-\rho^*|^2\le C(n+1)^{-2\alpha}+C\mathbb{E}[(\epsilon_0^{(n)})^2]+C\mathbb{E}\left[\frac{(W_t^{(n)})^2}{(p_n+t)^2}\right]+C\mathbb{E}|\phi_n-\rho^*|^2 \\
&\le C(n+1)^{-2\alpha}+C\mathbb{E}[(\epsilon_0^{(n)})^2]+C(R_0^{(n)})^2\mathbb{E}[(W_t^{(n)})^2]+C\mathbb{E}|\phi_n-\rho^*|^2.
\end{aligned} E ∣ M t ϕ n − ρ ∗ ∣ 2 ≤ C ( n + 1 ) − 2 α + C E [( ϵ 0 ( n ) ) 2 ] + C E [ ( p n + t ) 2 ( W t ( n ) ) 2 ] + C E ∣ ϕ n − ρ ∗ ∣ 2 ≤ C ( n + 1 ) − 2 α + C E [( ϵ 0 ( n ) ) 2 ] + C ( R 0 ( n ) ) 2 E [( W t ( n ) ) 2 ] + C E ∣ ϕ n − ρ ∗ ∣ 2 .
Here we used ( p n + t ) − 1 ≤ p n − 1 = R 0 ( n ) (p_n+t)^{-1}\le p_n^{-1}=R_0^{(n)} ( p n + t ) − 1 ≤ p n − 1 = R 0 ( n ) . Since E [ ( W t ( n ) ) 2 ] = t ≤ T \mathbb{E}[(W_t^{(n)})^2]=t\le T E [( W t ( n ) ) 2 ] = t ≤ T , by Lemma 1 and (39) ,
E ∣ M t ϕ n − ρ ∗ ∣ 2 ≤ C ( n + 1 ) − 2 α + C ( n + 1 ) − α + C L n [ ( n + 1 ) − 2 α + ( n + 1 ) − γ 2 ] . \mathbb{E}|M_t^{\phi_n}-\rho^*|^2\le C(n+1)^{-2\alpha}+C(n+1)^{-\alpha}+CL_n\left[(n+1)^{-2\alpha}+(n+1)^{-\gamma_2}\right]. E ∣ M t ϕ n − ρ ∗ ∣ 2 ≤ C ( n + 1 ) − 2 α + C ( n + 1 ) − α + C L n [ ( n + 1 ) − 2 α + ( n + 1 ) − γ 2 ] .
Since L n ≥ 1 L_n\ge1 L n ≥ 1 and ( n + 1 ) − 2 α ≤ ( n + 1 ) − α (n+1)^{-2\alpha}\le(n+1)^{-\alpha} ( n + 1 ) − 2 α ≤ ( n + 1 ) − α , the right-hand side can be controlled by
C L n [ 1 ( n + 1 ) α + 1 ( n + 1 ) γ 2 ] . CL_n\left[\frac{1}{(n+1)^\alpha}+\frac{1}{(n+1)^{\gamma_2}}\right]. C L n [ ( n + 1 ) α 1 + ( n + 1 ) γ 2 1 ] .
This proves (41) . Finally, the right-hand side of (41) converges to zero because α , γ 2 > 0 \alpha,\gamma_2>0 α , γ 2 > 0 and L n L_n L n grows only at order log log n \log\log n log log n . Thus the Cauchy–Schwarz inequality gives
∣ E [ M t ϕ n ] − ρ ∗ ∣ = ∣ E [ M t ϕ n − ρ ∗ ] ∣ ≤ ( E ∣ M t ϕ n − ρ ∗ ∣ 2 ) 1 / 2 ⟶ 0. \left|\mathbb{E}[M_t^{\phi_n}]-\rho^*\right|=\left|\mathbb{E}[M_t^{\phi_n}-\rho^*]\right|\le\left(\mathbb{E}|M_t^{\phi_n}-\rho^*|^2\right)^{1/2}\longrightarrow0. E [ M t ϕ n ] − ρ ∗ = E [ M t ϕ n − ρ ∗ ] ≤ ( E ∣ M t ϕ n − ρ ∗ ∣ 2 ) 1/2 ⟶ 0.
At the same time,
Var ( M t ϕ n ) = E ∣ M t ϕ n − E [ M t ϕ n ] ∣ 2 ≤ E ∣ M t ϕ n − ρ ∗ ∣ 2 ⟶ 0. \operatorname{Var}(M_t^{\phi_n})=\mathbb{E}\left|M_t^{\phi_n}-\mathbb{E}[M_t^{\phi_n}]\right|^2\le\mathbb{E}|M_t^{\phi_n}-\rho^*|^2\longrightarrow0. Var ( M t ϕ n ) = E M t ϕ n − E [ M t ϕ n ] 2 ≤ E ∣ M t ϕ n − ρ ∗ ∣ 2 ⟶ 0.
Proof of Lemma 2
Proof. We first compute the derivative of S R ∞ SR_\infty S R ∞ at ρ ∗ \rho^* ρ ∗ . At M = ρ ∗ M=\rho^* M = ρ ∗ , we have S R ∞ ( ρ ∗ ) = e ( ρ ∗ ) 2 T − 1 = S R o r SR_\infty(\rho^*)=\sqrt{e^{(\rho^*)^2T}-1}=SR_{\rm or} S R ∞ ( ρ ∗ ) = e ( ρ ∗ ) 2 T − 1 = S R or . Let h ( M ) : = S R ∞ ′ ( M ) / S R ∞ ( M ) h(M):=SR'_\infty(M)/SR_\infty(M) h ( M ) := S R ∞ ′ ( M ) / S R ∞ ( M ) . Using the explicit formula of S R ∞ SR_\infty S R ∞ , we get
h ( M ) = ρ ∗ T e ρ ∗ M T e ρ ∗ M T − 1 − M T e M 2 T e M 2 T − 1 . (111) h(M)=\frac{\rho^*T e^{\rho^*MT}}{e^{\rho^*MT}-1}-\frac{MT e^{M^2T}}{e^{M^2T}-1}. \tag*{(111)} h ( M ) = e ρ ∗ M T − 1 ρ ∗ T e ρ ∗ M T − e M 2 T − 1 M T e M 2 T . (111)
At M = ρ ∗ M=\rho^* M = ρ ∗ , h ( ρ ∗ ) = 0 h(\rho^*)=0 h ( ρ ∗ ) = 0 , and hence S R ∞ ′ ( ρ ∗ ) = 0 SR'_\infty(\rho^*)=0 S R ∞ ′ ( ρ ∗ ) = 0 . We next compute the second derivative. Differentiating (111) leads to
h ′ ( M ) = − ( ρ ∗ ) 2 T 2 e ρ ∗ M T ( e ρ ∗ M T − 1 ) 2 − T e M 2 T e M 2 T − 1 + 2 M 2 T 2 e M 2 T ( e M 2 T − 1 ) 2 . h'(M)=-\frac{(\rho^*)^2T^2e^{\rho^*MT}}{(e^{\rho^*MT}-1)^2}-\frac{Te^{M^2T}}{e^{M^2T}-1}+\frac{2M^2T^2e^{M^2T}}{(e^{M^2T}-1)^2}. h ′ ( M ) = − ( e ρ ∗ M T − 1 ) 2 ( ρ ∗ ) 2 T 2 e ρ ∗ M T − e M 2 T − 1 T e M 2 T + ( e M 2 T − 1 ) 2 2 M 2 T 2 e M 2 T .
Setting M = ρ ∗ M=\rho^* M = ρ ∗ , we get
h ′ ( ρ ∗ ) = ( ρ ∗ ) 2 T 2 e ( ρ ∗ ) 2 T ( e ( ρ ∗ ) 2 T − 1 ) 2 − T e ( ρ ∗ ) 2 T e ( ρ ∗ ) 2 T − 1 = − T e ( ρ ∗ ) 2 T ( e ( ρ ∗ ) 2 T − 1 − ( ρ ∗ ) 2 T ) ( e ( ρ ∗ ) 2 T − 1 ) 2 . h'(\rho^*)=\frac{(\rho^*)^2T^2e^{(\rho^*)^2T}}{(e^{(\rho^*)^2T}-1)^2}-\frac{Te^{(\rho^*)^2T}}{e^{(\rho^*)^2T}-1}=-T\frac{e^{(\rho^*)^2T}\left(e^{(\rho^*)^2T}-1-(\rho^*)^2T\right)}{(e^{(\rho^*)^2T}-1)^2}. h ′ ( ρ ∗ ) = ( e ( ρ ∗ ) 2 T − 1 ) 2 ( ρ ∗ ) 2 T 2 e ( ρ ∗ ) 2 T − e ( ρ ∗ ) 2 T − 1 T e ( ρ ∗ ) 2 T = − T ( e ( ρ ∗ ) 2 T − 1 ) 2 e ( ρ ∗ ) 2 T ( e ( ρ ∗ ) 2 T − 1 − ( ρ ∗ ) 2 T ) .
On the other hand, as S R ∞ ′ = S R ∞ h SR'_\infty=SR_\infty h S R ∞ ′ = S R ∞ h , it holds that S R ∞ ′ ′ = S R ∞ ( h 2 + h ′ ) SR''_\infty=SR_\infty(h^2+h') S R ∞ ′′ = S R ∞ ( h 2 + h ′ ) . From h ( ρ ∗ ) = 0 h(\rho^*)=0 h ( ρ ∗ ) = 0 and S R ∞ ( ρ ∗ ) = e ( ρ ∗ ) 2 T − 1 SR_\infty(\rho^*)=\sqrt{e^{(\rho^*)^2T}-1} S R ∞ ( ρ ∗ ) = e ( ρ ∗ ) 2 T − 1 , it follows that
S R ∞ ′ ′ ( ρ ∗ ) = − T e ( ρ ∗ ) 2 T ( e ( ρ ∗ ) 2 T − 1 − ( ρ ∗ ) 2 T ) ( e ( ρ ∗ ) 2 T − 1 ) 3 / 2 . SR''_\infty(\rho^*)=-T\frac{e^{(\rho^*)^2T}\left(e^{(\rho^*)^2T}-1-(\rho^*)^2T\right)}{(e^{(\rho^*)^2T}-1)^{3/2}}. S R ∞ ′′ ( ρ ∗ ) = − T ( e ( ρ ∗ ) 2 T − 1 ) 3/2 e ( ρ ∗ ) 2 T ( e ( ρ ∗ ) 2 T − 1 − ( ρ ∗ ) 2 T ) .
Strict convexity ensures e ( ρ ∗ ) 2 T − 1 − ( ρ ∗ ) 2 T > 0 e^{(\rho^*)^2T}-1-(\rho^*)^2T>0 e ( ρ ∗ ) 2 T − 1 − ( ρ ∗ ) 2 T > 0 , and it thus holds that S R ∞ ′ ′ ( ρ ∗ ) < 0 SR''_\infty(\rho^*)<0 S R ∞ ′′ ( ρ ∗ ) < 0 .
We next prove the local Taylor bound. In view of ρ ∗ ≠ 0 \rho^*\ne0 ρ ∗ = 0 , when ∣ M − ρ ∗ ∣ ≤ ∣ ρ ∗ ∣ / 4 |M-\rho^*|\le|\rho^*|/4 ∣ M − ρ ∗ ∣ ≤ ∣ ρ ∗ ∣/4 we have ∣ M ∣ ≥ 3 ∣ ρ ∗ ∣ / 4 > 0 |M|\ge3|\rho^*|/4>0 ∣ M ∣ ≥ 3∣ ρ ∗ ∣/4 > 0 . Hence S R ∞ SR_\infty S R ∞ is twice continuously differentiable on this closed neighborhood, and S R ∞ ′ ′ SR''_\infty S R ∞ ′′ is bounded. It holds by Taylor’s theorem that
∣ S R ∞ ( M ) − S R o r ∣ ≤ 1 2 sup ∣ z − ρ ∗ ∣ ≤ ∣ ρ ∗ ∣ / 4 ∣ S R ∞ ′ ′ ( z ) ∣ ∣ M − ρ ∗ ∣ 2 . |SR_\infty(M)-SR_{\rm or}|\le\frac{1}{2}\sup_{|z-\rho^*|\le|\rho^*|/4}|SR''_\infty(z)||M-\rho^*|^2. ∣ S R ∞ ( M ) − S R or ∣ ≤ 2 1 ∣ z − ρ ∗ ∣ ≤ ∣ ρ ∗ ∣/4 sup ∣ S R ∞ ′′ ( z ) ∣∣ M − ρ ∗ ∣ 2 .
Finally, we prove the ODE stability estimate. To make the continuous-dependence argument fully explicit, we write the coefficient ODEs from Section 4.2 below. For short, we write
U ( n ) : = ( p G ( n ) , q G ( n ) , A H ( n ) , B H ( n ) , C H ( n ) ) U^{(n)}:=\left(p_G^{(n)},q_G^{(n)},A_H^{(n)},B_H^{(n)},C_H^{(n)}\right) U ( n ) := ( p G ( n ) , q G ( n ) , A H ( n ) , B H ( n ) , C H ( n ) )
and omit the time subscript. It satisfies the common terminal condition U ( n ) ( T ) = 0 U^{(n)}(T)=0 U ( n ) ( T ) = 0 and
p ˙ G = R ( 1 + a ) p G + ρ ∗ a , \dot{p}_G=R(1+a)p_G+\rho^*a, p ˙ G = R ( 1 + a ) p G + ρ ∗ a ,
q ˙ G = − 1 2 R 2 p G 2 − R ( ρ ∗ − b ϕ ) p G + ρ ∗ b ϕ , \dot{q}_G=-\frac{1}{2}R^2p_G^2-R(\rho^*-b\phi)p_G+\rho^*b\phi, q ˙ G = − 2 1 R 2 p G 2 − R ( ρ ∗ − b ϕ ) p G + ρ ∗ b ϕ ,
A ˙ H = − R 2 A H 2 + 2 R ( 1 + 2 a ) A H − 2 a 2 , \dot{A}_H=-R^2A_H^2+2R(1+2a)A_H-2a^2, A ˙ H = − R 2 A H 2 + 2 R ( 1 + 2 a ) A H − 2 a 2 ,
B ˙ H = − R 2 A H B H − R ( ρ ∗ − 2 b ϕ ) A H + R ( 1 + 2 a ) B H − 2 a b ϕ + 2 ρ ∗ a , \dot{B}_H=-R^2A_HB_H-R(\rho^*-2b\phi)A_H+R(1+2a)B_H-2ab\phi+2\rho^*a, B ˙ H = − R 2 A H B H − R ( ρ ∗ − 2 b ϕ ) A H + R ( 1 + 2 a ) B H − 2 ab ϕ + 2 ρ ∗ a ,
C ˙ H = − 1 2 R 2 ( A H + B H 2 ) − R ( ρ ∗ − 2 b ϕ ) B H − b 2 ϕ 2 − V + 2 ρ ∗ b ϕ . \dot{C}_H=-\frac{1}{2}R^2(A_H+B_H^2)-R(\rho^*-2b\phi)B_H-b^2\phi^2-V+2\rho^*b\phi. C ˙ H = − 2 1 R 2 ( A H + B H 2 ) − R ( ρ ∗ − 2 b ϕ ) B H − b 2 ϕ 2 − V + 2 ρ ∗ b ϕ .
Here R = R t ( n ) R=R_t^{(n)} R = R t ( n ) , V = V t ( n ) V=V_t^{(n)} V = V t ( n ) , a = a t ( n ) a=a_t^{(n)} a = a t ( n ) , and b = b t ( n ) b=b_t^{(n)} b = b t ( n ) . Let F ( n ) ( t , U ; ϕ ) F^{(n)}(t,U;\phi) F ( n ) ( t , U ; ϕ ) denote the right-hand side of the ODE above. Substituting R = V = 0 R=V=0 R = V = 0 , a = a c a=a_c a = a c , and b = b c b=b_c b = b c gives the limiting vector field
F ( ∞ ) ( t , U ; ϕ ) = ( ρ ∗ a c , ρ ∗ b c ϕ , − 2 a c 2 , 2 a c ( ρ ∗ − b c ϕ ) , 2 ρ ∗ b c ϕ − b c 2 ϕ 2 ) , F^{(\infty)}(t,U;\phi)=\left(\rho^*a_c,\rho^*b_c\phi,-2a_c^2,2a_c(\rho^*-b_c\phi),2\rho^*b_c\phi-b_c^2\phi^2\right), F ( ∞ ) ( t , U ; ϕ ) = ( ρ ∗ a c , ρ ∗ b c ϕ , − 2 a c 2 , 2 a c ( ρ ∗ − b c ϕ ) , 2 ρ ∗ b c ϕ − b c 2 ϕ 2 ) ,
where the superscript ( ∞ ) (\infty) ( ∞ ) denotes the limiting system with R = V = 0 R=V=0 R = V = 0 . This vector field is independent of U U U , and the explicit solution under the same terminal condition is
p G ( ∞ ) ( t ) = − ρ ∗ a c ( T − t ) , (112) p_G^{(\infty)}(t)=-\rho^*a_c(T-t), \tag*{(112)} p G ( ∞ ) ( t ) = − ρ ∗ a c ( T − t ) , (112)
q G ( ∞ ) ( t ) = − ρ ∗ b c ϕ ( T − t ) , (113) q_G^{(\infty)}(t)=-\rho^*b_c\phi(T-t), \tag*{(113)} q G ( ∞ ) ( t ) = − ρ ∗ b c ϕ ( T − t ) , (113)
A H ( ∞ ) ( t ) = 2 a c 2 ( T − t ) , (114) A_H^{(\infty)}(t)=2a_c^2(T-t), \tag*{(114)} A H ( ∞ ) ( t ) = 2 a c 2 ( T − t ) , (114)
B H ( ∞ ) ( t ) = − 2 a c ( ρ ∗ − b c ϕ ) ( T − t ) , (115) B_H^{(\infty)}(t)=-2a_c(\rho^*-b_c\phi)(T-t), \tag*{(115)} B H ( ∞ ) ( t ) = − 2 a c ( ρ ∗ − b c ϕ ) ( T − t ) , (115)
C H ( ∞ ) ( t ) = ( b c 2 ϕ 2 − 2 ρ ∗ b c ϕ ) ( T − t ) , (116) C_H^{(\infty)}(t)=(b_c^2\phi^2-2\rho^*b_c\phi)(T-t), \tag*{(116)} C H ( ∞ ) ( t ) = ( b c 2 ϕ 2 − 2 ρ ∗ b c ϕ ) ( T − t ) , (116)
and write U ( ∞ ) : = ( p G ( ∞ ) , q G ( ∞ ) , A H ( ∞ ) , B H ( ∞ ) , C H ( ∞ ) ) U^{(\infty)} := (p_G^{(\infty)}, q_G^{(\infty)}, A_H^{(\infty)}, B_H^{(\infty)}, C_H^{(\infty)}) U ( ∞ ) := ( p G ( ∞ ) , q G ( ∞ ) , A H ( ∞ ) , B H ( ∞ ) , C H ( ∞ ) ) .
Let M : = M c ( m , ϕ ) = a c m + b c ϕ M := M_c(m,\phi) = a_cm + b_c\phi M := M c ( m , ϕ ) = a c m + b c ϕ . By (112)–(116), the normalized first and second moment factors corresponding to the limiting system are
G ( ∞ ) ( m , ϕ ) = exp { − ρ ∗ M T } , H ( ∞ ) ( m , ϕ ) = exp { M 2 T − 2 ρ ∗ M T } . G^{(\infty)}(m,\phi) = \exp\{-\rho^*MT\}, \qquad H^{(\infty)}(m,\phi) = \exp\{M^2T - 2\rho^*MT\}. G ( ∞ ) ( m , ϕ ) = exp { − ρ ∗ M T } , H ( ∞ ) ( m , ϕ ) = exp { M 2 T − 2 ρ ∗ M T } .
At the oracle point ( m , ϕ ) = ( ρ ∗ , ρ ∗ ) (m,\phi) = (\rho^*,\rho^*) ( m , ϕ ) = ( ρ ∗ , ρ ∗ ) , we have M = ρ ∗ M = \rho^* M = ρ ∗ , and hence
H ( ∞ ) − ( G ( ∞ ) ) 2 = e − 2 ( ρ ∗ ) 2 T ( e ( ρ ∗ ) 2 T − 1 ) > 0. H^{(\infty)} - \left(G^{(\infty)}\right)^2 = e^{-2(\rho^*)^2T}\left(e^{(\rho^*)^2T}-1\right) > 0. H ( ∞ ) − ( G ( ∞ ) ) 2 = e − 2 ( ρ ∗ ) 2 T ( e ( ρ ∗ ) 2 T − 1 ) > 0.
Since H ( ∞ ) ( m , ϕ ) − ( G ( ∞ ) ( m , ϕ ) ) 2 H^{(\infty)}(m,\phi) - \left(G^{(\infty)}(m,\phi)\right)^2 H ( ∞ ) ( m , ϕ ) − ( G ( ∞ ) ( m , ϕ ) ) 2 is continuous in ( m , ϕ ) (m,\phi) ( m , ϕ ) , one may choose δ O D E > 0 \delta_{\mathrm{ODE}} > 0 δ ODE > 0 and d 0 > 0 d_0 > 0 d 0 > 0 such that, whenever ∣ m − ρ ∗ ∣ ∨ ∣ ϕ − ρ ∗ ∣ ≤ δ O D E |m-\rho^*| \vee|\phi-\rho^*| \le\delta_{\mathrm{ODE}} ∣ m − ρ ∗ ∣ ∨ ∣ ϕ − ρ ∗ ∣ ≤ δ ODE , H ( ∞ ) ( m , ϕ ) − ( G ( ∞ ) ( m , ϕ ) ) 2 ≥ 2 d 0 H^{(\infty)}(m,\phi) - \left(G^{(\infty)}(m,\phi)\right)^2 \ge2d_0 H ( ∞ ) ( m , ϕ ) − ( G ( ∞ ) ( m , ϕ ) ) 2 ≥ 2 d 0 . In what follows, fix the neighborhood selected in this way: ∣ m − ρ ∗ ∣ ∨ ∣ ϕ − ρ ∗ ∣ ≤ δ O D E |m-\rho^*| \vee|\phi-\rho^*| \le\delta_{\mathrm{ODE}} ∣ m − ρ ∗ ∣ ∨ ∣ ϕ − ρ ∗ ∣ ≤ δ ODE . By continuity of the explicit solution above with respect to ( t , ϕ ) (t,\phi) ( t , ϕ ) , the family of trajectories Γ ∞ : = { U ( ∞ ) ( t ; ϕ ) : 0 ≤ t ≤ T , ∣ ϕ − ρ ∗ ∣ ≤ δ O D E } \Gamma_\infty:= \{U^{(\infty)}(t;\phi): 0 \le t \le T,\ |\phi-\rho^*| \le\delta_{\mathrm{ODE}}\} Γ ∞ := { U ( ∞ ) ( t ; ϕ ) : 0 ≤ t ≤ T , ∣ ϕ − ρ ∗ ∣ ≤ δ ODE } is compact. Choose ε K > 0 \varepsilon_K > 0 ε K > 0 and define the closed tubular neighborhood K : = { U ∈ R 5 : dist ( U , Γ ∞ ) ≤ ε K } \mathcal{K} := \{U \in\mathbb{R}^5 : \operatorname{dist}(U,\Gamma_\infty) \le\varepsilon_K\} K := { U ∈ R 5 : dist ( U , Γ ∞ ) ≤ ε K } . As Γ ∞ \Gamma_\infty Γ ∞ is compact, K \mathcal{K} K is also compact. Combining the uniform bounds on R R R , V V V , a a a , b b b with boundedness of K \mathcal{K} K , the polynomial vector field in U U U has a uniformly bounded Jacobian. Therefore there exists L < ∞ L < \infty L < ∞ , independent of n n n , t t t , and ϕ \phi ϕ in the neighborhood, such that for U , U ~ ∈ K U,\widetilde U \in\mathcal{K} U , U ∈ K ,
∣ F ( n ) ( t , U ; ϕ ) − F ( n ) ( t , U ~ ; ϕ ) ∣ ≤ L ∣ U − U ~ ∣ . \left|F^{(n)}(t,U;\phi)-F^{(n)}(t,\widetilde U;\phi)\right| \le L|U-\widetilde U|. F ( n ) ( t , U ; ϕ ) − F ( n ) ( t , U ; ϕ ) ≤ L ∣ U − U ∣.
On the other hand, using (71), (77), and (40) term by term, for U ∈ K U \in\mathcal{K} U ∈ K we have
∣ F p G ( n ) − F p G ( ∞ ) ∣ ≤ C ( R ∣ p G ∣ + ∣ a − a c ∣ ) , \left|F_{p_G}^{(n)}-F_{p_G}^{(\infty)}\right| \le C\left(R|p_G|+|a-a_c|\right), F p G ( n ) − F p G ( ∞ ) ≤ C ( R ∣ p G ∣ + ∣ a − a c ∣ ) ,
∣ F q G ( n ) − F q G ( ∞ ) ∣ ≤ C ( R 2 ∣ p G ∣ 2 + R ( 1 + ∣ p G ∣ ) + ∣ b − b c ∣ ) , \left|F_{q_G}^{(n)}-F_{q_G}^{(\infty)}\right| \le C\left(R^2|p_G|^2+R(1+|p_G|)+|b-b_c|\right), F q G ( n ) − F q G ( ∞ ) ≤ C ( R 2 ∣ p G ∣ 2 + R ( 1 + ∣ p G ∣ ) + ∣ b − b c ∣ ) ,
∣ F A H ( n ) − F A H ( ∞ ) ∣ ≤ C ( R 2 ∣ A H ∣ 2 + R ∣ A H ∣ + ∣ a − a c ∣ ) , \left|F_{A_H}^{(n)}-F_{A_H}^{(\infty)}\right| \le C\left(R^2|A_H|^2+R|A_H|+|a-a_c|\right), F A H ( n ) − F A H ( ∞ ) ≤ C ( R 2 ∣ A H ∣ 2 + R ∣ A H ∣ + ∣ a − a c ∣ ) ,
∣ F B H ( n ) − F B H ( ∞ ) ∣ ≤ C ( R 2 ∣ A H B H ∣ + R ( 1 + ∣ A H ∣ + ∣ B H ∣ ) + ∣ a − a c ∣ + ∣ b − b c ∣ ) , \left|F_{B_H}^{(n)}-F_{B_H}^{(\infty)}\right| \le C\left(R^2|A_HB_H|+R(1+|A_H|+|B_H|)+|a-a_c|+|b-b_c|\right), F B H ( n ) − F B H ( ∞ ) ≤ C ( R 2 ∣ A H B H ∣ + R ( 1 + ∣ A H ∣ + ∣ B H ∣ ) + ∣ a − a c ∣ + ∣ b − b c ∣ ) ,
∣ F C H ( n ) − F C H ( ∞ ) ∣ ≤ C ( R 2 ( 1 + ∣ A H ∣ + ∣ B H ∣ 2 ) + R ( 1 + ∣ B H ∣ ) + V + ∣ b − b c ∣ ) . \left|F_{C_H}^{(n)}-F_{C_H}^{(\infty)}\right| \le C\left(R^2(1+|A_H|+|B_H|^2)+R(1+|B_H|)+V+|b-b_c|\right). F C H ( n ) − F C H ( ∞ ) ≤ C ( R 2 ( 1 + ∣ A H ∣ + ∣ B H ∣ 2 ) + R ( 1 + ∣ B H ∣ ) + V + ∣ b − b c ∣ ) .
Since K \mathcal{K} K is bounded, the five estimates above jointly give
sup 0 ≤ t ≤ T , U ∈ K ∣ ϕ − ρ ∗ ∣ ≤ δ O D E ∣ F ( n ) ( t , U ; ϕ ) − F ( ∞ ) ( t , U ; ϕ ) ∣ ≤ C ( n + 1 ) α . \sup_{\substack{0\le t\le T,\ U\in\mathcal{K}\\ |\phi-\rho^*|\le\delta_{\mathrm{ODE}}}}
\left|F^{(n)}(t,U;\phi)-F^{(\infty)}(t,U;\phi)\right|
\le\frac{C}{(n+1)^\alpha}. 0 ≤ t ≤ T , U ∈ K ∣ ϕ − ρ ∗ ∣ ≤ δ ODE sup F ( n ) ( t , U ; ϕ ) − F ( ∞ ) ( t , U ; ϕ ) ≤ ( n + 1 ) α C .
Let [ t n ∗ , T ] [t_n^*,T] [ t n ∗ , T ] be the maximal backward interval from T T T on which U ( n ) U^{(n)} U ( n ) remains in K \mathcal{K} K . Since the two solutions have the same terminal values, on this interval
∣ U t ( n ) − U t ( ∞ ) ∣ ≤ ∫ t T ∣ F ( n ) ( s , U s ( n ) ; ϕ ) − F ( n ) ( s , U s ( ∞ ) ; ϕ ) ∣ d s \left|U_t^{(n)}-U_t^{(\infty)}\right|
\le\int_t^T\left|F^{(n)}(s,U_s^{(n)};\phi)-F^{(n)}(s,U_s^{(\infty)};\phi)\right|\,ds U t ( n ) − U t ( ∞ ) ≤ ∫ t T F ( n ) ( s , U s ( n ) ; ϕ ) − F ( n ) ( s , U s ( ∞ ) ; ϕ ) d s
+ ∫ t T ∣ F ( n ) ( s , U s ( ∞ ) ; ϕ ) − F ( ∞ ) ( s , U s ( ∞ ) ; ϕ ) ∣ d s ≤ L ∫ t T ∣ U s ( n ) − U s ( ∞ ) ∣ d s + C ( n + 1 ) α . \begin{aligned}
\qquad+\int_t^T\left|F^{(n)}(s,U_s^{(\infty)};\phi)-F^{(\infty)}(s,U_s^{(\infty)};\phi)\right|\,ds \\
&\le L\int_t^T\left|U_s^{(n)}-U_s^{(\infty)}\right|\,ds+\frac{C}{(n+1)^\alpha}.
\end{aligned} + ∫ t T F ( n ) ( s , U s ( ∞ ) ; ϕ ) − F ( ∞ ) ( s , U s ( ∞ ) ; ϕ ) d s ≤ L ∫ t T U s ( n ) − U s ( ∞ ) d s + ( n + 1 ) α C .
Reverse time in the terminal-value problem by setting s = T − t s=T-t s = T − t . Then the generalized Gronwall inequality [44 Lemma 2.7] gives
sup t n ∗ ≤ t ≤ T ∣ U t ( n ) − U t ( ∞ ) ∣ ≤ C e L T ( n + 1 ) − α . \sup_{t_n^*\le t\le T}\left|U_t^{(n)}-U_t^{(\infty)}\right|\le Ce^{LT}(n+1)^{-\alpha}. t n ∗ ≤ t ≤ T sup U t ( n ) − U t ( ∞ ) ≤ C e L T ( n + 1 ) − α .
Choose n 1 , S R n_{1,\mathrm{SR}} n 1 , SR so that the right-hand side is at most ε K / 2 \varepsilon_K/2 ε K /2 . Since U t ( ∞ ) ∈ Γ ∞ U_t^{(\infty)}\in\Gamma_\infty U t ( ∞ ) ∈ Γ ∞ , it follows that dist ( U t ( n ) , Γ ∞ ) ≤ ε K / 2 \operatorname{dist}(U_t^{(n)},\Gamma_\infty)\le\varepsilon_K/2 dist ( U t ( n ) , Γ ∞ ) ≤ ε K /2 , so U ( n ) U^{(n)} U ( n ) cannot reach ∂ K \partial\mathcal{K} ∂ K . The ODE continuation criterion ([44 Lemma 2.14 and Corollary 2.15] ) then implies that t n ∗ = 0 t_n^*=0 t n ∗ = 0 . Absorbing e L T e^{LT} e L T gives
sup 0 ≤ t ≤ T ∣ U t ( n ) − U t ( ∞ ) ∣ ≤ C ( n + 1 ) α . (117) \sup_{0\le t\le T}\left|U_t^{(n)}-U_t^{(\infty)}\right|\le\frac{C}{(n+1)^\alpha}. \tag*{(117)} 0 ≤ t ≤ T sup U t ( n ) − U t ( ∞ ) ≤ ( n + 1 ) α C . (117)
This also gives existence and uniform boundedness of the perturbed ODE solution. For k ∈ N 0 k \in\mathbb{N}_0 k ∈ N 0 , define the normalized first and second moment factors
G ( k ) ( m , ϕ ) : = exp { p G ( k ) ( 0 ) m + q G ( k ) ( 0 ) } , G^{(k)}(m,\phi) := \exp\{p_G^{(k)}(0)m + q_G^{(k)}(0)\}, G ( k ) ( m , ϕ ) := exp { p G ( k ) ( 0 ) m + q G ( k ) ( 0 )} ,
H ( k ) ( m , ϕ ) : = exp { 1 2 A H ( k ) ( 0 ) m 2 + B H ( k ) ( 0 ) m + C H ( k ) ( 0 ) } . H^{(k)}(m,\phi) := \exp\left\{\frac{1}{2}A_H^{(k)}(0)m^2 + B_H^{(k)}(0)m + C_H^{(k)}(0)\right\}. H ( k ) ( m , ϕ ) := exp { 2 1 A H ( k ) ( 0 ) m 2 + B H ( k ) ( 0 ) m + C H ( k ) ( 0 ) } .
For k = ∞ k=\infty k = ∞ , retain the definitions of G ( ∞ ) G^{(\infty)} G ( ∞ ) and H ( ∞ ) H^{(\infty)} H ( ∞ ) given above. On the chosen compact neighborhood of ( m , ϕ ) (m,\phi) ( m , ϕ ) , (117) gives
∣ log G ( n ) − log G ( ∞ ) ∣ + ∣ log H ( n ) − log H ( ∞ ) ∣ ≤ C ( n + 1 ) α . \left|\log G^{(n)}-\log G^{(\infty)}\right|+\left|\log H^{(n)}-\log H^{(\infty)}\right|\leq\frac{C}{(n+1)^\alpha}. log G ( n ) − log G ( ∞ ) + log H ( n ) − log H ( ∞ ) ≤ ( n + 1 ) α C .
These exponents are uniformly bounded on the compact neighborhood, and the exponential function is Lipschitz on bounded intervals. Hence
∣ G ( n ) − G ( ∞ ) ∣ + ∣ H ( n ) − H ( ∞ ) ∣ ≤ C ( n + 1 ) α . \left|G^{(n)}-G^{(\infty)}\right|+\left|H^{(n)}-H^{(\infty)}\right|\leq\frac{C}{(n+1)^\alpha}. G ( n ) − G ( ∞ ) + H ( n ) − H ( ∞ ) ≤ ( n + 1 ) α C .
Because w > x 0 w>x_0 w > x 0 , the common factor w − x 0 w-x_0 w − x 0 in the Sharpe ratio cancels exactly, and therefore
S ( m , ϕ ; R ( n ) ) = 1 − G ( n ) ( m , ϕ ) H ( n ) ( m , ϕ ) − G ( n ) ( m , ϕ ) 2 . S(m,\phi;R^{(n)})=\frac{1-G^{(n)}(m,\phi)}{\sqrt{H^{(n)}(m,\phi)-G^{(n)}(m,\phi)^2}}. S ( m , ϕ ; R ( n ) ) = H ( n ) ( m , ϕ ) − G ( n ) ( m , ϕ ) 2 1 − G ( n ) ( m , ϕ ) .
By the explicit expressions for G ( ∞ ) G^{(\infty)} G ( ∞ ) and H ( ∞ ) H^{(\infty)} H ( ∞ ) given above,
1 − G ( ∞ ) ( m , ϕ ) H ( ∞ ) ( m , ϕ ) − G ( ∞ ) ( m , ϕ ) 2 = e ρ ∗ M T − 1 e M 2 T − 1 = S R ∞ ( M ) . \frac{1-G^{(\infty)}(m,\phi)}{\sqrt{H^{(\infty)}(m,\phi)-G^{(\infty)}(m,\phi)^2}}=\frac{e^{\rho^*MT}-1}{\sqrt{e^{M^2T}-1}}=SR_{\infty}(M). H ( ∞ ) ( m , ϕ ) − G ( ∞ ) ( m , ϕ ) 2 1 − G ( ∞ ) ( m , ϕ ) = e M 2 T − 1 e ρ ∗ M T − 1 = S R ∞ ( M ) .
Moreover, by the choice of δ O D E \delta_{\mathrm{ODE}} δ ODE and d 0 d_0 d 0 above, H ( ∞ ) − ( G ( ∞ ) ) 2 ≥ 2 d 0 H^{(\infty)}-(G^{(\infty)})^2\geq2d_0 H ( ∞ ) − ( G ( ∞ ) ) 2 ≥ 2 d 0 . Since
∣ ( H ( n ) − ( G ( n ) ) 2 ) − ( H ( ∞ ) − ( G ( ∞ ) ) 2 ) ∣ ≤ ∣ H ( n ) − H ( ∞ ) ∣ + ∣ ( G ( n ) ) 2 − ( G ( ∞ ) ) 2 ∣ ≤ C ( n + 1 ) α , \begin{aligned}
\left|\left(H^{(n)}-(G^{(n)})^2\right)-\left(H^{(\infty)}-(G^{(\infty)})^2\right)\right|\leq\left|H^{(n)}-H^{(\infty)}\right|+\left|(G^{(n)})^2-(G^{(\infty)})^2\right| \\
&\leq\frac{C}{(n+1)^\alpha},
\end{aligned} ( H ( n ) − ( G ( n ) ) 2 ) − ( H ( ∞ ) − ( G ( ∞ ) ) 2 ) ≤ H ( n ) − H ( ∞ ) + ( G ( n ) ) 2 − ( G ( ∞ ) ) 2 ≤ ( n + 1 ) α C ,
take n 2 , S R n_{2,\mathrm{SR}} n 2 , SR such that, for all n ≥ n 2 , S R n\geq n_{2,\mathrm{SR}} n ≥ n 2 , SR , C / ( n + 1 ) α ≤ d 0 C/(n+1)^\alpha\leq d_0 C / ( n + 1 ) α ≤ d 0 . Then for all n ≥ max { n 1 , S R , n 2 , S R } n\geq\max\{n_{1,\mathrm{SR}},n_{2,\mathrm{SR}}\} n ≥ max { n 1 , SR , n 2 , SR } , we have H ( n ) − ( G ( n ) ) 2 ≥ d 0 H^{(n)}-(G^{(n)})^2\geq d_0 H ( n ) − ( G ( n ) ) 2 ≥ d 0 . Hence the map Ψ ( G , H ) : = H − G 2 \Psi(G,H):=\sqrt{H-G^2} Ψ ( G , H ) := H − G 2 is continuously differentiable on the compact set under consideration, and its gradient is uniformly bounded. The mean value theorem gives
∣ Ψ ( G ( n ) , H ( n ) ) − Ψ ( G ( ∞ ) , H ( ∞ ) ) ∣ ≤ C ( n + 1 ) α . \left|\Psi(G^{(n)},H^{(n)})-\Psi(G^{(\infty)},H^{(\infty)})\right|\leq\frac{C}{(n+1)^\alpha}. Ψ ( G ( n ) , H ( n ) ) − Ψ ( G ( ∞ ) , H ( ∞ ) ) ≤ ( n + 1 ) α C .
Combining this with the limiting identity above gives the convergence rate.
Proof of Theorem 7
Proof. Take δ : = min { ∣ ρ ∗ ∣ / 4 , δ O D E } \delta:=\min\{|\rho^*|/4,\delta_{\mathrm{ODE}}\} δ := min { ∣ ρ ∗ ∣/4 , δ ODE } , and define E n ( δ ) : = { ∣ m 0 ( n ) − ρ ∗ ∣ ≤ δ , ∣ ϕ n − ρ ∗ ∣ ≤ δ } E_n(\delta):=\{|m_0^{(n)}-\rho^*|\leq\delta,\ |\phi_n-\rho^*|\leq\delta\} E n ( δ ) := { ∣ m 0 ( n ) − ρ ∗ ∣ ≤ δ , ∣ ϕ n − ρ ∗ ∣ ≤ δ } . Denote r ϕ , n : = L n [ ( n + 1 ) − 2 α + ( n + 1 ) − γ 2 ] r_{\phi,n}:=\sqrt{L_n[(n+1)^{-2\alpha}+(n+1)^{-\gamma_2}]} r ϕ , n := L n [( n + 1 ) − 2 α + ( n + 1 ) − γ 2 ] . By Lemma 1 and (110) , we have e 0 ( n ) = O p ( ( n + 1 ) − α / 2 ) e_0^{(n)}=O_{\mathrm{p}}((n+1)^{-\alpha/2}) e 0 ( n ) = O p (( n + 1 ) − α /2 ) and ϕ n − ρ ∗ = O p ( r ϕ , n ) \phi_n-\rho^*=O_{\mathrm{p}}(r_{\phi,n}) ϕ n − ρ ∗ = O p ( r ϕ , n ) . Therefore
M ‾ c , n − ρ ∗ = a c e 0 ( n ) + b c ( ϕ n − ρ ∗ ) = O p ( ( n + 1 ) − α / 2 + r ϕ , n ) . \overline{M}_{c,n}-\rho^*=a_c e_0^{(n)}+b_c(\phi_n-\rho^*)=O_{\mathrm{p}}((n+1)^{-\alpha/2}+r_{\phi,n}). M c , n − ρ ∗ = a c e 0 ( n ) + b c ( ϕ n − ρ ∗ ) = O p (( n + 1 ) − α /2 + r ϕ , n ) .
On the event E n ( δ ) E_n(\delta) E n ( δ ) , by a c + b c = 1 a_c+b_c=1 a c + b c = 1 ,
∣ M ‾ c , n − ρ ∗ ∣ ≤ a c ∣ e 0 ( n ) ∣ + b c ∣ ϕ n − ρ ∗ ∣ ≤ δ ≤ ∣ ρ ∗ ∣ 4 . |\overline{M}_{c,n}-\rho^*|\leq a_c|e_0^{(n)}|+b_c|\phi_n-\rho^*|\leq\delta\leq\frac{|\rho^*|}{4}. ∣ M c , n − ρ ∗ ∣ ≤ a c ∣ e 0 ( n ) ∣ + b c ∣ ϕ n − ρ ∗ ∣ ≤ δ ≤ 4 ∣ ρ ∗ ∣ .
Thus, by the Taylor bound in Lemma 2,
1 E n ( δ ) ∣ S R ∞ ( M ‾ c , n ) − S R o r ∣ = O p ( ( n + 1 ) − α + r ϕ , n 2 ) . \mathbf{1}_{E_n(\delta)}|SR_{\infty}(\overline{M}_{c,n})-SR_{\mathrm{or}}|=O_{\mathrm{p}}\left((n+1)^{-\alpha}+r_{\phi,n}^2\right). 1 E n ( δ ) ∣ S R ∞ ( M c , n ) − S R or ∣ = O p ( ( n + 1 ) − α + r ϕ , n 2 ) .
On the other hand, since δ ≤ δ O D E \delta\le\delta_{\mathrm{ODE}} δ ≤ δ ODE , (44) gives
1 E n ( δ ) ∣ S R n − S R ∞ ( M c , n ) ∣ ≤ C ( n + 1 ) − α . \mathbf{1}_{\mathcal{E}_n(\delta)}|SR_n-SR_\infty(M_{c,n})|\le C(n+1)^{-\alpha}. 1 E n ( δ ) ∣ S R n − S R ∞ ( M c , n ) ∣ ≤ C ( n + 1 ) − α .
Combining the two displays and absorbing the term controlled by ( n + 1 ) − 2 α (n+1)^{-2\alpha} ( n + 1 ) − 2 α , we obtain
1 E n ( δ ) ∣ S R n − S R o r ∣ = O p ( L n ( n + 1 ) α + L n ( n + 1 ) γ 2 ) . \mathbf{1}_{\mathcal{E}_n(\delta)}|SR_n-SR_{\mathrm{or}}|=O_p\left(\frac{L_n}{(n+1)^\alpha}+\frac{L_n}{(n+1)^{\gamma_2}}\right). 1 E n ( δ ) ∣ S R n − S R or ∣ = O p ( ( n + 1 ) α L n + ( n + 1 ) γ 2 L n ) .
Write r S R , n : = L n ( n + 1 ) α + L n ( n + 1 ) γ 2 r_{SR,n}:=\frac{L_n}{(n+1)^\alpha}+\frac{L_n}{(n+1)^{\gamma_2}} r S R , n := ( n + 1 ) α L n + ( n + 1 ) γ 2 L n . For any ε > 0 \varepsilon>0 ε > 0 , take K < ∞ K<\infty K < ∞ such that
lim sup n → ∞ P ( 1 E n ( δ ) ∣ S R n − S R o r ∣ > K r S R , n ) ≤ ε . \limsup_{n\to\infty}\mathbb{P}\left(\mathbf{1}_{\mathcal{E}_n(\delta)}|SR_n-SR_{\mathrm{or}}|>Kr_{SR,n}\right)\le\varepsilon. n → ∞ lim sup P ( 1 E n ( δ ) ∣ S R n − S R or ∣ > K r S R , n ) ≤ ε .
Also, by Lemma 1 and Theorem 4,
P ( E n ( δ ) c ) ≤ P ( ∣ e 0 ( n ) ∣ > δ ) + P ( ∣ ϕ n − ρ ∗ ∣ > δ ) ⟶ 0. \mathbb{P}(\mathcal{E}_n(\delta)^c)\le\mathbb{P}(|e_0^{(n)}|>\delta)+\mathbb{P}(|\phi_n-\rho^*|>\delta)\longrightarrow0. P ( E n ( δ ) c ) ≤ P ( ∣ e 0 ( n ) ∣ > δ ) + P ( ∣ ϕ n − ρ ∗ ∣ > δ ) ⟶ 0.
Therefore
lim sup n → ∞ P ( ∣ S R n − S R o r ∣ > K r S R , n ) ≤ lim sup n → ∞ P ( 1 E n ( δ ) ∣ S R n − S R o r ∣ > K r S R , n ) + lim sup n → ∞ P ( E n ( δ ) c ) ≤ ε . \limsup_{n\to\infty}\mathbb{P}(|SR_n-SR_{\mathrm{or}}|>Kr_{SR,n})
\le\limsup_{n\to\infty}\mathbb{P}\left(\mathbf{1}_{\mathcal{E}_n(\delta)}|SR_n-SR_{\mathrm{or}}|>Kr_{SR,n}\right)
+\limsup_{n\to\infty}\mathbb{P}(\mathcal{E}_n(\delta)^c)\le\varepsilon. n → ∞ lim sup P ( ∣ S R n − S R or ∣ > K r S R , n ) ≤ n → ∞ lim sup P ( 1 E n ( δ ) ∣ S R n − S R or ∣ > K r S R , n ) + n → ∞ lim sup P ( E n ( δ ) c ) ≤ ε .
This is exactly (45) .
Proof of Theorem 6
We first have the next preparation.
Lemma 5. Let G ~ \widetilde{G} G and H ~ \widetilde{H} H be the sampled first- and second-moment factors of ( X T − w ) / ( x 0 − w ) (X_T-w)/(x_0-w) ( X T − w ) / ( x 0 − w ) under (67) . Episode and batch indices are suppressed in G ~ \widetilde{G} G and H ~ \widetilde{H} H . There exist constants R ∗ > 0 R_* > 0 R ∗ > 0 and C < ∞ C < \infty C < ∞ , depending on the fixed grid and model coefficients, such that all four moments are finite and positive for 0 < R ≤ R ∗ 0<R\le R_* 0 < R ≤ R ∗ and
∣ log G ~ G ( 0 , m ) ∣ ≤ C R ( 1 + ∣ m ∣ + ∣ ϕ ∣ ) , (118) \left|\log\frac{\widetilde{G}}{G(0,m)}\right|\le CR(1+|m|+|\phi|), \tag*{(118)} log G ( 0 , m ) G ≤ C R ( 1 + ∣ m ∣ + ∣ ϕ ∣ ) , (118) ∣ log H ~ H ( 0 , m ) ∣ ≤ C R ( 1 + m 2 + ϕ 2 ) . (119) \left|\log\frac{\widetilde{H}}{H(0,m)}\right|\le CR(1+m^2+\phi^2). \tag*{(119)} log H ( 0 , m ) H ≤ C R ( 1 + m 2 + ϕ 2 ) . (119) The constants are independent of m m m and ϕ \phi ϕ .
Proof. By (67) , Itô’s formula gives
Y T Y 0 = exp { − ∫ 0 T ( ρ t ∗ ρ t + 1 2 ρ t 2 ) d t − ∫ 0 T ρ t d W t } . \frac{Y_T}{Y_0}=\exp\left\{-\int_0^T\left(\rho_t^*\rho_t+\frac{1}{2}\rho_t^2\right)\,dt-\int_0^T\rho_t\,dW_t\right\}. Y 0 Y T = exp { − ∫ 0 T ( ρ t ∗ ρ t + 2 1 ρ t 2 ) d t − ∫ 0 T ρ t d W t } .
Condition throughout on the episode-start information F n \mathcal{F}_n F n . The conditional Gaussian sampling law gives
E [ ∫ 0 T ( ρ t − M t ϕ ) 2 d t ∣ F n ] = ∫ 0 T V t d t ≤ C R 0 . \mathbb{E}\left[\left.\int_0^T(\rho_t-M_t^\phi)^2dt\right|\mathcal{F}_n\right]=\int_0^T V_tdt\le CR_0. E [ ∫ 0 T ( ρ t − M t ϕ ) 2 d t F n ] = ∫ 0 T V t d t ≤ C R 0 .
Thus the centered action is square-integrable in time almost surely. Since V t V_t V t is deterministic given F n \mathcal{F}_n F n , the independent sampling construction in Algorithm 1 allows us to average first over the market noise:
log E [ Y T Y 0 ∣ F n , { ρ s − M s ϕ : 0 ≤ s ≤ T } ] = p ~ G ( 0 ) m + q ~ G ( 0 ) , \log\mathbb{E}\left[\left.\frac{Y_T}{Y_0}\right|\mathcal{F}_n,\{\rho_s-M_s^\phi:0\le s\le T\}\right]=\widetilde{p}_G(0)m+\widetilde{q}_G(0), log E [ Y 0 Y T F n , { ρ s − M s ϕ : 0 ≤ s ≤ T } ] = p G ( 0 ) m + q G ( 0 ) ,
log E [ ( Y T Y 0 ) 2 ∣ F n , { ρ s − M s ϕ : 0 ≤ s ≤ T } ] = 1 2 A ~ H ( 0 ) m 2 + B ~ H ( 0 ) m + C ~ H ( 0 ) . \log\mathbb{E}\left[\left.\left(\frac{Y_T}{Y_0}\right)^2\right|\mathcal{F}_n,\{\rho_s-M_s^\phi:0\le s\le T\}\right]=\frac{1}{2}\widetilde{A}_H(0)m^2+\widetilde{B}_H(0)m+\widetilde{C}_H(0). log E [ ( Y 0 Y T ) 2 F n , { ρ s − M s ϕ : 0 ≤ s ≤ T } ] = 2 1 A H ( 0 ) m 2 + B H ( 0 ) m + C H ( 0 ) .
The tilded coefficients depend on the centered sampling randomness. For the macroscopic coefficients, the A H A_H A H equation and terminal condition are independent of ϕ \phi ϕ . Thus the stability argument for (117) , with limit (114) , bounds A H A_H A H uniformly for small R 0 R_0 R 0 and all ϕ \phi ϕ . Together with (76) , the integral formulas A H A_H A H and B H B_H B H , this gives
sup t ≤ T ( ∣ p G ( t ) ∣ + ∣ A H ( t ) ∣ ) ≤ C , sup t ≤ T ∣ B H ( t ) ∣ ≤ C ( 1 + ∣ ϕ ∣ ) . \sup_{t\le T}\left(|p_G(t)|+|A_H(t)|\right)\le C,\qquad\sup_{t\le T}|B_H(t)|\le C(1+|\phi|). t ≤ T sup ( ∣ p G ( t ) ∣ + ∣ A H ( t ) ∣ ) ≤ C , t ≤ T sup ∣ B H ( t ) ∣ ≤ C ( 1 + ∣ ϕ ∣ ) .
Itô verification is justified by Novikov’s condition: the integrands are affine in the Gaussian filter state plus deterministic L 2 L^2 L 2 terms. Its conditional variance is O ( R 0 2 ) O(R_0^2) O ( R 0 2 ) by (57) , so Jensen’s inequality and Gaussian exponential moments give the required integrability for small R 0 R_0 R 0 . Matching the coefficients of m m m in the first-moment equations and m 2 m^2 m 2 in the second-moment equations gives the same terminal-value problems; uniqueness yields p ~ G ( t ) = p G ( t ) \widetilde{p}_G(t)=p_G(t) p G ( t ) = p G ( t ) and A ~ H ( t ) = A H ( t ) \widetilde{A}_H(t)=A_H(t) A H ( t ) = A H ( t ) . Variation of constants in the remaining coefficient equations gives
∣ q ~ G ( 0 ) − q G ( 0 ) ∣ + sup t ≤ T ∣ B ~ H ( t ) − B H ( t ) ∣ ≤ C ∫ 0 T ∣ ρ t − M t ϕ ∣ d t . \left|\widetilde{q}_G(0)-q_G(0)\right|+\sup_{t\leq T}\left|\widetilde{B}_H(t)-B_H(t)\right|\leq C\int_0^T\left|\rho_t-M_t^\phi\right|\,dt. ∣ q G ( 0 ) − q G ( 0 ) ∣ + t ≤ T sup B H ( t ) − B H ( t ) ≤ C ∫ 0 T ρ t − M t ϕ d t .
Both differences are centered linear functionals of ρ − M ϕ \rho-M^\phi ρ − M ϕ with uniformly bounded deterministic kernels. Moreover, C ~ H ( 0 ) − C H ( 0 ) + ∫ 0 T V t d t \widetilde{C}_H(0)-C_H(0)+\int_0^T V_t\,dt C H ( 0 ) − C H ( 0 ) + ∫ 0 T V t d t is the sum of a centered linear functional with deterministic kernel bounded by C ( 1 + ∣ ϕ ∣ ) C(1+|\phi|) C ( 1 + ∣ ϕ ∣ ) and a quadratic functional bounded in absolute value by C ∫ 0 T ( ρ t − M t ϕ ) 2 d t C\int_0^T(\rho_t-M_t^\phi)^2\,dt C ∫ 0 T ( ρ t − M t ϕ ) 2 d t . For fixed ( m , ϕ ) (m,\phi) ( m , ϕ ) , the tower property gives
G ~ ( 0 , m ) G ( 0 , m ) = E [ exp { q ~ G ( 0 ) − q G ( 0 ) } ∣ F n ] , (120) \frac{\widetilde{G}(0,m)}{G(0,m)}
=
\mathbb{E}\left[\exp\{\widetilde{q}_G(0)-q_G(0)\}\mid\mathcal{F}_n\right], \tag*{(120)} G ( 0 , m ) G ( 0 , m ) = E [ exp { q G ( 0 ) − q G ( 0 )} ∣ F n ] , (120)
H ~ ( 0 , m ) H ( 0 , m ) = E [ exp { m ( B ~ H ( 0 ) − B H ( 0 ) ) + C ~ H ( 0 ) − C H ( 0 ) } ∣ F n ] . (121) \frac{\widetilde{H}(0,m)}{H(0,m)}
=
\mathbb{E}\left[\exp\{m(\widetilde{B}_H(0)-B_H(0))+\widetilde{C}_H(0)-C_H(0)\}\mid\mathcal{F}_n\right]. \tag*{(121)} H ( 0 , m ) H ( 0 , m ) = E [ exp { m ( B H ( 0 ) − B H ( 0 )) + C H ( 0 ) − C H ( 0 )} ∣ F n ] . (121)
For every integrable f f f that is deterministic conditional on F n \mathcal{F}_n F n , weighted Jensen’s inequality and the conditional Gaussian law (74) yield
E [ exp { ∫ 0 T f t ( ρ t − M t ϕ ) d t } ∣ F n ] ≤ exp { 1 2 ( ∫ 0 T ∣ f t ∣ V t d t ) 2 } . \mathbb{E}\left[\left.\exp\left\{\int_0^T f_t(\rho_t-M_t^\phi)\,dt\right\}\right|\mathcal{F}_n\right]
\leq
\exp\left\{\frac{1}{2}\left(\int_0^T |f_t|\sqrt{V_t}\,dt\right)^2\right\}. E [ exp { ∫ 0 T f t ( ρ t − M t ϕ ) d t } F n ] ≤ exp ⎩ ⎨ ⎧ 2 1 ( ∫ 0 T ∣ f t ∣ V t d t ) 2 ⎭ ⎬ ⎫ .
Since the first exponent is a centered linear functional with a bounded coefficient and V t ≤ C R 0 V_t\leq CR_0 V t ≤ C R 0 ,
1 ≤ E [ e q ~ G ( 0 ) − q G ( 0 ) ∣ F n ] ≤ e C R 0 . 1\leq\mathbb{E}\left[\left.e^{\widetilde{q}_G(0)-q_G(0)}\right|\mathcal{F}_n\right]\leq e^{CR_0}. 1 ≤ E [ e q G ( 0 ) − q G ( 0 ) F n ] ≤ e C R 0 .
The second exponent has a centered linear part with kernel bounded by C ( 1 + ∣ m ∣ + ∣ ϕ ∣ ) C(1+|m|+|\phi|) C ( 1 + ∣ m ∣ + ∣ ϕ ∣ ) . Its quadratic part satisfies the preceding bound. For small R 0 R_0 R 0 ,
E [ exp { C ∫ 0 T ( ρ t − M t ϕ ) 2 d t } ∣ F n ] ≤ 1 T ∫ 0 T E [ e C T ( ρ t − M t ϕ ) 2 ∣ F n ] d t = 1 T ∫ 0 T ( 1 − 2 C T V t ) − 1 / 2 d t ≤ e C R 0 . \begin{aligned}
\mathbb{E}\left[\left.\exp\left\{C\int_0^T(\rho_t-M_t^\phi)^2\,dt\right\}\right|\mathcal{F}_n\right]
\leq
\frac{1}{T}\int_0^T\mathbb{E}\left[\left.e^{CT(\rho_t-M_t^\phi)^2}\right|\mathcal{F}_n\right]\,dt \\
&=
\frac{1}{T}\int_0^T(1-2CTV_t)^{-1/2}\,dt\leq e^{CR_0}.
\end{aligned} E [ exp { C ∫ 0 T ( ρ t − M t ϕ ) 2 d t } F n ] ≤ T 1 ∫ 0 T E [ e C T ( ρ t − M t ϕ ) 2 F n ] d t = T 1 ∫ 0 T ( 1 − 2 C T V t ) − 1/2 d t ≤ e C R 0 .
Thus Jensen’s inequality for the lower bound and Cauchy–Schwarz for the upper bound give
0 ≤ log G ~ ( 0 , m ) G ( 0 , m ) ≤ C R 0 , − C R 0 ≤ log H ~ ( 0 , m ) H ( 0 , m ) ≤ C R 0 ( 1 + m 2 + ϕ 2 ) . 0\leq\log\frac{\widetilde{G}(0,m)}{G(0,m)}\leq CR_0,\qquad
-CR_0\leq\log\frac{\widetilde{H}(0,m)}{H(0,m)}\leq CR_0(1+m^2+\phi^2). 0 ≤ log G ( 0 , m ) G ( 0 , m ) ≤ C R 0 , − C R 0 ≤ log H ( 0 , m ) H ( 0 , m ) ≤ C R 0 ( 1 + m 2 + ϕ 2 ) .
These estimates ensure finiteness for sufficiently small R ∗ R_* R ∗ ; positivity follows from the stochastic exponential representations. At R 0 = 0 R_0=0 R 0 = 0 , the action is constant, ρ t = a c m + b c ϕ \rho_t=a_cm+b_c\phi ρ t = a c m + b c ϕ , and (112) –(116) give
G ~ ( 0 , m ) ∣ R 0 = 0 = G ( 0 , m ) ∣ R 0 = 0 = G ( ∞ ) ( m , ϕ ) , H ~ ( 0 , m ) ∣ R 0 = 0 = H ( 0 , m ) ∣ R 0 = 0 = H ( ∞ ) ( m , ϕ ) . \left.\widetilde{G}(0,m)\right|_{R_0=0}
=
\left.G(0,m)\right|_{R_0=0}
=
G^{(\infty)}(m,\phi),
\qquad
\left.\widetilde{H}(0,m)\right|_{R_0=0}
=
\left.H(0,m)\right|_{R_0=0}
=
H^{(\infty)}(m,\phi). G ( 0 , m ) R 0 = 0 = G ( 0 , m ) ∣ R 0 = 0 = G ( ∞ ) ( m , ϕ ) , H ( 0 , m ) R 0 = 0 = H ( 0 , m ) ∣ R 0 = 0 = H ( ∞ ) ( m , ϕ ) .
The preceding logarithmic bounds prove (118) –(119) .
In the multiplier update, G G G , H H H and their ODE coefficients correspond to the macroscopic model with actor parameter ϕ ℓ \phi_\ell ϕ ℓ and covariance path R ( ℓ ) R^{(\ell)} R ( ℓ ) . Let F r w \mathcal{F}^w_r F r w denote the information before the start of the batch used for the ( r + 1 ) (r+1) ( r + 1 ) st multiplier update. Within this batch, the multiplier remains the F r w \mathcal{F}^w_r F r w -measurable variable w r w_r w r , and is updated after the episodes I r + 1 \mathcal{I}_{r+1} I r + 1 end. For ℓ ∈ I r + 1 \ell\in\mathcal{I}_{r+1} ℓ ∈ I r + 1 , let F ℓ \mathcal{F}_\ell F ℓ denote the information before episode ℓ \ell ℓ starts; therefore F r w ⊆ F ℓ \mathcal{F}^w_r\subseteq\mathcal{F}_\ell F r w ⊆ F ℓ . First define the episode-start normalized first-moment factor
G ~ w ( ℓ , r ) : = E [ X T ( ℓ ) − w r X 0 − w r ∣ F ℓ ] . \widetilde{G}^{(\ell,r)}_w:=\mathbb{E}\left[\left.\frac{X_T^{(\ell)}-w_r}{X_0-w_r}\right|\mathcal{F}_\ell\right]. G w ( ℓ , r ) := E [ X 0 − w r X T ( ℓ ) − w r F ℓ ] .
Next define the factor truly conditioned on the batch-start information:
G w ( ℓ , r ) : = E [ X T ( ℓ ) − w r x 0 − w r ∣ F r w ] = E [ G ~ w ( ℓ , r ) ∣ F r w ] , G ‾ w , r : = 1 N w ∑ ℓ ∈ I r + 1 G w ( ℓ , r ) . G_w^{(\ell,r)} := \mathbb{E}\left[\left.\frac{X_T^{(\ell)}-w_r}{x_0-w_r}\right|\mathcal{F}_r^w\right] = \mathbb{E}\left[\left.\widetilde{G}_w^{(\ell,r)}\right|\mathcal{F}_r^w\right], \qquad\overline{G}_{w,r} := \frac{1}{N_w}\sum_{\ell\in\mathcal{I}_{r+1}}G_w^{(\ell,r)}. G w ( ℓ , r ) := E [ x 0 − w r X T ( ℓ ) − w r F r w ] = E [ G w ( ℓ , r ) F r w ] , G w , r := N w 1 ℓ ∈ I r + 1 ∑ G w ( ℓ , r ) .
The second equality follows from the tower property, and G w ( ℓ , r ) G_w^{(\ell,r)} G w ( ℓ , r ) and G ‾ w , r \overline{G}_{w,r} G w , r are both F r w \mathcal{F}_r^w F r w -measurable. As long as x 0 ≠ w r x_0\ne w_r x 0 = w r , the terminal deviation X T ( ℓ ) − w r X_T^{(\ell)}-w_r X T ( ℓ ) − w r is homogeneous in the initial deviation x 0 − w r x_0-w_r x 0 − w r , and the normalized factor does not depend on w r w_r w r . Furthermore,
E [ X ‾ T , r + 1 − z ∣ F r w ] = ( 1 − G ‾ w , r ) w r + G ‾ w , r x 0 − z . (122) \mathbb{E}\left[\left.\overline{X}_{T,r+1}-z\right|\mathcal{F}_r^w\right]=(1-\overline{G}_{w,r})w_r+\overline{G}_{w,r}x_0-z. \tag*{(122)} E [ X T , r + 1 − z F r w ] = ( 1 − G w , r ) w r + G w , r x 0 − z . (122)
In the oracle limit,
G w ∗ : = exp { − ( ρ ∗ ) 2 T } , w ∗ : = z − G w ∗ x 0 1 − G w ∗ = z e ( ρ ∗ ) 2 T − x 0 e ( ρ ∗ ) 2 T − 1 . (123) G_w^* := \exp\left\{-(\rho^*)^2T\right\}, \qquad w^* := \frac{z-G_w^*x_0}{1-G_w^*}=\frac{z e^{(\rho^*)^2T}-x_0}{e^{(\rho^*)^2T}-1}. \tag*{(123)} G w ∗ := exp { − ( ρ ∗ ) 2 T } , w ∗ := 1 − G w ∗ z − G w ∗ x 0 = e ( ρ ∗ ) 2 T − 1 z e ( ρ ∗ ) 2 T − x 0 . (123)
For any terminal wealth in the window generated by episode-start parameters and using the same batch multiplier w r w_r w r , define the sampled second-moment factor and recall that
H ~ ( ℓ , r ) : = E [ ( X T ( ℓ ) − w r x 0 − w r ) 2 ∣ F ℓ ] , H ( 0 , m 0 ( ℓ ) ) = exp { 1 2 A H ℓ , r ( 0 ) ( m 0 ( ℓ ) ) 2 + B H ℓ , r ( 0 ) m 0 ( ℓ ) + C H ℓ , r ( 0 ) } . \widetilde{H}^{(\ell,r)}:=\mathbb{E}\left[\left.\left(\frac{X_T^{(\ell)}-w_r}{x_0-w_r}\right)^2\right|\mathcal{F}_{\ell}\right], \qquad H(0,m_0^{(\ell)})=\exp\left\{\frac{1}{2}A_{H\ell,r}(0)(m_0^{(\ell)})^2+B_{H\ell,r}(0)m_0^{(\ell)}+C_{H\ell,r}(0)\right\}. H ( ℓ , r ) := E ( x 0 − w r X T ( ℓ ) − w r ) 2 F ℓ , H ( 0 , m 0 ( ℓ ) ) = exp { 2 1 A H ℓ , r ( 0 ) ( m 0 ( ℓ ) ) 2 + B H ℓ , r ( 0 ) m 0 ( ℓ ) + C H ℓ , r ( 0 ) } .
It does not depend on w r w_r w r . Conditioning on the batch-start information, it follows from the tower property that
H w ( ℓ , r ) : = E [ H ~ ( ℓ , r ) ∣ F r w ] , E [ ( X T ( ℓ ) − w r ) 2 ∣ F r w ] = ( x 0 − w r ) 2 H w ( ℓ , r ) . (124) H_w^{(\ell,r)}:=\mathbb{E}\left[\left.\widetilde{H}^{(\ell,r)}\right|\mathcal{F}_r^w\right], \qquad\mathbb{E}\left[\left.(X_T^{(\ell)}-w_r)^2\right|\mathcal{F}_r^w\right]=(x_0-w_r)^2H_w^{(\ell,r)}. \tag*{(124)} H w ( ℓ , r ) := E [ H ( ℓ , r ) F r w ] , E [ ( X T ( ℓ ) − w r ) 2 F r w ] = ( x 0 − w r ) 2 H w ( ℓ , r ) . (124)
Choose ℓ 0 \ell_0 ℓ 0 so that the comparison lemma applies to all ℓ ≥ ℓ 0 \ell\ge\ell_0 ℓ ≥ ℓ 0 . For the Riccati coefficient A H ℓ , r A_{H\ell,r} A H ℓ , r , since the ODE for A H ℓ , r A_{H\ell,r} A H ℓ , r does not contain ϕ ℓ \phi_\ell ϕ ℓ , Lemma 2 gives the A H A_H A H -component estimate (117) without requiring a local restriction on ϕ \phi ϕ . Therefore, when ℓ \ell ℓ is sufficiently large, A H ℓ , r A_{H\ell,r} A H ℓ , r is uniformly close on [ 0 , T ] [0,T] [ 0 , T ] to A H ( ∞ ) ( t ) = 2 a c 2 ( T − t ) A_H^{(\infty)}(t)=2a_c^2(T-t) A H ( ∞ ) ( t ) = 2 a c 2 ( T − t ) , and hence is uniformly bounded. Thus there exists a deterministic constant C A < ∞ C_A<\infty C A < ∞ such that
sup r , ℓ ∈ I r + 1 , ℓ ≥ ℓ 0 sup 0 ≤ t ≤ T ∣ A H ℓ , r ( t ) ∣ ≤ C A . (125) \sup_{r,\ \ell\in\mathcal{I}_{r+1},\ \ell\ge\ell_0}\ \sup_{0\le t\le T}\left|A_{H\ell,r}(t)\right|\le C_A. \tag*{(125)} r , ℓ ∈ I r + 1 , ℓ ≥ ℓ 0 sup 0 ≤ t ≤ T sup ∣ A H ℓ , r ( t ) ∣ ≤ C A . (125)
Based on the formula above, the ODE for B H ℓ , r B_{H\ell,r} B H ℓ , r is linear in B H ℓ , r B_{H\ell,r} B H ℓ , r . Using (76) , ∣ ϕ ℓ ∣ ≤ c ℓ |\phi_\ell|\le c_\ell ∣ ϕ ℓ ∣ ≤ c ℓ , R t ( ℓ ) ≤ R 0 R_t^{(\ell)}\le R_0 R t ( ℓ ) ≤ R 0 , and the terminal condition B H ℓ , r ( T ) = 0 B_{H\ell,r}(T)=0 B H ℓ , r ( T ) = 0 , its backward integral form gives
∣ B H ℓ , r ( t ) ∣ ≤ C ∫ t T ∣ B H ℓ , r ( s ) ∣ d s + C ( 1 + c ℓ ) , 0 ≤ t ≤ T . |B_{H\ell,r}(t)|\le C\int_t^T|B_{H\ell,r}(s)|\,ds+C(1+c_\ell), \qquad0\le t\le T. ∣ B H ℓ , r ( t ) ∣ ≤ C ∫ t T ∣ B H ℓ , r ( s ) ∣ d s + C ( 1 + c ℓ ) , 0 ≤ t ≤ T .
The backward Gronwall inequality therefore gives
sup 0 ≤ t ≤ T ∣ B H ℓ , r ( t ) ∣ ≤ C ( 1 + c ℓ ) . (126) \sup_{0\le t\le T}|B_{H\ell,r}(t)|\le C(1+c_\ell). \tag*{(126)} 0 ≤ t ≤ T sup ∣ B H ℓ , r ( t ) ∣ ≤ C ( 1 + c ℓ ) . (126)
Finally, substituting (125) –(126) into the ODE for C H ℓ , r C_{H\ell,r} C H ℓ , r and using C H ℓ , r ( T ) = 0 C_{H\ell,r}(T)=0 C H ℓ , r ( T ) = 0 gives
∣ C H ℓ , r ( t ) ∣ ≤ C ∫ t T [ 1 + ∣ B H ℓ , r ( s ) ∣ 2 + c ℓ ∣ B H ℓ , r ( s ) ∣ + c ℓ 2 ] d s ≤ C ( 1 + c ℓ 2 ) . |C_{H\ell,r}(t)|\le C\int_t^T\left[1+|B_{H\ell,r}(s)|^2+c_\ell|B_{H\ell,r}(s)|+c_\ell^2\right]\,ds\le C(1+c_\ell^2). ∣ C H ℓ , r ( t ) ∣ ≤ C ∫ t T [ 1 + ∣ B H ℓ , r ( s ) ∣ 2 + c ℓ ∣ B H ℓ , r ( s ) ∣ + c ℓ 2 ] d s ≤ C ( 1 + c ℓ 2 ) .
Consequently, by ODE bounds and (119) , it holds that
H ~ ( ℓ , r ) ≤ H ( 0 , m 0 ( ℓ ) ) exp { C R 0 ( ℓ ) ( 1 + ∣ m 0 ( ℓ ) ∣ 2 + ϕ ℓ 2 ) } ≤ exp { C ( 1 + ∣ m 0 ( ℓ ) ∣ 2 + c ℓ 2 ) } . (127) \widetilde{H}^{(\ell,r)}\le H(0,m_0^{(\ell)})\exp\left\{CR_0^{(\ell)}\left(1+|m_0^{(\ell)}|^2+\phi_\ell^2\right)\right\}\le\exp\left\{C\left(1+|m_0^{(\ell)}|^2+c_\ell^2\right)\right\}. \tag*{(127)} H ( ℓ , r ) ≤ H ( 0 , m 0 ( ℓ ) ) exp { C R 0 ( ℓ ) ( 1 + ∣ m 0 ( ℓ ) ∣ 2 + ϕ ℓ 2 ) } ≤ exp { C ( 1 + ∣ m 0 ( ℓ ) ∣ 2 + c ℓ 2 ) } . (127)
Conditional on F r w \mathcal{F}_r^w F r w , the m 0 ( ℓ ) m_0^{(\ell)} m 0 ( ℓ ) in the window are Gaussian random variables whose means are eventually bounded and whose variances are O ( ( r + 1 ) − 2 α ) O((r+1)^{-2\alpha}) O (( r + 1 ) − 2 α ) . Since each batch contains only finitely many episodes, the Gaussian quadratic exponential moment formula and (127) show that there exists an a.s. finite random constant C H C_H C H such that eventually
Q r : = 1 + max ℓ ∈ I r + 1 H w ( ℓ , r ) ≤ C H exp { C c r + 1 2 N w } a.s. (128) Q_r:=1+\max_{\ell\in\mathcal{I}_{r+1}}H_w^{(\ell,r)}\le C_H\exp\left\{Cc_{r+1}^2N_w\right\}\quad\text{a.s.} \tag*{(128)} Q r := 1 + ℓ ∈ I r + 1 max H w ( ℓ , r ) ≤ C H exp { C c r + 1 2 N w } a.s. (128)
By w r ∈ W r w_r \in\mathcal{W}_r w r ∈ W r and (124) , we obtain
E [ ∣ X T ( ℓ ) ∣ 2 ∣ F r w ] ≤ C ( 1 + c w , r 2 ) Q r . \mathbb{E}\left[\left|X_T^{(\ell)}\right|^2 \mid\mathcal{F}_r^w\right] \le C(1+c_{w,r}^2)Q_r. E [ X T ( ℓ ) 2 ∣ F r w ] ≤ C ( 1 + c w , r 2 ) Q r .
By (93) and linearity of conditional expectation, applying Jensen’s inequality to the convex function x ↦ ∣ x ∣ 2 x \mapsto|x|^2 x ↦ ∣ x ∣ 2 , we have
∣ X ‾ T , r + 1 − E [ X ‾ T , r + 1 ∣ F r w ] ∣ 2 ≤ 1 N w ∑ ℓ ∈ T r + 1 ∣ X T ( ℓ ) − E [ X T ( ℓ ) ∣ F r w ] ∣ 2 . \left|\overline{X}_{T,r+1}-\mathbb{E}\left[\overline{X}_{T,r+1}\mid\mathcal{F}_r^w\right]\right|^2 \le\frac{1}{N_w}\sum_{\ell\in\mathcal{T}_{r+1}}\left|X_T^{(\ell)}-\mathbb{E}\left[X_T^{(\ell)}\mid\mathcal{F}_r^w\right]\right|^2. X T , r + 1 − E [ X T , r + 1 ∣ F r w ] 2 ≤ N w 1 ℓ ∈ T r + 1 ∑ X T ( ℓ ) − E [ X T ( ℓ ) ∣ F r w ] 2 .
Taking expectation conditional on F r w \mathcal{F}_r^w F r w and using the conditional variance identity,
E [ ∣ X ‾ T , r + 1 − E [ X ‾ T , r + 1 ∣ F r w ] ∣ 2 ∣ F r w ] ≤ 1 N w ∑ ℓ ∈ T r + 1 E [ ∣ X T ( ℓ ) − E [ X T ( ℓ ) ∣ F r w ] ∣ 2 ∣ F r w ] ≤ 1 N w ∑ ℓ ∈ T r + 1 E [ ∣ X T ( ℓ ) ∣ 2 ∣ F r w ] ≤ C ( 1 + c w , r 2 ) Q r . (129) \begin{aligned}
\mathbb{E}\left[\left|\overline{X}_{T,r+1}-\mathbb{E}\left[\overline{X}_{T,r+1}\mid\mathcal{F}_r^w\right]\right|^2\mid\mathcal{F}_r^w\right]\le\frac{1}{N_w}\sum_{\ell\in\mathcal{T}_{r+1}}\mathbb{E}\left[\left|X_T^{(\ell)}-\mathbb{E}\left[X_T^{(\ell)}\mid\mathcal{F}_r^w\right]\right|^2\mid\mathcal{F}_r^w\right] \\
&\le\frac{1}{N_w}\sum_{\ell\in\mathcal{T}_{r+1}}\mathbb{E}\left[\left|X_T^{(\ell)}\right|^2\mid\mathcal{F}_r^w\right] \\
&\le C(1+c_{w,r}^2)Q_r. \tag*{(129)}
\end{aligned} E [ X T , r + 1 − E [ X T , r + 1 ∣ F r w ] 2 ∣ F r w ] ≤ N w 1 ℓ ∈ T r + 1 ∑ E [ X T ( ℓ ) − E [ X T ( ℓ ) ∣ F r w ] 2 ∣ F r w ] ≤ N w 1 ℓ ∈ T r + 1 ∑ E [ X T ( ℓ ) 2 ∣ F r w ] ≤ C ( 1 + c w , r 2 ) Q r . (129)
Choose a deterministic batch index r 0 r_0 r 0 so that all episodes in batches r ≥ r 0 r\ge r_0 r ≥ r 0 have ℓ ≥ ℓ 0 \ell\ge\ell_0 ℓ ≥ ℓ 0 . Enlarge ℓ 0 \ell_0 ℓ 0 if necessary to make the within-batch Gaussian quadratic exponential moments finite, using their variances O ( ( r + 1 ) − 2 α ) O((r+1)^{-2\alpha}) O (( r + 1 ) − 2 α ) . Since γ 3 > 1 / 2 \gamma_3>1/2 γ 3 > 1/2 , using (128) gives
∑ r = r 0 ∞ ( α w ( r ) ) 2 ( 1 + c w , r 2 ) Q r < ∞ a.s. (130) \sum_{r=r_0}^{\infty}(\alpha_w(r))^2(1+c_{w,r}^2)Q_r<\infty\quad\text{a.s.} \tag*{(130)} r = r 0 ∑ ∞ ( α w ( r ) ) 2 ( 1 + c w , r 2 ) Q r < ∞ a.s. (130)
We next estimate the batch first-moment factor. Similarly, Jensen’s inequality gives
E [ ∣ G w , r − G w ∗ ∣ 2 ] ≤ 1 N w ∑ ℓ ∈ T r + 1 E [ ∣ G w ( ℓ , r ) − G w ∗ ∣ 2 ] . (131) \mathbb{E}\left[|G_{w,r}-G_w^*|^2\right]\le\frac{1}{N_w}\sum_{\ell\in\mathcal{T}_{r+1}}\mathbb{E}\left[\left|G_w^{(\ell,r)}-G_w^*\right|^2\right]. \tag*{(131)} E [ ∣ G w , r − G w ∗ ∣ 2 ] ≤ N w 1 ℓ ∈ T r + 1 ∑ E [ G w ( ℓ , r ) − G w ∗ 2 ] . (131)
By the conditional Jensen inequality and the tower property,
E [ ∣ G w ( ℓ , r ) − G w ∗ ∣ 2 ] ≤ E [ ∣ G ~ w ( ℓ , r ) − G w ∗ ∣ 2 ] . \mathbb{E}\left[\left|G_w^{(\ell,r)}-G_w^*\right|^2\right]\le\mathbb{E}\left[\left|\widetilde{G}_w^{(\ell,r)}-G_w^*\right|^2\right]. E [ G w ( ℓ , r ) − G w ∗ 2 ] ≤ E [ G w ( ℓ , r ) − G w ∗ 2 ] .
For the episode-start factor, the first-moment formula in the paper gives
G ( 0 , m 0 ( ℓ ) ) = exp { p G ℓ , r ( 0 ) m 0 ( ℓ ) + q G ℓ , r ( 0 ) } . G(0,m_0^{(\ell)})=\exp\left\{p_{G\ell,r}(0)m_0^{(\ell)}+q_{G\ell,r}(0)\right\}. G ( 0 , m 0 ( ℓ ) ) = exp { p G ℓ , r ( 0 ) m 0 ( ℓ ) + q G ℓ , r ( 0 ) } .
The first-moment coefficient ODE listed in the proof of Lemma 2 is triangular: the equation for p G p_G p G is linear, and after p G p_G p G is obtained, q G q_G q G is determined by backward integration. By (71) , (76) , ∣ ϕ ℓ ∣ ≤ c ℓ |\phi_\ell|\le c_\ell ∣ ϕ ℓ ∣ ≤ c ℓ and the backward Gronwall inequality, it holds that sup 0 ≤ t ≤ T ∣ p G ℓ , r ( t ) ∣ ≤ C \sup_{0\le t\le T}|p_{G\ell,r}(t)|\le C sup 0 ≤ t ≤ T ∣ p G ℓ , r ( t ) ∣ ≤ C and sup 0 ≤ t ≤ T ∣ q G ℓ , r ( t ) ∣ ≤ C ( 1 + c ℓ ) \sup_{0\le t\le T}|q_{G\ell,r}(t)|\le C(1+c_\ell) sup 0 ≤ t ≤ T ∣ q G ℓ , r ( t ) ∣ ≤ C ( 1 + c ℓ ) .
Next compare these two ODEs term by term with the R = 0 R=0 R = 0 limit in (112) –(113) using the same ϕ ℓ \phi_\ell ϕ ℓ . By (71) and (40) , the same backward integration estimate gives
sup 0 ≤ t ≤ T ∣ p G ℓ , r ( t ) − p G ( ∞ ) ( t ) ∣ ≤ C R 0 ( ℓ ) , sup 0 ≤ t ≤ T ∣ q G ℓ , r ( t ) − q G ( ∞ ) ( t ) ∣ ≤ C R 0 ( ℓ ) ( 1 + c ℓ ) . \sup_{0\le t\le T}|p_{G\ell,r}(t)-p_G^{(\infty)}(t)|\le CR_0^{(\ell)},\qquad\sup_{0\le t\le T}|q_{G\ell,r}(t)-q_G^{(\infty)}(t)|\le CR_0^{(\ell)}(1+c_\ell). 0 ≤ t ≤ T sup ∣ p G ℓ , r ( t ) − p G ( ∞ ) ( t ) ∣ ≤ C R 0 ( ℓ ) , 0 ≤ t ≤ T sup ∣ q G ℓ , r ( t ) − q G ( ∞ ) ( t ) ∣ ≤ C R 0 ( ℓ ) ( 1 + c ℓ ) .
Also, by a c + b c = 1 a_c+b_c=1 a c + b c = 1 ,
p G ( ∞ ) ( 0 ) m 0 ( ℓ ) + q G ( ∞ ) ( 0 ) + ( ρ ∗ ) 2 T = − ρ ∗ T [ a c ( m 0 ( ℓ ) − ρ ∗ ) + b c ( ϕ ℓ − ρ ∗ ) ] . p_G^{(\infty)}(0)m_0^{(\ell)}+q_G^{(\infty)}(0)+(\rho^*)^2T=-\rho^*T\left[a_c(m_0^{(\ell)}-\rho^*)+b_c(\phi_\ell-\rho^*)\right]. p G ( ∞ ) ( 0 ) m 0 ( ℓ ) + q G ( ∞ ) ( 0 ) + ( ρ ∗ ) 2 T = − ρ ∗ T [ a c ( m 0 ( ℓ ) − ρ ∗ ) + b c ( ϕ ℓ − ρ ∗ ) ] .
Therefore
∣ p G ℓ , r ( 0 ) m 0 ( ℓ ) + q G ℓ , r ( 0 ) + ( ρ ∗ ) 2 T ∣ ≤ C [ ∣ m 0 ( ℓ ) − ρ ∗ ∣ + ∣ ϕ ℓ − ρ ∗ ∣ + R 0 ( ℓ ) ( 1 + ∣ m 0 ( ℓ ) ∣ + c ℓ ) ] . \left|p_{G\ell,r}(0)m_0^{(\ell)}+q_{G\ell,r}(0)+(\rho^*)^2T\right|\le C\left[|m_0^{(\ell)}-\rho^*|+|\phi_\ell-\rho^*|+R_0^{(\ell)}\left(1+|m_0^{(\ell)}|+c_\ell\right)\right]. p G ℓ , r ( 0 ) m 0 ( ℓ ) + q G ℓ , r ( 0 ) + ( ρ ∗ ) 2 T ≤ C [ ∣ m 0 ( ℓ ) − ρ ∗ ∣ + ∣ ϕ ℓ − ρ ∗ ∣ + R 0 ( ℓ ) ( 1 + ∣ m 0 ( ℓ ) ∣ + c ℓ ) ] .
By (118) , the preceding exponent bound also holds for ∣ log G ~ w ( ℓ , r ) + ( ρ ∗ ) 2 T ∣ |\log\widetilde{G}_w^{(\ell,r)}+(\rho^*)^2T| ∣ log G w ( ℓ , r ) + ( ρ ∗ ) 2 T ∣ , after increasing C C C . The mean value theorem for the exponential map and the preceding display give
∣ G ~ w ( ℓ , r ) − G w ∗ ∣ 2 ≤ C exp { C ( ∣ m 0 ( ℓ ) − ρ ∗ ∣ + c ℓ ) } \left|\widetilde{G}_w^{(\ell,r)}-G_w^*\right|^2\le C\exp\left\{C\left(|m_0^{(\ell)}-\rho^*|+c_\ell\right)\right\} G w ( ℓ , r ) − G w ∗ 2 ≤ C exp { C ( ∣ m 0 ( ℓ ) − ρ ∗ ∣ + c ℓ ) }
× [ ∣ m 0 ( ℓ ) − ρ ∗ ∣ 2 + ∣ ϕ ℓ − ρ ∗ ∣ 2 + ( R 0 ( ℓ ) ) 2 ( 1 + ∣ m 0 ( ℓ ) − ρ ∗ ∣ 2 + c ℓ 2 ) ] . (132) \qquad\times\left[|m_0^{(\ell)}-\rho^*|^2+|\phi_\ell-\rho^*|^2+\left(R_0^{(\ell)}\right)^2\left(1+|m_0^{(\ell)}-\rho^*|^2+c_\ell^2\right)\right]. \tag*{(132)} × [ ∣ m 0 ( ℓ ) − ρ ∗ ∣ 2 + ∣ ϕ ℓ − ρ ∗ ∣ 2 + ( R 0 ( ℓ ) ) 2 ( 1 + ∣ m 0 ( ℓ ) − ρ ∗ ∣ 2 + c ℓ 2 ) ] . (132)
The same estimates hold with c ℓ c_{\ell} c ℓ replaced by ∣ ϕ ℓ ∣ |\phi_{\ell}| ∣ ϕ ℓ ∣ , giving the uniform local bound used below. By the explicit recursion for filtering and the conditional second-moment bound (103) for the actor, for every realized sample path satisfying m 0 ( r N w + 1 ) → ρ ∗ m_{0}^{(rN_{w}+1)} \to\rho^{*} m 0 ( r N w + 1 ) → ρ ∗ and ϕ r N w + 1 → ρ ∗ \phi_{rN_{w}+1} \to\rho^{*} ϕ r N w + 1 → ρ ∗ , we have
Δ r : = max ℓ ∈ T r + 1 E [ ∣ m 0 ( ℓ ) − ρ ∗ ∣ 2 + ∣ ϕ ℓ − ρ ∗ ∣ 2 + ( R 0 ( ℓ ) ) 2 | F r w ] ⟶ 0. \Delta_{r} := \max_{\ell\in\mathcal{T}_{r+1}} \mathbb{E}\left[|m_{0}^{(\ell)}-\rho^{*}|^{2}+|\phi_{\ell}-\rho^{*}|^{2}+(R_{0}^{(\ell)})^{2}\middle|\mathcal{F}_{r}^{w}\right] \longrightarrow0. Δ r := ℓ ∈ T r + 1 max E [ ∣ m 0 ( ℓ ) − ρ ∗ ∣ 2 + ∣ ϕ ℓ − ρ ∗ ∣ 2 + ( R 0 ( ℓ ) ) 2 F r w ] ⟶ 0.
Fix δ > 0 \delta>0 δ > 0 , and let A r ( δ ) : = { max ℓ ∈ T r + 1 ( ∣ m 0 ( ℓ ) − ρ ∗ ∣ ∨ ∣ ϕ ℓ − ρ ∗ ∣ ) ≤ δ } \mathcal{A}_{r}(\delta):=\left\{\max_{\ell\in\mathcal{T}_{r+1}}\left(|m_{0}^{(\ell)}-\rho^{*}|\vee|\phi_{\ell}-\rho^{*}|\right)\le\delta\right\} A r ( δ ) := { max ℓ ∈ T r + 1 ( ∣ m 0 ( ℓ ) − ρ ∗ ∣ ∨ ∣ ϕ ℓ − ρ ∗ ∣ ) ≤ δ } . Using the local Lipschitz bound on A r ( δ ) \mathcal{A}_{r}(\delta) A r ( δ ) gives
max ℓ ∈ T r + 1 E [ ∣ G ~ w ( ℓ , r ) − G w ∗ ∣ 2 1 A r ( δ ) | F r w ] ≤ C Δ r ⟶ 0 a.s. \max_{\ell\in\mathcal{T}_{r+1}}\mathbb{E}\left[|\widetilde{G}_{w}^{(\ell,r)}-G_{w}^{*}|^{2}\mathbf{1}_{\mathcal{A}_{r}(\delta)}\middle|\mathcal{F}_{r}^{w}\right]\le C\Delta_{r}\longrightarrow0\quad\text{a.s.} ℓ ∈ T r + 1 max E [ ∣ G w ( ℓ , r ) − G w ∗ ∣ 2 1 A r ( δ ) F r w ] ≤ C Δ r ⟶ 0 a.s.
On the other hand, batch-start convergence ensures that eventually a.s. every batch starts inside the δ / 4 \delta/4 δ /4 -neighborhood:
∣ m 0 ( r N w + 1 ) − ρ ∗ ∣ ∨ ∣ ϕ r N w + 1 − ρ ∗ ∣ ≤ δ / 4. |m_{0}^{(rN_{w}+1)}-\rho^{*}|\vee|\phi_{rN_{w}+1}-\rho^{*}|\le\delta/4. ∣ m 0 ( r N w + 1 ) − ρ ∗ ∣ ∨ ∣ ϕ r N w + 1 − ρ ∗ ∣ ≤ δ /4.
Thus, A r ( δ ) c \mathcal{A}_{r}(\delta)^{c} A r ( δ ) c can occur only if a filtering fluctuation or a cumulative actor increment within the window has magnitude at least 3 δ / 4 3\delta/4 3 δ /4 . We control these two cases separately. By the explicit recursion in the proof of Lemma 1, conditional on F r w \mathcal{F}_{r}^{w} F r w , the variables m 0 ( ℓ ) − ρ ∗ m_{0}^{(\ell)}-\rho^{*} m 0 ( ℓ ) − ρ ∗ in the window are Gaussian. Eventually a.s.,
max ℓ ∈ T r + 1 ∣ E [ m 0 ( ℓ ) − ρ ∗ ∣ F r w ] ∣ ≤ δ / 4 , max ℓ ∈ T r + 1 Var ( m 0 ( ℓ ) − ρ ∗ ∣ F r w ) ≤ C ( r + 1 ) − 2 α . \max_{\ell\in\mathcal{T}_{r+1}}\left|\mathbb{E}[m_{0}^{(\ell)}-\rho^{*}\mid\mathcal{F}_{r}^{w}]\right|\le\delta/4,\qquad
\max_{\ell\in\mathcal{T}_{r+1}}\operatorname{Var}(m_{0}^{(\ell)}-\rho^{*}\mid\mathcal{F}_{r}^{w})\le C(r+1)^{-2\alpha}. ℓ ∈ T r + 1 max E [ m 0 ( ℓ ) − ρ ∗ ∣ F r w ] ≤ δ /4 , ℓ ∈ T r + 1 max Var ( m 0 ( ℓ ) − ρ ∗ ∣ F r w ) ≤ C ( r + 1 ) − 2 α .
Therefore, for every ℓ ∈ T r + 1 \ell\in\mathcal{T}_{r+1} ℓ ∈ T r + 1 ,
P ( ∣ m 0 ( ℓ ) − ρ ∗ ∣ > δ | F r w ) ≤ P ( ∣ m 0 ( ℓ ) − ρ ∗ − E [ m 0 ( ℓ ) − ρ ∗ ∣ F r w ] ∣ > 3 δ 4 | F r w ) . \mathbb{P}\left(|m_{0}^{(\ell)}-\rho^{*}|>\delta\middle|\mathcal{F}_{r}^{w}\right)\le\mathbb{P}\left(\left|m_{0}^{(\ell)}-\rho^{*}-\mathbb{E}[m_{0}^{(\ell)}-\rho^{*}\mid\mathcal{F}_{r}^{w}]\right|>\frac{3\delta}{4}\middle|\mathcal{F}_{r}^{w}\right). P ( ∣ m 0 ( ℓ ) − ρ ∗ ∣ > δ F r w ) ≤ P ( m 0 ( ℓ ) − ρ ∗ − E [ m 0 ( ℓ ) − ρ ∗ ∣ F r w ] > 4 3 δ F r w ) .
The Gaussian tail bound then gives, for some κ 1 > 0 \kappa_{1}>0 κ 1 > 0 ,
P ( max ℓ ∈ T r + 1 ∣ m 0 ( ℓ ) − ρ ∗ ∣ > δ | F r w ) ≤ C e − 2 κ 1 ( r + 1 ) 2 α . \mathbb{P}\left(\max_{\ell\in\mathcal{T}_{r+1}}|m_{0}^{(\ell)}-\rho^{*}|>\delta\middle|\mathcal{F}_{r}^{w}\right)\le C e^{-2\kappa_{1}(r+1)^{2\alpha}}. P ( ℓ ∈ T r + 1 max ∣ m 0 ( ℓ ) − ρ ∗ ∣ > δ F r w ) ≤ C e − 2 κ 1 ( r + 1 ) 2 α .
On the other hand, since ϕ k ∈ Φ k ⊆ Φ k + 1 \phi_{k}\in\Phi_{k}\subseteq\Phi_{k+1} ϕ k ∈ Φ k ⊆ Φ k + 1 , we have ∣ ϕ k + 1 − ϕ k ∣ ≤ α ϕ ( k ) ∣ F ^ k + 1 c ( ϕ k , θ k ) ∣ |\phi_{k+1}-\phi_{k}|\le\alpha_{\phi}(k)|\widehat{F}_{k+1}^{c}(\phi_{k},\theta_{k})| ∣ ϕ k + 1 − ϕ k ∣ ≤ α ϕ ( k ) ∣ F k + 1 c ( ϕ k , θ k ) ∣ . Together with the batch-start bound, this yields
{ max ℓ ∈ T r + 1 ∣ ϕ ℓ − ρ ∗ ∣ > δ } ⊆ { ∑ k = r N w + 1 ( r + 1 ) N w − 1 α ϕ ( k ) ∣ F ^ k + 1 c ( ϕ k , θ k ) ∣ > 3 δ 4 } . \left\{\max_{\ell\in\mathcal{T}_{r+1}}|\phi_{\ell}-\rho^{*}|>\delta\right\}\subseteq\left\{\sum_{k=rN_{w}+1}^{(r+1)N_{w}-1}\alpha_{\phi}(k)|\widehat{F}_{k+1}^{c}(\phi_{k},\theta_{k})|>\frac{3\delta}{4}\right\}. { ℓ ∈ T r + 1 max ∣ ϕ ℓ − ρ ∗ ∣ > δ } ⊆ ⎩ ⎨ ⎧ k = r N w + 1 ∑ ( r + 1 ) N w − 1 α ϕ ( k ) ∣ F k + 1 c ( ϕ k , θ k ) ∣ > 4 3 δ ⎭ ⎬ ⎫ .
Therefore, the conditional Markov inequality and the Cauchy–Schwarz inequality give
P ( max ℓ ∈ T r + 1 ∣ ϕ ℓ − ρ ∗ ∣ > δ | F r w ) ≤ 16 9 δ 2 E [ ( ∑ k = r N w + 1 ( r + 1 ) N w − 1 α ϕ ( k ) ∣ F ^ k + 1 c ( ϕ k , θ k ) ∣ ) 2 | F r w ] ≤ 16 ( N w − 1 ) 9 δ 2 ∑ k = r N w + 1 ( r + 1 ) N w − 1 ( α ϕ ( k ) ) 2 E [ ∣ F ^ k + 1 c ( ϕ k , θ k ) ∣ 2 | F r w ] ≤ C ( r + 1 ) − 2 γ 2 ( 1 + c ( r + 1 ) N w 4 ) = O ( ( r + 1 ) − 2 κ 2 ) a.s. \begin{aligned}
\mathbb{P}\left(\max_{\ell\in\mathcal{T}_{r+1}}|\phi_{\ell}-\rho^{*}|>\delta\middle|\mathcal{F}_{r}^{w}\right)\le\frac{16}{9\delta^{2}}\mathbb{E}\left[\left(\sum_{k=rN_{w}+1}^{(r+1)N_{w}-1}\alpha_{\phi}(k)|\widehat{F}_{k+1}^{c}(\phi_{k},\theta_{k})|\right)^{2}\middle|\mathcal{F}_{r}^{w}\right] \\
&\le\frac{16(N_{w}-1)}{9\delta^{2}}\sum_{k=rN_{w}+1}^{(r+1)N_{w}-1}(\alpha_{\phi}(k))^{2}\mathbb{E}\left[|\widehat{F}_{k+1}^{c}(\phi_{k},\theta_{k})|^{2}\middle|\mathcal{F}_{r}^{w}\right] \\
&\le C(r+1)^{-2\gamma_{2}}\left(1+c_{(r+1)N_{w}}^{4}\right)=O((r+1)^{-2\kappa_{2}})\quad\text{a.s.}
\end{aligned} P ( ℓ ∈ T r + 1 max ∣ ϕ ℓ − ρ ∗ ∣ > δ F r w ) ≤ 9 δ 2 16 E k = r N w + 1 ∑ ( r + 1 ) N w − 1 α ϕ ( k ) ∣ F k + 1 c ( ϕ k , θ k ) ∣ 2 F r w ≤ 9 δ 2 16 ( N w − 1 ) k = r N w + 1 ∑ ( r + 1 ) N w − 1 ( α ϕ ( k ) ) 2 E [ ∣ F k + 1 c ( ϕ k , θ k ) ∣ 2 F r w ] ≤ C ( r + 1 ) − 2 γ 2 ( 1 + c ( r + 1 ) N w 4 ) = O (( r + 1 ) − 2 κ 2 ) a.s.
where 0 < κ 2 < γ 2 0<\kappa_{2}<\gamma_{2} 0 < κ 2 < γ 2 is arbitrary; the last inequality uses (103) and the tower property. In addition, (132) and the conditional Gaussian exponential-moment bound give eventually a.s.
∣ G ~ w ( ℓ , r ) − G w ∗ ∣ 2 ≤ C e c ( r + 1 ) N w + ∣ m 0 ( ℓ ) − ρ ∗ ∣ ( 1 + c ( r + 1 ) N w 2 ) ( 1 + ∣ m 0 ( ℓ ) − ρ ∗ ∣ 2 ) , |\widetilde{G}_{w}^{(\ell,r)}-G_{w}^{*}|^{2}\le Ce^{c(r+1)N_{w}+|m_{0}^{(\ell)}-\rho^{*}|}\left(1+c_{(r+1)N_{w}}^{2}\right)\left(1+|m_{0}^{(\ell)}-\rho^{*}|^{2}\right), ∣ G w ( ℓ , r ) − G w ∗ ∣ 2 ≤ C e c ( r + 1 ) N w + ∣ m 0 ( ℓ ) − ρ ∗ ∣ ( 1 + c ( r + 1 ) N w 2 ) ( 1 + ∣ m 0 ( ℓ ) − ρ ∗ ∣ 2 ) ,
max ℓ ∈ T r + 1 E [ e 2 C ∣ m 0 ( ℓ ) − ρ ∗ ∣ ( 1 + ∣ m 0 ( ℓ ) − ρ ∗ ∣ 2 ) 2 | F r w ] ≤ C . \max_{\ell\in\mathcal{T}_{r+1}}\mathbb{E}\left[e^{2C|m_{0}^{(\ell)}-\rho^{*}|}\left(1+|m_{0}^{(\ell)}-\rho^{*}|^{2}\right)^{2}\middle|\mathcal{F}_{r}^{w}\right]\le C. ℓ ∈ T r + 1 max E [ e 2 C ∣ m 0 ( ℓ ) − ρ ∗ ∣ ( 1 + ∣ m 0 ( ℓ ) − ρ ∗ ∣ 2 ) 2 F r w ] ≤ C .
Thus the conditional Cauchy–Schwarz inequality gives
max ℓ ∈ T r + 1 E [ ∣ G ~ w ( ℓ , r ) − G w ∗ ∣ 2 1 A r ( δ ) c | F r w ] ≤ exp { C c ( r + 1 ) N w } [ e − κ 1 ( r + 1 ) 2 α + O ( ( r + 1 ) − κ 2 ) ] , \max_{\ell\in\mathcal{T}_{r+1}}\mathbb{E}\left[|\widetilde{G}_{w}^{(\ell,r)}-G_{w}^{*}|^{2}\mathbf{1}_{\mathcal{A}_{r}(\delta)^{c}}\middle|\mathcal{F}_{r}^{w}\right]\le\exp\{Cc_{(r+1)N_{w}}\}\left[e^{-\kappa_{1}(r+1)^{2\alpha}}+O((r+1)^{-\kappa_{2}})\right], ℓ ∈ T r + 1 max E [ ∣ G w ( ℓ , r ) − G w ∗ ∣ 2 1 A r ( δ ) c F r w ] ≤ exp { C c ( r + 1 ) N w } [ e − κ 1 ( r + 1 ) 2 α + O (( r + 1 ) − κ 2 ) ] ,
The last display is clearly o ( 1 ) o(1) o ( 1 ) . Combining the two parts above and using the conditional Jensen inequality,
max ℓ ∈ I r + 1 ∣ G w ( ℓ , r ) − G w ∗ ∣ 2 ≤ max ℓ ∈ I r + 1 E [ ∣ G ~ w ( ℓ , r ) − G w ∗ ∣ 2 ∣ F r w ] ≤ C Δ r + o ( 1 ) ⟶ 0 a.s. \max_{\ell\in\mathcal{I}_{r+1}} \left|G_w^{(\ell,r)} - G_w^*\right|^2 \le\max_{\ell\in\mathcal{I}_{r+1}} \mathbb{E}\left[\left|\widetilde{G}_w^{(\ell,r)} - G_w^*\right|^2 \mid\mathcal{F}_r^w\right] \le C\Delta_r + o(1) \longrightarrow0 \quad\text{a.s.} ℓ ∈ I r + 1 max G w ( ℓ , r ) − G w ∗ 2 ≤ ℓ ∈ I r + 1 max E [ G w ( ℓ , r ) − G w ∗ 2 ∣ F r w ] ≤ C Δ r + o ( 1 ) ⟶ 0 a.s.
The finite average therefore satisfies G ‾ w , r → G w ∗ \overline{G}_{w,r} \to G_w^* G w , r → G w ∗ a.s. Thus we obtain
G ‾ w , r ⟶ G w ∗ a.s. (133) \overline{G}_{w,r} \longrightarrow G_w^* \quad\text{a.s.} \tag*{(133)} G w , r ⟶ G w ∗ a.s. (133)
Proof. It suffices to consider the tail r ≥ r 0 r \ge r_0 r ≥ r 0 , where the sampled-wealth conditional moments are finite. Write U w , r : = w r − w ∗ U_{w,r} := w_r - w^* U w , r := w r − w ∗ , ξ w , r + 1 : = X T , r + 1 − E [ X T , r + 1 ∣ F r w ] \xi_{w,r+1} := X_{T,r+1} - \mathbb{E}[X_{T,r+1} \mid\mathcal{F}_r^w] ξ w , r + 1 := X T , r + 1 − E [ X T , r + 1 ∣ F r w ] . By Theorem 6 and (123),
( 1 − G w ∗ ) w ∗ + G w ∗ x 0 − z = 0. (1-G_w^*)w^* + G_w^*x_0 - z = 0. ( 1 − G w ∗ ) w ∗ + G w ∗ x 0 − z = 0.
So the conditional mean in (122) can be written as
E [ X T , r + 1 − z ∣ F r w ] = ( 1 − G ‾ w , r ) U w , r + δ w , r , δ w , r : = ( G ‾ w , r − G w ∗ ) ( x 0 − w ∗ ) . (134) \mathbb{E}[X_{T,r+1} - z \mid\mathcal{F}_r^w] = (1-\overline{G}_{w,r})U_{w,r} + \delta_{w,r}, \qquad\delta_{w,r} := (\overline{G}_{w,r} - G_w^*)(x_0-w^*). \tag*{(134)} E [ X T , r + 1 − z ∣ F r w ] = ( 1 − G w , r ) U w , r + δ w , r , δ w , r := ( G w , r − G w ∗ ) ( x 0 − w ∗ ) . (134)
Since c w , r ↑ ∞ c_{w,r} \uparrow\infty c w , r ↑ ∞ , there exists a deterministic finite index r w r_w r w such that w ∗ ∈ W r w^* \in\mathcal{W}_r w ∗ ∈ W r for all r ≥ r w r \ge r_w r ≥ r w . Therefore, for all r ≥ r w r \ge r_w r ≥ r w , projection non-expansiveness gives
∣ U w , r + 1 ∣ 2 ≤ ∣ U w , r − α w ( r ) [ ( 1 − G ‾ w , r ) U w , r + δ w , r + ξ w , r + 1 ] ∣ 2 . |U_{w,r+1}|^2 \le\left|U_{w,r} - \alpha_w(r)\left[(1-\overline{G}_{w,r})U_{w,r}+\delta_{w,r}+\xi_{w,r+1}\right]\right|^2. ∣ U w , r + 1 ∣ 2 ≤ U w , r − α w ( r ) [ ( 1 − G w , r ) U w , r + δ w , r + ξ w , r + 1 ] 2 .
Since G ‾ w , r → G w ∗ < 1 \overline{G}_{w,r} \to G_w^* < 1 G w , r → G w ∗ < 1 , there exists a random but finite r w s t a b ≥ r w r_w^{\mathrm{stab}} \ge r_w r w stab ≥ r w such that, for all r ≥ r w s t a b r \ge r_w^{\mathrm{stab}} r ≥ r w stab ,
1 − G w ∗ 2 ≤ 1 − G ‾ w , r ≤ 1 + G w ∗ = : C G < ∞ . \frac{1-G_w^*}{2} \le1-\overline{G}_{w,r} \le1+G_w^* =: C_G < \infty. 2 1 − G w ∗ ≤ 1 − G w , r ≤ 1 + G w ∗ =: C G < ∞.
We again use the squared distance to a fixed neighborhood. Fix 0 < ε < ( w ∗ − w ) / 2 0 < \varepsilon< (w^*-w)/2 0 < ε < ( w ∗ − w ) /2 , and define
D ε w : = [ − ε , ε ] , V w , r ( ε ) : = ∣ U w , r − Π D ε w ( U w , r ) ∣ 2 . D_\varepsilon^w := [-\varepsilon,\varepsilon], \qquad V_{w,r}^{(\varepsilon)} := \left|U_{w,r}-\Pi_{D_\varepsilon^w}(U_{w,r})\right|^2. D ε w := [ − ε , ε ] , V w , r ( ε ) := U w , r − Π D ε w ( U w , r ) 2 .
Since c w , r ↑ ∞ c_{w,r} \uparrow\infty c w , r ↑ ∞ and w < w ∗ w < w^* w < w ∗ , eventually D ε w ⊆ W r + 1 − w ∗ D_\varepsilon^w \subseteq\mathcal{W}_{r+1}-w^* D ε w ⊆ W r + 1 − w ∗ . Applying (105)–(106) and taking conditional expectation given F r w \mathcal{F}_r^w F r w yields
E [ V w , r + 1 ( ε ) ∣ F r w ] ≤ V w , r ( ε ) − 2 α w ( r ) ( U w , r − Π D ε w ( U w , r ) ) [ ( 1 − G ‾ w , r ) U w , r + δ w , r ] + ( α w ( r ) ) 2 E [ ∣ ( 1 − G ‾ w , r ) U w , r + δ w , r + ξ w , r + 1 ∣ 2 ∣ F r w ] . (135) \mathbb{E}\left[V_{w,r+1}^{(\varepsilon)} \mid\mathcal{F}_r^w\right] \le V_{w,r}^{(\varepsilon)} - 2\alpha_w(r)\left(U_{w,r}-\Pi_{D_\varepsilon^w}(U_{w,r})\right)\left[(1-\overline{G}_{w,r})U_{w,r}+\delta_{w,r}\right]
+ \left(\alpha_w(r)\right)^2\mathbb{E}\left[\left|(1-\overline{G}_{w,r})U_{w,r}+\delta_{w,r}+\xi_{w,r+1}\right|^2 \mid\mathcal{F}_r^w\right]. \tag*{(135)} E [ V w , r + 1 ( ε ) ∣ F r w ] ≤ V w , r ( ε ) − 2 α w ( r ) ( U w , r − Π D ε w ( U w , r ) ) [ ( 1 − G w , r ) U w , r + δ w , r ] + ( α w ( r ) ) 2 E [ ( 1 − G w , r ) U w , r + δ w , r + ξ w , r + 1 2 ∣ F r w ] . (135)
Let κ w : = ( 1 − G w ∗ ) / 2 > 0 \kappa_w := (1-G_w^*)/2 > 0 κ w := ( 1 − G w ∗ ) /2 > 0 . By (133) and (134), eventually we have 1 − G ‾ w , r ≥ κ w 1-\overline{G}_{w,r} \ge\kappa_w 1 − G w , r ≥ κ w and ∣ δ w , r ∣ ≤ κ w ε / 2 |\delta_{w,r}| \le\kappa_w\varepsilon/2 ∣ δ w , r ∣ ≤ κ w ε /2 . As in the actor case, U w , r − Π D ε w ( U w , r ) U_{w,r}-\Pi_{D_\varepsilon^w}(U_{w,r}) U w , r − Π D ε w ( U w , r ) has the same sign as U w , r U_{w,r} U w , r when ∣ U w , r ∣ > ε |U_{w,r}|>\varepsilon ∣ U w , r ∣ > ε , and hence eventually
( U w , r − Π D ε w ( U w , r ) ) [ ( 1 − G ‾ w , r ) U w , r + δ w , r ] ≥ κ w V w , r ( ε ) . (136) \left(U_{w,r}-\Pi_{D_\varepsilon^w}(U_{w,r})\right)\left[(1-\overline{G}_{w,r})U_{w,r}+\delta_{w,r}\right] \ge\kappa_w V_{w,r}^{(\varepsilon)}. \tag*{(136)} ( U w , r − Π D ε w ( U w , r ) ) [ ( 1 − G w , r ) U w , r + δ w , r ] ≥ κ w V w , r ( ε ) . (136)
Also, by w r ∈ W r w_r \in\mathcal{W}_r w r ∈ W r , the eventual bound ∣ 1 − G ‾ w , r ∣ ≤ 1 + G w ∗ |1-\overline{G}_{w,r}| \le1+G_w^* ∣1 − G w , r ∣ ≤ 1 + G w ∗ , and (129),
E [ ∣ ( 1 − G ‾ w , r ) U w , r + δ w , r + ξ w , r + 1 ∣ 2 ∣ F r w ] ≤ C ( ( 1 + c w , r 2 ) Q r + ∣ δ w , r ∣ 2 ) . \mathbb{E}\left[\left|(1-\overline{G}_{w,r})U_{w,r}+\delta_{w,r}+\xi_{w,r+1}\right|^2 \mid\mathcal{F}_r^w\right] \le C\left((1+c_{w,r}^2)Q_r+|\delta_{w,r}|^2\right). E [ ( 1 − G w , r ) U w , r + δ w , r + ξ w , r + 1 2 ∣ F r w ] ≤ C ( ( 1 + c w , r 2 ) Q r + ∣ δ w , r ∣ 2 ) .
Finally, by (130), δ w , r → 0 \delta_{w,r}\to0 δ w , r → 0 , and ∑ r ( α w ( r ) ) 2 < ∞ \sum_r(\alpha_w(r))^2<\infty ∑ r ( α w ( r ) ) 2 < ∞ ,
∑ r C ( α w ( r ) ) 2 ( ( 1 + c w , r 2 ) Q r + ∣ δ w , r ∣ 2 ) < ∞ a.s. \sum_r C(\alpha_w(r))^2\left((1+c_{w,r}^2)Q_r+|\delta_{w,r}|^2\right)<\infty\quad\text{a.s.} r ∑ C ( α w ( r ) ) 2 ( ( 1 + c w , r 2 ) Q r + ∣ δ w , r ∣ 2 ) < ∞ a.s.
Therefore (135)–(136) gives
E [ V w , r + 1 ( ε ) ∣ F r w ] ≤ V w , r ( ε ) − 2 κ w α w ( r ) V w , r ( ε ) + C ( α w ( r ) ) 2 ( ( 1 + c w , r 2 ) Q r + ∣ δ w , r ∣ 2 ) . \mathbb{E}\left[V_{w,r+1}^{(\varepsilon)} \mid\mathcal{F}_r^w\right] \le V_{w,r}^{(\varepsilon)} - 2\kappa_w\alpha_w(r)V_{w,r}^{(\varepsilon)} + C(\alpha_w(r))^2\left((1+c_{w,r}^2)Q_r+|\delta_{w,r}|^2\right). E [ V w , r + 1 ( ε ) ∣ F r w ] ≤ V w , r ( ε ) − 2 κ w α w ( r ) V w , r ( ε ) + C ( α w ( r ) ) 2 ( ( 1 + c w , r 2 ) Q r + ∣ δ w , r ∣ 2 ) .
The Robbins–Siegmund theorem [38 ] and ∑ r α w ( r ) = ∞ \sum_r\alpha_w(r)=\infty ∑ r α w ( r ) = ∞ give V w , r ( ε ) → 0 V_{w,r}^{(\varepsilon)}\to0 V w , r ( ε ) → 0 a.s. Finally, take
ε m : = min { 1 m , w ∗ − w 4 } , m ≥ 1 , \varepsilon_m := \min\left\{\frac{1}{m},\frac{w^*-w}{4}\right\}, \qquad m\ge1, ε m := min { m 1 , 4 w ∗ − w } , m ≥ 1 ,
and intersect the countably many probability-one events to obtain U w , r → 0 U_{w,r}\to0 U w , r → 0 a.s., that is, w r → w ∗ w_r\to w^* w r → w ∗ a.s. \quad □ \square □