betaFeedback

WorksEconomics, Econometrics and Finance

Certifying Reward Alignment in Portfolio Reinforcement Learning

Abstract

Financial reinforcement learning often translates an investment objective into an additive risk-adjusted reward. An agent can optimise that reward accurately yet implement a different financial preference. We develop a decision audit for terminal mean-variance equilibrium that separates reward misspecification from learning error. In a finite exogenous-state market, we bound the gain from changing one action using actor, reward and continuation-critic residuals. The certificate ignores state-constant critic offsets, accommodates convex action constraints, and has an independent-sample extension that does not require the equilibrium policy. A continuation-aware actor-critic method estimates the future-policy correction from samples. Across nine controlled markets and twenty seeds, it reduces policy distance by and one-step deviation gain by relative to a matched myopic-reward network. A separate PPO comparison tests the same reward intervention within a standard optimiser. Historical tests cross seven covariance estimators and 10, 30 and 49 US industry portfolios over 432 months: joint reward calibration preserves each fitted mean-variance target, but does not improve realised investment performance. Together, these results provide a way to quantify whether a learned decision implements its stated financial objective, separately from how well that objective performs in a backtest.

CCS Concepts

• Computing methodologies → Reinforcement learning; • Applied computing → Economics.

Keywords

financial reinforcement learning, reward alignment, equilibrium, actor–critic, model validation

Introduction

An investment mandate specifies preferences over return, risk and costs; an RL agent maximises a reward. The translation can change decisions: replacing variance by a second moment penalises squared expected return; ignoring dependence removes a continuation hedge; correcting risk without scaling costs changes turnover preferences. Accurate reward optimisation or a favourable backtest does not establish mandate alignment.

Kolm and Ritter [18], pp. 90–91, place mandate-aligned reward specification before training. We ask whether the reward induces the intended decisions and the agent implements them. Our target is conditional terminal mean–variance equilibrium. When computable, it permits direct policy comparison; otherwise, we bound the improvement from changing one action while retaining the agent’s future policy.

Our residual certificate separates actor, reward and continuation errors. A zero actor gradient can conceal a wrong critic, while large state-constant critic errors can be harmless. Tracking state differences and their propagation distinguishes these cases and supplies an acceptance criterion under explicit assumptions. A continuation-aware learner estimates the correction from samples, without an equilibrium oracle in training.

Controlled actor–critic and PPO comparisons hold optimiser and data access fixed. Historical tests separately vary covariance and rewards: Markowitz [24] distinguishes beliefs from decisions, while DeMiguel et al. [7] show that estimation error can overwhelm diversification gains. Preserving a fitted objective need not improve realised returns.

Contributions. We contribute: (i) a certificate for finite-iterate equilibrium deviations, invariant to state-constant critic offsets and extendable to convex constraints and independent data; (ii) continuation-aware sample learning tested in matched neural and PPO comparisons; and (iii) a historical comparison separating alignment, estimation risk and performance. The new link is from reward and learning residuals to financial decision quality, building on established equilibrium learning, mean auxiliaries and concentration bounds.

Roadmap. Section 3 defines two distinct targets: terminal equilibrium and fitted one-period allocation. Sections 4–5 certify and learn the former. Section 6 tests sequential learning, then historical one-period calibration; Section 7 interprets validation decisions. Sections 8–10 provide technical and reproduction details.

Related work

Financial objectives and estimation. DeMiguel, Garlappi and Uppal [7], Tables 3–5, compare fourteen models with 1/N1/N. Proposition 1 and Figure 1 relate estimation requirements to dimension and opportunities, not a minimum asset count. Table 2 includes ten industries plus the market; we also test finer classifications. We implement linear shrinkage [19, 20], observed-factor covariance [9] and QuEST nonlinear shrinkage. Ledoit and Wolf [21], Corollary 1 and Eqs. (14)–(15), correct eigenvalues individually while retaining sample eigenvectors. Their Table 8 motivates holding long-only constraints and costs fixed.

Figure 1. Neural learning across twenty seeds and nine markets. Markets are averaged within seed; bands show descriptive mean ±1.96\pm1.96 standard errors, not simultaneous intervals. The certificate audits policy-relevant errors.

Gârleanu and Pedersen [11], Eq. (4) and Propositions 2–3, derive cost-aware dynamic allocation; DeMiguel et al. [8], Eq. (6) and Tables II–III, study conditional multifactor portfolios with costs. Their dynamic targets differ from our rolling one-period reference. Changing covariance or intentionally changing preferences is not reward misspecification.

Financial RL and reward design. Kolm and Ritter’s earlier hedging work [17], Eqs. (8)–(10), explicitly treats quadratic increment rewards as an approximation. Jiang et al. [16], Eq. (8), combine log return and variance; their 2024 formulation [15], Eq. (4), also considers costs. Habibnia and Soltanzadeh [14], Eq. (8), use asymmetric PnL with SAC, shorting and lending. Table 2 adapts reward formulations to common states; it does not reproduce these papers’ complete agents or backtests.

Objective-aware learning is established. The reward in Simon et al. [28], Eq. (17), is centred mean–variance utility. Gollart and Okhrin [12], Section 4.2.1, compare learned policies with an analytical benchmark. Cong et al. [4], Eqs. (19)–(20), permit differentiable trajectory objectives. Kolm and Ritter [18] survey the area and discuss validation against known policies (p. 87), so benchmark comparison itself is not new.

Equilibrium learning and certification. Precommitment [23] and equilibrium [1] solve different decision problems. Continuation hedges [5], equivalent objectives [2] and equilibrium RL [6] precede our work. Lesmana and Pun [22] develop subgame-perfect equilibrium RL and backward policy iteration; our implementation specialises these principles rather than replicating their algorithm. Guo et al. [13], Algorithms 1–3, already combine auxiliary rewards, deterministic policy gradients and fixed-point critic learning. We do not propose a new general equilibrium solver.

Sangadi et al. [26], Theorem 4.3, provide finite-sample actor–critic stationarity analysis for a discounted return mean–variance objective. Our target is instead a posteriori, statewise one-step deviation gain for a given finite-horizon policy. Seminorm learning and irrelevant critic offsets are also established [3]. We use that geometry to obtain a financial error certificate. The scalar reward-calibration identity below follows the mean auxiliary in MVPI [29]; the empirical Bernstein allowance follows Maurer and Pontil [25]. The contribution lies in the resulting decision audit, not in inventing these ingredients.

Objectives and reward calibration

Terminal and fitted references

Let ZtZ_t be a finite exogenous Markov state and let excess returns Rt∈RdR_t \in\mathbb{R}^d be jointly distributed with Zt+1Z_{t+1} conditional on ZtZ_t. Wealth obeys

Xt+1=sXt+RtTut(Zt),Ht=sT−t−1,(1)X_{t+1}=sX_t+R_t^\mathsf{T}u_t(Z_t), \qquad H_t=s^{T-t-1}, \tag*{(1)}

where s>0s>0 is deterministic. The financial criterion at each state is conditional terminal mean minus φ>0\varphi>0 times terminal variance. We consider deterministic wealth-independent policies and their one-step deviations, holding future policies fixed. This equilibrium criterion differs from maximising initial-date precommitment value.

For independent returns with mean mtm_t and covariance Σt≻0\Sigma_t \succ0, equilibrium is uteq=Σt−1mt/(2φHt)u_t^{\mathrm{eq}}=\Sigma_t^{-1}m_t/(2\varphi H_t). With Mt=Σt+mtmtTM_t=\Sigma_t+m_tm_t^\mathsf{T} and vt=mtTMt−1mtv_t=m_t^\mathsf{T}M_t^{-1}m_t, the additive reward

Yt−φ(1−vt)Yt2,Yt=HtRtTut,(2)Y_t-\varphi(1-v_t)Y_t^2, \qquad Y_t=H_tR_t^\mathsf{T}u_t, \tag*{(2)}

recovers that policy. This specialises prior equivalent-objective constructions [2]. Under dependence, continuation covariance is essential; Eq. (2) alone is insufficient.

For the raw family ΔX−κ(ΔX)2\Delta X-\kappa(\Delta X)^2, even optimal constant scalar tuning leaves a discrepancy. With independent constant moments and m≠0m\ne0, d=M−1md=M^{-1}m, v=mTdv=m^\mathsf{T}d, b=[2φ(1−v)]−1b=[2\varphi(1-v)]^{-1} and fixed T≥2T\ge2,

min⁡κ>0D=cT∣s−1∣+O((s−1)2),(3)\min_{\kappa>0}D=c_T|s-1|+O((s-1)^2), \tag*{(3)}
cT=∥d∥bT(T2−1)3+v(1−v)T(T−1)2.c_T=\lVert d\rVert b\sqrt{\frac{T(T^2-1)}{3}+\frac{v(1-v)T(T-1)}{2}}.

Here D2=∑tEπκ∥πκ(t,Xt)−uteq∥2D^2=\sum_t\mathbb{E}_{\pi^\kappa}\lVert\pi^\kappa(t,X_t)-u_t^{\mathrm{eq}}\rVert^2 uses the raw-reward optimum’s occupancy. Section 9 supplies the derivation. This obstruction concerns one reward family, not all additive rewards.

Constraints and deterministic costs

For historical data we use a separate, fitted one-period target

F(u)=mTu−φuTΣu−g(u),u∈C,(4)F(u)=m^\mathsf{T}u-\varphi u^\mathsf{T}\Sigma u-g(u), \qquad u\in\mathcal{C}, \tag*{(4)}

where C\mathcal{C} is the simplex and g(u)=λ∥u−p∥g(u)=\lambda\lVert u-p\rVert at pre-trade holdings pp. Its optimum u∗u^* is not the true historical market optimum. For a=mTu∗a=m^\mathsf{T}u^* and A=1+2φa>0A=1+2\varphi a>0, define

Qa(u)=mTu−φAuTMu−g(u)A.(5)Q_a(u)=m^\mathsf{T}u-\frac{\varphi}{A}u^\mathsf{T}Mu-\frac{g(u)}{A}. \tag*{(5)}

The identity

AQa(u)=F(u)−φ(mTu−a)2+φa2(6)AQ_a(u)=F(u)-\varphi(m^\mathsf{T}u-a)^2+\varphi a^2 \tag*{(6)}

proves that both objectives have the same maximiser. It is an application of the MVPI auxiliary [29], not a new general algorithm. Correcting risk alone introduces the extra term −2φag(u)-2\varphi ag(u) and can change the cost preference.

For any anchor aa with A>0A>0, an ϵ\epsilon-optimal u^\hat{u} for QaQ_a satisfies

F(u∗)−F(u^)≤φ(mTu∗−a)2+Aϵ.(7)F(u^*)-F(\hat{u})\le\varphi(m^\mathsf{T}u^*-a)^2+A\epsilon. \tag*{(7)}

This follows by comparing Eq. (6) at u∗u^* and u^\hat{u}. For convex C\mathcal{C}, gg and Σ≻0\Sigma\succ0, divide by φλmin⁡(Σ)\varphi\lambda_{\min}(\Sigma) to bound squared allocation distance. The result includes deterministic borrowing and shorting charges, provided all costs are scaled. Choosing a=mTpa=m^\mathsf{T}p gives one existing MVPI update without supplying u∗u^*; it is approximate rather than an exact calibration certificate.

Certifying a learned equilibrium

A decision-relevant critic error

Assume finite second moments, Σt(z)≻0\Sigma_t(z) \succ0, and that future innovations are conditionally independent of RtR_t given Zt+1Z_{t+1}. Let PtP_t denote the exogenous transition matrix. For a candidate policy uu, the expected remaining terminal excess gain satisfies

aT=0,at=Htmt⊤ut+Ptat+1.(8)a_T = 0,\qquad a_t = H_t m_t^\top u_t + P_t a_{t+1}. \tag*{(8)}

A one-step deviation to vv changes the terminal criterion through

Jt(v)=Htmt⊤v−φHt2v⊤Σtv−2φHtCov⁡t(Rt,at+1(Z′))⊤v+const.(9)J_t(v) = H_t m_t^\top v - \varphi H_t^2 v^\top\Sigma_t v - 2\varphi H_t \operatorname{Cov}_t(R_t,a_{t+1}(Z'))^\top v + \mathrm{const}. \tag*{(9)}

The conditional independence and wealth-independent continuation assumptions justify the covariance reduction. They do not hold for arbitrary latent-state trading policies.

For any critic ct(z)c_t(z), define residuals and a critic-based action gradient by

et=ct−Htmt⊤ut−Ptct+1,(10)e_t = c_t - H_t m_t^\top u_t - P_t c_{t+1}, \tag*{(10)}
gt=Htmt−2φHtCov⁡t(Rt,ct+1(Z′))−2φHt2Σtut.(11)g_t = H_t m_t - 2\varphi H_t \operatorname{Cov}_t(R_t,c_{t+1}(Z')) - 2\varphi H_t^2\Sigma_t u_t. \tag*{(11)}

Use span⁡(f)=max⁡zf(z)−min⁡zf(z)\operatorname{span}(f)=\max_z f(z)-\min_z f(z) and the Dobrushin coefficient

ρt=12max⁡z,w∑j∣Pt(z,j)−Pt(w,j)∣.\rho_t = \frac{1}{2}\max_{z,w}\sum_j \lvert P_t(z,j)-P_t(w,j)\rvert.

Set

ST=span⁡(cT),St=span⁡(et)+ρtSt+1.(12)S_T = \operatorname{span}(c_T),\qquad S_t = \operatorname{span}(e_t)+\rho_t S_{t+1}. \tag*{(12)}

No equilibrium policy appears in these quantities.

Theorem 1. Let Et(z)=sup⁡v[Jt(v)−Jt(ut(z))]E_t(z)=\sup_v[J_t(v)-J_t(u_t(z))]. Under the preceding assumptions, with unrestricted deviations,

Et(z)≤∥gt(z)∥Σt(z)−12φHt+φ2St+1.(13)\sqrt{E_t(z)} \le\frac{\lVert g_t(z)\rVert_{\Sigma_t(z)^{-1}}}{2\sqrt{\varphi}H_t}+\frac{\sqrt{\varphi}}{2}S_{t+1}. \tag*{(13)}

For a feasible policy on a fixed nonempty closed convex action set, replace the first term by Etc(z)\sqrt{E_t^c(z)}, the maximal feasible improvement in Eq. (9) with at+1a_{t+1} replaced by ct+1c_{t+1}, assuming its maximum is attained. The same bound holds.

Proof. Write δ=c−a\delta=c-a. Equations (8)–(10) give δt=et+Ptδt+1\delta_t=e_t+P_t\delta_{t+1}. Stochastic-kernel span contraction implies span⁡(δt)≤St\operatorname{span}(\delta_t)\le S_t. The true action gradient is dt=gt+2φHtCov⁡t(Rt,δt+1)d_t=g_t+2\varphi H_t\operatorname{Cov}_t(R_t,\delta_{t+1}). Completing the square yields Et=∥dt∥Σt−12/(4φHt2)E_t=\lVert d_t\rVert_{\Sigma_t^{-1}}^2/(4\varphi H_t^2). For any scalar DD, linear-projection residual variance is nonnegative, so

∥Cov⁡(R,D)∥Σ−12≤Var⁡(D)≤span⁡(D)2/4.\lVert\operatorname{Cov}(R,D)\rVert_{\Sigma^{-1}}^2\le\operatorname{Var}(D)\le\operatorname{span}(D)^2/4.

The triangle inequality proves Eq. (13).

For constraints, write J=Jc+b⊤v+constJ=J_c+b^\top v+\mathrm{const}, b=2φHCov⁡(R,δ)b=2\varphi H\operatorname{Cov}(R,\delta) and k=φH2k=\varphi H^2. If vcv_c maximises JcJ_c, strong concavity and its variational inequality imply Jc(v)−Jc(vc)≤−k∥v−vc∥Σ2J_c(v)-J_c(v_c)\le-k\lVert v-v_c\rVert_\Sigma^2 and ∥vc−u∥Σ≤Ec/k\lVert v_c-u\rVert_\Sigma\le\sqrt{E^c/k}. Thus

E≤Ec+∥b∥Σ−1Ec/k+∥b∥Σ−124k=(Ec+∥b∥Σ−12k)2.E\le E^c+\lVert b\rVert_{\Sigma^{-1}}\sqrt{E^c/k}+\frac{\lVert b\rVert_{\Sigma^{-1}}^2}{4k} =\left(\sqrt{E^c}+\frac{\lVert b\rVert_{\Sigma^{-1}}}{2\sqrt{k}}\right)^2.

Apply the preceding covariance bound. □

The maximum certified Et(z)E_t(z) bounds a one-step equilibrium deviation, not global-policy regret. Adding arbitrary time-dependent state-constant offsets to cc leaves the certificate unchanged. If ρt≤ρ<1\rho_t\le\rho<1 and residual spans are at most η\eta, the critic contribution depends on a geometric sum, rather than an unweighted horizon sum. This uses established seminorm geometry [3]. The vector result has no explicit dimension factor in the critic term; reliable covariance estimation remains dimension-dependent.

Reward error, actor error and sampled audits

If a learner follows a different action gradient hth_t, split gt=ht+(gt−ht)g_t=h_t+(g_t-h_t) in Eq. (13). This gives nonnegative actor, reward and critic terms, respectively. It is an upper bound, not an additive decomposition of realised returns. A small neural parameter gradient is insufficient: a saturated actor may have a small parameter derivative despite a large action gradient. At a constraint boundary, use the feasible improvement in Theorem 1 rather than an unconstrained gradient norm.

The exact-model certificate uses conditional moments. For scalar bounded returns, a separate independent-data construction estimates its gradient, variance and Bellman residuals using confidence bounds (Section 8). It conditions on a policy and critic fixed before the audit and does not use the equilibrium policy. If the variance lower bound is nonpositive, it returns an uninformative infinite bound. These conditions rule out silently applying the guarantee to reused training data or a dependent historical time series.

Learning and evaluation protocol

Continuation-aware sample learning

Two independent conditional observations (R1,Z1′)(R_1,Z_1'), (R2,Z2′)(R_2,Z_2') yield the reward

Lc(v)=H(R1+R2)⊤v2−φ2[H(R1−R2)⊤v+c(Z1′)−c(Z2′)]2.(14)L_c(v)=\frac{H(R_1+R_2)^\top v}{2}-\frac{\varphi}{2}\left[H(R_1-R_2)^\top v+c(Z_1')-c(Z_2')\right]^2. \tag*{(14)}

Its expected action gradient equals Eq. (11). A backward mean critic learns from targets HR⊤u+c(Z′)HR^\top u+c(Z'). This replaces oracle continuation by sampled policy evaluation, using established backward equilibrium-learning principles [22]. Paired covariance estimation itself is standard.

Learner pseudocode (shared neural variant). Given conditional samples, φ\varphi, TT and a fixed update budget:

  1. Initialise actor uθu_\theta, critic cψc_\psi and target cˉ\bar c, with zero terminal critics; freeze sample-based curvature weights.

  2. Draw minibatches in each (t,z)(t,z) cell. Regress cψc_\psi onto cell means of detached HR⊤uθ+cˉ(Z′)HR^\top u_\theta+\bar c(Z').

  3. Pair observations; with critic fixed, ascend curvature-weighted mean Lc(uθ)L_c(u_\theta), Eq. (14), in θ\theta.

  4. Update ψˉ←.98ψˉ+.02ψ\bar\psi\leftarrow.98\bar\psi+.02\psi; repeat. The centred variant centres critic outputs and TD targets across regimes at each time. The myopic control zeros step 3’s continuation difference.

  5. Freeze policy and critic; audit independently (§ 9). Equilibrium is used only for controlled evaluation.

Our controlled market has two observed regimes, transition probabilities (p,1−p)(p,1-p) and (1−p,p)(1-p,p), conditional edge means

(.09−.05.04−.02),\begin{pmatrix} .09 & -.05\\ .04 & -.02 \end{pmatrix},

and independent innovations ±.12\pm.12 with equal probabilities. We set s=1.02s=1.02, φ=1\varphi=1, X0=1X_0=1, p∈{.5,.8,.95}p\in\{.5,.8,.95\} and T∈{4,12,24}T\in\{4,12,24\}. Even p=.5p=.5 is not an iid-return experiment: returns remain jointly related to successor regimes. Exact moments and equilibrium are restricted to evaluation.

The tabular study crosses 128/512/2,048/8,192 samples per time/state with twenty seeds and three methods: myopic paired variance, continuation-aware actor updates and exact sampled-objective optimisation (2,160 records). Forty actor steps use one-quarter estimated inverse curvature; zero variance is rejected.

The neural study uses 2,048 conditional samples per time/state, shared actor and critic networks with two 32-unit tanh layers, and inputs consisting of normalised time and regime one-hot encoding. Adam uses learning rate .003, target-critic averaging .02 and batch size 64 per time/state. We retain checkpoints at 250, 1,000 and 3,000 updates. The three variants use myopic reward, continuation-aware reward with an ordinary mean critic, or continuation-aware reward with state-centred critic outputs and TD targets. Hyperparameters were fixed before the five-seed pilot and retained when extending to twenty seeds. All methods share samples and initialisations. Actor losses use the same frozen, sample-estimated inverse-curvature scaling. Actions do not affect the market, and all sampled returns are observed; this generative-data setting is stronger than a single historical trajectory.

A standard policy-optimisation comparison

We additionally use clipped PPO [27], Eq. (7), via Stable-Baselines3 2.9.0. The two arms differ only in whether Eq. (14) includes continuation covariance. Both use sample-only backward evaluation of the current deterministic policy, refreshed between rollouts; no exact equilibrium enters training. This is a structured PPO hybrid, not a reproduction of a published financial agent or a claim that PPO itself solves time inconsistency.

The protocol fixes T=12T = 12, p=.8/.95p = .8/.95, seeds 0–9, 32,768 environment steps, actions in [−3,3][-3, 3], and the same 2,048-sample conditional datasets. Each environment step draws a pair; the first successor advances the episode. Eight environments supply 512 observations per rollout, with ten epochs, batch size 128, learning rate 3×10−43 \times10^{-4}, clipping threshold .2, GAE parameter .95, discount 1, entropy coefficient 0, and separate 64-by-64 tanh actor/value networks. We evaluate the clipped deterministic deployment policy. Budgets are matched within PPO, not between PPO and direct-gradient learning.

Historical allocation and common metrics

Kenneth French’s value-weighted industry returns and factors [10], archived August 2026 vintage, provide 432 evaluation months from January 1990 to December 2025. The lagged window is 120 months; means are unshrunk excess-return means, φ=5\varphi= 5 and L1-cost coefficient λ=.001\lambda= .001. The data are revised research portfolios, not point-in-time investable funds. This experiment audits Eq. (4); it is not a backtest of the new regime-based RL learners.

Seven covariance models (sample, fixed 10% diagonal shrinkage, Ledoit–Wolf identity, constant correlation and nonlinear shrinkage, market factor and three Fama–French factors) are crossed with 10/30/49 industries. Within each model, rewards share target pre-trade holdings, moments and constraints. Cross-model comparisons use the diagonal-model target’s holdings. Own-state paths determine realised performance. These controls separate covariance changes, reward changes and portfolio drift.

MethodDDE\mathcal{E}Bound
A. Shared neural actor–critic
Myopic reward0.61020.003200.00642
Continuation, ordinary0.25070.000610.00223
Continuation, centred (ours)0.24930.000590.00163
B. Clipped PPO; separate matched comparison
PPO, myopic1.72800.023490.03279
PPO, continuation (ours)1.15250.011800.01723

Table 1. Controlled RL: A averages nine markets and twenty seeds (3,000 updates); B averages two markets and ten seeds (32,768 PPO steps). DD: policy distance; E\mathcal{E}: maximum feasible deviation gain. Compute is matched within panels only. Bold: continuation-aware method.

Controlled RL metrics are D=(∑tEuteq∥ut−uteq∥2)1/2D = \left(\sum_t \mathbb{E}_{u_t^{\mathrm{eq}}}\lVert u_t - u_t^{\mathrm{eq}}\rVert^2\right)^{1/2} under exact reference occupancy, starting in regime zero, and maximum one-step deviation gain. Initial terminal MV is secondary: equilibrium need not maximise the initial precommitment objective. Historical metrics are allocation distance, fitted regret, net excess-return MV, excess Sharpe and turnover. Paired neural uncertainty resamples seed-level averages across the nine fixed markets; historical uncertainty uses shared circular twelve-month blocks. Neither treats related market configurations as independent new datasets.

Results

Learning error and residual certification

The tabular continuation-aware learner’s average distance decreases from 1.436 at 128 samples to 0.1649 at 8,192, while myopic distance decreases from 1.497 to 0.5871. Corresponding maximum deviation gains at 8,192 are 0.000273 and 0.003319. More data reduces sampling error but does not remove an omitted continuation hedge. The maximum difference in distance between forty actor steps and exact optimisation of the same sampled objective is 5.24×10−55.24 \times10^{-5}.

Table 1 and Figure 1 report the neural comparison. At the fixed final budget, the centred continuation-aware variant reduces mean distance by 59.2% and deviation gain by 81.4% relative to the matched myopic network. The paired distance difference is −0.361 (95% seed-bootstrap interval [−0.390, −0.329]). Its additional distance improvement over the ordinary continuation critic is 0.57%; most of the improvement comes from the continuation term. The centring effect on distance is small: −0.00143, with paired interval [−0.00253, −0.00030]. Centring reduces the mean certificate by 27.1%; it has a larger effect on auditing than on the deployed policy. We report all seeds and do not select checkpoints using oracle quality.

With PPO, the continuation correction reduces mean distance by 33.3% and maximum deviation gain by 49.8%. The paired distance difference is −0.575 (95% seed-bootstrap interval [−0.739, −0.404]).

The PPO result supports a reward intervention within a fixed optimiser; it does not establish algorithmic superiority over the recent equilibrium-RL frameworks in § 2. Their continuous-time problems and model access differ from this benchmark.

A false-convergence construction has zero critic-based action gradients but true deviation gain 0.10263, bounded by 0.89674. State-constant critic offsets change bounds by less than 1.7×10−151.7 \times10^{-15} across twenty cases. Independent terminal-moment tests check random policies and active constraints; vector projection checks do not constitute high-dimensional neural experiments.

Auditing with independent samples

Twenty fixed tabular policies in the T=12T = 12, p=.8p = .8 market receive 32,768 independent audit observations per time/state, or 786,432 each. Figure 2 compares bounds for each policy. Mean actual maximum deviation is 0.000902 and the exact-model bound is 0.001257. A Hoeffding-only observable bound averages 0.036141. Empirical Bernstein bounds with confidence-bounded transition mixing reduce this to 0.004911. Every policy is covered by each construction. Coverage in these twenty cases is a numerical check, not the source of the confidence guarantee.

Figure 2. Independent-data audits of twenty fixed policies, each using 786,432 observations. Refinement tightens Hoeffding bounds. Coverage follows from the stated assumptions, not the plotted outcomes.

The Bernstein and mixing refinements were introduced after inspecting the Hoeffding bounds, so we report them as a post hoc extension. Each construction has its own 95% simultaneous-over-states guarantee for a fixed policy; no joint 95% guarantee across all policies or constructions is claimed. Large audit requirements and abstention when variance cannot be certified are substantive properties of the method.

Financial inputs and implemented preferences

Table 2 compares published reward formulations and covariance models on historical data. Its cross-estimator distances are changes in fitted models, not pure reward errors. The restricted Habibnia–Soltanzadeh comparator uses long-only unit-capital assetwise PnL, penalty 25 and cost multiplier 25/5; SAC, shorting and lending are not replicated. Its objective depends on marginal downside payoffs and can prefer allocations different from mean–variance diversification.

MethodSource / roleAllocation distanceReference gap (bp)Excess SharpeRealised MV (bp)
Fitted MV referenceMarkowitz [24]0.000000.00000.570−8.497
Minimum varianceMarkowitz [24]0.238439.02520.6455.274
MV, LW identityLedoit–Wolf (2004)0.005380.00550.572−8.442
MV, LW correlationLedoit–Wolf (2004)0.021960.11580.547−11.917
MV, market factorFan et al. [9]0.057111.04870.585−4.738
MV, three factorsFan et al. [9]0.048810.64070.555−8.532
Equal weightDeMiguel et al. [7]0.4349546.26260.622−11.893
MV, LW nonlinearLedoit–Wolf [21]0.011190.02090.559−9.578
Raw second momentKolm–Ritter (2019)0.008430.01480.573−7.182
MVPI adaptationZhang et al. [29]0.000000.00000.570−8.497
One MVPI stepZhang et al. [29]0.000140.00000.570−8.442
Assetwise PnL, restrictedHabibnia–Soltanzadeh (2024)0.7964540.25540.449−23.619
Log minus varianceJiang et al. [16]0.012860.02870.572−7.123
MV, sample covarianceSample control (2026)0.014490.03560.555−10.114
Log utilityLog-utility control (2026)0.5484839.78550.344−133.295
Unconstrained scalarAnalytical control (2026)0.005400.00430.568−8.882
Risk-only calibrationCalibration ablation (2026)0.005100.00330.571−8.365
Joint calibration (ours)This study (2026)0.000000.00000.570−8.497

Table 2. Ten industries, 1990–2025. Distances/gaps use common states and the fixed-diagonal reference; covariance rows change the model. Realised metrics use own-state paths. Sources are chronological; 2026 denotes implemented controls. Published formulations are adapted, not full agent replications. Bold identifies our construction, not investment superiority.

Across all twenty-one covariance/universe cases, joint calibration has mean distance below 1.60×10−71.60 \times10^{-7} from its own fitted reference. Raw distances range 0.00827–0.02249; transferring the unconstrained scalar formula worsens distance and fitted regret in all fourteen larger-universe cases. One existing MVPI update initialised at holdings reduces mean distance by 97.8–98.6%, without supplying the fitted optimum. These are calibration results, not a new MVPI algorithm.

Dimension increases N/LN/L from 0.083 to 0.250 and 0.408, with 65/495/1,274 distinct mean/covariance entries. The classifications also change the opportunity set, so this is not a causal experiment in dimension alone. DeMiguel et al.’s [7] 3,000/6,000-month examples are model-dependent illustrations, not universal requirements. Simple controls and shrinkage remain essential.

Sources: Markowitz [24]; Ledoit–Wolf [19, 20]; Fan et al. [9]; DeMiguel et al. [7]; Ledoit–Wolf [21]; Kolm–Ritter [17]; Zhang et al. [29]; Habibnia–Soltanzadeh [14]; Jiang et al. [16].

Minimum variance has the highest realised MV among quadratic comparators in all twenty-one cases, while joint calibration remains below raw. All eighteen cross-covariance target comparisons have paired 95% intervals containing zero, including simultaneous path-bootstrap bands. Improved alignment therefore does not establish improved historical investment performance. In 2,160 known-covariance fitted problems using the six estimators other than nonlinear shrinkage, factor structure helps when its residual assumptions hold; misspecification can worsen population regret while leaving exact reward calibration intact. A separate fifty-state borrowing/shorting test gives joint distance 1.51×10−71.51 \times10^{-7}, versus 0.00997 when financing charges are not scaled.

Financial interpretation and validation

A financial explanation should distinguish changed opportunities (covariance), changed preferences (reward), and implementation error (actor or critic). In the 49-industry market-factor case, changing covariance moves the target by 0.03992; within-model raw and joint reward discrepancies are 0.02174 and 1.36×10−71.36 \times10^{-7}. Combining them into one error score would obscure which intervention changed the decision.

Nonlinear shrinkage strengthens the estimation control. Ledoit and Wolf [21], Table 8, caution against transferring unconstrained performance rankings to long-only portfolios. With nonlinear shrinkage, joint allocation distances are 9.40×10−99.40 \times10^{-9}, 8.67×10−88.67 \times10^{-8} and 1.60×10−71.60 \times10^{-7} for 10/30/49 industries. Its 49-industry target improves realised MV over fixed diagonal shrinkage by 2.54 monthly basis points, but the paired 95% interval [−0.15, 5.20] includes zero.

The learning audit explains whether changing an action could improve the stated objective. State-constant critic errors are harmless; state-dependent errors can conceal non-equilibrium decisions. Feasible deviations respect boundary optima. This is decision-level validation, not attribution of hidden neural representations.

A worked validation decision. Let a mandate tolerate at most τ=0.001\tau= 0.001 in conditional terminal mean–variance improvement from any one-action change: ten basis points in the experiment’s normalised utility units, not ten basis points of realised return. Require a 95% simultaneous-over-time/state upper bound B≤τB \leq\tau. For the first saved audit policy (seed 0), 786,432 fresh observations give B=0.004992B = 0.004992 using Bernstein bounds and estimated mixing. The decision is inconclusive: the bound exceeds the tolerance, although the simulator’s true maximum gain, 0.000506, is below it. A separate mandate allowing τ=0.01\tau= 0.01 would pass the same bound; Hoeffding alone (B=0.041214B = 0.041214) would certify neither. These are illustrative tolerances applied retrospectively, not preregistered acceptance results. Operational use must fix the tolerance, policy and audit construction before inspecting audit outcomes; an infinite bound also requires abstention.

Scope of the evidence. The sequential experiments establish the correction’s effect in a small observed-state market with conditional sampling. The multiasset historical study instead establishes alignment with a fitted one-period objective; it does not validate the sequential learner or its independent-data guarantee on market histories. A realistic sequential multiasset test remains open. Unknown regimes, market impact, wealth-dependent policies and

@page 6/8 starts_mid=0 ends_mid=1

MethodSource / roleAllocation distanceReference gap (bp)Excess SharpeRealised MV (bp)
Fitted MV referenceMarkowitz [24]0.000000.00000.570−8.497
Minimum varianceMarkowitz [24]0.238439.02520.6455.274
MV, LW identityLedoit–Wolf (2004)0.005380.00550.572−8.442
MV, LW correlationLedoit–Wolf (2004)0.021960.11580.547−11.917
MV, market factorFan et al. [9]0.057111.04870.585−4.738
MV, three factorsFan et al. [9]0.048810.64070.555−8.532
Equal weightDeMiguel et al. [7]0.4349546.26260.622−11.893
MV, LW nonlinearLedoit–Wolf [21]0.011190.02090.559−9.578
Raw second momentKolm–Ritter (2019)0.008430.01480.573−7.182
MVPI adaptationZhang et al. [29]0.000000.00000.570−8.497
One MVPI stepZhang et al. [29]0.000140.00000.570−8.442
Assetwise PnL, restrictedHabibnia–Soltanzadeh (2024)0.7964540.25540.449−23.619
Log minus varianceJiang et al. [16]0.012860.02870.572−7.123
MV, sample covarianceSample control (2026)0.014490.03560.555−10.114
Log utilityLog-utility control (2026)0.5484839.78550.344−133.295
Unconstrained scalarAnalytical control (2026)0.005400.00430.568−8.882
Risk-only calibrationCalibration ablation (2026)0.005100.00330.571−8.365
Joint calibration (ours)This study (2026)0.000000.00000.570−8.497

Table 2 (PDF p. 6). Ten industries, 1990–2025. Distances/gaps use common states and the fixed-diagonal reference; covariance rows change the model. Realised metrics use own-state paths. Sources are chronological; 2026 denotes implemented controls. Published formulations are adapted, not full agent replications. Bold identifies our construction, not investment superiority.

Sources: Markowitz [24]; Ledoit–Wolf [19, 20]; Fan et al. [9]; DeMiguel et al. [7]; Ledoit–Wolf [21]; Kolm–Ritter [17]; Zhang et al. [29]; Habibnia–Soltanzadeh [14]; Jiang et al. [16].

Minimum variance has the highest realised MV among quadratic comparators in all twenty-one cases, while joint calibration remains below raw. All eighteen cross-covariance target comparisons have paired 95% intervals containing zero, including simultaneous path-bootstrap bands. Improved alignment therefore does not establish improved historical investment performance. In 2,160 known-covariance fitted problems using the six estimators other than nonlinear shrinkage, factor structure helps when its residual assumptions hold; misspecification can worsen population regret while leaving exact reward calibration intact. A separate fifty-state borrowing/shorting test gives joint distance 1.51×10−71.51 \times10^{-7}, versus 0.00997 when financing charges are not scaled.

Financial interpretation and validation

A financial explanation should distinguish changed opportunities (covariance), changed preferences (reward), and implementation error (actor or critic). In the 49-industry market-factor case, changing covariance moves the target by 0.03992; within-model raw and joint reward discrepancies are 0.02174 and 1.36×10−71.36 \times10^{-7}. Combining them into one error score would obscure which intervention changed the decision.

Nonlinear shrinkage strengthens the estimation control. Ledoit and Wolf [21], Table 8, caution against transferring unconstrained performance rankings to long-only portfolios. With nonlinear shrinkage, joint allocation distances are 9.40×10−99.40 \times10^{-9}, 8.67×10−88.67 \times10^{-8} and 1.60×10−71.60 \times10^{-7} for 10/30/49 industries. Its 49-industry target improves realised MV over fixed diagonal shrinkage by 2.54 monthly basis points, but the paired 95% interval [−0.15, 5.20] includes zero.

The learning audit explains whether changing an action could improve the stated objective. State-constant critic errors are harmless; state-dependent errors can conceal non-equilibrium decisions. Feasible deviations respect boundary optima. This is decision-level validation, not attribution of hidden neural representations.

A worked validation decision. Let a mandate tolerate at most τ=0.001\tau= 0.001 in conditional terminal mean–variance improvement from any one-action change: ten basis points in the experiment’s normalised utility units, not ten basis points of realised return. Require a 95% simultaneous-over-time/state upper bound B≤τB \leq\tau. For the first saved audit policy (seed 0), 786,432 fresh observations give B=0.004992B = 0.004992 using Bernstein bounds and estimated mixing. The decision is inconclusive: the bound exceeds the tolerance, although the simulator’s true maximum gain, 0.000506, is below it. A separate mandate allowing τ=0.01\tau= 0.01 would pass the same bound; Hoeffding alone (B=0.041214B = 0.041214) would certify neither. These are illustrative tolerances applied retrospectively, not preregistered acceptance results. Operational use must fix the tolerance, policy and audit construction before inspecting audit outcomes; an infinite bound also requires abstention.

Scope of the evidence. The sequential experiments establish the correction’s effect in a small observed-state market with conditional sampling. The multiasset historical study instead establishes alignment with a fitted one-period objective; it does not validate the sequential learner or its independent-data guarantee on market histories. A realistic sequential multiasset test remains open. Unknown regimes, market impact, wealth-dependent policies and transaction-cost states require additional continuation analysis, while dependent audit data require new statistical guarantees.

Figure 3. 49-industry covariance and reward interventions. Joint calibration preserves fitted preferences, without identifying true covariance or establishing superior own-state performance.

Kolm and Ritter [18], Exhibit 2 and pp. 90–91, distinguish reward specification, learning and decision quality. Our audit connects these stages within model-risk review, taking the mandate as given. Direct optimisation remains appropriate for tractable one-period allocation; sequential learning additionally needs an account of remaining decision error.

Independent-data certificate

Fix a scalar policy uu and critic cc independently of audit data, and assume known support ∣R∣≤B|R| \le B. At every (t,z)(t,z) obtain N=2nN = 2n independent conditional observations. Pair them and differentiate Eq. (14); each gradient sample has absolute bound

Mg=HB+2φHBspan⁡(ct+1)+4φH2B2∣u∣.M_g = HB + 2\varphi HB\operatorname{span}(c_{t+1}) + 4\varphi H^2B^2|u|.

The pair variable (R1−R2)2/2(R_1-R_2)^2/2 estimates conditional variance and lies in [0,2B2][0,2B^2]. Bellman targets have range at most 2HB∣u∣+span⁡(ct+1)2HB|u|+\operatorname{span}(c_{t+1}). Hoeffding radii with failure probability q=α/(3TK)q=\alpha/(3TK) give a gradient upper norm, variance lower bound and residual intervals [lz,hz][l_z,h_z]. Then span⁡(et)≤max⁡zhz−min⁡zlz\operatorname{span}(e_t)\le\max_z h_z-\min_z l_z; use ρt≤1\rho_t\le1 in Eq. (12). Substitute these bounds in Eq. (13). The result covers all time/state cells with probability at least 1−α1-\alpha, conditional on training.

For a variable of known range width WW and sample size nn, Maurer–Pontil [25], Theorem 4, gives the two-sided radius

2V^log⁡(4/q)n+7Wlog⁡(4/q)3(n−1).\sqrt{\frac{2\widehat{V}\log(4/q)}{n}+\frac{7W\log(4/q)}{3(n-1)}}.

where V^\widehat{V} uses denominator n−1n-1. Confidence intervals for each transition entry additionally give ρt≤min⁡(1,ρ^t+Kη)\rho_t\le\min(1,\widehat{\rho}_t+K\eta), where η\eta bounds each entry error. Include TK2TK^2 additional events in the union bound. The construction returns infinity if any variance lower bound is nonpositive. Adaptive policy selection on the audit set, dependent observations or unknown support require additional analysis; none is assumed here. The scalar implementation bounds unrestricted deviations, which also upper-bound constrained ones.

Raw-reward discrepancy

For constant independent moments with m≠0m\ne0, the quadratic Bellman solution has ut=αtx+βtu_t=\alpha_t x+\beta_t, with terminal pT=qT=0p_T=q_T=0 and gt=κ(s−1)−spt+1g_t=\kappa(s-1)-s p_{t+1}.

αt=−dgt/(κ−pt+1),βt=d(1+qt+1)/[2(κ−pt+1)],pt=[κpt+1−(1−ν)gt2]/(κ−pt+1),1+qt=(1+qt+1)[s−νgt/(κ−pt+1)].\begin{aligned} \alpha_t&=-d g_t/(\kappa-p_{t+1}),\\ \beta_t&=d(1+q_{t+1})/[2(\kappa-p_{t+1})],\\ p_t&=[\kappa p_{t+1}-(1-\nu)g_t^2]/(\kappa-p_{t+1}),\\ 1+q_t&=(1+q_{t+1})[s-\nu g_t/(\kappa-p_{t+1})]. \end{aligned}

Induction gives pt≤0p_t\le0, hence global concave action maximisation. Put e=s−1e=s-1 and h=(2κ)−1h=(2\kappa)^{-1}. Because pt/κp_t/\kappa, qtq_t are independent of κ\kappa, the Hessian in hh and D2D^2 is exactly quadratic in hh, analytic near e=0e=0. At zero it is T∥d∥2(h−b)2T\lVert d\rVert^2(h-b)^2, so its positive minimiser is analytic nearby.

Writing κ=φ(1−ν)(1+we)\kappa=\varphi(1-\nu)(1+we), the leading action error is d[−Xt+b((T−t−1)(2−ν)−w)]d[-X_t+b((T-t-1)(2-\nu)-w)]. At e=0e=0, mean wealth is x0+tvbx_0+tvb and variance tb2ν(1−ν)tb^2\nu(1-\nu). Minimisation gives w=(T−1)(1−ν)−x0/bw=(T-1)(1-\nu)-x_0/b; summing centred mean squares b2(T−1−2t)2b^2(T-1-2t)^2 and wealth variances yields Eq. (3). This is a fixed-horizon expansion, not a uniform joint limit. For s≠1s\ne1, exact matching is impossible: matching initially creates positive subsequent wealth variance, whereas the final raw policy has nonzero wealth slope −d(s−1)-d(s-1) and equilibrium is constant.

Reproduction and numerical checks

Conditional-sample generators use seeds 0–19, also controlling neural weights and minibatches; PPO uses seeds 0–9. Both PPO arms share data access and refresh frequency; continuation is fitted from training outcomes, not the analytic reference. The pilot, twenty-seed extension and PPO experiment are archived separately. Bootstrap intervals use 10,000 resamples of seed averages; all final-budget runs are retained.

Historical identity shrinkage uses centred ZZ, S0=ZTZ/LS_0=Z^\mathsf{T}Z/L and target tr⁡(S0)/N\operatorname{tr}(S_0)/N, with intensity from the empirical fourth moment; final covariance is converted to denominator L−1L-1. Constant-correlation shrinkage uses the plug-in formulas of Ledoit and Wolf [19]. Factor models use intercept OLS, market/SMB/HML and diagonal residual variances with denominator L−K−1L-K-1. This convention adapts Fan et al. [9]. Nonlinear shrinkage uses the covariance branch of QuEST v027 through a pinned Python port, with trace-constrained SLSQP fitting, tolerance 10−910^{-9}, and effective sample size L−1L-1 after demeaning. No additional rescaling is applied. This implements the estimator, not the original CRSP backtest. Missing and duplicate months are rejected. Holdings drift with gross asset returns; costs are charged separately at the stated first-order rate.

Historical uncertainty uses 1,000 shared circular twelve-month blocks, seed 8128, conditional on realised strategy paths; simultaneous covariance bands use the maximum centred deviation across eighteen comparisons. Quadratic optimisation is checked independently by a first-order linear-programme certificate in original objective units, with accepted gap at most 2×10−72 \times10^{-7}. Reconstruction verifies 67,392 historical decisions, 2,160 known-covariance fitted problems and 144 anchor checks. RL tests independently evaluate terminal moments, gradient/covariance identities, critic offsets, active boundaries and audit support.

Conclusion

The residual certificate connects learning errors to financial decision quality, respecting critic-offset invariance and convex constraints. Controlled actor–critic and PPO comparisons test continuation correction without an oracle target. Historical experiments separately test fitted reward alignment. Together they support validation that distinguishes objective fidelity from investment performance and explicitly abstains when audit evidence is insufficient.

Declarations

Funding and computational resources. This study received no project-specific research funding. No computing resources from Virginia Tech or the authors’ other affiliated universities were used in this study.

Code and data availability. The simulation experiments use data generated by the accompanying code. The historical analysis uses publicly available monthly value-weighted US industry portfolio returns and factors from Kenneth French’s Data Library [10], using the archived August 2026 vintage. Replication code, archived input data, experiment outputs and provenance records are available at https://github.com/DataismLab/Reward-alignment.

References

References

  1. [1]Suleyman Basak and Georgy Chabakauri. 2010. Dynamic Mean-Variance Asset Allocation. The Review of Financial Studies 23, 8 (2010), 2970–3016. doi:10.1093/rfs/hhq028DOI
  2. [2]Tomas Björk, Mariana Khapko, and Agatha Murgoci. 2016. Time Inconsistent Stochastic Control in Continuous Time: Theory and Examples. arXiv:1612.03650arxiv.org/abs/1612.03650
  3. [3]Zaiwei Chen, Sheng Zhang, Zhe Zhang, Shaan Ul Haque, and Siva Theja Maguluri. 2025. A Non-Asymptotic Theory of Seminorm Lyapunov Stability: From Deterministic to Stochastic Iterative Algorithms. arXiv:2502.14208arxiv.org/abs/2502.14208
  4. [4]Lin William Cong, Ke Tang, and Jingyuan Wang. 2026. AlphaPortfolio: Goal-Oriented Investment Management Through Deep Reinforcement Learning. NBER Working Paper 35195. National Bureau of Economic Research. doi:10.3386/w35195DOI
  5. [5]Christoph Cichowsky. 2013. Time-Consistent Mean-Variance Portfolio Selection in Discrete and Continuous Time. Finance and Stochastics 17, 2 (2013), 227–271. doi:10.1007/s00780-012-0189-9
  6. [6]Min Dai, Yuchao Dong, and Yanwei Jia. 2023. Learning Equilibrium Mean-Variance Strategy. Mathematical Finance 33, 4 (2023), 1166–1212. doi:10.1111/mafi.12402
  7. [7]Victor DeMiguel, Lorenzo Garlappi, and Raman Uppal. 2009. Optimal Versus Naive Diversification: How Inefficient is the 1/N Portfolio Strategy? The Review of Financial Studies 22, 5 (2009), 1915–1953. doi:10.1093/rfs/hhm075
  8. [8]Victor DeMiguel, Alberto Martín-Utrera, and Raman Uppal. 2024. A Multifactor Perspective on Volatility-Managed Portfolios. The Journal of Finance 79, 6 (2024), 3859–3891. doi:10.1111/jofi.13395DOI
  9. [9]Jianqing Fan, Yingying Fan, and Jinchi Lv. 2008. High Dimensional Covariance Matrix Estimation Using a Factor Model. Journal of Econometrics 147, 1 (2008), 186–197. doi:10.1016/j.jeconom.2008.09.017
  10. [10]Kenneth R. French. 2026. Data Library: Industry Portfolios and Fama–French Factors. Tuck School of Business, Dartmouth College. Retrieved October 1, 2026 from https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html August 2026 data vintage.
  11. [11]Nicolae Gârleanu and Lasse Heje Pedersen. 2013. Dynamic Trading with Predictable Returns and Transaction Costs. The Journal of Finance 68, 6 (2013), 2309–2340. doi:10.1111/jofi.12080DOI
  12. [12]Maximilian Gollart and Yarema Okhrin. 2025. A Reinforcement Learning Approach to Dynamic Portfolio Optimization. Annals of Operations Research (2025), 1–26. doi:10.1007/s10479-025-06649-x. Published online 25 June 2025.DOI
  13. [13]Xin Guo, Yijie Huang, and Xiang Yu. 2026. Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problems. arXiv:2606.11798arxiv.org/abs/2606.11798
  14. [14]Ali Habibnia and Mahdi Soltanzadeh. 2024. Optimizing Portfolio with Two-Sided Transactions and Lending: A Reinforcement Learning Framework. arXiv:2408.05382arxiv.org/abs/2408.05382
  15. [15]Yifu Jiang, Jose Olmo, and Majed Atwi. 2024. Deep Reinforcement Learning for Portfolio Selection. Global Finance Journal 62 (2024), 101016. doi:10.1016/j.gfj.2024.101016DOI
  16. [16]Yifu Jiang, Jose Olmo, and Majed Atwi. 2025. High-Dimensional Multi-Period Portfolio Allocation Using Deep Reinforcement Learning. International Review of Economics & Finance 98 (2025), 103996. doi:10.1016/j.iref.2025.103996DOI
  17. [17]Petter N. Kolm and Gordon Ritter. 2019. Dynamic Replication and Hedging: A Reinforcement Learning Approach. The Journal of Financial Data Science 1, 1 (2019), 159–171. doi:10.3905/jfds.2019.1.1.159DOI
  18. [18]Petter N. Kolm and Gordon Ritter. 2025. Reinforcement Learning for Asset and Portfolio Management. The Journal of Portfolio Management 52, 2 (2025), 81–95. doi:10.3905/jpm.2025.1.782DOI
  19. [19]Olivier Ledoit and Michael Wolf. 2004. Honey, I Shrunk the Sample Covariance Matrix. The Journal of Portfolio Management 30, 4 (2004), 110–119. doi:10.3905/jpm.2004.110DOI
  20. [20]Olivier Ledoit and Michael Wolf. 2004. A Well-Conditioned Estimator for Large-Dimensional Covariance Matrices. Journal of Multivariate Analysis 88, 2 (2004), 365–411. doi:10.1016/S0047-259X(03)00096-4DOI
  21. [21]Olivier Ledoit and Michael Wolf. 2017. Nonlinear Shrinkage of the Covariance Matrix for Portfolio Selection: Markowitz Meets Goldilocks. The Review of Financial Studies 30, 12 (2017), 4349–4388. doi:10.1093/rfs/hhx052DOI
  22. [22]Nixie Sapphira Lesmana and Chi Seng Pun. 2025. A Subgame Perfect Equilibrium Reinforcement Learning Approach to Time-Inconsistent Problems. SIAM Journal on Financial Mathematics 16, 1 (2025), 68–122. doi:10.1137/23M1594510DOI
  23. [23]Duan Li and Wan-Lung Ng. 2000. Optimal Dynamic Portfolio Selection: Multi-period Mean-Variance Formulation. Mathematical Finance 10, 3 (2000), 387–406. doi:10.1111/1467-9965.00100DOI
  24. [24]Harry Markowitz. 1952. Portfolio Selection. The Journal of Finance 7, 1 (1952), 77–91. doi:10.1111/j.1540-6261.1952.tb01525.xDOI
  25. [25]Andreas Maurer and Massimiliano Pontil. 2009. Empirical Bernstein Bounds and Sample Variance Penalization. In Proceedings of the 22nd Annual Conference on Learning Theory (COLT 2009). Montreal, Canada. https://www.cs.mcgill.ca/~colt2009/papers/012.pdfarxiv.org/abs/0907.3740
  26. [26]Tejaram Sangadi, L. A. Prashanth, and Krishna Jagannathan. 2025. A Finite-Sample Analysis of an Actor-Critic Algorithm for Mean-Variance Optimization in a Discounted MDP. arXiv:2406.07892 Version 2.arxiv.org/abs/2406.07892
  27. [27]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347arxiv.org/abs/1707.06347
  28. [28]Frederik Simon, Sebastian Weibels, and Tom Zimmerman. 2026. Deep Parametric Portfolio Policies. Management Science (2026), 1–19. doi:10.1287/mnsc.2025.00721 Articles in Advance.DOI
  29. [29]Shengtong Zhang, Bo Liu, and Shimon Whiteson. 2021. Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. AAAI Press, 10905–10913. doi:10.1609/aaai.v35i12.17302DOI

Paper details

vlnc:W_gjwpp335
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 2

A view counts once per browser per day, a download each time the PDF is requested.

  • Economics, Econometrics and Finance

Not recorded for this paper

View statement
Looking for related papers…
Contents