WorksEconomics, Econometrics and Finance
Certifying Reward Alignment in Portfolio Reinforcement Learning
Abstract
Financial reinforcement learning often translates an investment objective into an additive risk-adjusted reward. An agent can optimise that reward accurately yet implement a different financial preference. We develop a decision audit for terminal mean-variance equilibrium that separates reward misspecification from learning error. In a finite exogenous-state market, we bound the gain from changing one action using actor, reward and continuation-critic residuals. The certificate ignores state-constant critic offsets, accommodates convex action constraints, and has an independent-sample extension that does not require the equilibrium policy. A continuation-aware actor-critic method estimates the future-policy correction from samples. Across nine controlled markets and twenty seeds, it reduces policy distance by and one-step deviation gain by relative to a matched myopic-reward network. A separate PPO comparison tests the same reward intervention within a standard optimiser. Historical tests cross seven covariance estimators and 10, 30 and 49 US industry portfolios over 432 months: joint reward calibration preserves each fitted mean-variance target, but does not improve realised investment performance. Together, these results provide a way to quantify whether a learned decision implements its stated financial objective, separately from how well that objective performs in a backtest.
CCS Concepts
• Computing methodologies → Reinforcement learning; • Applied computing → Economics.
Keywords
financial reinforcement learning, reward alignment, equilibrium, actor–critic, model validation
Introduction
An investment mandate specifies preferences over return, risk and costs; an RL agent maximises a reward. The translation can change decisions: replacing variance by a second moment penalises squared expected return; ignoring dependence removes a continuation hedge; correcting risk without scaling costs changes turnover preferences. Accurate reward optimisation or a favourable backtest does not establish mandate alignment.
Kolm and Ritter [18], pp. 90–91, place mandate-aligned reward specification before training. We ask whether the reward induces the intended decisions and the agent implements them. Our target is conditional terminal mean–variance equilibrium. When computable, it permits direct policy comparison; otherwise, we bound the improvement from changing one action while retaining the agent’s future policy.
Our residual certificate separates actor, reward and continuation errors. A zero actor gradient can conceal a wrong critic, while large state-constant critic errors can be harmless. Tracking state differences and their propagation distinguishes these cases and supplies an acceptance criterion under explicit assumptions. A continuation-aware learner estimates the correction from samples, without an equilibrium oracle in training.
Controlled actor–critic and PPO comparisons hold optimiser and data access fixed. Historical tests separately vary covariance and rewards: Markowitz [24] distinguishes beliefs from decisions, while DeMiguel et al. [7] show that estimation error can overwhelm diversification gains. Preserving a fitted objective need not improve realised returns.
Contributions. We contribute: (i) a certificate for finite-iterate equilibrium deviations, invariant to state-constant critic offsets and extendable to convex constraints and independent data; (ii) continuation-aware sample learning tested in matched neural and PPO comparisons; and (iii) a historical comparison separating alignment, estimation risk and performance. The new link is from reward and learning residuals to financial decision quality, building on established equilibrium learning, mean auxiliaries and concentration bounds.
Roadmap. Section 3 defines two distinct targets: terminal equilibrium and fitted one-period allocation. Sections 4–5 certify and learn the former. Section 6 tests sequential learning, then historical one-period calibration; Section 7 interprets validation decisions. Sections 8–10 provide technical and reproduction details.
Related work
Financial objectives and estimation. DeMiguel, Garlappi and Uppal [7], Tables 3–5, compare fourteen models with . Proposition 1 and Figure 1 relate estimation requirements to dimension and opportunities, not a minimum asset count. Table 2 includes ten industries plus the market; we also test finer classifications. We implement linear shrinkage [19, 20], observed-factor covariance [9] and QuEST nonlinear shrinkage. Ledoit and Wolf [21], Corollary 1 and Eqs. (14)–(15), correct eigenvalues individually while retaining sample eigenvectors. Their Table 8 motivates holding long-only constraints and costs fixed.
Figure 1. Neural learning across twenty seeds and nine markets. Markets are averaged within seed; bands show descriptive mean standard errors, not simultaneous intervals. The certificate audits policy-relevant errors.
Gârleanu and Pedersen [11], Eq. (4) and Propositions 2–3, derive cost-aware dynamic allocation; DeMiguel et al. [8], Eq. (6) and Tables II–III, study conditional multifactor portfolios with costs. Their dynamic targets differ from our rolling one-period reference. Changing covariance or intentionally changing preferences is not reward misspecification.
Financial RL and reward design. Kolm and Ritter’s earlier hedging work [17], Eqs. (8)–(10), explicitly treats quadratic increment rewards as an approximation. Jiang et al. [16], Eq. (8), combine log return and variance; their 2024 formulation [15], Eq. (4), also considers costs. Habibnia and Soltanzadeh [14], Eq. (8), use asymmetric PnL with SAC, shorting and lending. Table 2 adapts reward formulations to common states; it does not reproduce these papers’ complete agents or backtests.
Objective-aware learning is established. The reward in Simon et al. [28], Eq. (17), is centred mean–variance utility. Gollart and Okhrin [12], Section 4.2.1, compare learned policies with an analytical benchmark. Cong et al. [4], Eqs. (19)–(20), permit differentiable trajectory objectives. Kolm and Ritter [18] survey the area and discuss validation against known policies (p. 87), so benchmark comparison itself is not new.
Equilibrium learning and certification. Precommitment [23] and equilibrium [1] solve different decision problems. Continuation hedges [5], equivalent objectives [2] and equilibrium RL [6] precede our work. Lesmana and Pun [22] develop subgame-perfect equilibrium RL and backward policy iteration; our implementation specialises these principles rather than replicating their algorithm. Guo et al. [13], Algorithms 1–3, already combine auxiliary rewards, deterministic policy gradients and fixed-point critic learning. We do not propose a new general equilibrium solver.
Sangadi et al. [26], Theorem 4.3, provide finite-sample actor–critic stationarity analysis for a discounted return mean–variance objective. Our target is instead a posteriori, statewise one-step deviation gain for a given finite-horizon policy. Seminorm learning and irrelevant critic offsets are also established [3]. We use that geometry to obtain a financial error certificate. The scalar reward-calibration identity below follows the mean auxiliary in MVPI [29]; the empirical Bernstein allowance follows Maurer and Pontil [25]. The contribution lies in the resulting decision audit, not in inventing these ingredients.
Objectives and reward calibration
Terminal and fitted references
Let be a finite exogenous Markov state and let excess returns be jointly distributed with conditional on . Wealth obeys
where is deterministic. The financial criterion at each state is conditional terminal mean minus times terminal variance. We consider deterministic wealth-independent policies and their one-step deviations, holding future policies fixed. This equilibrium criterion differs from maximising initial-date precommitment value.
For independent returns with mean and covariance , equilibrium is . With and , the additive reward
recovers that policy. This specialises prior equivalent-objective constructions [2]. Under dependence, continuation covariance is essential; Eq. (2) alone is insufficient.
For the raw family , even optimal constant scalar tuning leaves a discrepancy. With independent constant moments and , , , and fixed ,
Here uses the raw-reward optimum’s occupancy. Section 9 supplies the derivation. This obstruction concerns one reward family, not all additive rewards.
Constraints and deterministic costs
For historical data we use a separate, fitted one-period target
where is the simplex and at pre-trade holdings . Its optimum is not the true historical market optimum. For and , define
The identity
proves that both objectives have the same maximiser. It is an application of the MVPI auxiliary [29], not a new general algorithm. Correcting risk alone introduces the extra term and can change the cost preference.
For any anchor with , an -optimal for satisfies
This follows by comparing Eq. (6) at and . For convex , and , divide by to bound squared allocation distance. The result includes deterministic borrowing and shorting charges, provided all costs are scaled. Choosing gives one existing MVPI update without supplying ; it is approximate rather than an exact calibration certificate.
Certifying a learned equilibrium
A decision-relevant critic error
Assume finite second moments, , and that future innovations are conditionally independent of given . Let denote the exogenous transition matrix. For a candidate policy , the expected remaining terminal excess gain satisfies
A one-step deviation to changes the terminal criterion through
The conditional independence and wealth-independent continuation assumptions justify the covariance reduction. They do not hold for arbitrary latent-state trading policies.
For any critic , define residuals and a critic-based action gradient by
Use and the Dobrushin coefficient
Set
No equilibrium policy appears in these quantities.
Theorem 1. Let . Under the preceding assumptions, with unrestricted deviations,
For a feasible policy on a fixed nonempty closed convex action set, replace the first term by , the maximal feasible improvement in Eq. (9) with replaced by , assuming its maximum is attained. The same bound holds.
Proof. Write . Equations (8)–(10) give . Stochastic-kernel span contraction implies . The true action gradient is . Completing the square yields . For any scalar , linear-projection residual variance is nonnegative, so
The triangle inequality proves Eq. (13).
For constraints, write , and . If maximises , strong concavity and its variational inequality imply and . Thus
Apply the preceding covariance bound. □
The maximum certified bounds a one-step equilibrium deviation, not global-policy regret. Adding arbitrary time-dependent state-constant offsets to leaves the certificate unchanged. If and residual spans are at most , the critic contribution depends on a geometric sum, rather than an unweighted horizon sum. This uses established seminorm geometry [3]. The vector result has no explicit dimension factor in the critic term; reliable covariance estimation remains dimension-dependent.
Reward error, actor error and sampled audits
If a learner follows a different action gradient , split in Eq. (13). This gives nonnegative actor, reward and critic terms, respectively. It is an upper bound, not an additive decomposition of realised returns. A small neural parameter gradient is insufficient: a saturated actor may have a small parameter derivative despite a large action gradient. At a constraint boundary, use the feasible improvement in Theorem 1 rather than an unconstrained gradient norm.
The exact-model certificate uses conditional moments. For scalar bounded returns, a separate independent-data construction estimates its gradient, variance and Bellman residuals using confidence bounds (Section 8). It conditions on a policy and critic fixed before the audit and does not use the equilibrium policy. If the variance lower bound is nonpositive, it returns an uninformative infinite bound. These conditions rule out silently applying the guarantee to reused training data or a dependent historical time series.
Learning and evaluation protocol
Continuation-aware sample learning
Two independent conditional observations , yield the reward
Its expected action gradient equals Eq. (11). A backward mean critic learns from targets . This replaces oracle continuation by sampled policy evaluation, using established backward equilibrium-learning principles [22]. Paired covariance estimation itself is standard.
Learner pseudocode (shared neural variant). Given conditional samples, , and a fixed update budget:
Initialise actor , critic and target , with zero terminal critics; freeze sample-based curvature weights.
Draw minibatches in each cell. Regress onto cell means of detached .
Pair observations; with critic fixed, ascend curvature-weighted mean , Eq. (14), in .
Update ; repeat. The centred variant centres critic outputs and TD targets across regimes at each time. The myopic control zeros step 3’s continuation difference.
Freeze policy and critic; audit independently (§ 9). Equilibrium is used only for controlled evaluation.
Our controlled market has two observed regimes, transition probabilities and , conditional edge means
and independent innovations with equal probabilities. We set , , , and . Even is not an iid-return experiment: returns remain jointly related to successor regimes. Exact moments and equilibrium are restricted to evaluation.
The tabular study crosses 128/512/2,048/8,192 samples per time/state with twenty seeds and three methods: myopic paired variance, continuation-aware actor updates and exact sampled-objective optimisation (2,160 records). Forty actor steps use one-quarter estimated inverse curvature; zero variance is rejected.
The neural study uses 2,048 conditional samples per time/state, shared actor and critic networks with two 32-unit tanh layers, and inputs consisting of normalised time and regime one-hot encoding. Adam uses learning rate .003, target-critic averaging .02 and batch size 64 per time/state. We retain checkpoints at 250, 1,000 and 3,000 updates. The three variants use myopic reward, continuation-aware reward with an ordinary mean critic, or continuation-aware reward with state-centred critic outputs and TD targets. Hyperparameters were fixed before the five-seed pilot and retained when extending to twenty seeds. All methods share samples and initialisations. Actor losses use the same frozen, sample-estimated inverse-curvature scaling. Actions do not affect the market, and all sampled returns are observed; this generative-data setting is stronger than a single historical trajectory.
A standard policy-optimisation comparison
We additionally use clipped PPO [27], Eq. (7), via Stable-Baselines3 2.9.0. The two arms differ only in whether Eq. (14) includes continuation covariance. Both use sample-only backward evaluation of the current deterministic policy, refreshed between rollouts; no exact equilibrium enters training. This is a structured PPO hybrid, not a reproduction of a published financial agent or a claim that PPO itself solves time inconsistency.
The protocol fixes , , seeds 0–9, 32,768 environment steps, actions in , and the same 2,048-sample conditional datasets. Each environment step draws a pair; the first successor advances the episode. Eight environments supply 512 observations per rollout, with ten epochs, batch size 128, learning rate , clipping threshold .2, GAE parameter .95, discount 1, entropy coefficient 0, and separate 64-by-64 tanh actor/value networks. We evaluate the clipped deterministic deployment policy. Budgets are matched within PPO, not between PPO and direct-gradient learning.
Historical allocation and common metrics
Kenneth French’s value-weighted industry returns and factors [10], archived August 2026 vintage, provide 432 evaluation months from January 1990 to December 2025. The lagged window is 120 months; means are unshrunk excess-return means, and L1-cost coefficient . The data are revised research portfolios, not point-in-time investable funds. This experiment audits Eq. (4); it is not a backtest of the new regime-based RL learners.
Seven covariance models (sample, fixed 10% diagonal shrinkage, Ledoit–Wolf identity, constant correlation and nonlinear shrinkage, market factor and three Fama–French factors) are crossed with 10/30/49 industries. Within each model, rewards share target pre-trade holdings, moments and constraints. Cross-model comparisons use the diagonal-model target’s holdings. Own-state paths determine realised performance. These controls separate covariance changes, reward changes and portfolio drift.
| Method | Bound | ||
| A. Shared neural actor–critic | |||
| Myopic reward | 0.6102 | 0.00320 | 0.00642 |
| Continuation, ordinary | 0.2507 | 0.00061 | 0.00223 |
| Continuation, centred (ours) | 0.2493 | 0.00059 | 0.00163 |
| B. Clipped PPO; separate matched comparison | |||
| PPO, myopic | 1.7280 | 0.02349 | 0.03279 |
| PPO, continuation (ours) | 1.1525 | 0.01180 | 0.01723 |
Table 1. Controlled RL: A averages nine markets and twenty seeds (3,000 updates); B averages two markets and ten seeds (32,768 PPO steps). : policy distance; : maximum feasible deviation gain. Compute is matched within panels only. Bold: continuation-aware method.
Controlled RL metrics are under exact reference occupancy, starting in regime zero, and maximum one-step deviation gain. Initial terminal MV is secondary: equilibrium need not maximise the initial precommitment objective. Historical metrics are allocation distance, fitted regret, net excess-return MV, excess Sharpe and turnover. Paired neural uncertainty resamples seed-level averages across the nine fixed markets; historical uncertainty uses shared circular twelve-month blocks. Neither treats related market configurations as independent new datasets.
Results
Learning error and residual certification
The tabular continuation-aware learner’s average distance decreases from 1.436 at 128 samples to 0.1649 at 8,192, while myopic distance decreases from 1.497 to 0.5871. Corresponding maximum deviation gains at 8,192 are 0.000273 and 0.003319. More data reduces sampling error but does not remove an omitted continuation hedge. The maximum difference in distance between forty actor steps and exact optimisation of the same sampled objective is .
Table 1 and Figure 1 report the neural comparison. At the fixed final budget, the centred continuation-aware variant reduces mean distance by 59.2% and deviation gain by 81.4% relative to the matched myopic network. The paired distance difference is −0.361 (95% seed-bootstrap interval [−0.390, −0.329]). Its additional distance improvement over the ordinary continuation critic is 0.57%; most of the improvement comes from the continuation term. The centring effect on distance is small: −0.00143, with paired interval [−0.00253, −0.00030]. Centring reduces the mean certificate by 27.1%; it has a larger effect on auditing than on the deployed policy. We report all seeds and do not select checkpoints using oracle quality.
With PPO, the continuation correction reduces mean distance by 33.3% and maximum deviation gain by 49.8%. The paired distance difference is −0.575 (95% seed-bootstrap interval [−0.739, −0.404]).
The PPO result supports a reward intervention within a fixed optimiser; it does not establish algorithmic superiority over the recent equilibrium-RL frameworks in § 2. Their continuous-time problems and model access differ from this benchmark.
A false-convergence construction has zero critic-based action gradients but true deviation gain 0.10263, bounded by 0.89674. State-constant critic offsets change bounds by less than across twenty cases. Independent terminal-moment tests check random policies and active constraints; vector projection checks do not constitute high-dimensional neural experiments.
Auditing with independent samples
Twenty fixed tabular policies in the , market receive 32,768 independent audit observations per time/state, or 786,432 each. Figure 2 compares bounds for each policy. Mean actual maximum deviation is 0.000902 and the exact-model bound is 0.001257. A Hoeffding-only observable bound averages 0.036141. Empirical Bernstein bounds with confidence-bounded transition mixing reduce this to 0.004911. Every policy is covered by each construction. Coverage in these twenty cases is a numerical check, not the source of the confidence guarantee.
Figure 2. Independent-data audits of twenty fixed policies, each using 786,432 observations. Refinement tightens Hoeffding bounds. Coverage follows from the stated assumptions, not the plotted outcomes.
The Bernstein and mixing refinements were introduced after inspecting the Hoeffding bounds, so we report them as a post hoc extension. Each construction has its own 95% simultaneous-over-states guarantee for a fixed policy; no joint 95% guarantee across all policies or constructions is claimed. Large audit requirements and abstention when variance cannot be certified are substantive properties of the method.
Financial inputs and implemented preferences
Table 2 compares published reward formulations and covariance models on historical data. Its cross-estimator distances are changes in fitted models, not pure reward errors. The restricted Habibnia–Soltanzadeh comparator uses long-only unit-capital assetwise PnL, penalty 25 and cost multiplier 25/5; SAC, shorting and lending are not replicated. Its objective depends on marginal downside payoffs and can prefer allocations different from mean–variance diversification.
| Method | Source / role | Allocation distance | Reference gap (bp) | Excess Sharpe | Realised MV (bp) |
| Fitted MV reference | Markowitz [24] | 0.00000 | 0.0000 | 0.570 | −8.497 |
| Minimum variance | Markowitz [24] | 0.23843 | 9.0252 | 0.645 | 5.274 |
| MV, LW identity | Ledoit–Wolf (2004) | 0.00538 | 0.0055 | 0.572 | −8.442 |
| MV, LW correlation | Ledoit–Wolf (2004) | 0.02196 | 0.1158 | 0.547 | −11.917 |
| MV, market factor | Fan et al. [9] | 0.05711 | 1.0487 | 0.585 | −4.738 |
| MV, three factors | Fan et al. [9] | 0.04881 | 0.6407 | 0.555 | −8.532 |
| Equal weight | DeMiguel et al. [7] | 0.43495 | 46.2626 | 0.622 | −11.893 |
| MV, LW nonlinear | Ledoit–Wolf [21] | 0.01119 | 0.0209 | 0.559 | −9.578 |
| Raw second moment | Kolm–Ritter (2019) | 0.00843 | 0.0148 | 0.573 | −7.182 |
| MVPI adaptation | Zhang et al. [29] | 0.00000 | 0.0000 | 0.570 | −8.497 |
| One MVPI step | Zhang et al. [29] | 0.00014 | 0.0000 | 0.570 | −8.442 |
| Assetwise PnL, restricted | Habibnia–Soltanzadeh (2024) | 0.79645 | 40.2554 | 0.449 | −23.619 |
| Log minus variance | Jiang et al. [16] | 0.01286 | 0.0287 | 0.572 | −7.123 |
| MV, sample covariance | Sample control (2026) | 0.01449 | 0.0356 | 0.555 | −10.114 |
| Log utility | Log-utility control (2026) | 0.54848 | 39.7855 | 0.344 | −133.295 |
| Unconstrained scalar | Analytical control (2026) | 0.00540 | 0.0043 | 0.568 | −8.882 |
| Risk-only calibration | Calibration ablation (2026) | 0.00510 | 0.0033 | 0.571 | −8.365 |
| Joint calibration (ours) | This study (2026) | 0.00000 | 0.0000 | 0.570 | −8.497 |
Table 2. Ten industries, 1990–2025. Distances/gaps use common states and the fixed-diagonal reference; covariance rows change the model. Realised metrics use own-state paths. Sources are chronological; 2026 denotes implemented controls. Published formulations are adapted, not full agent replications. Bold identifies our construction, not investment superiority.
Across all twenty-one covariance/universe cases, joint calibration has mean distance below from its own fitted reference. Raw distances range 0.00827–0.02249; transferring the unconstrained scalar formula worsens distance and fitted regret in all fourteen larger-universe cases. One existing MVPI update initialised at holdings reduces mean distance by 97.8–98.6%, without supplying the fitted optimum. These are calibration results, not a new MVPI algorithm.
Dimension increases from 0.083 to 0.250 and 0.408, with 65/495/1,274 distinct mean/covariance entries. The classifications also change the opportunity set, so this is not a causal experiment in dimension alone. DeMiguel et al.’s [7] 3,000/6,000-month examples are model-dependent illustrations, not universal requirements. Simple controls and shrinkage remain essential.
Sources: Markowitz [24]; Ledoit–Wolf [19, 20]; Fan et al. [9]; DeMiguel et al. [7]; Ledoit–Wolf [21]; Kolm–Ritter [17]; Zhang et al. [29]; Habibnia–Soltanzadeh [14]; Jiang et al. [16].
Minimum variance has the highest realised MV among quadratic comparators in all twenty-one cases, while joint calibration remains below raw. All eighteen cross-covariance target comparisons have paired 95% intervals containing zero, including simultaneous path-bootstrap bands. Improved alignment therefore does not establish improved historical investment performance. In 2,160 known-covariance fitted problems using the six estimators other than nonlinear shrinkage, factor structure helps when its residual assumptions hold; misspecification can worsen population regret while leaving exact reward calibration intact. A separate fifty-state borrowing/shorting test gives joint distance , versus 0.00997 when financing charges are not scaled.
Financial interpretation and validation
A financial explanation should distinguish changed opportunities (covariance), changed preferences (reward), and implementation error (actor or critic). In the 49-industry market-factor case, changing covariance moves the target by 0.03992; within-model raw and joint reward discrepancies are 0.02174 and . Combining them into one error score would obscure which intervention changed the decision.
Nonlinear shrinkage strengthens the estimation control. Ledoit and Wolf [21], Table 8, caution against transferring unconstrained performance rankings to long-only portfolios. With nonlinear shrinkage, joint allocation distances are , and for 10/30/49 industries. Its 49-industry target improves realised MV over fixed diagonal shrinkage by 2.54 monthly basis points, but the paired 95% interval [−0.15, 5.20] includes zero.
The learning audit explains whether changing an action could improve the stated objective. State-constant critic errors are harmless; state-dependent errors can conceal non-equilibrium decisions. Feasible deviations respect boundary optima. This is decision-level validation, not attribution of hidden neural representations.
A worked validation decision. Let a mandate tolerate at most in conditional terminal mean–variance improvement from any one-action change: ten basis points in the experiment’s normalised utility units, not ten basis points of realised return. Require a 95% simultaneous-over-time/state upper bound . For the first saved audit policy (seed 0), 786,432 fresh observations give using Bernstein bounds and estimated mixing. The decision is inconclusive: the bound exceeds the tolerance, although the simulator’s true maximum gain, 0.000506, is below it. A separate mandate allowing would pass the same bound; Hoeffding alone () would certify neither. These are illustrative tolerances applied retrospectively, not preregistered acceptance results. Operational use must fix the tolerance, policy and audit construction before inspecting audit outcomes; an infinite bound also requires abstention.
Scope of the evidence. The sequential experiments establish the correction’s effect in a small observed-state market with conditional sampling. The multiasset historical study instead establishes alignment with a fitted one-period objective; it does not validate the sequential learner or its independent-data guarantee on market histories. A realistic sequential multiasset test remains open. Unknown regimes, market impact, wealth-dependent policies and
@page 6/8 starts_mid=0 ends_mid=1
| Method | Source / role | Allocation distance | Reference gap (bp) | Excess Sharpe | Realised MV (bp) |
| Fitted MV reference | Markowitz [24] | 0.00000 | 0.0000 | 0.570 | −8.497 |
| Minimum variance | Markowitz [24] | 0.23843 | 9.0252 | 0.645 | 5.274 |
| MV, LW identity | Ledoit–Wolf (2004) | 0.00538 | 0.0055 | 0.572 | −8.442 |
| MV, LW correlation | Ledoit–Wolf (2004) | 0.02196 | 0.1158 | 0.547 | −11.917 |
| MV, market factor | Fan et al. [9] | 0.05711 | 1.0487 | 0.585 | −4.738 |
| MV, three factors | Fan et al. [9] | 0.04881 | 0.6407 | 0.555 | −8.532 |
| Equal weight | DeMiguel et al. [7] | 0.43495 | 46.2626 | 0.622 | −11.893 |
| MV, LW nonlinear | Ledoit–Wolf [21] | 0.01119 | 0.0209 | 0.559 | −9.578 |
| Raw second moment | Kolm–Ritter (2019) | 0.00843 | 0.0148 | 0.573 | −7.182 |
| MVPI adaptation | Zhang et al. [29] | 0.00000 | 0.0000 | 0.570 | −8.497 |
| One MVPI step | Zhang et al. [29] | 0.00014 | 0.0000 | 0.570 | −8.442 |
| Assetwise PnL, restricted | Habibnia–Soltanzadeh (2024) | 0.79645 | 40.2554 | 0.449 | −23.619 |
| Log minus variance | Jiang et al. [16] | 0.01286 | 0.0287 | 0.572 | −7.123 |
| MV, sample covariance | Sample control (2026) | 0.01449 | 0.0356 | 0.555 | −10.114 |
| Log utility | Log-utility control (2026) | 0.54848 | 39.7855 | 0.344 | −133.295 |
| Unconstrained scalar | Analytical control (2026) | 0.00540 | 0.0043 | 0.568 | −8.882 |
| Risk-only calibration | Calibration ablation (2026) | 0.00510 | 0.0033 | 0.571 | −8.365 |
| Joint calibration (ours) | This study (2026) | 0.00000 | 0.0000 | 0.570 | −8.497 |
Table 2 (PDF p. 6). Ten industries, 1990–2025. Distances/gaps use common states and the fixed-diagonal reference; covariance rows change the model. Realised metrics use own-state paths. Sources are chronological; 2026 denotes implemented controls. Published formulations are adapted, not full agent replications. Bold identifies our construction, not investment superiority.
Sources: Markowitz [24]; Ledoit–Wolf [19, 20]; Fan et al. [9]; DeMiguel et al. [7]; Ledoit–Wolf [21]; Kolm–Ritter [17]; Zhang et al. [29]; Habibnia–Soltanzadeh [14]; Jiang et al. [16].
Minimum variance has the highest realised MV among quadratic comparators in all twenty-one cases, while joint calibration remains below raw. All eighteen cross-covariance target comparisons have paired 95% intervals containing zero, including simultaneous path-bootstrap bands. Improved alignment therefore does not establish improved historical investment performance. In 2,160 known-covariance fitted problems using the six estimators other than nonlinear shrinkage, factor structure helps when its residual assumptions hold; misspecification can worsen population regret while leaving exact reward calibration intact. A separate fifty-state borrowing/shorting test gives joint distance , versus 0.00997 when financing charges are not scaled.
Financial interpretation and validation
A financial explanation should distinguish changed opportunities (covariance), changed preferences (reward), and implementation error (actor or critic). In the 49-industry market-factor case, changing covariance moves the target by 0.03992; within-model raw and joint reward discrepancies are 0.02174 and . Combining them into one error score would obscure which intervention changed the decision.
Nonlinear shrinkage strengthens the estimation control. Ledoit and Wolf [21], Table 8, caution against transferring unconstrained performance rankings to long-only portfolios. With nonlinear shrinkage, joint allocation distances are , and for 10/30/49 industries. Its 49-industry target improves realised MV over fixed diagonal shrinkage by 2.54 monthly basis points, but the paired 95% interval [−0.15, 5.20] includes zero.
The learning audit explains whether changing an action could improve the stated objective. State-constant critic errors are harmless; state-dependent errors can conceal non-equilibrium decisions. Feasible deviations respect boundary optima. This is decision-level validation, not attribution of hidden neural representations.
A worked validation decision. Let a mandate tolerate at most in conditional terminal mean–variance improvement from any one-action change: ten basis points in the experiment’s normalised utility units, not ten basis points of realised return. Require a 95% simultaneous-over-time/state upper bound . For the first saved audit policy (seed 0), 786,432 fresh observations give using Bernstein bounds and estimated mixing. The decision is inconclusive: the bound exceeds the tolerance, although the simulator’s true maximum gain, 0.000506, is below it. A separate mandate allowing would pass the same bound; Hoeffding alone () would certify neither. These are illustrative tolerances applied retrospectively, not preregistered acceptance results. Operational use must fix the tolerance, policy and audit construction before inspecting audit outcomes; an infinite bound also requires abstention.
Scope of the evidence. The sequential experiments establish the correction’s effect in a small observed-state market with conditional sampling. The multiasset historical study instead establishes alignment with a fitted one-period objective; it does not validate the sequential learner or its independent-data guarantee on market histories. A realistic sequential multiasset test remains open. Unknown regimes, market impact, wealth-dependent policies and transaction-cost states require additional continuation analysis, while dependent audit data require new statistical guarantees.
Figure 3. 49-industry covariance and reward interventions. Joint calibration preserves fitted preferences, without identifying true covariance or establishing superior own-state performance.
Kolm and Ritter [18], Exhibit 2 and pp. 90–91, distinguish reward specification, learning and decision quality. Our audit connects these stages within model-risk review, taking the mandate as given. Direct optimisation remains appropriate for tractable one-period allocation; sequential learning additionally needs an account of remaining decision error.
Independent-data certificate
Fix a scalar policy and critic independently of audit data, and assume known support . At every obtain independent conditional observations. Pair them and differentiate Eq. (14); each gradient sample has absolute bound
The pair variable estimates conditional variance and lies in . Bellman targets have range at most . Hoeffding radii with failure probability give a gradient upper norm, variance lower bound and residual intervals . Then ; use in Eq. (12). Substitute these bounds in Eq. (13). The result covers all time/state cells with probability at least , conditional on training.
For a variable of known range width and sample size , Maurer–Pontil [25], Theorem 4, gives the two-sided radius
where uses denominator . Confidence intervals for each transition entry additionally give , where bounds each entry error. Include additional events in the union bound. The construction returns infinity if any variance lower bound is nonpositive. Adaptive policy selection on the audit set, dependent observations or unknown support require additional analysis; none is assumed here. The scalar implementation bounds unrestricted deviations, which also upper-bound constrained ones.
Raw-reward discrepancy
For constant independent moments with , the quadratic Bellman solution has , with terminal and .
Induction gives , hence global concave action maximisation. Put and . Because , are independent of , the Hessian in and is exactly quadratic in , analytic near . At zero it is , so its positive minimiser is analytic nearby.
Writing , the leading action error is . At , mean wealth is and variance . Minimisation gives ; summing centred mean squares and wealth variances yields Eq. (3). This is a fixed-horizon expansion, not a uniform joint limit. For , exact matching is impossible: matching initially creates positive subsequent wealth variance, whereas the final raw policy has nonzero wealth slope and equilibrium is constant.
Reproduction and numerical checks
Conditional-sample generators use seeds 0–19, also controlling neural weights and minibatches; PPO uses seeds 0–9. Both PPO arms share data access and refresh frequency; continuation is fitted from training outcomes, not the analytic reference. The pilot, twenty-seed extension and PPO experiment are archived separately. Bootstrap intervals use 10,000 resamples of seed averages; all final-budget runs are retained.
Historical identity shrinkage uses centred , and target , with intensity from the empirical fourth moment; final covariance is converted to denominator . Constant-correlation shrinkage uses the plug-in formulas of Ledoit and Wolf [19]. Factor models use intercept OLS, market/SMB/HML and diagonal residual variances with denominator . This convention adapts Fan et al. [9]. Nonlinear shrinkage uses the covariance branch of QuEST v027 through a pinned Python port, with trace-constrained SLSQP fitting, tolerance , and effective sample size after demeaning. No additional rescaling is applied. This implements the estimator, not the original CRSP backtest. Missing and duplicate months are rejected. Holdings drift with gross asset returns; costs are charged separately at the stated first-order rate.
Historical uncertainty uses 1,000 shared circular twelve-month blocks, seed 8128, conditional on realised strategy paths; simultaneous covariance bands use the maximum centred deviation across eighteen comparisons. Quadratic optimisation is checked independently by a first-order linear-programme certificate in original objective units, with accepted gap at most . Reconstruction verifies 67,392 historical decisions, 2,160 known-covariance fitted problems and 144 anchor checks. RL tests independently evaluate terminal moments, gradient/covariance identities, critic offsets, active boundaries and audit support.
Conclusion
The residual certificate connects learning errors to financial decision quality, respecting critic-offset invariance and convex constraints. Controlled actor–critic and PPO comparisons test continuation correction without an oracle target. Historical experiments separately test fitted reward alignment. Together they support validation that distinguishes objective fidelity from investment performance and explicitly abstains when audit evidence is insufficient.
Declarations
Funding and computational resources. This study received no project-specific research funding. No computing resources from Virginia Tech or the authors’ other affiliated universities were used in this study.
Code and data availability. The simulation experiments use data generated by the accompanying code. The historical analysis uses publicly available monthly value-weighted US industry portfolio returns and factors from Kenneth French’s Data Library [10], using the archived August 2026 vintage. Replication code, archived input data, experiment outputs and provenance records are available at https://github.com/DataismLab/Reward-alignment.
References
References
- [1]Suleyman Basak and Georgy Chabakauri. 2010. Dynamic Mean-Variance Asset Allocation. The Review of Financial Studies 23, 8 (2010), 2970–3016. doi:10.1093/rfs/hhq028DOI
- [2]Tomas Björk, Mariana Khapko, and Agatha Murgoci. 2016. Time Inconsistent Stochastic Control in Continuous Time: Theory and Examples. arXiv:1612.03650arxiv.org/abs/1612.03650
- [3]Zaiwei Chen, Sheng Zhang, Zhe Zhang, Shaan Ul Haque, and Siva Theja Maguluri. 2025. A Non-Asymptotic Theory of Seminorm Lyapunov Stability: From Deterministic to Stochastic Iterative Algorithms. arXiv:2502.14208arxiv.org/abs/2502.14208
- [4]Lin William Cong, Ke Tang, and Jingyuan Wang. 2026. AlphaPortfolio: Goal-Oriented Investment Management Through Deep Reinforcement Learning. NBER Working Paper 35195. National Bureau of Economic Research. doi:10.3386/w35195DOI
- [5]Christoph Cichowsky. 2013. Time-Consistent Mean-Variance Portfolio Selection in Discrete and Continuous Time. Finance and Stochastics 17, 2 (2013), 227–271. doi:10.1007/s00780-012-0189-9
- [6]Min Dai, Yuchao Dong, and Yanwei Jia. 2023. Learning Equilibrium Mean-Variance Strategy. Mathematical Finance 33, 4 (2023), 1166–1212. doi:10.1111/mafi.12402
- [7]Victor DeMiguel, Lorenzo Garlappi, and Raman Uppal. 2009. Optimal Versus Naive Diversification: How Inefficient is the 1/N Portfolio Strategy? The Review of Financial Studies 22, 5 (2009), 1915–1953. doi:10.1093/rfs/hhm075
- [8]Victor DeMiguel, Alberto Martín-Utrera, and Raman Uppal. 2024. A Multifactor Perspective on Volatility-Managed Portfolios. The Journal of Finance 79, 6 (2024), 3859–3891. doi:10.1111/jofi.13395DOI
- [9]Jianqing Fan, Yingying Fan, and Jinchi Lv. 2008. High Dimensional Covariance Matrix Estimation Using a Factor Model. Journal of Econometrics 147, 1 (2008), 186–197. doi:10.1016/j.jeconom.2008.09.017
- [10]Kenneth R. French. 2026. Data Library: Industry Portfolios and Fama–French Factors. Tuck School of Business, Dartmouth College. Retrieved October 1, 2026 from https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html August 2026 data vintage.
- [11]Nicolae Gârleanu and Lasse Heje Pedersen. 2013. Dynamic Trading with Predictable Returns and Transaction Costs. The Journal of Finance 68, 6 (2013), 2309–2340. doi:10.1111/jofi.12080DOI
- [12]Maximilian Gollart and Yarema Okhrin. 2025. A Reinforcement Learning Approach to Dynamic Portfolio Optimization. Annals of Operations Research (2025), 1–26. doi:10.1007/s10479-025-06649-x. Published online 25 June 2025.DOI
- [13]Xin Guo, Yijie Huang, and Xiang Yu. 2026. Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problems. arXiv:2606.11798arxiv.org/abs/2606.11798
- [14]Ali Habibnia and Mahdi Soltanzadeh. 2024. Optimizing Portfolio with Two-Sided Transactions and Lending: A Reinforcement Learning Framework. arXiv:2408.05382arxiv.org/abs/2408.05382
- [15]Yifu Jiang, Jose Olmo, and Majed Atwi. 2024. Deep Reinforcement Learning for Portfolio Selection. Global Finance Journal 62 (2024), 101016. doi:10.1016/j.gfj.2024.101016DOI
- [16]Yifu Jiang, Jose Olmo, and Majed Atwi. 2025. High-Dimensional Multi-Period Portfolio Allocation Using Deep Reinforcement Learning. International Review of Economics & Finance 98 (2025), 103996. doi:10.1016/j.iref.2025.103996DOI
- [17]Petter N. Kolm and Gordon Ritter. 2019. Dynamic Replication and Hedging: A Reinforcement Learning Approach. The Journal of Financial Data Science 1, 1 (2019), 159–171. doi:10.3905/jfds.2019.1.1.159DOI
- [18]Petter N. Kolm and Gordon Ritter. 2025. Reinforcement Learning for Asset and Portfolio Management. The Journal of Portfolio Management 52, 2 (2025), 81–95. doi:10.3905/jpm.2025.1.782DOI
- [19]Olivier Ledoit and Michael Wolf. 2004. Honey, I Shrunk the Sample Covariance Matrix. The Journal of Portfolio Management 30, 4 (2004), 110–119. doi:10.3905/jpm.2004.110DOI
- [20]Olivier Ledoit and Michael Wolf. 2004. A Well-Conditioned Estimator for Large-Dimensional Covariance Matrices. Journal of Multivariate Analysis 88, 2 (2004), 365–411. doi:10.1016/S0047-259X(03)00096-4DOI
- [21]Olivier Ledoit and Michael Wolf. 2017. Nonlinear Shrinkage of the Covariance Matrix for Portfolio Selection: Markowitz Meets Goldilocks. The Review of Financial Studies 30, 12 (2017), 4349–4388. doi:10.1093/rfs/hhx052DOI
- [22]Nixie Sapphira Lesmana and Chi Seng Pun. 2025. A Subgame Perfect Equilibrium Reinforcement Learning Approach to Time-Inconsistent Problems. SIAM Journal on Financial Mathematics 16, 1 (2025), 68–122. doi:10.1137/23M1594510DOI
- [23]Duan Li and Wan-Lung Ng. 2000. Optimal Dynamic Portfolio Selection: Multi-period Mean-Variance Formulation. Mathematical Finance 10, 3 (2000), 387–406. doi:10.1111/1467-9965.00100DOI
- [24]Harry Markowitz. 1952. Portfolio Selection. The Journal of Finance 7, 1 (1952), 77–91. doi:10.1111/j.1540-6261.1952.tb01525.xDOI
- [25]Andreas Maurer and Massimiliano Pontil. 2009. Empirical Bernstein Bounds and Sample Variance Penalization. In Proceedings of the 22nd Annual Conference on Learning Theory (COLT 2009). Montreal, Canada. https://www.cs.mcgill.ca/~colt2009/papers/012.pdfarxiv.org/abs/0907.3740
- [26]Tejaram Sangadi, L. A. Prashanth, and Krishna Jagannathan. 2025. A Finite-Sample Analysis of an Actor-Critic Algorithm for Mean-Variance Optimization in a Discounted MDP. arXiv:2406.07892 Version 2.arxiv.org/abs/2406.07892
- [27]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347arxiv.org/abs/1707.06347
- [28]Frederik Simon, Sebastian Weibels, and Tom Zimmerman. 2026. Deep Parametric Portfolio Policies. Management Science (2026), 1–19. doi:10.1287/mnsc.2025.00721 Articles in Advance.DOI
- [29]Shengtong Zhang, Bo Liu, and Shimon Whiteson. 2021. Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. AAAI Press, 10905–10913. doi:10.1609/aaai.v35i12.17302DOI


