betaFeedback

WorksComputer Science

GP–RL–MPC: Uncertainty-Aware Reinforcement Learning-Augmented Model Predictive Control with Gaussian Process Postdecision State-Value Functions

Abstract

We propose Gaussian Process Reinforcement Learning-Augmented Model Predictive Control (GP–RL–MPC), which uses a learned critic as the MPC terminal penalty to account for costs beyond the prediction horizon. Gaussian process regression (GPR) enables data-efficient critic learning from limited, noisy rollouts within optimistic approximate policy iteration (OAPI), with soft updates mitigating policy chattering. A tunable risk coefficient weights GP epistemic uncertainty in the terminal cost, regulating sensitivity to uncertain critic predictions. The postdecision state formulation shifts disturbance averaging offline to account for aleatoric uncertainty from the terminal transition onward. Under matching objectives and admissible actions, a one-step horizon recovers greedy value-space RL. After fitting GP hyperparameters by exact marginal likelihood, we construct a sparse deterministic training conditional (DTC) posterior using inducing points selected by pivoted Cholesky (PC), reducing online evaluation costs as the training dataset grows. In a continuous stirred-tank reactor (CSTR) subject to exogenous disturbances, GP–RL–MPC with an appropriately selected risk coefficient and policy screening reduces the sum of squared errors (SSE) by an average of 80.52% relative to an otherwise matched baseline MPC across five random seeds (sample standard deviation: 3.48%).

Introduction

Model predictive control (MPC) handles nonlinear dynamics, multivariable interactions, and operating constraints through repeated finite-horizon optimization (Lee et al., [8]). Computational demands and growing prediction uncertainty limit practical horizon lengths, making terminal costs important for accounting for long-term consequences beyond the prediction horizon. Under appropriate assumptions, suitable terminal ingredients can support recursive feasibility, closed-loop stability, and performance guarantees. However, constructing such terminal ingredients remains challenging, particularly for nonlinear stochastic systems.

Reinforcement learning (RL) offers a complementary approach by learning long-term cost information from closed-loop rollouts to approximate terminal values in MPC (Bertsekas, [3]). Previous studies have explored approximate dynamic programming (ADP) for control (Lee et al., [9]), RL-based parameterization of economic nonlinear MPC (NMPC) (Gros and Zanon, [4]), and terminal-value learning using approximate value iteration (Moreno-Mora et al., [14]), partially input-convex neural networks (PICNNs) (Li and Lee, [11]), and radial basis function (RBF) networks (Li and Lee, [12]). A key challenge in extending these approaches to stochastic systems is incorporating the effects of random state transitions into the learned continuation value without substantially increasing the computational burden of online MPC optimization.

The conventional (predecision) state-value function represents the expected cumulative cost conditioned on the current state, but evaluating it at a nominal terminal state omits averaging over disturbed terminal states. Explicitly computing this additional expectation online adds to the burden of stochastic MPC [13]. A postdecision state captures the known effect of the control action before the stochastic transition, allowing its value function to incorporate the expectation over successor states [19, 15]. Learning this function from disturbed rollouts shifts disturbance averaging offline while preserving deterministic online MPC optimization. A one-step horizon recovers greedy value-space RL, while longer horizons provide multistep lookahead.

Building on the postdecision state formulation, we develop GP–RL–MPC to improve short-horizon MPC performance while preserving its computational advantages. A Gaussian process (GP) critic learns the conditional expected cost-to-go from disturbed rollouts under a fixed policy, thereby incorporating future disturbance effects into postdecision value estimates evaluated at nominal MPC terminal states. The terminal penalty is defined as μGPp+κσGPp\mu_{\mathrm{GP}}^{p} + \kappa\sigma_{\mathrm{GP}}^{p}, where μGPp\mu_{\mathrm{GP}}^{p} and σGPp\sigma_{\mathrm{GP}}^{p} denote the GP posterior mean and latent posterior standard deviation, respectively. The tunable risk coefficient κ\kappa regulates the influence of critic epistemic uncertainty on control decisions [17].

Critic learning is integrated into an optimistic approximate policy iteration (OAPI) framework, with soft updates mitigating policy chattering [1]. GP hyperparameters are estimated by maximizing the exact marginal likelihood, after which pivoted Cholesky (PC) selects inducing points for a sparse deterministic training conditional (DTC) approximation [5, 16]. This sparse representation makes online posterior mean and variance evaluation costs depend primarily on the inducing-set size rather than the full training-set size, reducing computational costs while retaining estimates of critic epistemic uncertainty.

The main contributions of the proposed uncertainty-aware framework, illustrated in Figure 1, are as follows:

Overview of the proposed GP–RL–MPC algorithm

Figure 1. Overview of the proposed GP–RL–MPC algorithm.

  • Uncertainty propagation: A postdecision state-value critic that learns the expected effects of aleatoric uncertainty on the cost-to-go from disturbed rollouts, enabling nominal MPC to account for future disturbances without explicitly evaluating expectations during online optimization.

  • Uncertainty calibration: A computationally efficient GP critic that combines exact marginal-likelihood hyperparameter estimation with pivoted Cholesky (PC) inducing-point selection and deterministic training conditional (DTC) sparsification, reducing online computational costs while retaining estimates of epistemic uncertainty.

  • Uncertainty regulation: A terminal penalty that incorporates GP epistemic uncertainty through a tunable risk coefficient, combined with soft critic updates within optimistic approximate policy iteration, to promote reliable policy improvement while preserving the efficiency of short-horizon MPC.

Preliminaries

Problem Formulation

For nominal prediction, consider the discrete-time nonlinear model

xk+1=fs(xk,dk,uk),(1)x_{k+1}=f_s(x_k,d_k,u_k), \tag*{(1)}

where xk∈Rnxx_k \in\mathbb{R}^{n_x}, dk∈Rndd_k \in\mathbb{R}^{n_d}, and uk∈Rnuu_k \in\mathbb{R}^{n_u} denote the system state, information state, and manipulated input, respectively. The information state collects references, time-varying parameters, disturbances, and information related to model-plant mismatch. The controlled output is given by yk=h(xk)y_k=h(x_k), where h:Rnx→Rnyh:\mathbb{R}^{n_x}\to\mathbb{R}^{n_y}, assuming no direct input feedthrough. The stochastic components of the information state are modeled as zero-drift random walks, dk+1=dk+εkd_{k+1}=d_k+\varepsilon_k, where {εk}k≥0\{\varepsilon_k\}_{k\geq0} is a sequence of independent and identically distributed (i.i.d.) Gaussian innovations with εk∼N(0,Rε)\varepsilon_k\sim\mathcal{N}(0,R_\varepsilon). Known references and exogenous forecasts evolve according to prescribed trajectories.

Given xkx_k and dkd_k, or their estimates, the nominal NMPC problem is formulated as [7, 10]

JHp(xk):=min⁡{uk+l∣k}l=0Hp−1∑l=0Hp−1γlℓ(xk+l∣k,dk+l∣k,uk+l∣k)+γHpΨ(xk+Hp∣k,dk+Hp∣k),s.t.xk∣k=xk,dk∣k=dk,xk+l+1∣k=fs(xk+l∣k,dk+l∣k,uk+l∣k),dk+l+1∣k=dk+l∣k,xk+l∣k∈X,uk+l∣k∈U,xk+Hp∣k∈Xf.(2)\begin{aligned} J_{H_p}(x_k)&:=\min_{\{u_{k+l|k}\}_{l=0}^{H_p-1}}\sum_{l=0}^{H_p-1}\gamma^l\ell(x_{k+l|k},d_{k+l|k},u_{k+l|k})+\gamma^{H_p}\Psi(x_{k+H_p|k},d_{k+H_p|k}),\\ \mathrm{s.t.}\quad x_{k|k}&=x_k,\qquad d_{k|k}=d_k,\\ x_{k+l+1|k}&=f_s(x_{k+l|k},d_{k+l|k},u_{k+l|k}),\\ d_{k+l+1|k}&=d_{k+l|k},\\ x_{k+l|k}&\in\mathcal{X},\qquad u_{k+l|k}\in\mathcal{U},\\ x_{k+H_p|k}&\in\mathcal{X}_f. \tag*{(2)} \end{aligned}

Here, xk+l∣kx_{k+l|k}, dk+l∣kd_{k+l|k}, and uk+l∣ku_{k+l|k} denote the predicted state, information state, and manipulated input, respectively, at ll steps ahead, as computed at time kk, and HpH_p denotes the prediction horizon. Setting future stochastic innovations to zero yields certainty-equivalent (CE) predictions without explicitly propagating disturbance uncertainty. The admissible sets, X\mathcal{X} and U\mathcal{U}, are assumed to be closed and compact, and the stage cost ℓ(x,d,u)\ell(x,d,u) quantifies the tracking or economic performance [18]. While conventional deterministic MPC typically employs an undiscounted objective, we introduce a discount factor γ∈(0,1)\gamma\in(0,1) to ensure consistency with infinite-horizon value learning under persistent stochastic disturbances, assuming that the expected discounted cumulative cost is finite.

In conventional MPC, a terminal set Xf⊆X\mathcal{X}_f \subseteq\mathcal{X} and penalty Ψ\Psi can ensure recursive feasibility and closed-loop stability under suitable assumptions but are challenging to construct for nonlinear stochastic systems. We learn the terminal cost from closed-loop rollouts to capture long-term information while retaining a short prediction horizon for computational efficiency. In this study, the proposed controller enforces hard state and input constraints within the prediction horizon. The numerical results shown in § 4 demonstrate that the states of interest remain within their prescribed bounds with substantial safety margins [12]. Selected state constraints can be softened by penalizing bounded violations while retaining hard input constraints and outer state bounds. Using the same augmented stage cost in the MPC objective and Monte Carlo (MC) returns from disturbed rollouts incorporates expected future violation costs into the learned postdecision state-value function. No additional terminal violation penalty is required; Appendix § 7 provides the detailed formulation and its consistency with the value-learning targets.

Postdecision State-Value Functions

A postdecision state separates the known effects of a control action from the subsequent stochastic transition [20]. When the system dynamics admit such a decomposition, the state transition can be expressed as

skp=Fa(sk,uk),sk+1=Fw(skp,wk).(3)s_k^p = F_a(s_k,u_k), \qquad s_{k+1} = F_w(s_k^p,w_k). \tag*{(3)}

Here, sk∈Ss_k \in\mathcal{S} denotes the augmented predecision state, comprising the plant state xkx_k, information state dkd_k, and any additional variables, such as input memory or contextual information, required to satisfy the Markov property. The superscript pp denotes the postdecision state, and wkw_k represents the stochastic innovation realized after the control action. For the postdecision state to provide an exact representation, it must contain sufficient information to characterize the conditional distribution of the next predecision state, such that

P(sk+1∣sk,uk)=P(sk+1∣skp).(4)\mathbb{P}(s_{k+1} \mid s_k,u_k) = \mathbb{P}(s_{k+1} \mid s_k^p). \tag*{(4)}

For a fixed policy π\pi, the postdecision state-value function is defined as

Vπp(skp):=E[Vπ(sk+1)∣skp]=Eπ[Gk+1∣skp],(5)V_\pi^p(s_k^p) := \mathbb{E}[V_\pi(s_{k+1}) \mid s_k^p] = \mathbb{E}_\pi[G_{k+1} \mid s_k^p], \tag*{(5)}

where VπV_\pi is the conventional predecision state-value function and Gk+1:=∑j=k+1∞γj−k−1ℓ(sj,uj)G_{k+1} := \sum_{j=k+1}^{\infty}\gamma^{j-k-1}\ell(s_j,u_j) denotes the Monte Carlo (MC) return starting at time k+1k+1, with subsequent actions determined by π\pi. Thus, the postdecision state-value function captures the expected cumulative cost arising from the next stochastic transition and all subsequent disturbances, while excluding the current stage cost. Under the sufficiency condition, the Bellman expectation equation for a fixed deterministic policy π\pi becomes

Vπ(s)=ℓ(s,π(s))+γVπp(Fa(s,π(s))).(6)V_\pi(s) = \ell(s,\pi(s)) + \gamma V_\pi^p(F_a(s,\pi(s))). \tag*{(6)}

The corresponding one-step policy improvement is obtained by minimizing the right-hand side of Eq. (6) with respect to the current control input. Importantly, because the postdecision state-value function already incorporates the expectation over subsequent stochastic transitions, this minimization does not require explicit disturbance averaging. This property makes the postdecision formulation particularly attractive for integration with nominal MPC, allowing stochastic effects on future costs to be accounted for without increasing the complexity of online optimization. Further details on the construction of postdecision training targets and their consistency with nominal MPC terminal-state queries are provided in Appendix § 10.

Postdecision State-Value Learning as Optimistic Approximate Policy Iteration

Starting from a baseline MPC policy π(0)\pi^{(0)} without a terminal penalty, each policy iteration (PI) #n evaluates the preceding policy π(n−1)\pi^{(n-1)} using disturbed rollouts. The resulting data are used to update the GP critic, whose posterior mean μGPp,(n)\mu_{\mathrm{GP}}^{p,(n)} approximates the postdecision state-value function Vπ(n−1)pV_{\pi^{(n-1)}}^{p}. The GP posterior mean and latent posterior standard deviation are then combined to construct the uncertainty-regulated terminal value estimate Vθp,(n)V_{\theta}^{p,(n)}, which is incorporated into the constrained MPC optimization to obtain a candidate policy π(n)\pi^{(n)}. Thus, the critic learning serves as approximate policy evaluation, while the MPC optimization performs policy improvement using the learned postdecision state-value function Bertsekas [3]. This iterative procedure constitutes optimistic approximate policy iteration (OAPI) Bertsekas [2], in which policy improvement is performed following partial, sample-based policy evaluation rather than waiting for complete convergence of the value function. Such incomplete evaluation can lead to oscillations between successive policies, commonly referred to as policy chattering Li and Lee [11]. The conventional predecision state-value function and its relationship to the postdecision state-value function are reviewed in Appendix § 9.

Proposed GP–RL–MPC Algorithm

The proposed GP–RL–MPC algorithm integrates approximate policy evaluation and policy improvement by iteratively learning a postdecision state-value critic from disturbed closed-loop rollouts and updating the MPC policy through constrained nominal optimization. At PI #n, the preceding policy π(n−1)\pi^{(n-1)} is held fixed while disturbed rollouts are generated to obtain MC training pairs (χkp,(r),Gk(r))(\chi_k^{p,(r)}, G_k^{(r)}), where rr indexes rollouts and χkp,(r):=φ(skp,(r))\chi_k^{p,(r)} := \varphi(s_k^{p,(r)}) denotes the feature representation of the postdecision state used as input to the GP critic. These training pairs are pooled and reindexed to form the dataset Dn−1p={(χi,Gi)}i=1Npool\mathcal{D}_{n-1}^{p} = \{(\chi_i,G_i)\}_{i=1}^{N_{\mathrm{pool}}} for GP regression. The construction of the training pairs and their temporal alignment with the postdecision states are detailed in Appendix § 10. The resulting GP posterior mean μGPp,(n)\mu_{\mathrm{GP}}^{p,(n)} and latent posterior standard deviation σGPp,(n)\sigma_{\mathrm{GP}}^{p,(n)} are used to construct an uncertainty-regulated terminal penalty for the MPC optimization, yielding an updated control policy. Soft critic updates are employed to mitigate policy chattering across successive iterations. The complete GP–RL–MPC procedure is summarized in Algorithm 1.

Soft Updates in Value Space

At the first policy iteration, the GP critic is trained directly on the Monte Carlo returns GiG_i. From the second iteration onward, soft updates are introduced by constructing the regression targets as

yi(n)=(1−τ)μGPp,(n−1)(χip)+τGi,0<τ≤1,(7)y_i^{(n)} = (1-\tau)\mu_{\mathrm{GP}}^{p,(n-1)}(\chi_i^p) + \tau G_i,\qquad0 < \tau\le1, \tag*{(7)}

where τ\tau is the soft-update parameter and μGPp,(n−1)\mu_{\mathrm{GP}}^{p,(n-1)} is the posterior mean of the preceding GP critic, held fixed during target construction. The update is performed in the original cost units, blending the previous critic prediction with the newly observed MC return. Smaller values of τ\tau provide stronger damping of critic updates, thereby mitigating abrupt changes in the learned value function and the resulting MPC policy. Setting τ=1\tau= 1 recovers the raw MC targets, corresponding to a hard update.

The resulting input-target pairs are partitioned into training and diagnostic subsets using target-quantile stratification [12]. Input and output scalers are fitted using the initial training subset and subsequently held fixed to ensure consistent scaling across policy iterations. At each iteration, the GP hyperparameters are initialized from their values in the preceding iteration and refitted using the current training dataset. This procedure combines damped policy evaluation with consistent data preprocessing and warm-started GP training to improve the stability of successive policy updates.

Exact GPR Training and PC-DTC Deployment

We fit the full GP critic offline by maximizing the exact log marginal likelihood. With the fitted hyperparameters fixed, pivoted Cholesky (PC) selects M≪NM \ll N inducing points from the NN training inputs [5]. A deterministic training conditional (DTC) approximation incorporates information from all training pairs through this inducing representation [16]. The latent uncertainty excludes observation noise and represents critic epistemic uncertainty conditional on the fitted GP model. After precomputing the Cholesky factors and predictive mean coefficients offline, evaluating the mean and variance at one query requires O(M2+Mnχ)O(M^2 + Mn_\chi) operations, compared with O(N2+Nnχ)O(N^2 + Nn_\chi) for exact GPR using dense Cholesky factors, where nχn_\chi is the input dimension [17]. The regression model, inducing-point selection, and predictive formulas are detailed in Appendix § 8.

Uncertainty-Regulated Terminal Cost

Using the deployed GP posterior moments, we construct an uncertainty-regulated terminal cost based on an upper confidence bound (UCB) acquisition rule:

Vθp(χ):=μGPp(χ)+κσGPp(χ),(8)V_\theta^p(\chi) := \mu_{\mathrm{GP}}^p(\chi) + \kappa\sigma_{\mathrm{GP}}^p(\chi), \tag*{(8)}

where κ∈R\kappa\in\mathbb{R} controls the influence of epistemic uncertainty. Positive κ\kappa penalizes uncertain value estimates, yielding a more conservative controller; κ=0\kappa= 0 uses the posterior mean alone, and negative κ\kappa favors actions leading to uncertain regions, thereby encouraging exploration. Incorporating this approximation at the postdecision terminal feature χk+Hp−1∣kp\chi_{k+H_p-1|k}^p gives the GP–RL–MPC objective

JHpGP(sk;θ):=min⁡{uk+l∣k}l=0Hp−1{∑l=0Hp−1γlℓ(sk+l∣k,uk+l∣k)+μtermγHpVθp(χk+Hp−1∣kp)}.(9)J_{H_p}^{\mathrm{GP}}(s_k;\theta) := \min_{\{u_{k+l|k}\}_{l=0}^{H_p-1}} \left\{ \sum_{l=0}^{H_p-1} \gamma^l \ell(s_{k+l|k},u_{k+l|k}) + \mu_{\mathrm{term}}\gamma^{H_p} V_\theta^p\left(\chi_{k+H_p-1|k}^p\right) \right\}. \tag*{(9)}

subject to the dynamics, disturbance recursion, initialization, and state and input constraints in Eq. (2). The first optimal input defines the receding-horizon policy π(n)(sk):=uk∣k⋆,(n)\pi^{(n)}(s_k) := u_{k|k}^{\star,(n)}. After applying this input, the optimization is repeated at the next sampling instant. The terminal weight μterm>0\mu_{\mathrm{term}} > 0 adjusts the contribution of the learned value function, with μterm=1\mu_{\mathrm{term}} = 1 corresponding to its unscaled contribution. Setting the terminal approximation to zero recovers the baseline MPC policy π(0)\pi^{(0)}.

The stage cost penalizes tracking errors and input changes:

ℓ(sk,uk)=∥yk−rk∥μy2+∥uk−uk−1∥μu2,(10)\ell(s_k,u_k)=\lVert y_k-r_k\rVert_{\mu_y}^2+\lVert u_k-u_{k-1}\rVert_{\mu_u}^2, \tag*{(10)}

where μy,μu≥0\mu_y,\mu_u \ge0 are the tracking and input-move weights.

OAPI Procedure and Policy Assessment

At each optimistic approximate policy iteration (OAPI), the updated critic defines a new MPC policy, which is evaluated through closed-loop rollouts. For tracking applications, policy performance is measured by the mean rollout sum of squared errors (SSE):

Sn=Score⁡(π(n)):=1R∑r=1R∑k=0T−1∥yk(r)−rk∥22,(11)S_n=\operatorname{Score}(\pi^{(n)}):=\frac{1}{R}\sum_{r=1}^{R}\sum_{k=0}^{T-1}\lVert y_k^{(r)}-r_k\rVert_2^2, \tag*{(11)}

where RR is the number of rollouts collected under each fixed policy, and TT is the number of sampling intervals per rollout. Unlike the MPC objective, this evaluation metric is undiscounted and excludes input-move penalties, providing a direct measure of tracking performance. The same rollouts provide both the training batch for the next iteration and the score used for policy screening.

Pseudocode 1 summarizes the OAPI procedure. The routine Collect(π\pi) returns the postdecision training dataset and mean SSE obtained under policy π\pi, following the rollout protocol in Appendix § 10. At each iteration, the candidate policy is compared with the best accepted policy. A candidate is rejected if its mean SSE increases; otherwise, it is accepted, and iteration terminates when the percentage improvement falls below δPI\delta_{\mathrm{PI}} or the maximum number of iterations Nmax⁡N_{\max} is reached.

Experiments

Van de Vusse CSTR

We evaluate GP–RL–MPC on a Van de Vusse continuous stirred-tank reactor (CSTR), a classical process systems engineering (PSE) benchmark [11]. The reaction network consists of consecutive reactions A→B→CA \to B \to C, which produce and subsequently consume the desired intermediate BB, and a parallel reaction 2A→D2A \to D, which consumes reactant AA. Within each sampling interval, the plant dynamics are

x˙1=−k1x1−k3x12+(x1f−x1)uk,(12)\dot{x}_1=-k_1x_1-k_3x_1^2+(x_{1f}-x_1)u_k, \tag*{(12)}
x˙2=k1x1−k2x2−x2uk+ωkx2,(13)\dot{x}_2=k_1x_1-k_2x_2-x_2u_k+\omega_k^{x_2}, \tag*{(13)}
x˙1f=ωkx1f,(14)\dot{x}_{1f}=\omega_k^{x_{1f}}, \tag*{(14)}

where x1x_1 and x2x_2 denote the concentrations of AA and BB, respectively, x1fx_{1f} is the feed concentration, and uku_k is the dilution rate. The kinetic coefficients are k1k_1, k2k_2, and k3k_3. The stochastic forcing terms

Algorithm 1: OAPI for GP–RL–MPC with postdecision state-value learning

Input: Baseline π(0)\pi^{(0)}, shared rollout settings; maximum PI iterations Nmax⁡N_{\max}; improvement threshold δPI\delta_{\mathrm{PI}}; MPC and GPR settings.

Output: Selected controller πbest\pi_{\mathrm{best}}.

(D0p,S0)←Collect⁡(π(0))(\mathcal{D}^{p}_{0}, S_{0}) \leftarrow\operatorname{Collect}(\pi^{(0)})

n⋆←0n^{\star} \leftarrow0

for n=1,…,Nmax⁡n = 1,\ldots,N_{\max} do

&nbsp;&nbsp;&nbsp;&nbsp;if Sn⋆=0S_{n^{\star}} = 0 then break

&nbsp;&nbsp;&nbsp;&nbsp;Construct targets from Dn−1p\mathcal{D}^{p}_{n-1}: raw returns at n=1n = 1; otherwise Eq. (7)

&nbsp;&nbsp;&nbsp;&nbsp;Split data using target-quantile stratification

&nbsp;&nbsp;&nbsp;&nbsp;Standardize using scalers fitted on the initial training subset and held fixed thereafter

&nbsp;&nbsp;&nbsp;&nbsp;Fit θ(n)\theta^{(n)} by maximizing the exact GP marginal likelihood, warm-starting for n≥2n \ge2

&nbsp;&nbsp;&nbsp;&nbsp;Select PC inducing points and construct the DTC posterior using Eqs. (29) and (30)

&nbsp;&nbsp;&nbsp;&nbsp;Form Vθ(n)pV^{p}_{\theta^{(n)}} and π(n)\pi^{(n)} using Eqs. (8) and (9)

&nbsp;&nbsp;&nbsp;&nbsp;(Dnp,Sn)←Collect⁡(π(n))(\mathcal{D}^{p}_{n}, S_{n}) \leftarrow\operatorname{Collect}(\pi^{(n)})

&nbsp;&nbsp;&nbsp;&nbsp;Store the candidate policy, critic, and score

&nbsp;&nbsp;&nbsp;&nbsp;Δn←100(Sn⋆−Sn)/Sn⋆\Delta_{n} \leftarrow100(S_{n^{\star}} - S_{n})/S_{n^{\star}}

&nbsp;&nbsp;&nbsp;&nbsp;if Δn<0\Delta_{n} < 0 then

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Reject π(n)\pi^{(n)} and retain π(n⋆)\pi^{(n^{\star})} and its critic

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;break

&nbsp;&nbsp;&nbsp;&nbsp;else

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Accept candidate π(n)\pi^{(n)} and its critic; set n⋆←nn^{\star} \leftarrow n

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if Δn<δPI\Delta_{n} < \delta_{\mathrm{PI}} then break

&nbsp;&nbsp;&nbsp;&nbsp;end

end

return πbest=π(n⋆)\pi_{\mathrm{best}} = \pi^{(n^{\star})}

satisfy ωkx2∼N(0,σx22)\omega^{x_{2}}_{k} \sim\mathcal{N}(0,\sigma_{x_{2}}^{2}) and ωkx1f∼N(0,σx1f2)\omega^{x_{1f}}_{k} \sim\mathcal{N}(0,\sigma_{x_{1f}}^{2}), independently across channels and sampling intervals. Each disturbance is sampled once per interval and applied as a constant forcing rate under a zero-order hold (ZOH). Consequently, the feed concentration evolves as a random walk, while nominal MPC predictions assume zero future innovations. The plant parameters, controller settings, and operating conditions are summarized in Table 2 of Appendix § 11.

ParameterValueDescription
Plant parameters
k1k_150.0Rate constant for A→BA \to B [h−1\mathrm{h}^{-1}]
k2k_2100.0Rate constant for B→CB \to C [h−1\mathrm{h}^{-1}]
k3k_310.0Rate constant for 2A→D2A \to D [L mol−1 h−1\mathrm{L}\,\mathrm{mol}^{-1}\,\mathrm{h}^{-1}]
x1f,nomx_{1f,\mathrm{nom}}10.0Nominal feed concentration [mol L−1\mathrm{mol}\,\mathrm{L}^{-1}]
σx2\sigma_{x_2}2.0Product-concentration forcing-rate std. [mol L−1 h−1\mathrm{mol}\,\mathrm{L}^{-1}\,\mathrm{h}^{-1}]
σx1f\sigma_{x_{1f}}10.0Feed-concentration forcing-rate std. [mol L−1 h−1\mathrm{mol}\,\mathrm{L}^{-1}\,\mathrm{h}^{-1}]
Initial conditions and constraints
x1,0x_{1,0}2.15Initial concentration of AA [mol L−1\mathrm{mol}\,\mathrm{L}^{-1}]
x2,0x_{2,0}0.90Initial concentration of BB [mol L−1\mathrm{mol}\,\mathrm{L}^{-1}]
x1f,0x_{1f,0}10.00Initial feed concentration [mol L−1\mathrm{mol}\,\mathrm{L}^{-1}]
u−1u_{-1}19.63Input preceding the first control move [h−1\mathrm{h}^{-1}]
x1,min⁡x_{1,\min}0.0Lower bound on x1x_1 [mol L−1\mathrm{mol}\,\mathrm{L}^{-1}]
x1,max⁡x_{1,\max}7.0Upper bound on x1x_1 [mol L−1\mathrm{mol}\,\mathrm{L}^{-1}]
x2,min⁡x_{2,\min}0.0Lower bound on x2x_2 [mol L−1\mathrm{mol}\,\mathrm{L}^{-1}]
x2,max⁡x_{2,\max}2.4Upper bound on x2x_2 [mol L−1\mathrm{mol}\,\mathrm{L}^{-1}]
umin⁡u_{\min}0.0Lower dilution-rate bound [h−1\mathrm{h}^{-1}]
umax⁡u_{\max}60.0Upper dilution-rate bound [h−1\mathrm{h}^{-1}]
Controller and simulation parameters
HpH_p3Prediction horizon [steps]
TT1000Rollout length [sampling steps]
TsT_s0.005Sampling interval [h]
TintT_{\mathrm{int}}1.0Reference update interval [h]
γ\gamma0.98Discount factor
(μy,μu,μterm)(\mu_y,\mu_u,\mu_{\mathrm{term}})(102,1,1)(10^2,1,1)Tracking, input-move, and terminal weights
ζ\zeta0.99Effective-horizon return coverage factor
rmin⁡r_{\min}0.7Lower bound on reference [mol L−1\mathrm{mol}\,\mathrm{L}^{-1}]
rmax⁡r_{\max}0.9Upper bound on reference [mol L−1\mathrm{mol}\,\mathrm{L}^{-1}]
r0r_00.9Initial reference value [mol L−1\mathrm{mol}\,\mathrm{L}^{-1}]
RR5Number of rollouts per policy
Mmax⁡M_{\max}80Maximum number of PC-selected inducing points

Table 2. Plant, controller, and operating settings for the CSTR experiments.

The controlled output is the concentration of the desired product, yk=x2,ky_k = x_{2,k}, with reference rkr_k and stage cost defined by Eq. (10). The feed concentration is included in the information state as dk=x1f,kd_k = x_{1f,k}. The nominal postdecision state feature vector is

χkp:=φ(skp)=[xˉ1f,k+1∣krk+1∣kxˉ1,k+1∣kxˉ2,k+1∣k−rk+1∣k],(15)\chi_k^p := \varphi(s_k^p) = \begin{bmatrix} \bar{x}_{1f,k+1|k}\\ r_{k+1|k}\\ \bar{x}_{1,k+1|k}\\ \bar{x}_{2,k+1|k} - r_{k+1|k} \end{bmatrix}, \tag*{(15)}

where overbars denote nominal endpoints obtained from the current state and applied input with zero disturbance innovations. The GP employs a quadratic mean function, mβm_{\beta}, in the tracking-error coordinate, together with an automatic-relevance-determination (ARD) Matérn kernel defined over all four features.

The experiments compare GP–RL–MPC with baseline MPC, assess tracking improvements across accepted policy iterations, and examine how the risk coefficient κ\kappa affects policy-iteration performance.

Baseline Comparison and Policy Iteration Performance

We evaluate the final incumbent GP–RL–MPC controllers with κ=3.0\kappa= 3.0 against a baseline MPC controller with zero terminal cost. Both use the same prediction horizon, transition model, stage cost, and constraints. Initial conditions, reference schedules, and disturbance realizations are matched across controllers and PI rounds, and each SSE score is averaged over R=5R = 5 rollouts. Table 1 summarizes the final incumbent performance after five to eight accepted PI rounds, during which SSE decreases monotonically. The reductions relative to baseline MPC, calculated as 100(1−Sn⋆/S0)100(1 - S_{n^\star}/S_0), range from 77.07% to 85.63% with a mean of 80.52% and a sample standard deviation of 3.48%. The consistently large improvements across all five seeds demonstrate the effectiveness of the learned terminal cost in compensating for the limited prediction horizon. These results also provide empirical support for reliable policy improvement within the OAPI framework. The corresponding closed-loop tracking responses are shown in Figure 3 in Appendix F.

SeedAccepted PIMPC SSE S0S_0Final SSE Sn⋆S_{n^\star}SSE Reduction (%)
173.200.4685.63
252.300.4082.43
362.180.4778.38
481.810.4277.07
552.490.5279.07
Mean ±\pm SD———80.52 ±\pm 3.48

Table 1. Final incumbent GP–RL–MPC performance with κ=3.0\kappa= 3.0.

Multi-seed closed-loop tracking responses

Figure 3. Multi-seed closed-loop tracking responses: (a)–(e): Baseline MPC; (f)–(j): Final incumbent GP–RL–MPC controllers with κ=3.0\kappa= 3.0. Each panel overlays RR rollouts.

Effect of the Risk Coefficient

To investigate the role of epistemic uncertainty in terminal-cost optimization, we compare three risk coefficients, κ∈{−3.0,0.0,3.0}\kappa\in\{-3.0, 0.0, 3.0\}. These correspond to an optimistic lower confidence bound, the posterior mean, and a conservative upper confidence bound, respectively, with κ=±3.0\kappa= \pm3.0 representing the bounds of a nominal 99.7% pointwise Gaussian credible interval. Figure 2(a)–(Figure 2e) shows the resulting mean rollout SSE trajectories across policy iterations for the five seed configurations.

Multi-seed analysis of the risk coefficient

Figure 2. Multi-seed analysis of the risk coefficient: (a)–(e): Mean rollout SSE across policy iterations. (f): Normalized SSE at the end of each initial non-increasing segment.

The screening-enabled configuration with κ=3.0\kappa= 3.0 achieves sustained, monotonically non-increasing SSE across accepted PI rounds. In contrast, the κ=0\kappa= 0 and κ=−3.0\kappa= -3.0 configurations exhibit non-monotonic behavior, with generally larger fluctuations under κ=−3.0\kappa= -3.0. These latter two configurations are evaluated as diagnostic runs without screening, each continued for the same number of PI rounds as its corresponding κ=3.0\kappa= 3.0 run. Although mean-based optimization (κ=0.0\kappa= 0.0) and optimistic optimization (κ=−3.0\kappa= -3.0) occasionally achieve lower SSE, these improvements are neither predictable nor consistently sustained. For example, in Figure 2(c), κ=0\kappa= 0 achieves an SSE of 0.39 at PI #6, compared with 0.47 for κ=3.0\kappa= 3.0, but only after several intermediate performance reversals. Such behavior reflects the sensitivity of policy improvement to the accuracy of the learned critic, which depends on the disturbance realizations and the coverage of the relevant state-action space by the GP training data. In particular, optimistic terminal-cost estimates may favor decisions in poorly sampled regions where critic predictions are less reliable.

To compare the consistency of improvement across risk coefficients, Figure 2(f) summarizes the initial non-increasing segments of the SSE trajectories, starting at PI #1. The score at the end of each segment is normalized by its corresponding baseline score S0S_0. The resulting means and sample standard deviations of normalized SSE are 0.19±0.030.19 \pm0.03, 0.54±0.370.54 \pm0.37, and 0.71±0.280.71 \pm0.28 for κ=3.0\kappa= 3.0, 0.00.0, and −3.0-3.0, respectively. Under this segment-based comparison, κ=3.0\kappa= 3.0 achieves the lowest mean normalized SSE and the smallest variability across seeds.

Conclusion

We developed GP–RL–MPC, an uncertainty-aware framework integrating uncertainty propagation, calibration, and regulation to improve short-horizon MPC performance while preserving its computational advantages. For uncertainty propagation, the GP postdecision state-value critic approximates expected cost-to-go under aleatoric uncertainty from disturbed MC rollouts aligned with nominal terminal queries. This formulation shifts disturbance averaging offline and avoids explicit online expectation calculations. For uncertainty calibration, exact marginal-likelihood hyperparameter fitting is followed by PC-DTC sparse deployment, reducing kernel redundancy and online evaluation costs while maintaining reliable uncertainty estimates. For uncertainty regulation, the UCB-based terminal cost uses the latent posterior standard deviation to provide tunable regulation of critic epistemic uncertainty in control decisions. Numerical results provide empirical support for consistent and sustained policy improvement under the OAPI procedure. Future work will extend GP–RL–MPC to economic optimization and control.

Generative AI Use

Generative AI tools were used to assist with writing. The authors reviewed and verified the AI-assisted content and take responsibility for the accuracy and integrity of the submitted work.

Soft-Constraint Violation Penalties

As an illustrative example, consider softening only the upper product-concentration bound x2,max⁡s=2.4 mol L−1x^{s}_{2,\max}=2.4\ \mathrm{mol}\ \mathrm{L}^{-1}. For l=0,…,Hp−1l=0,\ldots,H_p-1, replace the corresponding hard upper bound in X\mathcal{X} with

x2,k+l∣k≤x2,max⁡s+ξk+l∣k,0≤ξk+l∣k≤ξmax⁡,(16)x_{2,k+l\mid k}\le x^{s}_{2,\max}+\xi_{k+l\mid k}, \qquad 0\le\xi_{k+l\mid k}\le\xi_{\max}, \tag*{(16)}

where ξmax⁡>0\xi_{\max}>0 limits the admissible violation. This retains the hard outer bound x2,k+l∣k≤x2,max⁡s+ξmax⁡x_{2,k+l\mid k}\le x^{s}_{2,\max}+\xi_{\max}. All other state and input bounds remain unchanged.

Let [a]+:=max⁡(a,0)[a]_{+}:=\max(a,0) and define

pc(x):=ρc[x2−x2,max⁡s]+,ρc>0,p_{\mathrm{c}}(x):=\rho_{\mathrm{c}}[x_{2}-x^{s}_{2,\max}]_{+}, \qquad\rho_{\mathrm{c}}>0,
ℓc(s,u):=ℓ(s,u)+pc(x),(17)\ell_{\mathrm{c}}(s,u):=\ell(s,u)+p_{\mathrm{c}}(x), \tag*{(17)}

where ℓ\ell is the tracking and input-move cost in Eq. (10). When the slack variable appears only in Eq. (16) and its linear penalty, the minimizing value is ξk+l∣k=[x2,k+l∣k−x2,max⁡s]+\xi_{k+l\mid k}=[x_{2,k+l\mid k}-x^{s}_{2,\max}]_{+}. Thus, within the admissible outer bound, the slack formulation is equivalent to using ℓc\ell_{\mathrm{c}}.

The resulting GP–RL–MPC objective becomes

JHp,cGP(sk;θ):=min⁡{uk+l∣k}l=0Hp−1{∑l=0Hp−1γl[ℓ(sk+l∣k,uk+l∣k)+ρcξk+l∣k]+μtermγHpVθp(xk+Hp−1∣kp)},(18)J^{\mathrm{GP}}_{H_p,\mathrm{c}}(s_k;\theta):= \min_{\{u_{k+l\mid k}\}_{l=0}^{H_p-1}} \left\{ \sum_{l=0}^{H_p-1}\gamma^l \left[\ell(s_{k+l\mid k},u_{k+l\mid k})+\rho_{\mathrm{c}}\xi_{k+l\mid k}\right] +\mu_{\mathrm{term}}\gamma^{H_p}V^{p}_{\theta}(x^{p}_{k+H_p-1\mid k}) \right\}, \tag*{(18)}

subject to the nominal dynamics, initialization, remaining hard bounds, and the soft constraints in Eq. (16). The horizon penalties are evaluated on nominal predictions.

For critic learning, the same augmented stage cost is evaluated along the disturbed closed-loop rollout:

Gk+1c:=∑j=k+1T−1γj−k−1ℓc(sj,uj),G^{\mathrm{c}}_{k+1}:=\sum_{j=k+1}^{T-1}\gamma^{j-k-1}\ell_{\mathrm{c}}(s_j,u_j),
Gjc=ℓc(sj,uj)+γGj+1c,GTc=0.(19)G^{\mathrm{c}}_j=\ell_{\mathrm{c}}(s_j,u_j)+\gamma G^{\mathrm{c}}_{j+1}, \qquad G^{\mathrm{c}}_T=0. \tag*{(19)}

Following the postdecision state-value formulation, each nominal postdecision state feature χkp\chi^p_k is paired with the disturbed return Gk+1cG^{\mathrm{c}}_{k+1}, preserving the indexing and truncation conventions in Appendix D. Violation penalties are evaluated using the full realized constraint violation, even when it exceeds ξmax⁡\xi_{\max}. When this extension is enabled, baseline and learned policies use the same penalized cost ℓc\ell_{\mathrm{c}}, and GicG^{\mathrm{c}}_i replaces GiG_i in the MC target construction and value-space soft updates.

Because the learned postdecision value incorporates the augmented stage costs from k+Hpk+H_p onward, including constraint-violation penalties, no additional terminal violation penalty is required. Any hard terminal set Xf\mathcal{X}_f, if imposed, remains a separate constraint and is not enforced by the learned terminal value. Moreover, the soft constraints and violation penalties imposed on nominal predictions do not guarantee constraint satisfaction for the disturbed plant.

Gaussian Process Regression and PC-DTC Deployment

This appendix summarizes exact Gaussian process regression (GPR) and the PC-DTC approximation deployed in Section 3.2. All inputs and regression targets below are expressed in standardized coordinates, and policy-iteration indices are suppressed unless explicitly shown.

Exact Gaussian Process Regression

Let χ∈Rnχ\chi\in\mathbb{R}^{n_\chi} denote an input and y∈Ry \in\mathbb{R} the associated regression target. A GP prior on the latent function vv is

v∼GP(mβ,k),(20)v \sim\mathcal{GP}(m_\beta,k), \tag*{(20)}

where mβm_\beta is the parameterized mean function and kk is the covariance kernel. With Gaussian likelihood noise of variance σn2>0\sigma_n^2 > 0, the observation model is

yi=v(χi)+νi,νi∼i.i.d.N(0,σn2).(21)y_i = v(\chi_i) + \nu_i,\qquad\nu_i \mathrel{\overset{\mathrm{i.i.d.}}{\sim}} \mathcal{N}(0,\sigma_n^2). \tag*{(21)}

For NN training pairs, write X=(χ1,…,χN)X = (\chi_1,\ldots,\chi_N) and define

y=(y1,…,yN)T,\mathbf{y} = (y_1,\ldots,y_N)^{\mathsf{T}},
m=(mβ(χ1),…,mβ(χN))T,\mathbf{m} = (m_\beta(\chi_1),\ldots,m_\beta(\chi_N))^{\mathsf{T}},
[K]ij=k(χi,χj),Ky=K+σn2I.(22)[K]_{ij} = k(\chi_i,\chi_j),\qquad K_y = K + \sigma_n^2 I. \tag*{(22)}

For a query χ∗\chi_\ast, let k(X,χ∗)=(k(χ1,χ∗),…,k(χN,χ∗))T∈RN×1k(X,\chi_\ast) = (k(\chi_1,\chi_\ast),\ldots,k(\chi_N,\chi_\ast))^{\mathsf{T}} \in\mathbb{R}^{N \times1}. Conditional on the fitted parameters, the exact latent posterior mean and variance are Rasmussen and Williams, [17]

μGP(χ∗)=mβ(χ∗)+k(X,χ∗)TKy−1(y−m),(23)\mu_{\mathrm{GP}}(\chi_\ast) = m_\beta(\chi_\ast) + k(X,\chi_\ast)^{\mathsf{T}}K_y^{-1}(\mathbf{y}-\mathbf{m}), \tag*{(23)}
σGP2(χ∗)=k(χ∗,χ∗)−k(X,χ∗)TKy−1k(X,χ∗).(24)\sigma_{\mathrm{GP}}^2(\chi_\ast) = k(\chi_\ast,\chi_\ast) - k(X,\chi_\ast)^{\mathsf{T}}K_y^{-1}k(X,\chi_\ast). \tag*{(24)}

We use an automatic-relevance-determination (ARD) Matérn-3/2 kernel,

k(χ,χ′)=σf2(1+3a(χ,χ′))exp⁡(−3a(χ,χ′)),(25)k(\chi,\chi^\prime) = \sigma_f^2(1+\sqrt{3}a(\chi,\chi^\prime))\exp(-\sqrt{3}a(\chi,\chi^\prime)), \tag*{(25)}

with scaled distance

a(χ,χ′)=∑j=1nχ(χj−χj′)2λj2.(26)a(\chi,\chi^\prime) = \sqrt{\sum_{j=1}^{n_\chi}\frac{(\chi_j-\chi_j^\prime)^2}{\lambda_j^2}}. \tag*{(26)}

The signal variance σf2>0\sigma_f^2 > 0 sets the covariance scale, while the length scales λj>0\lambda_j > 0 control variation along each input coordinate. Larger length scales correspond to slower variation along the associated coordinates.

The posterior mean approaches mβ(χ∗)m_\beta(\chi_\ast) as the cross-covariances vanish far from the training inputs. The latent posterior variance represents model-based epistemic uncertainty conditional on the fitted GP assumptions and parameters. It excludes observation-noise variance and does not directly quantify stochastic return variability or plant risk.

PC-DTC Deployment

For critic learning, let DN={(χip,yi)}i=1N\mathcal{D}_N = \{(\chi_i^p,y_i)\}_{i=1}^{N} denote the current training subset. The regression targets are constructed as described in § 3.1 and subsequently standardized. The parameter vector θ\theta, comprising the mean function, kernel, and likelihood parameters, is fitted using Adam to maximize the exact GP log marginal likelihood. The PC-DTC approximation is constructed only after fitting, with these parameters held fixed.

For input collections UU and VV, let KUVK_{UV} denote the kernel matrix with entries [KUV]ij=k(Ui,Vj)[K_{UV}]_{ij} = k(U_i,V_j). For a single input χ\chi, let k(Z,χ)k(Z,\chi) denote the column vector of covariances between χ\chi and the inducing points in ZZ. Starting with Z=∅Z = \varnothing, PC sequentially selects the unselected training input having the largest residual conditional prior variance (Harbrecht et al., [5]),

ρi(Z)=k(χip,χip)−k(Z,χip)TKZZ−1k(Z,χip),(27)\rho_i(Z) = k(\chi_i^p,\chi_i^p) - k(Z,\chi_i^p)^\mathsf{T}K_{ZZ}^{-1}k(Z,\chi_i^p), \tag*{(27)}

where the second term is defined as zero when Z=∅Z = \varnothing. The residual ρi(Z)\rho_i(Z) measures the prior variance at χip\chi_i^p unexplained by the current inducing set. Selection terminates when the inducing-point budget is reached, the total residual variance falls below a prescribed fraction of its initial value, or the maximum residual variance becomes numerically negligible. This yields an inducing set of size M≤Mmax⁡M \le M_{\max}, typically with M≪NM \ll N.

DTC uses the selected inducing points to approximate the training covariance while incorporating information from all training pairs (Quinonero-Candela and Rasmussen, [16]). Write X=(χ1p,…,χNp)X = (\chi_1^p,\ldots,\chi_N^p), y=(y1,…,yN)Ty = (y_1,\ldots,y_N)^\mathsf{T}, and mβ(X)=(mβ(χ1p),…,mβ(χNp))Tm_\beta(X) = (m_\beta(\chi_1^p),\ldots,m_\beta(\chi_N^p))^\mathsf{T}. Define

B=KZZ+δjitI,A=KZXKXZ+σn2B,(28)B = K_{ZZ} + \delta_{\mathrm{jit}}I,\qquad A = K_{ZX}K_{XZ} + \sigma_n^2B, \tag*{(28)}

where δjit>0\delta_{\mathrm{jit}} > 0 provides numerical regularization for the Cholesky factorization. Standard DTC is recovered as δjit→0\delta_{\mathrm{jit}} \to0. For Z={z1,…,zM}Z = \{z_1,\ldots,z_M\} and a query χ∗\chi_*, the inducing-point covariance vector is k(Z,χ∗):=(k(z1,χ∗),…,k(zM,χ∗))T∈RM×1k(Z,\chi_*) := (k(z_1,\chi_*),\ldots,k(z_M,\chi_*))^\mathsf{T} \in\mathbb{R}^{M \times1}. The resulting predictive mean and latent variance are

μDTC(χ∗)=mβ(χ∗)+k(Z,χ∗)TA−1KZX(y−mβ(X)),(29)\mu_{\mathrm{DTC}}(\chi_*) = m_\beta(\chi_*) + k(Z,\chi_*)^\mathsf{T}A^{-1}K_{ZX}(y-m_\beta(X)), \tag*{(29)}
σDTC2(χ∗)=k(χ∗,χ∗)−k(Z,χ∗)TB−1k(Z,χ∗)+σn2k(Z,χ∗)TA−1k(Z,χ∗).(30)\sigma_{\mathrm{DTC}}^2(\chi_*) = k(\chi_*,\chi_*) - k(Z,\chi_*)^\mathsf{T}B^{-1}k(Z,\chi_*) + \sigma_n^2 k(Z,\chi_*)^\mathsf{T}A^{-1}k(Z,\chi_*). \tag*{(30)}

The variance combines a regularized residual conditional variance, represented by the first two terms, with posterior uncertainty in the inducing variables, represented by the third term. No observation-noise variance is added to the latent prediction.

The deployed mean μGPp,(n)\mu_{\mathrm{GP}}^{p,(n)} and latent posterior standard deviation σGPp,(n)\sigma_{\mathrm{GP}}^{p,(n)} are obtained by transforming μDTC\mu_{\mathrm{DTC}} and the square root of σDTC2\sigma_{\mathrm{DTC}}^2 back to the original cost units. These deployed moments replace the exact posterior moments in Eqs. (23) and (24) when constructing the terminal penalty in Eq. (8).

Predecision and Postdecision State-Value Functions

Let s∈Ss \in\mathcal{S} denote the augmented Markov state and ℓ(s,u)\ell(s,u) the stage cost. For a fixed stationary policy π:S→U\pi:\mathcal{S} \to\mathcal{U}, the standard (predecision) discounted state-value function is defined as (Sutton and Barto, [19])

Vπ(s):=Eπ[∑j=0∞γjℓ(sj,π(sj))∣s0=s],(31)V_\pi(s) := \mathbb{E}_\pi\left[\left.\sum_{j=0}^{\infty}\gamma^j\ell(s_j,\pi(s_j))\right|s_0=s\right], \tag*{(31)}

assuming finite expected discounted cost. MC value fitting estimates this expectation conditional on the current state. Nevertheless, evaluating it at a nominal terminal state neglects the uncertainty in the stochastic transition leading to that state. Accounting for this uncertainty requires an additional expectation over possible terminal states, which can be approximated through explicit uncertainty propagation [6].

To clarify the distinction, consider the final transition from k+Hp−1k + H_p - 1 to k+Hpk + H_p, with all preceding horizon predictions treated nominally. Under the sufficiency condition in Eq. (4), Eqs. (3) and (5) give

μtermγHpE[Vπ(Fw(sk+Hp−1∣kp,wk+Hp−1))|sk+Hp−1∣kp]=μtermγHpVπp(sk+Hp−1∣kp).(32)\mu_{\mathrm{term}} \gamma^{H_p} \mathbb{E}\left[V_{\pi}\left(F_{\mathrm{w}}\left(s^{p}_{k+H_p-1|k}, w_{k+H_p-1}\right)\right)\middle|s^{p}_{k+H_p-1|k}\right] = \mu_{\mathrm{term}} \gamma^{H_p} V_{\pi}^{p}\left(s^{p}_{k+H_p-1|k}\right). \tag*{(32)}

Here, VπV_{\pi} accounts for costs from k+Hpk + H_p onward conditional on the realized successor state, while the outer expectation averages over the final innovation that generates that state. The postdecision state-value function VπpV_{\pi}^{p} incorporates both sources of stochastic expectation, allowing direct nominal terminal evaluation without an additional online expectation. In contrast, evaluating VπV_{\pi} at the nominal predecision terminal state sˉk+Hp∣k\bar{s}_{k+H_p|k} generally neglects the expectation over the final stochastic transition.

The training pairs in Eq. (33) associate nominal postdecision state features with MC returns beginning at the corresponding disturbed successor states. Regression therefore approximates the final-transition expectation together with the subsequent expected continuation cost. With the implemented features and finite-rollout targets, this is an approximation to Eq. (32), with target damping applied through Eq. (7). The GP posterior mean estimates the expected continuation cost, while the uncertainty-regulated terminal function in Eq. (8) additionally accounts for critic epistemic uncertainty. Predictions within the MPC horizon remain certainty-equivalent (CE), as specified in Eq. (9).

In practical implementation, training rollouts can be generated using a simulator or world model to reduce experimental costs. Plant data can then be used for model calibration, validation, and subsequent refinement. The learned critic approximates expected returns under the simulated dynamics and disturbance distribution. Increasing the number of MC rollouts reduces sampling error but cannot eliminate bias arising from model-plant mismatch. Reliable deployment consequently requires validation over the intended operating region and, when necessary, iterative refinement of the dynamics and disturbance models using plant data. Importantly, the GP posterior uncertainty characterizes uncertainty in the learned critic conditional on the assumed model and training data; it does not directly quantify simulator-plant discrepancy. Robustness to such mismatch must therefore be assessed separately.

Postdecision State Data, Targets, and Terminal Evaluation

Fix a rollout-generating policy π\pi and collect RR disturbed rollouts, each containing TT stage costs. Initial states, previous inputs, reference schedules, and controller and estimator initialization are matched across rollouts and policies. Disturbance seed pairs differ across rollouts and are reused across policies. For clarity, PI and rollout superscripts are omitted, and returns are computed separately within each rollout.

In the implementation, we define sk+l∣kp:=sˉk+l∣ks^{p}_{k+l|k} := \bar{s}_{k+l|k} within prediction horizon HpH_p, where the overbar denotes a nominal prediction based on the predecision states, with disturbance innovations set to zero. Thus, sk+l∣kps^{p}_{k+l|k} is indexed by the generating action at k+lk + l and represents the nominal endpoint at k+l+1k + l + 1. Let φ\varphi denote the critic feature map and define χkp=φ(sk∣kp)=φ(sˉk+1∣k)\chi^{p}_{k} = \varphi(s^{p}_{k|k}) = \varphi(\bar{s}_{k+1|k}). Pair this nominal feature with the cost-to-go from the actual disturbed trajectory:

(χkp,Gk+1),Gk+1:=∑j=k+1T−1γj−k−1ℓ(sj,uj).(33)(\chi_k^p, G_{k+1}), \qquad G_{k+1} := \sum_{j=k+1}^{T-1} \gamma^{j-k-1}\ell(s_j,u_j). \tag*{(33)}

Returns are computed backward using

GT=0,Gj=ℓ(sj,uj)+γGj+1.(34)G_T = 0, \qquad G_j = \ell(s_j,u_j) + \gamma G_{j+1}. \tag*{(34)}

Each target excludes the current stage cost ℓ(sk,uk)\ell(s_k,u_k) and contains no additional leading discount. The first training pair is (χ0p,G1)(\chi_0^p,G_1), so G0G_0 is not used as a target and the initial stage cost ℓ(s0,u0)\ell(s_0,u_0) is excluded from the continuation targets. The boundary pair (χT−1p,GT)(\chi_{T-1}^p,G_T) is also omitted because its zero target is imposed by finite-rollout truncation. To reduce truncation bias, discard the final NeffN_{\mathrm{eff}} remaining pairs within each rollout. For 0<γ<10 < \gamma< 1, define

Neff=⌈log⁡(1−ζ)log⁡γ⌉,0<ζ<1.(35)N_{\mathrm{eff}} = \left\lceil\frac{\log(1-\zeta)}{\log\gamma} \right\rceil, \qquad0 < \zeta< 1. \tag*{(35)}

Here, ζ\zeta specifies discount-weight coverage, with γNeff≤1−ζ\gamma^{N_{\mathrm{eff}}} \le1-\zeta. The retained training pairs correspond to action indices from k=0k=0 through k=T−2−Neffk=T-2-N_{\mathrm{eff}} within each rollout. For T>Neff+1T>N_{\mathrm{eff}}+1, concatenate the retained pairs from all rollouts and reindex them as

Dp={(χip,Gi)}i=1Npool,Npool=R(T−1−Neff).(36)\mathcal{D}^p = \left\{(\chi_i^p,G_i)\right\}_{i=1}^{N_{\mathrm{pool}}}, \qquad N_{\mathrm{pool}} = R(T-1-N_{\mathrm{eff}}). \tag*{(36)}

Here, ii indexes pooled pairs rather than time, each GiG_i denotes the corresponding continuation return Gk+1G_{k+1} from its source rollout, and repeated samples are retained. Each critic fit uses the current policy-specific batch. The dataset stores raw returns; soft targets and parameter warm starts are applied as described in § 3.1.

Using the uncertainty-regulated terminal estimate VθpV_\theta^p defined in Eq. (8), the terminal query and its objective contribution are

χk+Hp−1∣kp=φ(sˉk+Hp∣k),μtermγHpVθp(χk+Hp−1∣kp).(37)\chi_{k+H_p-1|k}^p = \varphi(\bar{s}_{k+H_p|k}), \qquad\mu_{\mathrm{term}}\gamma^{H_p}V_\theta^p(\chi_{k+H_p-1|k}^p). \tag*{(37)}

The horizon sum includes stage costs from kk through k+Hp−1k+H_p-1, while the continuation term is aligned with costs beginning at k+Hpk+H_p. GPR can estimate conditional return means without requiring identical feature vectors to recur. The fitted GP mean therefore provides a feature-based approximation under the sampled rollout distribution, with target damping applied during fitting.

Experiment Configurations

Table 2 summarizes the plant, controller, and operating settings. Both controllers are implemented using CasADi with IPOPT. No separate terminal set is imposed beyond the prescribed state bounds. The disturbed plant and nominal one-step predictor use CVODES with absolute and relative tolerances of 10−1010^{-10}. Full augmented-state feedback is assumed without additional measurement noise. The reference is initially 0.9 mol L−1^{-1} and subsequently sampled uniformly from [0.7, 0.9] mol L−1^{-1} at each reference update; its known future values are supplied to controllers. Reference schedules and disturbance seed pairs are reused across policies, with all rollout initializations reset as described in Appendix § 10.

Critic data are split into training and diagnostic subsets in an 80%:20% ratio using ten target-quantile strata. We apply τ=0.1\tau= 0.1 for soft updates, fixed first-round scalers, and parameter warm starts. The mean is mβ(χ)=β2χerr2+β1χerr+β0m_\beta(\chi) = \beta_2\chi_{\mathrm{err}}^2 + \beta_1\chi_{\mathrm{err}} + \beta_0, where χerr\chi_{\mathrm{err}} is the standardized tracking-error feature. Adam uses a learning rate of 10−310^{-3} for at most 5000 steps, stopping after 200 consecutive steps without a training-loss improvement exceeding 10−410^{-4}. PC uses a remaining-trace tolerance of 10−310^{-3} and a residual-variance cutoff of 10−1210^{-12}; DTC uses δjit=10−8\delta_{\mathrm{jit}} = 10^{-8} in standardized coordinates. Screened runs terminate at the first SSE deterioration or upon exhausting the prescribed PI budget; rejected candidates are excluded from the accepted sequence. Diagnostic runs use the matched iteration budgets described in § 4.3.

Closed-Loop Trajectories

Figure 3 compares the five-seed closed-loop trajectories under the matched settings described in § 4.2. GP–RL–MPC tracks the varying setpoints more closely and reduces persistent offsets relative to baseline MPC.

References

  1. [1]Dimitri P. Bertsekas. Reinforcement Learning and Optimal Control. Athena Scientific, Belmont, MA, 2019. ISBN 978-1-886529-39-7.
  2. [2]Dimitri P. Bertsekas. Abstract Dynamic Programming. Athena Scientific, Belmont, MA, 3rd edition, 2022. ISBN 978-1-886529-47-2.
  3. [3]Dimitri P. Bertsekas. Model predictive control and reinforcement learning: A unified framework based on dynamic programming. IFAC-PapersOnLine, 58(18):363–383, 2024. doi: 10.1016/j.ifacol.2024.09.056.DOI
  4. [4]Sébastien Gros and Mario Zanon. Data-driven economic NMPC using reinforcement learning. IEEE Transactions on Automatic Control, 65(2):636–648, 2020. doi: 10.1109/TAC.2019.2913768.DOI
  5. [5]Helmut Harbrecht, Michael Peters, and Reinhold Schneider. On the low-rank approximation by the pivoted Cholesky decomposition. Applied Numerical Mathematics, 62(4):428–440, 2012. doi: 10.1016/j.apnum.2011.10.001.DOI
  6. [6]Daniel Landgraf, Andreas Völz, and Knut Graichen. A software framework for stochastic model predictive control of nonlinear continuous-time systems (GRAMPC-S). Optimization and Engineering, 27(1):193–228, 2026. doi: 10.1007/s11081-025-10006-z.DOI
  7. [7]Jay H. Lee. Model predictive control: Review of the three decades of development. International Journal of Control, Automation and Systems, 9(3):415–424, 2011. doi: 10.1007/s12555-011-0300-6.DOI
  8. [8]Jay H. Lee, Niket S. Kaisare, and Carlos E. García. Model Predictive Control: Fundamentals and Practice. John Wiley & Sons, 2026. ISBN 978-1-394-33329-5. doi: 10.1002/978139433325.DOI
  9. [9]Jong Min Lee, Niket S. Kaisare, and Jay H. Lee. Choice of approximator and design of penalty function for an approximate dynamic programming based control approach. Journal of Process Control, 16(2):135–156, 2006. doi: 10.1016/j.jprocont.2005.04.010.DOI
  10. [10]Jon Li and Jay H. Lee. Deep reinforcement learning integrated NARX-model predictive control. In Proceedings of the 2025 AIChE Annual Meeting, Boston, MA, USA, November 2025. American Institute of Chemical Engineers. ISBN 978-0-8169-1125-7. URL https://proceedings.aiche.org/conferences/aiche-annual-meeting/2025/proceeding/paper/integrating-deep-reinforcement. Paper 109b.
  11. [11]Jon Li and Jay H. Lee. Reinforcement learning-enhanced model predictive control using an iteratively learned PICNN terminal value function. Computers & Chemical Engineering, 216:109879, 2026a. doi: 10.1016/j.compchemeng.2026.109879.DOI
  12. [12]Jon Li and Jay H. Lee. RBF–RL–MPC: Finite localized value functions for reinforcement learning-based model predictive control. Journal of Process Control, 168:103884, 2026b. doi: 10.1016/j.jprocont.2026.103884.DOI
  13. [13]Ali Mesbah. Stochastic model predictive control: An overview and perspectives for future research. IEEE Control Systems, 36(6):30–44, 2016. doi: 10.1109/MCS.2016.2602087.DOI
  14. [14]Francisco Moreno-Mora, Lukas Beckenbach, and Stefan Streif. Predictive control with learning-based terminal costs using approximate value iteration. IFAC-PapersOnLine, 56(2):3874–3879, 2023. doi: 10.1016/j.ifacol.2023.10.1320.
  15. [15]Warren B. Powell. Perspectives of approximate dynamic programming. Annals of Operations Research, 241(1–2):319–356, 2016. doi: 10.1007/s10479-012-1077-6.
  16. [16]Joaquin Quiñonero-Candela and Carl Edward Rasmussen. A unifying view of sparse approximate Gaussian process regression. Journal of Machine Learning Research, 6(65):1939–1959, 2005. URL https://www.jmlr.org/papers/v6/quinonero-candela05a.html.
  17. [17]Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA, 2006. ISBN 978-0-262-18253-9. doi: 10.7551/mitpress/3206.001.0001.DOI
  18. [18]James B. Rawlings, David Q. Mayne, and Moritz M. Diehl. Model Predictive Control: Theory, Computation, and Design. Nob Hill Publishing, Santa Barbara, CA, 2nd edition, 2020. ISBN 978-0-9759377-5-4.
  19. [19]Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 2nd edition, 2018. ISBN 978-0-262-03924-6.
  20. [20]Wee Chin Wong and Jay H. Lee. Postdecision-state-based approximate dynamic programming for robust predictive control of constrained stochastic processes. Industrial & Engineering Chemistry Research, 50(3):1389–1399, 2011. doi: 10.1021/ie1007882.DOI

Paper details

vlnc:W_rjbjjnar
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Generative AI tools

View statement
Looking for related papers…
Contents