WorksComputer Science
GP–RL–MPC: Uncertainty-Aware Reinforcement Learning-Augmented Model Predictive Control with Gaussian Process Postdecision State-Value Functions
Abstract
We propose Gaussian Process Reinforcement Learning-Augmented Model Predictive Control (GP–RL–MPC), which uses a learned critic as the MPC terminal penalty to account for costs beyond the prediction horizon. Gaussian process regression (GPR) enables data-efficient critic learning from limited, noisy rollouts within optimistic approximate policy iteration (OAPI), with soft updates mitigating policy chattering. A tunable risk coefficient weights GP epistemic uncertainty in the terminal cost, regulating sensitivity to uncertain critic predictions. The postdecision state formulation shifts disturbance averaging offline to account for aleatoric uncertainty from the terminal transition onward. Under matching objectives and admissible actions, a one-step horizon recovers greedy value-space RL. After fitting GP hyperparameters by exact marginal likelihood, we construct a sparse deterministic training conditional (DTC) posterior using inducing points selected by pivoted Cholesky (PC), reducing online evaluation costs as the training dataset grows. In a continuous stirred-tank reactor (CSTR) subject to exogenous disturbances, GP–RL–MPC with an appropriately selected risk coefficient and policy screening reduces the sum of squared errors (SSE) by an average of 80.52% relative to an otherwise matched baseline MPC across five random seeds (sample standard deviation: 3.48%).
Introduction
Model predictive control (MPC) handles nonlinear dynamics, multivariable interactions, and operating constraints through repeated finite-horizon optimization (Lee et al., [8]). Computational demands and growing prediction uncertainty limit practical horizon lengths, making terminal costs important for accounting for long-term consequences beyond the prediction horizon. Under appropriate assumptions, suitable terminal ingredients can support recursive feasibility, closed-loop stability, and performance guarantees. However, constructing such terminal ingredients remains challenging, particularly for nonlinear stochastic systems.
Reinforcement learning (RL) offers a complementary approach by learning long-term cost information from closed-loop rollouts to approximate terminal values in MPC (Bertsekas, [3]). Previous studies have explored approximate dynamic programming (ADP) for control (Lee et al., [9]), RL-based parameterization of economic nonlinear MPC (NMPC) (Gros and Zanon, [4]), and terminal-value learning using approximate value iteration (Moreno-Mora et al., [14]), partially input-convex neural networks (PICNNs) (Li and Lee, [11]), and radial basis function (RBF) networks (Li and Lee, [12]). A key challenge in extending these approaches to stochastic systems is incorporating the effects of random state transitions into the learned continuation value without substantially increasing the computational burden of online MPC optimization.
The conventional (predecision) state-value function represents the expected cumulative cost conditioned on the current state, but evaluating it at a nominal terminal state omits averaging over disturbed terminal states. Explicitly computing this additional expectation online adds to the burden of stochastic MPC [13]. A postdecision state captures the known effect of the control action before the stochastic transition, allowing its value function to incorporate the expectation over successor states [19, 15]. Learning this function from disturbed rollouts shifts disturbance averaging offline while preserving deterministic online MPC optimization. A one-step horizon recovers greedy value-space RL, while longer horizons provide multistep lookahead.
Building on the postdecision state formulation, we develop GP–RL–MPC to improve short-horizon MPC performance while preserving its computational advantages. A Gaussian process (GP) critic learns the conditional expected cost-to-go from disturbed rollouts under a fixed policy, thereby incorporating future disturbance effects into postdecision value estimates evaluated at nominal MPC terminal states. The terminal penalty is defined as , where and denote the GP posterior mean and latent posterior standard deviation, respectively. The tunable risk coefficient regulates the influence of critic epistemic uncertainty on control decisions [17].
Critic learning is integrated into an optimistic approximate policy iteration (OAPI) framework, with soft updates mitigating policy chattering [1]. GP hyperparameters are estimated by maximizing the exact marginal likelihood, after which pivoted Cholesky (PC) selects inducing points for a sparse deterministic training conditional (DTC) approximation [5, 16]. This sparse representation makes online posterior mean and variance evaluation costs depend primarily on the inducing-set size rather than the full training-set size, reducing computational costs while retaining estimates of critic epistemic uncertainty.
The main contributions of the proposed uncertainty-aware framework, illustrated in Figure 1, are as follows:

Figure 1. Overview of the proposed GP–RL–MPC algorithm.
Uncertainty propagation: A postdecision state-value critic that learns the expected effects of aleatoric uncertainty on the cost-to-go from disturbed rollouts, enabling nominal MPC to account for future disturbances without explicitly evaluating expectations during online optimization.
Uncertainty calibration: A computationally efficient GP critic that combines exact marginal-likelihood hyperparameter estimation with pivoted Cholesky (PC) inducing-point selection and deterministic training conditional (DTC) sparsification, reducing online computational costs while retaining estimates of epistemic uncertainty.
Uncertainty regulation: A terminal penalty that incorporates GP epistemic uncertainty through a tunable risk coefficient, combined with soft critic updates within optimistic approximate policy iteration, to promote reliable policy improvement while preserving the efficiency of short-horizon MPC.
Preliminaries
Problem Formulation
For nominal prediction, consider the discrete-time nonlinear model
where , , and denote the system state, information state, and manipulated input, respectively. The information state collects references, time-varying parameters, disturbances, and information related to model-plant mismatch. The controlled output is given by , where , assuming no direct input feedthrough. The stochastic components of the information state are modeled as zero-drift random walks, , where is a sequence of independent and identically distributed (i.i.d.) Gaussian innovations with . Known references and exogenous forecasts evolve according to prescribed trajectories.
Given and , or their estimates, the nominal NMPC problem is formulated as [7, 10]
Here, , , and denote the predicted state, information state, and manipulated input, respectively, at steps ahead, as computed at time , and denotes the prediction horizon. Setting future stochastic innovations to zero yields certainty-equivalent (CE) predictions without explicitly propagating disturbance uncertainty. The admissible sets, and , are assumed to be closed and compact, and the stage cost quantifies the tracking or economic performance [18]. While conventional deterministic MPC typically employs an undiscounted objective, we introduce a discount factor to ensure consistency with infinite-horizon value learning under persistent stochastic disturbances, assuming that the expected discounted cumulative cost is finite.
In conventional MPC, a terminal set and penalty can ensure recursive feasibility and closed-loop stability under suitable assumptions but are challenging to construct for nonlinear stochastic systems. We learn the terminal cost from closed-loop rollouts to capture long-term information while retaining a short prediction horizon for computational efficiency. In this study, the proposed controller enforces hard state and input constraints within the prediction horizon. The numerical results shown in § 4 demonstrate that the states of interest remain within their prescribed bounds with substantial safety margins [12]. Selected state constraints can be softened by penalizing bounded violations while retaining hard input constraints and outer state bounds. Using the same augmented stage cost in the MPC objective and Monte Carlo (MC) returns from disturbed rollouts incorporates expected future violation costs into the learned postdecision state-value function. No additional terminal violation penalty is required; Appendix § 7 provides the detailed formulation and its consistency with the value-learning targets.
Postdecision State-Value Functions
A postdecision state separates the known effects of a control action from the subsequent stochastic transition [20]. When the system dynamics admit such a decomposition, the state transition can be expressed as
Here, denotes the augmented predecision state, comprising the plant state , information state , and any additional variables, such as input memory or contextual information, required to satisfy the Markov property. The superscript denotes the postdecision state, and represents the stochastic innovation realized after the control action. For the postdecision state to provide an exact representation, it must contain sufficient information to characterize the conditional distribution of the next predecision state, such that
For a fixed policy , the postdecision state-value function is defined as
where is the conventional predecision state-value function and denotes the Monte Carlo (MC) return starting at time , with subsequent actions determined by . Thus, the postdecision state-value function captures the expected cumulative cost arising from the next stochastic transition and all subsequent disturbances, while excluding the current stage cost. Under the sufficiency condition, the Bellman expectation equation for a fixed deterministic policy becomes
The corresponding one-step policy improvement is obtained by minimizing the right-hand side of Eq. (6) with respect to the current control input. Importantly, because the postdecision state-value function already incorporates the expectation over subsequent stochastic transitions, this minimization does not require explicit disturbance averaging. This property makes the postdecision formulation particularly attractive for integration with nominal MPC, allowing stochastic effects on future costs to be accounted for without increasing the complexity of online optimization. Further details on the construction of postdecision training targets and their consistency with nominal MPC terminal-state queries are provided in Appendix § 10.
Postdecision State-Value Learning as Optimistic Approximate Policy Iteration
Starting from a baseline MPC policy without a terminal penalty, each policy iteration (PI) #n evaluates the preceding policy using disturbed rollouts. The resulting data are used to update the GP critic, whose posterior mean approximates the postdecision state-value function . The GP posterior mean and latent posterior standard deviation are then combined to construct the uncertainty-regulated terminal value estimate , which is incorporated into the constrained MPC optimization to obtain a candidate policy . Thus, the critic learning serves as approximate policy evaluation, while the MPC optimization performs policy improvement using the learned postdecision state-value function Bertsekas [3]. This iterative procedure constitutes optimistic approximate policy iteration (OAPI) Bertsekas [2], in which policy improvement is performed following partial, sample-based policy evaluation rather than waiting for complete convergence of the value function. Such incomplete evaluation can lead to oscillations between successive policies, commonly referred to as policy chattering Li and Lee [11]. The conventional predecision state-value function and its relationship to the postdecision state-value function are reviewed in Appendix § 9.
Proposed GP–RL–MPC Algorithm
The proposed GP–RL–MPC algorithm integrates approximate policy evaluation and policy improvement by iteratively learning a postdecision state-value critic from disturbed closed-loop rollouts and updating the MPC policy through constrained nominal optimization. At PI #n, the preceding policy is held fixed while disturbed rollouts are generated to obtain MC training pairs , where indexes rollouts and denotes the feature representation of the postdecision state used as input to the GP critic. These training pairs are pooled and reindexed to form the dataset for GP regression. The construction of the training pairs and their temporal alignment with the postdecision states are detailed in Appendix § 10. The resulting GP posterior mean and latent posterior standard deviation are used to construct an uncertainty-regulated terminal penalty for the MPC optimization, yielding an updated control policy. Soft critic updates are employed to mitigate policy chattering across successive iterations. The complete GP–RL–MPC procedure is summarized in Algorithm 1.
Soft Updates in Value Space
At the first policy iteration, the GP critic is trained directly on the Monte Carlo returns . From the second iteration onward, soft updates are introduced by constructing the regression targets as
where is the soft-update parameter and is the posterior mean of the preceding GP critic, held fixed during target construction. The update is performed in the original cost units, blending the previous critic prediction with the newly observed MC return. Smaller values of provide stronger damping of critic updates, thereby mitigating abrupt changes in the learned value function and the resulting MPC policy. Setting recovers the raw MC targets, corresponding to a hard update.
The resulting input-target pairs are partitioned into training and diagnostic subsets using target-quantile stratification [12]. Input and output scalers are fitted using the initial training subset and subsequently held fixed to ensure consistent scaling across policy iterations. At each iteration, the GP hyperparameters are initialized from their values in the preceding iteration and refitted using the current training dataset. This procedure combines damped policy evaluation with consistent data preprocessing and warm-started GP training to improve the stability of successive policy updates.
Exact GPR Training and PC-DTC Deployment
We fit the full GP critic offline by maximizing the exact log marginal likelihood. With the fitted hyperparameters fixed, pivoted Cholesky (PC) selects inducing points from the training inputs [5]. A deterministic training conditional (DTC) approximation incorporates information from all training pairs through this inducing representation [16]. The latent uncertainty excludes observation noise and represents critic epistemic uncertainty conditional on the fitted GP model. After precomputing the Cholesky factors and predictive mean coefficients offline, evaluating the mean and variance at one query requires operations, compared with for exact GPR using dense Cholesky factors, where is the input dimension [17]. The regression model, inducing-point selection, and predictive formulas are detailed in Appendix § 8.
Uncertainty-Regulated Terminal Cost
Using the deployed GP posterior moments, we construct an uncertainty-regulated terminal cost based on an upper confidence bound (UCB) acquisition rule:
where controls the influence of epistemic uncertainty. Positive penalizes uncertain value estimates, yielding a more conservative controller; uses the posterior mean alone, and negative favors actions leading to uncertain regions, thereby encouraging exploration. Incorporating this approximation at the postdecision terminal feature gives the GP–RL–MPC objective
subject to the dynamics, disturbance recursion, initialization, and state and input constraints in Eq. (2). The first optimal input defines the receding-horizon policy . After applying this input, the optimization is repeated at the next sampling instant. The terminal weight adjusts the contribution of the learned value function, with corresponding to its unscaled contribution. Setting the terminal approximation to zero recovers the baseline MPC policy .
The stage cost penalizes tracking errors and input changes:
where are the tracking and input-move weights.
OAPI Procedure and Policy Assessment
At each optimistic approximate policy iteration (OAPI), the updated critic defines a new MPC policy, which is evaluated through closed-loop rollouts. For tracking applications, policy performance is measured by the mean rollout sum of squared errors (SSE):
where is the number of rollouts collected under each fixed policy, and is the number of sampling intervals per rollout. Unlike the MPC objective, this evaluation metric is undiscounted and excludes input-move penalties, providing a direct measure of tracking performance. The same rollouts provide both the training batch for the next iteration and the score used for policy screening.
Pseudocode 1 summarizes the OAPI procedure. The routine Collect() returns the postdecision training dataset and mean SSE obtained under policy , following the rollout protocol in Appendix § 10. At each iteration, the candidate policy is compared with the best accepted policy. A candidate is rejected if its mean SSE increases; otherwise, it is accepted, and iteration terminates when the percentage improvement falls below or the maximum number of iterations is reached.
Experiments
Van de Vusse CSTR
We evaluate GP–RL–MPC on a Van de Vusse continuous stirred-tank reactor (CSTR), a classical process systems engineering (PSE) benchmark [11]. The reaction network consists of consecutive reactions , which produce and subsequently consume the desired intermediate , and a parallel reaction , which consumes reactant . Within each sampling interval, the plant dynamics are
where and denote the concentrations of and , respectively, is the feed concentration, and is the dilution rate. The kinetic coefficients are , , and . The stochastic forcing terms
Algorithm 1: OAPI for GP–RL–MPC with postdecision state-value learning
Input: Baseline , shared rollout settings; maximum PI iterations ; improvement threshold ; MPC and GPR settings.
Output: Selected controller .
for do
if then break
Construct targets from : raw returns at ; otherwise Eq. (7)
Split data using target-quantile stratification
Standardize using scalers fitted on the initial training subset and held fixed thereafter
Fit by maximizing the exact GP marginal likelihood, warm-starting for
Select PC inducing points and construct the DTC posterior using Eqs. (29) and (30)
Form and using Eqs. (8) and (9)
Store the candidate policy, critic, and score
if then
Reject and retain and its critic
break
else
Accept candidate and its critic; set
if then break
end
end
return
satisfy and , independently across channels and sampling intervals. Each disturbance is sampled once per interval and applied as a constant forcing rate under a zero-order hold (ZOH). Consequently, the feed concentration evolves as a random walk, while nominal MPC predictions assume zero future innovations. The plant parameters, controller settings, and operating conditions are summarized in Table 2 of Appendix § 11.
| Parameter | Value | Description |
| Plant parameters | ||
| 50.0 | Rate constant for [] | |
| 100.0 | Rate constant for [] | |
| 10.0 | Rate constant for [] | |
| 10.0 | Nominal feed concentration [] | |
| 2.0 | Product-concentration forcing-rate std. [] | |
| 10.0 | Feed-concentration forcing-rate std. [] | |
| Initial conditions and constraints | ||
| 2.15 | Initial concentration of [] | |
| 0.90 | Initial concentration of [] | |
| 10.00 | Initial feed concentration [] | |
| 19.63 | Input preceding the first control move [] | |
| 0.0 | Lower bound on [] | |
| 7.0 | Upper bound on [] | |
| 0.0 | Lower bound on [] | |
| 2.4 | Upper bound on [] | |
| 0.0 | Lower dilution-rate bound [] | |
| 60.0 | Upper dilution-rate bound [] | |
| Controller and simulation parameters | ||
| 3 | Prediction horizon [steps] | |
| 1000 | Rollout length [sampling steps] | |
| 0.005 | Sampling interval [h] | |
| 1.0 | Reference update interval [h] | |
| 0.98 | Discount factor | |
| Tracking, input-move, and terminal weights | ||
| 0.99 | Effective-horizon return coverage factor | |
| 0.7 | Lower bound on reference [] | |
| 0.9 | Upper bound on reference [] | |
| 0.9 | Initial reference value [] | |
| 5 | Number of rollouts per policy | |
| 80 | Maximum number of PC-selected inducing points |
Table 2. Plant, controller, and operating settings for the CSTR experiments.
The controlled output is the concentration of the desired product, , with reference and stage cost defined by Eq. (10). The feed concentration is included in the information state as . The nominal postdecision state feature vector is
where overbars denote nominal endpoints obtained from the current state and applied input with zero disturbance innovations. The GP employs a quadratic mean function, , in the tracking-error coordinate, together with an automatic-relevance-determination (ARD) Matérn kernel defined over all four features.
The experiments compare GP–RL–MPC with baseline MPC, assess tracking improvements across accepted policy iterations, and examine how the risk coefficient affects policy-iteration performance.
Baseline Comparison and Policy Iteration Performance
We evaluate the final incumbent GP–RL–MPC controllers with against a baseline MPC controller with zero terminal cost. Both use the same prediction horizon, transition model, stage cost, and constraints. Initial conditions, reference schedules, and disturbance realizations are matched across controllers and PI rounds, and each SSE score is averaged over rollouts. Table 1 summarizes the final incumbent performance after five to eight accepted PI rounds, during which SSE decreases monotonically. The reductions relative to baseline MPC, calculated as , range from 77.07% to 85.63% with a mean of 80.52% and a sample standard deviation of 3.48%. The consistently large improvements across all five seeds demonstrate the effectiveness of the learned terminal cost in compensating for the limited prediction horizon. These results also provide empirical support for reliable policy improvement within the OAPI framework. The corresponding closed-loop tracking responses are shown in Figure 3 in Appendix F.
| Seed | Accepted PI | MPC SSE | Final SSE | SSE Reduction (%) |
| 1 | 7 | 3.20 | 0.46 | 85.63 |
| 2 | 5 | 2.30 | 0.40 | 82.43 |
| 3 | 6 | 2.18 | 0.47 | 78.38 |
| 4 | 8 | 1.81 | 0.42 | 77.07 |
| 5 | 5 | 2.49 | 0.52 | 79.07 |
| Mean SD | — | — | — | 80.52 3.48 |
Table 1. Final incumbent GP–RL–MPC performance with .

Figure 3. Multi-seed closed-loop tracking responses: (a)–(e): Baseline MPC; (f)–(j): Final incumbent GP–RL–MPC controllers with . Each panel overlays rollouts.
Effect of the Risk Coefficient
To investigate the role of epistemic uncertainty in terminal-cost optimization, we compare three risk coefficients, . These correspond to an optimistic lower confidence bound, the posterior mean, and a conservative upper confidence bound, respectively, with representing the bounds of a nominal 99.7% pointwise Gaussian credible interval. Figure 2(a)–(Figure 2e) shows the resulting mean rollout SSE trajectories across policy iterations for the five seed configurations.

Figure 2. Multi-seed analysis of the risk coefficient: (a)–(e): Mean rollout SSE across policy iterations. (f): Normalized SSE at the end of each initial non-increasing segment.
The screening-enabled configuration with achieves sustained, monotonically non-increasing SSE across accepted PI rounds. In contrast, the and configurations exhibit non-monotonic behavior, with generally larger fluctuations under . These latter two configurations are evaluated as diagnostic runs without screening, each continued for the same number of PI rounds as its corresponding run. Although mean-based optimization () and optimistic optimization () occasionally achieve lower SSE, these improvements are neither predictable nor consistently sustained. For example, in Figure 2(c), achieves an SSE of 0.39 at PI #6, compared with 0.47 for , but only after several intermediate performance reversals. Such behavior reflects the sensitivity of policy improvement to the accuracy of the learned critic, which depends on the disturbance realizations and the coverage of the relevant state-action space by the GP training data. In particular, optimistic terminal-cost estimates may favor decisions in poorly sampled regions where critic predictions are less reliable.
To compare the consistency of improvement across risk coefficients, Figure 2(f) summarizes the initial non-increasing segments of the SSE trajectories, starting at PI #1. The score at the end of each segment is normalized by its corresponding baseline score . The resulting means and sample standard deviations of normalized SSE are , , and for , , and , respectively. Under this segment-based comparison, achieves the lowest mean normalized SSE and the smallest variability across seeds.
Conclusion
We developed GP–RL–MPC, an uncertainty-aware framework integrating uncertainty propagation, calibration, and regulation to improve short-horizon MPC performance while preserving its computational advantages. For uncertainty propagation, the GP postdecision state-value critic approximates expected cost-to-go under aleatoric uncertainty from disturbed MC rollouts aligned with nominal terminal queries. This formulation shifts disturbance averaging offline and avoids explicit online expectation calculations. For uncertainty calibration, exact marginal-likelihood hyperparameter fitting is followed by PC-DTC sparse deployment, reducing kernel redundancy and online evaluation costs while maintaining reliable uncertainty estimates. For uncertainty regulation, the UCB-based terminal cost uses the latent posterior standard deviation to provide tunable regulation of critic epistemic uncertainty in control decisions. Numerical results provide empirical support for consistent and sustained policy improvement under the OAPI procedure. Future work will extend GP–RL–MPC to economic optimization and control.
Generative AI Use
Generative AI tools were used to assist with writing. The authors reviewed and verified the AI-assisted content and take responsibility for the accuracy and integrity of the submitted work.
Soft-Constraint Violation Penalties
As an illustrative example, consider softening only the upper product-concentration bound . For , replace the corresponding hard upper bound in with
where limits the admissible violation. This retains the hard outer bound . All other state and input bounds remain unchanged.
Let and define
where is the tracking and input-move cost in Eq. (10). When the slack variable appears only in Eq. (16) and its linear penalty, the minimizing value is . Thus, within the admissible outer bound, the slack formulation is equivalent to using .
The resulting GP–RL–MPC objective becomes
subject to the nominal dynamics, initialization, remaining hard bounds, and the soft constraints in Eq. (16). The horizon penalties are evaluated on nominal predictions.
For critic learning, the same augmented stage cost is evaluated along the disturbed closed-loop rollout:
Following the postdecision state-value formulation, each nominal postdecision state feature is paired with the disturbed return , preserving the indexing and truncation conventions in Appendix D. Violation penalties are evaluated using the full realized constraint violation, even when it exceeds . When this extension is enabled, baseline and learned policies use the same penalized cost , and replaces in the MC target construction and value-space soft updates.
Because the learned postdecision value incorporates the augmented stage costs from onward, including constraint-violation penalties, no additional terminal violation penalty is required. Any hard terminal set , if imposed, remains a separate constraint and is not enforced by the learned terminal value. Moreover, the soft constraints and violation penalties imposed on nominal predictions do not guarantee constraint satisfaction for the disturbed plant.
Gaussian Process Regression and PC-DTC Deployment
This appendix summarizes exact Gaussian process regression (GPR) and the PC-DTC approximation deployed in Section 3.2. All inputs and regression targets below are expressed in standardized coordinates, and policy-iteration indices are suppressed unless explicitly shown.
Exact Gaussian Process Regression
Let denote an input and the associated regression target. A GP prior on the latent function is
where is the parameterized mean function and is the covariance kernel. With Gaussian likelihood noise of variance , the observation model is
For training pairs, write and define
For a query , let . Conditional on the fitted parameters, the exact latent posterior mean and variance are Rasmussen and Williams, [17]
We use an automatic-relevance-determination (ARD) Matérn-3/2 kernel,
with scaled distance
The signal variance sets the covariance scale, while the length scales control variation along each input coordinate. Larger length scales correspond to slower variation along the associated coordinates.
The posterior mean approaches as the cross-covariances vanish far from the training inputs. The latent posterior variance represents model-based epistemic uncertainty conditional on the fitted GP assumptions and parameters. It excludes observation-noise variance and does not directly quantify stochastic return variability or plant risk.
PC-DTC Deployment
For critic learning, let denote the current training subset. The regression targets are constructed as described in § 3.1 and subsequently standardized. The parameter vector , comprising the mean function, kernel, and likelihood parameters, is fitted using Adam to maximize the exact GP log marginal likelihood. The PC-DTC approximation is constructed only after fitting, with these parameters held fixed.
For input collections and , let denote the kernel matrix with entries . For a single input , let denote the column vector of covariances between and the inducing points in . Starting with , PC sequentially selects the unselected training input having the largest residual conditional prior variance (Harbrecht et al., [5]),
where the second term is defined as zero when . The residual measures the prior variance at unexplained by the current inducing set. Selection terminates when the inducing-point budget is reached, the total residual variance falls below a prescribed fraction of its initial value, or the maximum residual variance becomes numerically negligible. This yields an inducing set of size , typically with .
DTC uses the selected inducing points to approximate the training covariance while incorporating information from all training pairs (Quinonero-Candela and Rasmussen, [16]). Write , , and . Define
where provides numerical regularization for the Cholesky factorization. Standard DTC is recovered as . For and a query , the inducing-point covariance vector is . The resulting predictive mean and latent variance are
The variance combines a regularized residual conditional variance, represented by the first two terms, with posterior uncertainty in the inducing variables, represented by the third term. No observation-noise variance is added to the latent prediction.
The deployed mean and latent posterior standard deviation are obtained by transforming and the square root of back to the original cost units. These deployed moments replace the exact posterior moments in Eqs. (23) and (24) when constructing the terminal penalty in Eq. (8).
Predecision and Postdecision State-Value Functions
Let denote the augmented Markov state and the stage cost. For a fixed stationary policy , the standard (predecision) discounted state-value function is defined as (Sutton and Barto, [19])
assuming finite expected discounted cost. MC value fitting estimates this expectation conditional on the current state. Nevertheless, evaluating it at a nominal terminal state neglects the uncertainty in the stochastic transition leading to that state. Accounting for this uncertainty requires an additional expectation over possible terminal states, which can be approximated through explicit uncertainty propagation [6].
To clarify the distinction, consider the final transition from to , with all preceding horizon predictions treated nominally. Under the sufficiency condition in Eq. (4), Eqs. (3) and (5) give
Here, accounts for costs from onward conditional on the realized successor state, while the outer expectation averages over the final innovation that generates that state. The postdecision state-value function incorporates both sources of stochastic expectation, allowing direct nominal terminal evaluation without an additional online expectation. In contrast, evaluating at the nominal predecision terminal state generally neglects the expectation over the final stochastic transition.
The training pairs in Eq. (33) associate nominal postdecision state features with MC returns beginning at the corresponding disturbed successor states. Regression therefore approximates the final-transition expectation together with the subsequent expected continuation cost. With the implemented features and finite-rollout targets, this is an approximation to Eq. (32), with target damping applied through Eq. (7). The GP posterior mean estimates the expected continuation cost, while the uncertainty-regulated terminal function in Eq. (8) additionally accounts for critic epistemic uncertainty. Predictions within the MPC horizon remain certainty-equivalent (CE), as specified in Eq. (9).
In practical implementation, training rollouts can be generated using a simulator or world model to reduce experimental costs. Plant data can then be used for model calibration, validation, and subsequent refinement. The learned critic approximates expected returns under the simulated dynamics and disturbance distribution. Increasing the number of MC rollouts reduces sampling error but cannot eliminate bias arising from model-plant mismatch. Reliable deployment consequently requires validation over the intended operating region and, when necessary, iterative refinement of the dynamics and disturbance models using plant data. Importantly, the GP posterior uncertainty characterizes uncertainty in the learned critic conditional on the assumed model and training data; it does not directly quantify simulator-plant discrepancy. Robustness to such mismatch must therefore be assessed separately.
Postdecision State Data, Targets, and Terminal Evaluation
Fix a rollout-generating policy and collect disturbed rollouts, each containing stage costs. Initial states, previous inputs, reference schedules, and controller and estimator initialization are matched across rollouts and policies. Disturbance seed pairs differ across rollouts and are reused across policies. For clarity, PI and rollout superscripts are omitted, and returns are computed separately within each rollout.
In the implementation, we define within prediction horizon , where the overbar denotes a nominal prediction based on the predecision states, with disturbance innovations set to zero. Thus, is indexed by the generating action at and represents the nominal endpoint at . Let denote the critic feature map and define . Pair this nominal feature with the cost-to-go from the actual disturbed trajectory:
Returns are computed backward using
Each target excludes the current stage cost and contains no additional leading discount. The first training pair is , so is not used as a target and the initial stage cost is excluded from the continuation targets. The boundary pair is also omitted because its zero target is imposed by finite-rollout truncation. To reduce truncation bias, discard the final remaining pairs within each rollout. For , define
Here, specifies discount-weight coverage, with . The retained training pairs correspond to action indices from through within each rollout. For , concatenate the retained pairs from all rollouts and reindex them as
Here, indexes pooled pairs rather than time, each denotes the corresponding continuation return from its source rollout, and repeated samples are retained. Each critic fit uses the current policy-specific batch. The dataset stores raw returns; soft targets and parameter warm starts are applied as described in § 3.1.
Using the uncertainty-regulated terminal estimate defined in Eq. (8), the terminal query and its objective contribution are
The horizon sum includes stage costs from through , while the continuation term is aligned with costs beginning at . GPR can estimate conditional return means without requiring identical feature vectors to recur. The fitted GP mean therefore provides a feature-based approximation under the sampled rollout distribution, with target damping applied during fitting.
Experiment Configurations
Table 2 summarizes the plant, controller, and operating settings. Both controllers are implemented using CasADi with IPOPT. No separate terminal set is imposed beyond the prescribed state bounds. The disturbed plant and nominal one-step predictor use CVODES with absolute and relative tolerances of . Full augmented-state feedback is assumed without additional measurement noise. The reference is initially 0.9 mol L and subsequently sampled uniformly from [0.7, 0.9] mol L at each reference update; its known future values are supplied to controllers. Reference schedules and disturbance seed pairs are reused across policies, with all rollout initializations reset as described in Appendix § 10.
Critic data are split into training and diagnostic subsets in an 80%:20% ratio using ten target-quantile strata. We apply for soft updates, fixed first-round scalers, and parameter warm starts. The mean is , where is the standardized tracking-error feature. Adam uses a learning rate of for at most 5000 steps, stopping after 200 consecutive steps without a training-loss improvement exceeding . PC uses a remaining-trace tolerance of and a residual-variance cutoff of ; DTC uses in standardized coordinates. Screened runs terminate at the first SSE deterioration or upon exhausting the prescribed PI budget; rejected candidates are excluded from the accepted sequence. Diagnostic runs use the matched iteration budgets described in § 4.3.
Closed-Loop Trajectories
Figure 3 compares the five-seed closed-loop trajectories under the matched settings described in § 4.2. GP–RL–MPC tracks the varying setpoints more closely and reduces persistent offsets relative to baseline MPC.
References
- [1]Dimitri P. Bertsekas. Reinforcement Learning and Optimal Control. Athena Scientific, Belmont, MA, 2019. ISBN 978-1-886529-39-7.
- [2]Dimitri P. Bertsekas. Abstract Dynamic Programming. Athena Scientific, Belmont, MA, 3rd edition, 2022. ISBN 978-1-886529-47-2.
- [3]Dimitri P. Bertsekas. Model predictive control and reinforcement learning: A unified framework based on dynamic programming. IFAC-PapersOnLine, 58(18):363–383, 2024. doi: 10.1016/j.ifacol.2024.09.056.DOI
- [4]Sébastien Gros and Mario Zanon. Data-driven economic NMPC using reinforcement learning. IEEE Transactions on Automatic Control, 65(2):636–648, 2020. doi: 10.1109/TAC.2019.2913768.DOI
- [5]Helmut Harbrecht, Michael Peters, and Reinhold Schneider. On the low-rank approximation by the pivoted Cholesky decomposition. Applied Numerical Mathematics, 62(4):428–440, 2012. doi: 10.1016/j.apnum.2011.10.001.DOI
- [6]Daniel Landgraf, Andreas Völz, and Knut Graichen. A software framework for stochastic model predictive control of nonlinear continuous-time systems (GRAMPC-S). Optimization and Engineering, 27(1):193–228, 2026. doi: 10.1007/s11081-025-10006-z.DOI
- [7]Jay H. Lee. Model predictive control: Review of the three decades of development. International Journal of Control, Automation and Systems, 9(3):415–424, 2011. doi: 10.1007/s12555-011-0300-6.DOI
- [8]Jay H. Lee, Niket S. Kaisare, and Carlos E. García. Model Predictive Control: Fundamentals and Practice. John Wiley & Sons, 2026. ISBN 978-1-394-33329-5. doi: 10.1002/978139433325.DOI
- [9]Jong Min Lee, Niket S. Kaisare, and Jay H. Lee. Choice of approximator and design of penalty function for an approximate dynamic programming based control approach. Journal of Process Control, 16(2):135–156, 2006. doi: 10.1016/j.jprocont.2005.04.010.DOI
- [10]Jon Li and Jay H. Lee. Deep reinforcement learning integrated NARX-model predictive control. In Proceedings of the 2025 AIChE Annual Meeting, Boston, MA, USA, November 2025. American Institute of Chemical Engineers. ISBN 978-0-8169-1125-7. URL https://proceedings.aiche.org/conferences/aiche-annual-meeting/2025/proceeding/paper/integrating-deep-reinforcement. Paper 109b.
- [11]Jon Li and Jay H. Lee. Reinforcement learning-enhanced model predictive control using an iteratively learned PICNN terminal value function. Computers & Chemical Engineering, 216:109879, 2026a. doi: 10.1016/j.compchemeng.2026.109879.DOI
- [12]Jon Li and Jay H. Lee. RBF–RL–MPC: Finite localized value functions for reinforcement learning-based model predictive control. Journal of Process Control, 168:103884, 2026b. doi: 10.1016/j.jprocont.2026.103884.DOI
- [13]Ali Mesbah. Stochastic model predictive control: An overview and perspectives for future research. IEEE Control Systems, 36(6):30–44, 2016. doi: 10.1109/MCS.2016.2602087.DOI
- [14]Francisco Moreno-Mora, Lukas Beckenbach, and Stefan Streif. Predictive control with learning-based terminal costs using approximate value iteration. IFAC-PapersOnLine, 56(2):3874–3879, 2023. doi: 10.1016/j.ifacol.2023.10.1320.
- [15]Warren B. Powell. Perspectives of approximate dynamic programming. Annals of Operations Research, 241(1–2):319–356, 2016. doi: 10.1007/s10479-012-1077-6.
- [16]Joaquin Quiñonero-Candela and Carl Edward Rasmussen. A unifying view of sparse approximate Gaussian process regression. Journal of Machine Learning Research, 6(65):1939–1959, 2005. URL https://www.jmlr.org/papers/v6/quinonero-candela05a.html.
- [17]Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA, 2006. ISBN 978-0-262-18253-9. doi: 10.7551/mitpress/3206.001.0001.DOI
- [18]James B. Rawlings, David Q. Mayne, and Moritz M. Diehl. Model Predictive Control: Theory, Computation, and Design. Nob Hill Publishing, Santa Barbara, CA, 2nd edition, 2020. ISBN 978-0-9759377-5-4.
- [19]Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 2nd edition, 2018. ISBN 978-0-262-03924-6.
- [20]Wee Chin Wong and Jay H. Lee. Postdecision-state-based approximate dynamic programming for robust predictive control of constrained stochastic processes. Industrial & Engineering Chemistry Research, 50(3):1389–1399, 2011. doi: 10.1021/ie1007882.DOI