betaFeedback

WorksEngineering

Reinforcement Twinning via Twin-Calibrated Bayesian Policy Search and Replay-Driven Identification

Abstract

The feasibility of updating and synchronising digital twins of actuated systems from operational data depends strongly on how the system is controlled and, consequently, on the data generated during operation. This establishes a direct link between control actuation and model identifiability. This work presents a reinforcement-twinning framework that exploits this link through three computational components. A Gaussian-process surrogate of the physical return combines the twin prediction, used as a calibrated trend, with a discrepancy learned from measured returns, allowing Bayesian policy search to exploit the twin without being tied to its predictions. Candidate policies are proposed around the policy that is optimal for the twin, while measured returns can expand this region. The resulting physical episodes are replayed on the twin after observer-based synchronisation with sparse measurements, and its parameters are identified by combining Bayesian optimisation with gradient-based refinement. Selected information-oriented episodes deploy an excitation policy to improve parameter identifiability. The framework is demonstrated on a Burgers-flow analogue of dynamic-induction control in wind farms, with a deliberately misspecified model structure. Despite substantially biased return predictions, the twin provides a useful starting policy and search region, placing the guided search ahead of unguided Bayesian optimisation from its first physical evaluations. This advantage is maintained when the search region is allowed to expand in response to changes in measured returns, whereas searches over-constrained by the twin-defined region are eventually overtaken by the unguided alternatives. The results support reinforcement twinning as a framework for jointly adapting models and control policies when physical interaction is limited and model discrepancy cannot be neglected.

Introduction

Digital twins have become a central paradigm for cyber-physical systems and Industry 4.0, linking a physical process to a continually updated virtual model through sensing, communication, and embedded computation [1, 2, 3, 4]. Unlike a standalone model, a twin remains connected to operational data and evolves with the system it represents, supporting prediction, optimisation, and decision-making as new measurements become available [5, 6, 7, 8, 9]. In actuated systems, control and twinning are naturally coupled: the twin can guide control, while the applied actions determine the trajectories explored by the physical system and therefore the data available to update the twin. This creates the possibility of reciprocal learning between model and controller.

Realising this reciprocity requires addressing two complementary problems. First, the twin must be synchronised, identified, or updated from trajectories generated during operation, which may provide only sparse and unevenly informative data. Second, the controller must exploit useful model guidance without becoming overly constrained by imperfect predictions. Between these two lies informative actuation: actions can be chosen not only for control performance, but also to improve identifiability, reduce parameter uncertainty, or reveal model discrepancy. The literature intersected by these problems is vast, and the following offers only a concise overview of the main ideas and approaches most closely related to the present work.

Model Identification and Assimilation

Updating and synchronising a digital twin from sparse and noisy operational measurements can be a formidable challenge, especially when the twin is governed by partial differential equations (PDEs). Parameter identification then requires estimating the full initial state, propagating it under the recorded forcing and control actions, and comparing the resulting prediction with measurements that typically observe only a small part of that state.

For PDE-governed systems, this generally requires data assimilation, namely the reconstruction of the underlying fields from sparse measurements. The resulting inverse problem can be poorly conditioned, depending on sensor placement and on the sensitivity of the dynamics to the model parameters [10, 11, 12, 13, 14, 15, 16]. When reconstruction and prediction must also be performed fast enough for real-time forecasting, dimensionality reduction and mappings

Technical details of the Burgers benchmark

to low-dimensional latent representations can become necessary (see, e.g. [17]). An additional layer of complexity arises when state and parameter estimation are carried out jointly [18, 19, 20, 21, 22]: errors in the reconstructed state can be absorbed into the inferred parameters, while model error can in turn bias the state estimate unless it is accounted for simultaneously [23].

Finally, for actuated systems, identifiability also depends on the trajectories generated by the applied control. This is the well-known problem of persistent excitation and optimal experimental design [24, 25, 26, 27, 28, 29, 30]: parameters can only be identified if the system is driven through sufficiently informative dynamics. The control input therefore becomes part of the identification problem and can be chosen deliberately to improve identifiability, for example by reducing parameter uncertainty or exposing model discrepancy.

Recent studies design such experiments directly on the model [31, 32]. Sequential Bayesian experimental design has similarly been formulated as a Markov decision process and addressed through approximate dynamic programming, policy-gradient methods, and learned design policies based on expected information gain [33, 34, 35, 36, 37, 38]. In learning-based control, active system identification selects inputs or exploration policies using Fisher-information or information-gain criteria [39, 40, 41, 42, 43, 44], while reinforcement learning has recently been used to steer sensors for data acquisition in digital twins [45]. Together, these approaches show how controlled interaction can improve model identification.

Model-Based and Model-Free Control

Finding control actions that optimise a prescribed objective for a known dynamical model is the classical problem of optimal control [46, 47, 48, 49]. When the model is imperfect, however, the challenge is to exploit its predictions without making control performance overly dependent on its accuracy.

Self-tuning and adaptive control [50, 51, 52, 53] seek to reduce this dependence by updating the model and/or controller online as new measurements become available. At the other extreme, model-free methods seek to solve the control problem directly from measured system performance, without relying on an explicit dynamical model [54, 55, 56, 57, 58]. This avoids bias from an inaccurate model, but generally requires more interaction with the physical system and potentially lengthy, costly, or even unsafe trial-and-error experimentation.

Model-based and hybrid reinforcement-learning approaches bridge these extremes by using learned dynamics models to plan, generate synthetic experience, or guide control improvement [22, 59, 60, 61, 62, 63, 64, 65]. For distributed systems, these models are often reduced-order representations learned from data, allowing feedback controllers to be trained offline and deployed in real time [66]. Such model-based guidance can substantially reduce the number of physical interactions compared with purely model-free approaches [67, 68, 69, 70, 71], but it also introduces a dependence on model quality, since model errors can bias the resulting control strategy. The central challenge is therefore to exploit useful information from the model without allowing its inaccuracies to dominate the search.

Dual Control and Reinforcement Twinning

Model identification and control meet when actions are valued both for the performance they produce and for the information they generate. This is the classical dual-control problem, originally formulated by Feldbaum [72, 73]. Feldbaum showed that an optimal decision may sacrifice immediate performance for information that improves later decisions. This idea is closely related to the exploration–exploitation trade-off in reinforcement learning, where exploratory interactions can improve value estimates or policies used for subsequent control [55, 74, 75, 76, 77]. In dual control, however, this exploration is explicitly valued through its effect on future uncertainty and performance. Exact dual control requires propagating both the system state and uncertainty about the model, which is tractable only for relatively small problems [78, 79, 80, 81]. Practical formulations rely on approximations, for example within model predictive control or Bayesian policy search [82, 83, 84, 85, 86].

Reinforcement Twinning (RT) [87, 88] was introduced from a related perspective, coupling model-based and model-free learning through a continuously adapted digital twin. As in dual control, physical interactions have a dual role: they produce control performance while also generating information that can improve subsequent decisions. In RT, however, this reciprocity is organised explicitly around the twin. The twin provides model-based guidance to the controller, while physical interaction supplies data used both to improve control and to update the twin.

Earlier realisations were primarily control-oriented. In the original formulation, an identified twin supported a model-based controller developed alongside a model-free actor–critic agent, with information transferred between the two when either learning process stagnated. Although the twin was continually updated from trajectories generated during operation, its main role remained to support control.

Contributions of This Work

The present work strengthens the reciprocal connection between model improvement and control in two ways. First, trajectories generated by the controller are replayed to synchronise and update the twin. The twin can in turn be used to synthesise excitation policies for selected episodes, explicitly to improve parameter identification, bringing the experimental-design perspective discussed above into the control loop. Second, measured control performance is used to calibrate the twin’s prediction of the return and to learn its discrepancy with the physical system, so that the twin influences policy search only to the extent supported by physical evidence. This second component builds on the construction of Kennedy and O’Hagan, in which a response is modelled as the scaled prediction of a less accurate model plus a Gaussian-process discrepancy [89, 90]. The same construction has been used to calibrate flow and PDE models against measurements [91, 92], and underlies multi-fidelity surrogates in which a cheap model is corrected using a small number of expensive evaluations [93, 94, 95, 96, 97, 98], including approaches for correcting optima computed on a low-fidelity model [99]. Here, the twin plays the role of the low-fidelity model and the measured returns that of the high-fidelity data. A similar division of roles determines where new policies are proposed: the twin defines a region around the policy that is optimal for it, while measured returns can expand this region when the search approaches its boundary. The resulting framework therefore allows the twin to guide control without over-committing to its predictions, while both the observed trajectories and their measured performance contribute to improving the twin.

The framework is demonstrated on a one-dimensional Burgers-flow analogue of distributed energy extraction, motivated by dynamic-induction control in wind farms [100, 101]. The twin is deliberately misspecified through a persistent, policy-dependent structural discrepancy, and the physical system is observed only through sparse sensors.

The paper is organised as follows. Section 2 introduces the general RT setting and its main principles. Section 3 details the numerical implementation of each operation of the resulting algorithm: initialisation, synthesis of the model-based reference, the return surrogate, selection of performance and excitation policies, state synchronisation, and replay identification of the twin parameters. Section 4 defines the Burgers benchmark and its numerical set-up, and Section 5 presents the results. Section 6 summarises the work and discusses its main limitations and possible extensions.

Reinforcement Twinning

Reinforcement twinning targets actuated systems for which extensive trial-and-error experimentation on the physical system is impractical and only an imperfect model is available. This model depends on uncertain parameters that can be continuously refined from operational data, and is therefore referred to as a digital twin (hereinafter the twin, for brevity).

The simultaneous improvement of the control policy and the twin leverages data collected over sequences of interactions called episodes, following the loop in Figure 1. Physical episodes are replayed on the twin for state synchronisation and parameter identification, while counterfactual rollouts are used to evaluate candidate performance and excitation policies without interacting with the physical system. These virtual evaluations guide the next physical interaction, whose data then drive subsequent policy and twin updates. The remainder of this section introduces the physical and twin dynamics, defines the learning objectives, and assembles these components into the reinforcement-twinning loop. Throughout, subscript ∙\bullet denotes physical quantities, and subscript o\mathrm{o} denotes twin quantities, physical episodes are indexed by nn, and twin updates by mm.

Reinforcement-twinning loop

Figure 1. Reinforcement-twinning loop. Physical environment (left, •): the deployed policy (1) drives a physical episode (2), whose record (3) feeds the replay buffer and, for performance episodes, the archive of measured returns. Digital twin (right, ◦): factual replay (4) synchronises the twin with the data and refines its parameters; counterfactual rollouts (5) provide the virtual returns, the model-based reference and the informativeness of candidate excitation policies; with the measured returns, these determine the next interaction (6). All components are presented in Sections 2 and 3.

Physical System and Digital Twin

Each episode consists of NN interactions with the physical system, occurring at the sampling times tk=kΔtt_k = k\Delta t, with k=0,…,N−1k = 0,\ldots,N-1 and Δt\Delta t the sampling period. The physical system and its digital twin are treated as discrete dynamical systems evolving side by side as

s∙,k+1=F∙(s∙,k,zk,a∙,k),so,k+1=Fo(so,k,z^k,ao,k;θp),oˉ∙,k=h(s∙,k,a∙,k−1),o∙,k=oˉ∙,k+ε∙,k,oˉo,k=h(so,k,ao,k−1),oo,k=oˉo,k+εo,k,χ∙,k+1=Gπ(χ∙,k,o∙,k),χo,k+1=Gπ(χo,k,oo,k),a∙,k=πo(n)(o∙,k,χ∙,k),ao,k=πo(oo,k,χo,k),s∙,0∼I0(hidden),so,0=s^o,0a(analysed).(1)\begin{aligned} \mathbf{s}_{\bullet,k+1} &= F_{\bullet}(\mathbf{s}_{\bullet,k},\mathbf{z}_k,\mathbf{a}_{\bullet,k}),\\ \mathbf{s}_{\mathrm{o},k+1} &= F_{\mathrm{o}}(\mathbf{s}_{\mathrm{o},k},\hat{\mathbf{z}}_k,\mathbf{a}_{\mathrm{o},k};\boldsymbol{\theta}_{\mathrm{p}}),\\ \bar{\mathbf{o}}_{\bullet,k} &= h(\mathbf{s}_{\bullet,k},\mathbf{a}_{\bullet,k-1}), \qquad\mathbf{o}_{\bullet,k} = \bar{\mathbf{o}}_{\bullet,k}+\boldsymbol{\varepsilon}_{\bullet,k},\\ \bar{\mathbf{o}}_{\mathrm{o},k} &= h(\mathbf{s}_{\mathrm{o},k},\mathbf{a}_{\mathrm{o},k-1}), \qquad\mathbf{o}_{\mathrm{o},k} = \bar{\mathbf{o}}_{\mathrm{o},k}+\boldsymbol{\varepsilon}_{\mathrm{o},k},\\ \boldsymbol{\chi}_{\bullet,k+1} &= G_{\boldsymbol{\pi}}(\boldsymbol{\chi}_{\bullet,k},\mathbf{o}_{\bullet,k}),\\ \boldsymbol{\chi}_{\mathrm{o},k+1} &= G_{\boldsymbol{\pi}}(\boldsymbol{\chi}_{\mathrm{o},k},\mathbf{o}_{\mathrm{o},k}),\\ \mathbf{a}_{\bullet,k} &= \boldsymbol{\pi}^{(n)}_{\mathrm{o}}(\mathbf{o}_{\bullet,k},\boldsymbol{\chi}_{\bullet,k}),\\ \mathbf{a}_{\mathrm{o},k} &= \boldsymbol{\pi}_{\mathrm{o}}(\mathbf{o}_{\mathrm{o},k},\boldsymbol{\chi}_{\mathrm{o},k}),\\ \mathbf{s}_{\bullet,0} &\sim\mathcal{I}_0\quad\text{(hidden)},\\ \mathbf{s}_{\mathrm{o},0} &= \hat{\mathbf{s}}_{\mathrm{o},0}^{a}\quad\text{(analysed)}. \tag*{(1)} \end{aligned}

Each pair of lines gives the physical and twin forms of the dynamics, the observation, the information state of the policy, the action and the initial condition. These are briefly described below to introduce the main components and their assembly into the algorithm, introduced in Section 2.3. Section 3 enters into the details of each.

Dynamics The physical system evolves from a hidden state s∙,k∈Rns\mathbf{s}_{\bullet,k} \in\mathbb{R}^{n_s} through an unknown (nonlinear) transition operator F∙F_{\bullet}. Both the real system and the twin receive at each time step kk a control action ak∈Rna\mathbf{a}_k \in\mathbb{R}^{n_a}, governed by a control policy, and an exogenous forcing. The physical system is subject to the forcing zk∈Rnz\mathbf{z}_k \in\mathbb{R}^{n_z} itself, a stochastic input realised from a process Pz\mathcal{P}_{\mathbf{z}} while the twin receives in input its measurement z^k\hat{\mathbf{z}}_k, which may be a component of o∙,k\mathbf{o}_{\bullet,k}; the measurement error of this input thus adds to the discrepancy between the two systems.

Unmeasured inputs would require an input estimator, not considered here. All inputs at kk are considered to be applied over [tk,tk+1)[t_k,t_{k+1}). The state is assumed or made Markovian, by augmenting it with any system memory such as actuator dynamics or delays, whereas the memory of the policy is kept in the separate information state χk\boldsymbol{\chi}_k. The twin has a state so,k∈Rns,o\mathbf{s}_{\mathrm{o},k} \in\mathbb{R}^{n_{s,\mathrm{o}}}, whose dimension might differ from nsn_s, and evolves under the partially known operator FoF_{\mathrm{o}}, with fixed structure and uncertain twin parameters θp∈Θp⊂Rnθp\boldsymbol{\theta}_{\mathrm{p}} \in\Theta_{\mathrm{p}} \subset\mathbb{R}^{n_{\theta_{\mathrm{p}}}}. It may be structurally misspecified, in which case no θp\boldsymbol{\theta}_{\mathrm{p}} gives Fo≡F∙F_{\mathrm{o}} \equiv F_{\bullet}; identification then yields the best in-family approximation, leaving a residual discrepancy that may depend on the policy. In the following, θp(m)\theta_{\mathrm{p}}^{(m)} denotes the twin parameters at the mm-th scheduled twin update, while θp\theta_{\mathrm{p}} without superscript refers to the same parameters as a free variable.

Observations

The observation operator hh maps the state and the previously applied action to a noise-free output. On the physical system, oˉ∙,k\bar{\mathbf{o}}_{\bullet,k} is latent: only the noisy measurement o∙,k∈Rno\mathbf{o}_{\bullet,k} \in\mathbb{R}^{n_o} is available to the controller, typically with no≪nsn_o \ll n_s. The measurement error ϵ∙,k∈Rno\boldsymbol{\epsilon}_{\bullet,k} \in\mathbb{R}^{n_o} is zero-mean and follows a known noise model, denoted compactly by its covariance Σo∈Rno×no\boldsymbol{\Sigma}_o \in\mathbb{R}^{n_o \times n_o}; the model may include correlations across components or time. The dependence of hh on the previously applied action accounts for measurements that respond directly to the command, such as actuator states, while keeping the observation at tkt_k available before the action at tkt_k is chosen. Applying the same observation operator to the twin state gives the noise-free prediction oˉ∘,k\bar{\mathbf{o}}_{\circ,k}. The operator may also depend on the twin parameters, as in the test case of § 5.2.

In factual replay, no synthetic observation noise is drawn: oˉ∘,k\bar{\mathbf{o}}_{\circ,k} is compared directly with the recorded physical measurement o∙,k\mathbf{o}_{\bullet,k}. The resulting residual therefore contains both physical measurement noise and any discrepancy between the physical system and the twin. In counterfactual rollouts, synthetic noise ϵ∘,k\boldsymbol{\epsilon}_{\circ,k} is instead generated from the physical noise model to form o∘,k\mathbf{o}_{\circ,k}. Both the candidate policy and the reward receive this observation, as do their physical counterparts. A complete noise trajectory is fixed for each entry of the scoring bank, ensuring that all candidate policies are evaluated under the same noise realisations. When a measured exogenous input zˉk\bar{\mathbf{z}}_k is itself a component of the observation, the input record and the synthetic observation must be constructed consistently, without adding its measurement noise twice.

Initial condition

The physical state at the start of the episode is hidden and only partially observed, so the initial state of the twin cannot be set directly. It is instead reconstructed by an observer Ψ∘\Psi_{\circ} from the measurements, inputs, and actions recorded over a window Wn−\mathcal{W}_n^{-} of LaL_a steps preceding the episode, indexed by negative kk,

References

  1. [1]E. A. Lee. Cyber physical systems: Design challenges. In 2008 11th IEEE International Symposium on Object and Component-Oriented Real-Time Distributed Computing, pages 363–369, 2008. doi: 10.1109/ISORC.2008.25.DOI
  2. [2]E. A. Lee and S. A. Seshia. Introduction to Embedded Systems: A Cyber-Physical Systems Approach. MIT Press, 2015.
  3. [3]H. Kagermann, W. Wahlster, and J. Helbig. Recommendations for implementing the strategic initiative industrie 4.0. Final report of the industrie 4.0 working group, National Academy of Science and Engineering, 2013.
  4. [4]I. Ficili, M. Giacobbe, G. Tricomi, and A. Puliafito. From sensors to data intelligence: Leveraging iot, cloud, and edge computing with ai. Sensors, 25(6):1763, March 2025. doi: 10.3390/s25061763.DOI
  5. [5]D. Jones, C. Snider, A. Nassehi, J. Yon, and B. Hicks. Characterising the digital twin: A systematic literature review. CIRP Journal of Manufacturing Science and Technology, 29:36–52, 2020. doi: 10.1016/j.cirpj.2020.02.002.DOI
  6. [6]M. Bokhtiar Al Zami, S. Shaon, V. Khanh Quy, and D. C. Nguyen. Digital twin in industries: A comprehensive survey. IEEE Access, 13:47291–47336, 2025. doi: 10.1109/ACCESS.2025.3551532.
  7. [7]P. Korenhof, V. Blok, and S. Kloppenburg. Steering representations—towards a critical understanding of digital twins. Philosophy & Technology, 34(4):1751–1773, October 2021. doi: 10.1007/s13347-021-00484-1.DOI
  8. [8]D. J. Wagg, C. Burr, J. Shepherd, Z. Xuereb Conti, M. Enzer, and S. Niederer. The philosophical foundations of digital twinning. Data-Centric Engineering, 6:e12, 2025. doi: 10.1017/dce.2025.4.DOI
  9. [9]M. Torzoni, M. Tezzele, S. Mariani, A. Manzoni, and K. E. Willcox. A digital twin framework for civil engineering structures. Computer Methods in Applied Mechanics and Engineering, 418:116584, 2024. doi: 10.1016/j.cma.2023.116584.DOI
  10. [10]the state of turbulence: Fundamental difficulties of flow reconstruction. Physical Review Fluids, 6(10):100501, 2021. doi: 10.1103/PhysRevFluids.6.100501.DOI
  11. [11]T. A. Zaki. Turbulence from an observer perspective. Annual Review of Fluid Mechanics, 57(1):311–334, 2025. doi: 10.1146/annurev-fluid-030424-114735.DOI
  12. [12]J. Hoepffner, M. Chevalier, T. R. Bewley, and D. S. Henningson. State estimation in wall-bounded flow systems. Part 1. Perturbed laminar flows. Journal of Fluid Mechanics, 534:263–294, 2005. doi: 10.1017/S0022112005004210.DOI
  13. [13]C. H. Colburn, J. B. Cessna, and T. R. Bewley. State estimation in wall-bounded flow systems. Part 3. The ensemble Kalman filter. Journal of Fluid Mechanics, 682:289–303, 2011. doi: 10.1017/jfm.2011.222.
  14. [14]V. Mons, J.-C. Chassaing, T. Gomez, and P. Sagaut. Reconstruction of unsteady viscous flows using data assimilation schemes. Journal of Computational Physics, 316:255–280, 2016. doi: 10.1016/j.jcp.2016.04.022.DOI
  15. [15]K. Manoahr, B. W. Brunton, J. N. Kutz, and S. L. Brunton. Data-driven sparse sensor placement for reconstruction: Demonstrating the benefits of exploiting known patterns. IEEE Control Systems Magazine, 38(3):63–86, 2018. doi: 10.1109/MCS.2018.2810460.
  16. [16]K. Fukami, R. Maulik, N. Ramachandra, K. Fukagata, and K. Taira. Global field reconstruction from sparse sensors with Voronoi tessellation-assisted deep learning. Nature Machine Intelligence, 3(11):945–951, 2021. doi: 10.1038/s42256-021-00402-2.DOI
  17. [17]E. Özalp, A. Nóvoa, and L. Magri. Real-time forecasting of chaotic dynamics from sparse data and autoencoders. Computer Methods in Applied Mechanics and Engineering, 450:118600, 2026. doi: 10.1016/j.cma.2025.118600.DOI
  18. [18]M. Asch, M. Bocquet, and M. Nodet. Data Assimilation: Methods, Algorithms, and Applications. Fundamentals of Algorithms. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2016. doi: 10.1137/1.9781611974546.DOI
  19. [19]P. Moireau, D. Chapelle, and P. Le Tallec. Joint state and parameter estimation for distributed mechanical systems. Computer Methods in Applied Mechanics and Engineering, 197(6–8):659–677, 2008. doi: 10.1016/j.cma.2007.08.021.DOI
  20. [20]L. D. Smith and G. A. Gottwald. Data assimilation for networks of coupled oscillators: inferring unknown model parameters from partial observations. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 481(2313):20240813, May 2025. doi: 10.1098/rspa.2024.0813.DOI
  21. [21]P. A. Marques, S. Ahizi, and M. A. Mendez. Real-time data assimilation for the thermodynamic modeling of cryogenic storage tanks. Energy, 302:131739, September 2024. doi: 10.1016/j.energy.2024.131739.
  22. [22]D. E. Ozan, A. Nóvoa, G. Rigas, and L. Magri. Data-assimilated model-informed reinforcement learning. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 481(2327):20250476, December 2025. doi: 10.1098/rspa.2025.0476.DOI
  23. [23]A. Nóvoa, A. Racca, and L. Magri. Inferring unknown unknowns: Regularized bias-aware ensemble Kalman filter. Computer Methods in Applied Mechanics and Engineering, 418:116502, 2024. doi: 10.1016/j.cma.2023.116502.DOI
  24. [24]G. C. Goodwin and R. L. Payne. Dynamic System Iden- tification: Experiment Design and Data Analysis. Academic Press, 1977.
  25. [25]L. Ljung. System Identification: Theory for the User. Prentice Hall, 2nd edition, 1999.
  26. [26]R. K. Mehra. Optimal input signals for parameter estimation in dynamic systems—survey and new results. IEEE Transactions on Automatic Control, 19(6):753–768, 1974. doi: 10.1109/TAC.1974.1100701.DOI
  27. [27]É. Walter and L. Pronzato. Identification of Parametric Models from Experimental Data. Communications and Control Engineering. Springer, London, UK, 1997.
  28. [28]M. Gevers. Identification for control: From the early achievements to the revival of experiment design. European Journal of Control, 11(4–5):335–352, 2005. doi: 10.3166/ejc.11.335-352.DOI
  29. [29]H. Hjalmarsson. From experiment design to closed-loop control. Automatica, 41(3):393–438, 2005. doi: 10.1016/j.automatica.2004.11.021.DOI
  30. [30]A. Chiuso and G. Pillonetto. System identification: A machine learning perspective. Annual Review of Control, Robotics, and Autonomous Systems, 2(1):281–304, 2019. doi: 10.1146/annurev-control-053018-023744.DOI
  31. [31]K. Bhattacharya, L. Cao, and A. M. Stuart. Optimal experimental design for reliable learning of history-dependent constitutive laws. Computer Methods in Applied Mechanics and Engineering, 457:119022, 2026. doi: 10.1016/j.cma.2026.119022.DOI
  32. [32]H. Yang, C. Chen, and J.-L. Wu. Active learning of model discrepancy with Bayesian experimental design. Computer Methods in Applied Mechanics and Engineering, 446:118198, 2025. doi: 10.1016/j.cma.2025.118198.DOI
  33. [33]X. Huan and Y. M. Marzouk. Sequential Bayesian optimal experimental design via approximate dynamic programming. arXiv preprint arXiv:1604.08320, 2016.arxiv.org/abs/1604.08320
  34. [34]W. Shen and X. Huan. Bayesian sequential optimal experimental design for nonlinear models using policy gradient reinforcement learning. Computer Methods in Applied Mechanics and Engineering, 416:116304, 2023. doi: 10.1016/j.cma.2023.116304.DOI
  35. [35]W. Shen, J. Dong, and X. Huan. Variational sequential optimal experimental design using reinforcement learning. Computer Methods in Applied Mechanics and Engineering, 444:118068, 2025. doi: 10.1016/j.cma.2025.118068.DOI
  36. [36]A. Foster, D. R. Ivanova, I. Malik, and T. Rainforth. Deep adaptive design: Amortizing sequential Bayesian experimental design. In Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pages 3384–3395. PMLR, 2021.arxiv.org/abs/2103.02438
  37. [37]T. Blau, E. V. Bonilla, I. Chades, and A. Dezfouli. Optimizing sequential experimental design with deep reinforcement learning. In Proceedings of the 39th International Conference on Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Research, pages 2107–2128. PMLR, 2022.arxiv.org/abs/2202.00821
  38. [38]T. Rainforth, A. Foster, D. R. Ivanova, and F. Bickford Smith. Modern Bayesian experimental design. Statistical Science, 39(1):100–114, 2024. doi: 10.1214/23-STS915.DOI
  39. [39]A. Wagenmaker and K. Jamieson. Active learning for identification of linear dynamical systems. In Proceedings of the 33rd Conference on Learning Theory (COLT), volume 125 of Proceedings of Machine Learning Research, pages 3487–3582, 2020.arxiv.org/abs/2002.00495
  40. [40]H. Mania, M. I. Jordan, and B. Recht. Active learning for nonlinear system identification with guarantees. Journal of Machine Learning Research, 23(32):1–30, 2022.
  41. [41]M. Buisson-Fenet, F. Solowjow, and S. Trimpe. Actively learning Gaussian process dynamics. In Proceedings of the 2nd Conference on Learning for Dynamics and Control (L4DC), volume 120 of Proceedings of Machine Learning Research, pages 5–15. PMLR, 2020.
  42. [42]B. Sukhija, L. Treven, C. Sancaktar, S. Blaes, S. Coros, and A. Krause. Optimistic active exploration of dynamical systems. In Advances in Neural Information Processing Systems 36, pages 38122–38153, 2023. doi: 10.52202/075280-1656.DOI
  43. [43]V. Mehta, B. Paria, J. Schneider, S. Ermon, and W. Neiswanger. An experimental design perspective on model-based reinforcement learning. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2112.05244
  44. [44]M. Memmel, A. Wagenmaker, C. Zhu, D. Fox, and A. Gupta. ASID: Active exploration for system identification in robotic manipulation. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2404.12308
  45. [45]C. O. Ogbodo, T. J. Rogers, M. Dal Borgo, and D. J. Wagg. Adaptive sensor steering strategy using deep reinforcement learning for dynamic data acquisition in digital twins. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 482(2329):20250326, January 2026. doi: 10.1098/rspa.2025.0326.DOI
  46. [46]R. F. Stengel. Optimal Control and Estimation. Dover Publications, 1994.
  47. [47]D. P. Bertsekas. Dynamic Programming and Optimal Control, Vol. I. Athena Scientific, Belmont, MA, 4 edition, 2012.
  48. [48]J. B. Rawlings, D. Q. Mayne, and M. M. Diehl. Model Predictive Control: Theory, Computation, and Design. Nob Hill Publishing, 2nd edition, 2017.
  49. [49]M. D. Gunzburger. Perspectives in Flow Control and Optimization. SIAM, 2002.
  50. [50]K. J. Åström and B. Wittenmark. On self-tuning regulators. Automatica, 9(2):185–199, 1973. doi: 10.1016/0005-1098(73)90073-3.DOI
  51. [51]K. J. Åström and B. Wittenmark. Adaptive Control. Dover Publications, 2008.
  52. [52]A. M. Annaswamy and A. L. Fradkov. A historical perspective of adaptive control and learning. Annual Reviews in Control, 52:18–41, 2021. doi: 10.1016/j.arcontrol.2021.10.014.DOI
  53. [53]N. Matni, A. Proutiere, A. Rantzer, and S. Tu. From self-tuning regulators to reinforcement learning and back again. In 2019 IEEE 58th Conference on Decision and Control (CDC), pages 3724–3740. IEEE, December 2019. doi: 10.1109/cdc40024.2019.9029916.DOI
  54. [54]H. Hjalmarsson, M. Gevers, S. Gunnarsson, and O. Lequin. Iterative feedback tuning: theory and applications. IEEE Control Systems Magazine, 18(4):26–41, 1998. doi: 10.1109/37.710876.DOI
  55. [55]R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2 edition, 2018.
  56. [56]M. A. Bucci, O. Semeraro, A. Allauzen, G. Wisniewski, L. Cordier, and L. Mathelin. Control of chaotic sys- tems by deep reinforcement learning. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 475(2231):20190351, November 2019. doi: 10.1098/rspa.2019.0351.
  57. [57]F. Pino, L. Schena, J. Rabault, and M. A. Mendez. Comparative analysis of machine learning methods for active flow control. Journal of Fluid Mechanics, 958:A39, March 2023. doi: 10.1017/jfm.2023.76.DOI
  58. [58]F. Pino, E. Fracchia, B. Scheid, and M. A. Mendez. Integral modeling and reinforcement learning control of three-dimensional liquid metal coating on a moving substrate. Physical Review Fluids, 11(4):044003, April 2026. doi: 10.1103/77zf-ks57.DOI
  59. [59]R. S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin, 2(4):160–163, 1991. doi: 10.1145/122344.122377.DOI
  60. [60]M. P. Deisenroth and C. E. Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on Machine Learning (ICML), pages 465–472, 2011.
  61. [61]A. Nagabandhi, G. Kahn, R. S. Fearing, and S. Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA1), pages 7559–7566. IEEE, May 2018. doi: 10.1109/icra.2018.8463189.
  62. [62]T. M. Moerland, J. Broekens, A. Plaat, and C. M. Jonker. Model-based reinforcement learning: A survey. Foundations and Trends in Machine Learning, 16(1):1–118, 2023. doi: 10.1561/2200000086.DOI
  63. [63]S. Bansal, R. Calandra, K. Chua, S. Levine, and C. Tomlin. MBMF: Model-based priors for model-free reinforcement learning. arXiv preprint arXiv:1709.03153, 2017.arxiv.org/abs/1709.03153
  64. [64]T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll, J. A. Ojea, E. Solowjow, and S. Levine. Residual reinforcement learning for robot control. In 2019 International Conference on Robotics and Automation (ICRA), pages 6023–6029. IEEE, May 2019. doi: 10.1109/icra.2019.8794127.DOI
  65. [65]X.-Y. Li and J.-X. Wang. Physics-informed dyna-style model-based deep reinforcement learning for dynamic control. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 477(2255):20210168, November 2021. doi: 10.1098/rspa.2021.0168.
  66. [66]M. Tomasetto, F. Braghin, and A. Manzoni. Latent feedback control of distributed systems in multiple scenarios through deep learning-based reduced order models. Computer Methods in Applied Mechanics and Engineering, 442:118030, 2025. doi: 10.1016/j.cma.2025.118030.DOI
  67. [67]S. Werner and S. Peitz. Numerical evidence for sample efficiency of model-based over model-free reinforcement learning control of partial differential equations. In 2024 European Control Conference (ECC), pages 2965–2971. IEEE, June 2024. doi: 10.23919/ecc64448.2024.10590945.DOI
  68. [68]S. W. R. Werner and B. Peherstorfer. System stabilization with policy optimization on unstable latent manifolds. Computer Methods in Applied Mechanics and Engineering, 433:117483, 2025. doi: 10.1016/j.cma.2024.117483.DOI
  69. [69]A. Rai, R. Antonova, F. Meier, and C. G. Atkeson. Using simulation to improve sample-efficiency of Bayesian optimization for bipedal robots. Journal of Machine Learning Research, 20(49):1–24, 2019.
  70. [70]M. Nobar, J. Keller, A. Rupenyan, M. Khosravi, and J. Lygeros. Guided bayesian optimization: Data-efficient controller tuning with digital twin. IEEE Transactions on Automation Science and Engineering, 22:11304–11317, 2025. doi: 10.1109/tase.2024.3454176.DOI
  71. [71]M. Nobar, J. Keller, A. Forino, J. Lygeros, and A. Rupenyan. Guided multi-fidelity bayesian optimization for data-driven controller tuning with digital twins. IEEE Robotics and Automation Letters, 11(5):5294–5301, 2026. doi: 10.1109/LRA.2026.3671557.DOI
  72. [72]A. A. Feldbaum. Dual control theory. Automation and Remote Control, 21:874–880, 1960.
  73. [73]A. A. Feldbäum. Dual control theory problems. IFAC Proceedings Volumes, 1(2):541–550, 1963. doi: 10.1016/S1474-6670(17)69687-3.DOI
  74. [74]G. A. Rummery and M. Niranjan. On-line Q-learning using connectionist systems. Technical Report CUED/F-INFENG/TR 166, Department of Engineering, University of Cambridge, 1994.
  75. [75]V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015. doi: 10.1038/nature14236.
  76. [76]T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016.arxiv.org/abs/1509.02971
  77. [77]T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning (ICML), pages 1861–1870, 2018.
  78. [78]Y. Bar-Shalom and E. Tse. Dual effect, certainty equivalence, and separation in stochastic control. IEEE Transactions on Automatic Control, 19(5):494–500, 1974. doi: 10.1109/TAC.1974.1100635.
  79. [79]B. Bernhardsson. Dual control of a first-order system with two possible gains. International Journal of Adaptive Control and Signal Processing, 3(1):15–22, 1989. doi: 10.1002/acs.4480030103.
  80. [80]K. J. Åström and A. Helmersson. Dual control of an integrator with unknown gain. Computers & Mathematics with Applications, 12(6, Part A):653–662, 1986. doi: 10.1016/0898-1221(86)90052-0.DOI
  81. [81]N. M. Filatov and H. Unbehauen. Survey of adaptive dual control methods. IEE Proceedings – Control Theory and Applications, 147(1):118–128, 2000. doi: 10.1049/ip-cta:200000107.DOI
  82. [82]B. Wittenmark. Adaptive dual control methods: An overview. IFAC Proceedings Volumes, 28(13):67–72, 1995. doi: 10.1016/S1474-6670(17)45327-4.DOI
  83. [83]A. Mesbah. Stochastic model predictive control with active uncertainty learning: A survey on dual control. Annual Reviews in Control, 45:107–117, 2018. doi: 10.1016/j.arcontrol.2017.11.001.DOI
  84. [84]E. D. Klenske and P. Hennig. Dual control for approx imate Bayesian reinforcement learning. Journal of Machine Learning Research, 17(127):1–30, 2016.
  85. [85]L. Arcari, L. Hewing, M. Schlichting, and M. N. Zeilinger. Dual stochastic MPC for systems with parametric and structural uncertainty. In Proceedings of the 2nd Conference on Learning for Dynamics and Control (L4DC), volume 120 of Proceedings of Machine Learning Research, pages 894–903, 2020.
  86. [86]W.-H. Chen, C. Rhodes, and C. Liu. Dual control for exploitation and exploration (DCEE) in autonomous search. Automatica, 133:109851, 2021. doi: 10.1016/j.automatica.2021.109851.DOI
  87. [87]L. Schena, P. A. Marques, R. Poletti, S. Ahizi, J. Van den Berghe, and M. A. Mendez. Reinforcement twinning: From digital twins to model-based reinforcement learning. Journal of Computational Science, 82:102421, October 2024. doi: 10.1016/j.jocs.2024.102421.DOI
  88. [88]R. Poletti, L. Schena, L. Koloszar, J. Degroote, and M. A. Mendez. Reinforcement twinning for hybrid control of flapping-wing drones. arXiv preprint arXiv:2505.18201, 2025.arxiv.org/abs/2505.18201
  89. [89]M. C. Kennedy and A. O’Hagan. Predicting the output from a complex computer code when fast approximations are available. Biometrika, 87(1):1–13, 2000. doi: 10.1093/biomet/87.1.1.DOI
  90. [90]M. C. Kennedy and A. O’Hagan. Bayesian calibration of computer models. Journal of the Royal Statistical Society: Series B, 63(3):425–464, 2001. doi: 10.1111/1467-9868.00294.DOI
  91. [91]S. Guillas, N. Glover, and L. Malki-Epshtein. Bayesian calibration of the k–ε turbulence model for a CFD model of street canyon flow. Computer Methods in Applied Mechanics and Engineering, 279:536–553, 2014. doi: 10.1016/j.cma.2014.06.008.DOI
  92. [92]M. Girolami, E. Febrianto, G. Yin, and F. Cirak. The statistical finite element method (statFEM) for coherent synthesis of observation data and model predictions. Computer Methods in Applied Mechanics and Engineering, 375:113533, 2021. doi: 10.1016/j.cma.2020.113533.DOI
  93. [93]A. I. J. Forrester, A. Sóbestér, and A. J. Keane. Multi-fidelity optimization via surrogate modelling. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 463(2088):3251–3269, 2007. doi: 10.1098/rspa.2007.1900.DOI
  94. [94]L. Le Gratiét and J. Garnier. Recursive co-kriging model for design of computer experiments with multiple levels of fidelity. International Journal for Uncertainty Quantification, 4(5):365–386, 2014. doi: 10.1615/Int.J.UncertaintyQuantification.20140106914.
  95. [95]L. Bonfiglio, P. Perdikari, S. Brizzolara, and G. E. Karniadakis. Multi-fidelity optimization of super-cavitating hydrofoils. Computer Methods in Applied Mechanics and Engineering, 332:63–85, 2018. doi: 10.1016/j.cma.2017.12.009.DOI
  96. [96]B. Peherstorfer, K. Willcox, and M. Gunzburger. Survey of multifidelity methods in uncertainty propagation, inference, and optimization. SIAM Review, 60(3):550–591, 2018. doi: 10.1137/16M1082469.DOI
  97. [97]Z. Z. Foumani, M. Shishehbor, A. Yousefpour, and R. Bostanabad. Multi-fidelity cost-aware Bayesian optimization. Computer Methods in Applied Mechanics and Engineering, 407:115937, 2023. doi: 10.1016/j.cma.2023.115937.DOI
  98. [98]M. Fiore, A. Bresciani, M. A. Mendez, and J. van Beeck. Predicting wind loads on container ships in harbour environments through multi-fidelity modelling. Ocean Engineering, 364:126928, 2026. doi: 10.1016/j.oceaneng.2026.126928.DOI
  99. [99]J. Hart and B. van Bloemen Waanders. Hyper-differential sensitivity analysis with respect to model discrepancy: Optimal solution updating. Computer Methods in Applied Mechanics and Engineering, 412:116082, 2023. doi: 10.1016/j.cma.2023.116082.DOI
  100. [100]W. Munters and J. Meyers. An optimal control framework for dynamic induction control of wind farms and their interaction with the atmospheric boundary layer. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 375(2091):20160100, March 2017. doi: 10.1098/rsta.2016.0100.DOI
  101. [101]W. Munters and J. Meyers. Towards practical dynamic induction control of wind farms: analysis of optimally controlled wind-farm boundary layers and sinusoidal induction control of first-row turbines. Wind Energy Science, 3(1):409–425, June 2018. doi: 10.5194/wes-3-409-2018.DOI
  102. [102]J. Močkus. On Bayesian methods for seeking the extremum. In G. I. Marchuk, editor, Optimization Techniques IFIP Technical Conference Novosibirsk, July 1–7, 1974, pages 400–404, Berlin, Heidelberg, 1975. Springer Berlin Heidelberg.
  103. [103]D. R. Jones, M. Schonlau, and W. J. Welch. Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13(4):455–492, 1998. doi: 10.1023/A:1008306431147.DOI
  104. [104]R. Calandra, A. Seyfarth, J. Peters, and M. P. Deisenroth. Bayesian optimization for learning gaits under uncertainty: An experimental comparison on a dynamic bipedal walker. Annals of Mathematics and Artificial Intelligence, 76(1–2):5–23, 2016. doi: 10.1007/s10472-015-9463-9.DOI
  105. [105]A. Y. Ng and M. I. Jordan. PEGASUS: A policy search method for large MDPs and POMDPs. In Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence (UAI), pages 406–415, 2000.
  106. [106]P. Abbeel, M. Quigley, and A. Y. Ng. Using inaccurate models in reinforcement learning. In Proceedings of the 23rd International Conference on Machine Learning (ICML), pages 1–8, 2006. doi: 10.1145/1143844.1143845.DOI
  107. [107]M. P. Deisenroth, G. Neumann, and J. Peters. A survey on policy search for robotics. Foundations and Trends in Robotics, 2(1–2):1–142, 2013. doi: 10.1561/2300000021.
  108. [108]Y. Gao, T. Yu, and J. Li. Bayesian optimization with local search. In Machine Learning, Optimization, and Data Science (LOD 2020), Lecture Notes in Computer Science, pages 350–361, Cham, 2020. Springer. doi: 10.1007/978-3-030-64580-9_30.DOI
  109. [109]L. Kyriakidis, M. A. Mendez, and M. Bähr. A hybrid algorithm based on Bayesian optimization and interior point optimizer for optimal operation of energy conversion systems. Energy, 312:133416, 2024. doi: 10.1016/j.energy.2024.133416.DOI
  110. [110]S. J. Reddi, S. Kale, and S. Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations (ICLR), 2018.
  111. [111]A. Griewank and A. Walther. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation. SIAM, Philadelphia, PA, 2nd edition, 2008. doi: 10.1137/1.9780898717761.DOI
  112. [112]A. O’Hagan. Curve fitting and optimal design for prediction. Journal of the Royal Statistical Society: Series B, 40 (1):1–24, 1978. doi: 10.1111/j.2517-6161.1978.tb01643.x.DOI
  113. [113]T. J. Santner, B. J. Williams, and W. I. Notz. The Design and Analysis of Computer Experiments. Springer, New York, 2nd edition, 2018. doi: 10.1007/978-1-4939-8847-1.DOI
  114. [114]M. Poloczek, J. Wang, and P. I. Frazier. Multi-information source optimization. In Advances in Neural Information Processing Systems, volume 30, pages 4288–4298, 2017.arxiv.org/abs/1603.00389
  115. [115]C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006. doi: 10.7551/mitpress/3206.001.0001.DOI
  116. [116]Y. Cao, M. A. Brubaker, D. J. Fleet, and A. Hertzmann. Efficient optimization for sparse gaussian process regression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(12):2415–2427, December 2015. doi: 10.1109/tpami.2015.2424873.DOI
  117. [117]V. Picheny, T. Wagner, and D. Ginsbourger. A benchmark of kriging-based infill criteria for noisy optimization. Structural and Multidisciplinary Optimization, 48 (3):607–626, 2013. doi: 10.1007/s00158-013-0919-4.DOI
  118. [118]D. Eriksson, M. Pearce, J. R. Gardner, R. Turner, and M. Poloczek. Scalable global optimization via local Bayesian optimization. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019.arxiv.org/abs/1910.01739
  119. [119]J. A. Frederik, R. Weber, S. Cacciola, F. Campagnolo, A. Croce, C. Bottasso, and J.-W. van Wingerden. Periodic dynamic induction control of wind farms: proving the potential in simulations and wind tunnel experiments. Wind Energy Science, 5(1):245–257, 2020. doi: 10.5194/wes-5-245-2020.DOI
  120. [120]P. Clark Di Leoni, A. Mazzino, and L. Biferale. Synchronization to big data: nudging the Navier–Stokes equations for data assimilation of turbulent flows. Physical Review X, 10(1):011023, 2020. doi: 10.1103/PhysRevX.10.011023.DOI
  121. [121]M. Zauner, V. Mons, O. Marquet, and B. Leclaire. Nudging-based data assimilation of the turbulent flow around a square cylinder. Journal of Fluid Mechanics, 937:A38, 2022. doi: 10.1017/jfm.2022.133.DOI

Paper details

vlnc:W_vnr3smh6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Engineering

Claude, ChatGPT

View statement
Looking for related papers…
Contents