betaFeedback

WorksMathematics

Active learning for data-driven reduced models of parametric differential systems with Bayesian operator inference

Abstract

This work develops an active learning framework to intelligently enrich data-driven reduced-order models (ROMs) of parametric dynamical systems, which can serve as the foundation of virtual assets in a digital twin. Data-driven ROMs are explainable, computationally efficient scientific machine learning models that aim to preserve the underlying physics of complex dynamical simulations. Since the quality of data-driven ROMs is sensitive to the quality of the limited training data, we seek to identify training parameters for which using the associated training data results in the best possible parametric ROM. Our approach uses the operator inference methodology, a regression-based strategy which can be tailored to particular parametric structure for a large class of problems. We establish a probabilistic version of parametric operator inference, casting the learning problem as a Bayesian linear regression. Prediction uncertainties stemming from the resulting probabilistic ROM solutions are used to design a sequential adaptive sampling scheme to select new training parameter vectors that promote ROM stability and accuracy globally in the parameter domain. We conduct numerical experiments for several nonlinear parametric systems of partial differential equations and compare the results to ROMs trained on random parameter samples. The results demonstrate that the proposed adaptive sampling strategy consistently yields more stable and accurate ROMs than random sampling does under the same computational budget.

Introduction

Numerical simulation of complex physical phenomena is a core enabling technology for digital twins, which are comprised of physical and virtual assets with a two-way flow of information: data from the physical asset is used to construct and/or calibrate the virtual asset (a numerical model), while numerical predictions from the virtual asset are used for control or decision-making for the physical asset [55]. To be viable for practical application, the virtual asset must be able to produce predictions rapidly and reliably; however, the underlying physics that are of interest for digital twin applications can typically only be accurately simulated using a large number of degrees of freedom, leading to computationally expensive numerical simulations. The explainability and computational efficiency of decisions made by the digital twin play a key role in safety-critical applications, making explainable artificial intelligence an essential ingredient [33]. Model reduction techniques are one such explainable scientific machine learning technique that construct low-dimensional systems, called reduced-order models (ROMs), to serve as computationally inexpensive surrogates for a high-dimensional physics simulation [6, 28]. This paper introduces a technique for adaptively constructing ROMs to emulate systems with parametric dependence, that is, systems whose behavior varies with some set of parameters, usually representing physical properties. We focus on systems where the parametric dependence manifests in the operators defining the model, not merely in initial conditions or external inputs. Building parametric ROMs requires training data at different parameter combinations, which can incur significant training computational cost. Moreover, state-of-the-art parametric ROMs—especially those learned directly from data—are not typically equipped with automated methods for determining how many and which parameters to select for training data generation within a limited training budget. Our goal is to develop a framework for intelligently selecting training parameters in order to learn a high-quality ROM with as few training data as possible, in particular in the context of data-driven model reduction.

Classical model reduction techniques, including those that are designed for parametric systems, intrusively construct a ROM by compressing a large numerical model, called the full-order model (FOM), through direct projection [6]. On the other hand, non-intrusive data-driven model reduction techniques, such as dynamic mode decomposition (DMD) [8, 65] and operator inference (OpInf) [28, 45, 57], construct a ROM by learning from available simulation data while imposing appropriate model structure without needing access to the implementation details of the FOM. The non-intrusive nature of these procedures eases the implementation burden and thus widens the applicability of model reduction. Since ROMs of parametric systems have application in large-scale optimization [28, 74], inverse problems [28, 48], and uncertainty quantification [13], data-driven parametric model reduction is a particularly attractive prospect in the context of the digital twin framework [46]. Examples of such techniques include the Loewner framework [38], interpolation-based methods [24, 77], parametric DMD [2, 19, 37, 49, 64], parametric OpInf [21, 42, 54, 72], Gaussian process surrogate modeling [13, 34], and deep learning [23, 25, 39]. Stability is a key issue for data-driven ROMs, and several recent works focus on embedding physical constraints or structure to promote or guarantee stability in OpInf ROMs. The work in [75] designs ROM structure to emulate precise coupling relationships between physical variables, while [3, 4] constrains learned operators to follow a hierarchical nested structure related to direct projection. Energy conservation for Hamiltonian systems is addressed in [32, 69, 72], second-order systems for structural or Lagrangian dynamics are examined in [17, 27, 68], and the works in [20, 29] obtain Lyapunov-based stability guarantees for quadratic ROMs. Other methods post-process the learned operators [63] or state predictions at runtime [43]; see also [44] for an analysis of stability of nonlinear polynomial models, including data-driven ROMs. Most of these stability-promoting methods are currently developed for non-parametric problems only. Our method leverages Tikhonov-regularized parametric OpInf [54], not only because of its parametric structure, but also due to a natural connection to Bayesian inference [41, 70].

Parametric data-driven models are only as good as the data they are trained on, and we can only obtain limited training data from expensive-to-evaluate complex systems due to training cost constraints. Therefore, what training data to obtain for optimal learning of the parametric ROMs within a given training budget is important to understand, since not all observations are equally useful. Active learning (or sequential optimal experimental design) is a way to select new training data given some existing knowledge about the current approximation, such as prediction uncertainty. In this work, the goal is to improve the global accuracy of the ROM by adaptively sampling the parameter space and, hence, selecting new training data. Some early work for active learning methods for global accuracy include maximizing mean squared error metrics [62], entropy [15, 62, 50], and maximizing the expected reduction in predictive variance [14] as acquisition functions for selecting the next training sample. The standard active learning literature uses Gaussian process regression (or Kriging) [61] as the surrogate model. There is limited work on active learning for adaptively updating parametric ROMs since they lack a measure of uncertainty in the predictions. Traditionally, greedy algorithms based on a posteriori error estimators were used to generate snapshots for adaptive reduced basis construction [9, 31, 30, 60]; however, these methods are limited to the context of intrusive ROMs with certain structure that give rise to the error estimators. Finally, there are recent works on adaptive sampling for parametric ROMs [7, 76] and for surrogate models fitted in the reduced input-output dimensions [36] that utilize additional surrogate models for error estimates to drive the parameter selection. In this work, we introduce an active learning method based on the prediction uncertainty from a probabilistic surrogate model, without the need for additional surrogates to provide ROM error estimates. Specifically, we propose a Bayesian parametric ROM equipped prediction uncertainty, which we use to design an active learning framework for adaptively selecting new training parameters. The major contributions of this work are (i) a Bayesian parametric OpInf ROM method for affine-parametric systems that provides uncertainty in spatiotemporal predictions with respect to underlying model parameters and (ii) acquisition functions targeting stability and global accuracy for active learning in ROMs.

We develop a probabilistic parametric ROM, building on our previous work in affine-parametric OpInf [54] and Bayesian OpInf [35]. We derive closed form expressions for the posterior distribution of the affine reduced operators for the proposed Bayesian parametric OpInf. The Bayesian parametric OpInf is a probabilistic ROM that provides a measure of uncertainty in the predictions. The statistics obtained from the posterior ROM are used to formulate the proposed acquisition functions of the probability of ROM instability and the total prediction variance. Our iterative procedure does not directly enforce global stability for the entire parameter space at each step, but such stability follows naturally within in few iterations due to our choice of acquisition functions. We present numerical experiments on a heat equation with nonlinear reaction term and a two-dimensional viscous Burgers’ problem for shock propagation. The results show that the proposed active learning methodology significantly results in satisfactory stability and accuracy at a fraction of the training computational cost when compared to random sampling of the parameter space.

The remainder of the paper is organized as follows. Section 2 describes the proposed Bayesian parametric ROM for affine-parametric systems. Section 3 uses the Bayesian parametric ROM to develop an active learning scheme for iteratively updating the training data for parametric OpInf ROMs. Implementation details and customization options are also discussed. The methodology is demonstrated on two problems in Section 4: a parametric diffusion-reaction equation with nonlinear reaction term, and a two-dimensional Burgers’ equation with variable wave speed. Finally, Section 5 makes some concluding remarks and outlines avenues for future work.

Bayesian parametric operator inference

This section briefly reviews operator inference for affine-parametric systems as developed in [54, 73] in Section 2.1. Then we present the proposed Bayesian parametric operator inference method that provides prediction uncertainties in Section 2.2. The technique gives rise to a regression problem, which is endowed with a probabilistic interpretation to form a Bayesian linear regression.

Non-intrusive reduced models for affine-parametric system

Let ξ∈P⊂Rp\xi\in\mathcal{P} \subset\mathbb{R}^{p} denote a vector of pp variables parameterizing a time-dependent partial differential equation (PDE). After introducing a spatial discretization, a finite-dimensional numerical model for the PDE may be written generically as

ddtq(t;ξ)=f(q(t),ξ),q(t0;ξ)=q0(ξ),t∈[t0,tf],(1)\frac{\mathrm{d}}{\mathrm{d}t}\mathbf{q}(t;\xi)=\mathbf{f}(\mathbf{q}(t),\xi),\qquad\mathbf{q}(t_{0};\xi)=\mathbf{q}_{0}(\xi),\qquad t\in[t_{0},t_{f}], \tag*{(1)}

where q(t;ξ)∈RN\mathbf{q}(t;\xi)\in\mathbb{R}^{N} is the time-dependent spatial discretization of the PDE state. This is the full-order model (FOM). The state depends implicitly on the parameter vector ξ\xi through the function f:RN×P→RN\mathbf{f}:\mathbb{R}^{N}\times\mathcal{P}\to\mathbb{R}^{N} which governs the dynamics, and through the initial condition q0(ξ)∈RN\mathbf{q}_{0}(\xi)\in\mathbb{R}^{N}. The function f\mathbf{f} may also depend on time-dependent system inputs, for example arising from nonstationary boundary conditions or source terms, but we neglect these for presentation clarity.

Many systems of interest can be written with the following parametric polynomial structure (we suppress the dependence of q(t;ξ)\mathbf{q}(t;\xi) on tt and ξ\xi going forward):

dqdt=c(ξ)+A(ξ)q+H(ξ)[q⊗q],(2)\frac{\mathrm{d}\mathbf{q}}{\mathrm{d}t}=\mathbf{c}(\xi)+\mathbf{A}(\xi)\mathbf{q}+\mathbf{H}(\xi)[\mathbf{q}\otimes\mathbf{q}], \tag*{(2)}
c(ξ)∈RN,A(ξ)∈RN×N,H(ξ)∈RN×N2,(3)\mathbf{c}(\xi)\in\mathbb{R}^{N},\qquad\mathbf{A}(\xi)\in\mathbb{R}^{N\times N},\qquad\mathbf{H}(\xi)\in\mathbb{R}^{N\times N^{2}}, \tag*{(3)}

where ⊗\otimes denotes the Kronecker product [71]. This work considers the class of problems where each of the “operators” in (2.2b) depend affinely on the parameter vector ξ\boldsymbol{\xi}, meaning there exist c(1),…,c(nc)∈RN\mathbf{c}^{(1)},\ldots,\mathbf{c}^{(n_c)} \in\mathbb{R}^{N}, A(1),…,A(na)∈RN×N\mathbf{A}^{(1)},\ldots,\mathbf{A}^{(n_a)} \in\mathbb{R}^{N\times N}, and H(1),…,H(nh)∈RN×N2\mathbf{H}^{(1)},\ldots,\mathbf{H}^{(n_h)} \in\mathbb{R}^{N\times N^2} such that

c(ξ)=∑ℓ=1ncθc(ℓ)(ξ)c(ℓ),A(ξ)=∑ℓ=1naθa(ℓ)(ξ)A(ℓ),\mathbf{c}(\boldsymbol{\xi}) = \sum_{\ell=1}^{n_c}\theta_c^{(\ell)}(\boldsymbol{\xi})\mathbf{c}^{(\ell)}, \qquad\mathbf{A}(\boldsymbol{\xi}) = \sum_{\ell=1}^{n_a}\theta_a^{(\ell)}(\boldsymbol{\xi})\mathbf{A}^{(\ell)},
H(ξ)=∑ℓ=1nhθh(ℓ)(ξ)H(ℓ),(4)\mathbf{H}(\boldsymbol{\xi}) = \sum_{\ell=1}^{n_h}\theta_h^{(\ell)}(\boldsymbol{\xi})\mathbf{H}^{(\ell)}, \tag*{(4)}

where each θx(ℓ):P→R\theta_x^{(\ell)}:\mathcal{P}\to\mathbb{R} is a scalar-valued function of the parameter vector. In many cases, including the numerical examples presented later, the number of terms ncn_c, nan_a, and nhn_h in each expansion is small.

State-polynomial, affine-parametric structure such as in (2.2) is advantageous for projection-based model reduction because linear dimensionality reduction preserves this structure. Given a matrix V∈RN×rV \in\mathbb{R}^{N\times r} with orthonormal columns, making the low-dimensional approximation q≈Vq^\mathbf{q}\approx V\hat{\mathbf{q}} for some q^=q^(t;ξ)∈Rr\hat{\mathbf{q}}=\hat{\mathbf{q}}(t;\boldsymbol{\xi})\in\mathbb{R}^{r} leads to the low-dimensional evolution equations ddtq^=VTf(Vq^,ξ)\frac{\mathrm{d}}{\mathrm{d}t}\hat{\mathbf{q}}=V^{T}\mathbf{f}(V\hat{\mathbf{q}},\boldsymbol{\xi}). For (2.2), this can be written as

dq^dt=c^(ξ)+A^(ξ)q^+H^(ξ)[q^⊗q^],(5)\frac{\mathrm{d}\hat{\mathbf{q}}}{\mathrm{d}t}=\hat{\mathbf{c}}(\boldsymbol{\xi})+\hat{\mathbf{A}}(\boldsymbol{\xi})\hat{\mathbf{q}}+\hat{\mathbf{H}}(\boldsymbol{\xi})[\hat{\mathbf{q}}\otimes\hat{\mathbf{q}}], \tag*{(5)}
c^(ξ)∈Rr,A^(ξ)∈Rr×r,H^(ξ)∈Rr×r2.(6)\hat{\mathbf{c}}(\boldsymbol{\xi})\in\mathbb{R}^{r},\qquad\hat{\mathbf{A}}(\boldsymbol{\xi})\in\mathbb{R}^{r\times r},\qquad\hat{\mathbf{H}}(\boldsymbol{\xi})\in\mathbb{R}^{r\times r^{2}}. \tag*{(6)}

Critically, the affine-parametric structure (2.2c) is preserved in the reduced system matrices: there exist c^(1),…,c^(nc)∈Rr\hat{\mathbf{c}}^{(1)},\ldots,\hat{\mathbf{c}}^{(n_c)}\in\mathbb{R}^{r}, A^(1),…,A^(na)∈Rr×r\hat{\mathbf{A}}^{(1)},\ldots,\hat{\mathbf{A}}^{(n_a)}\in\mathbb{R}^{r\times r}, and H^(1),…,H^(nh)∈Rr×r2\hat{\mathbf{H}}^{(1)},\ldots,\hat{\mathbf{H}}^{(n_h)}\in\mathbb{R}^{r\times r^{2}} with

c^(ξ)=∑ℓ=1ncθc(ℓ)(ξ)c^(ℓ),A^(ξ)=∑ℓ=1naθa(ℓ)(ξ)A^(ℓ),\hat{\mathbf{c}}(\boldsymbol{\xi})=\sum_{\ell=1}^{n_c}\theta_c^{(\ell)}(\boldsymbol{\xi})\hat{\mathbf{c}}^{(\ell)},\qquad\hat{\mathbf{A}}(\boldsymbol{\xi})=\sum_{\ell=1}^{n_a}\theta_a^{(\ell)}(\boldsymbol{\xi})\hat{\mathbf{A}}^{(\ell)},
H^(ξ)=∑ℓ=1nhθh(ℓ)(ξ)H^(ℓ),(7)\hat{\mathbf{H}}(\boldsymbol{\xi})=\sum_{\ell=1}^{n_h}\theta_h^{(\ell)}(\boldsymbol{\xi})\hat{\mathbf{H}}^{(\ell)}, \tag*{(7)}

where each expansion has the same number of terms nxn_x and the same coefficient functions θx(ℓ)\theta_x^{(\ell)} as in (2.2c). This means that if the polynomial form and the affine-parametric structure of a high-dimensional model are known, we can stipulate the same structure in a low-dimensional space to design an effective ROM. The main advantage is that if r≪Nr\ll N, solving (2.3) is much less computationally expensive than solving (2.2).

Remark 2.1 (Dimensionality reduction). The linearity of the approximation q≈Vq^\mathbf{q}\approx V\hat{\mathbf{q}} is the reason that the structure of (2.2) is preserved in (2.3). If a shifted approximation is made, i.e., q≈Vq^+qˉ\mathbf{q}\approx V\hat{\mathbf{q}}+\bar{\mathbf{q}} for a fixed nonzero reference vector qˉ∈RN\bar{\mathbf{q}}\in\mathbb{R}^{N}, additional terms arise in the reduced dynamics, but the new structure is straightforward to derive. The situation is further complicated by partially nonlinear approximations, such as quadratic manifolds [5, 26, 40, 67], but the model form can usually still be written out explicitly. However, when using a fully nonlinear approximation, for example with autoencoders [11, 12, 47], it is no longer straightforward to write down the expected parametric structure of the reduced dynamics.

Operator inference (OpInf) [28, 45, 57] is a non-intrusive strategy for learning ROMs with polynomial structure. In the original work [57], parametric models of the form (2.2a)–(2.2b) are constructed by learning nonparametric models for each of several training parameter instances, then the learned models are interpolated in parameter space to extrapolate to new parameter instances. Later, [54, 73] extended the methodology to the case of affine-parametric structure as in (2.2c). Affine-parametric OpInf (pOpInf) learns ROMs that can be written generally as

dq^dt=O^d(q^;ξ),q^(t0)=q^0,t∈[t0,tf],(8)\frac{\mathrm{d}\hat{\mathbf{q}}}{\mathrm{d}t}=\hat{\mathbf{O}}\mathrm{d}(\hat{\mathbf{q}};\boldsymbol{\xi}),\qquad\hat{\mathbf{q}}(t_0)=\hat{\mathbf{q}}_0,\qquad t\in[t_0,t_f], \tag*{(8)}

Bayesian framework for affine-parametric operator inference

where the operator matrix O^∈Rr×d(r)\widehat{\mathbf{O}} \in\mathbb{R}^{r \times d(r)} contains the (parameter-independent) vectors and matrices defining the ROM and d(q^;ξ)∈Rd(r)\mathbf{d}(\widehat{\mathbf{q}};\boldsymbol{\xi}) \in\mathbb{R}^{d(r)} encodes the polynomial and parametric structure of the system. In the case of (2.3), we have

O^=[c^(1)⋯c^(nc)A^(1)⋯A^(na)H^(1)⋯H^(nh)],(9)\widehat{\mathbf{O}} = \left[ \begin{array}{cccccccc} \widehat{\mathbf{c}}^{(1)} & \cdots& \widehat{\mathbf{c}}^{(n_c)} & \widehat{\mathbf{A}}^{(1)} & \cdots& \widehat{\mathbf{A}}^{(n_a)} & \widehat{\mathbf{H}}^{(1)} & \cdots& \widehat{\mathbf{H}}^{(n_h)} \end{array} \right], \tag*{(9)}
d(q^,u;ξ)=[θc(1)(ξ)⋯θc(nc)(ξ)θa(1)(ξ)q^T⋯θh(nh)(ξ)[q^⊗q^]T]T,(10)\mathbf{d}(\widehat{\mathbf{q}},\mathbf{u};\boldsymbol{\xi}) = \left[ \theta_c^{(1)}(\boldsymbol{\xi}) \quad\cdots\quad\theta_c^{(n_c)}(\boldsymbol{\xi}) \quad\theta_a^{(1)}(\boldsymbol{\xi})\widehat{\mathbf{q}}^{\mathsf{T}} \quad\cdots\quad\theta_h^{(n_h)}(\boldsymbol{\xi})[\widehat{\mathbf{q}}\otimes\widehat{\mathbf{q}}]^{\mathsf{T}} \right]^{\mathsf{T}}, \tag*{(10)}

with d(r)=nc+rna+r2nhd(r) = n_c + r n_a + r^2 n_h. The vector-valued function d\mathbf{d} is known a priori because the structure of the ROM (2.4) is stipulated to match the structure of a classical projection-based ROM, which, in the case being examined, inherits the structure of the original model. The operator matrix O^\widehat{\mathbf{O}}, on the other hand, is not known a priori in cases where the high-dimensional operators (e.g., A(1),…,A(na)\mathbf{A}^{(1)}, \ldots, \mathbf{A}^{(n_a)}) are unknown or inaccessible. The goal is to learn an appropriate O^\widehat{\mathbf{O}} from data, as we now explain.

Consider a limited number of training parameter instances ξ1,…,ξnp\boldsymbol{\xi}_1,\ldots,\boldsymbol{\xi}_{n_p} for which solution data are available from the FOM (2.2). Let {qij}j=1nt\{\mathbf{q}_{ij}\}_{j=1}^{n_t} be snapshots of the state solution of (2.2) at ntn_t time instances t1,…,tntt_1,\ldots,t_{n_t} with parameter vector ξ=ξi\boldsymbol{\xi}=\boldsymbol{\xi}_i. These training states are compressed to produce reduced snapshots q^ij=VTqij\widehat{\mathbf{q}}_{ij}=\mathbf{V}^{\mathsf{T}}\mathbf{q}_{ij} and then used to estimate time derivatives q^˙ij≈ddtq(t;ξi)∣t=tj\dot{\widehat{\mathbf{q}}}_{ij} \approx\left.\frac{\mathrm{d}}{\mathrm{d}t}\mathbf{q}(t;\boldsymbol{\xi}_i)\right|_{t=t_j}, for example with finite differences. The operator matrix is selected by solving the regression problem

min⁡O^∑i=1np∑j=1nt∥O^d(q^ij;ξi)−q^˙ij∥22,(11)\min_{\widehat{\mathbf{O}}} \sum_{i=1}^{n_p}\sum_{j=1}^{n_t}\left\|\widehat{\mathbf{O}}\mathbf{d}(\widehat{\mathbf{q}}_{ij};\boldsymbol{\xi}_i)-\dot{\widehat{\mathbf{q}}}_{ij}\right\|_2^2, \tag*{(11)}

which minimizes the residual of (2.4) with respect to available observations. Letting di,j=d(q^ij,ξi)\mathbf{d}_{i,j}=\mathbf{d}(\widehat{\mathbf{q}}_{ij},\boldsymbol{\xi}_i) and defining the data matrices

Q^˙=[q^˙1,1⋯q^˙1,ntq^˙2,1⋯q^˙np,nt]∈Rr×npnt,(12)\dot{\widehat{\mathbf{Q}}} = \left[\begin{array}{cccccc} \dot{\widehat{\mathbf{q}}}_{1,1} & \cdots& \dot{\widehat{\mathbf{q}}}_{1,n_t} & \dot{\widehat{\mathbf{q}}}_{2,1} & \cdots& \dot{\widehat{\mathbf{q}}}_{n_p,n_t} \end{array}\right] \in\mathbb{R}^{r \times n_p n_t}, \tag*{(12)}
D=[d1,1⋯d1,ntd2,1⋯dnp,nt]T∈Rnpnt×d(r),(13)\mathbf{D} = \left[\begin{array}{cccccc} \mathbf{d}_{1,1} & \cdots& \mathbf{d}_{1,n_t} & \mathbf{d}_{2,1} & \cdots& \mathbf{d}_{n_p,n_t} \end{array}\right]^{\mathsf{T}} \in\mathbb{R}^{n_p n_t \times d(r)}, \tag*{(13)}

we can write (2.6a) in the standard linear least-squares form

min⁡O^∥DO^T−Q^˙T∥F2.(14)\min_{\widehat{\mathbf{O}}} \left\|\mathbf{D}\widehat{\mathbf{O}}^{\mathsf{T}}-\dot{\widehat{\mathbf{Q}}}^{\mathsf{T}}\right\|_{\mathrm{F}}^2. \tag*{(14)}

Furthermore, this regression decouples into rr independent least-squares problems. Letting o^k∈Rd(r)\widehat{\mathbf{o}}_k \in\mathbb{R}^{d(r)} and zk∈Rnpnt\mathbf{z}_k \in\mathbb{R}^{n_p n_t} denote the kk-th rows of O^\widehat{\mathbf{O}} and Q^˙\dot{\widehat{\mathbf{Q}}}, respectively, (2.6d) can be written as

min⁡o^k∥Do^k−zk∥22,k=1,…,r,\min_{\widehat{\mathbf{o}}_k}\left\|\mathbf{D}\widehat{\mathbf{o}}_k-\mathbf{z}_k\right\|_2^2,\qquad k=1,\ldots,r,
O^=[o^1⋯o^r]T,\widehat{\mathbf{O}}=\left[\begin{array}{ccc}\widehat{\mathbf{o}}_1 & \cdots& \widehat{\mathbf{o}}_r\end{array}\right]^{\mathsf{T}},
Q^˙=[z1⋯zr]T,(15)\dot{\widehat{\mathbf{Q}}}=\left[\begin{array}{ccc}\mathbf{z}_1 & \cdots& \mathbf{z}_r\end{array}\right]^{\mathsf{T}}, \tag*{(15)}

where each subproblem can be solved independent of the others.

Data-driven ROMs (2.4) generated through the pOpInf regression (2.6) are not guaranteed to be stable. One standard method for promoting model stability is to add a regularization term to the regression, replacing (2.6e) with, for instance, a Tikhonov-regularized (ridge) regression,

min⁡o^k{∥Do^k−zk∥22+∥Γko^k∥22},k=1,…,r,(16)\min_{\widehat{\mathbf{o}}_k}\left\{\left\|\mathbf{D}\widehat{\mathbf{o}}_k-\mathbf{z}_k\right\|_2^2+\left\|\mathbf{\Gamma}_k\widehat{\mathbf{o}}_k\right\|_2^2\right\},\qquad k=1,\ldots,r, \tag*{(16)}

where Γk∈Rd(r)×d(r)\mathbf{\Gamma}_k \in\mathbb{R}^{d(r)\times d(r)}. The regularization matrix Γk\mathbf{\Gamma}_k is often parameterized with a small number of hyperparameters and designed so that like terms are penalized equally [53, 59, 63]. Alternative strategies for promoting or even enforcing stability typically involve constraining the optimization problem (2.6), for example to preserve certain structure [3, 17, 20, 27, 29, 32, 69, 72, 75]. The lack of constraints in (2.7) and our choice of Tikhonov regularization does not guarantee stability, but it admits a critical connection to Bayesian inference [41, 70] that enables a framework for constructing probabilistic ROMs.

Bayesian framework for affine-parametric operator inference

We now present the Bayesian parametric OpInf method that can account for underlying model errors to provide prediction uncertainties. The reduced snapshot time derivatives q^˙ij\dot{\hat{\mathbf{q}}}_{ij} are typically estimated from the reduced snapshots q^ij\hat{\mathbf{q}}_{ij}, incurring errors to which the regression problems (2.6) and (16) are sensitive. Modeling the estimation error as uncorrelated Gaussian noise, we obtain the Bayesian linear regression

zk=Do^k+ϵk,p(ϵk)=N(ϵk∣0npnt,σk2I),k=1,…,r,(17)\mathbf{z}_k = \mathbf{D}\hat{\mathbf{o}}_k + \boldsymbol{\epsilon}_k,\qquad p(\boldsymbol{\epsilon}_k) = \mathcal{N}(\boldsymbol{\epsilon}_k \mid\mathbf{0}_{n_p n_t},\sigma_k^2\mathbf{I}),\qquad k=1,\ldots,r, \tag*{(17)}

in which 0npnt∈Rnpnt\mathbf{0}_{n_p n_t} \in\mathbb{R}^{n_p n_t} denotes the zero vector and I\mathbf{I} is the npnt×npntn_p n_t \times n_p n_t identity matrix. Each σk2>0\sigma_k^2 > 0 describes the noise variance in the regression for o^k\hat{\mathbf{o}}_k, the kk-th row of the operator matrix O^\hat{\mathbf{O}}. The vector ϵk∈Rnpnt\boldsymbol{\epsilon}_k \in\mathbb{R}^{n_p n_t} corresponds to noise over the ntn_t time instances for each of the npn_p training parameter samples. The noise is not correlated in time or across parameter values. The formulation builds on our previous work in [35]. The noise model in (17) assumes that the errors in time-derivatives are uncorrelated across time and parameter values. While we estimate the time-derivatives non-sequentially from the data, finite difference approximations generally share neighboring observations that introduce some correlation in the time-derivative errors. However, in this work, we adopt the independent Gaussian noise model as a simplifying approximation (i.e., without defining a specific temporal correlation structure in the trajectories), enabling computationally tractable Bayesian inference.

We immediately have a Gaussian likelihood function for o^k\hat{\mathbf{o}}_k given data in D\mathbf{D}:

πlike(o^k)=p(zk∣D,o^k,σk2)=N(zk∣Do^k,σk2I)\pi_{\mathrm{like}}(\hat{\mathbf{o}}_k)=p(\mathbf{z}_k\mid\mathbf{D},\hat{\mathbf{o}}_k,\sigma_k^2)=\mathcal{N}(\mathbf{z}_k\mid\mathbf{D}\hat{\mathbf{o}}_k,\sigma_k^2\mathbf{I})
∝exp⁡(−12σk2∥Do^k−zk∥22).(18)\propto\exp\left(-\frac{1}{2\sigma_k^2}\left\|\mathbf{D}\hat{\mathbf{o}}_k-\mathbf{z}_k\right\|_2^2\right). \tag*{(18)}

To facilitate a Bayesian inference, we impose a Gaussian prior on the operator vector, namely

πprior(o^k)=p(o^k∣γk,σk2)=N(o^k∣0d(r),σk2diag⁡(γk)−1)\begin{aligned} \pi_{\mathrm{prior}}(\hat{\mathbf{o}}_k)=p(\hat{\mathbf{o}}_k\mid\boldsymbol{\gamma}_k,\sigma_k^2) \\ &=\mathcal{N}\left(\hat{\mathbf{o}}_k\mid\mathbf{0}_{d(r)},\sigma_k^2\operatorname{diag}(\boldsymbol{\gamma}_k)^{-1}\right) \end{aligned}
∝exp⁡(−12σk2o^kTdiag⁡(γk)o^k)=exp⁡(−12σk2∥diag⁡(γk)1/2o^k∥22).(19)\propto\exp\left(-\frac{1}{2\sigma_k^2}\hat{\mathbf{o}}_k^{\mathsf{T}}\operatorname{diag}(\boldsymbol{\gamma}_k)\hat{\mathbf{o}}_k\right)=\exp\left(-\frac{1}{2\sigma_k^2}\left\|\operatorname{diag}(\boldsymbol{\gamma}_k)^{1/2}\hat{\mathbf{o}}_k\right\|_2^2\right). \tag*{(19)}

where γ1,…,γr∈Rd(r)\gamma_1,\ldots,\gamma_r \in\mathbb{R}^{d(r)}, 0d(r)∈Rd(r)\mathbf{0}_{d(r)} \in\mathbb{R}^{d(r)} is the zero vector, and σk2diag⁡(γk)−1\sigma_k^2\operatorname{diag}(\boldsymbol{\gamma}_k)^{-1} is the prior covariance for o^k\hat{\mathbf{o}}_k, k=1,…,rk=1,\ldots,r. The scalar likelihood variance σk2\sigma_k^2 is included in the prior for convenience in matching to terms in (18). Equipping the prior with a diagonal covariance implies that the entries of the operator vector o^k\hat{\mathbf{o}}_k are uncorrelated, and the zero prior mean disincentives the entries of o^k\hat{\mathbf{o}}_k from becoming too large. Note that this also follows from the explicit connection between Tikhonov regularization and Bayesian linear regression [41, 70].

The goal now is to establish a probability distribution for o^k\hat{\mathbf{o}}_k given training data for D\mathbf{D} and zk\mathbf{z}_k. Applying Bayes’ rule, we obtain

πpost(o^k)∝πlike(o^k)πprior(o^k)\pi_{\mathrm{post}}(\hat{\mathbf{o}}_k)\propto\pi_{\mathrm{like}}(\hat{\mathbf{o}}_k)\pi_{\mathrm{prior}}(\hat{\mathbf{o}}_k)
∝exp⁡(−12σk2∥Do^k−zk∥22−12σk2∥diag⁡(γk)1/2o^k∥22)\propto\exp\left(-\frac{1}{2\sigma_k^2}\left\|\mathbf{D}\hat{\mathbf{o}}_k-\mathbf{z}_k\right\|_2^2-\frac{1}{2\sigma_k^2}\left\|\operatorname{diag}(\boldsymbol{\gamma}_k)^{1/2}\hat{\mathbf{o}}_k\right\|_2^2\right)
∝exp⁡(−12(o^k−μk)TΣk−1(o^k−μk)),(20)\propto\exp\left(-\frac{1}{2}(\hat{\mathbf{o}}_k-\boldsymbol{\mu}_k)^{\mathsf{T}}\boldsymbol{\Sigma}_k^{-1}(\hat{\mathbf{o}}_k-\boldsymbol{\mu}_k)\right), \tag*{(20)}

that is, πpost(o^k)∝N(o^k∣μk,Σk)\pi_{\mathrm{post}}(\hat{\mathbf{o}}_k)\propto\mathcal{N}(\hat{\mathbf{o}}_k\mid\boldsymbol{\mu}_k,\boldsymbol{\Sigma}_k), with posterior mean and covariance

μk=(DTD+diag⁡(γk))−1DTzk=argmin⁡μ{∥Dμ−zk∥22+∥Γkμ∥22},(21)\boldsymbol{\mu}_k=(\mathbf{D}^{\mathsf{T}}\mathbf{D}+\operatorname{diag}(\boldsymbol{\gamma}_k))^{-1}\mathbf{D}^{\mathsf{T}}\mathbf{z}_k=\underset{\boldsymbol{\mu}}{\operatorname{argmin}}\left\{\|\mathbf{D}\boldsymbol{\mu}-\mathbf{z}_k\|_2^2+\|\boldsymbol{\Gamma}_k\boldsymbol{\mu}\|_2^2\right\}, \tag*{(21)}
Σk=σk2(DTD+diag⁡(γk))−1=σk2(DTD+ΓkTΓk)−1.(22)\boldsymbol{\Sigma}_k=\sigma_k^2(\mathbf{D}^{\mathsf{T}}\mathbf{D}+\operatorname{diag}(\boldsymbol{\gamma}_k))^{-1}=\sigma_k^2(\mathbf{D}^{\mathsf{T}}\mathbf{D}+\boldsymbol{\Gamma}_k^{\mathsf{T}}\boldsymbol{\Gamma}_k)^{-1}. \tag*{(22)}

Algorithm 2.1 Bayesian affine-parametric operator inference.

1: procedure pBayesOpInf(
    Training parameter vectors $\xi_1,\ldots,\xi_{n_p}\in\mathcal{P}$,
    Compressed states $\{\hat{\mathbf{q}}_{ij}\}_{j=1}^{n_t}\subset\mathbb{R}^r$ corresponding to $\xi_i$ for $i=1,\ldots,n_p$,
    Parametric ROM structure function $\mathbf{d}=\mathbf{d}(\hat{\mathbf{q}};\xi)$,
    Number of posterior draws $n_d>0$
)
    # Set up the OpInf regression.
2:  $\dot{\hat{\mathbf{Q}}}=[\mathbf{z}_1\ \cdots\ \mathbf{z}_r]^{\mathsf{T}}\leftarrow\operatorname{ddt}(\hat{\mathbf{q}}_{ij}\ \forall i,j)$                                      # Time derivatives {xref}`(2.6b)`.
3:  $\mathbf{D}\leftarrow\operatorname{datamatrix}(\mathbf{d}(\hat{\mathbf{q}}_{ij};\xi_i)\ \forall i,j)$                                           # Data matrix {xref}`(2.6c)`.
4:  $\mathbf{\Gamma}\leftarrow\operatorname{regselect}(\hat{\mathbf{q}}_{ij},\xi_{ij}\ \forall i,j)$                                               # Select regularization matrix.

    # Construct the operator matrix distribution.
5:  for $k=1,\ldots,r$ do
6:      $\boldsymbol{\mu}_k\leftarrow\underset{\boldsymbol{\mu}}{\operatorname{argmin}}\{\lVert\mathbf{D}\boldsymbol{\mu}-\mathbf{z}_k\rVert_2^2+\lVert\mathbf{\Gamma}\boldsymbol{\mu}\rVert_2^2\}$                                      # Posterior mean for $\hat{\mathbf{o}}_k$.
7:      $\sigma_k^2\leftarrow\frac{1}{n_pn_t}\left(\lVert\mathbf{D}\boldsymbol{\mu}_k-\mathbf{z}_k\rVert_2^2+\lVert\mathbf{\Gamma}\boldsymbol{\mu}_k\rVert_2^2\right)$
8:      $\mathbf{\Sigma}_k\leftarrow\sigma_k^2(\mathbf{D}^{\mathsf{T}}\mathbf{D}+\mathbf{\Gamma}^{\mathsf{T}}\mathbf{\Gamma})^{-1}$                                      # Posterior covariance for $\hat{\mathbf{o}}_k$.

    # Draw from the operator matrix distribution.
9:  for $\ell=1,\ldots,n_d$ do
10:     for $k=1,\ldots,r$ do
11:         $\hat{\mathbf{o}}_k^{(\ell)}\leftarrow\operatorname{draw\ from}\mathcal{N}(\boldsymbol{\mu}_k,\mathbf{\Sigma}_k)$
12:     $\hat{\mathbf{O}}^{(\ell)}\leftarrow[\hat{\mathbf{o}}_1^{(\ell)}\ \cdots\ \hat{\mathbf{o}}_r^{(\ell)}]^{\mathsf{T}}$
13: return $\hat{\mathbf{O}}^{(1)},\ldots,\hat{\mathbf{O}}^{(n_d)}$

wherein Γk=diag⁡(γk)1/2\mathbf{\Gamma}_k=\operatorname{diag}(\gamma_k)^{1/2}. A maximum marginal likelihood argument yields optimal variance values

σk2=1npnt(∥Dμk−zk∥22+∥Γkμk∥22),(23)\sigma_k^2=\frac{1}{n_pn_t}\left(\lVert\mathbf{D}\boldsymbol{\mu}_k-\mathbf{z}_k\rVert_2^2+\lVert\mathbf{\Gamma}_k\boldsymbol{\mu}_k\rVert_2^2\right), \tag*{(23)}

see [35] for the derivation of this estimator, noting that μk\boldsymbol{\mu}_k solves the regression (16) with regularization matrix Γk\mathbf{\Gamma}_k. It is also worth pointing out that, though the Bayesian framework in this subsection shares the same mathematical derivation as that in [35], the assembly of O^\hat{\mathbf{O}} takes the affine parameter-dependency into account, and the data matrices D\mathbf{D} and Q^˙\dot{\hat{\mathbf{Q}}} include solution data evaluated at different parameter vectors.

The definitions in (2.12) establish a probability distribution for the operator matrix O^\hat{\mathbf{O}} in the parametric model (8). To generate a single sample from this distribution, draw o^k(1)\hat{\mathbf{o}}_k^{(1)} from N(μk,Σk)\mathcal{N}(\boldsymbol{\mu}_k,\mathbf{\Sigma}_k) for each k=1,…,rk=1,\ldots,r and stack the results to form O^(1)=[o^1(1) ⋯ o^r(1)]T\hat{\mathbf{O}}^{(1)}=[\hat{\mathbf{o}}_1^{(1)}\ \cdots\ \hat{\mathbf{o}}_r^{(1)}]^{\mathsf{T}}. Note that this is a computationally inexpensive task. Since the vector-valued function d(q^;ξ)\mathbf{d}(\hat{\mathbf{q}};\xi) is pre-specified, fixing O^=O^(1)\hat{\mathbf{O}}=\hat{\mathbf{O}}^{(1)} fully defines (8) as a deterministic ROM which can be solved for any parameter vector ξ∈P\xi\in\mathcal{P}. On the other hand, considering O^\hat{\mathbf{O}} as a matrix-valued random variable defines (8) as a probabilistic ROM whose state solution, which we denote q^rom(t;ξ)\hat{\mathbf{q}}_{\mathrm{rom}}(t;\xi), is a stochastic process. The moments of the state solution can be estimated efficiently with Monte Carlo sampling and will be used shortly to drive our active learning procedure.

The only outstanding information needed to construct a probabilistic ROM as described above is the prior hyperparameters γ1,…,γr\gamma_1,\ldots,\gamma_r or, equivalently, the diagonal regularization matrices Γ1,…,Γr\mathbf{\Gamma}_1,\ldots,\mathbf{\Gamma}_r. To select these, we adapt the approaches from [35, 52, 53, 54], which set γ1=γ2=⋯=γr\gamma_1=\gamma_2=\cdots=\gamma_r and parameterize Γ=diag⁡(γk)\mathbf{\Gamma}=\operatorname{diag}(\gamma_k) so that nonlinear terms are regularized separately from remaining terms. In the quadratic model (2.3) with operator matrix (9), this results in the total regularization term

∑k=1r∥O^k∥22=λ1(∑ℓ=1nc∥c^ℓ∥22+∑ℓ=1na∥A^ℓ∥F2)+λ2∑ℓ=1nh∥H^ℓ∥F2.(24)\sum_{k=1}^{r}\left\lVert\widehat{\mathbf{O}}_{k}\right\rVert_{2}^{2} =\lambda_{1}\left(\sum_{\ell=1}^{n_{c}}\left\lVert\widehat{\mathbf{c}}_{\ell}\right\rVert_{2}^{2} +\sum_{\ell=1}^{n_{a}}\left\lVert\widehat{\mathbf{A}}_{\ell}\right\rVert_{F}^{2}\right) +\lambda_{2}\sum_{\ell=1}^{n_{h}}\left\lVert\widehat{\mathbf{H}}_{\ell}\right\rVert_{F}^{2}. \tag*{(24)}

The scalars λ1,λ2≥0\lambda_{1},\lambda_{2} \ge0 are selected through a grid search that prioritizes stability. Given a pair of candidate regularization values (λ1,λ2)(\lambda_{1},\lambda_{2}), we construct the operator matrix posterior (20)–(2.12) and, for each training parameter vector ξi\boldsymbol{\xi}_{i}, sample the posterior nd>0n_{d} > 0 times and solve the corresponding deterministic ROMs at that parameter vector. If any of the ndn_{d} draws result in an unstable ROM, the candidate pair is discarded. In this context, stability refers to numerical stability over the time domain of interest, measured by whether the ROM state exceeds a large predetermined value. If all ROM solutions are stable in this sense, they are averaged over the draws and compared to the training snapshots. More precisely, we draw operator matrices O^(1),…,O^(nd)\widehat{\mathbf{O}}^{(1)}, \ldots, \widehat{\mathbf{O}}^{(n_{d})} and estimate the process mean with the sample mean

E[q^rom(t;ξi)]≈q^‾rom(t;ξi)=1nd∑ℓ=1ndq^(t;ξi;O^(ℓ)),(25)\mathbb{E}\left[\widehat{\mathbf{q}}_{\mathrm{rom}}(t;\boldsymbol{\xi}_{i})\right]\approx\overline{\widehat{\mathbf{q}}}_{\mathrm{rom}}(t;\boldsymbol{\xi}_{i})=\frac{1}{n_{d}}\sum_{\ell=1}^{n_{d}}\widehat{\mathbf{q}}\left(t;\boldsymbol{\xi}_{i};\widehat{\mathbf{O}}^{(\ell)}\right), \tag*{(25)}

where q^(t;ξi;O^(ℓ))\widehat{\mathbf{q}}(t;\boldsymbol{\xi}_{i};\widehat{\mathbf{O}}^{(\ell)}) is the solution to (8) with O^=O^(ℓ)\widehat{\mathbf{O}}=\widehat{\mathbf{O}}^{(\ell)} and ξ=ξi\boldsymbol{\xi}=\boldsymbol{\xi}_{i}. Then, we compute the average relative training reconstruction error of the sample mean,

Etrain=1npnt∑i=1np∑j=1nt∥q^i,j−q^‾rom(tj;ξi)∥22∥q^i,j∥22.(26)\mathcal{E}_{\mathrm{train}}=\frac{1}{n_{p}n_{t}}\sum_{i=1}^{n_{p}}\sum_{j=1}^{n_{t}}\frac{\left\lVert\widehat{\mathbf{q}}_{i,j}-\overline{\widehat{\mathbf{q}}}_{\mathrm{rom}}(t_{j};\boldsymbol{\xi}_{i})\right\rVert_{2}^{2}}{\left\lVert\widehat{\mathbf{q}}_{i,j}\right\rVert_{2}^{2}}. \tag*{(26)}

The regularization hyperparameters (λ1,λ2)(\lambda_{1},\lambda_{2}) that result in the lowest training error (26) without any unstable draws are selected to define the final probabilistic ROM. This procedure can be carried out through a grid search and fine tuned with derivative-free optimization techniques. The approach balances stability with accuracy at the training parameter vectors: though there is no guarantee that all possible draws of the probabilistic ROM will be stable at the training parameter vectors, we observe empirically in the numerical experiments that if a modest nd=20n_{d}=20 draws are stable, additional draws tend to be stable. However, stability at the training parameter vectors does not guarantee stability at non-training parameter vectors.

Algorithm 2.1 details the procedure for constructing and sampling from the posterior operator matrix distribution (20). The regselect() procedure in section 2.2 represents the regularization selection process described in the previous paragraph.

Active learning for Bayesian parametric OpInf ROMs

The performance quality of data-driven models, including operator inference ROMs, depends strongly on the quality of the training data. In the model reduction setting, acquiring new training data amounts to either solving an expensive FOM or performing a physical experiment and collecting observations. The goal of this section is to present a novel active learning method that can use Bayesian pOpInf to iteratively determine new points in the parameter domain such that new training data corresponding to these points are optimal in some sense, allowing us to learn accurate pOpInf ROMs with as little training data as is practical and provide prediction uncertainties.

Acquisition functions for new training parameters

Given a probabilistic pOpInf ROM constructed through Algorithm 2.1, our goal is to choose a new parameter vector, not yet used for training, at which to solve the FOM and supplement the training set. There are two main considerations to use as selection criteria: (i) ROM stability and (ii) global accuracy of predictions. The regularization selection procedure promotes ROM stability at the training parameter vectors. However, we

Algorithm 3.1 Posterior sampling and acquisition evaluation with parametric BayesOpInf.

  1. procedure BayesOpInfAcquisition( Test parameter vector ξ~\tilde{\boldsymbol{\xi}}, Operator matrix realizations O^(1),…,O^(nd)∈Rr×d(r)\widehat{\mathbf{O}}^{(1)},\ldots,\widehat{\mathbf{O}}^{(n_d)}\in\mathbb{R}^{r\times d(r)}, Parametric ROM structure function d=d(q^;ξ)\mathbf{d}=\mathbf{d}(\hat{\mathbf{q}};\boldsymbol{\xi}), )

  2. nα←0n_{\alpha}\leftarrow0    # Counter for number of unstable draws.

  3. S←{}S\leftarrow\{\}    # Set for the indices of stable draws.

  4. for ℓ=1,…,nd\ell=1,\ldots,n_d do

  5.     try

  6.         Q~(ℓ)←integrate dq^dt=O^(ℓ)d(q^(t);ξ~)\widetilde{\mathbf{Q}}^{(\ell)}\leftarrow\text{integrate }\frac{\mathrm{d}\hat{\mathbf{q}}}{\mathrm{d}t}=\widehat{\mathbf{O}}^{(\ell)}\mathbf{d}(\hat{\mathbf{q}}(t);\tilde{\boldsymbol{\xi}})

  7.         Add ℓ\ell to the indexing set SS

  8.     except time integration is numerically unstable

  9.         nα←nα+1n_{\alpha}\leftarrow n_{\alpha}+1

  10. α←nα/nd\alpha\leftarrow n_{\alpha}/n_d    # Probability of ROM instability.

  11. if α=100%\alpha=100\% (all ndn_d time integrations unstable) then

  12.     ω←None\omega\leftarrow\text{None}    # No accuracy acquisition values.

  13. else

  14.     ω←φ^(ξ~;{Q~(ℓ)}ℓ∈S)\omega\leftarrow\hat{\varphi}(\tilde{\boldsymbol{\xi}};\{\widetilde{\mathbf{Q}}^{(\ell)}\}_{\ell\in S})    # Global accuracy acquisition value (30).

  15. return α,ω\alpha,\omega

do not have guarantees for ROM performance in terms of accuracy, uncertainty, or stability at non-training parameter vectors, nor do we know beforehand how adding a particular parameter vector to the training set might improve ROM performance throughout the parameter domain. We choose parameter vectors based on two proposed active learning acquisition functions that target the stability and the variance of the current Bayesian parametric ROM.

The first acquisition function, ψ:P→[0,1]\psi:\mathcal{P}\to[0,1], is the probability of ROM instability, since an unstable surrogate model cannot be used for prediction or uncertainty estimation. Determining the stability of a probabilistic ROM analytically is challenging, but for a fixed parameter vector ξ~\tilde{\boldsymbol{\xi}}, stability can be estimated empirically by drawing nd>0n_d>0 ROM samples, integrating each sampled deterministic ROM in time at ξ=ξ~\boldsymbol{\xi}=\tilde{\boldsymbol{\xi}}, and counting the number of times nαn_{\alpha} that the integration becomes numerically unstable. The probability of ROM instability, α=ψ(ξ~)∈[0,1]\alpha=\psi(\tilde{\boldsymbol{\xi}})\in[0,1], can then be estimated as

α=nαnd,(27)\alpha=\frac{n_{\alpha}}{n_d}, \tag*{(27)}

which is a Monte Carlo estimator at ξ=ξ~\boldsymbol{\xi}=\tilde{\boldsymbol{\xi}}. This strategy is computationally tractable for moderate ndn_d because each ROM sample is, by design, cheap to assemble and solve.

Our second acquisition function, φ:P→R≥0\varphi:\mathcal{P}\to\mathbb{R}_{\ge0}, targets global accuracy of the ROM by scoring parameter candidates based on the prediction uncertainties obtained from stable realizations of the Bayesian parametric ROM. We start by defining an ideal acquisition function based on the accuracy of the ROM as

φideal(ξ)=E[1T∫0T∥q(t;ξ)−q~rom(t;ξ)∥22 dt],(28)\varphi_{\mathrm{ideal}}(\boldsymbol{\xi})=\mathbb{E}\left[\frac{1}{T}\int_{0}^{T}\left\|\mathbf{q}(t;\boldsymbol{\xi})-\widetilde{\mathbf{q}}_{\mathrm{rom}}(t;\boldsymbol{\xi})\right\|_{2}^{2}\,\mathrm{d}t\right], \tag*{(28)}

where q\mathbf{q} is the solution of the FOM and q~rom=Vq^rom\widetilde{\mathbf{q}}_{\mathrm{rom}}=\mathbf{V}\hat{\mathbf{q}}_{\mathrm{rom}} is the ROM solution mapped to the original state space. Maximizing φideal\varphi_{\mathrm{ideal}} would find the point in the parameter domain where the ROM is least accurate in expectation. However, φideal(ξ)\varphi_{\mathrm{ideal}}(\boldsymbol{\xi}) cannot be computed without first computing the full-order solution q(t;ξ)\mathbf{q}(t;\boldsymbol{\xi}). Instead, we derive an alternative based on the prediction variance of the Bayesian parametric ROM solution.

For a vector random variable x=[x1 ⋯ xnx]T∈Rnx\mathbf{x}=[x_1\ \cdots\ x_{n_x}]^{\mathsf{T}}\in\mathbb{R}^{n_x}, recall the identity

E[xTx]=E[x]TE[x]+tr⁡(Cov⁡[x])=∥E[x]∥22+∑i=1nxVar⁡[xi].\mathbb{E}[\mathbf{x}^{\mathsf{T}}\mathbf{x}]=\mathbb{E}[\mathbf{x}]^{\mathsf{T}}\mathbb{E}[\mathbf{x}]+\operatorname{tr}\left(\operatorname{Cov}[\mathbf{x}]\right)=\|\mathbb{E}[\mathbf{x}]\|_2^2+\sum_{i=1}^{n_x}\operatorname{Var}[x_i].

Suppressing for the moment the dependence of q\mathbf{q} and q~\widetilde{\mathbf{q}} on tt and ξ\boldsymbol{\xi},

ϕideal(ξ)=E[1T∫0T∥q−q~∥22 dt]=1T∫0TE[(q−q~)T(q−q~)] dt=1T∫0TqTq−2qTE[q~]+E[q~Tq~] dt=1T∫0TqTq−2qTE[q~]+E[q~]TE[q~]+tr⁡(Cov⁡[q~]) dt=1T∫0T∥q−E[q~]∥22 dt+1T∫0T∑i=1nxVar⁡[q~i] dt=ϕbias(ξ)+ϕvar(ξ),\begin{aligned} \phi_{\mathrm{ideal}}(\boldsymbol{\xi})&=\mathbb{E}\left[\frac{1}{T}\int_0^T\|\mathbf{q}-\widetilde{\mathbf{q}}\|_2^2\,dt\right]\\ &=\frac{1}{T}\int_0^T\mathbb{E}\left[(\mathbf{q}-\widetilde{\mathbf{q}})^{\mathsf{T}}(\mathbf{q}-\widetilde{\mathbf{q}})\right]\,dt\\ &=\frac{1}{T}\int_0^T\mathbf{q}^{\mathsf{T}}\mathbf{q}-2\mathbf{q}^{\mathsf{T}}\mathbb{E}[\widetilde{\mathbf{q}}]+\mathbb{E}[\widetilde{\mathbf{q}}^{\mathsf{T}}\widetilde{\mathbf{q}}]\,dt\\ &=\frac{1}{T}\int_0^T\mathbf{q}^{\mathsf{T}}\mathbf{q}-2\mathbf{q}^{\mathsf{T}}\mathbb{E}[\widetilde{\mathbf{q}}]+\mathbb{E}[\widetilde{\mathbf{q}}]^{\mathsf{T}}\mathbb{E}[\widetilde{\mathbf{q}}]+\operatorname{tr}\left(\operatorname{Cov}[\widetilde{\mathbf{q}}]\right)\,dt\\ &=\frac{1}{T}\int_0^T\|\mathbf{q}-\mathbb{E}[\widetilde{\mathbf{q}}]\|_2^2\,dt+\frac{1}{T}\int_0^T\sum_{i=1}^{n_x}\operatorname{Var}[\widetilde{q}_i]\,dt\\ &=\phi_{\mathrm{bias}}(\boldsymbol{\xi})+\phi_{\mathrm{var}}(\boldsymbol{\xi}), \end{aligned}

in which q~i(t;ξ)\widetilde{q}_i(t;\boldsymbol{\xi}) the ii-th entry of the ROM solution in the original state space, and where

ϕbias(ξ)=1T∫0T∥q(t;ξ)−E[q~(t;ξ)]∥22 dt,\phi_{\mathrm{bias}}(\boldsymbol{\xi})=\frac{1}{T}\int_0^T\|\mathbf{q}(t;\boldsymbol{\xi})-\mathbb{E}[\widetilde{\mathbf{q}}(t;\boldsymbol{\xi})]\|_2^2\,dt,
ϕvar(ξ)=1T∫0T∑i=1nxVar⁡[q~i(t;ξ)] dt.\phi_{\mathrm{var}}(\boldsymbol{\xi})=\frac{1}{T}\int_0^T\sum_{i=1}^{n_x}\operatorname{Var}[\widetilde{q}_i(t;\boldsymbol{\xi})]\,dt.

While ϕbias(ξ)\phi_{\mathrm{bias}}(\boldsymbol{\xi}) depends on q(t;ξ)\mathbf{q}(t;\boldsymbol{\xi}), ϕvar(ξ)\phi_{\mathrm{var}}(\boldsymbol{\xi}) depends on ROM solutions only and can therefore be computed efficiently. Furthermore, letting Vi,kV_{i,k} denote the entries of the basis matrix V\mathbf{V}, we have q~i=∑k=1rVi,kq^k\widetilde{q}_i=\sum_{k=1}^{r}V_{i,k}\widehat{q}_k, where q^k(t;ξ)\widehat{q}_k(t;\boldsymbol{\xi}) denotes the kk-th entry of the vector-valued stochastic process q^rom\widehat{\mathbf{q}}_{\mathrm{rom}}. Then

∑i=1nxVar⁡[q~i(t,ξ)]=∑i=1nx∑k=1rVar⁡[Vi,kq^k(t;ξ)]=∑i=1nx∑k=1rVi,k2Var⁡[q^k(t;ξ)]=∑k=1rVar⁡[q^k(t;ξ)],(29)\begin{aligned} \sum_{i=1}^{n_x}\operatorname{Var}[\widetilde{q}_i(t,\boldsymbol{\xi})]&=\sum_{i=1}^{n_x}\sum_{k=1}^{r}\operatorname{Var}\left[V_{i,k}\widehat{q}_k(t;\boldsymbol{\xi})\right]\\ &=\sum_{i=1}^{n_x}\sum_{k=1}^{r}V_{i,k}^2\operatorname{Var}[\widehat{q}_k(t;\boldsymbol{\xi})]\\ &=\sum_{k=1}^{r}\operatorname{Var}[\widehat{q}_k(t;\boldsymbol{\xi})], \tag*{(29)} \end{aligned}

since ∑i=1nxVi,k2=1\sum_{i=1}^{n_x}V_{i,k}^2=1. Hence, defining

ϕ^(ξ)=1T∫0T∑k=1rVar⁡[q^k(t,ξ)] dt,(30)\widehat{\phi}(\boldsymbol{\xi})=\frac{1}{T}\int_0^T\sum_{k=1}^{r}\operatorname{Var}[\widehat{q}_k(t,\boldsymbol{\xi})]\,dt, \tag*{(30)}

we have ϕ^(ξ)=ϕvar(ξ)\widehat{\phi}(\boldsymbol{\xi})=\phi_{\mathrm{var}}(\boldsymbol{\xi}). In other words, the total variance in the reduced space equals the total variance in the original state space. The acquisition function (3.4) is a computationally efficient measure of the total variance of the ROM solution at the given parameter vector, and this is the metric that we base our adaptive parameter selection strategy on.

Remark 3.1 (Shifted approximation). If the low-dimensional state approximation is given by q≈q‾+Vq^\mathbf{q}\approx\overline{\mathbf{q}}+\mathbf{V}\widehat{\mathbf{q}} as in Remark 2.1, the equality ϕ^(ξ)=ϕvar(ξ)\widehat{\phi}(\boldsymbol{\xi})=\phi_{\mathrm{var}}(\boldsymbol{\xi}) still holds because variance is translational. Specifically, we have q~i=q‾i+∑k=1rVi,kq^k\widetilde{q}_i=\overline{q}_i+\sum_{k=1}^{r}V_{i,k}\widehat{q}_k, and Var⁡[q‾i+Vi,kq^k(t;ξ)]=Vi,k2Var⁡[q^k(t;ξ)]\operatorname{Var}\left[\overline{q}_i+V_{i,k}\widehat{q}_k(t;\boldsymbol{\xi})\right]=V_{i,k}^2\operatorname{Var}\left[\widehat{q}_k(t;\boldsymbol{\xi})\right], by which (3.3) holds.

Algorithm 3.2  Adaptive sampling for (Bayesian) parametric OpInf ROMs

1: procedure NextSample(
       Parameter candidates $\tilde{\boldsymbol{\xi}}_1,\ldots,\tilde{\boldsymbol{\xi}}_{\tilde{n}_p}\in\mathcal{P}$
       Operator matrix realizations $\widehat{\mathbf{O}}^{(1)},\ldots,\widehat{\mathbf{O}}^{(n_d)}\in\mathbb{R}^{r\times d(r)}$
       Parametric ROM structure function $\mathbf{d}=\mathbf{d}(\hat{\mathbf{q}};\boldsymbol{\xi})$
   )
2:     $\mathcal{I}\leftarrow\{1,2,\ldots,\tilde{n}_p\}$                         # Indices of all candidates.
3:     for $i=1,\ldots,\tilde{n}_p$ do
4:         $\alpha_i,\omega_i\leftarrow\operatorname{BayesOpInfAcquisition}(\tilde{\boldsymbol{\xi}}_i,\widehat{\mathbf{O}}^{(1)},\ldots,\widehat{\mathbf{O}}^{(n_d)},\mathbf{d})$   # Algorithm 3.1.
5:         $\mathcal{U}\leftarrow\{i\in\mathcal{I}\mid\alpha_i\geq\alpha_j\ \forall j\in\mathcal{I}\}$   # Collect maximal unstable ROM indices.
6:         if $\alpha_i<100\%\ \forall i\in\mathcal{U}$ (some draws stable for every candidate) then
7:             $i^*\leftarrow\operatorname*{arg\,max}_{i\in\mathcal{U}}\omega_i$   # Index maximizing acquisition.
8:         else ($\alpha_i=100\%\ \forall i\in\mathcal{U}$, all draws unstable for at least one candidate)
9:             $i^*\leftarrow$ sample randomly from $\mathcal{U}$   # No stability.
10:    return $\tilde{\boldsymbol{\xi}}_{i^*}$

Algorithm 3.1 provides a routine for scoring a parameter candidate in terms of stability and accuracy. Given draws from the operator matrix posterior, the corresponding deterministic ROMs are solved to produce state trajectories if possible or, otherwise, to indicate that the ROM is unstable. The probability of instability ψ\psi is estimated as α=nα/nd\alpha=n_\alpha/n_d, and stable solutions are used to estimate the value of the variance-based acquisition function φ\varphi. In our experiments, we use a numerical approximant of (30),

φ(ξ)=φ^(ξ)≈∑j=1nt∑k=1rVar⁡(ℓ)[q^k(ℓ)(tj;ξ)],(31)\varphi(\boldsymbol{\xi})=\hat{\varphi}(\boldsymbol{\xi})\approx\sum_{j=1}^{n_t}\sum_{k=1}^{r}\operatorname{Var}_{(\ell)}\left[\hat{q}^{(\ell)}_k(t_j;\boldsymbol{\xi})\right], \tag*{(31)}

in which q^k(ℓ)(tj;ξ)\hat{q}^{(\ell)}_k(t_j;\boldsymbol{\xi}) is the solution of the ℓ\ell-th deterministic ROM at time t=tjt=t_j for some discretization of the time domain t1,t2,…,tntt_1,t_2,\ldots,t_{n_t}, and where Var⁡(ℓ)\operatorname{Var}_{(\ell)} indicates the sample variance over the sampling index ℓ\ell. The variance is only taken over stable draws (the set SS in Algorithm 3.1); consequently, this value cannot be computed if all deterministic ROM draws turn out to be unstable.

Algorithm and implementation details

Let ξ~1,…,ξ~n~p∈P\tilde{\boldsymbol{\xi}}_1,\ldots,\tilde{\boldsymbol{\xi}}_{\tilde{n}_p}\in\mathcal{P} be a finite set of candidate parameter vectors at which we have the option to solve the FOM (2.2) to obtain training data for the pOpInf regression (2.6). With an initial index i1∈I={1,…,n~p}i_1\in\mathcal{I}=\{1,\ldots,\tilde{n}_p\}, our goal is to select a distinct index i2i_2 such that using training data corresponding to the parameters (ξ1,ξ2)=(ξ~i1,ξ~i2)(\boldsymbol{\xi}_1,\boldsymbol{\xi}_2)=(\tilde{\boldsymbol{\xi}}_{i_1},\tilde{\boldsymbol{\xi}}_{i_2}) results in the best possible pOpInf ROM among those that are trained with ξ~i1\tilde{\boldsymbol{\xi}}_{i_1} and one other parameter vector. We then seek an optimal third training parameter vector, and so on, until exhausting a computational budget for the FOM evaluations.

Our proposed procedure, which uses the stability and variance-based acquisition criteria from the previous section, is outlined in Algorithm 3.2. First, Algorithm 3.1 is evaluated for each candidate parameter vector, resulting in pairs (αi,ωi)(\alpha_i,\omega_i) for the probability of instability and variance-based acquisition values corresponding to each ξ~i\tilde{\boldsymbol{\xi}}_i. The indexing set I\mathcal{I} is downselected to those indices U\mathcal{U} corresponding to the parameter candidates that maximize the empirical probability of unstable behavior, that is, αi≥αj\alpha_i\geq\alpha_j for all i∈Ui\in\mathcal{U} and j∈Ij\in\mathcal{I}. From these remaining indices, the index i∗i^* is chosen which corresponds to the greatest acquisition value, meaning ωi∗≥ωj\omega_{i^*}\geq\omega_j for all j∈Uj\in\mathcal{U}. The procedure accounts for several scenarios:

  1. If all ROM samples are stable for every parameter candidate, then αi=0\alpha_i=0 for each i∈Ii\in\mathcal{I} and therefore U=I\mathcal{U}=\mathcal{I}. Hence, the selection of i∗i^* is determined exclusively using the acquisition values ωi\omega_i, not via the stability indicators αi\alpha_i.

  1. If there is a single parameter candidate index iui_u for which the instability indicator αiu\alpha_{i_u} is strictly greater than all other instability indicators, then U={iu}\mathcal{U}=\{i_u\} is a singleton set, and i∗=iui^*=i_u is always chosen no matter the corresponding variance-based acquisition value ωiu\omega_{i_u}. In other words, if there is a single “most unstable” parameter vector, the corresponding index is always chosen.

  1. If there are several parameter candidate indices which maximize the stability indicator αi\alpha_i, then U\mathcal{U} is a subset of I\mathcal{I}. This means that both acquisition functions are used—first stability, then variance.

  1. If, for at least one parameter candidate, every ROM sample is unstable, the variance-based acquisition value (3.5) cannot be computed at that parameter (there are no stable solutions to take the variance of), and the winning index i∗i^* must be chosen randomly from the set U\mathcal{U} (§ 3.2). From a practical point of view, this may indicate an issue with the problem setup, such as the quality of the low-dimensional state approximation or an inappropriate ROM structure.

In the numerical experiments that follow, these cases tend to occur in reverse order: with only few training parameter vectors, the ROM may be highly unstable at many parameter candidates; after sampling a few additional parameter vectors, the ROM tends to be unstable at only a few candidates; finally, given data for a sufficient number of training parameter vectors, the resulting ROM is stable throughout the domain and can then be improved strategically based on the variance-based acquisition function alone.

The entire active learning workflow is described in Algorithm 3.3. After generating an initial training set, a new parameter is selected via Algorithm 3.2, the basis and ROM operators are recomputed, and the process repeats. The total number of FOM solves, which dominates the total computational cost, is the number of initial training parameters npn_p plus the number of times nbn_b that the active learning loop is repeated.

Numerical results

In this section, we apply our active learning approach to learn probabilistic ROMs for two parametric partial differential equations (PDEs) with affine-parametric structure. Both examples are prototypical, yet they highlight the difficulty of constructing parametric ROMs that effectively capture the effects of the parameters on the solution, even with moderately benign parameter spaces, without an effective sampling strategy. The codes for these experiments are available at github.com/McQLab/ActiveBayesOpInf.

Algorithm 3.3 Active learning procedure for ROM learning.

1: procedure ActiveLearningROM(
    Training parameter vectors ξ₁, . . . , ξₙₚ ∈ 𝒫,
    Number of time steps nₜ,
    Budget for additional full-order solves n_b,
    Parametric ROM structure function d = d(q̂; ξ),
    Number of posterior draws n_d > 0,
    Parameter candidates ξ̃₁, . . . , ξ̃ₙ̃ₚ ∈ 𝒫
)
    # Generate initial training dataset.
2:  for i = 1, . . . , nₚ do
3:      {qᵢⱼ}ⱼ₌₁ⁿᵗ ← Solve (2.1) with parameter value ξᵢ over nₜ time steps

    # Active learning loop.
4:  for k = 1, . . . , n_b do
5:      q̄, V ∈ ℝᴺˣʳ ← Reference vector, basis matrix from [ q₁₁ q₁₂ · · · qₙₚₙₜ ]
6:      {q̂ᵢⱼ}ⁿᵖ,ⁿᵗᵢ₌₁,ⱼ₌₁ ← {Vᵀ(qᵢⱼ − q̄)}ⁿᵖ,ⁿᵗᵢ₌₁,ⱼ₌₁                         # Compress training data.
7:      Ô⁽¹⁾, . . . , Ô⁽ⁿᵈ⁾ ← pBayesOpInf({ξᵢ}ⁿᵖᵢ₌₁, {q̂ᵢⱼ}ⁿᵖ,ⁿᵗᵢ₌₁,ⱼ₌₁, d, n_d)       # Alg. 2.1.
8:      ξ̃ᵢ* ← NextSample({ξ̃ᵢ}ⁿ̃ᵖᵢ₌₁, Ô⁽¹⁾, . . . , Ô⁽ⁿᵈ⁾, d)                         # Alg. 3.2.

    # Add selected parameter to training set.
9:  nₚ ← nₚ + 1
10: ξₙₚ ← ξ̃ᵢ*
11: {qₙₚⱼ}ⱼ₌₁ⁿᵗ ← Solve (2.1) with parameter value ξᵢ* over nₜ time steps

    # Remove selected parameter from candidate set.
12: {ξ̃ᵢ}ⁿ̃ᵖᵢ₌₁ ← {. . . , ξ̃ᵢ*−₁, ξ̃ᵢ*+₁, . . .}
13: ñₚ ← ñₚ − 1
14: return Ô⁽¹⁾, . . . , Ô⁽ⁿᵈ⁾

Heat equation with nonlinear reaction

Consider the following diffusive PDE defined over a one-dimensional spatial domain,

∂q∂t=κ∂2q∂x2−ρq2,x∈(0,L), t∈(0,tf],q(0,t)=0,q(L,t)=1,t∈(0,tf],q(x,0)=q0(x),x∈[0,L],(32)\begin{aligned} \frac{\partial q}{\partial t} &= \kappa\frac{\partial^2 q}{\partial x^2} - \rho q^2, && x \in(0,L),\ t \in(0,t_f],\\ q(0,t) &= 0,\quad q(L,t) = 1, && t \in(0,t_f],\\ q(x,0) &= q_0(x), && x \in[0,L], \tag*{(32)} \end{aligned}

where κ>0\kappa> 0 and ρ≥0\rho\ge0 are diffusion and reaction coefficients, respectively, L>0L > 0 is the length of the spatial domain, tf>0t_f > 0 is the final time, and q0:[0,L]→Rq_0 : [0,L] \to\mathbb{R} is the initial condition. We consider a parameter-independent, non-polynomial initial condition that matches the constant Dirichlet boundary conditions,

q0(x)=x(1−x)(6e−x(1−x)2−10exsin⁡(x/6))+x.(33)q_0(x) = x(1-x)(6e^{-x}(1-x)^2 - 10e^x\sin(x/6)) + x. \tag*{(33)}

A uniform second-order central finite difference discretization of (32) with NN degrees of freedom yields a semi-discrete model

dqdt=κc(1)+κA(1)q+ρH(1)[q⊗q],(34)\frac{\mathrm{d}q}{\mathrm{d}t} = \kappa c^{(1)} + \kappa A^{(1)}q + \rho H^{(1)}[q \otimes q], \tag*{(34)}

where q(t;ξ),c(1)∈RNq(t;\xi), c^{(1)} \in\mathbb{R}^N, A(1)∈RN×NA^{(1)} \in\mathbb{R}^{N \times N}, and H(1)∈RN×N2H^{(1)} \in\mathbb{R}^{N \times N^2}. This is the generic FOM (2.2) with parameter vector ξ=(κ,ρ)\xi= (\kappa,\rho), coefficient functions θc(1)(ξ)=θa(1)(ξ)=κ\theta_c^{(1)}(\xi) = \theta_a^{(1)}(\xi) = \kappa and θh(1)(ξ)=ρ\theta_h^{(1)}(\xi) = \rho, with nc=na=nh=1n_c = n_a = n_h = 1. In our experiments, we use L=1L = 1 as the length of the spatial domain, set N=500N = 500, and consider the time domain t∈[0,1]t \in[0,1] with nt=101n_t = 101 discrete temporal points. Figure 1 shows FOM solutions for several parameter vectors.

Figure 1. Solutions to the full-order diffusion-reaction model (34) for values of the parameter vector ξ=(κ,ρ)\boldsymbol{\xi}=(\kappa,\rho) at the corners of the parameter domain P=[10−3,10−1]×[1,5]\mathcal{P}=[10^{-3},10^{-1}]\times[1,5]. The initial condition and time domain are the same for all experiments.

Our goal in this experiment is to construct a ROM that accurately mirrors the FOM over the parameter domain (κ,ρ)∈P=[10−3,10−1]×[1,5](\kappa,\rho) \in\mathcal{P} = [10^{-3},10^{-1}] \times[1,5]. To exactly preserve the boundary conditions, we consider affine approximations of the discretized state, q≈qˉ+Vq^\mathbf{q} \approx\bar{\mathbf{q}} + \mathbf{V}\hat{\mathbf{q}}, where qˉ\bar{\mathbf{q}} is a reference vector such that q(t)−qˉ\mathbf{q}(t) - \bar{\mathbf{q}} corresponds to homogeneous Dirichlet conditions. The corresponding ROM has the form

dq^dt=κc^(1)+ρc^(2)+(κA^(1)+ρA^(2))q^+ρH^(1)[q^⊗q^],(35)\frac{\mathrm{d}\hat{\mathbf{q}}}{\mathrm{d}t} = \kappa\hat{\mathbf{c}}^{(1)} + \rho\hat{\mathbf{c}}^{(2)} + \left(\kappa\hat{\mathbf{A}}^{(1)} + \rho\hat{\mathbf{A}}^{(2)}\right)\hat{\mathbf{q}} + \rho\hat{\mathbf{H}}^{(1)}[\hat{\mathbf{q}} \otimes\hat{\mathbf{q}}], \tag*{(35)}

where c^(1),c^(2)∈Rr\hat{\mathbf{c}}^{(1)}, \hat{\mathbf{c}}^{(2)} \in\mathbb{R}^{r}, A^(1),A^(2)∈Rr×r\hat{\mathbf{A}}^{(1)}, \hat{\mathbf{A}}^{(2)} \in\mathbb{R}^{r \times r}, and H^(1)∈Rr×r2\hat{\mathbf{H}}^{(1)} \in\mathbb{R}^{r \times r^{2}}. This is the generic ROM (2.3) with coefficient functions inherited from (4.3), plus additional coefficient functions θc(2)(ξ)=θa(2)(ξ)=ρ\theta_{c}^{(2)}(\boldsymbol{\xi}) = \theta_{a}^{(2)}(\boldsymbol{\xi}) = \rho arising due to the nonzero reference vector qˉ\bar{\mathbf{q}}. To write this as a pOpInf ROM dq^dt=O^d(q^,ξ)\frac{\mathrm{d}\hat{\mathbf{q}}}{\mathrm{d}t} = \hat{\mathbf{O}}\mathbf{d}(\hat{\mathbf{q}},\boldsymbol{\xi}) akin to (2.4), we define

O^=[c^(1) c^(2) A^(1) A^(2) H^(1)]∈Rr×d(r),d(q^,ξ)=[κρκq^ρq^ρ(q^⊗q^)]∈Rd(r),(36)\hat{\mathbf{O}} = \left[\hat{\mathbf{c}}^{(1)}\ \hat{\mathbf{c}}^{(2)}\ \hat{\mathbf{A}}^{(1)}\ \hat{\mathbf{A}}^{(2)}\ \hat{\mathbf{H}}^{(1)}\right] \in\mathbb{R}^{r \times d(r)}, \qquad \mathbf{d}(\hat{\mathbf{q}},\boldsymbol{\xi}) = \begin{bmatrix} \kappa\\ \rho\\ \kappa\hat{\mathbf{q}} \\ \rho\hat{\mathbf{q}} \\ \rho(\hat{\mathbf{q}} \otimes\hat{\mathbf{q}}) \end{bmatrix} \in\mathbb{R}^{d(r)}, \tag*{(36)}

with data dimension d(r)=2+2r+r2d(r) = 2 + 2r + r^{2}.

Remark 4.1 (Compressed Kronecker product). The quadratic mapping q^↦H^(1)[q^⊗q^]\hat{\mathbf{q}} \mapsto\hat{\mathbf{H}}^{(1)}[\hat{\mathbf{q}} \otimes\hat{\mathbf{q}}] with H^∈Rr×r2\hat{\mathbf{H}} \in\mathbb{R}^{r \times r^{2}} can be represented with a compressed version of the Kronecker product and a r×r(r+1)/2r \times r(r + 1)/2 matrix to avoid redundant computations, see, e.g., [51 Appendix B] for details. Hence, in practice, the data dimension for (4.5) is reduced to d(r)=2+2r+r(r+1)/2d(r) = 2 + 2r + r(r + 1)/2.

To assess our adaptive sampling strategy, we discretize the two-dimensional parameter domain P\mathcal{P} with a 20×2020 \times20 grid G⊂P\mathcal{G} \subset\mathcal{P} of parameter candidates, logarithmically spaced in κ\kappa and uniformly spaced in ρ\rho. We run 50 trials where, in each trial, a single parameter sample is randomly selected from the parameter grid G\mathcal{G} to generate an initial training data set via the FOM. From that data, we learn a Bayesian parametric ROM with the procedure from Section 2, then select a new training parameter vector using the sampling strategy of Section 3. The FOM is then solved at the new parameter sample to generate additional training data, a new ROM is constructed, and another sample is selected. This procedure continues until, in each trial, np=10n_p = 10 training parameters have been selected. A new POD basis matrix V\mathbf{V} is generated at each iteration by calculating SVD of the (shifted) snapshot matrix with mean-zero columns qj−qˉ\mathbf{q}_j - \bar{\mathbf{q}}, where the qj\mathbf{q}_j are the training snapshots and qˉ\bar{\mathbf{q}} is the average training snapshot. Letting σ1≥σ2≥⋯≥σK\sigma_1 \geq\sigma_2 \geq\cdots\geq\sigma_K be the resulting singular values, the reduced dimension rr is selected as the smallest integer such that the residual singular value energy,

1 - \left(\sum_{i=1}^{r} \sigma_i^2 \middle/ \sum_{k=1}^{K} \sigma_k^2\right) = \sum_{i=r+1}^{K} \sigma_i^2 \middle/ \sum_{k=1}^{K} \sigma_k^2,

is less than 10−610^{-6}. The POD basis matrix V\mathbf{V} then consists of the first rr left singular vectors.

Remark 4.2 (Unstructured candidate sets). The set of parameter candidates G\mathcal{G} is chosen in this example to be a regular grid in order to facilitate visualizing the results and establish a fair comparison with realistic random sampling (described shortly). However, the set G\mathcal{G} need not represent a grid or any other kind of geometric structure; all that Algorithm 3.3 requires is a generic set of candidates to evaluate and select from.

At each iteration of the procedure, we evaluate the learned ROM on each of the 400 parameter vectors in the grid G\mathcal{G} and compute the L2L^2 error relative to the FOM, which is related to the ideal acquisition function φideal\varphi_{\mathrm{ideal}} in (28). Specifically, the relative ROM error is defined by

relative ROM error=∫0T∥q(t;ξ)−E[q~rom(t;ξ)]∥22 dt∫0T∥q(t;ξ)∥22 dt(37)\text{relative ROM error} = \frac{\int_{0}^{T} \lVert\mathbf{q}(t;\boldsymbol{\xi})-\mathbb{E}[\widetilde{\mathbf{q}}_{\mathrm{rom}}(t;\boldsymbol{\xi})]\rVert_{2}^{2}\,\mathrm{d}t}{\int_{0}^{T} \lVert\mathbf{q}(t;\boldsymbol{\xi})\rVert_{2}^{2}\,\mathrm{d}t} \tag*{(37)}

which is a function of the parameter vector ξ\boldsymbol{\xi}. The time integrals are estimated with the trapezoidal rule over the discrete time domain and the expectation is estimated with a sample mean over nd=50n_d=50 draws. Recall that this error is unavailable without evaluating the FOM; our goal is for this error to decrease globally—without our having access to it—as parameter samples are strategically added to the training set.

Figure 2 displays the relative ROM error after training on data corresponding to np=1n_p=1, np=5n_p=5, and np=9n_p=9 parameter samples for a single trial. The ROM error tends to be smallest near training parameter samples and is often largest at the boundaries of the parameter domain. Figure 2 also shows the variance-based acquisition values φ(ξ)\varphi(\boldsymbol{\xi}), which are used to adaptively determine the next parameter sample. Although the acquisition function is based on variance, not actual error, the parameters that maximize the acquisition function tend to also maximize or nearly maximize the ROM error. Hence, the adaptive sampling procedure successfully selects parameter vectors where the current ROM performs poorly. As npn_p increases, the ROM error decreases globally across P\mathcal{P}. The variance-based acquisition values do not always decrease monotonically as new parameter samples are added, but these values are typically much smaller for larger npn_p than for smaller npn_p. This indicates that as the ROM becomes more accurate, it also reports more confidence in its predictions.

Acquisition values and relative ROM errors for three training sample sizes

Figure 2. Acquisition values φ(ξ)\varphi(\boldsymbol{\xi}) (top) and relative L2L^2-norm ROM errors (bottom) after training on npn_p adaptively chosen parameter samples for the problem (4.1).

We also compare our adaptive sampling procedure to random sampling. For each of the 50 experimental trials (each with different initial training samples), we select np=10n_p=10 training parameters from the 20×2020 \times20 grid G⊂P\mathcal{G} \subset\mathcal{P} via discrete Latin hypercube sampling while ensuring that the initial samples for each trial are the same as were used in the corresponding adaptive sampling experiments. Figure 3 shows the samples from three different trials for both the adaptive sampling strategy and random sampling. The Latin hypercube sampling covers the space P\mathcal{P} better, while the adaptively chosen samples tend toward the corners and boundaries of P\mathcal{P}. The exact training parameters chosen by the adaptive sampling vary from trial to trial, depending on the initial parameter sample, but many adaptive samples are always chosen from the region of P\mathcal{P} with the largest reaction coefficient ρ\rho and, especially, with large ρ\rho but small diffusion coefficient κ\kappa. Note that this corner of P\mathcal{P} is where the model (4.1) is the most nonlinear, since ρ\rho is attached to the reaction term and κ\kappa scales the linear diffusion operator. On the other hand, the adaptive sampling scheme almost always also selects one sample from the opposite corner of P\mathcal{P}, where κ\kappa is largest and ρ\rho is smallest—the area of P\mathcal{P} where (4.1) is the most linear. Clearly, the adaptive sampling scheme avoids replicating uniform or space-covering sampling, instead focusing on particular regions of the parameter domain.

Figure 3. Parameter samples selected from a 20×2020 \times20 grid G\mathcal{G} over the parameter domain P\mathcal{P} using adaptive sampling (top) and Latin hypercube sampling (bottom) for the problem (4.1). Results are shown for three trials, each with a different random initial training parameter vector (blue).

Next, we discuss how the sampling strategies compare in terms of ROM stability. In each trial and at each iteration, we record the probability of ROM instability αi=ψ(ξi)∈[0,1]\alpha_i = \psi(\boldsymbol{\xi}_i) \in[0,1] for each parameter candidate ξi\boldsymbol{\xi}_i in the grid G\mathcal{G}. We then calculate the percentage of parameters for which αi>0\alpha_i > 0, that is, the frequency with which any of the posterior ROM draws were unstable at a given parameter vector:

β=1400∑i=1400βi,βi={1if αi>0,0if αi=0.\beta= \frac{1}{400}\sum_{i=1}^{400}\beta_i,\qquad \beta_i = \begin{cases} 1 & \text{if }\alpha_i > 0,\\ 0 & \text{if }\alpha_i = 0. \end{cases}

Figure 4 shows these β\beta values as a function of the number of training parameters npn_p, averaged over the 50 trials for each sampling method. We also display the maximum over the trials (pointwise for each npn_p). With only one training parameter sample, the resulting parametric ROM is unstable (for at least one posterior realization) over about 65% of the parameter grid on average, while at least one initial parameter sample resulted in a ROM that was unstable over all of G\mathcal{G}. There are two key insights from Figure 4: (i) the average and maximal instability percentages decrease faster with adaptive sampling than with random sampling, and (ii) adaptive sampling achieves total stability for np≥7n_p \ge7, while at least one of the trials for random sampling results in some instabilities for all np≤10n_p \le10. With np=6n_p = 6, random sampling often results in ROMs that are stable over the entire parameter domain, but the same average rate of stability can be achieved with half as many samples through adaptive sampling.

Figure 4. Percentages of the 400 training parameter candidates where any instabilities are observed from 50 posterior draws of the probabilistic ROM, as a function of the number of training parameters npn_p, for the problem (32). The results are averaged and maximized over 50 trials with different random initial parameter samples.

We conclude this example by analyzing the accuracy of the expected value of ROM predictions. Figure 2 plots the relative error (4.6) over the parameter domain for a few select cases; to measure the ROM error over the entire parameter space, we consider a total L2L^2 error,

Etotal=(∑ξi∈G(∫0T∥q(t;ξi)−E[q~rom(t;ξi)]∥22 dt)2∑ξi∈G(∫0T∥q(t;ξi)∥22 dt)2)1/2.(38)\mathcal{E}_{\mathrm{total}} = \left( \frac{ \sum_{\boldsymbol{\xi}_i \in\mathcal{G}} \left( \int_0^T \left\lVert\boldsymbol{q}(t;\boldsymbol{\xi}_i)- \mathbb{E}[\widetilde{\boldsymbol{q}}_{\mathrm{rom}}(t;\boldsymbol{\xi}_i)] \right\rVert_2^2\,\mathrm{d}t \right)^2 }{ \sum_{\boldsymbol{\xi}_i \in\mathcal{G}} \left( \int_0^T \left\lVert\boldsymbol{q}(t;\boldsymbol{\xi}_i)\right\rVert_2^2\,\mathrm{d}t \right)^2 } \right)^{1/2}. \tag*{(38)}

This value is calculated for each of the 50 trials, and the results are aggregated in Figure 5, which shows the 5% and 95% quantiles and the geometric mean of the Etotal\mathcal{E}_{\mathrm{total}} values. After only np=2n_p = 2 training samples, the adaptive sampling ROMs perform at least one order of magnitude better on average than the random sampling ROMs. Furthermore, the range of errors contracts quickly in the adaptive sampling case, so that with np=7n_p = 7 training samples the ROM error is always below 1% (and sometimes below 0.1%). By contrast, in the random sampling case, the ROM error spans a wide range of at least three orders of magnitude for all np≤9n_p \le9. Even with np=10n_p = 10 samples, in at least one instance the random sampling ROM still only achieves 10% total relative error. The difference in the results shown in Figure 5 is primarily due to the manner in which the adaptive sampling method targets ROM performance, not because of the quality of the underlying basis. To demonstrate this, Figure 6 shows the total projection error,

Figure 5. Total relative L2L^2 ROM errors Etotal\mathcal{E}_{\mathrm{total}} as a function of the number of training parameter samples, aggregated across 50 trials with different initial samples, for the problem (32). The shaded regions show the interquantile range of 5%–95% of trials.

Figure 6. Total relative L2L^2 projection errors EprojE_{\mathrm{proj}} as a function of the number of training parameter samples, aggregated across 50 trials with different initial samples, for the problem (4.1). The shaded regions show the interquantile range of 5%–95% of trials.

Eproj=(∑ξi∈G(∫0T∥q(t;ξi)−VVT(q(t;ξi)−qˉ)+qˉ∥22 dt)∑ξi∈G(∫0T∥q(t;ξi)∥22 dt))1/2,(39)\mathcal{E}_{\mathrm{proj}} = \left(\frac{\sum_{\xi_i\in\mathcal{G}}\left(\int_0^T \left\lVert\mathbf{q}(t;\xi_i)-\mathbf{V}\mathbf{V}^{\mathsf{T}}\left(\mathbf{q}(t;\xi_i)-\bar{\mathbf{q}}\right)+\bar{\mathbf{q}}\right\rVert_2^2\,\mathrm{d}t\right)}{\sum_{\xi_i\in\mathcal{G}}\left(\int_0^T \left\lVert\mathbf{q}(t;\xi_i)\right\rVert_2^2\,\mathrm{d}t\right)}\right)^{1/2}, \tag*{(39)}

which depends only on the basis matrix V\mathbf{V}, not the quality of the ensuing ROM. Recall that the basis size is determined at each iteration by the residual singular value energy. This means that the basis may grow when derived from a more diverse set of training data: in these experiments, the initial basis with np=1n_p = 1 training parameter has r=4r = 4 or 5, which increases to about r=8r = 8 for np=2n_p = 2, and settles at r=10r = 10 or r=11r = 11 after additional samples. This is the main reason that the projection error first decreases, then plateaus, as npn_p increases. The average rate of decrease is highly similar in adaptive and random sampling, and the variance in the projection errors is smaller for the adaptive sampling, which tends to focus on the same regions of P\mathcal{P} in each trial. This shows that the adaptive sampling method does not merely target a better basis matrix V\mathbf{V}; rather, it focuses specifically on improving the performance of the resulting ROM.

Viscous Burgers' equation

Consider next the two-dimensional viscous Burgers’ equations over the spatial domain Ω=[0,2]×[0,2]\Omega= [0,2] \times[0,2],

∂u∂t+u∂u∂x+v∂u∂y=ν(∂2u∂x2+∂2u∂y2),∂v∂t+u∂v∂x+v∂v∂y=ν(∂2v∂x2+∂2v∂y2),(40)\begin{aligned} \frac{\partial u}{\partial t} + u\frac{\partial u}{\partial x} + v\frac{\partial u}{\partial y} &= \nu\left(\frac{\partial^2 u}{\partial x^2} + \frac{\partial^2 u}{\partial y^2}\right), \\ \frac{\partial v}{\partial t} + u\frac{\partial v}{\partial x} + v\frac{\partial v}{\partial y} &= \nu\left(\frac{\partial^2 v}{\partial x^2} + \frac{\partial^2 v}{\partial y^2}\right), \tag*{(40)} \end{aligned}

where u(x,y,t)u(x,y,t) and v(x,y,t)v(x,y,t) are the velocity in the xx and yy directions, respectively, and ν∈P={0.001,0.025}\nu\in\mathcal{P} = \{0.001, 0.025\} is the kinematic viscosity, which is the parameter of interest. We consider homogeneous Dirichlet boundary conditions,

u(x,y,t)=0,v(x,y,t)=0,∀(x,y)∈∂Ω, t≥0,(41)u(x,y,t) = 0,\qquad v(x,y,t) = 0,\qquad\forall(x,y) \in\partial\Omega,\ t \ge0, \tag*{(41)}

and define the initial velocity fields as partially overlapping square pulses:

u(x,y,0)={2,0.5≤x≤1.0, 0.5≤y≤1.0,0,elsewhere,v(x,y,0)={4,0.25≤x≤0.75, 0.25≤y≤0.75,0,elsewhere.(42)u(x,y,0) = \begin{cases} 2, & 0.5 \le x \le1.0,\ 0.5 \le y \le1.0, \\ 0, & \text{elsewhere}, \end{cases} \qquad v(x,y,0) = \begin{cases} 4, & 0.25 \le x \le0.75,\ 0.25 \le y \le0.75, \\ 0, & \text{elsewhere}. \end{cases} \tag*{(42)}

This setup produces a localized velocity disturbance that spreads and diffuses over time, with stronger diffusion when ν\nu is larger and more shock-like behavior for small ν\nu.

Discretizing (4.9a) with finite differences over nxn_x spatial degrees of freedom yields a semi-discrete model

dqdt=νA(1)q+H(1)[q⊗q],q=[uv],(43)\frac{\mathrm{d}\mathbf{q}}{\mathrm{d}t} = \nu\mathbf{A}^{(1)}\mathbf{q} + \mathbf{H}^{(1)}[\mathbf{q}\otimes\mathbf{q}],\qquad\mathbf{q} = \begin{bmatrix} \mathbf{u} \\ \mathbf{v} \end{bmatrix}, \tag*{(43)}

in which u(t;ν)\mathbf{u}(t;\nu), v(t;ν)∈Rnx\mathbf{v}(t;\nu) \in\mathbb{R}^{n_x} are the spatial discretizations of uu and vv, respectively, and where q(t;ν)∈RN\mathbf{q}(t;\nu) \in\mathbb{R}^{N}, A(1)∈RN×N\mathbf{A}^{(1)} \in\mathbb{R}^{N\times N} and H(1)∈RN×N2\mathbf{H}^{(1)} \in\mathbb{R}^{N\times N^2} with N=2nxN = 2n_x. This is the FOM (2.2) with parameter “vector” ξ=ν\xi= \nu, coefficient functions θa(1)(ξ)=ν\theta_a^{(1)}(\xi) = \nu and θh(1)(ξ)=1\theta_h^{(1)}(\xi) = 1, with nc=0n_c = 0 (no constant term) and na=nh=1n_a = n_h = 1. Our experiments use nx=1012=10,201n_x=101^2=10{,}201 (101 uniformly spaced points in each spatial direction) so that the FOM has N=2nx=20,402N=2n_x=20{,}402 total degrees of freedom, along with the time domain t∈[0,1]t\in[0,1] with nt=100n_t=100 temporal measurements. Figure 7 shows the initial conditions (42) and solutions to the FOM (43) at t=0.5t=0.5 for the smallest and largest values of ν\nu. Although this problem has only a one-dimensional parameter space, it is more difficult to tackle with standard model reduction methods than the previous problem due to the shock-like behavior that occurs with small values of ν\nu.

Solutions to the full-order Burgers’ equation at the initial time and for two different values of $\nu$ at time $t=0.5$

Figure 7. Solutions to the full-order Burgers’ equation (4.9) at the initial time (left) and for two different values of ν\nu at time t=0.5t=0.5 (middle and right). With smaller ν\nu, the solutions are more shock-like (middle); when ν\nu is larger, the solutions are more diffusive (right).

Because the boundary conditions (41) are homogeneous, we consider a linear low-dimensional approximation of the discrete state, q≈Vq^\mathbf{q}\approx\mathbf{V}\widehat{\mathbf{q}} (i.e., the reference vector is q‾=0\overline{\mathbf{q}}=0). This leads to a ROM which inherits the parametric and polynomial structure of the FOM exactly,

dq^dt=νA^(1)q^+H^(1)[q^⊗q^],(44)\frac{\mathrm{d}\widehat{\mathbf{q}}}{\mathrm{d}t}=\nu\widehat{\mathbf{A}}^{(1)}\widehat{\mathbf{q}}+\widehat{\mathbf{H}}^{(1)}[\widehat{\mathbf{q}}\otimes\widehat{\mathbf{q}}], \tag*{(44)}

where A^(1)∈Rr×r\widehat{\mathbf{A}}^{(1)}\in\mathbb{R}^{r\times r}, H^(1)∈Rr×r2\widehat{\mathbf{H}}^{(1)}\in\mathbb{R}^{r\times r^2}. This ROM can be written in the form of (8) by defining

O^=[A^(1) H^(1)]∈Rr×d(r),d(q^,ξ)=[νq^q^⊗q^]∈Rd(r),(45)\widehat{\mathbf{O}}=\left[\widehat{\mathbf{A}}^{(1)}\ \widehat{\mathbf{H}}^{(1)}\right]\in\mathbb{R}^{r\times d(r)},\qquad \mathbf{d}(\widehat{\mathbf{q}},\xi)=\begin{bmatrix}\nu\widehat{\mathbf{q}}\\ \widehat{\mathbf{q}}\otimes\widehat{\mathbf{q}}\end{bmatrix}\in\mathbb{R}^{d(r)}, \tag*{(45)}

where the data dimension is d(r)=r+r2d(r)=r+r^2.

For our experiments, we discretize the one-dimensional parameter domain P\mathcal{P} with a uniformly spaced grid G⊂P\mathcal{G}\subset\mathcal{P} of 50 logarithmically spaced parameter candidates. As in the previous experiment, we run 50 trials, each with a different initial condition, in which we iteratively learn a Bayesian parametric ROM, adaptively select a new training parameter value, and repeat until obtaining np=10n_p=10 samples. The POD basis matrix V\mathbf{V} is constructed with the rr principal left singular vectors of the snapshot data where rr is the smallest integer such that the cumulative energy

∑i=1rσi2∑k=1Kσk2\frac{\sum_{i=1}^{r}\sigma_i^2}{\sum_{k=1}^{K}\sigma_k^2}

exceeds 99.5%, where σ1≥σ2≥⋯≥σK\sigma_1 \ge\sigma_2 \ge\cdots\ge\sigma_K are the singular values of the snapshot matrix, as before. The following L1L^1-L2L^2 norm is appropriate for the shock-like phenomena in this problem:

∥q∥L1(Ω)2=∥(uv)∥L1(Ω)2=∫Ωu(x)2+v(x)2 dx,(46)\|q\|_{L^1(\Omega)}^2 = \left\|\begin{pmatrix} u \\ v \end{pmatrix}\right\|_{L^1(\Omega)}^2 = \int_{\Omega}\sqrt{u(x)^2+v(x)^2}\,dx, \tag*{(46)}

which we use to define the relative ROM error, total relative error, and relative projection error analogous to (4.6), (4.7), and (4.8), respectively.

Figure 8 shows the relative ROM and projection errors and the variance-based acquisition values in a single trial when there are np=1n_p = 1, 55, and 99 training parameter values. In this representative trial, when np=1n_p = 1 the ROM error is very near the projection error at the solitary training parameter value, but the ROM (and the basis) generalizes poorly to other parameter values. After additional parameter samples, however, both the projection error and the ROM error drop to a reasonable range throughout P\mathcal{P}, with the ROM nearly achieving the projection error at some (but not all) parameter values. The adaptive sampling scheme tends to select parameters near the boundaries of the parameter domain, though there are exceptions. As the number of training parameter samples npn_p increases, the acquisition values decrease and become better indicators for the trends in the ROM error. Thus, with more data the ROM becomes more confident in general but is also equipped with an informative uncertainty indicator.

Figure 8. Acquisition values φ(ν)\varphi(\nu) (top) and relative ROM and projection errors (bottom) after training on npn_p adaptively chosen parameter samples in the problem (4.9). The vertical lines show the locations of the parameter samples.

We repeat the procedure used to generate Figure 4 to analyze ROM stability in this problem, resulting in Figure 9. For np=2n_p = 2, random sampling produces slightly more stable ROMs than adaptive sampling on average, likely because the second adaptively chosen training parameter sample is often the smallest or largest possible value for ν\nu, and hence as different as possible from the initial parameter sample. On the other hand, stability over the entire parameter domain is difficult to achieve with random sampling, but is achieved in every trial with np=5n_p = 5 adaptively chosen training parameter samples. In terms of ROM error, shown in Figure 10, the total error is slightly better on average with adaptive sampling than with random sampling for np≥3n_p \ge3. The more important observation, however, is that adaptive sampling leads to accurate ROMs much more consistently than random sampling does. While some random sampling trials result in more accurate ROMs than the adaptive sampling ROMs, other random sampling trials also result in accuracy that is worse by about 10% compared to the adaptive sampling ROMs. As in the previous experiment, the projection errors, shown in Figure 11, are largely the same with both sampling strategies, but with adaptive sampling resulting in less variance across trials.

Figure 9. Percentages of the 50 training parameter candidates where any instabilities are observed from 50 posterior draws of the probabilistic ROM, as a function of the number of training parameters npn_p, for the problem (4.9). The results are averaged and maximized over 50 trials with different random initial parameter samples.

Figure 10. Total relative ROM errors EtotalE_{\mathrm{total}} as a function of the number of training parameter samples, aggregated across 50 trials with different initial samples, for the problem (4.9). The shaded regions show the interquantile range of 5%–95% of trials.

Figure 11. Total relative projection errors EprojE_{\mathrm{proj}} as a function of the number of training parameter samples, aggregated across 50 trials with different initial samples, for the problem (4.9). The shaded regions show the interquantile range of 5%–95% of trials.

Conclusion

We have introduced an uncertainty-aware active learning method for constructing data-driven parametric reduced-order models, tackling the challenge that ROM accuracy is sensitive to training data while full-order simulations are costly and hence the amount of available training data is limited. Our approach adaptively selects training parameters based on two proposed acquisition functions ensuring model stability and targeting regions of maximum predictive uncertainty. Numerical experiments on two nonlinear partial differential equations for thermal analysis and shock propagation demonstrate that our approach outperforms principled random sampling, achieving comparable ROM accuracy and superior stability with only about half as many full-order solves. Our approach shows significantly more robust results when repeated over different initial training sets, that is, the adaptive sampling scheme consistently leads to stable and accurate ROMs with as few samples as possible. While our applications focused on parameters appearing affinely in the operators, the methodology readily extends to parameters in initial conditions or input terms. Overall, we provide a general active learning framework for data-driven model reduction that leverages prediction uncertainty to produce efficient, robust, and uncertainty-informed reduced order models.

Future work may include exploring alternate acquisition functions for goal-oriented design, moving beyond greedy sequential selection to lookahead and batch selection strategies, and extending the framework to other ROM types beyond affine-structured operator inference. In particular, OpInf methods that guarantee stability, for example those in [20, 29], may improve our framework by allowing the adaptive sampling to focus on minimizing uncertainty rather than establishing stability. However, such an approach would require 1) extending these stability-focused methods to the parametric setting, and 2) constraining the posterior distribution from which the ROM is constructed to yield desired stability properties (for instance, enforcing symmetry or spectral properties in individual operators). Both of these goals remains a subject for future work. In general, however, the proposed active learning acquisition functions are directly applicable to any ROMs that provide prediction uncertainty estimates.

Another important challenge to address is working with higher-dimensional parameter spaces. Our proposed method is applicable to any set of parameter candidates (a strict grid layout is not required, see Remark 4.2), and sampling techniques such as sparse grid selection [10, 22, 58] can be used to construct a reasonable candidate set even in higher dimensions. However, constructing parametric ROMs that remain accurate over high-dimensional parameter domains, without resorting to domain decomposition or interpolation-based ROMs, remains an open challenge [6, 54]. Approaches for reducing the dimensionality of the parameter space may be paired with our framework since no geometric structure is required in the set of candidates. A related bottleneck for POD-based ROMs is that they can only be as good as the dimensionality reduction driven by linear manifolds, a problem that is exacerbated when the system behavior changes significantly with the parameters. Incorporating more sophisticated dimensionality reduction through nonlinear manifolds [5, 26, 40, 67], time- or parameter-adaptive bases [1, 16, 18], or by jointly optimizing the bases with ROM operators [20, 56, 66] are all potential directions to further improve the framework, especially for problems featuring higher-dimensional parameters.

Acknowledgments

A.C. acknowledges support from AFOSR grant FA9550-24-1-0327 under the Multidisciplinary University Research Initiatives (MURI), the NSF FDT-Biotech award 2436499, and the Texas Institute for Electronics (TIE) DARPA #HR00112430347.

References

  1. [1]D. AMSALLEM AND C. FARHAT, Interpolation method for adapting reduced-order models and application to aeroelasticity, AIAA Journal, 46 (2008), pp. 1803–1813, https://doi.org/10.2514/1.35374.
  2. [2]F. Andreuzzi, N. Demo, and G. Rozza, A dynamic mode decomposition extension for the forecasting of parametric dynamical systems, SIAM Journal on Applied Dynamical Systems, 22 (2023), pp. 2432–2458, https://doi.org/10.1137/22M1481658.
  3. [3]N. Aretz and K. Willcox, Enforcing structure in data-driven reduced modeling through nested Operator Inference, in 63rd IEEE Conference on Decision and Control (CDC), 2024, https://doi.org/10.1109/CDC56724.2024.10885857.
  4. [4]N. Aretz and K. Willcox, Nested operator inference for adaptive data-driven learning of reduced-order models, Advances in Computational Mathematics, 52 (2026), p. 49, https://doi.org/10.1007/s10444-026-10322-7.
  5. [5]J. Barnett and C. Farhat, Quadratic approximation manifold for mitigating the Kolmogorov barrier in nonlinear projection-based model order reduction, Journal of Computational Physics, 464 (2022), p. 111348, https://doi.org/10.1016/j.jcp.2022.111348.
  6. [6]P. Benner, S. Gügercin, and K. Willcox, A survey of projection-based model reduction methods for parametric dynamical systems, SIAM Review, 57 (2015), pp. 483–531, https://doi.org/10.1137/130932715.
  7. [7]D. Blais, S. Nadarajah, and C. Biondic, Goal-oriented adaptive sampling for projection-based reduced-order models, Computers & Fluids, 290 (2025), p. 106568, https://doi.org/10.1016/j.compfluid.2025.106568.
  8. [8]S. L. Brunton, J. L. Proctor, J. H. Tu, and J. N. Kutz, Compressed sensing and dynamic mode decomposition, Journal of Computational Dynamics, 2 (2015), p. 165, https://doi.org/10.3934/jcd.2015002.
  9. [9]A. Buffa, Y. Maday, A. T. Patera, C. Prud’homme, and G. Turinici, A priori convergence of the greedy algorithm for the parametrized reduced basis method, ESAIM: M2AN, 46 (2012), pp. 595–603, https://doi.org/10.1051/m2an/2011056, https://doi.org/10.1051/m2an/2011056.
  10. [10]H.-J. Bungartz and M. Griebel, Sparse grids, Acta Numerica, 13 (2004), p. 147–269, https://doi.org/10.1017/S0962492904000182.
  11. [11]K. Champion, B. Lusch, J. N. Kutz, and S. L. Brunton, Data-driven discovery of coordinates and governing equations, Proceedings of the National Academy of Sciences, 116 (2019), pp. 22445–22451, https://doi.org/10.1073/pnas.1906995116.
  12. [12]P. Y. Chen, J. Xiang, D. H. Cho, Y. Chang, G. A. Pershing, H. T. Maia, M. M. Chiaramonte, K. T. Carlberg, and E. Grinspun, CROM: Continuous reduced-order modeling of PDEs using implicit neural representations, in The Eleventh International Conference on Learning Representations, 2023, https://openreview.net/forum?id=FU0Rz1tG80g.arxiv.org/abs/2206.02607
  13. [13]L. Cicci, S. Fresca, M. Guo, A. Manzoni, and P. Zunino, Uncertainty quantification for nonlinear solid mechanics using reduced order models with Gaussian process regression, Computers & Mathematics with Applications, 149 (2023), pp. 1–23, https://doi.org/10.1016/j.camwa.2023.08.016.
  14. [14]D. A. Cohn, Z. Ghahramani, and M. I. Jordan, Active learning with statistical models, Journal of Artificial Intelligence Research, 4 (1996), pp. 129–145, https://doi.org/10.1613/jair.295.
  15. [15]C. Currin, T. Mitchell, M. Morris, and D. Ylvisaker, A Bayesian approach to the design and analysis of computer experiments, tech. report, Oak Ridge National Lab., TN (USA), 1988, https://doi.org/10.2172/814584.
  16. [16]J. DeChant, R. Geelen, S. A. McQuarrie, and J. Guilleminot, A dynamic subspace approach for low-rank approximation of large-scale nonlinear systems, arXiv, 2605.26025 (2026), https://doi.org/10.48550/arXiv.2605.26025.
  17. [17]P. den Boef, D. Manvelyan, J. Maubach, W. Schilders, and N. van de Wouw, Stable sparse operator inference for nonlinear structural dynamics, arXiv, 2407.21672 (2024), https://doi.org/10.48550/arXiv.2407.21672.
  18. [18]K. R. M. dos Santos, D. G. Giovanis, K. Kontolati, D. Loukrezis, and M. D. Shields, Grassmannian diffusion maps based surrogate modeling via geometric harmonics, International Journal for Numerical Methods in Engineering, 123 (2022), pp. 3507–3529, https://doi.org/10.1002/nme.6977.
  19. [19]J. Duan and J. S. Hesthaven, Non-intrusive data-driven reduced-order modeling for time-dependent parametrized problems, Journal of Computational Physics, 497 (2024), p. 112621, https://doi.org/10.1016/j.jcp.2023.112621.
  20. [20]C. J. Errico, A. Padovan, and D. J. Bodony, GasNiTROM: Model reduction via non-intrusive optimization of oblique projection operators and guaranteed-stable latent-space dynamics, arXiv, 2603.21254 (2026), https://doi.org/10.48550/arXiv.2603.21254.
  21. [21]I.-G. Farcas, R. Gundeviа, R. Munipalli, and K. E. Willcox, Parametric non-intrusive reduced-order models via operator inference for large-scale rotating detonation engine simulations, in AIAA SciTech 2023 Forum, 2023, p. 0172, https://doi.org/10.2514/6.2023-0172.
  22. [22]I.-G. Farcas, B. Uekermann, T. Neckel, and H.-J. Bungartz, Nonintrusive uncertainty analysis of fluid-structure interaction with spatially adaptive sparse grids and polynomial chaos expansion, SIAM Journal on Scientific Computing, 40 (2018), pp. B457–B482, https://doi.org/10.1137/16M1093975.
  23. [23]N. R. Franco, A. Manzoni, and P. Zunino, A deep learning approach to reduced order modelling of parameter dependent partial differential equations, Mathematics of Computation, 92 (2023), pp. 483–524, https://doi.org/10.1090/mcom/3781.
  24. [24]T. Franz, R. Zimmermann, S. Görtz, and N. Karcher, Interpolation-based reduced-order modelling for steady transonic flows via manifold learning, International Journal of Computational Fluid Dynamics, 28 (2014), pp. 106–121, https://doi.org/10.1080/10618562.2014.918695.
  25. [25]S. Fresca, L. Dede’, and A. Manzoni, A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized PDEs, Journal of Scientific Computing, 87 (2021), p. 61, https://doi.org/10.1007/s10915-021-01462-7.
  26. [26]R. Geelen, S. Wright, and K. Willcox, Operator inference for non-intrusive model reduction with quadratic manifolds, Computer Methods in Applied Mechanics and Engineering, 403 (2023), p. 115717, https://doi.org/10.1016/j.cma.2022.115717.
  27. [27]Y. Geng, J. Singh, L. Ju, B. Kramer, and Z. Wang, Gradient preserving Operator Inference: Data-driven reduced-order models for equations with gradient structure, Computer Methods in Applied Mechanics and Engineering, 427 (2024), p. 117033, https://doi.org/10.1016/j.cma.2024.117033.
  28. [28]O. Ghattas and K. Willcox, Learning physics-based models from data: Perspectives from inverse problems and model reduction, Acta Numerica, 30 (2021), pp. 445–554, https://doi.org/10.1017/S0962492921000064.
  29. [29]P. Goyal, I. Pontes Duff, and P. Benner, Guaranteed stable quadratic models and their applications in SINDy and operator inference, Physica D: Nonlinear Phenomena, 483 (2025), p. 134893, https://doi.org/10.1016/j.physd.2025.134893.
  30. [30]M. A. Grepl, Y. Maday, N. C. Nguyen, and A. T. Patera, Efficient reduced-basis treatment of nonaffine and nonlinear partial differential equations, ESAIM: Mathematical Modelling and Numerical Analysis, 41 (2007), pp. 575–605, https://doi.org/10.1051/m2an:2007031.
  31. [31]M. A. Grepl and A. T. Patera, A posteriori error bounds for reduced-basis approximations of parametrized parabolic partial differential equations, ESAIM: Mathematical Modelling and Numerical Analysis, 39 (2005), pp. 157–181, https://doi.org/10.1051/m2an:2005006.
  32. [32]A. Gruber and I. Tezaur, Variationally consistent Hamiltonian model reduction, SIAM Journal on Applied Dynamical Systems, 24 (2025), pp. 376–414, https://doi.org/10.1137/24M1652490.
  33. [33]D. Gunning and D. W. Aha, DARPA’s explainable artificial intelligence (XAI) program, AI Magazine, 40 (2019), pp. 44–58, https://doi.org/10.1609/aimag.v40i2.2850.
  34. [34]M. Guo and J. S. Hesthaven, Reduced order modeling for nonlinear structural analysis using Gaussian process regression, Computer Methods in Applied Mechanics and Engineering, 341 (2018), pp. 807–826, https://doi.org/10.1016/j.cma.2018.07.017.
  35. [35]M. Guo, S. A. McQuarrie, and K. Willcox, Bayesian operator inference for data-driven reduced-order modeling, Computer Methods in Applied Mechanics and Engineering, 402 (2022), p. 115336, https://doi.org/10.1016/j.cma.2022.115336.
  36. [36]Y. Guo, P. Nath, S. Mahadevan, and P. Witherell, Active learning for adaptive surrogate model improvement in high-dimensional problems, Structural and Multidisciplinary Optimization, 67 (2024), p. 122, https://doi.org/10.1007/s00158-024-03816-9.
  37. [37]Q. A. Huhn, M. E. Tano, J. C. Ragusa, and Y. Choi, Parametric dynamic mode decomposition for reduced order modeling, Journal of Computational Physics, 475 (2023), p. 111852, https://doi.org/10.1016/j.jcp.2022.111852.
  38. [38]A. C. Ionita and A. C. Antoulas, Data-driven parametrized model reduction in the Loewner framework, SIAM Journal on Scientific Computing, 36 (2014), pp. A984–A1007, https://doi.org/10.1137/130914619.
  39. [39]A. Ivagnes, N. Demo, and G. Rozza, Towards a machine learning pipeline in reduced order modelling for inverse problems: Neural networks for boundary parametrization, dimensionality reduction and solution manifold approximation, Journal of Scientific Computing, 95 (2023), p. 23, https://doi.org/10.1007/s10915-023-02142-4.
  40. [40]S. Jain, P. Tiso, J. B. Rutzmoser, and D. J. Rixen, A quadratic manifold for model order reduction of nonlinear structural dynamics, Computers & Structures, 188 (2017), pp. 80–94, https://doi.org/10.1016/j.compstruc.2017.04.005.
  41. [41]J. P. Kaipio and E. Somersalo, Statistical and Computational Inverse Problems, vol. 160 of Applied Mathematical Sciences, Springer, 2005.DOI
  42. [42]P. Khodabakhshi and K. E. Willcox, Non-intrusive data-driven model reduction for differential–algebraic equations derived from lifting transformations, Computer Methods in Applied Mechanics and Engineering, 389 (2022), p. 114296, https://doi.org/10.1016/j.cma.2021.114296.
  43. [43]H. Kim and B. Kramer, Physically consistent predictive reduced-order modeling by enhancing operator inference with state constraints, Journal of Computational Physics, 543 (2025), p. 114418, https://doi.org/10.1016/j.jcp.2025.114418.
  44. [44]B. Kramer, Stability domains for quadratic-bilinear reduced-order models, SIAM Journal on Applied Dynamical Systems, 20 (2021), p. 981–996, https://doi.org/10.1137/20M1364849.
  45. [45]B. Kramer, B. Peherstorfer, and K. Willcox, Learning nonlinear reduced models from data with operator inference, Annual Review of Fluid Mechanics, 56 (2024), pp. 521–548, https://doi.org/10.1146/annurev-fluid-121021-025220.
  46. [46]B. Kramer and E. Qian, Reduced-order modeling for engineering systems: Survey and opportunities for digital twins, Structural and Multidisciplinary Optimization, (2026), p. 141, https://doi.org/10.1007/s00158-026-04310-0.
  47. [47]K. Lee and K. T. Carlberg, Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders, Journal of Computational Physics, 404 (2020), p. 108973, https://doi.org/10.1016/j.jcp.2019.108973.
  48. [48]C. Lieberman, K. Willcox, and O. Ghattas, Parameter and state model reduction for large-scale statistical inverse problems, SIAM Journal on Scientific Computing, 32 (2010), pp. 2523–2542, https://doi.org/10.1137/090775622.
  49. [49]Y. Lin, Z. Gao, Y. Chen, and X. Sun, A dynamic mode decomposition based reduced-order model for parameterized time-dependent partial differential equations, Journal of Scientific Computing, 95 (2023), p. 70, https://doi.org/10.1007/s10915-023-02200-x.
  50. [50]D. J. MacKay, Bayesian methods for adaptive models, PhD thesis, California Institute of Technology, 1992.DOI
  51. [51]S. A. McQuarrie, Data-driven parametric reduced-order models: Operator inference for reactive flow applications, PhD thesis, The University of Texas at Austin, 2023, https://doi.org/10.26153/tsw/50172.
  52. [52]S. A. McQuarrie, A. Chaudhuri, K. E. Willcox, and M. Guo, Bayesian learning with Gaussian processes for low-dimensional representations of time-dependent nonlinear systems, Physica D: Nonlinear Phenomena, 475 (2025), p. 134572, https://doi.org/10.1016/j.physd.2025.134572.
  53. [53]S. A. McQuarrie, C. Huang, and K. Willcox, Data-driven reduced-order models via regularised operator inference for a single-injector combustion process, Journal of the Royal Society of New Zealand, 51 (2021), pp. 194–211, https://doi.org/10.1080/03036758.2020.1863237.
  54. [54]S. A. McQuarrie, P. Khodabakhs̄i, and K. Willcox, Nonintrusive reduced-order models for parametric partial differential equations via data-driven operator inference, SIAM Journal on Scientific Computing, 45 (2023), pp. A1917–A1946, https://doi.org/10.1137/21M1452810.
  55. [55]National Academy of Engineering and National Academies of Sciences, Engineering, and Medicine, Foundational Research Gaps and Future Directions for Digital Twins, The National Academies Press, Washington, DC, 2024, https://doi.org/10.17226/26894.
  56. [56]A. Padovan, B. Vollmer, and D. J. Bodony, Data-driven model reduction via non-intrusive optimization of projection operators and reduced-order dynamics, SIAM Journal on Applied Dynamical Systems, 23 (2024), pp. 3052–3076, https://doi.org/10.1137/24M1628414.
  57. [57]B. Peherstorfer and K. Willcox, Data-driven operator inference for nonintrusive projection-based model reduction, Computer Methods in Applied Mechanics and Engineering, 306 (2016), pp. 196–215, https://doi.org/10.1016/j.cma.2016.03.025.
  58. [58]B. Peherstorfer, K. Willcox, and M. Gunzburger, Survey of multifidelity methods in uncertainty propagation, inference, and optimization, SIAM Review, 60 (2018), pp. 550–591, https://doi.org/10.1137/16M1082469.
  59. [59]E. Qian, I.-G. Farcas, and K. Willcox, Reduced operator inference for nonlinear partial differential equations, SIAM Journal on Scientific Computing, 44 (2022), pp. A1934–A1959, https://doi.org/10.1137/21M1393972.
  60. [60]A. Quarteroni, G. Rozza, and A. Manzoni, Certified reduced basis approximation for parametrized partial differential equations and applications, Journal of Mathematics in Industry, 1 (2011), p. 3, https://doi.org/10.1186/2190-5983-1-3.
  61. [61]C. E. Rasmussen and C. K. Williams, Gaussian Processes for Machine Learning, MIT Press, Cambridge, MA, 2006.DOI
  62. [62]J. Sacks, W. J. Welch, T. J. Mitchell, and H. P. Wynn, Design and analysis of computer experiments, Statistical science, 4 (1989), pp. 409–423, https://doi.org/10.1214/ss/1177012413.
  63. [63]N. Sawant, B. Kramer, and B. Peherstorfer, Physics-informed regularization and structure preservation for learning stable reduced models from data with operator inference, Computer Methods in Applied Mechanics and Engineering, 404 (2023), p. 115836, https://doi.org/10.1016/j.cma.2022.115836.
  64. [64]T. Sayadi, P. J. Schmid, F. Richecoeur, and D. Durox, Parametrized data-driven decomposition for bifurcation analysis, with application to thermo-acoustically unstable systems, Physics of Fluids, 27 (2015), https://doi.org/10.1063/1.4913868.
  65. [65]P. J. Schmid, Dynamic mode decomposition and its variants, Annual Review of Fluid Mechanics, 54 (2022), pp. 225–254, https://doi.org/10.1146/annurev-fluid-030121-015835.
  66. [66]P. Schwerdtner, P. Mohan, J. Bessac, M. T. H. de Frahan, and B. Peherstorfer, Operator inference aware quadratic manifolds with isotropic reduced coordinates for nonintrusive model reduction, 2025, https://doi.org/10.48550/arXiv.2507.20463.
  67. [67]P. Schwerdtner and B. Peherstorfer, Greedy construction of quadratic manifolds for nonlinear dimensionality reduction and nonlinear model reduction, arXiv, 2403.06732 (2024), https://doi.org/10.48550/arXiv.2403.06732.
  68. [68]H. Sharma, D. A. Najera-Flores, M. D. Todd, and B. Kramer, Lagrangian operator inference enhanced with structure-preserving machine learning for nonintrusive model reduction of mechanical systems, Computer Methods in Applied Mechanics and Engineering, 423 (2024), p. 116865, https://doi.org/10.1016/j.cma.2024.116865.
  69. [69]H. Sharma, Z. Wang, and B. Kramer, Hamiltonian operator inference: Physics-preserving learning of reduced-order models for canonical Hamiltonian systems, Physica D: Nonlinear Phenomena, 431 (2022), p. 133122, https://doi.org/10.1016/j.physd.2021.133122.
  70. [70]A. M. Stuart, Inverse problems: A Bayesian perspective, Acta Numerica, 19 (2010), p. 451–559, https://doi.org/10.1017/S0962492910000061.
  71. [71]C. F. Van Loan, The ubiquitous Kronecker product, Journal of Computational and Applied Mathematics, 123 (2000), pp. 85–100, https://doi.org/10.1016/S0377-0427(00)00393-9.
  72. [72]A. Vijaywargiya, S. A. McQuarrie, and A. Gruber, Tensor parametric Hamiltonian operator inference, arXiv, 2502.10888 (2025), https://doi.org/10.48550/arXiv.2502.10888.
  73. [73]S. Yildiz, P. Goyal, P. Benner, and B. Karašözen, Learning reduced-order dynamics for parametrized shallow water equations from data, International Journal for Numerical Methods in Fluids, 93 (2021), pp. 2803–2821, https://doi.org/10.1002/fld.4998.
  74. [74]M. J. Zahr and C. Farhat, Progressive construction of a parametric reduced-order model for PDE-constrained optimization, International Journal for Numerical Methods in Engineering, 102 (2015), pp. 1111–1135, https://doi.org/10.1002/nme.4770.
  75. [75]B. G. Zastrow, A. Chaudhuri, K. E. Willcox, A. Ashley, and M. C. Henson, Block-structured operator inference for coupled multiphysics model reduction, AIAA Journal, Digital (2025), pp. 1–16, https://doi.org/10.2514/1.J065798.
  76. [76]Q. Zhuang, D. Hartmann, H.-J. Bungartz, and J. M. Lorenzi, Active-learning-based nonintrusive model order reduction, Data-Centric Engineering, 4 (2023), p. e2, https://doi.org/10.1017/dce.2022.39.
  77. [77]R. Zimmermann, Manifold interpolation, in Model Order Reduction Volume 1: System- and Data-Driven Methods and Algorithms, P. Benner, S. Grivet-Talocia, A. Quarteroni, G. Rozza, W. Schilders, and L. M. Silva, eds., Berlin, Boston, 2021, De Gruyter, pp. 229–274, https://doi.org/10.1515/9783110498967–007.

Paper details

vlnc:W_bxyh737g
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents