betaFeedback

WorksComputer Science

Evolutionary Optimization of Physics-Informed Neural Networks: Advancing Generalizability by the Baldwin Effect

Abstract

Physics-informed neural networks (PINNs) are at the forefront of scientific machine learning, making possible the creation of machine intelligence that is cognizant of physical laws and able to accurately simulate them. However, today's PINNs are often trained for a single physics task and require computationally expensive re-training for each new task, even for tasks from similar physics domains. To address this limitation, this paper proposes a pioneering approach to advance the generalizability of PINNs through the framework of Baldwinian evolution. Drawing inspiration from the neurodevelopment of precocial species that have evolved to learn, predict and react quickly to their environment, we envision PINNs that are pre-wired with connection strengths inducing strong biases towards efficient learning of physics. A novel two-stage stochastic programming formulation coupling evolutionary selection pressure (based on proficiency over a distribution of physics tasks) with lifetime learning (to specialize on a sampled subset of those tasks) is proposed to instantiate the Baldwin effect. The evolved Baldwinian-PINNs demonstrate fast and physics-compliant prediction capabilities across a range of empirically challenging problem instances with more than an order of magnitude improvement in prediction accuracy at a fraction of the computation cost compared to state-of-the-art gradient-based meta-learning methods. For example, when solving the diffusion-reaction equation, a 70x improvement in accuracy was obtained while taking 700x less computational time. This paper thus marks a leap forward in the meta-learning of PINNs as generalizable physics solvers. Sample codes are available at https://github.com/chiuph/Baldwinian-PINN.

Introduction

The emerging field of scientific machine learning seeks to create more accurate, data-efficient, and explainable machine intelligence for science and engineering. The direct incorporation of mathematically expressible laws of nature into

learned models to ensure physically consistent predictions is an appealing proposition, as evidenced by the proliferation of physics-informed neural networks (PINNs) across multiple scientific domains since seminal work by Raissi et al. [1]. The key concept is to utilize physics-based mathematical relations or constraints as a regularization loss (aka physics-informed loss). This physics-informed loss is amenable to various forms of scientific knowledge and theories, including fundamental ordinary or partial differential equations (ODEs or PDEs). It flexibly incorporates scientific discoveries accumulated across centuries into the machine intelligence models of today across diverse scientific and engineering disciplines [2–6].

However, PINNs remain limited in their ability to generalize across physics scenarios. Contrary to its promise, a PINN does not guarantee compliance with physics when used for new scenarios unseen during training, e.g., variations in PDE parameters, initial conditions (ICs) or boundary conditions (BCs) that lie outside the confines of their training. Instead, these predictions remain physics-agnostic and may experience similar negative implications for reliability as typical data-driven models.

In principle, physics-compliant predictions for any new scenario can be achieved by performing physics-informed retraining—an attractive feature of PINNs—even without labelled data. However, the additional training can be cost-prohibitive as physics-informed learning is much more difficult than data-driven learning due to the ruggedness of the loss landscapes [7–10]. This has motivated the exploration of transfer learning techniques where connection strengths from similar (source) physics scenarios are used to facilitate accurate learning of solutions for new, harder problems [11–13]. The related notion of meta-learning seeks to discover an optimized initialization of a model to enable rapid adaptation to a new test task with minimal training [11]. Nonetheless, most transfer- and meta-learned PINNs proposed to date still require the aforementioned physics-informed retraining (with a substantial number of optimization iterations) to achieve more accurate solutions, and are therefore not ideal for applications that call for repeated, fast evaluations. A method to arrive at a generalizable PINN, one that can provide fast and accurate physics prediction/simulation across a varied set of unseen scenarios, remains elusive.

In search of machine intelligence that generalizes for physically-consistent simulations of varied processes in the natural world, this paper studies the meta-learning of PINNs through the lens of neural Baldwinism—an expression of the Baldwin effect in the evolution of brains and intelligence [14]. Inspiration is drawn from the transmission of knowledge and predispositions across generations in precocial species, whereby their young are “born ready” with strong learning biases to perform a wide range of tasks. In order to pre-wire such learning ability into the initial connection strengths of a neural network, we examine an algorithmic realization of neural Baldwinism in the context of PINNs. The essence of Baldwinism lies in the phenomenon that characters learned by individual organisms of a group during their lifetime may eventually, under selection pressure, get reinforced by associated hereditary characters [15]. Analogously, Baldwinian neuroevolution of PINNs consists of an outer evolutionary optimization loop in which populations of PINN models are collectively exposed to a wide range of physics tasks sampled from a probability distribution over tasks of interest. Models with higher propensity to perform a random subset of those tasks well in an inner lifetime learning loop are evaluated as being fitter for survival, thus inducing a selection pressure towards connection strengths that encode stronger learning biases. The parallels between neural Baldwinism in nature and that of PINNs is depicted in Figure 1. The mathematical formulation of the problem resembles a two-stage stochastic program [16, 17], where the solution to lifetime learning enables the PINN to rapidly specialize to a specific physics task at test time.

Figure 1. Schematic diagram of (a) Baldwinian evolution in nature and (b) evolving machine intelligence for learning physics with Baldwinian-PINNs. In nature, the Baldwin effect describes how learned traits are eventually reinforced in the genetic makeup of a population of organisms through natural selection. Equivalently, a population of Baldwinian-PINNs evolves over generations by being exposed to a broad distribution of physics tasks, gradually reinforcing traits promoting accurate physics learning into their genetic makeup. The evolved Baldwinian-PINNs are inherently equipped with strong learning biases to accurately solve any physics tasks over a broad task distribution.

Harnessing evolutionary procedures to optimize neural networks lends much greater versatility relative to other meta-learning approaches in terms of jointly crafting network architectures, initial network connection strengths, as well as learning algorithms, all through the use of potentially non-differentiable fitness functions [18]. Such biological “neuroevolution” [19, 20] precludes the need for explicit parameterization of tasks, facilitating generalization over task distributions comprising any broad mix of ODE/PDEs, ICs, and BCs. The physics-informed lifetime learning of the neural network can be accelerated by reduction to a least-squares learning problem in its final layer, making Baldwinian neuroevolution computationally feasible. Such a least-squares formulation yields a closed-form result by means of the Tikhonov regularization [21] and guarantees zero physics-informed loss given a sufficiently overparameterized network. The closed-form expression vastly reduces (or even eliminates) the need for iterative parameter updates to enable extremely fast lifetime learning of desired physics. Critically, the evolutionary search procedure is inherently highly parallelizable, thereby allowing for efficient meta-learning at scale by capitalizing on state-of-the-art advances in multi-CPU/GPU hardware infrastructure.

Neural Baldwinism thus makes it possible to achieve generalizable PINNs that are “genetically equipped” to perform well over a wide range of physics tasks. The evolved models (referred to as Baldwinian-PINNs) are demonstrated in this study to be broadly applicable to the simulation of families of linear and nonlinear ODE/PDEs encompassing diverse physical phenomena such as particle kinematics, heat and mass transfer, and reaction-diffusion. These Baldwinian-PINNs are capable of fast and accurate physics-aware predictions on 大发快三是不是 previously unseen tasks, demonstrating up to several orders of magnitude computation speedup along with an order of magnitude improvement in prediction accuracy relative to recent meta-learned PINNs [22, 23].

The key contributions of this paper are summarized below.

  • This is the first study to unveil Baldwinian evolution as a compelling route towards discovering neural nets with high capacity to learn diverse physics tasks, thereby advancing generalizability in PINNs.

  • An instantiation of the Baldwinian evolution framework is proposed through a novel two-stage stochastic programming formulation, wherein the first stage evolves the initial layers of a generalizable PINN model and the second stage trains its final layers to specialize to any new physics task (analogous to lifetime learning).

  • Comprehensive numerical experimentation and analysis shows that this methodology produces PINNs that effectively learn and predict across multiple PDE problems spanning different physics domains, demonstrating significant advancements in terms of speed and accuracy over models meta-learned by gradient descent.

The remainder of the paper is organized as follows. The basic problem setup for a PINN, the notion of generalization over physics tasks, and related work are presented in Section II. The proposed methodology for evolving Baldwinian-PINNs is detailed in Section III. Extensive numerical assessment of the method is carried out in Section IV over a range of linear and nonlinear ODE/PDEs. The paper is concluded in Section V with a discussion of future research directions.

Preliminaries

Problem setup

1) Single PINN problem: For simplicity of exposition, let us consider a problem with single spatial dimension xx, time dimension tt, and a single variable of interest uu. In general, PINNs can learn a mapping between the input variables (x,t)(x,t) and the output variable uu while satisfying specified governing equations representing the physical phenomenon or dynamical process of interest:

PDE:Nϑ[u(x,t)]=h(x,t),x∈Ω, t∈(0,T]IC:u(x,t=0)=u0(x),x∈ΩBC:B[u(x,t)]=g(x,t),x∈∂Ω, t∈(0,T](1)\begin{aligned} \mathrm{PDE}: \quad&\mathcal{N}_{\vartheta}[u(x,t)] = h(x,t), \qquad x \in\Omega,\ t \in(0,T] \\ \mathrm{IC}: \quad&u(x,t=0) = u_0(x), \qquad x \in\Omega\\ \mathrm{BC}: \quad&\mathcal{B}[u(x,t)] = g(x,t), \qquad x \in\partial\Omega,\ t \in(0,T] \tag*{(1)} \end{aligned}

where the general differential operator Nϑ[u(x,t)]\mathcal{N}_{\vartheta}[u(x,t)] can include linear and/or nonlinear combinations of temporal and spatial derivatives and PDE parameters ϑ\vartheta, and h(x,t)h(x,t) is an arbitrary source term in the domain x∈Ωx \in\Omega, t∈(0,T]t \in(0,T]. The IC (Eq. 1b) specifies the initial state, u0(x)u_0(x), at time t=0t=0, and the BC (Eq. 1c) specifies that B[u(x,t)]\mathcal{B}[u(x,t)] equates to g(x,t)g(x,t) at the domain boundary ∂Ω\partial\Omega.

Crucially, individual PINN models arrive at an accurate and physics-compliant prediction u(x,t)u(x,t) for a single target scenario by minimizing the discrepancy between Eq. 1 and the model’s prediction during training.

**2) Generalizable neural physics solver: ** While most PINN models are trained to solve a specific physics task, there is increasing interest in generalizable neural physics solvers, i.e. models that can be flexibly applied to multiple physics problems once trained. In this context, we can consider a physical phenomena of interest that is represented by a set of training tasks belonging to some underlying task-distribution p(T)p(\mathcal{T}), e.g., a family of PDEs spanning different PDE parameters ϑ\vartheta, different ICs u0(x)u_0(x) and/or different BCs g(x,t)g(x,t).

Hence, the goal is to discover generalizable PINN models capable of fast, accurate, and physics-aware predictions on unseen scenarios, i.e., any new task from the distribution, Ti∼p(T)\mathcal{T}_i \sim p(\mathcal{T}), by learning the underlying governing physics. In the context of meta-learning, the learning objective is to use training tasks from p(T)p(\mathcal{T}) to find network initializations that are most amenable to a quick and accurate solution for the PINN loss, thereby accelerating the solution of multiple related physics problems at test time and providing a potential route to a generalizable neural physics solver.

PINN models are usually trained to solve a single, specific physics task. However, recent studies on meta-learning of PINNs to solve different physics scenarios as separate tasks have emerged, although no work has been reported from the neuroevolution perspective to our knowledge.

Most of the meta-learning PINN approaches reported in literature use weight interpolation as the basic framework. The simplest instantiation of this approach is to first train independent PINN models for each task and then interpolate across model weights for the new task [22, 24]. Several interpolation methods such as the Gaussian Process (GP) and Radial Basis Function (RBF) have been studied and shown to improve physics-informed learning on new tasks [22]. In other studies, interpolation is learned through the use of hypernetworks, whereby the hypernetworks and PINN are trained simultaneously from all the tasks [25, 26]. Other variants include encoding tasks as latent variables and passing them into the input layer of the PINN model [27].

However, there are two major drawbacks to such weight interpolation frameworks. Firstly, these methods rely heavily on task parameterization to perform interpolation and operate under the assumption of smoothness. This means that all the train tasks (and the new task) must adhere to this parameterization requirement. In addition, these methods do not capitalize on the principal characteristic of PINN, which is the potential for physics-informed learning (retraining or fine-tuning) for a new task, in their meta-learning formulation. The data-free nature of physics-informed fine-tuning is not exploited during the meta-learning phase, given that there is no guiding principle on how the fine-tuning towards any new task should be performed given the interpolated weights.

The model-agnostic meta-learning (MAML) [28] framework can theoretically overcome the limitations of weight interpolation methods. MAML and its variant, Reptile [29], aim to learn an optimal weight initialization that can be quickly fine-tuned on new tasks without the need for interpolation [23]. An illustration of MAML for PINN is shown in

Figure 2(a). To reduce the computation cost of meta-learning, task-specific physics-informed learning is limited to one or a few gradient descent updates from the initialization. This essentially assumes that the model performance after one step (or a few steps of) gradient descent update from the initialization is already indicative of the learning performance. While this assumption may be appropriate in data-driven models, PINN models are much more challenging to optimize. Hence, they require more thorough training (e.g., a large number of updates, even with the use of 2nd-order gradient descent methods) to achieve a good convergence. Moreover, gradient-based MAML methods are prone to getting stuck in a local minimum and struggle to find a good initialization for a diverse set of tasks, as the gradients can be noisy or even deceptive, i.e., requiring going against it to reach the optimum. Hence, recent studies have shown poor MAML results, potentially due to non-convergence during meta-learning or insufficient physics-informed fine-tuning performed at test time [22, 25].

Meta-learning PINN with Baldwinian neuroevolution versus MAML, and Baldwinian-PINN architecture

Figure 2. (a) Meta-learning PINN with Baldwinian neuroevolution (right) versus MAML (left). In MAML, the initial weights WW are learned using gradient-based method, such that they can be quickly fine-tuned (physics-informed learning) on new tasks. Although the task-specific fine-tuning is limited to one or a few gradient descent updates during training, such amount of fine-tuning is usually insufficient for a PINN at test time. In Baldwinian neuroevolution, the weight distribution in the pre-final nonlinear hidden layers w~\widetilde{w} and learning hyperparameters λ\lambda are jointly evolved. The task-specific physics-informed learning is performed on the final layer weights ww (segregated from w~\widetilde{w}) with a 1-step Tikhonov regularization operation at both training and test time (for linear ODE/PDEs). (b) Schematic of Baldwinian-PINNs architecture used in present study and procedure to obtain nonlinear hidden layers’ weights w~\widetilde{w}’s from the evolved network hyperparameter θ\theta for learning task-specific outputs.

Proposed Methodology

We address the above limitations with a meta-learning PINN framework based on Baldwinian neuroevolution. A novel evolutionary algorithm is crafted to jointly optimize the weight distribution in the network’s hidden layers alongside other hyperparameters that are essential for achieving optimal performance on downstream physics-informed learning tasks. We limit our task-specific learning to the final layer, allowing us to solve linear problems with a 1-step Tikhonov regularization operation (or a few steps for nonlinear problems) during both the meta-learning process and test time, in a consistent manner. Importantly, once given the nonlinear hidden layers’ weights and learning hyperparameters, the Tikhonov regularization is exact and fast (does not require further gradient descent updates). An illustration of the Baldwinian evolution of PINNs is shown in Figure 2(a).

The Baldwin effect as a two-stage stochastic program

Without loss of architectural generality, Baldwinian-PINNs can be assigned the form of a multilayer perceptron (MLP) with proven representation capacity [30, 31], thereby ensuring the ability to learn a diversity of tasks. Its output u(x,t)u(x,t) can be written as:

u(x,t)=∑jwjfj(x,t;w~)(2)u(x,t) = \sum_j w_j f_j(x,t;\widetilde{w}) \tag*{(2)}

where w=[…wj…]T\mathbf{w} = [\ldots w_j \ldots]^{\mathrm{T}} are the final layer weights, and fj(x,t;w~)f_j(x,t;\widetilde{w})’s represent nonlinear projections of the input variables with the hidden layers’ weights w~\widetilde{w}. The connection strengths w~\widetilde{w} are typically learned during PINN training. However, in the proposed implementation of Baldwinian-PINNs as detailed below, they are drawn from a probability distribution defined by a dimensionally reduced set of network hyperparameters θ\theta—see example in Section III-D1 and Figure 2(b)—that are assigned at birth as per Baldwinian evolution.

The model’s reaction to a given environment—i.e., finding the best set of wjw_j’s such that the model’s output satisfies Eq. 1 for a specific physics task Ti\mathcal{T}_i—is the focus of a typical PINN described in Section II-A. This is reduced to a physics-informed least-squares problem in this work:

w∗=arg⁡min⁡w~(Aw~Tiw−bTi)T(Aw~Tiw−bTi)+λwTw(3)\mathbf{w}^{*} = \underset{\widetilde{w}}{\arg\min}\left(\mathbf{A}_{\widetilde{w}}^{\mathcal{T}_i}\mathbf{w}-\mathbf{b}^{\mathcal{T}_i}\right)^{\mathrm{T}}\left(\mathbf{A}_{\widetilde{w}}^{\mathcal{T}_i}\mathbf{w}-\mathbf{b}^{\mathcal{T}_i}\right)+\lambda\mathbf{w}^{\mathrm{T}}\mathbf{w} \tag*{(3)}

where Aw~Tiw\mathbf{A}_{\widetilde{w}}^{\mathcal{T}_i}\mathbf{w} is obtained by substituting the model’s output into the left hand side of Eq. 1 for a given set of collocation points, and bTi\mathbf{b}^{\mathcal{T}_i} represents the corresponding right hand side of Eq. 1. Eq. (3) yields a closed-form solution w∗\mathbf{w}^{*} when using the Moore-Penrose generalized inverse, permitting extremely fast learning (order of milli-seconds in our experiments) for physics-compliant prediction. Similar least-squares formulations studied in the literature have been shown to be competitive with, or faster than, widely-used numerical solvers such as the finite element method [32, 33]. The learning hyperparameter λ≥0\lambda\ge0 reduces the L2L^2-norm of the least-squares solution, thereby improving the solution numerically. Detailed derivations, including information on the construction of matrix Aw~Ti\mathbf{A}_{\widetilde{w}}^{\mathcal{T}_i} and vector bTi\mathbf{b}^{\mathcal{T}_i}, are provided in Section III-C.

Going beyond specialization to a single physics task by means of Eq. (3), we formulate the search for Baldwinian-PINN models that can generalize to a whole family of PDEs as the following two-stage stochastic optimization problem [17]:

min⁡θ,λ ETi∼p(T)Ew~∼pθ(w~)[κLSElLSE(w∗)+κMSElMSE(w∗)]\underset{\theta,\lambda}{\min}\ \mathbb{E}_{\mathcal{T}_i\sim p(\mathcal{T})}\mathbb{E}_{\widetilde{w}\sim p_{\theta}(\widetilde{w})}\left[\kappa_{LSE}l_{LSE}(\mathbf{w}^{*})+\kappa_{MSE}l_{MSE}(\mathbf{w}^{*})\right]
subject to κLSE≥0, κMSE≥0(4)\text{subject to }\kappa_{LSE}\ge0,\ \kappa_{MSE}\ge0 \tag*{(4)}

where w∗\mathbf{w}^{*} is the solution of the second stage problem defined earlier in Eq. (3), which provides optimal final layer weights that allow the model to specialize to any realization of task Ti∼p(T)\mathcal{T}_i\sim p(\mathcal{T}) for the given network’s w~∼pθ(w~)\widetilde{w}\sim p_{\theta}(\widetilde{w}).

The first stage optimization objective is defined by the weighted sum of the physics learning proficiency, i.e., sum of squared residuals or least-squares error (LSE),

lLSE(w∗)=(Aw~Tiw∗−bTi)T(Aw~Tiw∗−bTi)(5)l_{LSE}(\mathbf{w}^{*})=\left(\mathbf{A}_{\widetilde{w}}^{\mathcal{T}_i}\mathbf{w}^{*}-\mathbf{b}^{\mathcal{T}_i}\right)^{\mathrm{T}}\left(\mathbf{A}_{\widetilde{w}}^{\mathcal{T}_i}\mathbf{w}^{*}-\mathbf{b}^{\mathcal{T}_i}\right) \tag*{(5)}

and the actual predictive performance, i.e., mean squared error (MSE),

lMSE(w∗)=1n∑s=1n(uslabel−∑jwj∗fj(xs,ts;w~))2(6)l_{MSE}(\mathbf{w}^{*})=\frac{1}{n}\sum_{s=1}^{n}\left(u_s^{label}-\sum_j w_j^{*}f_j(x_s,t_s;\widetilde{w})\right)^2 \tag*{(6)}

given labelled data uslabelu_s^{label}, s=1,…,ns=1,\ldots,n, over the task distribution p(T)p(\mathcal{T}) and the network connections’ distribution pθ(w~)p_{\theta}(\widetilde{w}).

In what follows, we present a novel procedure for solving the two-stage stochastic program via an evolutionary algorithm. The hyperparameters θ,λ\theta,\lambda are evolved from one generation to the next, with their fitness evaluations for the first stage problem conditioned on lifetime learning to obtain the optimal w∗\mathbf{w}^{*} from the second stage problem. Note that while the outcome of lifetime learning influences the selection pressure acting on the evolving hyperparameters, it does not directly alter the genetic makeup of the hyperparameters within a generation. As such, our overall method exhibits a clear connection with the evolutionary principles of Baldwinism.

Baldwinian neuroevolution

The Baldwinian neuroevolution procedure to solve the two-stage stochastic programming problem defined in Section III-A is described in Algorithm 1 and Algorithm 2. Recall from Figure 2(b) that a Baldwinian-PINN is represented by (θ,λ)(\theta,\lambda), i.e., distribution of network weights and biases and lifetime learning hyperparameters, during neuroevolution.

Algorithm 1 Baldwinian neuroevolution of PINNs

INPUT: training tasks distribution, p(T)p(\mathcal{T})

OUTPUT: best solution found (i.e., Baldwinian-PINN model’s weights and biases in nonlinear hidden-layers and lifetime learning hyperparameters (θ,λ)(\theta,\lambda))

Require: F\mathcal{F}: procedure to return fitness based on lifetime learning performance of individual given a batch of tasks (described in detail in Algorithm 2)

  1. Initialize population P\mathcal{P}

  2. while not done do

  3. Sample a new batch of offspring (θg,λg)(\theta^g,\lambda^g), g=1,…,npopg=1,\ldots,n_{pop} from the population P\mathcal{P}

  4. Sample batch of tasks Ti∼p(T)\mathcal{T}_i\sim p(\mathcal{T}), i=1,…,ntaski=1,\ldots,n_{task}

  5. for all (θg,λg)(\theta^g,\lambda^g) do

  6. fg=F(θg,λg,{Ti,i=1,…,ntask})f^g=\mathcal{F}(\theta^g,\lambda^g,\{\mathcal{T}_i,i=1,\ldots,n_{task}\})

  7. end for

  8. Update population P\mathcal{P} based on fitness of individuals: {(θg,λg,fg),g=1,…,npop}\{(\theta^g,\lambda^g,f^g),g=1,\ldots,n_{pop}\}

  9. end while

  10. Return best individual (θ,λ)(\theta,\lambda) found in population P\mathcal{P}

The Baldwinian neuroevolution procedure described in Algorithm 1 is generic for evolutionary optimization methods. The algorithm initializes a population P\mathcal{P} of Baldwinian-PINN

Algorithm 2 Baldwinian-PINN lifetime learning and fitness calculation F\mathcal{F}

INPUT: network and lifetime learning hyperparameters (θ,λ)(\theta,\lambda), batch of tasks Ti\mathcal{T}_i, i=1,…,ntaski=1,\ldots,n_{\mathrm{task}}

OUTPUT: fitness ff

Require: G\mathcal{G}: procedure to sample weights and biases in nonlinear hidden-layers (exemplified in Section III-D1 and Figure. Figure 2(b))

Require: C\mathcal{C}: procedure to construct least squares problem on a set of collocation points based on underlying physics (PDEs, BCs, ICs) of the task (described in detail in Section III-C)

  1. Sample hidden layers’ weights and biases w~=G(θ)\widetilde{\mathbf{w}}=\mathcal{G}(\theta)

  2. for all Ti\mathcal{T}_i do

  3. Construct least squares matrix and vector: (A,b)=C(Ti,w~)(\mathbf{A},\mathbf{b})=\mathcal{C}(\mathcal{T}_i,\widetilde{\mathbf{w}})

  4. Compute least squares solution as per Eq. (8):

    w∗=(λI+ATA)−1Ab\mathbf{w}^{*}=(\lambda\mathbf{I}+\mathbf{A}^{\mathsf{T}}\mathbf{A})^{-1}\mathbf{A}\mathbf{b}
  5. Compute least squares error (LSE) as per Eq. (5):

    lLSETi=(Aw∗−b)T(Aw∗−b)l_{\mathrm{LSE}}^{\mathcal{T}_i}=(\mathbf{A}\mathbf{w}^{*}-\mathbf{b})^{\mathsf{T}}(\mathbf{A}\mathbf{w}^{*}-\mathbf{b})
  6. Compute mean squared error (MSE) as per Eq. (6):

    lMSETi=1n∑sn(ulabel−∑jwj∗fj(xs,ts;w~j))2l_{\mathrm{MSE}}^{\mathcal{T}_i}=\frac{1}{n}\sum_s^{n}\left(u_{\mathrm{label}}-\sum_j w_j^{*}f_j(x_s,t_s;\widetilde{\mathbf{w}}_j)\right)^2
  7. end for

  8. Compute overall fitness: f=−(κLSE∑TilLSETi+κMSE∑TilMSETi)f=-(\kappa_{\mathrm{LSE}}\sum_{\mathcal{T}_i}l_{\mathrm{LSE}}^{\mathcal{T}_i}+\kappa_{\mathrm{MSE}}\sum_{\mathcal{T}_i}l_{\mathrm{MSE}}^{\mathcal{T}_i})

models given by different weights and biases in the nonlinear hidden-layers and lifetime learning hyperparameters (θ,λ)(\theta,\lambda). For a probabilistic model-based evolutionary algorithm, P\mathcal{P} is commonly represented by a search distribution. In each generation, npopn_{\mathrm{pop}} new offspring individuals are sampled from the search distribution (probabilistic model-based EAs) or through crossover/mutation (traditional EAs), and their fitness are evaluated for a batch of tasks randomly sampled from the training task distribution p(T)p(\mathcal{T}). The fitness evaluation procedure F\mathcal{F} gives the lifetime learning outcome fgf^g of these offspring individuals (θg,λg)(\theta^g,\lambda^g), g=1,…,npopg=1,\ldots,n_{\mathrm{pop}} for the given tasks. In line with the essence of Baldwinism, the lifetime learning procedure (described in detail in Algorithm 2) does not alter the genetic makeup (θ,λ)(\theta,\lambda) of the individuals. The outcome of the lifetime learning procedure specifies the fitness which creates the selection pressure influencing the evolution of the population, but is not directly inherited by the next population (unlike Lamarckian evolution). The Baldwinian neuroevolution algorithm iteratively adapts P\mathcal{P} towards offspring with better fitness until the convergence criteria, e.g., a pre-determined number of generations or fitness value (trade-off between computation resource and convergence), is met.

Given an individual (θ,λ)(\theta,\lambda) sampled from the search distribution and a batch of tasks sampled from the training distribution, Ti∼p(T)\mathcal{T}_i\sim p(\mathcal{T}), i=1,…,ntaski=1,\ldots,n_{\mathrm{task}}, the procedure F\mathcal{F} to return fitness is detailed in Algorithm 2. It starts with a procedure G\mathcal{G} to populate nonlinear hidden-layers’ weights and biases w~\widetilde{\mathbf{w}} of a Baldwinian-PINN from the sampled individual θ\theta. In the present study, Baldwinian-PINNs are designed to have w~\widetilde{\mathbf{w}} fixed at birth to random values drawn from a probability

distribution defined by a dimensionally reduced set of network hyperparameters θ\theta, akin to a randomized neural networks setup [17, 34]. This procedure is exemplified in Section III-D1. Then, lifetime learning of the Baldwinian-PINN is performed to obtain the optimal network weights w∗\mathbf{w}^{*} in the final layer, for each of the sampled tasks Ti\mathcal{T}_i. It involves procedure C\mathcal{C} to construct least squares problem, i.e., matrix and vector (A,b)(\mathbf{A},\mathbf{b}), on a fixed set of collocation points based on underlying physics (PDEs, BCs, ICs) of the task. This procedure is detailed in Section III-C. We choose the collocation points to coincide with the location of labelled data for MSE computation, although this is not a prerequisite. The overall fitness ff is computed by aggregating the LSE and MSE based on lifetime learning outcomes for all the sampled tasks.

In the spirit of the Baldwin effect, the task-specific final layer w∗\mathbf{w}^{*} is the outcome of lifetime learning and not directly inherited by the next generation of offspring [35]; only the hyperparameters (θ,λ)(\theta,\lambda) are subjected to evolutionary variation and inheritance.

Baldwinian-PINN lifetime learning procedure

The lifetime learning procedure of the Baldwinian-PINNs occurs only in the linear final layer of the network, i.e., finding the best set of wjw_j’s such that the output u(x,t)=∑jwjfj(x,t;w~)u(x,t)=\sum_j w_jf_j(x,t;\widetilde{\mathbf{w}}) satisfies the governing equations in Eq. 1 for a specific task. Given a set of collocation points (xipde,tipde)(x_i^{pde},t_i^{pde}), i=1,…,npdei=1,\ldots,n_{pde}, (xiic,0)(x_i^{ic},0), i=1,…,nici=1,\ldots,n_{ic}, (xibc,tibc)(x_i^{bc},t_i^{bc}), i=1,…,nbci=1,\ldots,n_{bc} sampled from the respective domain, the following system of equations can be formed:

[⋯Nθ[fj(x1pde,t1pde;w~)]⋯⋯Nθ[fj(xnpdepde,tnpdepde;w~)]⋯⋯fj(x1ic,0;w~)⋯⋯fj(xnicic,0;w~)⋯⋯B[fj(x1bc,t1bc;w~)]⋯⋯B[fj(xnbcbc,tnbcbc;w~)]⋯][⋮wj⋮]=[h(x1pde,t1pde)⋮h(xnpdepde,tnpdepde)u0(x1ic)⋮u0(xnicic)g(x1bc,t1bc)⋮g(xnbcbc,tnbcbc)](7)\begin{bmatrix} \cdots& \mathcal{N}_{\theta}[f_j(x_1^{pde},t_1^{pde};\widetilde{\mathbf{w}})] & \cdots\\ \cdots& \mathcal{N}_{\theta}[f_j(x_{n_{pde}}^{pde},t_{n_{pde}}^{pde};\widetilde{\mathbf{w}})] & \cdots\\ \cdots& f_j(x_1^{ic},0;\widetilde{\mathbf{w}}) & \cdots\\ \cdots& f_j(x_{n_{ic}}^{ic},0;\widetilde{\mathbf{w}}) & \cdots\\ \cdots& \mathcal{B}[f_j(x_1^{bc},t_1^{bc};\widetilde{\mathbf{w}})] & \cdots\\ \cdots& \mathcal{B}[f_j(x_{n_{bc}}^{bc},t_{n_{bc}}^{bc};\widetilde{\mathbf{w}})] & \cdots \end{bmatrix} \begin{bmatrix} \vdots\\ w_j\\ \vdots \end{bmatrix} = \begin{bmatrix} h(x_1^{pde},t_1^{pde})\\ \vdots\\ h(x_{n_{pde}}^{pde},t_{n_{pde}}^{pde})\\ u_0(x_1^{ic})\\ \vdots\\ u_0(x_{n_{ic}}^{ic})\\ g(x_1^{bc},t_1^{bc})\\ \vdots\\ g(x_{n_{bc}}^{bc},t_{n_{bc}}^{bc}) \tag*{(7)} \end{bmatrix}

Aw=b\mathbf{A}\mathbf{w}=\mathbf{b}

Note that the derivatives required to construct A\mathbf{A} can be easily computed by automatic differentiation [36]. The best-fit solution to the above system of linear equations with unknown w=[…wj…]T\mathbf{w}=[\ldots w_j\ldots]^{\mathsf{T}} can be obtained by means of the Tikhonov regularization:

w∗=(λI+ATA)−1ATb(8)\mathbf{w}^{*}=(\lambda\mathbf{I}+\mathbf{A}^{\mathsf{T}}\mathbf{A})^{-1}\mathbf{A}^{\mathsf{T}}\mathbf{b} \tag*{(8)}

where the learning hyperparameter λ≥0\lambda\geq0 reduces the L2L^2-norm of the least-squares solution, thereby improving the solution numerically. Such a least-squares formulation yields a closed-form result in a single computation step for any linear PDE (i.e., the governing equations in Eq. 1 are all linear with respect to uu) which encapsulates a wide class of physics phenomena in the natural world. For nonlinear PDEs, iterative methods can be used for arriving at optimized wjw_j’s in Baldwinian-PINNs (detailed in Suppl. S.II). Hence, the Baldwinian-PINNs’ lifetime learning is broadly applicable to both linear and nonlinear PDEs, with the pseudoinverse formulation permitting extremely fast computation for physics-compliant prediction. It is worth emphasizing that each Baldwinian-PINN undergoes lifetime learning in order to produce accurate, physics-compliant predictions on a single, new physics task, hence, the fast nature of this procedure permits flexible and rapid prediction for each new task on-demand. In addition, there is no requirement on prior labelled data for any new task as the learning can be entirely physics-informed (as per conventional PINNs).

Implementation details

Baldwinian-PINN architecture

In the present study, Baldwinian-PINNs are designed to have weights and biases in the nonlinear hidden-layers w~\tilde{w} fixed to random values at birth. They are akin to a randomized neural networks setup [17, 34] and drawn from a probability distribution defined by a dimensionally reduced set of network hyperparameters θ\theta. Both normal and uniform distributions for the weights and biases are possible for the randomized PINNs [32, 33]. Similarly, smooth nonlinear activations such as sinsin, softplussoftplus, and tanhtanh are common in PINN literature and have their own merits. For greater flexibility, we apply both distributions for setting weights and biases and all three activations to different hidden layer blocks in the Baldwinian-PINN models.

Our base Baldwinian-PINN model is depicted in Figure 2(b). It’s hidden layer architecture is segmented into 3×2=63 \times2 = 6 unique blocks, with the weights and biases of the first 3 blocks sampled from normal distributions and the weights and biases of the other 3 blocks sampled from uniform distributions. Each block has a fixed number of neurons (nneuron=150n_{neuron} = 150 or 200). Assuming 2 input variables (x,t)(x,t), we can write the output fjf_j for all the neurons j=1,…,nneuronj = 1,\ldots,n_{neuron} in a nonlinear hidden-layer block as:

yj=w~j3b−2x+w~j3b−1t+w~j3b(9)y_j = \tilde{w}_j^{3b-2}x + \tilde{w}_j^{3b-1}t + \tilde{w}_j^{3b} \tag*{(9)}
fj(x,t;w~j)=φb(yj)(10)f_j(x,t;\tilde{w}_j) = \varphi^b(y_j) \tag*{(10)}

for each of the blocks b=1,…,6b = 1,\ldots,6, where the activation φb\varphi^b can be sinsin (b=1,4b = 1,4), softplussoftplus (b=2,5b = 2,5), or tanhtanh (b=3,6b = 3,6). The w~ji\tilde{w}_j^i’s are weights and biases with their own distributional mean mim^i and spread sis^i, i=1,…,18i = 1,\ldots,18. Their values can be obtained by the following sampling procedure:

w~ji∼N(0,1)orw~ji∼U(−1,1)(11)\tilde{w}_j^i \sim\mathcal{N}(0,1) \quad\text{or} \quad\tilde{w}_j^i \sim\mathcal{U}(-1,1) \tag*{(11)}
w~ji←w~ji×si+mi(12)\tilde{w}_j^i \leftarrow\tilde{w}_j^i \times s^i + m^i \tag*{(12)}

Given the Baldwinian-PINNs’ configuration, a set of network hyperparameters θ=(m1,s1,…,m18,s18)\theta= (m^1,s^1,\ldots,m^{18},s^{18}) control the distributional mean and spread of the weights and biases in different blocks. Since the neuroevolution only searches the distribution parameters for groups of weights instead of evolving each individual weight in the nonlinear hidden layer, the reduced dimensionality can be even more effectively searched by today’s evolutionary algorithms.

While the above description is for a single hidden layer, we note that the Baldwinian-PINN framework is not restricted to a single hidden layer. We include additional examples in Suppl. S.IV.B to show that the methodology also works for deeper neural architectures.

Evolutionary algorithm

In the majority of this study, we employ the covariance matrix adaptation evolution strategy (CMA-ES) [37] for evolving (θ,λ)(\theta,\lambda), although results in Suppl. S.IV.B show the extensibility of this framework to other neuroevolution algorithms. As an instantiation of information-geometric optimization algorithms [38], CMA-ES represents the population of (θ,λ)(\theta,\lambda) in P\mathcal{P} using a multivariate normal search distribution, initialized with zero mean and standard deviation (std.) as tuning hyperparameter. It iteratively adapts the search distribution based on the rank-based fitness landscape until the convergence criteria is met. Our experiments show that the performance of Baldwinian neuroevolution is robust across a range of CMA-ES hyperparameters such as population size and initial standard deviation (std.) of search distribution, and the number of tasks sampled for fitness evaluation per iteration. Hence, a robust setting that shows good convergence in fitness across different types of problems is chosen based on initial experiments.

The network weights and biases can take any value from (−∞,∞)(-\infty,\infty), hence there is no restriction to the continuous search space of θ\theta representing their distributional means and spreads. The learning hyperparameter λ≥0\lambda\geq0 can be evolved in continuous search space and its absolute value is then used for computing the least-squares solution. In our implementation with CMA-ES, (θ,λ)(\theta,\lambda) share the same initial standard deviation, and we scale the learning hyperparameter by a factor, i.e., λ→1e−4×abs(λ)\lambda\to1\mathrm{e}{-4}\times abs(\lambda), to improve the performance of the Tikhonov regularization.

In our preliminary experiments, we found it helpful to have the LMSEL_{MSE} component in the optimization objective even though the learning of future test tasks remains solely physics-informed. We set κLSE=κMSE=1\kappa_{LSE} = \kappa_{MSE} = 1 as default for the computation of overall fitness unless there is a huge difference in magnitude between LSEL_{SE} and LMSEL_{MSE} during Baldwinian neuroevolution. In addition, we can multiply the BC/IC rows in both A and b to re-balance the importance between PDE and BC/IC errors in the least-squares solution.

As this study focuses on the paradigm of Baldwinian neuroevolution as a pathway towards generalizable neural physics solvers, other combinations of evolutionary optimization or lifetime learning algorithms may be used. One key advantage of evolutionary optimization is that the fitness evaluations (population size npop×n_{pop} \times number of random tasks ntaskn_{task}) required each iteration can be easily parallelized across multiple GPUs to fully harvest any hardware advantage. In particular, we utilized the JAX framework to harness previously reported performance improvements for automatic differentiation and linear algebra operations [39, 40]. The experimental study is performed on a workstation with an Intel Xeon W-2275 Processor and 2 NVIDIA GeForce RTX 3090 GPUs.

Experimental Studies

We examine the efficacy of Baldwinian neuroevolution for learning physics (Algorithm 1 and Algorithm 2 in Section III-B) as formalized in a two-stage stochastic programming problem in Section III-A. Several ODE/PDE problems which are representative of real-world phenomena are used to demonstrate Baldwinian neuroevolution for physics in the following sections. Table I summarizes neuroevolution configurations and Baldwinian-PINN lifetime learning performance on their respective test tasks, including computational time for both neuroevolution and lifetime learning phases.

ProblemTask distributionNo. train tasksBatch size for task, ntaskn_{task}Population size, npopn_{pop}Max. iterationInitial std.Search dimension for (θ,λ)(\theta,\lambda)Time1^{1} (s)dim⁡(w)\dim(w)Sample size (npde,nic,nbc)(n_{pde}, n_{ic}, n_{bc})Nonlinear iterations, NNNo. test tasksTime2^{2} (s)MSE3^{3}
1Convection-diffusion x→ux \to u2010202001252509001001, -, 2-1100.0055.8e−9±9.9e−95.8\mathrm{e}{-9} \pm9.9\mathrm{e}{-9}
2Family of linear PDEs (x,t)→u(x,t) \to u10815205005371083012005151, 101, 50 (10201,101,100)4(10201, 101, 100)^{4}-870.1227.1e−4±3.1e−37.1\mathrm{e}{-4} \pm3.1\mathrm{e}{-3}
31D Poisson’s equation x→ux \to u6010201001251209001001, -, 2-600.0191.0e−9±5.8e−91.0\mathrm{e}{-9} \pm5.8\mathrm{e}{-9}
42D Poisson’s equation (x,y)→u(x,y) \to u10020201001372709001089, -, 128 (16641,−,512)5(16641, -, 512)^{5}-1000.1328.9e−12±1.0e−118.9\mathrm{e}{-12} \pm1.0\mathrm{e}{-11}
5Helmholtz equation (x,y)→u(x,y) \to u2010204005385209001024, -, 124 (16384,−,508)5(16384, -, 508)^{5}-600.1311.6e−5±5.1e−51.6\mathrm{e}{-5} \pm5.1\mathrm{e}{-5}
6Kinematics t→(x,y)t \to(x,y)15030204000.52516001800101, 1, -151000.0392.2e−8±1.3e−72.2\mathrm{e}{-8} \pm1.3\mathrm{e}{-7}
7Burgers’ equation (x,t)→u(x,t) \to u16520200137524090013107, 257, 100 (25957,257,200)5(25957, 257, 200)^{5}5 (10)6(10)^{6}321.962.3e−7±9.6e−72.3\mathrm{e}{-7} \pm9.6\mathrm{e}{-7}
8Nonlinear heat (x,t)→u(x,t) \to u135201005374409001600, 64, 50 (25600,256,200)5(25600, 256, 200)^{5}5640.871.3e−7±4.1e−71.3\mathrm{e}{-7} \pm4.1\mathrm{e}{-7}
9Allen-Cahn equation (x,y)→u(x,y) \to u168201000.5375109001024, -, 128 (16384,−,512)5(16384, -, 512)^{5}5 (10)6(10)^{6}321.192.0e−7±6.1e−72.0\mathrm{e}{-7} \pm6.1\mathrm{e}{-7}
10Diffusion-reaction (x,y)→u(x,y) \to u2210201001376409001024, -, 128 (16384,−,512)5(16384, -, 512)^{5}5640.581.2e−8±8.1e−81.2\mathrm{e}{-8} \pm8.1\mathrm{e}{-8}
116D diffusion-reaction (x,y)→u(x,y) \to u178201001375109001024, -, 128 (16384,−,512)5(16384, -, 512)^{5}51000.601.6e−8±2.5e−81.6\mathrm{e}{-8} \pm2.5\mathrm{e}{-8}
12Fluid flow (x,y)→(u,v,p)(x,y) \to(u,v,p)74204000.1661586811522401, -, 1965180.9032.1e−4±2.4e−42.1\mathrm{e}{-4} \pm2.4\mathrm{e}{-4}

Table I. Summary of Baldwinian neuroevolution and Baldwinian-PINN’s lifetime learning configurations, and generalization performance on PINN problems in experimental study.

Learning to solve and generalize linear ODE/PDEs

1) Convection-diffusion: The steady-state convection-diffusion equation is a ubiquitous physics model that describes the final distribution of a scalar quantity (e.g. mass, energy, or temperature) in the presence of convective transport and diffusion [41]. Solutions to this physics are key to characterization and design of many systems, including microfluidic chip cooling in electronics [42]. The 1D equation is defined as:

αdudx−d2udx2=0,x∈[0,1](13)\alpha\frac{du}{dx} - \frac{d^{2}u}{dx^{2}} = 0,\quad x \in[0,1] \tag*{(13)}

subject to BCs u(x=0)=0u(x = 0) = 0; u(x=1)=1u(x = 1) = 1. These real-world problems have characteristic physics that vary with non-dimensional constants such as the Peclet number PePe (ratio of convection to diffusion) [43]. Hence, it is helpful to learn a PINN model that can return u(x)u(x) for a diverse range of PePe-related problems (determined by α\alpha here). The training tasks consist of α={5,10,...,100}\alpha= \{5, 10, ..., 100\}, encompassing both smoother

output patterns at lower α\alpha and very high gradient patterns at higher α\alpha, with the latter being challenging for PINNs to learn by both stochastic gradient descent (SGD) [8, 12] and classical numerical methods [44].

The predictive performance of the learned model is evaluated for an unseen range of test tasks, i.e., α={1,2,...,110}\alpha= \{1, 2, ..., 110\}. The efficacy of Baldwinian neuroevolution is demonstrated in Figure 3, with the successful evolution of Baldwinian-PINNs which can learn an extremely accurate solution on new test tasks in milli-seconds. The learned solutions can achieve an average MSE of 5.8e−9±9.9e−95.8\mathrm{e}{-9}_{\pm9.9\mathrm{e}{-9}} (n=110n = 110 tasks ×\times 5 individual runs) after 200 neuroevolution iterations.

Figure 3. (a) Solution of 20 individual Baldwinian-PINN models sampled from the CMA-ES search distribution (initial std. = 1), for unseen convection-diffusion tasks α={4,27,108}\alpha= \{4, 27, 108\}. The task is more challenging with increasing α\alpha. Baldwinian neuroevolution is effective for evolving good Baldwinian-PINN models which can generalize across different difficulties. (b) Baldwinian neuroevolution demonstrates effective LSE and MSE convergences on convection-diffusion problem, for different CMA-ES initial std. values (best std. = 1). The bold lines indicate their median convergence path from 5 individual runs, and the shaded areas indicate their interquartile ranges.

2) Family of linear PDEs (convection, diffusion, and dispersion): Next, we extend Baldwinian-PINNs to a family of linear PDEs. This PDE family is a further generalization of the convection-diffusion equation:

dudt+αdudx−γd2udx2+δd3udx3=q(x,t),x∈[0,1],t∈(0,2](14)\frac{du}{dt} + \alpha\frac{du}{dx} - \gamma\frac{d^{2}u}{dx^{2}} + \delta\frac{d^{3}u}{dx^{3}} = q(x,t), \qquad x \in[0,1],\quad t \in(0,2] \tag*{(14)}
q(x,t)=∑j=1JAjsin⁡(ωjt+2πljxL+φj)(15)q(x,t) = \sum_{j=1}^{J} A_j \sin\left(\omega_j t + \frac{2\pi l_j x}{L} + \varphi_j\right) \tag*{(15)}

with IC u(x,0)=q(x,0)u(x,0) = q(x,0) and periodic BC u(0,t)=u(1,t)u(0,t) = u(1,t). eq:12 models the time evolution of a scalar quantity in the presence of physics phenomena such as convection (dudx\frac{du}{dx} component), diffusion (d2udx2\frac{d^{2}u}{dx^{2}} component), and dispersion (d3udx3\frac{d^{3}u}{dx^{3}} component), and a rich diversity of dynamical processes can be generated from different PDE and IC combinations [45, 46].

q(x,t)q(x,t) is a source term, with different q(x,t=0)q(x,t = 0) being the corresponding IC profiles. We consider the following PDE scenarios: α=1\alpha= 1, γ={0,5e−4,1e−3}\gamma= \{0, 5\mathrm{e}{-4}, 1\mathrm{e}{-3}\}, δ={0,5e−4,1e−3}\delta= \{0, 5\mathrm{e}{-4}, 1\mathrm{e}{-3}\}. The ratio between α\alpha, γ\gamma, and δ\delta determine non-dimensional constants (e.g. PePe), and consequently, the systems’ characteristic physics. q(x,t)q(x,t) comprises scenarios with J=5J = 5, L=6L = 6 and coefficients sampled uniformly from Aj∈[−0.8,0.8]A_j \in[-0.8, 0.8], ωj∈[−2,2]\omega_j \in[-2, 2], lj∈[0,1,2,3,4]l_j \in[0, 1, 2, 3, 4], φj∈[−π,π]\varphi_j \in[-\pi, \pi]. The training set comprises 108 tasks with different PDE and IC combinations.

The effective generalization of a Baldwinian-PINN to an entire PDE family with diverse output patterns is demonstrated on 2 task scenarios: S1 shows successful learning of u(x,t)u(x,t) for t∈[0,2]t \in[0, 2] on unseen set of PDEs, which include changes to PDE parameters (γ\gamma and δ\delta) and the source term / IC qq; while S2 shows effective extrapolation of solution to a longer time domain, i.e., t∈[0,4]t \in[0, 4]. For both scenarios, the Baldwinian-PINNs learn the solution accurately in milli-seconds. The

average MSE given by a successfully evolved Baldwinian-PINN over all test tasks (n=87n = 87) for S1 and S2 are 1.06e−5±1.60e−51.06\mathrm{e}{-5} \pm1.60\mathrm{e}{-5} and 1.72e−5±3.02e−51.72\mathrm{e}{-5} \pm3.02\mathrm{e}{-5}, respectively. Illustrative results are in Figure 4.

Figure 4. (a) Schematic to illustrate new tasks arising from family of linear PDEs problem: S1 change to new PDE and IC profile for t∈[0,2]t \in[0, 2] (same time domain as train tasks), and S2 projection to longer time domain t∈[0,4]t \in[0, 4]. (b) Solution for unseen linear PDE tasks obtained by best evolved Baldwinian-PINN sampled from the center of CMA-ES search distribution after 500 iterations with initial std. = 5 (they are visually indistinguishable from the ground truth). (c) The mean MSE over n=87n = 87 test tasks for 2 test scenarios described in (a) are below 5e−55\mathrm{e}{-5}. (d) Baldwinian neuroevolution demonstrates effective LSE and MSE convergences for different CMA-ES initial std. values, with a superior performance given by std. = 5 and 10. The bold lines indicate their median convergence path from 5 individual runs, and the shaded areas indicate their interquartile ranges.

Interestingly, the Baldwinian-PINNs maintain good accuracy on tasks from S2, whereby the evolved Baldwinian-PINN model first predicts u(x,t)u(x,t) for t∈[0,2]t \in[0, 2], before using the solution u(x,t=2)u(x,t = 2) as new IC for t∈[2,4]t \in[2, 4]. This is achievable because Baldwinian neuroevolution does not require parameterization for the tasks, and Baldwinian-PINNs can generalize to new ICs, BCs, and PDE source terms. This is tricky for existing meta-PINN methods as they require interpolation across a potentially infinitely large distribution of tasks (e.g. possible BCs or ICs), in contrast to the lifetime learning encapsulated in the Baldwinian paradigm. Additional results illustrating the ability of Baldwinian-PINNs to perform well despite variations in the lifetime learning task objectives (e.g., solving for different time durations) are in Suppl. S.III.A, further emphasizing the merits of Baldwinian neuroevolution for physics.

3) Additional linear ODE/PDE problems: The Baldwinian neuroevolution of physics is further demonstrated on 3 linear problems (Problems 3-5), namely 1D Poisson’s equation, 2D Poisson’s equation, and Helmholtz equation (see Suppl. S.IV). Suppl. Table S1 enumerates the accuracy advantages of Baldwinian-PINN relative to results reported by recent meta-learning PINN works [23, 25]. Suppl. S.IV also presents additional visualization results from Baldwinian-PINN, other instantiations of the proposed Baldwinian neuroevolution framework (such as natural evolution strategies (NES) variant [47]), and empirical study of Baldwinian-PINN performance when training-task labels are noisy or scarce, demonstrating the Baldwinian-PINNs’ versatility and generalizability.

ProblemMethod / ModelRemarks
NRPINNHyper-LR-PINNBaldwinian-PINN
MAEMSEMAEMSEMAEMSE
3 1D Poisson’s equation5.1e−45.1\mathrm{e}{-4}1---2.5e−72.5\mathrm{e}{-7}8.4e−148.4\mathrm{e}{-14}Test task: α1=1,α2=1,α3=0.1,α4=0,ω1=0.7,ω2=1.5\alpha_1 = 1, \alpha_2 = 1, \alpha_3 = 0.1, \alpha_4 = 0, \omega_1 = 0.7, \omega_2 = 1.5, x∈[−10,10]x \in[-10, 10]<br><br>NRPINN: 60 train tasks from α1=1,α2=1,α3=0.1,α4=0,ω1∈[0,1],ω2∈[0.2]\alpha_1 = 1, \alpha_2 = 1, \alpha_3 = 0.1, \alpha_4 = 0, \omega_1 \in[0, 1], \omega_2 \in[0.2] as per [5]<br><br>Baldwinian-PINN: 60 train tasks from α\alpha’s ∈[0,4],ω\in[0, 4], \omega’s ∈[0,4]\in[0, 4]
4 2D Poisson’s equation6.4e−76.4\mathrm{e}{-7}2---3.0e−73.0\mathrm{e}{-7}2.0e−132.0\mathrm{e}{-13}Test task: a1=0.15,a2=0.18,a3=0.20,a4=0.31,a5=0.43,a6=0.56,a7=0.70,a8=0.80,b1=0.34,b2=0.31,b3=0.65,b4=0.86,b5=0.65,b6=0.38,b7=0.64,b8=0.12,c1=0.84,c2=1.07,c3=1.12,c4=0.83,c5=1.12,c6=1.11,c7=0.99,c8=0.91,dj=0.01,x∈[0,1],y∈[0,1]a_1 = 0.15, a_2 = 0.18, a_3 = 0.20, a_4 = 0.31, a_5 = 0.43, a_6 = 0.56, a_7 = 0.70, a_8 = 0.80, b_1 = 0.34, b_2 = 0.31, b_3 = 0.65, b_4 = 0.86, b_5 = 0.65, b_6 = 0.38, b_7 = 0.64, b_8 = 0.12, c_1 = 0.84, c_2 = 1.07, c_3 = 1.12, c_4 = 0.83, c_5 = 1.12, c_6 = 1.11, c_7 = 0.99, c_8 = 0.91, d_j = 0.01, x \in[0, 1], y \in[0, 1]<br><br>NRPINN: 100 train tasks from J∈{1,5,10},aj∈[0.1,0.9],bj∈[0.1,0.9],cj∈[0.8,1.2],dj=0.01J \in\{1, 5, 10\}, a_j \in[0.1, 0.9], b_j \in[0.1, 0.9], c_j \in[0.8, 1.2], d_j = 0.01 as per [5]<br><br>Baldwinian-PINN: 100 train tasks from J∈[1,10],aj∈[−0.6,0.6],bj∈[−0.6,0.6],cj∈[0.5,2],dj∈[0.005,0.02]J \in[1, 10], a_j \in[-0.6, 0.6], b_j \in[-0.6, 0.6], c_j \in[0.5, 2], d_j \in[0.005, 0.02]
5 Helmholtz equation--2.8e−22.8\mathrm{e}{-2}3-1.5e−41.5\mathrm{e}{-4}4.7e−84.7\mathrm{e}{-8}Test task: α1=2.5,α2=2.5,x∈[−1,1],y∈[−1,1]\alpha_1 = 2.5, \alpha_2 = 2.5, x \in[-1, 1], y \in[-1, 1]<br><br>Hyper-LR-PINN: train tasks from α∈[2,3]\alpha\in[2, 3] with interval 0.1 (α=α1=α2\alpha= \alpha_1 = \alpha_2) as per [6]<br><br>Baldwinian-PINN: 20 train tasks from α1∈[0.1,6],α2∈[0.1,6]\alpha_1 \in[0.1, 6], \alpha_2 \in[0.1, 6]

Table S1. Table S1. Comparison with baseline meta-learning PINN models on single test task for linear ODE/PDE problems.

Learning to solve and generalize nonlinear ODE/PDEs

1) Kinematics: Extending beyond linear ODE/PDEs, Baldwinian-PINNs are applied to nonlinear kinematics equations. Assuming a ball is thrown at specific launch angle a0a_0 and initial velocity vel0vel_0, the following 2D kinematics equations describe the ball’s motion under the influence of gravity gg and air resistance RR:

d2xdt2+Rdxdt=0,t∈(0,T](16)\frac{d^2x}{dt^2} + R\frac{dx}{dt} = 0, \qquad t \in(0,T] \tag*{(16)}
d2ydt2+Rdydt=−g,t∈(0,T](17)\frac{d^2y}{dt^2} + R\frac{dy}{dt} = -g, \qquad t \in(0,T] \tag*{(17)}

subject to ICs x(t=0)=0x(t=0)=0, dxdt(t=0)=vel0×cos⁡(a0π180)\frac{dx}{dt}(t=0)=vel_0 \times\cos\left(\frac{a_0\pi}{180}\right); y(t=0)=0y(t=0)=0, dydt(t=0)=vel0×sin⁡(a0π180)\frac{dy}{dt}(t=0)=vel_0 \times\sin\left(\frac{a_0\pi}{180}\right). The air resistance R=12ρCdAmVR=\frac{1}{2}\frac{\rho C_d A}{m}V is related to air density ρ\rho, object properties (drag coefficient CdC_d, cross-sectional area AA, and mass mm), and object velocity V=(dx/dt)2+(dy/dt)2V=\sqrt{\left(dx/dt\right)^2+\left(dy/dt\right)^2}, hence the equations are nonlinear with respect to xx and yy.

The 150 training tasks comprise different launch angles a0∈[15,85]a_0 \in[15,85], initial velocity vel0∈[10,110]vel_0 \in[10,110], and object properties Cd∈[0.2,0.7]C_d \in[0.2,0.7], A∈[0.00145,0.045]A \in[0.00145,0.045], m∈[0.046,0.6]m \in[0.046,0.6] as may be representative of different projectiles (e.g., baseball, basketball). gg and ρ\rho are assumed to be 9.8 and 1.3 respectively.

The Baldwinian-PINN learns the horizontal and vertical position x(t)x(t) and y(t)y(t) of the object via iterative least-squares computation (see Section III-C) with a fixed number of nonlinear iterations N=15N=15. 100 test tasks encompassing different a0∈[5,90]a_0 \in[5,90], vel0∈[8,98]vel_0 \in[8,98], and CdC_d are constructed to assess generalizability. Results presented in Figure 5a-Figure 5b show that Baldwinian-PINNs learn very accurate solutions (MSE =2.2e−8±1.3e−7=2.2\mathrm{e}{-8}\pm1.3\mathrm{e}{-7}) in milli-seconds.

Baldwinian-PINN lifetime learning and convergence for unseen kinematics and fluid flow tasks

Figure 5. (a) Solution of 20 individual Baldwinian-PINN models sampled from the CMA-ES search distribution (initial std. = 0.5), for unseen kinematics tasks. (c) Solution of Baldwinian-PINN model sampled from the center of NES search distribution after 400 iterations (initial std. = 0.5), for unseen fluid flow (N-S equations) tasks. The velocity magnitude predictions are visually indistinguishable from the ground truth, even as Reynolds number (Re) increases. Their mid-section uu and vv-velocity profiles (dashed lines) show excellent agreement with the classic benchmark results (marked points) from Ghia [48]. Baldwinian neuroevolution demonstrates effective LSE and MSE convergence on both (b) kinematics and (d) fluid flow problem, for different CMA-ES initial std. values. The bold lines indicate their median convergence path from 5 individual runs, and the shaded areas indicate their interquartile ranges.

2) Set of nonlinear PDEs: The Baldwinian neuroevolution of physics is further demonstrated on 5 nonlinear PDE problems (Problems 7-11) such as the Burger’s, nonlinear Allen-Cahn and nonlinear reaction-diffusion equations, as per recent meta-learning PINN study [22] and described in Suppl. S.V. That study compares several meta-learning PINNs based on weight interpolation methods and MAML.

A key difference between this study and meta-learning of PINNs in [22] is the availability of data for training tasks. On the test tasks, the evolved Baldwinian-PINNs can achieve ∼1\sim1 order of magnitude lower relative error compared to [22]. Crucially, the computation time for Baldwinian-PINN for new predictions is at most two seconds whereas existing meta-learning approaches may take more than 500 seconds (2 orders of magnitude acceleration). For example, when solving the diffusion-reaction equation, a 70x improvement in accuracy was achieved relative to other state-of-the-art approaches, while taking 700x less computational time. Complete quantitative results are summarized in Suppl. Table S2.

Problem (no. test task)Random Error¹Random Time²MAML Error¹MAML Time²Center Error¹Center Time²Multitask Error¹Multitask Time²LMC Error¹LMC Time²RBF (multiquadric) Error¹RBF (multiquadric) Time²Polynomial Error¹Polynomial Time²Baldwinian-PINN Error³Baldwinian-PINN Time⁴
7 Burgers’ equation (n=32n = 32)1.2e−3±1.8e−31.2\mathrm{e}{-3} \pm1.8\mathrm{e}{-3}105±29105 \pm291.7e−3±3.2e−31.7\mathrm{e}{-3} \pm3.2\mathrm{e}{-3}125±38125 \pm389e−4±5e−49\mathrm{e}{-4} \pm5\mathrm{e}{-4}56±4856 \pm487e−4±2e−47\mathrm{e}{-4} \pm2\mathrm{e}{-4}24±2124 \pm218e−4±4e−48\mathrm{e}{-4} \pm4\mathrm{e}{-4}40±2740 \pm278e−4±4e−48\mathrm{e}{-4} \pm4\mathrm{e}{-4}7±87 \pm87e−4±3e−47\mathrm{e}{-4} \pm3\mathrm{e}{-4}7±167 \pm163.8e−4±6.9e−4\mathbf{3.8\mathrm{e}{-4}} \pm6.9\mathrm{e}{-4}1.96±0.01\mathbf{1.96} \pm0.01
8 nonlinear heat (n=64n = 64)5.2e−3±3.9e−35.2\mathrm{e}{-3} \pm3.9\mathrm{e}{-3}156±42156 \pm424.5e−3±3.1e−34.5\mathrm{e}{-3} \pm3.1\mathrm{e}{-3}188±43188 \pm434.5e−3±2.4e−34.5\mathrm{e}{-3} \pm2.4\mathrm{e}{-3}96±4196 \pm414.8e−3±3.6e−34.8\mathrm{e}{-3} \pm3.6\mathrm{e}{-3}49±3049 \pm304.9e−3±3.3e−34.9\mathrm{e}{-3} \pm3.3\mathrm{e}{-3}59±2859 \pm284.4e−3±2.2e−34.4\mathrm{e}{-3} \pm2.2\mathrm{e}{-3}35±3035 \pm304.6e−3±3.2e−34.6\mathrm{e}{-3} \pm3.2\mathrm{e}{-3}38±3038 \pm306.0e−4±7.6e−4\mathbf{6.0\mathrm{e}{-4}} \pm7.6\mathrm{e}{-4}0.87±0.02\mathbf{0.87} \pm0.02
9 Allen-Cahn equation (n=32n = 32)1.5e−2±1.2e−21.5\mathrm{e}{-2} \pm1.2\mathrm{e}{-2}496±161496 \pm161——1.2e−2±4.6e−31.2\mathrm{e}{-2} \pm4.6\mathrm{e}{-3}201±83201 \pm831.2e−2±4.7e−31.2\mathrm{e}{-2} \pm4.7\mathrm{e}{-3}120±44120 \pm441.1e−2±4.0e−31.1\mathrm{e}{-2} \pm4.0\mathrm{e}{-3}120±21120 \pm211.2e−2±5.3e−31.2\mathrm{e}{-2} \pm5.3\mathrm{e}{-3}68±4668 \pm461.2e−2±4.2e−31.2\mathrm{e}{-2} \pm4.2\mathrm{e}{-3}44±1944 \pm199.5e−4±1.5e−3\mathbf{9.5\mathrm{e}{-4}} \pm1.5\mathrm{e}{-3}1.19±0.01\mathbf{1.19} \pm0.01
10 Diffusion-reaction (n=64n = 64)1.1e−2±6.2e−31.1\mathrm{e}{-2} \pm6.2\mathrm{e}{-3}1073±2061073 \pm206——9.5e−3±5.9e−39.5\mathrm{e}{-3} \pm5.9\mathrm{e}{-3}426±159426 \pm1599.0e−3±5.4e−39.0\mathrm{e}{-3} \pm5.4\mathrm{e}{-3}243±93243 \pm939.1e−3±5.8e−39.1\mathrm{e}{-3} \pm5.8\mathrm{e}{-3}302±127302 \pm1278.1e−3±4.6e−38.1\mathrm{e}{-3} \pm4.6\mathrm{e}{-3}280±161280 \pm1618.5e−3±5.1e−38.5\mathrm{e}{-3} \pm5.1\mathrm{e}{-3}249±129249 \pm1291.3e−4±2.0e−4\mathbf{1.3\mathrm{e}{-4}} \pm2.0\mathrm{e}{-4}0.58±0.02\mathbf{0.58} \pm0.02
11 6D diffusion-reaction (n=100n = 100)2.2e−3±1.3e−32.2\mathrm{e}{-3} \pm1.3\mathrm{e}{-3}612±255612 \pm255——1.8e−3±1.7e−31.8\mathrm{e}{-3} \pm1.7\mathrm{e}{-3}494±222494 \pm2221.7e−3±1.9e−31.7\mathrm{e}{-3} \pm1.9\mathrm{e}{-3}431±200431 \pm2001.5e−3±1.8e−31.5\mathrm{e}{-3} \pm1.8\mathrm{e}{-3}428±201428 \pm2011.7e−3±2.0e−31.7\mathrm{e}{-3} \pm2.0\mathrm{e}{-3}375±168375 \pm1681.8e−3±2.0e−31.8\mathrm{e}{-3} \pm2.0\mathrm{e}{-3}496±213496 \pm2131.9e−4±1.5e−4\mathbf{1.9\mathrm{e}{-4}} \pm1.5\mathrm{e}{-4}0.60±0.03\mathbf{0.60} \pm0.03

Table S2. Comparison with baseline meta-learning PINN models on nonlinear benchmark problems.

The versatility of Baldwinian neuroevolution is also key here as we can accelerate the Baldwinian-PINN’s lifetime learning by using a much coarser discretization and smaller number of nonlinear iterations during training while still learning a good solution on test tasks with finer (e.g., 16×) discretization and more (e.g., 2×) nonlinear iterations.

3) Navier-Stokes (N-S) equations for fluid flow: We explore a more challenging physics scenario in computational fluid dynamics, the lid-driven cavity flow problem. The fluid flow inside a 2D unit square, x∈[0,1]x \in[0,1], y∈[0,1]y \in[0,1], is driven by the top lid velocity (ulid=1u_{\mathrm{lid}}=1), and governed by the steady-state incompressible N-S equations for dependent variables velocity uu, vv and pressure pp:

∇⋅u⃗=0(18)\nabla\cdot\vec{u}=0 \tag*{(18)}
(u⃗⋅∇)u=1Re∇2u−∇p(19)(\vec{u}\cdot\nabla)u=\frac{1}{Re}\nabla^2u-\nabla p \tag*{(19)}

Complex flow structures emerge at moderate to high Reynolds numbers, making this a challenging benchmark for PINNs, which may require hours of training with additional techniques such as transfer or curriculum learning to achieve convergence.

The Baldwinian neuroevolution of physics is performed on 7 training tasks Re∈[1,800]Re \in[1,800]. After 400 neuroevolution iterations, the evolved Baldwinian-PINNs can generalize to new solutions Re∈[5,1000]Re \in[5,1000] with an average MSE of 2.1e−4±2.4e−42.1\mathrm{e}{-4}\pm2.4\mathrm{e}{-4} (n=18n=18 tasks × 5 individual runs). Results presented in Figure 5c-Figure 5d and Suppl. S.VI Figure S9 demonstrate excellent agreement between Baldwinian-PINN predictions and the numerical benchmark from Ghia et al. [48]. Baldwinian-PINNs learn accurate solutions to the N-S equations in under one second, highlighting the potential to address difficult physics problems characterized by strong nonlinearity and strong coupling among multiple variables.

Baldwinian-PINN prediction versus ground truth and absolute error maps for unseen fluid flow tasks

Figure S9. Baldwinian-PINN prediction vs. ground truth and their absolute error maps for unseen fluid flow (N-S equations) tasks on Re∈[5,1000]Re \in[5, 1000]. The MSE of the solutions obtained by the best evolved Baldwinian-PINN (sampled from the center of NES search distribution after 400 iterations with initial std. = 0.5) compared to numerical simulation ground truths are 4.4e−54.4\mathrm{e}{-5} (Re=5Re=5), 3.4e−53.4\mathrm{e}{-5} (Re=25Re=25), 3.5e−53.5\mathrm{e}{-5} (Re=100Re=100), 5.1e−55.1\mathrm{e}{-5} (Re=400Re=400), and 1.8e−41.8\mathrm{e}{-4} (Re=1000Re=1000), respectively.

Analysis on effectiveness of Baldwinian neuroevolution

We further investigate the advantages of Baldwinian neuroevolution via an ablation study using the convection-diffusion and kinematics examples.

Briefly, we explore deep and shallow MLP architectures for baseline SGD-trained DNN and PINN models for comparison with the Baldwinian-PINNs: 1. the deep architecture consists of similar total number of network weights as the corresponding Baldwinian-PINN models, but distributed across multiple nonlinear hidden layers with smaller number of nodes; 2. the shallow architecture has single nonlinear hidden layer and same number of nodes as the corresponding Baldwinian-PINN model but this necessitates more total network weights because of the additional input variables.

Each architecture also consists of 2 variants: a. tanh activation for the nonlinear hidden layers whereby the network weights are initialized by Xavier method; b. sin activation for the nonlinear hidden layers whereby the network weights are initialized by He method.

This notation is maintained when referencing the respective model performance in Figure 6. For example, the models labelled as DNN-1a and PINN-2b refer to the corresponding baseline DNN model with a deep architecture and tanh activation layer and baseline PINN model with a shallow architecture (single hidden layer) and sin activation layer respectively. Additional model descriptions are in Suppl. S.VII.

Error distributions of DNN, PINN, and Baldwinian-PINN models for convection-diffusion and kinematics tasks

Figure 6. The generalization performance of the DNN, PINN, and Baldwinian-PINN models are compared, across (a) 110 convection-diffusion and (b) 100 kinematics test tasks. Model-1a/2a: deep/shallow architecture with tanh activations (network weights initialized by Xavier method); Model-1b/2b: deep/shallow architecture with sin activations (weights initialized by He method). The MSE results from each of the DNN and PINN models are pooled from 3 initial learning rate configurations × 5 individual runs. The MSE results from Baldwinian-PINN model are pooled from 5 initial std. values × 5 individual runs.

1) Direct prediction across tasks with parametric DNNs / PINNs: As a baseline, DNN and PINN models are trained with SGD (ADAM) based on data-driven loss and PINN (data and physics) loss respectively, and applied to predictions for unseen tasks based on interpolation. We explore different MLP configurations as per Suppl. Table S3 in order to more fairly compare the best performance across different models.

ProblemModelArchitectureActivationInitializationNo. weights (final layer)Batch size for taskMax. iterationLearning rate1
11aDeeptanhXavier2750 (50)1050,0005e-4 / 5e-3 / 5e-2
11bDeepsinHe2750 (50)1050,0005e-4 / 5e-3 / 5e-2
12aShallowtanhXavier3600 (900)1050,0005e-4 / 5e-3 / 5e-2
12bShallowsinHe3600 (900)1050,0005e-4 / 5e-3 / 5e-2
61aDeeptanhXavier3930 (60)30150,0005e-4 / 5e-3 / 5e-2
61bDeepsinHe3930 (60)30150,0005e-4 / 5e-3 / 5e-2
62aShallowtanhXavier6300 (1800)30150,0005e-4 / 5e-3 / 5e-2
62bShallowsinHe6300 (1800)30150,0005e-4 / 5e-3 / 5e-2

Table S3. DNN / PINN MODEL AND TRAINING CONFIGURATIONS USED IN ABLATION STUDY.

Critically, when making a direct prediction for a new task, the respective baseline DNN and PINN models do not explicitly incorporate (and guarantee compliance with) the known physics prior. Although the parametric PINN can learn a more physically-consistent prediction from the physics loss during training, compliance is not guaranteed for a new task.

In addition, the new tasks must follow an a priori determined input parameterization for interpolation. For example, the DNN and PINN models for convection-diffusion need to be a priori set-up with (x,α)(x, \alpha) as inputs to enable predictions across different α\alpha’s. In contrast, the Baldwinian-PINN is parameterization-agnostic, and does not require any input parameterization for predictions of new tasks.

The direct prediction performance of DNNs and PINNs with different model configurations are summarized in Figure 6 and Suppl. Table S4. These baseline DNN and PINN predictions have high generalization errors (several orders of magnitude higher than Baldwinian-PINN). As explained above, PINNs do not necessarily outperform DNNs for direct prediction on test tasks (e.g., MSE for convection-diffusion for different model configurations range from 1.7e−41.7\mathrm{e}{-4} to 1.2e−21.2\mathrm{e}{-2} for DNN, and from 1.9e−31.9\mathrm{e}{-3} to 3.4e−13.4\mathrm{e}{-1} for PINN), suggesting that PINN training by itself does not always guarantee generalization in the absence of additional physics-informed fine-tuning.

2) Prediction across tasks with parametric DNNs / PINNs and physics-informed fine-tuning: In order to assess the utility of a physics-informed fine-tuning step, we optimize the final layer weights of the SGD-trained baseline DNNs and PINNs with the Tikhonov regularization at test time. This fine-tuning procedure is almost identical to the Baldwinian-PINN’s lifetime learning.

The key difference is that the pre-final nonlinear hidden layers weight distribution and learning hyperparameter λ\lambda in Baldwinian-PINNs are jointly learned by evolution to share among tasks. In contrast, nonlinear hidden layers’ weights for the DNNs and PINNs are SGD-learned from train tasks. As the Tikhonov regularization solution is highly dependent on λ\lambda, we need to perform a grid search across λ={1e−2,1e−4,1e−6,1e−8,1e−10,1e−12,0}\lambda= \{1\mathrm{e}{-2}, 1\mathrm{e}{-4}, 1\mathrm{e}{-6}, 1\mathrm{e}{-8}, 1\mathrm{e}{-10}, 1\mathrm{e}{-12}, 0\} to determine the best λ\lambda during test time and present the solution associated with the lowest LSE. The results after Tikhonov regularization are compared in Figure 6 and Suppl. Table S4.

Results show that physics-informed fine-tuning via Tikhonov regularization can improve the accuracy of both data-trained DNN and physics-trained PINN models at test time. However, their results are not as generalizable (i.e., the improvements are isolated to certain model and task instances) as the Baldwinian-PINN (which has been optimized for generating accurate solution over task distribution).

For example, we observe a significant improvement for the nonlinear kinematics problem (i.e., the best MSE for DNN and PINN improves from 3.0 to 4.4e−54.4\mathrm{e}{-5} and 1.3 to 7.9e−67.9\mathrm{e}{-6} respectively after Tikhonov regularization), potentially because the training distribution is sparser relative to the larger variation in output patterns (see Suppl. S.VIII).

In contrast, fine-tuning via Tikhonov regularization doesn’t produce better results than direct prediction for the convection-diffusion problem. The MSE with and without Tikhonov regularization are 2.3e−42.3\mathrm{e}{-4} and 2.8e−22.8\mathrm{e}{-2}, and 1.9e−31.9\mathrm{e}{-3} and 7.2e−37.2\mathrm{e}{-3} for the best-configured DNN and PINN respectively. Tikhonov regularization cannot jointly minimize both PDE and IC/BC errors to sufficiently small levels, leading to inferior performance than direct prediction for some tasks.

While the best DNN model (after Tikhonov regularization) can be more accurate than the corresponding Baldwinian-PINN on some convection-diffusion tasks at lower α\alpha, solution quality deteriorates quickly at larger α\alpha. Interestingly, PINNs with the same configuration (after Tikhonov regularization) show the opposite trend, highlighting the challenge of obtaining a generalizable model for entire task distributions through SGD, in contrast to Baldwinian neuroevolution.

Overall, these results suggest that an additional step of physics-informed learning (i.e. Tikhonov regularization) on any new task can be beneficial. However, SGD-trained PINN and DNN models learn pre-final nonlinear hidden layers that are highly variable in their suitability for the physics-informed Tikhonov regularization-based fine-tuning across new tasks, in contrast to the proposed Baldwinian-PINNs. Hence, these results highlight the advantages of Baldwinian-PINN framework as a complete solution to the challenges encountered in generalizing with physics-informed learning.

Conclusion

In this paper, we study the Baldwin effect as a novel means to advancing the generalizability of PINNs over a family of governing differential equations. The Baldwin effect is instantiated through a two-stage stochastic programming formulation, wherein the first stage evolves the initial layers of a generalizable PINN model, and the second stage trains its final layer to specialize to any new physics task (analogous to lifetime learning). The Baldwinian neuroevolution biases pre-final hidden layers’ weight distribution so that final layer specialization is both fast and accurate, preserving (near) constant-time solves while allowing capacity to scale. Although the pre-final hidden layers are not fine-tuned per task, the Baldwinian-PINN’s capacity can be effectively scaled—by increasing width and/or depth—to handle more complex physics (e.g., fluid flow) while preserving fast test-time learning.

Our method is demonstrated to be broadly applicable to the learning of different linear and nonlinear ODE/PDEs encompassing diverse physical phenomena such as convection-diffusion, particle kinematics, heat and mass transfer, and fluid flow. Relative to recent meta-learned PINNs, Baldwinian-PINNs can accelerate the physics-aware predictions by several orders of magnitude, while improving the prediction accuracy by up to one order. A Baldwinian-PINN is thus in the image of a precocial species with accelerated learning ability at birth.

The lifetime learning encapsulated in the Baldwinian paradigm does not require a priori parameterization for the task scenarios, permitting both flexible generalization to new ICs, BCs, and PDE source terms and variations in the lifetime learning task objectives (e.g., different domain and/or sample size). This allows Baldwinian-PINNs to be useful in applications when large number of evaluations with a priori unknown input conditions are sought, e.g. generative design or what-if analysis. In addition, results in Section IV-A2 suggest that Baldwinian-PINNs could be suited to the continual modelling of dynamical systems through their versatility in handling ICs and ability to rapidly model and stitch time windows together with minimal error [49].

In the context of recent interest in foundation models for scientific machine learning [50], specifically through the use of neural operators or neural PDE solvers, our experiments showing accurate and fast generalization across families of linear and nonlinear ODE/PDEs suggest that Baldwinian learning can be an alternate route to such flexible and generalizable machine intelligence models. It will be interesting to test the limits to which Baldwinian-PINNs can learn across broad classes of physics phenomena and/or differential operators in future work. Hidden-layer fine-tuning adds flexibility but forfeits the closed-form update and re-introduces slow, nonconvex optimization. For large task shifts (out-of-distribution), a light hidden-layer adaptation (e.g., a few gradient-descent steps on a low-rank adapter) are a potential practical extension that trades some speed for additional flexibility.

Lastly, while the experiments in this work focus on optimizing the center and spread of the probability distributions that sample the initial weights in the neural network layers, this can be easily extended to incorporate other state-of-the-art neural architecture search approaches, which directly optimize the graph structure of the node connections. Our experiments indicate that significant improvement relative to other recent meta-learning PINN works can already be observed even when we only optimize the center and spread of the weight sampling distributions under the Baldwinian-PINN framework. The proposed Baldwinian-PINN framework can be seamlessly extended in future work, e.g. via integration with other state-of-the-art neural architecture search algorithms.

Acknowledgment

This research was in part supported by the National Research Foundation, Singapore through the AI Singapore Programme, under the project “AI-based urban cooling technology development” (Award No. AISG3-TC-2024-014-SGKR). Abhishek Gupta was supported by the Ramanujan Fellowship from the Anusandhan National Research Foundation, Government of India (Grant No. RJF/2022/000115).

Supplementary Material for Evolutionary Optimization of Physics-Informed Neural Networks: Advancing Generalizability by the Baldwin Effect

Jian Cheng Wong; Chin Chun Ooi; Abhishek Gupta; Pao-Hsiung Chiu; Joshua Shao Zheng Low; My Ha Dao; Yew-Soon Ong

Data Generation

In this study, the majority of the problems have corresponding analytical solutions as described in the problem setups. In addition, the ground truth for the PDE family (Problem 2) and Burgers’ equation (Problem 7) is obtained by a high-resolution finite volume scheme. To alleviate convection instability, the dispersion-relation-preserving (DRP) finite volume scheme with a universal limiter has been utilized [1 S2]. Other spatial derivative terms are discretized by central difference. For the temporal term, the second-order TVD Runge Kutta scheme [3] is employed. For the incompressible Navier-Stokes (N-S) equations (Problem 12), the dispersion-relation-preserving (DRP) finite volume scheme, together with improved divergence-free-condition compensated framework [1], is employed to produce high-fidelity solutions as well as ensure the velocity-pressure coupling.

For the PDE family equation, the spatial resolution, Δx\Delta x, is 1/400, while temporal resolution, Δt\Delta t, is 5e−5Δx5\mathrm{e}{-5}\Delta x; For the Burgers’ equation, the spatial resolution, Δx\Delta x, is 1/512, while temporal resolution Δt\Delta t is 1e−2Δx1\mathrm{e}{-2}\Delta x; for the incompressible N-S equations, the spatial resolution, Δx\Delta x and Δy\Delta y, are 1/200.

Lagging of Coefficient Method for Nonlinear PDEs

For nonlinear PDEs, iterative methods can be used for arriving at optimized Baldwinian-PINNs. In this work, we use a lagging of coefficient approach which is common in numerical methods [4]. Briefly, we approximately linearize the nonlinear term(s) in Eq. 1 by substituting the output u(x,t)=∑jwjfj(x,t;w~)u(x,t)=\sum_j w_jf_j(x,t;\widetilde{w}) obtained from previous step, and

Jian Cheng Wong, Chin Chun Ooi, and Pao-Hsiung Chiu are with the Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore (e-mail: [email protected]; [email protected]; [email protected]).

Abhishek Gupta is with the School of Mechanical Sciences, Indian Institute of Technology Goa, India (e-mail: [email protected]).

Joshua Shao Zheng Low is with the College of Computing and Data Science, Nanyang Technological University, Singapore (e-mail: [email protected]).

My Ha Dao is with the Technology Centre for Offshore and Marine, Singapore (e-mail: [email protected]).

Yew-Soon Ong is with the Agency for Science, Technology and Research, Singapore, and is also with the College of Computing and Data Science, Nanyang Technological University, Singapore (e-mail: [email protected]).

iteratively solve Eq. (8) to update wjw_j’s for a fixed number of steps, NN, or until a convergence criterion is reached.

The nonlinear equation (∂2u∂x2+∂2u∂y2)+u(1−u2)=f\left(\frac{\partial^2u}{\partial x^2}+\frac{\partial^2u}{\partial y^2}\right)+u(1-u^2)=f is used to demonstrate the lagging of coefficient method for computing the wjw_j’s in Baldwinian-PINNs. We approximately linearize the nonlinear term u(1−u2)u(1-u^2) as u(1−u~2)u(1-\widetilde{u}^2) with u~=0\widetilde{u}=0 being the initial guess solution at first iteration. The (ii-th PDE sample, jj-th neuron) entry of the least squares matrix AA in Eq. (7) (§ 3-C) now becomes:

Nθ[fj(xipde,yipde;w~j)]=d2fj(xipde,yipde;w~j)dx2+d2fj(xipde,yipde;w~j)dy2+fj(xipde,yipde;w~j)(1−u~2)(20)\mathcal{N}_{\theta}[f_j(x_i^{pde},y_i^{pde};\widetilde{w}_j)] = \frac{d^2f_j(x_i^{pde},y_i^{pde};\widetilde{w}_j)}{dx^2} + \frac{d^2f_j(x_i^{pde},y_i^{pde};\widetilde{w}_j)}{dy^2} + f_j(x_i^{pde},y_i^{pde};\widetilde{w}_j)(1-\widetilde{u}^2) \tag*{(20)}

Hence, the Tikhonov regularization solution ww can be obtained from the linearized version of the equation. The solution obtained from past iteration is used to compute u~(x,y)=∑jwjfj(x,y;w~j)\widetilde{u}(x,y)=\sum_jw_jf_j(x,y;\widetilde{w}_j) for the next iteration, until the solution ww reaches a specified convergence criterion or reaches a pre-determined number of iterations, NN.

Additional Results for Family of Linear PDE Problem

Prediction on new time interval

The experimental results in Section IV-A2 show that Baldwinian-PINNs trained on a set of linear PDE tasks for t∈[0,2]t\in[0,2], are capable of learning time-dependent solution u(x,t)u(x,t) on a set of test tasks for unseen PDEs and ICs, as well as for a longer (2×)(2\times) time domain by performing the learning twice, i.e., for t∈[0,2]t\in[0,2] and then using the learned solution u(x,t=2)u(x,t=2) as new IC for t∈[2,4]t\in[2,4]. Recall that the average MSE given by the best evolved Baldwinian-PINN model (sampled from the center of CMA-ES search distribution from the best run) over all test tasks for t∈[0,2]t\in[0,2] and t∈[0,4]t\in[0,4] are 1.06e−5±1.60e−51.06\mathrm{e}{-5}\pm1.60\mathrm{e}{-5} and 1.72e−5±2.30e−51.72\mathrm{e}{-5}\pm2.30\mathrm{e}{-5}, respectively.

We further demonstrate the versatility of Baldwinian-PINNs by changing the time domain of interest in the test tasks to t∈[0,3]t\in[0,3]. From the best evolved Baldwinian-PINN model, we can obtain the prediction on new tasks in the following ways: P0 the solution for the original time domain t∈[0,2]t\in[0,2] via physics-based lifetime learning, and the extended time t∈[2,3]t \in[2, 3] [2, 3] based on neural network interpolation; P1 the solution for original and extended time domain t∈[0,3]t \in[0, 3] altogether via physics-based lifetime learning; P2 physics-based learning of the solution for the first 2s time t∈[0,2]t \in[0, 2], before using the learned solution u(x,t=2)u(x, t = 2) as new IC for next 2s time t∈[2,4]t \in[2, 4]; P3 physics-based learning of the solution for the first 2s time t∈[0,2]t \in[0, 2], before using the learned solution u(x,t=2)u(x, t = 2) as new IC for next time domain t∈[2,3]t \in[2, 3].

Figure S1 shows the Baldwinian-PINN’s solution obtained using different ways P0-P3 on selected test tasks. Their MSE results over all test tasks for t∈[0,3]t \in[0, 3] are 5.35e−2±6.08e−25.35\mathrm{e}{-2} \mathbin{\pm} 6.08\mathrm{e}{-2} (P0), 4.24e−5±1.04e−54.24\mathrm{e}{-5} \mathbin{\pm} 1.04\mathrm{e}{-5} (P1), 1.47e−5±2.32e−51.47\mathrm{e}{-5} \mathbin{\pm} 2.32\mathrm{e}{-5} (P2), and 1.43e−5±2.22e−51.43\mathrm{e}{-5} \mathbin{\pm} 2.22\mathrm{e}{-5} (P3), respectively. The performance of P0 is significantly worse than P1-P4, as expected, because of the physics-agnostic extrapolation. The Baldwinian-PINNs can flexibly learn the solutions on extended time domain in a single Tikhonov regularization solve (P1), although the most accurate solutions are given by P2 and P3.

Figure S1. Solution for unseen family of linear PDE tasks on t∈[0,3]t \in[0, 3] obtained by best evolved Baldwinian-PINN (sampled from the center of CMA-ES search distribution after 500 iterations with initial std. = 5) using different ways P0-P3.

Performance variation across runs

The Baldwinian neuroevolution outcome has the most variation in accuracy across individual runs, on the family of linear PDEs problem relative to the other problems in the experimental studies. The Baldwinian-PINN MSE results over all test tasks for t∈[0,4]t \in[0, 4] obtained by 5 different runs are 5.3e−5±1.8e−45.3\mathrm{e}{-5} \mathbin{\pm} 1.8\mathrm{e}{-4}, 1.7e−5±3.0e−51.7\mathrm{e}{-5} \mathbin{\pm} 3.0\mathrm{e}{-5}, 2.0e−3±5.6e−32.0\mathrm{e}{-3} \mathbin{\pm} 5.6\mathrm{e}{-3}, 1.5e−3±3.5e−31.5\mathrm{e}{-3} \mathbin{\pm} 3.5\mathrm{e}{-3}, and 2.8e−5±9.1e−52.8\mathrm{e}{-5} \mathbin{\pm} 9.1\mathrm{e}{-5}, respectively. Figure S2 and Figure S3 compare the solutions of 2 Baldwinian-PINNs obtained from separate Baldwinian neuroevolution runs, on the same 5 test tasks selected at different levels of accuracy along the MSE spectrum (pooled from n=87n = 87 test tasks ×\times 5 individual runs with initial std. = 5). In all our other experiments, the overall variation across runs remains fairly small.

Baldwinian-PINN solutions and ground truths for Run-2

Figure S2. Baldwinian-PINN’s solution (run no. 2) vs. ground truth on 5 selected family of linear PDE tasks, and the position of their accuracy along the MSE spectrum (pooled from n=87n = 87 test tasks ×\times 5 individual runs with initial std. = 5).

Baldwinian-PINN solutions and ground truths for Run-4

Figure S3. Baldwinian-PINN’s solution (run no. 4) vs. ground truth on 5 selected family of linear PDE tasks, and the position of their accuracy along the MSE spectrum (pooled from n=87n = 87 test tasks ×\times 5 individual runs with initial std. = 5).

Studies on Set of Linear ODE/PDE Problems

The additional linear ODE/PDE problems (Problems 3-5) are described below.

  1. 1D Poisson’s equation: The 1D Poisson’s equation consists of 60 randomly sampled train tasks α\alpha’s ∈[0,4]\in[0,4], ω\omega’s ∈[0,4]\in[0,4] and 60 randomly sampled test tasks α\alpha’s ∈[−5,5]\in[-5,5], ω\omega’s ∈[−5,5]\in[-5,5] for the PDE/BC parameters (α1,α2,α3,α4,ω1,ω2)(\alpha_1,\alpha_2,\alpha_3,\alpha_4,\omega_1,\omega_2):

d2udx2=q,x∈[−10,10](21)\frac{d^2u}{dx^2}=q,\quad x\in[-10,10] \tag*{(21)}

where the exact solution u(x;α1,α2,α3,α4,ω1,ω2)=α1sin⁡(ω1x)+α2sin⁡(ω2x)−α3x+α4u(x;\alpha_1,\alpha_2,\alpha_3,\alpha_4,\omega_1,\omega_2)=\alpha_1\sin(\omega_1x)+\alpha_2\sin(\omega_2x)-\alpha_3x+\alpha_4 is used to derive the corresponding BCs and source term qq.

We further test the evolved Baldwinian-PINN on a single test task (α1=1,α2=1,α3=0.1,α4=0,ω1=0.7,ω2=1.5)(\alpha_1=1,\alpha_2=1,\alpha_3=0.1,\alpha_4=0,\omega_1=0.7,\omega_2=1.5) as described in [5].

  1. 2D Poisson’s equation: The 2D Poisson’s equation is represented by:

−(d2udx2+d2udy2)=q(x,y),x∈[−1,1],y∈[−1,1](22)-\left(\frac{d^2u}{dx^2}+\frac{d^2u}{dy^2}\right)=q(x,y),\quad x\in[-1,1],\quad y\in[-1,1] \tag*{(22)}
q(x,y)=∑j=1Jcjexp⁡((x−aj)2+(y−bj)2dj)(23)q(x,y)=\sum_{j=1}^{J}c_j\exp\left(\frac{(x-a_j)^2+(y-b_j)^2}{d_j}\right) \tag*{(23)}

subject to BCs u(x=−1)=0u(x=-1)=0, u(x=1)=0u(x=1)=0, u(y=−1)=0u(y=-1)=0, u(y=1)=0u(y=1)=0. The heat source q(x,y)q(x,y) is generated by the following scenarios: the number of heat source J∈[1,10]J\in[1,10], and coefficients sampled uniformly from aj∈[−0.6,0.6]a_j\in[-0.6,0.6], bj∈[−0.6,0.6]b_j\in[-0.6,0.6], cj∈[0.5,2]c_j\in[0.5,2], and dj∈[0.005,0.02]d_j\in[0.005,0.02]. The training and test sets both comprise 100 tasks with different source q(x,y)q(x,y) scenarios.

We further test the evolved Baldwinian-PINN on a single test task as described in [5], which is subject to a different domain x∈[0,1]x\in[0,1], y∈[0,1]y\in[0,1] and BCs u(x=0)=0u(x=0)=0, u(x=1)=0u(x=1)=0, u(y=0)=0u(y=0)=0, u(y=1)=0u(y=1)=0, with J=8J=8 and aj∈[0.1,0.9]a_j\in[0.1,0.9], bj∈[0.1,0.9]b_j\in[0.1,0.9], cj∈[0.8,1.2]c_j\in[0.8,1.2], and dj=0.01d_j=0.01.

  1. Helmholtz equation: The Helmholtz equation consists of 20 randomly sampled train tasks and 20 randomly sampled test tasks from the PDE/BC parameters α1∈(0,6]\alpha_1\in(0,6], α2∈(0,6]\alpha_2\in(0,6]:

(d2udx2+d2udy2)+u2=q,x∈[−1,1],y∈[−1,1](24)\left(\frac{d^2u}{dx^2}+\frac{d^2u}{dy^2}\right)+u^2=q,\quad x\in[-1,1],\quad y\in[-1,1] \tag*{(24)}

where the exact solution u(x,y;α1,α2)=sin⁡(α1πx)sin⁡(α2πy)u(x,y;\alpha_1,\alpha_2)=\sin(\alpha_1\pi x)\sin(\alpha_2\pi y) is used to derive the corresponding BCs and source term qq [6, 7].

We further test the evolved Baldwinian-PINN on a single test task (α1=2.5,α2=2.5)(\alpha_1=2.5,\alpha_2=2.5) as described in [6].

Results for set of linear ODE/PDE problems

Table S1 compares the generalization performance between Baldwinian-PINNs and baseline meta-learning PINN models (NRPINN [5] and Hyper-LR-PINN [6]) on linear benchmark problems.

Figure S4 and Figure S5 provide visualization of the Baldwinian-PINN solutions and errors for the 2D Poisson’s and Helmholtz tasks, respectively. The visualization results show good performance on the diverse set of PDE tasks, as exhibited by source patterns and frequencies. Note that the train and test task distributions as used in this work have much greater diversity than those studied in prior meta-learning PINN works such as [S5–S7]. Our PDE parameters’ range for the Helmholtz problem is 6 times larger than [6, 7], thereby encapsulating a much broader frequency spectrum, including tasks in the higher frequency range which are also more challenging to learn.

To demonstrate the diversity of the tasks, we compare the performance of different Baldwinian-PINN models, with the first one learning from n=10n=10 lower frequency train tasks α1,α2∈(0,1]\alpha_1,\alpha_2\in(0,1] for 200 neuroevolution iterations (train MSE <1e−11<1\mathrm{e}{-11}); while the second one is learning from n=10n=10 higher frequency train tasks α1,α2∈[5,6]\alpha_1,\alpha_2\in[5,6] for 200 neuroevolution iterations (train MSE <1e−6<1\mathrm{e}{-6}). Both low- and high-frequency-learned models are applied to the same set of n=60n=60 test tasks drawing from the full frequency range α1,α2∈(0,6]\alpha_1,\alpha_2\in(0,6], and their test MSE results are 4.8e−2 ±1.5e−14.8\mathrm{e}{-2}\,{}^{\pm1.5\mathrm{e}{-1}} and 3.8e−2 ±8.7e−23.8\mathrm{e}{-2}\,{}^{\pm8.7\mathrm{e}{-2}}, respectively.

Their test MSE results are 2-3 orders of magnitudes higher than the 1.6e−5 ±5.1e−51.6\mathrm{e}{-5}\,{}^{\pm5.1\mathrm{e}{-5}} achieved by the original Baldwinian-PINN. Note that the original model is learned from n=20n=20 train tasks α1,α2∈(0,6]\alpha_1,\alpha_2\in(0,6] for 400 neuroevolution iterations (train MSE <1e−5<1\mathrm{e}{-5}).

Figure S6 provides the comparison of test MSE distributions and visualization of selected (worst to best along the MSE spectrum) Baldwinian-PINN solutions and errors for the low- and high-frequency-learned Baldwinian-PINN models. From the results, we can observe that the low-frequency-learned model tends to give inaccurate physics-informed predictions for the test tasks from the other side of the frequency spectrum (i.e., high and mixed frequencies), and vice versa.

Figure S6. The MSE distributions on n=60n = 60 Helmholtz test tasks for the original Baldwinian-PINN (evolved by NES to meta-learn from n=20n = 20 train tasks α1∈(0,6]\alpha_1 \in(0, 6], α2∈(0,6]\alpha_2 \in(0, 6]) and its variants, i.e., using a different set of train tasks, or a different evolutionary algorithm, or both different evolutionary algorithm and neural architecture. The solutions and errors for selected (worst to best along the MSE spectrum) Helmholtz test tasks are visualized for both low- and high-frequency-learned Baldwinian-PINN models, and also the NES-evolved Baldwinian-PINN.

These results highlight the effectiveness of the Baldwinian neuroevolution for generating Baldwinian-PINN models that are “genetically equipped” to perform well over diverse task distribution pertaining to the training environment, e.g., low frequency, high frequency, or full frequency spectrum, as represented in the population.

Baldwinian-PINN with different evolutionary algorithms and neural architectures

The CMA-ES algorithm used in our experimental study is merely an instantiation of the proposed Baldwinian neuroevolution framework for meta-learning PINN. The outer-loop evolution procedure is agnostic to, and can be seamlessly switched to other state-of-the-art evolutionary search algorithms.

In this section, we present the results when Baldwinian neuroevolution is carried out by 3 alternate algorithms: simple genetic algorithm (GA) [8], policy gradients with parameter-based exploration (PGPE) [9], and natural evolution strategies (NES) variant [10], using the Helmholtz example. Note that the same optimization settings (e.g., initial standard deviation, population size, neuroevolution iterations) as the original (CAM-ES-evolved) Baldwinian-PINN are used (i.e., we do not fine-tune the settings for each algorithm). The search dimension consists of 36 network hyperparameters controlling the distributional mean and spread of the weights and biases, the learning hyperparameter λ\lambda, and the loss importance hyperparameter λpde\lambda_{pde} for re-balancing the relative importance between PDE and BC/IC errors in the least-squares solution.

Figure S4. Baldwinian-PINN solutions selected (worst to best along the MSE spectrum) from the n=60n = 60 2D Poisson’s test tasks, and an additional test task (subject to a different domain x∈[0,1]x \in[0, 1], y∈[0,1]y \in[0, 1]) described in [5], showing good performance on the diverse set of tasks exhibited by the source patterns.

Their test MSE values are 1.1±8.1e−11.1 \pm8.1\mathrm{e}{-1}, 3.0e−5±1.5e−43.0\mathrm{e}{-5} \pm1.5\mathrm{e}{-4}, and 2.9e−8±9.8e−82.9\mathrm{e}{-8} \pm9.8\mathrm{e}{-8}, respectively. The simple GA performs much worse than the original Baldwinian-PINN. On the other hand, the NES-evolved Baldwinian-PINN significantly outperforms the CMA-ES-evolved Baldwinian-PINN, with more than 2 orders of magnitude MSE improvement on test tasks. Their MSE distributions on n=60n = 60 test tasks are compared in Figure S6, together with visualization of the selected (worst to best along the MSE spectrum) solutions and errors obtained by the NES-evolved Baldwinian-PINN.

Similarly, the Baldwinian-PINN neural architecture described in § 3-D1 is merely an instantiation of the proposed Baldwinian neuroevolution framework for meta-learning PINN. The extension from a simple, single hidden layer neural architecture to a deeper, more complex neural architecture is straightforward.

To demonstrate this, we construct a MLP with multiple hidden layers with a mix of sin and softplus activation functions, and additional skip connections from the early hidden layers to the final layer. The total number of weight parameters in this deeper Baldwinian-PINN is 68,480 (including 1280 weights in the final layer). Note that the learned final layer is the outcome of lifetime learning through the Tikhonov regularization computation, whereas these 67,200 weight parameters before the final layer are sampled from 32 normal distributions. There are 64 distributional hyperparameters, 1 learning hyperparameter λ\lambda, and 1 loss importance hyperparameter λpde\lambda_{pde} evolved by the NES algorithm, using the same optimization settings (e.g., initial standard deviation, population size, neuroevolution iterations) as the original Baldwinian-PINN. For this study, the original Baldwinian-PINN architecture consists of 3600 weight parameters (including 900 weights in the final layer).

Despite having a much larger number of weight parameters in the deeper Baldwinian-PINN model, Baldwinian neuroevolution managed to arrive at a good set of distributional hyperparameters for the model to make a fast and accurate physics-based prediction on the test tasks (test MSE=2.6e−7±6.7e−72.6\mathrm{e}{-7} \pm6.7\mathrm{e}{-7}), in the Helmholtz problem.

The results described in this section underscore the promise of the Baldwinian neuroevolution framework for meta-learning PINN. Further improvements to the search algorithm and neural architecture design may boost the Baldwinian-PINN’s learning speed and accuracy on other complex physics problems.

Figure S5. Baldwinian-PINN solutions selected (worst to best along the MSE spectrum) from the n=60n = 60 Helmholtz test tasks, and an additional test task (α1=2.5\alpha_1 = 2.5, α2=2.5\alpha_2 = 2.5) described in [6]. Baldwinian-PINN shows good performance on the diverse set of PDE tasks exhibited by the frequencies.

Noisy or scarce training-task labels

Our lifetime-learning step is entirely physics-based, requires no labels at test time, and is solved in closed form with Tikhonov regularization; this inherently resists noisy or scarce data. When labels in the training tasks are noisy or sparse, we can rebalance the physics-data trade-off in the loss function by tuning the loss weights κLSE\kappa_{LSE} and κMSE\kappa_{MSE} to further strengthen the physics prior.

Empirically, Figure S7 evaluates Baldwinian-PINNs’ lifetime-learning performance on a set of 1D Poisson’s equation tasks (n=60n = 60) when the training-task labels are noisy or scarce. As observation noise increases from 0 to 100% under both zero-mean Gaussian noise and positively biased noise scenarios, the test error rises smoothly rather than catastrophically. Meanwhile, test accuracy remains low even when only 1–10% of points from the training tasks are observed, because the physics residual and BC/IC rows constrain the solution and meta-learning can be effectively guided by a few quality measurements, demonstrating that Baldwinian-PINNs tolerate non-ideal data and can operate under severe data scarcity.

Robustness to noisy and sparse observations on 1D Poisson’s problem

Figure S7. Robustness to noisy and sparse observations on 1D Poisson’s problem. Top: distribution of test MSE obtained by Baldwinian-PINNs when noisy or sparse training-task labels are used. The training data noise level ranges from 0% (baseline) to 100% for both zero-mean “Gaussian” noise and positively biased “Gaussian (+ve)” noise scenarios, whereas the number of sparse observations ranges from 100% (baseline) to 1%. Bottom: examples of training tasks with corresponding noisy (25% Gaussian; 100% Gaussian +ve) and sparse (1%) observation scenarios. Baldwinian-PINNs degrade gracefully with noise and exhibit low error with very limited observations.

Studies on Set of Nonlinear PDE Problems

The nonlinear PDE problems (Problems 7-11) used for demonstrating the Baldwinian neuroevolution of physics are described below.

  1. Burgers’ equation: The Burgers’ equation [12] consists of 16 randomly sampled train tasks and 32 uniformly sampled test tasks from the PDE parameter γ∈[5e−3,5e−2]\gamma\in[5\mathrm{e}{-3}, 5\mathrm{e}{-2}]:

dudt+ududx−γd2udx2=0,x∈[−1,1],t∈(0,1](25)\frac{du}{dt}+u\frac{du}{dx}-\gamma\frac{d^2u}{dx^2}=0, \qquad x\in[-1,1],\quad t\in(0,1] \tag*{(25)}

subject to IC u(x,t=0)=−sin⁡(πx)u(x,t=0)=-\sin(\pi x).

  1. Nonlinear heat equation: The nonlinear heat equation consists of 13 randomly sampled train tasks and 64 uniformly sampled test tasks from the PDE parameters γ∈[1,π]\gamma\in[1,\pi], k∈[1,π]k \in[1,\pi]:

dudt−γd2udx2+ktanh⁡(u)=q,x∈[−1,1],t∈(0,1](26)\frac{du}{dt}-\gamma\frac{d^2u}{dx^2}+k\tanh(u)=q, \qquad x\in[-1,1],\quad t\in(0,1] \tag*{(26)}

where the exact solution u(x,t;γ,k)=ksin⁡(πx)exp⁡(−πkx2)exp⁡(−πt2)u(x,t;\gamma,k)=k\sin(\pi x)\exp(-\pi kx^2)\exp(-\pi t^2) is used to derive the corresponding IC, BCs and source term qq.

  1. Nonlinear Allen-Cahn equation: The nonlinear Allen-Cahn equation consists of 16 randomly sampled train tasks and 32 uniformly sampled test tasks from the PDE parameter γ∈(0,π]\gamma\in(0,\pi]:

γ(d2udx2+d2udy2)+u(u2−1)=q,x∈[−1,1],y∈[−1,1](27)\gamma\left(\frac{d^2u}{dx^2}+\frac{d^2u}{dy^2}\right)+u(u^2-1)=q, \qquad x\in[-1,1],\quad y\in[-1,1] \tag*{(27)}

where the exact solution u(x,y;γ)=exp⁡(−γ(x+0.7))sin⁡(πx)sin⁡(πy)u(x,y;\gamma)=\exp(-\gamma(x+0.7))\sin(\pi x)\sin(\pi y) is used to derive the corresponding BCs and source term qq.

  1. Nonlinear diffusion-reaction equation: The nonlinear diffusion-reaction equation [13] consists of 22 randomly sampled train tasks and 64 uniformly sampled test tasks from the PDE parameter γ∈[1,π]\gamma\in[1,\pi], k∈[1,π]k \in[1,\pi]:

γ(d2udx2+d2udy2)+ku2=q,x∈[−1,1],y∈[−1,1](28)\gamma\left(\frac{d^2u}{dx^2}+\frac{d^2u}{dy^2}\right)+ku^2=q, \qquad x\in[-1,1],\quad y\in[-1,1] \tag*{(28)}

where the exact solution u(x,y;γ,k)=ksin⁡(πx)sin⁡(πy)exp⁡(−γkx2+y2)u(x,y;\gamma,k)=k\sin(\pi x)\sin(\pi y)\exp\left(-\gamma\sqrt{kx^2+y^2}\right) is used to derive the corresponding BCs and source term qq.

  1. 6D parametric diffusion-reaction: The 6D parametric diffusion-reaction problem consists of 17 randomly sampled train tasks and 100 randomly sampled test tasks from the PDE/BC parameters (α1,α2,ω1,ω2,ω3,ω4)(\alpha_1,\alpha_2,\omega_1,\omega_2,\omega_3,\omega_4), α\alpha’s ∈[0.1,1]\in[0.1,1], ω\omega’s ∈[1,5]\in[1,5]:

(d2udx2+d2udy2)+u(1−u2)=q,x∈[−1,1],y∈[−1,1](29)\left(\frac{d^2u}{dx^2}+\frac{d^2u}{dy^2}\right)+u(1-u^2)=q, \qquad x\in[-1,1],\quad y\in[-1,1] \tag*{(29)}

where the exact solution u(x,y;α1,α2,ω1,ω2,ω3,ω4)=α1tanh⁡(ω1x)tanh⁡(ω2y)+α2sin⁡(ω3x)sin⁡(ω4y)u(x,y;\alpha_1,\alpha_2,\omega_1,\omega_2,\omega_3,\omega_4)=\alpha_1\tanh(\omega_1x)\tanh(\omega_2y)+\alpha_2\sin(\omega_3x)\sin(\omega_4y) is used to derive the corresponding BCs and source term qq.

Results for set of nonlinear PDE problems

As per [S11], the spatio-temporal domain in Problems 7-8 is uniformly discretized into 256×100256\times100, and the 2D spatial domain in Problems 9-11 is uniformly discretized into 128×128128\times128, for the test tasks. The comparison of generalization performance and compute cost (i.e., lifetime learning for Baldwinian-PINNs or gradient fine-tuning for baselines) between Baldwinian-PINN and several meta-learned PINN models (based on results reported in [S11]) are summarized in Table S2.

Figure S8 provides additional visualization results for the 5 nonlinear PDE problems described above, showing the corresponding Baldwinian-PINN solutions with the worst and median accuracy along the MSE spectrum (pooled from all test tasks ×\times 5 individual runs) for each problem.

Additional Results for Navier-Stokes (N-S) Equations for Fluid Flow

Our Baldwinian-PINN model for the fluid flow problem consists of multiple hidden layers with a mix of sin, tanh, and softplus activation functions, with skip connections from the early hidden layers to the final layer. The total number of weight parameters is 133,504 (including 1152 weights in the final layer). Figure S9 provides additional visualization comparing Baldwinian-PINN predictions and ground truth for the lid-driven cavity flow across unseen Reynolds numbers Re={5,25,100,400,1000}Re=\{5,25,100,400,1000\}, showing uu- and vv-velocity fields alongside absolute error maps. Errors remain uniformly small across the domain and over two orders of magnitude in ReRe, indicating strong generalization of the Baldwinian-PINN model to markedly different flow regimes.

Figure S8. Baldwinian-PINN solutions with the worst and (approximately) average accuracy along the MSE spectrum (pooled from all test tasks × 5 individual runs), for 5 nonlinear PDE problems.

Baseline SGD-Trained DNN and PINN Models Used in Ablation Study

Consider the general inputs (x,t,ϑ)(x,t,\vartheta) for the spatial-temporal domain and task parameter ϑ\vartheta. The data-driven loss function of a SGD-trained DNN model computes the MSE between the DNN output uDNN(xi,ti,ϑi)u_{\mathrm{DNN}}(x_i,t_i,\vartheta_i) against the target uilabelu_i^{label} over i=1,…,ni=1,\ldots,n labelled data pooled from a batch of training tasks:

lDNN=ldata=1n∑i=1n(uilabel−uDNN(xi,ti,ϑi))2(30)l_{\mathrm{DNN}}=l_{data}=\frac{1}{n}\sum_{i=1}^{n}\left(u_i^{label}-u_{\mathrm{DNN}}(x_i,t_i,\vartheta_i)\right)^2 \tag*{(30)}

The loss function of a (baseline) SGD-trained PINN model is defined as:

lPINN=λdata ldata+λpde lpde+λic lic+λbc lbc(31)l_{\mathrm{PINN}}=\lambda_{data}\,l_{data}+\lambda_{pde}\,l_{pde}+\lambda_{ic}\,l_{ic}+\lambda_{bc}\,l_{bc} \tag*{(31)}
lpde=1npde∑i=1npde(Nϑ[uPINN(xipde,tipde,ϑi)]−h(xipde,tipde))2(32)l_{pde}=\frac{1}{n_{pde}}\sum_{i=1}^{n_{pde}}\left(\mathcal{N}_{\vartheta}\left[u_{\mathrm{PINN}}(x_i^{pde},t_i^{pde},\vartheta_i)\right]-h(x_i^{pde},t_i^{pde})\right)^2 \tag*{(32)}
lic=1nic∑i=1nic(uPINN(xiic,0,θi)−u0(xiic))2(33)l_{ic}=\frac{1}{n_{ic}}\sum_{i=1}^{n_{ic}}\left(u_{\mathrm{PINN}}(x_i^{ic},0,\theta_i)-u_0(x_i^{ic})\right)^2 \tag*{(33)}
lbc=1nbc∑i=1nbc(B[uPINN(xibc,tibc,θi)]−g(xibc,tibc))2(34)l_{bc}=\frac{1}{n_{bc}}\sum_{i=1}^{n_{bc}}\left(\mathcal{B}\left[u_{\mathrm{PINN}}(x_i^{bc},t_i^{bc},\theta_i)\right]-g(x_i^{bc},t_i^{bc})\right)^2 \tag*{(34)}

such that the PINN output uPINN(xi,ti,ϑi)u_{\mathrm{PINN}}(x_i,t_i,\vartheta_i) satisfies PDE, IC, and BC for a set of training samples (xipde,tipde,ϑi)(x_i^{pde},t_i^{pde},\vartheta_i), i=1,…,npdei=1,\ldots,n_{pde}, (xiic,0,ϑi)(x_i^{ic},0,\vartheta_i), i=1,…,nici=1,\ldots,n_{ic}, (xibc,tibc,ϑi)(x_i^{bc},t_i^{bc},\vartheta_i), i=1,…,nbci=1,\ldots,n_{bc} from the respective domain and task, in addition to minimizing the MSE from the labelled data. SGD-trained PINNs typically converge much slower than DNN because of the additional loss terms. We perform a coarse search for the loss balancing parameters (λdata,λpde,λic,λbc)(\lambda_{data},\lambda_{pde},\lambda_{ic},\lambda_{bc}) to improve the convergence of the PINN loss.

We tested a range of learning rate schedules, e.g., an initial learning rate ={5e−4,1e−3,1e−2}=\{5\mathrm{e}{-4},1\mathrm{e}{-3},1\mathrm{e}{-2}\} for the first 40% of training iterations followed by cosine decay towards 1e−61\mathrm{e}{-6}. Table S3 and Table S4 give the DNN / PINN model and training configurations and their subsequent performance on test tasks for the ablation study.

Problem (no. test task)ModelDNN 5e-4 DNNDNN 5e-4 Tik-RegDNN 5e-3 DNNDNN 5e-3 Tik-RegDNN 5e-2 DNNDNN 5e-2 Tik-RegPINN 5e-4 PINNPINN 5e-4 Tik-RegPINN 5e-3 PINNPINN 5e-3 Tik-RegPINN 5e-2 PINNPINN 5e-2 Tik-Reg
1 Convection-diffusion (x,α)→u(x,\alpha) \to u (n=110)(n = 110)1a3.5e−4±2.2e−33.5\mathrm{e}{-4} \pm2.2\mathrm{e}{-3}1.5e−1±1.1e−11.5\mathrm{e}{-1} \pm1.1\mathrm{e}{-1}2.3e−4±1.8e−32.3\mathrm{e}{-4} \pm1.8\mathrm{e}{-3}3.8e−2±7.6e−23.8\mathrm{e}{-2} \pm7.6\mathrm{e}{-2}1.2e−2±2.6e−21.2\mathrm{e}{-2} \pm2.6\mathrm{e}{-2}2.3e−1±2.8e−22.3\mathrm{e}{-1} \pm2.8\mathrm{e}{-2}4.6e−3±2.0e−24.6\mathrm{e}{-3} \pm2.0\mathrm{e}{-2}1.4e−2±4.3e−21.4\mathrm{e}{-2} \pm4.3\mathrm{e}{-2}1.9e−3±1.1e−21.9\mathrm{e}{-3} \pm1.1\mathrm{e}{-2}7.2e−3±2.5e−27.2\mathrm{e}{-3} \pm2.5\mathrm{e}{-2}2.2e−1±5.6e−22.2\mathrm{e}{-1} \pm5.6\mathrm{e}{-2}2.3e−1±2.8e−22.3\mathrm{e}{-1} \pm2.8\mathrm{e}{-2}
1b6.9e−4±2.2e−36.9\mathrm{e}{-4} \pm2.2\mathrm{e}{-3}1.8e−1±1.0e−11.8\mathrm{e}{-1} \pm1.0\mathrm{e}{-1}1.7e−4±1.0e−31.7\mathrm{e}{-4} \pm1.0\mathrm{e}{-3}1.3e−1±1.2e−11.3\mathrm{e}{-1} \pm1.2\mathrm{e}{-1}1.9e−3±7.4e−31.9\mathrm{e}{-3} \pm7.4\mathrm{e}{-3}2.2e−2±3.6e−22.2\mathrm{e}{-2} \pm3.6\mathrm{e}{-2}1.3e−1±9.3e−21.3\mathrm{e}{-1} \pm9.3\mathrm{e}{-2}1.6e−1±1.1e−11.6\mathrm{e}{-1} \pm1.1\mathrm{e}{-1}2.1e−2±3.5e−22.1\mathrm{e}{-2} \pm3.5\mathrm{e}{-2}1.1e−1±1.1e−11.1\mathrm{e}{-1} \pm1.1\mathrm{e}{-1}3.4e−1±3.7e−13.4\mathrm{e}{-1} \pm3.7\mathrm{e}{-1}1.9e−1±9.6e−21.9\mathrm{e}{-1} \pm9.6\mathrm{e}{-2}
2a7.1e−3±6.8e−37.1\mathrm{e}{-3} \pm6.8\mathrm{e}{-3}2.3e−1±4.7e−22.3\mathrm{e}{-1} \pm4.7\mathrm{e}{-2}4.5e−3±1.3e−24.5\mathrm{e}{-3} \pm1.3\mathrm{e}{-2}2.2e−1±5.3e−22.2\mathrm{e}{-1} \pm5.3\mathrm{e}{-2}2.3e−3±4.0e−32.3\mathrm{e}{-3} \pm4.0\mathrm{e}{-3}2.1e−1±7.2e−22.1\mathrm{e}{-1} \pm7.2\mathrm{e}{-2}2.2e−1±4.5e−22.2\mathrm{e}{-1} \pm4.5\mathrm{e}{-2}2.3e−1±4.9e−22.3\mathrm{e}{-1} \pm4.9\mathrm{e}{-2}1.9e−1±7.0e−21.9\mathrm{e}{-1} \pm7.0\mathrm{e}{-2}2.2e−1±5.6e−22.2\mathrm{e}{-1} \pm5.6\mathrm{e}{-2}2.0e−1±6.5e−22.0\mathrm{e}{-1} \pm6.5\mathrm{e}{-2}2.1e−1±7.8e−22.1\mathrm{e}{-1} \pm7.8\mathrm{e}{-2}
2b6.0e−3±7.1e−36.0\mathrm{e}{-3} \pm7.1\mathrm{e}{-3}2.2e−1±5.7e−22.2\mathrm{e}{-1} \pm5.7\mathrm{e}{-2}3.8e−3±5.0e−33.8\mathrm{e}{-3} \pm5.0\mathrm{e}{-3}2.2e−1±6.1e−22.2\mathrm{e}{-1} \pm6.1\mathrm{e}{-2}2.5e−3±5.2e−32.5\mathrm{e}{-3} \pm5.2\mathrm{e}{-3}2.1e−1±7.7e−22.1\mathrm{e}{-1} \pm7.7\mathrm{e}{-2}2.2e−1±5.1e−22.2\mathrm{e}{-1} \pm5.1\mathrm{e}{-2}2.2e−1±5.8e−22.2\mathrm{e}{-1} \pm5.8\mathrm{e}{-2}2.2e−1±5.2e−22.2\mathrm{e}{-1} \pm5.2\mathrm{e}{-2}2.2e−1±5.9e−22.2\mathrm{e}{-1} \pm5.9\mathrm{e}{-2}2.1e−1±5.8e−22.1\mathrm{e}{-1} \pm5.8\mathrm{e}{-2}2.2e−1±6.1e−22.2\mathrm{e}{-1} \pm6.1\mathrm{e}{-2}
6 Kinematics (t,vel0,a0,Cd,A,m)→(x,y)(t, vel_0, a_0, C_d, A, m) \to(x,y) (n=100)(n = 100)1a8.5e+0±2.8e+18.5\mathrm{e}{+0} \pm2.8\mathrm{e}{+1}2.9e−2±4.2e−12.9\mathrm{e}{-2} \pm4.2\mathrm{e}{-1}3.7e+1±8.8e+13.7\mathrm{e}{+1} \pm8.8\mathrm{e}{+1}9.4e+2±3.5e+39.4\mathrm{e}{+2} \pm3.5\mathrm{e}{+3}2.0e+2±6.2e+22.0\mathrm{e}{+2} \pm6.2\mathrm{e}{+2}2.6e+3±4.6e+32.6\mathrm{e}{+3} \pm4.6\mathrm{e}{+3}1.1e+1±3.7e+11.1\mathrm{e}{+1} \pm3.7\mathrm{e}{+1}2.5e+2±3.3e+32.5\mathrm{e}{+2} \pm3.3\mathrm{e}{+3}1.9e+1±6.0e+11.9\mathrm{e}{+1} \pm6.0\mathrm{e}{+1}4.8e+1±6.3e+24.8\mathrm{e}{+1} \pm6.3\mathrm{e}{+2}2.5e+3±4.5e+32.5\mathrm{e}{+3} \pm4.5\mathrm{e}{+3}2.6e+3±4.6e+32.6\mathrm{e}{+3} \pm4.6\mathrm{e}{+3}
1b3.5e+0±1.4e+13.5\mathrm{e}{+0} \pm1.4\mathrm{e}{+1}3.0e−3±4.5e−23.0\mathrm{e}{-3} \pm4.5\mathrm{e}{-2}1.9e+1±6.7e+11.9\mathrm{e}{+1} \pm6.7\mathrm{e}{+1}3.2e−4±2.9e−33.2\mathrm{e}{-4} \pm2.9\mathrm{e}{-3}5.2e+0±9.1e+05.2\mathrm{e}{+0} \pm9.1\mathrm{e}{+0}2.6e+3±4.6e+32.6\mathrm{e}{+3} \pm4.6\mathrm{e}{+3}1.3e+0±5.7e+01.3\mathrm{e}{+0} \pm5.7\mathrm{e}{+0}7.9e−6±1.1e−47.9\mathrm{e}{-6} \pm1.1\mathrm{e}{-4}3.5e+0±1.4e+13.5\mathrm{e}{+0} \pm1.4\mathrm{e}{+1}1.6e−5±1.1e−41.6\mathrm{e}{-5} \pm1.1\mathrm{e}{-4}3.0e+3±3.5e+33.0\mathrm{e}{+3} \pm3.5\mathrm{e}{+3}8.1e−2±1.6e+08.1\mathrm{e}{-2} \pm1.6\mathrm{e}{+0}
2a3.0e+0±4.3e+13.0\mathrm{e}{+0} \pm4.3\mathrm{e}{+1}4.4e−5±2.5e−44.4\mathrm{e}{-5} \pm2.5\mathrm{e}{-4}7.9e+0±1.6e+17.9\mathrm{e}{+0} \pm1.6\mathrm{e}{+1}2.4e+1±3.8e+22.4\mathrm{e}{+1} \pm3.8\mathrm{e}{+2}1.8e+1±2.7e+11.8\mathrm{e}{+1} \pm2.7\mathrm{e}{+1}2.3e+2±1.6e+32.3\mathrm{e}{+2} \pm1.6\mathrm{e}{+3}2.8e+1±5.5e+12.8\mathrm{e}{+1} \pm5.5\mathrm{e}{+1}9.4e−3±1.7e−19.4\mathrm{e}{-3} \pm1.7\mathrm{e}{-1}3.3e+0±6.5e+03.3\mathrm{e}{+0} \pm6.5\mathrm{e}{+0}3.1e−4±2.8e−33.1\mathrm{e}{-4} \pm2.8\mathrm{e}{-3}1.5e+1±2.9e+11.5\mathrm{e}{+1} \pm2.9\mathrm{e}{+1}1.2e−2±2.7e−11.2\mathrm{e}{-2} \pm2.7\mathrm{e}{-1}
2b1.5e+1±2.7e+11.5\mathrm{e}{+1} \pm2.7\mathrm{e}{+1}5.2e−2±5.4e−15.2\mathrm{e}{-2} \pm5.4\mathrm{e}{-1}2.1e+0±3.5e+02.1\mathrm{e}{+0} \pm3.5\mathrm{e}{+0}2.0e−1±2.1e+02.0\mathrm{e}{-1} \pm2.1\mathrm{e}{+0}5.0e+2±7.7e+25.0\mathrm{e}{+2} \pm7.7\mathrm{e}{+2}1.4e−3±7.9e−31.4\mathrm{e}{-3} \pm7.9\mathrm{e}{-3}2.2e+1±4.4e+12.2\mathrm{e}{+1} \pm4.4\mathrm{e}{+1}4.7e−2±3.1e−14.7\mathrm{e}{-2} \pm3.1\mathrm{e}{-1}4.0e+0±8.7e+04.0\mathrm{e}{+0} \pm8.7\mathrm{e}{+0}5.2e−3±3.5e−25.2\mathrm{e}{-3} \pm3.5\mathrm{e}{-2}7.8e+1±1.1e+27.8\mathrm{e}{+1} \pm1.1\mathrm{e}{+2}5.9e−3±3.9e−25.9\mathrm{e}{-3} \pm3.9\mathrm{e}{-2}

Table S4. Table S4. GENERALIZATION PERFORMANCE OF DNN / PINN MODELS IN ABLATION STUDY.

Linear and Nonlinear ODE: Task Diversity

Figure S10 provides additional visualization results for the convection-diffusion (Problem 1), 1D Poisson’s (Problem 3), and nonlinear kinematics (Problem 6) tasks, showing the diversity of task distributions on which we have demonstrated the utility of the current study’s Baldwinian-PINN.

Figure S10. The distribution of convection-diffusion, 1D Poisson’s, and nonlinear kinematics tasks for training and test.

References

  1. [1]P.-H. Chiu, “An improved divergence-free-condition compensated method for solving incompressible flows on collocated grids,” Computers & Fluids, vol. 162, pp. 39–54, 2018.DOI
  2. [3]S. Gottlieb and C.-W. Shu, “Total variation diminishing Runge-kutta schemes,” Mathematics of Computation, vol. 67, pp. 73–85, 1998.DOI
  3. [4]D. Anderson, J. C. Tannéhill, R. H. Pletcher, R. Munipalli, and V. Shankar, Computational fluid mechanics and heat transfer. Boca Raton, Florida, USA: CRC press, 2020.
  4. [5]X. Liu, X. Zhang, W. Peng, W. Zhou, and W. Yao, “A novel meta-learning initialization method for physics-informed neural networks,” Neural Computing and Applications, vol. 34, no. 17, pp. 14511–14534, 2022.DOI
  5. [6]W. Cho, K. Lee, D. Rim, and N. Park, “Hypernetwork-based meta-learning for low-rank physics-informed neural networks,” Advances in Neural Information Processing Systems, vol. 36, 2024.
  6. [7]M. Toloubidokhti, Y. Ye, R. Missel, X. Jiang, N. Kumar, R. Shrestha, and L. Wang, “Dats: Difficulty-aware task sampler for meta-learning physics-informed neural networks,” in The Twelfth International Conference on Learning Representations, 2023.
  7. [8]Y. Tang, Y. Tian, and D. Ha, “Evojax: Hardware-accelerated neuroevolution,” arXiv preprint arXiv:2202.05008, 2022.DOI
  8. [9]F. Sehnke, C. Osendorfer, T. Rückstieß, A. Graves, J. Peters, and J. Schmidhuber, “Parameter-exploring policy gradients,” Neural Networks, vol. 23, no. 4, pp. 551–559, 2010.DOI
  9. [10]M. Nomura and I. Ono, “Fast moving natural evolution strategy for high-dimensional problems,” in 2022 IEEE Congress on Evolutionary Computation (CEC). IEEE, 2022, pp. 1–8.arxiv.org/abs/2201.11422
  10. [12]J. Bec and K. Khanin, “Burgers turbulence,” Physics reports, vol. 447, no. 1-2, pp. 1–66, 2007.DOI
  11. [13]C. Rao, P. Ren, Q. Wang, O. Buyukozturk, H. Sun, and Y. Liu, “Encoding physics to learn reaction–diffusion processes,” Nature Machine Intelligence, vol. 5, no. 7, pp. 765–779, 2023.arxiv.org/abs/2106.04781

Paper details

vlnc:W_ezk2tteg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents