betaFeedback

WorksComputer Science

Enhanced accuracy through ensembling of randomly initialized auto-regressive models for time-dependent PDEs

Abstract

Systems governed by partial differential equations (PDEs) require computationally intensive numerical solvers to predict spatiotemporal field evolution. While machine learning (ML) surrogates offer faster solutions, autoregressive inference with ML models suffer from error accumulation over successive predictions, limiting their long-term accuracy. We propose a deep ensemble framework to address this challenge, where multiple ML surrogate models with random weight initializations are trained in parallel and aggregated during inference. This approach leverages the diversity of model predictions to mitigate error propagation while retaining the autoregressive strategies ability to capture the system's time dependent relations. We validate the framework on three PDE-driven dynamical systems - stress evolution in heterogeneous microstructures, Gray-Scott reaction-diffusion, and planetary-scale shallow water system - demonstrating consistent reduction in error accumulation over time compared to individual models. Critically, the method requires only a few time steps as input, enabling full trajectory predictions with inference times significantly faster than numerical solvers. Our results highlight the robustness of ensemble methods in diverse physical systems and their potential as efficient and accurate alternatives to traditional solvers. The codes for this work are available on GitHub (https://github.com/Graham-Brady-Research-Group/AutoregressiveEnsemble_SpatioTemporal_Evolution).

Graphical Abstract

Enhanced accuracy through ensembling of randomly initialized auto-regressive models for time-dependent PDEs

Ishan Khurjekar, Indrashish Saha, Lori Graham-Brady, Somdatta Goswami

Graphical abstract showing data preparation, parallel ensemble model training, and inference through full trajectory autoregressive prediction

Figure 1.

Email addresses: [email protected] (Ishan Khurjekar), [email protected] (Indrashish Saha), [email protected] (Lori Graham-Brady), [email protected] (Somdatta Goswami)

Enhanced accuracy through ensembling of randomly initialized autoregressive models for time-dependent PDEs

Ishan Khurjekar; Indrashish Saha; Lori Graham-Brady; Somdatta Goswami

Highlights

  • An ensemble of machine learning surrogate models is initialized with random weights and trained in parallel for spatiotemporal field evolution tasks.

  • The ensemble approach leads to reduced error accumulation for the field evolution task in different PDE-driven dynamic systems.

keywords: time-dependent PDEs; surrogate model; autoregressive prediction; model ensemble; error accumulation reduction; mixture of experts

Introduction

Accurate simulation of complex physical systems often relies on solving time-dependent partial differential equations (PDEs) using traditional numerical methods (finite difference, finite element, etc.), but these approaches can be prohibitively expensive when high resolution or many repeated queries are required. In many engineering and scientific applications (e.g. design optimization, uncertainty quantification), it is therefore desirable to build fast data-driven surrogate models that approximate the PDE solution operator. Deep neural networks have emerged as a powerful class of surrogates, capable of learning high-dimensional mappings from current system states to future states with far less online computation than conventional solvers. These ML-based simulators can leverage large training datasets to encode complex dynamics and thus provide flexible, real-time prediction of spatiotemporal evolution.

For time-dependent problems, a common strategy is to train an autoregressive neural model that predicts the next time step (or increment) given the current state, and then to apply it recursively to generate a full time trajectory. This one-step training approach is often more accurate and grounded in the physics of the system than trying to predict far-future states directly. However, a well-known challenge is that small one-step errors accumulate over repeated predictions. In other words, the iterative use of a learned time update operator tends to compound its own prediction errors, degrading stability and accuracy when the model is rolled out for long horizons.

To address this challenge, several architectural innovations have been proposed. Recurrent networks (LSTM, GRU), attention-based transformers, and memory-augmented models have been introduced to improve temporal coherence. In the context of PDEs, neural operator (NO) frameworks such as DeepONet [1] and Fourier Neural Operator [2] offer a means to preserve spatial structures while modeling temporal evolution. Yet, NOs too have shown vulnerability to error accumulation in autoregressive settings [3]. Consequently, recent work has sought to enhance these models via architectural modifications [4, 5], hybrid schemes with numerical integrators [6], and RNN-based sequence learners on top of NO outputs [7, 8]. While effective, these solutions often introduce architectural complexity and increase tuning burden.

In this work, we propose a conceptually simple yet effective alternative: mitigating long-term prediction error through deep ensembling. Rather than modifying the architecture, we train multiple instances of the same autoregressive model, each with a different random initialization. Due to the stochastic nature of training (e.g., with stochastic gradient descent), these models converge to different local minima and yield diverse prediction behaviors [9, 10]. By averaging the predictions from this ensemble at each time step, we reduce individual model biases and suppress cumulative error. This approach is grounded in classical ensemble learning theory, which shows that aggregating diverse learners improves robustness and generalization [11]. While ensemble methods such as bagging, boosting, and Mixture of Experts (MoE) often rely on explicit diversity through data splits or gating mechanisms [12], we show that random initialization alone can induce sufficient diversity to enhance performance. Deep ensembles have been shown to improve accuracy, uncertainty calibration, and out-of-distribution robustness [13, 14, 15, 16]; we adapt this principle for the specific challenge of long-horizon PDE forecasting.

We evaluate our ensemble strategy on three representative PDE systems: (i) stress field evolution in a two-phase microstructure under increasing strain (solid mechanics), (ii) the shallow water equations (fluid dynamics), and (iii) the Gray-Scott reaction–diffusion (chemical kinetics). These problems span a range of dynamical behaviors - from smooth deformations to wave propagation and pattern formation. In all cases, our ensemble-based surrogates consistently outperform single-model rollouts in long-horizon accuracy, without requiring any architectural modification, memory augmentation, or additional training objectives.

Methods

Consider a 2D physical field p(x,y,t)\mathbf{p}(x,y,t) defined over a continuous spatiotemporal domain, subjected to an external time-dependent forcing function e(t)e(t). Numerical solutions typically assume a discretization of p(x,y,t)\mathbf{p}(x,y,t) as p(x,y,t)\mathbf{p}(\mathbf{x},\mathbf{y},t), where the components of vectors x\mathbf{x}, y\mathbf{y}, and tt are xix_i: i∈1,2,…,Nxi \in1,2,\ldots,N_x, yjy_j: j∈1,2,…,Nyj \in1,2,\ldots,N_y, and tkt_k: k∈1,…,NTk \in1,\ldots,N_T, Nx×NyN_x \times N_y are the spatial grid dimensions and NTN_T is the number of time steps. Assuming uniform discretization xi=iΔxx_i = i\Delta x, yj=jΔyy_j = j\Delta y, tk=kΔtt_k = k\Delta t, where Δx\Delta x, Δy\Delta y, and Δt\Delta t are the spatial and temporal resolutions, respectively. The forcing function e(t)e(t) is similarly discretized as e(t)e(t), or e(tk)e(t_k). In particular, numerical solvers attempt to find the solution at the next future state p(x,y,tk+1)\mathbf{p}(\mathbf{x},\mathbf{y},t_{k+1}) based on the solution history p(x,y,tk)\mathbf{p}(\mathbf{x},\mathbf{y},t_k): k∈1,…,NT−1k \in1,\ldots,N_T-1 and the forcing function e(tk+1)e(t_{k+1}): k∈1,…,NT−1k \in1,\ldots,N_T-1. Representing the full-time history of both the solution and the forcing function is typically infeasible; therefore, the solution is simplified through the use of state variables that provide a reduced-order representation of the forcing function and solution history. Such an approach allows the solution p(x,y,tk+1)\mathbf{p}(\mathbf{x},\mathbf{y},t_{k+1}) to be found based only on the solution at the previous time step tkt_k, the current values of the state variables, and the current value of the forcing function e(tk+1)e(t_{k+1}). Even with this simplified representation, calculation of the solution and the state variables at each time step is typically expensive, for example, requiring solutions involving high-dimensional matrices and iterative solvers.

More recently, ML representations of the solution have proven to be much more efficient, with accuracy approaching that of the physics-based numerical solvers. Researchers have proposed sequence-to-sequence mapping approaches using transformer [17, 18, 19, 20], CNNs [21], or Long Short Term Memory (LSTM) [22] architectures to predict full trajectory rollouts. But such an approach assumes that the external conditions for the full simulation time are known in advance, limiting the real-world applicability. A more realistic approach is to predict the solution field for one time step at a time, using the historical information from previous time steps (autoregressive).

Specifically, consider an autoregressive ML surrogate model trained for the temporal evolution of the solution field. Mathematically, this is expressed as:

p(x,y,tk+1)=M{p(x,y,tk),e(tk+1)},(1)\mathbf{p}(\mathbf{x},\mathbf{y},t_{k+1}) = \mathcal{M}\{\mathbf{p}(\mathbf{x},\mathbf{y},t_k), e(t_{k+1})\}, \tag*{(1)}

where M\mathcal{M} denotes the ML-model. To avoid the need to calculate and track state variables, the accuracy of this autoregressive model is improved by considering not just one previous time step, but the previous LL time step history, assuming that this subset of time steps contains a sufficient amount of the information necessary to make an accurate prediction for the next time step [23, 24]. This transforms Eq. (1) into,

p(x,y,tk+1)=M{p(x,y,tk−L:k),e(tk−L:k+1)},(2)\mathbf{p}(\mathbf{x},\mathbf{y},t_{k+1}) = \mathcal{M}\{\mathbf{p}(\mathbf{x},\mathbf{y},t_{k-L:k}), e(t_{k-L:k+1})\}, \tag*{(2)}

where p(x,y,tk−L:k)\mathbf{p}(\mathbf{x},\mathbf{y},t_{k-L:k}) are the 2D maps of the physical field in the previous LL time steps and e(tk−L:k+1)e(t_{k-L:k+1}) is the subset of the forcing function values in the LL previous time steps and the current time step.

Schematic representation of the Diffusion-inspired Temporal Transformer Operator for spatiotemporal field prediction

Figure 1 (PDF p. 7). Schematic representation of the Diffusion-inspired Temporal Transformer Operator for spatiotemporal field prediction.

The training data, Dtrain\mathbf{D}_{train}, is formed as tuples of the form: {p(x,y,tk−(L−1):k),e(tk−(L−1):k+1),p(x,y,tk+1)} ∀ k∈(L,NT−1)\{\mathbf{p}(\mathbf{x},\mathbf{y},t_{k-(L-1):k}), e(t_{k-(L-1):k+1}), \mathbf{p}(\mathbf{x},\mathbf{y},t_{k+1})\}\ \forall\ k \in(L,N_T-1). This creates a total of Ns∗(NT−L)N_s * (N_T-L) training samples, where NsN_s is the total number of unique trajectories simulated (either with variation only in initial conditions, external time-dependent variable, or both). The model is trained by empirical risk minimization,

M∗=arg⁡min⁡M∈M∑pi∈DtrainL(M(pi),pi),(3)\mathcal{M}^{*} = \underset{\mathcal{M}\in\mathbb{M}}{\arg\min}\sum_{\mathbf{p}_i\in\mathbf{D}_{train}}\mathcal{L}(\mathcal{M}(\mathbf{p}_i),\mathbf{p}_i), \tag*{(3)}

where M∗\mathcal{M}^{*} is the choice of model parametrized by a set of weights within the entire model space M\mathbb{M} that minimizes an appropriate loss function L(⋅)\mathcal{L}(\cdot) evaluated over all the training data pi∈Dtrain\mathbf{p}_i \in\mathbf{D}_{train}. The mean squared error is chosen as the loss function in this work.

In this work, we utilize the Diffusion-inspired Temporal Transformer Operator (DiTTO) for temporal evolution of fields [25]. In this approach, the time-dependent forcing function is encoded via a separate feedforward neural network (FNN) and this embedding is multiplied with each of the UNet intermediate layers, conditioning the UNet on the time-dependent variable.

Each block in the UNet consists of a convolutional layer with a residual connection (ResNet block [26]). This is followed by a temporal attention layer that captures dependencies across temporal features (physical fields at consecutive time instants represented as channels of an image). The attention mechanism [27] is denoted as,

attn⁡=softmax⁡(Q⋅KTdemb)V,(4)\operatorname{attn} = \operatorname{softmax}\left(\frac{\mathbf{Q} \cdot\mathbf{K}^{T}}{\sqrt{d_{emb}}}\right)\mathbf{V}, \tag*{(4)}

where Q\mathbf{Q}, K\mathbf{K}, V\mathbf{V}, and dembd_{emb} are the query, key, value, and the embedding dimensionality, respectively. The matrices Q\mathbf{Q}, K\mathbf{K}, and V\mathbf{V} are essentially learnable representations of the input features obtained to extract self-dependencies across either the spatial or temporal dimensions of the physical fields. The embedding dimensionality normalizes the attention to avoid scaling issues and stabilize training. A detailed illustration of DiTTO architecture is shown in Fig. 1.

At inference time, the ground truth for the first LL time-steps is input to the model M\mathcal{M}. The model predicts the field at the next time step. This is repeated recursively, with the model taking the LL recent model predictions to predict the successive time-step. At the end of the inference process, the entire trajectory prediction, p^(x,y,tL+1:NT)\hat{\mathbf{p}}(\mathbf{x}, y, t_{L+1:N_T}), is obtained.

Deep ensemble for autoregressive predictions

In autoregressive models for time-dependent systems, the predicted output at one time step is fed back as input for the next. While effective, this approach introduces a compounding error mechanism - small inaccuracies at early steps can propagate and grow over time. One remedy is to use a deep ensemble approach: an ensemble of models, each trained independently on the same dataset but initialized with different random weights [13]. Rather than selecting a single “best” model, retaining all trained models in an ensemble offers several advantages: (i) different initializations explore distinct regions of the function space, resulting in diverse yet accurate approximations; (ii) error accumulation in autoregressive inference is mitigated through averaging; and (iii) no computational effort is wasted, as all trained models contribute to the final prediction.

Theoretical support for deep ensemble comprising of deep neural networks highlights their ability to capture multiple function modes due to random initialization and non-convex loss landscapes [28]. Previous work has

Algorithm 1 Ensemble Approach for Accurate Spatiotemporal Prediction.

Training Phase:

Step 0: Prepare the training dataset DtrD_{\mathrm{tr}} as a collection of tuples:

Dtr={(p(x,y,tk−L:k), e(tk−L:k+1), p(x,y,tk+1))}∀ k∈[L,NT−1]D_{\mathrm{tr}} = \{(\mathbf{p}(\mathbf{x},\mathbf{y},t_{k-L:k}),\,\mathbf{e}(t_{k-L:k+1}),\,\mathbf{p}(\mathbf{x},\mathbf{y},t_{k+1}))\}\quad\forall\, k \in[L,N_T-1]

Step 1: Train an ensemble of NN base models {Mi}i=1N\{\mathcal{M}_i\}_{i=1}^{N}, each initialized with different random weights. Each model Mi\mathcal{M}_i is trained on the same dataset DtrD_{\mathrm{tr}} to learn the mapping from inputs to the next time step prediction.

Inference Phase:

Step 2: Provide the ensemble with initial ground truth fields for the first LL time steps:

p(x,y,t1:L)\mathbf{p}(\mathbf{x},\mathbf{y},t_{1:L})

for k=Lk = L to NT−1N_T - 1 do

    for i=1i = 1 to NN do

        Each model Mi\mathcal{M}_i predicts p(x,y,tk+1)\mathbf{p}(\mathbf{x},\mathbf{y},t_{k+1}) from p(x,y,tk−L+1:k)\mathbf{p}(\mathbf{x},\mathbf{y},t_{k-L+1:k}):

p^i(x,y,tk+1)=Mi(p(x,y,tk−L+1:k))\hat{\mathbf{p}}_i(\mathbf{x},\mathbf{y},t_{k+1}) = \mathcal{M}_i\left(\mathbf{p}(\mathbf{x},\mathbf{y},t_{k-L+1:k})\right)

    end for

    Compute the ensemble-averaged prediction:

p^final(x,y,tk+1)=1N∑i=1Np^i(x,y,tk+1)\hat{\mathbf{p}}_{\mathrm{final}}(\mathbf{x},\mathbf{y},t_{k+1}) = \frac{1}{N}\sum_{i=1}^{N}\hat{\mathbf{p}}_i(\mathbf{x},\mathbf{y},t_{k+1})

    Append p^final(x,y,tk+1)\hat{\mathbf{p}}_{\mathrm{final}}(\mathbf{x},\mathbf{y},t_{k+1}) to the input sequence for the next prediction.

end for

Output: The ensemble-generated spatiotemporal predictions:

{p^final(x,y,tL+1),p^final(x,y,tL+2),…,p^final(x,y,tNT)}\{\hat{\mathbf{p}}_{\mathrm{final}}(\mathbf{x},\mathbf{y},t_{L+1}),\hat{\mathbf{p}}_{\mathrm{final}}(\mathbf{x},\mathbf{y},t_{L+2}),\ldots,\hat{\mathbf{p}}_{\mathrm{final}}(\mathbf{x},\mathbf{y},t_{N_T})\}

explored ensembles in the context of neural operators for PDEs, using independently trained trunk networks to enhance generalization in static (single time-step) predictions [29]. Building on this idea, we extend the deep ensemble paradigm to the dynamic setting, demonstrating its effectiveness for a range of spatiotemporal field evolution problems. Our approach is outlined in Algorithm 1.

To explain the error improvement using the deep ensemble approach, let each model prediction be written as,

Mn(z)=f(z)+ϵn(z),(5)\mathcal{M}_n(z) = f(z) + \epsilon_n(z), \tag*{(5)}

where M(z)\mathcal{M}(z) is the output of the nthn^{th} base model for a given input zz, f(z)f(z) is the true output for zz, and ϵn(z)\epsilon_n(z) is the error by the nthn^{th} base model for the input. The mean squared error for each model (MSEn\mathrm{MSE}_n) is denoted as,

Ez[ϵn(z)2]=Ez[(Mn(z)−f(z))2],(6)\mathbb{E}_z[\epsilon_n(z)^2] = \mathbb{E}_z[(\mathcal{M}_n(z) - f(z))^2], \tag*{(6)}

where Ez(⋅)\mathbb{E}_z(\cdot) is the expectation operator. The MSE of the ensemble can be written as,

MSEens=Ez[(1N∑i=1NMi(z)−f(z))2],(7)\mathrm{MSE}_{ens} = \mathbb{E}_z\left[\left(\frac{1}{N}\sum_{i=1}^{N}\mathcal{M}_i(z) - f(z)\right)^2\right], \tag*{(7)}

This expression can be decomposed into the correlated and uncorrelated components as

MSEens=1N2∑i=1NEz[(Mi(z)−f(z))2]+1N2∑i≠jEz[(Mi(z)−f(z))(Mj(z)−f(z))].(8)\mathrm{MSE}_{ens} = \frac{1}{N^2}\sum_{i=1}^{N}\mathbb{E}_z[(\mathcal{M}_i(z)-f(z))^2] + \frac{1}{N^2}\sum_{i\ne j}\mathbb{E}_z[(\mathcal{M}_i(z)-f(z))(\mathcal{M}_j(z)-f(z))]. \tag*{(8)}

Consider the hypothetical case that all model errors have zero mean (zero bias assumption) and are uncorrelated (independent). Then the ensemble error is only the first term in Eq. (8),

MSEenslower=1N2∑i=1NEz[(Mi(z)−f(z))2]=1N(1N∑i=1NMSEi)(9)\mathrm{MSE}^{lower}_{ens} = \frac{1}{N^2}\sum_{i=1}^{N}\mathbb{E}_z[(\mathcal{M}_i(z)-f(z))^2] = \frac{1}{N}\left(\frac{1}{N}\sum_{i=1}^{N}\mathrm{MSE}_i\right) \tag*{(9)}

which is exactly 1/N1/N times the MSE averaged over all individual models. This is the theoretical lower bound on the ensemble error. Conversely, if all models are perfectly correlated (i.e., errors are identical), the ensemble MSE reduces to the mean of individual MSEs (see [30] for proof):

MSEensupper=1N∑i=1NMSEi(10)\mathrm{MSE}^{upper}_{ens} = \frac{1}{N} \sum_{i=1}^{N} \mathrm{MSE}_{i} \tag*{(10)}

Eqns. (9) and (10) together bound the ensemble MSE. Given that our approach has models with the same architecture and trained on the same data, it is unlikely that the models are uncorrelated. Hence, the total ensemble MSE shall also contain the term due to interaction between models as given in Eq. (8). The diversity between different models (second term in Eq. (8)) determines the relative performance of the ensemble approach. This diversity depends on the training data, individual base models, and the overall optimization process, and analyzing that is beyond the scope of this work. Nevertheless, we demonstrate for a variety of spatiotemporal datasets, that the deep ensemble approach outperforms all the individual base models.

Datasets

In this work, we have demonstrated the deep ensemble approach on three high-dimensional PDEs. The details for all 3 datasets are summarized in Table 1. Each dataset is discussed in detail in the subsequent subsections. Each dataset has different spatiotemporal variations and hence presents a unique set of challenges.

DatasetTrajectoriesTimestepsInput fieldOutput field
2-phase microstructure (stress field: σxx\sigma_{xx})12033σxx\sigma_{xx} history + load pathσxx\sigma_{xx} future time-step
Gray-Scott diffusion (species conc: A)120033A historyA future time-step
Shallow Water Equation (velocity field: ux\mathbf{u}_x)12033ux\mathbf{u}_x historyux\mathbf{u}_x future time-step

Table 1. Details of the spatiotemporal datasets considered in this work

Stress field prediction in a 2-Phase microstructure

The first dataset comprises 2D two-phase microstructures subjected to time-dependent external loading (dynamic load paths), resulting in a spatiotemporally varying stress field. Spatial variation arises from the random placement of elastic fibers within the microstructure, while temporal variation is driven by the evolving external load. This setup, previously studied in [21], was used to validate a UNet-based model for quasi-static prediction of stress and strain under fixed load paths.

In this work, we consider multiple realizations of such microstructures, each containing 20 identical elastic fibers randomly embedded within an elastic perfectly-plastic matrix. The fibers, of constant radius, occupy 40% of the microstructural volume. Each configuration is subjected to uniaxial (u1(t))(u_1(t)) and shear (u2(t))(u_2(t)) displacements on the right boundary, as illustrated in Figure 2. The left boundary is pinned, and the top and bottom edges are traction-free.

2-phase microstructure dataset

Figure 2. 2-phase microstructure dataset: The top row illustrates a representative two-phase microstructure, the applied boundary conditions, and a sample load path generated using Algorithm 2. The bottom row shows the evolution of the stress tensor component (σxx\sigma_{xx}) at selected time steps t=1t = 1, T/3T/3, 2T/32T/3, TT with T=100T = 100 for the load path shown in solid deep blue line.

To simplify, both tensile and shear components follow randomly generated monotonically increasing load paths, generated as described in Algorithm 2. Each load path begins at zero strain, with the final strain sampled from a uniform distribution. A few intermediate time points are sampled to form a strictly increasing sequence, and a cubic interpolator [31] is applied to obtain a smooth monotonic curve. This procedure is repeated to generate a diverse set of load paths, as illustrated in Figure 2.

Algorithm 2 Load Path Generation for 2-Phase Microstructure.

Initialize:

Number of discrete time steps T=100T = 100

Number of control points Nctrl=5N_{\mathrm{ctrl}} = 5

Initialize strain array L∈RTL \in\mathbb{R}^{T} with zeros

Initialize time array X∈RTX \in\mathbb{R}^{T} with zeros

Set L[1]=0L[1] = 0, X[1]=0X[1] = 0

Sample final strain: L[T]∼U(0.07,0.11)L[T] \sim\mathcal{U}(0.07, 0.11)

Set X[T]=TX[T] = T

Sample intermediate control points:

for i=2i = 2 to Nctrl−1N_{\mathrm{ctrl}} - 1 do

Sample L[i]∼U(L[i−1],L[T])L[i] \sim\mathcal{U}(L[i - 1], L[T]) ▷ Monotonic strain

Sample X[i]∼U(X[i−1],X[T])X[i] \sim\mathcal{U}(X[i - 1], X[T]) ▷ Monotonic time

end for

Fit interpolator:

Fit a monotonic cubic interpolator (e.g., Fritsch–Carlson) to {X[1:Nctrl],L[1:Nctrl]}\{X[1:N_{\mathrm{ctrl}}], L[1:N_{\mathrm{ctrl}}]\}

Evaluate interpolator to obtain full strain trajectory L[1:T]L[1:T]

Stress distributions are calculated using the ABAQUS finite element software [32], assuming a J2J_2 plasticity model without hardening, where the yield function is defined as,

f(S)=3J2−σy,(11)f(\mathbf{S}) = \sqrt{3J_2} - \sigma_y, \tag*{(11)}

and J2J_2 is defined as,

J2=12S:S(12)J_2 = \frac{1}{2}\mathbf{S}:\mathbf{S} \tag*{(12)}

where S\mathbf{S} is the deviatoric part of the Cauchy stress tensor and σy=210\sigma_y = 210 MPa is the yield strength of the material. The Young’s modulus for the matrix and fiber are set to 3.2 GPa and 87 GPa, respectively. The Poisson’s ratio for the matrix and fiber are set to 0.35 and 0.20, respectively. Additionally, the model assumes small-strain conditions, which allows the additive decomposition of the strain tensor into elastic and plastic components as shown in Eq. (13).

ε=εe+εp(13)\varepsilon= \varepsilon_e + \varepsilon_p \tag*{(13)}

The simulations use an irregular mesh with 86000 elements on average to resolve the smaller regions between two fibers. The full dataset can be found at [33]. A mesh convergence study of this finite element model can be found in [21]. After the simulation, the stress fields obtained are interpolated on a regular square grid (Nx×Ny=128×128N_x \times N_y = 128 \times128) for implementation in the ML model. For model training, the first time instant (t=0t = 0) corresponding to no external strain is discarded and then we subsample the load path by 3 (i.e., NT=33N_T = 33).

One example of a generated load path and its corresponding stress field simulation (with dimensions Nx×Ny×NTN_x \times N_y \times N_T) is illustrated in Figure 2, where intermediate time steps are uniformly spaced. The time axis is normalized, i.e., each load path is represented by TT discrete values between 0 and 1, which are then used as input to the simulation model. The solid blue curve indicates the specific load path for which the spatial stress field evolution is visualized. Stress maps at selected time instances t=1t = 1, T/3T/3, 2T/32T/3, TT, marked by red circles, are shown alongside the load path. For training and testing the model, the stress field is subsampled in time by a factor of 3, resulting in Nt=33N_t = 33 total time steps. Obtaining full-field stress data from ABAQUS for a single time-step requires 1.6s, while the inference time for the deep ensemble approach is 9x faster. In this work, we focus on predicting the evolution of a single stress component (σxx\sigma_{xx}); however, the framework can be easily extended to include other stress tensor components.

Gray-Scott reaction diffusion

The Gray-Scott reaction diffusion model [34] describes systems that are commonly found in many biological and chemical species where chemical reactions interact with a spatial diffusion process. Let us consider two chemical species AA and BB interacting with each other in a closed domain. The Gray-Scott reaction diffusion model describes the variation of chemical concentration of both species over space and time as:

Gray-Scott reaction diffusion dataset: Species A concentration

Figure 3. Gray-Scott reaction diffusion dataset: Species A concentration shown for a random initial condition and Gray-Scott parameters F=0.098F = 0.098, K=0.057K = 0.057 at time steps t=1t = 1, T/3T/3, 2T/32T/3, TT, with T=33T = 33 on the simulation trajectory.

∂A∂t=δAΔA−AB2+f(1−A),(14)\frac{\partial A}{\partial t} = \delta_A \Delta A - AB^2 + f(1-A), \tag*{(14)}
∂B∂t=δBΔB+AB2−(f+k)B.(15)\frac{\partial B}{\partial t} = \delta_B \Delta B + AB^2 - (f+k)B. \tag*{(15)}

In the above system, ff and kk are the feed and kill parameters, respectively. Specifically, ff controls the rate at which species AA is added to the system, and kk controls the rate at which species BB is removed from the system. δA\delta_A and δB\delta_B control the diffusion rate of each species and are set to 2×10−52 \times10^{-5} and 1×10−51 \times10^{-5}, respectively, for the simulations.

The simulation data is obtained from [35]. Eqs. (14) - (15) are simulated with a higher order spectral difference method on the spatial domain [−1,−1]×[1,1][-1, -1] \times[1, 1] with periodic boundary conditions and Δt=10s\Delta t = 10s. At each time step, the simulation is represented as a regular grid of size 128×128128 \times128, with 200 initial conditions (random Gaussian clusters), and 6 different combinations of ff and kk parameters are considered for the simulations. This leads to a diverse dataset (200×6=1200200 \times6 = 1200 unique patterns). The details of the 6 combinations of ff and kk parameters are detailed in Table. Table 2. We demonstrate our approach for predicting the evolution of the species A chemical concentration for the first Nt=33N_t = 33 simulation timesteps.

FKPattern
0.0140.054Gliders
0.0980.057Bubbles
0.0290.057Maze
0.030.062Spots
0.0580.065Worms
0.0180.051Spirals

Table 2. Parameter combinations considered for the dataset generation of the Gray-Scott reaction diffusion system.

Shallow Water Equation

The shallow water equations are a 2D approximation of a 3D incompressible fluid flow when horizontal length scales are much larger than vertical length scales. They are obtained by numerically depth integrating the incompressible Navier-Stokes equation. These equations are crucial to validate atmospheric dynamics [36]. Specifically, we consider the rotating forced hyperviscous spherical shallow water equation,

∂u∂t=−u⋅∇u−g∇h−ν∇4u−2Ω×u,(16)\frac{\partial\mathbf{u}}{\partial t} = -\mathbf{u} \cdot\nabla\mathbf{u} - g\nabla h - \nu\nabla^{4}\mathbf{u} - 2\Omega\times\mathbf{u}, \tag*{(16)}
∂h∂t=−H∇⋅u−∇⋅(hu)−ν∇4h+F,(17)\frac{\partial h}{\partial t} = -H\nabla\cdot\mathbf{u} - \nabla\cdot(h\mathbf{u}) - \nu\nabla^{4}h + F, \tag*{(17)}

where u\mathbf{u} is the velocity vector field, hh is the deviation of pressure surface height from the mean, HH is the mean height, Ω\Omega is the Coriolis factor, and FF is the forcing term to induce double periodicity (daily and annual). The ∇4\nabla^{4} term corresponds to the hyperviscosity term. Spherical boundary conditions and periodic forcing term are adopted in this simulation to create a simpler proxy for real-world atmospheric dynamics. The initial conditions are sampled from hPa 500 level in ERA5 [37].

The dataset used in this work has been previously introduced in an atmospherical dynamics prediction study using Fourier Neural Operator (FNO) [38] and a comprehensive dataset study [35]. The dataset consists of 120 trajectories with Δt=1\Delta t = 1 hour in simulation time. The data is simulated on an equiangular grid with ϕ∈[0,2π]\phi\in[0, 2\pi] and θ∈[0,π]\theta\in[0, \pi]. At each time-step, the high-dimensional simulation is represented as a 256×512256 \times512 2D image corresponding to the equiangular spatial domain. We demonstrate our approach for the evolution of the velocity vector field component uxu_x for the first Nt=33N_t = 33 simulation timesteps.

Shallow-water equation results at four time instants with color scale

Figure 4. Shallow-water equation results (Shallow Water equation dataset): The velocity field component uxu_x is visualized at four key time instants along the simulation trajectory - t={1,T3,2T3,T}t = \{1, \frac{T}{3}, \frac{2T}{3}, T\}, with T=33T = 33. These snapshots capture the spatiotemporal evolution of the flow dynamics throughout the simulation.

Temporal information processing

As explained in § 4, the base models in the deep ensemble are conditioned on time via an embedding of the temporal information. For each dataset, the temporal input has two parts: time-IDs and the external forcing input. For the 2-phase microstructure dataset, the temporal input is a 1D vector of size 2(L+1)2(L + 1), consisting of the external forcing input values e(tk−(L−1):k+1)e(t_{k-(L-1):k+1}) - that is, the load values over the previous LL time steps and the next time step, along with the corresponding discrete time IDs (tk−(L−1):k+1)(t_{k-(L-1):k+1}), as illustrated in Fig. 1.

For the Gray-Scott reaction-diffusion dataset, the temporal input is a 1D vector of size (L+1)+2(L + 1) + 2, comprising the previous LL time IDs plus the next one and the external forcing input (comprising of parameters ff and kk values from Eqs. (14)–(15)). Note that while ff and kk are constant over time, their combination with different initial conditions leads to a distinct set of patterns.

For the Shallow Water equation dataset, the temporal input is a 1D vector of size (L+1)(L + 1), representing the previous LL and the next time IDs. There is no external forcing input in this case, as the only source of variability is due to differences in initial conditions.

The final temporal embedding u(t)u(t) is defined as:

u(t)=FNNemb(t)∥FNNvar(e(t)),(18)u(t) = \mathrm{FNN}_{\mathrm{emb}}(t) \mathbin{\|} \mathrm{FNN}_{\mathrm{var}}(e(t)), \tag*{(18)}

where FNNemb(t)\mathrm{FNN}_{\mathrm{emb}}(t) is the time-ID embedding, FNNvar(e(t))\mathrm{FNN}_{\mathrm{var}}(e(t)) is the forcing input embedding, and ∥\| denotes vector concatenation.

The component FNNemb(t)\mathrm{FNN}_{\mathrm{emb}}(t) encodes time IDs using a sinusoidal positional embedding - used in NLP to represent token positions-followed by two fully connected layers with Gaussian Error Linear Unit (GeLU) activation. The resulting output is a 1D vector of size 1×321 \times32 (output dimension obtained after hyperparameter tuning). The sinusoidal embedding is computed as:

si(t)=sin⁡(t10000i/dt)∥cos⁡(t10000i/dt),for i=1,…,dt,(19)s_i(t) = \sin\left(\frac{t}{10000^{i/d_t}}\right) \mathbin{\|} \cos\left(\frac{t}{10000^{i/d_t}}\right), \quad\text{for } i = 1, \ldots, d_t, \tag*{(19)}

where dtd_t is the embedding dimension.

The external forcing input e(t)e(t) is embedded using FNNvar(e(t))\mathrm{FNN}_{\mathrm{var}}(e(t)), which consists of two linear layers with a GeLU activation in between. The output is also a 1D vector of size 1×321 \times32 (output dimension obtained after hyperparameter tuning). As described in Sec. 2, the complete temporal embedding u(t)u(t) is injected into each intermediate layer of the UNet. This is achieved by reshaping u(t)u(t) and passing it through a 2D convolutional layer, ensuring the number of filters matches the channel size of the intermediate UNet output. For the Shallow Water equation dataset, which lacks time-varying external inputs, the temporal embedding simplifies to only the time-ID embedding (FNNemb(t)\mathrm{FNN}_{\mathrm{emb}}(t)).

Deep Ensemble training

For the 2-phase microstructure dataset, the time-varying load paths differ between the training and testing datasets. In contrast, for the Gray-Scott and Shallow Water equation datasets, the variability arises from different initial conditions in training and testing data. The training datasets are standardized for faster convergence during training. The test data are standardized with the training data statistics.

During training, we use a batch size of 8. At test time, predictions are generated in an autoregressive manner to reconstruct the full trajectory. Accordingly, the test-time batch size is set to Nt−LN_t - L, where NtN_t is the trajectory length for the given dataset and LL is the temporal history length. We set L=3L = 3 (number of previous time-steps to be used as model input) for all datasets. In general, we recommend choosing 3≤L≤103 \le L \le10 to balance the trade-off between predictive accuracy and the amount of ground truth required at test time.

To mitigate overfitting, we apply L2L_2-norm regularization to the model weights during training. All base models are trained for 100 epochs using the Adam optimizer with a learning rate of 0.0001 and default hyperparameters. Our primary goal is not to optimize performance for each individual dataset, but rather to demonstrate the effectiveness of the ensemble approach compared to standalone models. Training for 100 epochs is sufficient to ensure convergence to a stable optimum.

It is noted that for a fair comparison of numerical solvers with data-driven surrogate models, the model training time should also be accounted for. Since the deep ensemble models are randomly initialized and do not share any parameters, they can be trained/used in parallel on multiple GPUs. In this work, the models are trained on a GPU cluster node (each node has 4 Nvidia A100 GPUs with 40GB of memory each). For the 3 datasets, the average training time on GPU is ≈4.5\approx4.5 hours for one model. After training, the models can be used at inference time for every new test sample without any retraining.

Results

The performance of the deep ensemble approach for all 3 datasets is compared qualitatively by visualizing the physical field predictions at select timesteps on the simulation trajectory. A quantitative comparison of the deep ensemble approach and individual models is presented next. The relative L2\mathcal{L}^2 norm error (RLE) is defined as,

RLE⁡(p(x,y,t),p^(x,y,t))=1Ntest∑i=1Ntest∥pi(x,y,t)−p^i(x,y,t)∥2∥pi(x,y,t)∥2(20)\operatorname{RLE}(\mathbf{p}(\mathbf{x},\mathbf{y},t), \widehat{\mathbf{p}}(\mathbf{x},\mathbf{y},t)) = \frac{1}{N_{\mathrm{test}}} \sum_{i=1}^{N_{\mathrm{test}}} \frac{\lVert\mathbf{p}_i(\mathbf{x},\mathbf{y},t) - \widehat{\mathbf{p}}_i(\mathbf{x},\mathbf{y},t) \rVert_2}{\lVert\mathbf{p}_i(\mathbf{x},\mathbf{y},t) \rVert_2} \tag*{(20)}

The mean absolute error (MAE) is defined as,

MAE⁡(p(x,y,t),p^(x,y,t))=1Ntest∑i=1Ntest1Nx∗Ny∑x,y∣pi(x,y,t)−p^i(x,y,t)∣(21)\operatorname{MAE}(\mathbf{p}(\mathbf{x},\mathbf{y},t), \widehat{\mathbf{p}}(\mathbf{x},\mathbf{y},t)) = \frac{1}{N_{\mathrm{test}}} \sum_{i=1}^{N_{\mathrm{test}}} \frac{1}{N_x * N_y} \sum_{x,y} \left| \mathbf{p}_i(\mathbf{x},\mathbf{y},t) - \widehat{\mathbf{p}}_i(\mathbf{x},\mathbf{y},t) \right| \tag*{(21)}

where NtestN_{\mathrm{test}} is the size of the test dataset. Both metrics are computed separately at each time step tt since we are interested in evaluating the performance of the approaches over time.

2-phase microstructure

Figure 5 shows the predictions with the deep ensemble approach at selected time-steps along the simulation trajectory. There is a remarkable similarity between the ground truth simulations (reference finite element solution) and the ensemble predictions. The absolute error is low except at a few points on the fiber-matrix interfaces.

2-phase microstructure stress field prediction with the deep ensemble, reference FEM solution, and absolute error map

Figure 5. 2-phase microstructure stress field prediction with the deep ensemble (top), reference FEM solution (middle row), and absolute error map (bottom) at t={L+1,T4,T2,T}t = \{L + 1, \frac{T}{4}, \frac{T}{2}, T\} time steps of the simulation trajectory.

The performance metrics with the deep ensemble approach are illustrated in Figure 6. As illustrated in Figure 6(a), at the end of the simulation trajectory, RLE metric with the deep ensemble approach is 0.0987, which is 18 to 28% less than the RLE range with individual models (0.120 to 0.137). Similarly, Figure 6(b) demonstrates that at the end of the simulation trajectory, the MAE metric is at least 20% less than the best-performing individual model (Model 2 in this case). Both comparisons show that the deep ensemble approach helps to mitigate error accumulation in an autoregressive setting.

Performance metrics for the 2-phase microstructure dataset

Figure 6. Performance metrics for the 2-phase microstructure dataset (a). RLE and (b). MAE.

Gray-Scott reaction diffusion

The results for the Gray-Scott diffusion reaction dataset, illustrated in (Figure 7), show more significant errors in the prediction. Specifically, a few of the patterns are mispredicted by the data-driven model at later timesteps (t=25t=25 onwards), with the data-driven model predicting regions of very low species concentration that are not present in the ground-truth data. The deep ensemble approach improves the model predictions, with the RLE at the end of the trajectory being 0.085, which is 5 to 30% less than the RLE metric range with individual models (0.095 to 0.12), see Figure 8(a). Similarly, the MAE metric with the deep ensemble approach is at least 14% less compared to an individual model (best performing Model-3), as illustrated in Figure 8(b).

Gray-Scott reaction species concentration with the deep ensemble

Figure 7. Gray-Scott reaction species concentration with the deep ensemble (top row), reference analytical solution (middle row), and absolute error map (bottom row) at t={L+1,T4,T2,3T4,T}t = \{L + 1, \frac{T}{4}, \frac{T}{2}, \frac{3T}{4}, T\} time steps of the simulation trajectory.

Performance metrics for the Gray-Scott diffusion reaction dataset

Figure 8. Performance metrics for the Gray-Scott diffusion reaction dataset (a). RLE and (b). MAE.

Shallow water equation

Figure 9 shows the predictions with the deep ensemble approach for the Shallow Water equation dataset. This illustration shows higher error in regions with complicated patterns (top center of the spatial dimensions). Towards the end of the simulation trajectory, the data-driven model is unable to capture all the patterns in the data. Indeed, researchers have demonstrated the inability of data-driven surrogate models to capture high-frequency patterns in physical systems [39] (spectral bias). Note that it is inherently impossible for a data-driven surrogate model to capture all the subtleties of the physical system. The objective of this work is to showcase the advantage of using the deep ensemble instead of individual models.

Shallow Water equation velocity field prediction with the deep ensemble, reference FEM solution, and absolute error map

Figure 9. Shallow Water equation velocity field prediction with the deep ensemble (top row), reference FEM solution (middle row), and absolute error map (bottom row) at t={L+1,T4,T2,T}t = \{L + 1, \frac{T}{4}, \frac{T}{2}, T\} time steps of the simulation trajectory with L=3L = 3, T=33T = 33 being the time history input to the model and total simulation steps respectively.

Due to the complicated nature of the dataset, we expect the error metrics to be higher. As illustrated in Figure 10(a), with the deep ensemble approach, the RLE at the end of the simulation trajectory is 0.73, which is 10 to 30% less than the RLE range with individual models (0.81 to 1.018). The MAE metric with the deep ensemble approach is at least 15% less compared to the individual models (best performing model 4), as illustrated in Figure 10(b).

Performance metrics for the Shallow Water equation dataset

Figure 10. Performance metrics for the Shallow Water equation dataset (a). RLE and (b). MAE.

Analysis of the deep ensemble framework

Analysis of the deep ensemble framework

As noted in § 4.1, the ensemble error decreases as the correlation between individual model errors decreases. Given that all the base models share the same architecture and parameters, the difference in performance can be attributed to the complexity of the individual datasets. This likely gives rise to different degrees of independence between individual models. Figure 11 compares the best and worst performing models v/s the deep ensemble approach for all three datasets (in terms of MAE metric). On average over all 3 datasets, the deep ensemble approach has a 33% lesser MAE compared to the worst-performing individual model and 15% lesser MAE compared to the best-performing model. This comparison underscores the observation that the deep ensemble approach always outperforms the best individual model and, on average, provides substantial improvement in performance.

Performance comparison of the deep ensemble approach and model with highest MAE and lowest MAE

Figure 11. Performance comparison of the deep ensemble approach and model with highest MAE and lowest MAE respectively for: 2-phase microstructure dataset (a, d); Gray Scott diffusion reaction dataset (b, e); Shallow Water equation dataset (c, f).

Clearly, the results using the deep ensemble approach will improve on average, as the number of base models in the ensemble is increased, but the requisite computational effort will increase. To investigate this effect, Figure 12 illustrates the performance metrics for a varying number of base models in the deep ensemble for the Gray-Scott reaction diffusion dataset. We observe that an increasing number of base models leads to improved performance (increasing the number of models from 2 to 8 improves the MAE by 28%). However, after a saturation point, further increases in the number of models (8 to 16) do not lead to substantial improvement in performance. The number of base models in the deep ensemble has to be chosen based on a tradeoff between improvement in performance and time and memory requirement to train and store all base models respectively.

Performance comparison plots for different numbers of base models in the ensemble approach, showing RLE and MAE

Figure 12. Performance comparison for different numbers of base models in the ensemble approach for Gray-Scott diffusion reaction dataset: (a). RLE and (b). MAE.

Conclusions

Traditional numerical solvers for time-dependent PDEs provide high-fidelity results but are often too computationally intensive for real-time or many-query applications. While ML surrogates offer a scalable alternative, autoregressive implementations are prone to error accumulation over long horizons—compromising their reliability in practical deployments.

In this work, we demonstrate that a simple ensemble strategy—based solely on random weight initialization—can significantly reduce long-term prediction error in autoregressive ML surrogates. By training multiple models on the same data with different initializations and averaging their predictions, we effectively mitigate the compounding errors typical of single-model rollouts. Across three diverse PDE systems—elastic stress evolution, shallow water dynamics, and reaction-diffusion chemistry—our ensemble approach improves predictive stability by 15–33%, without requiring any architectural modifications, memory modules, or retraining overhead. The effectiveness of our method stems from its ability to harness the stochasticity of neural network training. Each model in the ensemble converges to a different local minimum, and their aggregated predictions offer improved generalization and robustness to initialization-induced bias. The result is a lightweight, architecture-agnostic enhancement that retains the computational speed of ML surrogates, making it particularly attractive for real-time simulation in scientific and engineering workflows.

Looking forward, our ensembling framework opens up several directions for future work. One avenue is to integrate physics-informed constraints or inductive biases to further regularize ensemble outputs. Another promising direction is to explore ensemble diversity not just through random initialization but also via controlled variations in hyperparameters, architectures, or data augmentation—pushing the method toward more adaptive and robust surrogate modeling.

Acknowledgements

I.K., S.G., and L.G.B. are supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, grant under Award Number DE-SC0024162. I.S. and L.G.B were supported by the Army Research Laboratory under Cooperative Agreement Number W911NF-12-2-0023, W911NF-12-2-0022, and W911NF-22-2-0014. The computational experiments were carried out at the Advanced Research Computing at Hopkins (ARCH) core facility (rockfish.jhu.edu) hosted by Johns Hopkins University, which is supported by the National Science Foundation (NSF) grant number OAC 1920103. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes, notwithstanding any copyright notation herein.

References

References

  1. [1]L. Lu, P. Jin, G. Pang, Z. Zhang, G. E. Karniadakis, Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, Nature machine intelligence 3 (2021) 218–229.
  2. [2]Z. Li, N. B. Kovachki, K. Azizzadenesheli, K. Bhattacharya, A. Stuart, A. Anandkumar, et al., Fourier Neural Operator for Parametric Partial Differential Equations, in: International Conference on Learning Representations, 2021.
  3. [3]S. Goswami, A. D. Jagtap, H. Babaee, B. T. Susi, G. E. Karniadakis, Learning stiff chemical kinetics using extended deep neural operators, Computer Methods in Applied Mechanics and Engineering 419 (2024) 116674.
  4. [4]W. Diab, M. Al-Kobaisi, Temporal Neural Operator for Modeling Time-Dependent Physical Phenomena, arXiv preprint arXiv:2504.20249 (2025).arxiv.org/abs/2504.20249
  5. [5]H. Robinson, S. Pawar, A. Rasheed, O. San, Physics guided neural networks for modelling of non-linear dynamics, Neural Networks 154 (2022) 333–345.
  6. [6]D. Nayak, S. Goswami, TI-DeepONet: Learnable Time Integration for Stable Long-Term Extrapolation, arXiv preprint arXiv:2505.17341 (2025).arxiv.org/abs/2505.17341
  7. [7]K. Michałowska, S. Goswami, G. E. Karniadakis, S. Riemer-Sørensen, Neural operator learning for long-time integration in dynamical systems with recurrent neural networks, in: 2024 International Joint Conference on Neural Networks (IJCNN), IEEE, 2024, pp. 1–8.
  8. [8]J. He, S. Kushwaha, J. Park, S. Koric, D. Abueidda, I. Jasiuk, Sequential deep operator networks (S-deeponet) for predicting full-field solutions under time-dependent loads, Engineering Applications of Artificial Intelligence 127 (2024) 107258.DOI
  9. [9]I. Sutskever, J. Martens, G. Dahl, G. Hinton, On the importance of initialization and momentum in deep learning, in: International conference on machine learning, PMLR, 2013, pp. 1139–1147.
  10. [10]J. Huang, H.-T. Yau, Dynamics of deep neural networks and neural tangent hierarchy, in: International conference on machine learning, PMLR, 2020, pp. 4542–4551.arxiv.org/abs/1909.08156
  11. [11]L. Breiman, Random forests, Machine learning 45 (2001) 5–32.
  12. [12]R. A. Jacobs, M. I. Jordan, S. J. Nowlan, G. E. Hinton, Adaptive mixtures of local experts, Neural computation 3 (1991) 79–87.DOI
  13. [13]B. Lakshminarayanan, A. Pritzel, C. Blundell, Simple and scalable predictive uncertainty estimation using deep ensembles, Advances in neural information processing systems 30 (2017).arxiv.org/abs/1612.01474
  14. [14]Z. Zou, Z. Wang, G. E. Karniadakis, Learning and discovering multiple solutions using physics-informed neural networks with random initialization and deep ensemble, arXiv preprint arXiv:2503.06320 (2025).arxiv.org/abs/2503.06320
  15. [15]I. D. Khurjekar, J. B. Harley, Sim-to-real localization: Environment resilient deep ensemble learning for guided wave damage localization, The Journal of the Acoustical Society of America 151 (2022) 1325–1336.DOI
  16. [16]T. Tripura, S. Chakraborty, A foundational neural operator that continuously learns without forgetting, arXiv preprint arXiv:2310.18885 (2023).arxiv.org/abs/2310.18885
  17. [17]Z. Li, D. Shu, A. Barati Farimani, Scalable transformer for PDE surrogate modeling, Advances in Neural Information Processing Systems 36 (2023) 28010–28039.arxiv.org/abs/2305.17560
  18. [18]A. Hemmasian, A. B. Farimani, Multi-scale time-stepping of partial differential equations with transformers, Computer Methods in Applied Mechanics and Engineering 426 (2024) 116983.DOI
  19. [19]H. Wu, H. Luo, H. Wang, J. Wang, M. Long, Transolver: A fast transformer solver for pdes on general geometries, arXiv preprint arXiv:2402.02366 (2024).arxiv.org/abs/2402.02366
  20. [20]N. Geneva, N. Zabaras, Transformers for modeling physical systems, Neural Networks 146 (2022) 272–289.
  21. [21]I. Saha, A. Gupta, L. Graham-Brady, Prediction of local elasto-plastic stress and strain fields in a two-phase composite microstructure using a deep convolutional neural network, Computer Methods in Applied Mechanics and Engineering 421 (2024) 116816.DOI
  22. [22]K. Michalowska, S. Goswami, G. E. Karniadakis, S. Riemer-Sørensen, DON-LSTM: Multi-resolution learning with DeepONets and long short-term memory neural networks, arXiv preprint arXiv:2310.02491 (2023).
  23. [23]V. Oommen, K. Shukla, S. Goswami, R. Dingreville, G. E. Karniadakis, Learning two-phase microstructure evolution using neural operators and autoencoder architectures, npj Computational Materials 8 (2022) 190.arxiv.org/abs/2204.07230
  24. [24]Q. Cao, S. Goswami, T. Tripura, S. Chakraborty, G. E. Karniadakis, Deep neural operators can predict the real-time response of floating offshore structures under irregular waves, Computers & Structures 291 (2024) 107228.arxiv.org/abs/2302.06667
  25. [25]O. Ovadia, V. Oommen, A. Kahana, A. Peyvan, E. Turkel, G. E. Karniadakis, Real-time inference and extrapolation via a diffusion-inspired temporal transformer operator (ditto), arXiv preprint arXiv:2307.09072 (2023).arxiv.org/abs/2307.09072
  26. [26]K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.arxiv.org/abs/1512.03385
  27. [27]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017).arxiv.org/abs/1706.03762
  28. [28]S. Fort, H. Hu, B. Lakshminarayanan, Deep ensembles: A loss landscape perspective, arXiv preprint arXiv:1912.02757 (2019).arxiv.org/abs/1912.02757
  29. [29]R. Sharma, V. Shankar, Ensemble and mixture-of-experts deeponets for operator learning, arXiv preprint arXiv:2405.11907 (2024).arxiv.org/abs/2405.11907
  30. [30]G. Brown, Ensemble Learning, Springer US, Boston, MA, 2010, pp. 312–320. URL: https://doi.org/10.1007/978-0-387-30164-8_252. doi:10.1007/978-0-387-30164-8_252.
  31. [31]F. N. Fritsch, J. Butland, A method for constructing local monotone piecewise cubic interpolants, SIAM journal on scientific and statistical computing 5 (1984) 300–304.DOI
  32. [32]E. J. Barbero, Finite element analysis of composite materials using Abaqus®, CRC press, 2023.DOI
  33. [33]I. D. Khurjekar, I. Saha, A. Gupta, L. Graham-Brady, S. Goswami, Data and code associated with the publication: Full field stress response in 2-phase microstructures under dynamic loading, 2025. URL: https://doi.org/10.7281/T1X5ZYPI. doi:10.7281/T1X5ZYPI.
  34. [34]P. Gray, S. K. Scott, Autocatalytic reactions in the isothermal, continuous stirred tank reactor: Oscillations and instabilities in the system a+2b→ 3b; b→ c, Chemical Engineering Science 39 (1984) 1087–1097.
  35. [35]R. Ohana, M. McCabe, L. Meyer, R. Morel, F. Agocs, M. Beneitez, M. Berger, B. Burkhart, S. Dalziel, D. Fielding, et al., The well: a large-scale collection of diverse physics simulations for machine learning, Advances in Neural Information Processing Systems 37 (2025) 44989–45037.arxiv.org/abs/2412.00568
  36. [36]D. L. Williamson, J. B. Drake, J. J. Hack, R. Jakob, P. N. Swarztrauber, A standard test set for numerical approximations to the shallow water equations in spherical geometry, Journal of computational physics 102 (1992) 211–224.
  37. [37]H. Hersbach, B. Bell, P. Berrisford, S. Hirahara, A. Horányi, J. Muñoz-Sabater, J. Nicolas, C. Peubey, R. Radu, D. Schepers, et al., The era5 global reanalysis, Quarterly journal of the royal meteorological society 146 (2020) 1999–2049.
  38. [38]M. McCabe, P. Harrington, S. Subramanian, J. Brown, Towards stability of autoregressive neural operators, arXiv preprint arXiv:2306.10619 (2023).arxiv.org/abs/2306.10619
  39. [39]S. Wang, X. Yu, P. Perdikarıs, When and why pinns fail to train: A neural tangent kernel perspective, Journal of Computational Physics 449 (2022) 110768.arxiv.org/abs/2007.14527

Paper details

vlnc:W_ztmqxkk3
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents