betaFeedback

WorksComputer Science

Benchmarking Multi-fidelity Neural Operators on Complex PDE Problems with Non-trivial Fidelity Differences

Abstract

Solving PDE-governed physical problems is computationally expensive, limiting the availability of high-fidelity (HF) data for training neural operators, which typically require large datasets. Multi-fidelity learning addresses this by combining abundant low-fidelity (LF) data with limited HF samples. To evaluate several multi-fidelity neural operator strategies, including two-step, residual, intermediate, and transfer learning, we conduct a systematic assessment across four PDE test cases: two standard grid-based problems, a modified Darcy flow with LF-HF governing equation discrepancies, and an unsteady smoke inflow with temporal dynamics. Most prior studies focus on grid-based LF-HF differences, but real-world discrepancies, such as those between RANS and LES models in CFD, can be more complex. To capture these more challenging scenarios, we introduce two new test cases specifically designed to emulate nontrivial LF-HF mismatch beyond simple grid-resolution differences, enabling a more realistic assessment of multi-fidelity strategies under substantial model discrepancies. We find that methods directly feeding LF predictions into HF models are prone to error amplification under large LF-HF discrepancies, particularly in autoregressive settings. In contrast, transfer learning, which uses LF-trained weights to initialise the HF model, consistently improves accuracy by providing a robust prior refined with HF data. These results highlight the importance of establishing transfer learning as a reliable approach for efficient multi-fidelity PDE surrogate modelling.

Highlights

On Multifidelity Neural Operator for Data-Driven PDE Predictions

Ghifari Adam Faza, Kemas Zakaria, Pramudita Satria Palar, Keivan Shariatmadar, Lavi Rizki Zuhal, Hans Hallez, David Moens

  • Training a machine learning-based PDE solver, such as a neural operator (NO), requires a considerable amount of high-quality and high-fidelity data to generalise the solution accurately. However, access to high-fidelity data is costly and often limited. Thus, a multi-fidelity learning strategy offers a solution to address this problem.

  • There are multiple strategies and approaches for constructing a multi-fidelity learning model for a neural network-based model. This paper compares and evaluates different approaches to enable multi-fidelity learning in the neural operator model.

  • We observed that many recent studies assume that multi-fidelity data arise primarily from differences in discretisation resolution. While this is a valid and practical scenario, the resulting gap between high-fidelity and low-fidelity data is often relatively small. Consequently, we argue that the impact of incorporating lower-fidelity data in such cases may be limited and less representative of broader multi-fidelity learning challenges.

  • In this paper, we propose a modification of the current multi-fidelity test case baseline to resemble real-world cases. We also propose a more complex test case which includes a time-dependent component.

Introduction

Modelling engineering and science problems often involves solving complex partial differential equations (PDE) systems, or through experiments. However, the computational complexity and resource-intensive nature of these methods frequently result in prohibitively expensive single-configuration analyses. Such constraints severely limit the number of samples that can be generated, creating significant challenges for critical downstream tasks, including reliability assessment, design optimisation, and inverse engineering problems. Thus, faster alternatives are needed to address this problem. While hardware acceleration has improved the efficiency of numerical PDE solvers [1, 2], data-driven methods are gaining popularity as a viable alternative. These surrogate models, such as support vector regression (SVR) [3], Gaussian processes (GP) [4], and neural networks (NN) [5, 6], learn input-output relationships from existing data, providing efficient and scalable solutions.

As scientific problems grow increasingly complex, such as accurately modelling full-field PDE solutions, the need for models capable of capturing intricate spatial and temporal patterns has grown significantly. Traditional methods such as SVR and Gaussian processes GP, often paired with reduced-order modelling techniques like proper orthogonal decomposition (POD) [7], have been employed to tackle such challenges. However, recently neural networks have emerged as a powerful and flexible alternative, owing to their high expressive capacity, advances in computational resources, scalable architectures, and access to large datasets. These qualities make them especially suitable for complex predictive tasks in scientific machine learning. Applications of neural networks in this domain include solving inverse problems governed by PDEs [8, 9, 10], data-driven discovery [11, 12], and design optimisation [13, 14]. The great flexibility of NN architecture is the dominant factor that drives the superiority of neural networks. For example, classical artificial neural networks (ANNs), or multi-layer perceptrons (MLPs) [15, 16], use stacked layers of perceptrons with linear transformations and nonlinear activations. When dealing with complex data types such as images, the standard nonlinear layer is then replaced by convolutional layers [17], allowing the model to more effectively capture local spatial structures.

Neural networks have long been used as surrogate models, but the introduction of physics-informed neural networks (PINNs) [18] marked a significant shift in this approach. Unlike standard neural networks, PINNs incorporate external information, such as physical laws, directly into the loss function. This integration allows the model to learn more efficiently and accurately. Notably, while PINNs modify the loss function, they do not alter the network’s architecture or layer structure. Deep operator network (DeepONet) [19], on the other hand, learns nonlinear continuous operators via two sub-networks, namely, trunk-net and branch-net. The former is used to encode the location of the output functions, and the latter is used to encode the input function at a fixed number of sensors/discretisation. More recently, the neural operator (NO) [20] has emerged as a resolution-invariant alternative to DeepONet. At its core, NO leverages an integral operator layer, enabling it to learn mappings between infinite-dimensional function spaces, offering greater flexibility and generalisation across different resolutions.

Advancements in neural network-based models have significantly improved their ability to capture complex physical phenomena. However, these models require large volumes of high-quality training data to achieve accurate learning, leading to substantial computational and financial costs for dataset generation. Addressing this challenge necessitates a more efficient approach. One way to mitigate computational expenses is by using coarser discretisation grids or simplified governing equations to generate lower-fidelity solutions. However, these lower-fidelity simulations often lack accuracy and fail to capture detailed phenomena that their high-fidelity counterparts are able to resolve, such as small-scale turbulence in CFD simulations. Despite this limitation, lower-fidelity solutions generally preserve the overall structure of the high-fidelity solutions. Given the ease of generating large quantities of lower-fidelity data and the constraints on producing high-fidelity datasets, a natural strategy is to use lower-fidelity data to learn the general structure of PDE solutions while leveraging limited high-fidelity data to refine finer details. This approach, known as multi-fidelity learning, enables models to achieve greater predictive accuracy than single-fidelity models trained with only a small amount of high-fidelity data.

The concept of multi-fidelity learning is well-established in other machine learning techniques, such as Gaussian process regression [21, 22] (co-Kriging) and polynomial chaos expansion [23] with applications in design optimisation [24, 25]. In the context of neural-network-based models, several approaches such as intermediate output architecture [26, 27], multi-step architectures [27], and transfer learning method [28] have been proposed to tackle the multi-fidelity learning problems. Multi-fidelity learning has effectively been utilised in neural operators, such as multi-fidelity neural operators with transfer learning [29, 30], and multi-fidelity wavelet neural operators with residual value architecture [31]. These studies have successfully showcased the benefits of multi-fidelity learning across various test cases. However, in all the multi-fidelity neural operator test cases we found [29, 32, 30, 31], fidelity variations are introduced solely through differences in grid resolution, where lower-fidelity solutions resemble higher-fidelity ones but at a coarser scale. While such cases are valid examples of multi-fidelity learning, they do not capture variations in the governing equations. For instance, distinct flow separation characteristics around an airfoil at high angles of attack [33], as modelled by Reynolds-averaged Navier-Stokes (RANS) and detached eddy simulation (LES), cannot be replicated merely by adjusting grid resolution. This highlights the need for multi-fidelity approaches that account for fundamental differences in the underlying physics.

In this study, we assess various architectures and methodologies for multi-fidelity learning in neural operators. First, we perform a comparative and ablation study on different architectures using benchmark test cases, including the one-dimensional stochastic Poisson equation and the two-dimensional Darcy flow at varying resolutions. Then, to resemble multi-fidelity in terms of governing equations, we propose a modification on the existing 2-dimensional Darcy flow and perform a comparative study on different architectures and strategies. Lastly, we propose a new benchmark multi-fidelity test case on unsteady smoke inflow, which includes the time-dependent component. Specifically, we compared several approaches for multi-fidelity NO learning, namely, the intermediate architecture, the multi-step architecture, and the transfer learning. The remainder of the paper is organised as follows: Section 2 provides an overview of the Neural Operator (NO) model and multi-fidelity learning. Section 3 introduces the benchmark test cases along with our proposed test case. Finally, Section 4 presents the concluding remarks.

Multifidelity Neural Operator

This section provides a brief overview of the NO model and multi-fidelity learning. The brief overview of the theoretical foundation of neural operators provides a mathematical primer to the readers on the concept of learning mappings between two infinite-dimensional spaces of functions. We also provide a quick review of multi-fidelity learning, primarily on NN-based models, to provide the reader with the general idea of learning with limited high-fidelity data.

Neural Operator

Consider the generic form of PDE:

(Lau)(x)=f(x),x∈D,u(x)=0,x∈∂D,(1)\begin{aligned} (\mathcal{L}_a u)(x) &= f(x), \qquad x \in D,\\ u(x) &= 0, \qquad x \in\partial D, \tag*{(1)} \end{aligned}

where a∈Aa \in\mathcal{A}, f∈U∗f \in\mathcal{U}^{*} with dd-dimensional solution domain D∈RdD \in\mathbb{R}^{d} and boundary ∂D\partial D. We assume that the solution u:D→Rduu : D \to\mathbb{R}^{d_u} and the input parameter a:D→Rdaa : D \to\mathbb{R}^{d_a} live in Banach spaces U\mathcal{U} and A\mathcal{A}, respectively. Expression La:A→L(U;U∗)\mathcal{L}_a : \mathcal{A} \to\mathcal{L}(\mathcal{U};\mathcal{U}^{*}) is a mapping from the parameter Banach space A\mathcal{A} to the space of linear operators U\mathcal{U} to its dual U∗\mathcal{U}^{*}. An operator G†:=La−1f:A→UG^{\dagger} := \mathcal{L}_a^{-1}f : \mathcal{A} \to\mathcal{U} is defined to map the input parameter to the solution a↦ua \mapsto u. Suppose we have a set of NN observations {(a(i),u(i))}i=1N\{(a^{(i)},u^{(i)})\}_{i=1}^{N}, the goal is to build an approximation of G†G^{\dagger} by constructing:

Gθ:A→U,θ∈Rp,(2)G_{\theta} : \mathcal{A} \to\mathcal{U}, \qquad\theta\in\mathbb{R}^{p}, \tag*{(2)}

then optimising parameter θ∈Rp\theta\in\mathbb{R}^{p}, where pp is the dimensionality of the parameter, so that Gθ†≈G†G_{\theta}^{\dagger} \approx G^{\dagger}.

Neural operators (NO) [20] are a class of neural network models that are designed to learn grid-independent problems in infinite-dimensional function space, in contrast to the classical neural network models that learn the mapping between finite-dimensional vector spaces. The neural operator model is generally composed of linear integral operators and non-linear activation functions that are developed specifically to tackle the learning problem between Banach spaces. The NO is formulated as an iterative architecture which resembles a multi-layered neural network:

a(x)↦v0↦v1↦⋯↦vl↦u(x),(3)a(x) \mapsto v_0 \mapsto v_1 \mapsto\cdots\mapsto v_l \mapsto u(x), \tag*{(3)}

where vjv_j for j=0,1,…,lj = 0,1,\ldots,l is a sequence of functions that transforms the input to its latent representation. The input a∈Aa \in\mathcal{A} is initially transformed into a higher dimensional representation v0(x)=P(a(x))v_0(x) = P(a(x)) through a local operator PP (also known as the lifting layer [20]), which usually a shallow fully-connected NN is used in this task. The output u(x)=Q(vl(x))u(x) = Q(v_l(x)) corresponds to the projection of the final latent space into the output space, where this step is also typically parameterised by a shallow fully connected NN.

The ability of neural operator models to learn the mappings between infinite-dimensional function spaces is made possible by utilising integral operators inside of each iteration update vj↦vj+1v_j \mapsto v_{j+1}. The update is defined as the non-local integral operator K\mathcal{K} and a nonlinear activation function σ\sigma,

vj+1(x):=σ(Wvj(x)+(K(vj))(x))(4)v_{j+1}(x) := \sigma\left(Wv_j(x) + \left(\mathcal{K}(v_j)\right)(x)\right) \tag*{(4)}

The linear integral operator is given by.

(K(vj))(x)=∫Djκ(j)(x,y)vj(y) dyj(y)≈∑i=1Nκ(x,yi)vj(yi)Δyi,(5)\left(\mathcal{K}(v_j)\right)(x) = \int_{D_j} \kappa^{(j)}(x,y)v_j(y)\,dy_j(y) \approx\sum_{i=1}^{N}\kappa(x,y_i)v_j(y_i)\Delta y_i, \tag*{(5)}

where vj(⋅)v_j(\cdot) is the input function to the operator block, and κ(x,y)\kappa(x,y) is the learnable kernel between two points in the output and input domain. Then, the linear integral operator is followed by non-linear activation functions such as Gaussian error linear units (GeLU) [34], sigmoid linear units (SiLU) [35], or rectified linear units (ReLU) [36].

The kernel integrator in neural operator models is available in many forms of different design options, such as the graph neural operator (GNO) [37], low-rank neural operator (LNO) [20], Fourier neural operator (FNO) [38], and wavelet neural operator (WNO) [31], each offering distinct advantages, disadvantages, and use cases. For the sake of brevity, this paper focuses exclusively on the FNO formulation. However, in some of the experiments, WNO was also implemented.

FNO [38] uses the Fourier transform as the kernel integral transformation. This method is effective in tackling challenging PDE problems, such as learning solutions in fluid dynamics problems [38]. To define the kernel integrator in FNO, let F\mathcal{F} be the Fourier transform of a function f:D→Rdvf:D \to\mathbb{R}^{d_v} and F−1\mathcal{F}^{-1} is its inverse,

(Ff)j(k)=∫Dfj(x)e−2iπ(x,k)dx(6)(\mathcal{F}f)_j(k)=\int_D f_j(x)e^{-2i\pi(x,k)}\mathrm{d}x \tag*{(6)}
(F−1f)j(x)=∫Dfj(k)e2iπ(x,k)dk,(\mathcal{F}^{-1}f)_j(x)=\int_D f_j(k)e^{2i\pi(x,k)}\mathrm{d}k,

for j=1,…,dvj=1,\ldots,d_v where i=−1i=\sqrt{-1} is the imaginary unit. Thus, we define the Fourier kernel integrator as:

(K(a;ϕ)vt)(x)=F−1(F(κϕ)⋅F(vt))(x),∀x∈D,(7)(\mathcal{K}(a;\phi)v_t)(x)=\mathcal{F}^{-1}(\mathcal{F}(\kappa_\phi)\cdot\mathcal{F}(v_t))(x),\quad\forall x\in D, \tag*{(7)}

where κϕ\kappa_\phi is parameterised in Fourier space.

When defining the model, we typically retain only the first NmN_m modes, where NmN_m is a user-defined hyperparameter. This means that we are not using all Fourier modes from the Fourier transform result; instead, we discard the high-frequency modes. Choosing the right number of NmN_m might be a unique problem for FNO. Since the higher frequency modes are interpreted as the non-linearity term in the model, one might suggest that we keep high enough amounts of modes in the model. Thus, this results in more efficient computation without sacrificing too much accuracy.

The schematic figure of the FNO is given in Figure 1. The full architecture of the neural operator (a) resembles what we have described in Eq. (3), and the Fourier integral kernel from Eq. (4) is illustrated in Figure 1(b). The bottom part of the schematic 1(b) describes the Fourier integral kernel, and the top part illustrates the bias. The operation RR in the Fourier integral kernel denotes that we retain the first NmN_m modes from the Fourier transformation operation. It is also worth noticing that the bias WW in Figure 1(b) is modelled as dd-dimensional 1×11 \times1 convolutional layer, where dd refers to the spatial dimension of the problem.

Figure 1. FNO schematics: (a) full architecture of FNO. (b) Fourier layers.

Multi-fidelity learning

Multiple levels of simulation fidelity are often found in PDE-based systems. The goal of multi-fidelity simulation lies in balancing accuracy and computational efficiency. High-fidelity (HF) simulations provide detailed and highly accurate results but require significant computational resources. In contrast, low-fidelity (LF) simulations are more computationally efficient, though they trade off some degree of accuracy and detail. In engineering applications, the availability of multi-level simulation fidelity has been exploited to assist the process of design optimisation [24, 25]. In this matter, several methods have been implemented to perform multi-fidelity modelling, such as model fusion [39], space mapping [40], and shape-preserving response prediction [41].

Before the utilisation of neural networks as a surrogate model became popular due to the advent of computational resources, the multi-fidelity learning approach has been implemented in other popular surrogate models such as linear regression [42, 43], Gaussian process regression [21, 22], and polynomial chaos expansion (PCE) regression [23]. Despite different multi-fidelity surrogate model (MFSM) methods, approaches, and applications that have been developed, the core concept of multi-fidelity learning in surrogate models is the exploitation of the correlation between the HF and LF data. One of the most used representations is the comprehensive correction [44], described as:

y^HF=ρ(x)⋅yLF(x)+δ(x),(8)\hat{y}_{HF} = \rho(x) \cdot y_{LF}(x) + \delta(x), \tag*{(8)}

where ρ(x)\rho(x) represents the multiplicative correlation between multi-fidelity data, and δ(x)\delta(x) represents the additive correlation. The problem with this representation is that it only handles linear correlations between the lower and higher fidelity. However, the correlation between LF and HF data might go beyond linear correlation, such as the mixed convection flows past a cylinder [45, 46]. To capture the nonlinear correlation, a generalised autoregressive scheme is proposed in [26], which is expressed as:

y^HF=F(yLF)+δ(x),(9)\hat{y}_{HF} = F(y_{LF}) + \delta(x), \tag*{(9)}

where F(⋅)F(\cdot) is an unknown, possibly nonlinear, function that maps between the low and the high-fidelity data. In terms of function mapping, Eq. (9) rewrites as:

y^HF=F(x,yLF;θ).(10)\hat{y}_{HF} = \mathcal{F}(x, y_{LF}; \theta). \tag*{(10)}

In this case, [26] use a neural network to represent F(⋅)\mathcal{F}(\cdot).

To represent the non-linear mapping function with a neural network, the mapping function F(⋅)\mathcal{F}(\cdot) is then decomposed into linear and nonlinear parts:

F=Fl+Fnl,(11)\mathcal{F} = \mathcal{F}_l + \mathcal{F}_{nl}, \tag*{(11)}

where Fl\mathcal{F}_l and Fnl\mathcal{F}_{nl} are the linear and nonlinear terms in F\mathcal{F}, respectively. Thus, the correlation is now written as:

x^HF=γFl(x,xLF;θ)+(1−γ)Fnl(x,yLF;θ),γ∈[0,1],(12)\hat{x}_{HF} = \gamma\mathcal{F}_l(x, x_{LF}; \theta) + (1 - \gamma)\mathcal{F}_{nl}(x, y_{LF}; \theta), \quad\gamma\in[0, 1], \tag*{(12)}

where γ\gamma is a user-defined hyperparameter.

Similarly, Guo et al. [27] proposed a multi-fidelity neural network architecture. In the proposed architecture, instead of decomposing the mapping function F(⋅)\mathcal{F}(\cdot) to the linear and nonlinear terms as shown in Equation (11), they directly model the function F(x,yLF;θ)\mathcal{F}(x, y_{LF}; \theta). This architecture uses the same input layer for the LF and HF data. Then, at the end of the network, the output layer of the HF is given, and the output layer of the LF data is located in the middle of the hidden layers. Thus, this architecture is called the ”intermediate architecture”. Since the LF and HF data are processed simultaneously in a single pass, two different training loss components are introduced, namely the MSELFMSE_{LF} and MSEHFMSE_{HF}:

MSELF=1NLF∑i=1NLF(yLF(i)−y^LF(xLF(i)))2,andMSE_{LF} = \frac{1}{N_{LF}}\sum_{i=1}^{N_{LF}}\left(y_{LF}^{(i)} - \hat{y}_{LF}(x_{LF}^{(i)})\right)^2, \quad\text{and}
MSEHF=1NHF∑i=1NHF(yHF(i)−y^HF(xHF(i)))2.(13)MSE_{HF} = \frac{1}{N_{HF}}\sum_{i=1}^{N_{HF}}\left(y_{HF}^{(i)} - \hat{y}_{HF}(x_{HF}^{(i)})\right)^2. \tag*{(13)}

Where y(i)y^{(i)} indicates the ground truth and y^(x(i))\hat{y}(x^{(i)}) indicates the predicted output y^\hat{y} given xx. The subscripts LFLF and HFHF are used to indicate the low fidelity and high fidelity data, respectively. A weighted sum is then used to aggregate the two loss functions to account for objectives of different scales. Thus, the loss function in this architecture is written as:

L=αMSEHF+(1−α)MSELF+λ∥θ∥2,(14)\mathcal{L} = \alpha MSE_{HF} + (1 - \alpha)MSE_{LF} + \lambda\lVert\theta\rVert^2, \tag*{(14)}

where α∈[0,1]\alpha\in[0, 1] is the weighting factor of the loss function and λ∥θ∥2\lambda\lVert\theta\rVert^2 acts as the regularisation term of the parameters θ\theta.

Intermediate architecture

Inspired by [27], we implement the intermediate multi-fidelity neural network architecture in the neural operator. Illustrated in Figure 2, the intermediate multi-fidelity neural operator (MFNO) consists of 2 neural operators that are connected with a fully connected layer in which one of the nodes in the layer is the lower-fidelity output. The model is trained by minimising the loss function defined in Eq. (14), where the α\alpha is a user-defined hyperparameter since the lower and higher-fidelity neural operator shares the same input layer. Both training datasets TLF={(xLF(i),yLF(i)):1≤i≤NLF}\mathcal{T}_{LF} = \{(x_{LF}^{(i)}, y_{LF}^{(i)}) : 1 \leq i \leq N_{LF}\} and THF={(xHF(i),yHF(i)):1≤i≤NHF}\mathcal{T}_{HF} = \{(x_{HF}^{(i)}, y_{HF}^{(i)}) : 1 \leq i \leq N_{HF}\} are concatenated before training the model.

Figure 2. Intermediate multi-fidelity neural operator architecture

Multi-step architecture

Another approach, also used to define a multi-fidelity neural operator is by employing two distinct neural operators to model these functions. In this multi-step [27] approach, the first neural operator NOLFNO_{LF} is trained on the lower-fidelity training set TLF\mathcal{T}_{LF}. Then a second neural operator NOHFNO_{HF} approximates the high-fidelity function y^HF=FHF(xHF,yLF)\hat{y}_{HF} = \mathcal{F}_{HF}(x_{HF}, y_{LF}) based on the HF input xHFx_{HF} and the LF output y^LF\hat{y}_{LF} at the same locations with yHFy_{HF}. Since the LF and HF data may be generated independently, the yLFy_{LF} at location xHFx_{HF} might not be directly available. Thus, the data has to be evaluated from the lower fidelity neural operator, y^LF=FLF(xHF)\hat{y}_{LF} = \mathcal{F}_{LF}(x_{HF}). The implementation of the two-step multi-fidelity neural operator is illustrated in Figure 3.

Figure 3. Two-step multi-fidelity neural operator architecture

An alternative form of two-step multi-fidelity learning is based on the residual learning approach. Proposed in [31], the residual MFNO use a similar architecture to the two-step MFNO as shown in Figure 3. However, the second neural operator learns the residual value instead of the high-fidelity output, defined as:

r(x)=yHF(x)−yLF(x).(15)r(x) = y_{HF}(x) - y_{LF}(x). \tag*{(15)}

The goal of using residual learning, rather than directly mapping to the high-fidelity (HF) solution, is to simplify the learning task. Since there is generally some correlation between the low-fidelity (LF) and HF solutions, a degree of feature similarity can be expected. In some cases, however, this correlation may be weak or dominated by a systematic bias. In such scenarios, learning the residual, the difference between the HF and LF outputs, allows the model to focus on capturing the remaining complexity not already represented by the LF solution. This approach can lead to improved performance and data efficiency, as supported by [31]. The residual MFNO architecture is depicted in Figure 4.

Figure 4. Residual multi-fidelity neural operator architecture

Transfer learning

Another widely used approach in multi-fidelity modelling is transfer learning [29, 30]. The central idea is to leverage knowledge gained from training on low-fidelity (LF) data to improve performance on high-fidelity (HF) tasks. This is achieved through a two-stage process: pre-training and fine-tuning. During the pre-training phase, a model is trained using a large set of inexpensive LF data TLF\mathcal{T}_{LF} following a standard neural operator training procedure [20] yielding a set of optimum LF NO parameters θLF\theta_{LF}. This set of optimum LF NO parameters is then transferred to the initialised HF NO as a starting point for the fine-tuning process. A limited amount of expensive, high-fidelity data is then employed to train the HF NO in the fine-tuning process. After this fine-tuning process, we then obtain the multi-fidelity model in the form of the fine-tuned HF NO model. The schematic of the transfer learning process is given in Figure 5.

Figure 5. Transfer learning process schematics.

Experiment results

We present four test cases. The first two, taken from [31, 47], are the 1-dimensional stochastic Poisson equation and 2-dimensional Darcy flow in a triangular domain act as a benchmark. Then, we present the modified version of the 2-dimensional Darcy flow as our third test case. Finally, we will evaluate the models based on the unsteady smoke inflow problem [48]. The main goal of this test case is to assess the multi-fidelity model performance on time-series data with a complex problem, in this case, an unsteady laminar flow.

1-dimensional stochastic Poisson equation

Following [31], we consider the second-order stochastic Poisson’s equation in one dimension as our first test case. The stochastic differential equation (SDE) is formulated as:

d2u(x)dx2=20g(x),x∈[0,1],u(x=0)=u(x=1)=0,g(x)∼GP(m(x),k(x,x′))(16)\frac{d^2u(x)}{dx^2}=20g(x),\quad x\in[0,1], \\ u(x=0)=u(x=1)=0, \\ g(x)\sim\mathcal{GP}(m(x),k(x,x')) \tag*{(16)}

where g(x)g(x) is the spatially random forcing function, modelled as a Gaussian process (GP\mathcal{GP}) with mean m(x)m(x) set to be 0, and the radial basis function kernel k(x,x′)=exp⁡(−(x−x′)2/(2l2))k(x,x')=\exp(-(x-x')^2/(2l^2)) where the kernel length scale of the GP is set to be l=0.1l=0.1. The HF dataset is generated using a grid size Δx=0.01\Delta x=0.01, while the LF dataset grid size is Δx=0.1\Delta x=0.1. An illustration of the differences between LF and HF data is presented in Figure 6. As shown, the LF data captures the overall trend of the HF solution but lacks finer details present in the more accurate HF output.

Figure 6. An example of the difference between LF and HF data in a 1-dimensional stochastic Poisson equation.

We vary the number of training samples on the LF and HF models to investigate the effect of the number of training samples on the multi-fidelity model. Each experiment is repeated 15 times with different combinations of testing and training dataset splits. We found that the FNO yields a better performance compared to other kernels, such as the Wavelet neural operator (WNO), in this test case. However, since the selection of the NO kernel is not the focus of this paper, the comparison between kernels is not shown in this paper. The LF and HF model hyperparameters are given in Table 1. The choice of the hyperparameters is arbitrary. In this experiment, we found that the hyperparameters in Table 1 yield a balance between accuracy and computational cost.

Lower-fidelityHigher-fidelity
NO kernelFourierFourier
Number of NO layers42
FC layers width6464
Fourier modes1616
Learning rate1×10−31 \times10^{-3}1×10−31 \times10^{-3}
Weight Decay1×10−41 \times10^{-4}1×10−41 \times10^{-4}
Epochs200200

Table 1. 1-dimensional multi-fidelity FNO hyperparameters.

The experiments were conducted mainly to investigate the behaviour of the models given different training samples in both LF and HF. The result in Table 2 is interpreted as two experiments. First, we set a constant value for the number of HF training samples to 50 and gradually increased the LF training sample number from 50 to 150. Second, we set ntrainn_{\mathrm{train}} LF to be constant at 150 while gradually increasing HF from 15 to 50. While we have access to many HF data, we keep the highest number of HF training samples to 50 to emulate real-world multi-fidelity cases. In this test case, we provide the high-fidelity baseline, which is trained only on the corresponding high-fidelity test sample.

ntrainn_{\mathrm{train}}2-stepsResidualIntermediateTransfer learningHF baseline
LFHF
50504.43×10−2±9.35×10−34.43 \times10^{-2} \pm9.35 \times10^{-3}6.83×10−2±1.49×10−26.83 \times10^{-2} \pm1.49 \times10^{-2}5.60×10−2±1.21×10−25.60 \times10^{-2} \pm1.21 \times10^{-2}4.80×10−2±1.06×10−24.80 \times10^{-2} \pm1.06 \times10^{-2}5.71×10−2±1.23×10−25.71 \times10^{-2} \pm1.23 \times10^{-2}
100503.07×10−2±5.96×10−33.07 \times10^{-2} \pm5.96 \times10^{-3}3.54×10−2±7.40×10−33.54 \times10^{-2} \pm7.40 \times10^{-3}5.43×10−2±1.42×10−25.43 \times10^{-2} \pm1.42 \times10^{-2}3.85×10−2±9.68×10−33.85 \times10^{-2} \pm9.68 \times10^{-3}5.71×10−2±1.23×10−25.71 \times10^{-2} \pm1.23 \times10^{-2}
150502.58×10−2±3.55×10−32.58 \times10^{-2} \pm3.55 \times10^{-3}2.69×10−2±7.06×10−32.69 \times10^{-2} \pm7.06 \times10^{-3}7.97×10−2±1.66×10−27.97 \times10^{-2} \pm1.66 \times10^{-2}3.40×10−2±6.88×10−33.40 \times10^{-2} \pm6.88 \times10^{-3}5.71×10−2±1.23×10−25.71 \times10^{-2} \pm1.23 \times10^{-2}
150253.93×10−2±1.00×10−23.93 \times10^{-2} \pm1.00 \times10^{-2}3.31×10−2±6.33×10−33.31 \times10^{-2} \pm6.33 \times10^{-3}6.84×10−2±2.13×10−26.84 \times10^{-2} \pm2.13 \times10^{-2}3.85×10−2±7.92×10−33.85 \times10^{-2} \pm7.92 \times10^{-3}1.24×10−1±3.02×10−21.24 \times10^{-1} \pm3.02 \times10^{-2}
150155.99×10−2±1.07×10−25.99 \times10^{-2} \pm1.07 \times10^{-2}3.95×10−2±4.91×10−33.95 \times10^{-2} \pm4.91 \times10^{-3}7.83×10−2±2.92×10−27.83 \times10^{-2} \pm2.92 \times10^{-2}3.96×10−2±6.01×10−33.96 \times10^{-2} \pm6.01 \times10^{-3}2.11×10−1±5.67×10−22.11 \times10^{-1} \pm5.67 \times10^{-2}

Table 2. 1-dimensional multi-fidelity FNO RMSE

The result shows that the intermediate architecture is inferior compared to the 2-step and residual architectures. The 2-step and residual architecture seem to have their advantages in particular cases by a small margin from each other. Meanwhile, the transfer learning, while not giving an outstanding performance in this test case, has the most robust results concerning the LF and HF variation. A single realisation of the prediction on the test dataset is given in Figure 7.

Figure 7. Single realisation on 1-dimensional stochastic Poisson equation test data.

2-dimensional triangular Darcy flow

The Darcy equation is often used to describe flow through a porous medium in geotechnical and civil engineering. Following [31, 47], we consider this problem in the 2-dimensional triangular domain with a notch. The mathematical expression of the Darcy equation is given by:

−∇⋅(a(x,y)∇u(x,y))=f(x,y); x,y∈[0,1],-\nabla\cdot\left(a(x,y)\nabla u(x,y)\right) = f(x,y); \ x,y \in[0,1],
u(x,y)∣∂D=GP(0,K((x,y),(x′,y′)))(17)u(x,y)\big|_{\partial D} = \mathcal{GP}\left(0,\mathcal{K}\left((x,y),(x',y')\right)\right) \tag*{(17)}

where a(x,y)∈Ra(x,y) \in\mathbb{R} is the permeability field, u(x,y)∈Ru(x,y) \in\mathbb{R} is the pressure field, and f(x,y)∈Rf(x,y) \in\mathbb{R} is the source term. The synthetic training and testing data are generated by solving Eq. (17) for different boundary conditions. Different boundary conditions u(x,y)∣∂Du(x,y)\big|_{\partial D} are generated from a Gaussian process with radial basis function kernel, defined as:

K((x,y),(x′,y′))=exp⁡(−(x−x′)22lx2−(y−y′)22ly2),(18)\mathcal{K}\left((x,y),(x',y')\right) = \exp\left(-\frac{(x-x')^2}{2l_x^2} - \frac{(y-y')^2}{2l_y^2}\right), \tag*{(18)}

where the lengthscale parameter lxl_x and lyl_y are set as 0.2. The permeability and the source field are assumed constant with a(x,y)=0.1a(x,y)=0.1 and f(x,y)=−1f(x,y)=-1 respectively.

Lu et al. [47] used the MATLAB PDE Toolbox to solve the problem numerically. However, in this paper, we use the generated dataset by Tripura et al. [31], where they already modified the original problem to be suited for a multi-fidelity problem. The different levels of fidelity are achieved through variation in the discretisation grid, where the HF dataset is obtained from a fine grid with maximum and minimum element edge lengths set to hmax=0.028h_{max}=0.028 and hmin=0.026h_{min}=0.026. Meanwhile, the LF dataset is obtained from a coarser grid with maximum and minimum element edge lengths set to hmax=0.18h_{max}=0.18 and hmin=0.16h_{min}=0.16. However, as shown in Figure 8, the difference between LF and HF results is not significant.

Figure 8. An example of the difference between LF and HF data in the original 2-dimensional Darcy flow problem.

To train the model, we use the model hyperparameters as given in Table 3. Here, we follow the WNO hyperparameters used in [31], where the biorthogonal filter is set to near_sym_a and the quarter shift filter is set to qshift_a. However, the other hyperparameter settings are arbitrary. Similar to section 3.1, we conduct the experiments across different combinations of LF and HF training samples. We repeat the experiments with 10 different combinations of testing and training dataset splits for consistency.

Lower-fidelityHigher-fidelity
NO kernelWaveletWavelet
Number of NO layers44
FC layers width6464
Decomposition level22
Biorthogonal filtersnear_sym_anear_sym_a
Quarter shift filtersqshift_aqshift_a
Learning rate1×10−31 \times10^{-3}1×10−31 \times10^{-3}
Weight Decay1×10−41 \times10^{-4}1×10−41 \times10^{-4}
Epochs200200

Table 3. 2-dimensional multi-fidelity WNO hyperparameters.

The results in Table 4 indicate that transfer learning and the two-step architecture outperform all other methods and are the only approaches that consistently surpass the high-fidelity (HF) baseline across all training sample configurations. We observe that when high-fidelity data is limited, leveraging lower-fidelity data significantly enhances prediction accuracy. However, when sufficiently high-fidelity training data is available, in this case 50, incorporating lower-fidelity data does not always improve prediction accuracy for residual and intermediate models. This provides us with an insight that while multi-fidelity is promising, we have to be careful with the model and hyperparameter selection. Otherwise, instead of enhancing the model performance, we could end up degrading the prediction model. Figure 9 presents a realisation plot illustrating these findings. It is important to note that, although some models outperform others, the overall prediction results remain fairly accurate. This is evidenced by the errors, which are all within the same order of magnitude, as shown in Table 4. Furthermore, the error patterns presented in Figure 9 appear to be purely random, suggesting no systematic bias across the models.

ntrainn_{\mathrm{train}}LFHF2-stepsResidualIntermediateTransfer learningHF baseline
50502.28×10−2±2.35×10−32.28 \times10^{-2} \pm2.35 \times10^{-3}3.50×10−2±2.25×10−33.50 \times10^{-2} \pm2.25 \times10^{-3}3.64×10−2±3.56×10−33.64 \times10^{-2} \pm3.56 \times10^{-3}2.32×10−2±2.49×10−32.32 \times10^{-2} \pm2.49 \times10^{-3}3.14×10−2±4.54×10−33.14 \times10^{-2} \pm4.54 \times10^{-3}
100502.25×10−2±3.38×10−32.25 \times10^{-2} \pm3.38 \times10^{-3}3.18×10−2±2.10×10−33.18 \times10^{-2} \pm2.10 \times10^{-3}3.91×10−2±2.46×10−33.91 \times10^{-2} \pm2.46 \times10^{-3}1.96×10−2±1.00×10−31.96 \times10^{-2} \pm1.00 \times10^{-3}3.14×10−2±4.54×10−33.14 \times10^{-2} \pm4.54 \times10^{-3}
150502.23×10−2±4.34×10−32.23 \times10^{-2} \pm4.34 \times10^{-3}2.63×10−2±1.76×10−32.63 \times10^{-2} \pm1.76 \times10^{-3}3.67×10−2±3.79×10−33.67 \times10^{-2} \pm3.79 \times10^{-3}1.74×10−2±1.59×10−31.74 \times10^{-2} \pm1.59 \times10^{-3}3.14×10−2±4.54×10−33.14 \times10^{-2} \pm4.54 \times10^{-3}
150252.77×10−2±4.55×10−32.77 \times10^{-2} \pm4.55 \times10^{-3}2.76×10−2±1.95×10−32.76 \times10^{-2} \pm1.95 \times10^{-3}5.00×10−2±4.50×10−35.00 \times10^{-2} \pm4.50 \times10^{-3}2.49×10−2±3.13×10−32.49 \times10^{-2} \pm3.13 \times10^{-3}8.86×10−2±1.81×10−28.86 \times10^{-2} \pm1.81 \times10^{-2}
150152.90×10−2±8.80×10−32.90 \times10^{-2} \pm8.80 \times10^{-3}2.98×10−2±8.48×10−32.98 \times10^{-2} \pm8.48 \times10^{-3}5.68×10−2±1.32×10−25.68 \times10^{-2} \pm1.32 \times10^{-2}2.70×10−2±3.59×10−32.70 \times10^{-2} \pm3.59 \times10^{-3}2.27×10−1±4.32×10−22.27 \times10^{-1} \pm4.32 \times10^{-2}

Table 4. 2-dimensional Darcy flow multi-fidelity WNO RMSE

Figure 9. Prediction and error plot of a single data point for each model on 2D Darcy flow in a triangular domain with a notch.

Modified 2-dimensional triangular Darcy flow

The standard Darcy flow problem described in § 4.2 assumes that multi-fidelity data is obtained through differences in mesh resolution. In this setup, lower-fidelity data is generated by solving the same numerical formulation on a coarser grid or mesh, resulting in a mean absolute error of 0.02647 between the low-fidelity (LF) and high-fidelity (HF) data, as shown in Figure 8. While this approach is a valid form of multi-fidelity learning, real-world multi-fidelity data often exhibits more complex relationships, leading to greater discrepancies between LF and HF solutions. For example, in computational fluid dynamics (CFD) problems, the user can specify different assumptions or turbulence equations, which significantly impact the solution. As an illustration, Figure 6 in [33] highlights the differences between turbulence models such as Reynolds-averaged Navier-Stokes (RANS), delayed detached eddy simulation (DDES), and zonal detached eddy simulation (ZDES), demonstrating how variations in governing equations lead to substantial differences in predicted flow behaviour.

To emulate the behaviour of multi-fidelity in the governing equation, we propose to modify the 2-dimensional triangular Darcy flow in [31]. Instead of decreasing the resolution to achieve lower fidelity, we decompose the solution using a proper orthogonal decomposition (POD) [7] method. The snapshot matrix A\mathbf{A} of the solution is decomposed through a lower rank approximation, defined as:

A≈UΣkVT,(19)\mathbf{A} \approx\mathbf{U}\boldsymbol{\Sigma}_{k}\mathbf{V}^{T}, \tag*{(19)}

where U\mathbf{U} is an N×kN \times k orthogonal matrix, V\mathbf{V} is an m×km \times k orthogonal matrix, and Σk\boldsymbol{\Sigma}_{k} is a k×kk \times k matrix with all elements zero except its diagonal. In this case, we only retain the two most dominant modes, resulting in a reduced-order snapshot matrix A2A_2.

A2=UΣ2VT(20)A_2 = U\Sigma_2 V^T \tag*{(20)}

With this approach, we preserve the overall structure of the solution while selectively removing finer details in the lower-fidelity data, mimicking the multi-fidelity scenario based on governing equations, as seen in fluid dynamics problems. This method significantly increases the discrepancy, measured by mean absolute error between the low-fidelity (LF) and high-fidelity (HF) data to 0.151, compared to 0.02647 in the original LF setup. An example realisation is illustrated in Figure 10.

Figure 10. Comparison between the original high-fidelity data, original lower-fidelity data, and the reduced-order lower-fidelity data.

We adopt the same model hyperparameters as the two-dimensional WNO configuration specified in Table 3. To ensure robustness, each experiment is repeated with 10 different random splits of training and testing data. The results, summarized in Table 5, show that transfer learning consistently outperforms the other methods across various combinations of low-fidelity (LF) and high-fidelity (HF) training data. The benefits of incorporating LF data are particularly evident when the HF data is limited. In such scenarios, LF data significantly enhances prediction accuracy. However, when a sufficient amount of HF training data is available, the advantage of incorporating LF data diminishes and, in some cases, may even slightly degrade performance relative to the HF-only baseline. In this test case, we found a more pronounced effect of the benefit of transfer learning compared to the other methods when the LF data is less accurate.

ntrainn_{\mathrm{train}} LFntrainn_{\mathrm{train}} HF2-stepsResidualIntermediateTransfer learningHF baseline
50505.35×10−2±6.14×10−35.35 \times10^{-2} \pm6.14 \times10^{-3}1.54×10−1±1.08×10−21.54 \times10^{-1} \pm1.08 \times10^{-2}4.52×10−2±3.63×10−34.52 \times10^{-2} \pm3.63 \times10^{-3}2.76×10−2±3.08×10−32.76 \times10^{-2} \pm3.08 \times10^{-3}3.14×10−2±4.54×10−33.14 \times10^{-2} \pm4.54 \times10^{-3}
100505.65×10−2±6.74×10−35.65 \times10^{-2} \pm6.74 \times10^{-3}1.32×10−1±4.87×10−31.32 \times10^{-1} \pm4.87 \times10^{-3}6.82×10−2±6.58×10−36.82 \times10^{-2} \pm6.58 \times10^{-3}2.74×10−2±4.38×10−32.74 \times10^{-2} \pm4.38 \times10^{-3}3.14×10−2±4.54×10−33.14 \times10^{-2} \pm4.54 \times10^{-3}
150505.51×10−2±8.08×10−35.51 \times10^{-2} \pm8.08 \times10^{-3}1.20×10−1±7.88×10−31.20 \times10^{-1} \pm7.88 \times10^{-3}6.78×10−2±9.51×10−36.78 \times10^{-2} \pm9.51 \times10^{-3}2.48×10−2±1.51×10−32.48 \times10^{-2} \pm1.51 \times10^{-3}3.14×10−2±4.54×10−33.14 \times10^{-2} \pm4.54 \times10^{-3}
150259.47×10−2±1.11×10−29.47 \times10^{-2} \pm1.11 \times10^{-2}1.59×10−1±1.91×10−21.59 \times10^{-1} \pm1.91 \times10^{-2}9.26×10−2±5.04×10−39.26 \times10^{-2} \pm5.04 \times10^{-3}4.63×10−2±6.39×10−34.63 \times10^{-2} \pm6.39 \times10^{-3}8.86×10−2±1.81×10−28.86 \times10^{-2} \pm1.81 \times10^{-2}
150151.65×10−1±3.16×10−21.65 \times10^{-1} \pm3.16 \times10^{-2}3.23×10−1±1.51×10−13.23 \times10^{-1} \pm1.51 \times10^{-1}1.27×10−1±2.57×10−21.27 \times10^{-1} \pm2.57 \times10^{-2}7.62×10−2±6.65×10−37.62 \times10^{-2} \pm6.65 \times10^{-3}2.27×10−1±4.32×10−22.27 \times10^{-1} \pm4.32 \times10^{-2}

Table 5. 2-dimensional modified Darcy flow multi-fidelity WNO RMSE.

Figure 11 illustrates a sample realization for a single test case. For both the intermediate and residual models, the largest prediction errors occur near the top-left and bottom-left corners of the domain. This appears to be coincidental, as the overall error patterns are random and do not exhibit any consistent spatial structure.

Figure 11. Prediction and error plot for each model on modified 2D Darcy flow in triangular domain with a notch.

To further analyse the models’ performance under varying levels of discrepancy between low-fidelity (LF) and high-fidelity (HF) data, we examine the relationship between the LF-HF discrepancy and the mean absolute error (MAE) of the predictions compared to the HF ground truth. As illustrated in Figure 12, prediction errors generally increase with larger LF-HF discrepancies, a trend that is expected. However, the extent to which the discrepancy affects model performance varies. Notably, the transfer learning approach appears to be the least sensitive to increasing discrepancies. While we do not provide a theoretical justification, we hypothesise that this robustness may stem from the way LF information is incorporated. In the intermediate, residual, and two-step models, LF data is directly injected into the input of the HF model, which may confuse the network when the discrepancy is large. In contrast, transfer learning leverages LF-trained weights as initialisation for the HF model. This approach allows the HF model to start with a rough understanding of the solution’s general structure, which is then refined during training on HF data, making it more resilient to LF-HF mismatch.

Figure 12. Discrepancy vs mean absolute error (MAE) plot for the 2-dimensional Darcy flow problem. Blue circle indicates the original data points from section 3.2, and the orange circle indicates data points from section 3.3.

We also investigate how different models perform under varying levels of discrepancy between LF and HF data. The first scenario, presented in § 4.2, features relatively low LF-HF discrepancy, while the second scenario in § 4.3 introduces a higher discrepancy by discarding more detailed information in the LF data. As shown in Figure 13, the LF training losses exhibit a clear gap between the original and modified datasets, an expected outcome due to the reduced complexity in the modified LF data. Interestingly, the differences become more pronounced during the HF training phase. The residual model performs well on the original dataset but struggles significantly on the modified data, suggesting its sensitivity to LF-HF mismatch. In contrast, the transfer learning model demonstrates strong performance across both scenarios, as indicated by the similar final loss values. The two-step model lies between the two; it performs better than the residual model on the modified data, but does not match the robustness of transfer learning. For the intermediate model, which jointly trains on LF and HF data in a single pass, training loss is plotted only up to epoch 200. All models are trained for a total of 200 epochs. For those employing separate LF and HF phases (e.g., two-step, residual, and transfer learning models), HF training is plotted immediately following LF training for visualisation purposes, although in practice, these phases may occur separately. Lastly, the HF baseline yields identical training curves for both datasets, as the HF data is unchanged.

Figure 13. Training history for Darcy flow problem. Blue lines represent training histories using the original dataset described in § 4.2, while orange lines correspond to training histories using the modified dataset from § 4.3.

Unsteady smoke inflow

The multi-fidelity unsteady smoke inflow dataset, introduced in [48], models the time-dependent concentration of smoke particles within a rectangular domain. It consists of 125 unique inlet configurations, each simulated over 126 timesteps. The inlet has a fixed radius of 3 units, and its location is sampled using Latin Hypercube Sampling (LHS), constrained within the ranges [4, 28] along the horizontal axis and [4, 35] along the vertical axis. The objective is to learn the temporal evolution of smoke particle concentration for varying inlet positions. To this end, the model is trained on data from different inlet configurations across all timesteps and tested by predicting the concentration evolution over the first 50 timesteps for the unseen inlet locations. The smoke inflow problem itself is driven by the Navier-Stokes equation, defined as:

∂u∂t=−(u⋅∇)u−ν∇2u−1ρ∇p+g,∂s∂t+(u⋅∇)s=α∇2s+i.(21)\frac{\partial u}{\partial t} = -(\mathbf{u} \cdot\nabla)\mathbf{u} - \nu\nabla^{2}\mathbf{u} - \frac{1}{\rho}\nabla p + g, \\ \frac{\partial s}{\partial t} + (\mathbf{u} \cdot\nabla)s = \alpha\nabla^{2}s + i. \tag*{(21)}

Where u\mathbf{u} is the velocity component, ρ\rho and pp denote the fluid density and pressure, α\alpha denotes the particle diffusivity, ss and ii are the smoke particle concentration and the particle concentration inflow, respectively. The flow simulations were conducted using PhiFlow [49] package, where the simulations are set to be incompressible, viscous, and laminar. The authors follow the standard simulation settings with viscosity ν=0.1\nu= 0.1 and density ρ=1.0\rho= 1.0, described in the PhiFlow smoke plume tutorial.1

The low-fidelity data is represented on a 96×12096 \times120 grid, while the high-fidelity data uses a finer 256×320256 \times320 grid. Although fidelity levels were controlled by grid resolution, the discrepancy between LF and HF data is notably larger than in the 2D Darcy flow, as shown in Figure 10 between the Original HF and LF. The unsteady smoke inflow problem, illustrated in Figure 14, shows that the LF simulations are able to capture the general trend of the flow, but somehow fail to predict the exact value.

Figure 14. Low and high-fidelity smoke inflow data with different inlet locations at timestep t=75t = 75.

Given the high computational cost to train the model, our experiments were limited to 100 low-fidelity (LF) and 10 high-fidelity (HF) training samples, with each experiment repeated five times. Since the problem is time-dependent, the number of rollouts during training plays a crucial role [50, 51]. However, constrained by hardware limitations, we restrict the rollout to a single step and use only a subset of the time domain, from t=0t = 0 to t=50t = 50, rather than using full timesteps for training. The remaining hyperparameters for the LF and HF models are listed in Table 6.

Lower-fidelityHigher-fidelity
NO kernelFourierFourier
Number of NO layers22
FC layers width3232
Fourier modes1616
Rollout number33
Learning rate4×10−44 \times10^{-4}5×10−45 \times10^{-4}
Weight Decay1×10−31 \times10^{-3}1×10−21 \times10^{-2}
Epochs200200

Table 6. Smoke inflow FNO models hyperparameters.

The model was trained on a single NVIDIA A100 GPU. However, due to the problem’s complexity, we must limit certain training hyperparameters, particularly for the HF model, such as the rollout number and batch size. Specifically, the HF model is trained with a rollout number of one and a maximum batch size of 25. To keep the model size manageable for the available hardware, the fully connected layer width and the number of Fourier modes were limited to 32 and 16, respectively. For the same reason, we did not evaluate the intermediate model in this test case, as it integrates both LF and HF inputs, requiring a larger batch size that exceeds our hardware constraints. Also, since the 2-step model performs better in most 2D test cases than the residual model, we only compare the 2-step model, transfer learning, and the HF baseline.

During training, a single LF model is shared across all multi-fidelity approaches to ensure a fair comparison. Consequently, the LF model’s loss history is not included in Figure 15. The HF training phase is repeated five times with different random seeds and training sample selections to assess consistency. As illustrated in Figure 15, the resulting loss values across models show no significant variation, indicating that all models are capable of effectively learning from the data.

Figure 15. HF training phase loss history for smoke inflow problem.

The model is evaluated on 25 unseen inlet locations using the high-fidelity (HF) dataset. Unlike the training loss history, the evaluation results, shown in Figure 16, demonstrate that the transfer learning approach consistently outperforms the HF baseline across all timesteps. In contrast, while having a similar profile for training loss, the 2-step model fails to generalise to unseen data, indicated by larger numbers of errors. This suggests that incorporating the LF model output as an additional input feature in the HF model tends to degrade prediction performance rather than enhance it.

Figure 16. RMSE at each timestep for each multi-fidelity strategy.

Given the temporal nature of the data, we further analyse how discrepancies between LF and HF evolve over time to better understand the models’ behavior. In Figure 17, we present the discrepancy computed across all inlet locations in the dataset at each time step. The figure indicates that the discrepancy generally increases as time progresses. However, this increasing divergence is not the sole challenge. Due to the sequential nature of the problem, both the training and inference phases rely on an autoregressive strategy, where predictions from previous time steps are used as inputs for subsequent steps. This setup can lead to error accumulation over time. For multi-fidelity models that are particularly sensitive to discrepancies, such as the two-step model, this compounding effect can significantly degrade performance.

Figure 17. Data discrepancy vs time for the smoke inflow data.

Conclusion

As PDE simulation problems grow in complexity, generating high-fidelity (HF) data for training surrogate models like Neural Operators (NOs) becomes increasingly costly. To mitigate this challenge, multi-fidelity learning has emerged as a promising strategy to leverage both low- and high-fidelity data during training. In this paper, we investigate several multi-fidelity approaches for NOs, including transfer learning, a two-step model, a residual-enhanced two-step model, and an intermediate architecture. These methods are evaluated against a baseline NO trained solely on HF data, using a series of test cases with increasing complexity. Our results show that while most multi-fidelity strategies perform comparably to the HF baseline on simpler tasks, their benefits diminish as the problem becomes more challenging. Notably, among all methods, only the transfer learning approach consistently outperforms the HF baseline across all scenarios.

Further analysis under varying LF-HF discrepancies revealed that strategies incorporating LF information directly into the HF model input are more vulnerable to performance degradation, especially when discrepancies are large. This vulnerability likely arises from the conflicting information that hinders effective learning. In contrast, transfer learning, by using LF-trained weights to initialise the HF model, provides a better learning process. It offers a generalised prior that is refined through HF training, making it more robust to LF-HF mismatch. This resilience becomes especially critical in complex, temporally unsteady problems where an autoregressive scheme can accumulates errors over time. In such settings, direct LF incorporation strategies struggle, whereas transfer learning maintains stable and reliable performance.

In addition to model exploration, we introduce two test cases situated at the higher end of the complexity spectrum. These were motivated by the current scarcity of complex multi-fidelity benchmarks. We observed that many existing multi-fidelity studies primarily focus on fidelity differences arising from grid resolution. While such cases are valid, they capture only a narrow subset of the broader multi-fidelity learning landscape. To address this, we propose a modification to the existing two-dimensional Darcy flow dataset by systematically reducing the physical complexity of the low-fidelity data using proper orthogonal decomposition (POD). Additionally, we introduce a new test case involving two-dimensional unsteady laminar smoke inflow, designed to evaluate the models’ ability to capture dynamic, time-evolving phenomena.

While this paper demonstrates the potential of multi-fidelity modeling for unsteady flow problems, the topic remains a rich and promising avenue for future research. One important direction is to investigate more effective implementations of multi-fidelity strategies, particularly in complex temporal problems such as the smoke inflow scenario. For instance, understanding why certain strategies fail to generalize to unseen data warrants further analysis. Exploring alternative neural operator architectures could also provide valuable insights and improved performance. Additionally, the current smoke inflow test case does not reflect fidelity differences arising from variations in governing equations. Developing new benchmark cases that incorporate such discrepancies would be a valuable contribution. Another promising direction is to explore uncertainty quantification in both low-fidelity and high-fidelity models, specifically, how different sources of uncertainty interact and propagate through multi-fidelity frameworks.

Acknowledgments

This work was supported by granted H2020 FETOPEN-2018-2019-2020-01 European project, Epistemic AI under grant agreement No. 964505 (E-pi).

References

  1. [1]P. Huthwaite, Accelerated finite element elastodynamic simulations using the gpu, Journal of Computational Physics 257 (2014) 687–707. doi:10.1016/j.jcp.2013.10.017. URL http://dx.doi.org/10.1016/j.jcp.2013.10.017
  2. [2]M. Bernardini, D. Modesti, F. Salvadore, S. Pirozzoli, Streams: A high-fidelity accelerated solver for direct numerical simulation of compressible turbulent flows, Computer Physics Communications 263 (2021) 107906. doi:10.1016/j.cpc.2021.107906. URL http://dx.doi.org/10.1016/j.cpc.2021.107906
  3. [3]H. Drucker, C. J. C. Burges, L. Kaufman, A. Smola, V. Vapnik, Support vector regression machines, in: M. Mozer, M. Jordan, T. Petsche (Eds.), Advances in Neural Information Processing Systems, Vol. 9, MIT Press, 1996, p. 1.
  4. [4]C. E. Rasmussen, C. K. I. Williams, Gaussian processes for machine learning, Adaptive Computation and Machine Learning series, MIT Press, London, England, 2005.DOI
  5. [5]R. K. Tripathy, R. Bilionis, Deep uq: Learning deep neural network surrogate models for high dimensional uncertainty quantification, Journal of computational physics 375 (2018) 565–588.DOI
  6. [6]G. Sun, S. Wang, A review of the artificial neural network surrogate modeling in aerodynamic design, Proceedings of the Institution of Mechanical Engineers, Part G: Journal of Aerospace Engineering 233 (16) (2019) 5863–5872.DOI
  7. [7]A. Chatterjee, An introduction to the proper orthogonal decomposition, Current Science 78 (7) (2000) 808–817. URL http://www.jstor.org/stable/24103957
  8. [8]S. Pakravan, P. A. Mistani, M. A. Aragon-Calvo, F. Gibou, Solving inverse-pde problems with physics-aware neural networks, Journal of Computational Physics 440 (2021) 110414. doi:10.1016/j.jcp.2021.110414. URL http://dx.doi.org/10.1016/j.jcp.2021.110414
  9. [9]R. Molinaro, Y. Yang, B. Engquist, S. Mishra, Neural inverse operators for solving pde inverse problems (2023). arXiv:2301.11167. URL https://arxiv.org/abs/2301.11167
  10. [10]J. Berg, K. Nyström, Neural networks as smooth priors for inverse problems for pdes, Journal of Computational Mathematics and Data Science 1 (2021) 100008. doi:10.1016/j.jcmds.2021.100008. URL http://dx.doi.org/10.1016/j.jcmds.2021.100008
  11. [11]M. Raissi, P. Perdikaris, G. E. Karniadakis, Multistep neural networks for data-driven discovery of nonlinear dynamical systems (2018). arXiv:1801.01236. URL https://arxiv.org/abs/1801.01236
  12. [12]S. Pan, K. Duraisamy, Data-driven discovery of closure models, SIAM Journal on Applied Dynamical Systems 17 (4) (2018) 2381–2413. doi:10.1137/18m1177263. URL http://dx.doi.org/10.1137/18M1177263
  13. [13]M. M. Rai, N. K. Madavan, Aerodynamic design using neural networks, AIAA Journal 38 (1) (2000) 173–182. doi:10.2514/2.938. URL http://dx.doi.org/10.2514/2.938
  14. [14]X. Du, P. He, J. R. Martins, Rapid airfoil design optimization via neural networks-based parameterization and surrogate modeling, Aerospace Science and Technology 113 (2021) 106701. doi:10.1016/j.ast.2021.106701. URL http://dx.doi.org/10.1016/j.ast.2021.106701
  15. [15]F. Murtagh, Multilayer perceptrons for classification and regression, Neurocomputing 2 (5–6) (1991) 183–197. doi:10.1016/0925-2312(91)90023-5. URL http://dx.doi.org/10.1016/0925-2312(91)90023-5
  16. [16]M.-C. Popescu, V. E. Balas, L. Perescu-Popescu, N. Mastorakis, Multilayer perceptron and neural networks, WSEAS Trans. Cir. and Sys. 8 (7) (2009) 579–588.DOI
  17. [17]Y. Lecun, Y. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE 86 (11) (1998) 2278–2324. doi:10.1109/5.726791 URL http://dx.doi.org/10.1109/5.726791
  18. [18]M. Raissi, P. Perdikaris, G. E. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational Physics 378 (2019) 686–707. doi:10.1016/j.jcp.2018.10.045. URL http://dx.doi.org/10.1016/j.jcp.2018.10.045
  19. [19]L. Lu, P. Jin, G. Pang, Z. Zhang, G. E. Karniadakis, Learning nonlinear operators via deeponet based on the universal approximation theorem of operators, Nature Machine Intelligence 3 (3) (2021) 218–229. doi:10.1038/s42256-021-00302-5. URL http://dx.doi.org/10.1038/s42256-021-00302-5
  20. [20]N. Kovachki, Z. Li, B. Liu, K. Azizzadenesheli, K. Bhattacharya, A. Stuart, A. Anandkumar, Neural operator: Learning maps between function spaces (2021). doi:10.48550/ARXIV.2108.08481. URL https://arxiv.org/abs/2108.08481
  21. [21]D. E. Myers, Matrix formulation of co-kriging, Journal of the International Association for Mathematical Geology 14 (3) (1982) 249–257. doi:10.1007/bf01032887. URL http://dx.doi.org/10.1007/BF01032887
  22. [22]A. I. Forrester, A. Só́bester, A. J. Keane, Multi-fidelity optimization via surrogate modelling, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 463 (2088) (2007) 3251–3269. doi:10.1098/rspa.2007.1900. URL http://dx.doi.org/10.1098/rspa.2007.1900
  23. [23]P. S. Palar, T. Tsuchiya, G. T. Parks, Multi-fidelity non-intrusive polynomial chaos based on regression, Computer Methods in Applied Mechanics and Engineering 305 (2016) 579–606. doi:10.1016/j.cma.2016.03.022. URL http://dx.doi.org/10.1016/j.cma.2016.03.022
  24. [24]S. Koziel, A. Bekasiewicz, I. Couckuyt, T. Dhaene, Efficient multi-objective simulation-driven antenna design using co-kriging, IEEE Transactions on Antennas and Propagation 62 (11) (2014) 5900–5905. doi:10.1109/tap.2014.2354673. URL http://dx.doi.org/10.1109/TAP.2014.2354673
  25. [25]X. Liu, W. Zhao, D. Wan, Multi-fidelity co-kriging surrogate model for ship hull form optimization, Ocean Engineering 243 (2022) 110239. doi:10.1016/j.oceaneng.2021.110239. URL http://dx.doi.org/10.1016/j.oceaneng.2021.110239
  26. [26]M. Meng, G. E. Karniadakis, A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse pde problems, Journal of Computational Physics 401 (2020) 109020. doi:10.1016/j.jcp.2019.109020. URL http://dx.doi.org/10.1016/j.jcp.2019.109020
  27. [27]M. Guo, A. Manzi, M. Amendt, P. Conti, J. S. Hesthaven, Multi-fidelity regression using artificial neural networks: Efficient approximation of parameter-dependent output quantities, Computer Methods in Applied Mechanics and Engineering 389 (2022) 114378. doi:10.1016/j.cma.2021.114378. URL http://dx.doi.org/10.1016/j.cma.2021.114378
  28. [28]S. Chakraborty, Transfer learning based multi-fidelity physics informed deep neural network, Journal of Computational Physics 426 (2021) 109942. doi:10.1016/j.jcp.2020.109942. URL http://dx.doi.org/10.1016/j.jcp.2020.109942
  29. [29]Y. Lyu, X. Zhao, G. X. Kong, X. Kang, W. Yao, Multi-fidelity prediction of fluid flow based on transfer learning using fourier neural operator, Physics of Fluids 35 (7) (jul 2023). doi:10.1063/5.0155555. URL http://dx.doi.org/10.1063/5.0155555
  30. [30]H. Tang, Q. Kong, J. P. Morris, Multi-fidelity fourier neural operator for fast modeling of large-scale geological carbon storage (2024). arXiv:2308.09113. URL https://arxiv.org/abs/2308.09113
  31. [31]T. Tripura, A. Thakur, S. Chakraborty, Multi-fidelity wavelet neural operator surrogate model for time-independent and time-dependent reliability analysis, Probabilistic Engineering Mechanics 77 (2024) 103672. doi:10.1016/j.probengmech.2024.103672. URL http://dx.doi.org/10.1016/j.probengmech.2024.103672
  32. [32]A. A. Howard, M. Perego, G. E. Karniadakis, P. Stinis, Multifidelity deep operator networks for data-driven and physics-informed problems, Journal of Computational Physics 493 (2023) 112462. doi:10.1016/j.jcp.2023.112462. URL http://dx.doi.org/10.1016/j.jcp.2023.112462
  33. [33]N. Renard, S. Deck, ZONAL DETACHED EDDY SIMULATION OF UNSTEADY AIRFOIL AERODYNAMICS, in: AERO 2020+1 - 55th 3AF International Conference on Applied Conference, Poitiers (virtuel), France, 2021, p. 1. URL https://hal.science/hal-03206330
  34. [34]D. Hendrycks, K. Gimpel, Gaussian error linear units (gelus) (2023). arXiv:1606.08415. URL https://arxiv.org/abs/1606.08415
  35. [35]S. Elfwing, E. Uchibe, K. Doya, Sigmoid-weighted linear units for neural network function approximation in reinforcement learning (2017). arXiv:1702.03118. URL https://arxiv.org/abs/1702.03118
  36. [36]A. F. Agarap, Deep learning using rectified linear units (relu) (2019). arXiv:1803.08375. URL https://arxiv.org/abs/1803.08375
  37. [37]Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, A. Anandkumar, Neural operator: Graph kernel network for partial differential equations (2020). arXiv:2003.03485. URL https://arxiv.org/abs/2003.03485
  38. [38]Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, A. Anandkumar, Fourier neural operator for parametric partial differential equations (2021). arXiv:2010.08895. URL https://arxiv.org/abs/2010.08895
  39. [39]M. Eldred, A. Giunta, S. Collis, Second-Order Corrections for Surrogate-Based Optimization with Model Hierarchies, American Institute of Aeronautics and Astronautics, 2004, Ch. 1, p. 1. arXiv:https://arc.aiaa.org/doi/pdf/10.2514/6.2004-4457, doi:10.2514/6.2004-4457. URL https://arc.aiaa.org/doi/abs/10.2514/6.2004-4457
  40. [40]T. D. Robinson, M. S. Eldred, K. E. Willcox, R. Haimes, Surrogate-based optimization using multifidelity models with variable parameterization and corrected space mapping, AIAA Journal 46 (11) (2008) 2814–2822. doi:10.2514/1.36043. URL http://dx.doi.org/10.2514/1.36043
  41. [41]L. Leifsson, S. Koziel, Multi-fidelity design optimization of transonic airfoils using physics-based surrogate modeling and shape-preserving response prediction, Journal of Computational Science 1 (2) (2010) 98–106. doi:10.1016/j.jocs.2010.03.007. URL http://dx.doi.org/10.1016/j.jocs.2010.03.007
  42. [42]Y. Zhang, N. H. Kim, C. Park, R. T. Haftka, Multifidelity surrogate based on single linear regression, AIAA Journal 56 (12) (2018) 4944–4952. doi:10.2514/1.J057299. URL http://dx.doi.org/10.2514/1.J057299
  43. [43]M. G. Fernández-Godino, S. Dubreuil, N. Bartoli, C. Gogu, S. Balachandar, R. T. Haftka, Linear regression-based multifidelity surrogate for disturbance amplification in multiphase explosion, Structural and Multidisciplinary Optimization 60 (6) (2019) 2205–2220. doi:10.1007/s00158-019-02387-4. URL http://dx.doi.org/10.1007/s00158-019-02387-4
  44. [44]M. G. Giselle Fernández-Godino, Review of multi-fidelity models, Advances in Computational Science and Engineering 1 (4) (2023) 351–400. doi:10.3934/acse.2023015. URL http://dx.doi.org/10.3934/acse.2023015
  45. [45]H. Babaee, P. Perdikarīs, C. Chrysostomidis, G. E. Karniadakis, Multi-fidelity modelling of mixed convection based on experimental correlations and numerical simulations, Journal of Fluid Mechanics 809 (2016) 895–917. doi:10.1017/jfm.2016.718. URL http://dx.doi.org/10.1017/jfm.2016.718
  46. [46]P. Perdikaris, M. Raissi, A. D. Manfredi, N. Lawrence, G. E. Karniadakis, Nonlinear information fusion algorithms for data-efficient multi-fidelity modelling, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 473 (2198) (2017) 20160751. doi:10.1098/rspa.2016.0751. URL http://dx.doi.org/10.1098/rspa.2016.0751
  47. [47]L. Lu, X. Meng, S. Cai, Z. Mao, S. Goswami, Z. Zhang, G. E. Karniadakis, A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data, Computer Methods in Applied Mechanics and Engineering 393 (2022) 114778. doi:10.1016/j.cma.2022.114778. URL http://dx.doi.org/10.1016/j.cma.2022.114778
  48. [48]K. Zakaria, A. N. Sinusuka, P. Palar, L. Zuhal, Prediction of smoke-inflow using recursive fourier neural operator dataset (2025). doi:10.5281/ZENODO.15629890. URL https://zenodo.org/doi/10.5281/zenodo.15629890
  49. [49]P. Holl, N. Thuerey, Φflow (PhiFlow): Differentiable simulations for pytorch, tensorflow and jax, in: International Conference on Machine Learning, PMLR, 2024.
  50. [50]P. Lippe, B. S. Veeling, P. Perdikaris, R. E. Turner, J. Brandstetter, PDE-Refiner: Achieving Accurate Long Rollouts with Temporal Neural PDE Solvers, in: Thirty-seventh Conference on Neural Information Processing Systems, 2023, p. 1. URL https://openreview.net/forum?id=Qv64681lWSarxiv.org/abs/2308.05732
  51. [51]G. Kohl, L.-W. Chen, N. Thuerey, Benchmarking autoregressive conditional diffusion models for turbulent flow simulation (2024). arXiv:2309.01745. URL https://arxiv.org/abs/2309.01745

Paper details

vlnc:W_zzfvt63m
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents