betaFeedback

WorksComputer Science

Structure-Preserving Operator Learning

Abstract

Learning complex dynamics driven by partial differential equations directly from data holds great promise for fast and accurate simulations of complex physical systems. In most cases, this problem can be formulated as an operator learning task, where one aims to learn the operator representing the physics of interest, which entails discretization of the continuous system. However, preserving key continuous properties at the discrete level, such as boundary conditions, and addressing physical systems with complex geometries is challenging for most existing approaches. We introduce a family of operator learning architectures, structure-preserving operator networks (SPONs), that allows to preserve key mathematical and physical properties of the continuous system by leveraging finite element (FE) discretizations of the input-output spaces. SPONs are encode-process-decode architectures that are end-to-end differentiable, where the encoder and decoder follows from the discretizations of the input-output spaces. SPONs can operate on complex geometries, enforce certain boundary conditions exactly, and offer theoretical guarantees. Our framework provides a flexible way of devising structure-preserving architectures tailored to specific applications, and offers an explicit trade-off between performance and efficiency, all thanks to the FE discretization of the input-output spaces. Additionally, we introduce a multigrid-inspired SPON architecture that yields improved performance at higher efficiency. Finally, we release a software to automate the design and training of SPON architectures.

Introduction

Partial differential equations (PDEs) underpin the modeling of many complex systems across science and engineering. However, traditional approaches such as the finite element method (FEM) are, in most cases, notoriously expensive, demand tailored solver configurations for each specific PDE, and cannot deal with scenarios where the underlying PDE modeling the system is unknown. Operator learning aims to address these limitations by approximating operators G:U→V\mathcal{G} : \mathcal{U} \to\mathcal{V} governed by PDEs, such as solution operators, directly from observational data [39] (Boullé & Townsend 2024), where U\mathcal{U} and V\mathcal{V} are infinite-dimensional function spaces. Operator learning has been successfully applied across different areas, including weather forecasting [40, 55, 36] or continuum mechanics [64].

Several operator learning architectures have been proposed, ranging from graph networks, which leverage relational inductive biases [56, 19], to neural operators, which rely on discretizations of integral operators defined on infinite-dimensional spaces [38, 46], and physics-based architectures [48] (Bélbute-Peres et al. 2020), where the PDE serves as an inductive bias to encode physical prior knowledge. However, these techniques often consider pointwise discretizations of the input and output functions that discard the continuous mathematical structure of the function spaces considered, leading to inconsistencies between the continuous and discrete representation of the operator, which deteriorate the operator approximation [4]. Hence, structural properties at the continuous level, such as symmetries, boundary conditions, or conservation laws, may not be preserved at the discrete level. In addition, most existing approaches discard the topological information of the underlying domain, thereby restricting them to simple geometries or meshes [45, 48].

In contrast, the finite-element method carries the topological information of the domain, and extensive literature has been devoted to structure-preserving discretizations, such as the finite element and exterior calculus framework (FEEC) [3]. While different approaches have been proposed for enforcing constraints in operator learning [35], a generic and consistent framework for designing operator learning architectures with structure-preserving spatial discretizations is lacking. Throughout this work, we denote structure-preserving discretizations as numerical methods that preserve, on the discrete level, key geometric, topological, and algebraic structures possessed by the original continuous system.

We introduce a family of structure-preserving operator learning architectures, called structure-preserving operator networks (SPON), that are expressed as encoder-processor-decoder models. The encoder and decoder result from the finite element (FE) discretization of the input-output spaces and the processor operates on FE degrees of freedom. As a result, SPON architectures are capable of naturally preserving key properties of the continuous operator thanks to FE discretizations, which can be tailored to specific scientific applications. Moreover, structure-preserving operator networks can operate on complex geometries and meshes, preserve certain boundary conditions exactly at the discrete level, while offering theoretical guarantees on the approximation error. We also demonstrate that our framework exhibits mesh-invariant capabilities and provide an explicit way to control the operator aliasing error via the discretization employed. Our framework achieves higher accuracy than several state-of-the-art architectures on a classical benchmark. Finally, we introduce a multigrid-based SPON that can be scaled to large problems and captures long-range information, while achieving high performance and accuracy.

Main contributions. Our main contributions are summarized as follows.

  1. We propose a generic and flexible framework for operator learning that combines the finite element method with the encode-process-decode paradigm, allowing for the preservation of key properties of the continuous system at the discrete level using finite element discretizations tailored to the PDE of interest. Our framework can be used on complex meshes, for time-dependent and steady problems, and comes with theoretical guarantees.

  1. We introduce a multigrid-based structure-preserving operator network (SPON-MG) that combines multilevel message passing GNNs with finite element mapping operators. SPON-MG achieves greater accuracy with significantly higher efficiency while greatly reducing the number of parameters needed, resulting in lower memory usage and improved latency.

We also release an open-source library at https://github.com/sponproject/spon interfacing with the Firedrake FE software [30] for building structure-preserving operator networks using state-of-the-art FE discretizations.

Background and Related Work

Neural operators. Kovachki et al. [38] introduced neural operators as infinite-dimensional generalizations of neural networks to discretize and approximate operators associated with PDEs. Several architectures have been proposed and all result from a specific parametrization of the integral kernel (Boullé & Townsend, 2024). Examples include Fourier neural operators (FNO) [45], which employ Fourier convolutional kernels, DeepONet (Lu et al., 2021), that learns the mapping between Hilbert spaces to a finite-dimensional latent space using encoder-decoder architectures [39], and Boullé et al. [11] that learns Green’s functions. Such approaches consider samples of the input-output functions at point values or on a tensor-product grid and often discard the intrinsic structures of the underlying continuous PDE systems, leading to aliasing errors [4]. In contrast, we consider a FEM discretization of the input-output spaces, which allows preserving key properties of the continuous spaces via the discretization, and are applicable to complex meshes. Notably, the aliasing error can be explicitly controlled by the choice of discretization.

Operator learning and FEM. Several related works explored connections between operator learning and the finite element method (FEM) [21, 27, 32, 42, 62]. The MgNO introduced by He et al. [32] uses a multigrid approach to discretize integral operators on simple geometries with tensor-product grids, and requires custom-defined convolution kernels to enforce specific boundary conditions. Franco et al. [27] proposed a mesh-informed neural network for operator learning that uses a dense feedforward model along with a mesh that uses a pruning strategy to dismiss far points. Finally, Lee et al. [42] considered the predictions of the neural operator at degrees of freedom, in cases where they coincide with the mesh vertex nodes, and used continuous Lagrange elements of degree one (CG1_1).

GNN simulators. GNN-based methods have also been proposed for learning operators driven by PDEs. Pfaff et al. [56] introduced an encode-process-decode GNN architecture (Battaglia et al. [5]) followed by a time integrator capable of learning mesh-based quantities. In (Brandstetter et al. [19]), a message passing neural PDE solver is considered to learn solution operators, along with a stabilization technique to train autoregressive operators. Finally, Belbute-Peres et al. (2020) embedded a differentiable CFD solver into a GNN to improve generalization. Our framework contrasts with these GNN architectures by using FEM discretizations to design a family of GNN architectures that preserve continuous structure of the operator at the discrete level, while being compatible with existing autoregressive techniques for time modeling.

Physics-based approaches. Different works explored the use of physics-based inductive biases for machine learning algorithms. Examples include the use of the PDE as a regularization term in the loss ([48]), constraining the architecture to enforce certain boundary conditions ([59]), or the design of neural networks that comply with thermodynamics principles ([33]) or preserve structures of kinetic collision operators ([43]). In contrast, our approach relies on a FEM-based inductive bias that allows preserving mathematical properties of the continuous operator at the discrete level, offers theoretical guarantees, and allows tackling problems defined on complex geometries. Notably, the use of FEM discretizations facilitates the combination of complex physics-based inductive biases with SPON models ([17]).

Method

Our main motivation is to learn a (typically nonlinear) operator G:U→V\mathcal{G} : \mathcal{U} \to\mathcal{V} associated with a PDE (e.g., solution operator or inverse problem), where U\mathcal{U} and V\mathcal{V} are Hilbert spaces of functions defined on a bounded domain Ω⊂Rd\Omega\subset\mathbb{R}^{d} in spatial dimension d∈{1,2,3}d \in\{1, 2, 3\}. For simplicity, we consider U\mathcal{U} and V\mathcal{V} to be defined on the same domain Ω\Omega, but different bounded domains and spatial dimensions may be considered. The spaces U\mathcal{U} and V\mathcal{V} arising from such problems are typically infinite-dimensional and need to be discretized. Similarly to FEM, we consider a mesh M\mathcal{M} of the domain Ω\Omega, and two finite-dimensional spaces Uh\mathcal{U}_{h} and Vh\mathcal{V}_{h} arising from a suitable discretization of the spaces U\mathcal{U} and V\mathcal{V}. We introduce a framework for designing operator learning architectures, which we refer to as structure-preserving operator networks, that approximate G\mathcal{G} on the discretized spaces Uh\mathcal{U}_{h} and Vh\mathcal{V}_{h}.

Structure-preserving operator network

We define a structure-preserving operator network (SPON) Sθ\mathcal{S}_{\theta} between the finite-dimensional functions spaces Uh\mathcal{U}_{h} and Vh\mathcal{V}_{h} of dimensions nn and mm as

Sθ(f)=D∘Pθ∘E(f),f∈Uh,(1)\mathcal{S}_{\theta}(f) = \mathcal{D} \circ\mathcal{P}_{\theta} \circ\mathcal{E}(f), \qquad f \in\mathcal{U}_{h}, \tag*{(1)}

where E\mathcal{E} and D\mathcal{D} denote the encoder and decoder, while Pθ:Rn→Rm\mathcal{P}_{\theta}: \mathbb{R}^{n} \to\mathbb{R}^{m} is a learnable model of parameters θ\theta, referred to as the processor (see Figure 1).

Figure 1. Schematic diagram of a structure-preserving operator network architecture.

Encoder. The encoder maps an input function f∈Uhf \in\mathcal{U}_{h} to its degrees of freedom in the finite element space Uh=span⁡(φ1,…,φn)\mathcal{U}_{h} = \operatorname{span}(\varphi_{1}, \ldots, \varphi_{n}) as

E(f)=(f1,…,fn),(2)\mathcal{E}(f) = (f_{1}, \ldots, f_{n}), \tag*{(2)}

where fi=⟨f,φi⟩f_{i} = \langle f, \varphi_{i}\rangle denotes the Galerkin projection onto the ii-th basis function φi\varphi_{i}.

Decoder. The decoder maps the predicted degrees of freedom in Vh\mathcal{V}_h to the reconstructed solution u∈Vhu \in\mathcal{V}_h as

D(u1,…,um)=u,(3)\mathcal{D}(u_1,\ldots,u_m)=u, \tag*{(3)}

where u(x)=∑i=1muiϕi(x)u(x)=\sum_{i=1}^{m}u_i\phi_i(x), for x∈Ωx\in\Omega, and with (ϕi)1≤i≤m(\phi_i)_{1\leq i\leq m} a basis of Vh\mathcal{V}_h.

The structure-preserving operator network framework combines the finite element method with the encode-process-decode paradigm. The finite element method can be seen as an encode-process-decode approach, with encoder E\mathcal{E} and decoder D\mathcal{D}, and where the processor numerically solves the discretized system posed on the degrees of freedom (DoFs). On the other hand, classical encode-process-decode architectures [5] consider learnable models for the encoder, processor, and decoder. In contrast, only the processor Pθ\mathcal{P}_{\theta} is learnable in our approach. In that sense, our framework can be seen as an operator learning approach over a structured latent space.

SPON architectures separate the concerns of the latent space representation, which relies on the finite element discretization, from the learning of the operator, which is delegated to the processor. The FEM-based encoder and decoder allow to leverage the rich literature on efficient structure-preserving finite element discretizations of PDE systems to preserve the structure of the input-output spaces. In particular, several key properties are naturally preserved for SPON architectures. This includes the preservation of the function space of the output, which is always Vh\mathcal{V}_h independently of Pθ\mathcal{P}_{\theta}. In fact, for conforming FE discretizations [18 Chapt. 2], the output also lies in V\mathcal{V} as in this is case we have Vh⊂V\mathcal{V}_h\subset\mathcal{V}. Additionally, SPON architectures can impose boundary conditions such as Dirichlet exactly at the discrete level. Other mathematical and physical properties may be satisfied at the discrete level using structure-preserving FE discretizations [3].

The choice of the discretized spaces is problem-specific, and different discretizations can be employed to incorporate prior information about the regularity of the input-output functions. Structure-preserving operator networks also inherit other FEM benefits, such as allowing arbitrary geometric decompositions, which facilitate the use of complex geometries, while offering an easy way to strike favorable trade-offs between training/inference time and accuracy through mesh refinement or higher-order spatial discretizations. One can, however, still provide pointwise values as inputs to the encoder and then use a suitable Galerkin projection to obtain the corresponding degrees of freedom.

Relational inductive bias. Since the spaces Uh\mathcal{U}_h and Vh\mathcal{V}_h are finite element spaces, the basis functions (φi)1≤i≤n(\varphi_i)_{1\leq i\leq n} and (ϕi)1≤i≤m(\phi_i)_{1\leq i\leq m} are typically piecewise polynomial functions with compact support, which results in sparse representations as most basis functions decouple, i.e., their supports do not intersect. More precisely, the value of a given function f∈Uhf\in\mathcal{U}_h at a point x∈Ωx\in\Omega depends only on the small set of basis functions whose supports contain xx. A sparse representation induces a graph whose vertices correspond to the degrees of freedom (fi)1≤i≤n(f_i)_{1\leq i\leq n} of Uh\mathcal{U}_h, and where two vertices fif_i and fjf_j admits an edge only if the corresponding basis functions φi\varphi_i and φj\varphi_j have overlapping support (see Appendices A.1 and A.2 for details). We use this graph representation as a relational inductive bias for the processor Pθ\mathcal{P}_{\theta} that maps the graph associated with the degrees of freedom of Uh\mathcal{U}_h to the ones of Vh\mathcal{V}_h, as illustrated in Figure 1. This motivates the use of graph neural network architectures for Pθ\mathcal{P}_{\theta}.

Boundary conditions. The function spaces U\mathcal{U} and V\mathcal{V} may be equipped with boundary conditions. For example, V=H01(Ω)\mathcal{V}=H_0^1(\Omega) contains functions that vanish on the boundary ∂Ω\partial\Omega. These continuous structures need to be preserved at the discrete level. Structure-preserving operator networks automatically conserve such boundary conditions at the discrete level, independently of the choice of the processor Pθ\mathcal{P}_{\theta}. This is achieved by the decoder, as outlined in Figure 1, which enforces boundary conditions directly on the degrees of freedom. This allows to output functions that satisfy boundary conditions such as Dirichlet conditions exactly (see Appendix A.3).

Time-dependent operators. While our framework primarily addresses the spatial discretization of operators between function spaces, SPON architectures can also be used for time-dependent problems. The SPON “function-to-function” interface provides a composable and flexible way to model time-dependent operators and can seamlessly be integrated with standard temporal modeling approaches, such as autoregressive modeling [40] (Pfatt et al. 2021), neural operators [45], or temporal bundling [19].

Zero-shot super resolution. The structure-preserving operator network Sθ\mathcal{S}_{\theta} produces a finite element function u∈Vhu\in\mathcal{V}_h, which can be evaluated at any point xx in the geometrical domain Ω\Omega via u(x)=∑i=1muiϕi(x)u(x)=\sum_{i=1}^{m}u_i\phi_i(x), independently of the mesh and resolution it was trained on. This powerful property results from the FE discretization of U\mathcal{U} and V\mathcal{V} and holds even for complex geometries. Such a property is highly desirable to transfer solutions between different meshes and space discretizations, e.g., for zero-shot super resolution, and yields architectures that can operate across different resolutions. Let Sθ:Uh→VhS_\theta: U_h \to V_h, we can construct the SPON interpolation operator S^θ:Xh→Yh\hat{S}_\theta: X_h \to Y_h:

S^θ=P∘Sθ∘R,(4)\hat{S}_\theta= \mathcal{P} \circ S_\theta\circ\mathcal{R}, \tag*{(4)}

where XhX_h and YhY_h are discrete spaces defined on a different resolution and/or different finite element discretization than UhU_h and VhV_h, and with R:Xh→Uh\mathcal{R} : X_h \to U_h and P:Vh→Yh\mathcal{P} : V_h \to Y_h the corresponding finite element interpolation operators. In the zero-shot super resolution case, R\mathcal{R} and P\mathcal{P} are merely the restriction and prolongation operators. Note that S^θ\hat{S}_\theta and SθS_\theta have the same number of parameters.

A multigrid-inspired processor

The latent graph representations of SPONs promote the use of GNN architectures for the processor. The corresponding graphs intrinsically depend on the mesh and the chosen finite element discretizations. For example, higher-resolution meshes or higher-order discretizations lead to larger graphs. However, the use of standard GNNs such as message-passing architectures on large graphs is doomed by significantly higher computational cost and information bottleneck, i.e., the model cannot capture long-range information since a GNN with mm layers can only capture information up to mm hops away. While increasing the number of message passings helps propagate information through the graph, it also increases the computational load and may lead to over-smoothing.

To address these limitations, we introduce a multigrid-inspired processor PθMG\mathcal{P}^{MG}_\theta that operates on a hierarchy of meshes and function spaces to provide greater accuracy with higher efficiency. Our processor yields a structure-preserving operator network that can scale to highly resolved meshes and/or high-order discretizations, while efficiently capturing long-range dependencies between distant regions in the domain. Our multilevel processor combines lightweight message passing architectures (φi)1≤i≤N(\varphi_i)_{1 \leq i \leq N} at each level, which models information exchange at different length scales, with a larger graph-based architecture ψ\psi that facilitates the propagation of information at the coarse level. The computational load is delegated to the coarse model ψ\psi, which performs message-passing updates that are significantly cheaper than the fine level, thereby increasing the computational efficiency. This processor contrasts with existing multiscale GNN approaches [26, 44, 40], where each scale is only defined by a given mesh resolution. Our approach combines the mesh resolution with the FE discretization of the input-output spaces, associating each level with a pair of function spaces. It may be described as a functional multilevel message passing since the latent features across the different scales can all be associated with a given function in a known function space. Then, mapping latent features from one space to another is achieved using appropriate operators between the FE spaces.

We define Rhi:Uhi→Uhi+1\mathcal{R}_h^i : U_h^i \to U_h^{i+1}, Phi:Vhi+1→Vhi\mathcal{P}_h^i : V_h^{i+1} \to V_h^i, and Ihi:Uhi→Vhi\mathcal{I}_h^i : U_h^i \to V_h^i, the restriction, prolongation, and interpolation operators, respectively. These operators are used across the processor architecture to map latent features. This is achieved through a sparse matrix-vector product, where the matrix results from the discretization of the operator, and the vector contains the latent degrees of freedom. This additional inductive bias drastically decreases the number of parameters needed, as the matrices do not have to be learned, unlike other approaches [44], and reduces the model latency.

Notably, the matrices induced by (Rhii)i(\mathcal{R}^{i}_{h_i})_i, (Phii)i(\mathcal{P}^{i}_{h_i})_i, and (Ihii)i(\mathcal{I}^{i}_{h_i})_i are sparse, which reduces memory and allows for larger batch sizes, leading to higher throughput. The architecture of PθMG\mathcal{P}^{MG}_{\theta}, outlined in Figure 2, is inspired by the V-cycle multigrid algorithm and multilevel message passing methods. It consists of a downward pass, a coarse update, and an upstream pass. See Appendix § 9 for more details.

Figure 2. Diagram of the multigrid processor PθMG\mathcal{P}^{MG}_\theta for 3 levels (right) with the corresponding mesh hierarchy (left). PθMG\mathcal{P}^{MG}_\theta takes in fˉ\bar{f}, the input DoFs, and predicts uˉ\bar{u}, the DoFs of the output.

Downward pass. Let 2≤i≤N−12 \le i \le N-1, the downward pass is defined as zi↓=φi(Rhi−1i−1zi−1↓)z_i^{\downarrow} = \varphi_i(\mathcal{R}^{i-1}_{h_{i-1}}z_{i-1}^{\downarrow}), along with z1↓=φ1(fˉ)z_1^{\downarrow} = \varphi_1(\bar{f}) and zN↓=RhN−1N−1zN−1↓z_N^{\downarrow} = \mathcal{R}^{N-1}_{h_{N-1}}z_{N-1}^{\downarrow}.

Coarse update. The coarse update zN↑=ψ(zN↓)z_N^{\uparrow} = \psi(z_N^{\downarrow}) consists of a forward pass through a messaging-passing-based model ψ\psi with linear encoding and decoding.

Upward pass. Let 1≤i≤N−11 \le i \le N-1, the upward pass is defined as zi↑=φi(Ii−1zi+1↓⊕Pizi+1↑)z_i^{\uparrow} = \varphi_i(\mathcal{I}_{i-1}z_{i+1}^{\downarrow} \oplus\mathcal{P}_{i}z_{i+1}^{\uparrow}), where ⊕\oplus is a differentiable aggregator such as averaging, or a learnable linear combination. We then return uˉ=z1↑\bar{u} = z_1^{\uparrow}.

Approximation error

This section provides an error estimate for the approximation of a Lipschitz continuous operator G\mathcal{G} by a structure-preserving operator network Sθ\mathcal{S}_{\theta}. Let Ω1⊂Rn1\Omega_1 \subset\mathbb{R}^{n_1} and Ω2⊂Rn2\Omega_2 \subset\mathbb{R}^{n_2} be two open bounded domains, U\mathcal{U} be a compact subset of a Hilbert space of functions Hk1(Ω1)H^{k_1}(\Omega_1), for k1≥0k_1 \ge0, defined as U={f∈Hk1(Ω1),∥f∥≤1}\mathcal{U} = \{f \in H^{k_1}(\Omega_1), \|f\| \le1\}, and V=Hk2(Ω2)\mathcal{V} = H^{k_2}(\Omega_2) for some k2≥0k_2 \ge0. Let Uh1\mathcal{U}_{h_1} and Vh2\mathcal{V}_{h_2} be two finite element spaces of functions defined on regular (in the sense of Brenner & Scott 2008, Def. 4.4.13) meshes Mh1\mathcal{M}_{h_1} and Mh2\mathcal{M}_{h_2} of Ω1\Omega_1 and Ω2\Omega_2, respectively, with polynomial degree k1k_1 and k2k_2, and maximum diameter h1h_1 and h2h_2. We assume that Uh1\mathcal{U}_{h_1} and Vh2\mathcal{V}_{h_2} are conforming finite element spaces (i.e., Uh1⊂U\mathcal{U}_{h_1} \subset\mathcal{U} and Vh2⊂V\mathcal{V}_{h_2} \subset\mathcal{V}) satisfying the standard finite element hypotheses of Brenner & Scott (2008, Thm. 4.4.4). In addition, we denote by PU:U→Uh1P_{\mathcal{U}} : \mathcal{U} \to\mathcal{U}_{h_1} the Galerkin interpolation.

Theorem 1 (Approximation bound). Let G:Hs1(Ω1)→V\mathcal{G} : H^{s_1}(\Omega_1) \to\mathcal{V} be a Lipschitz continuous operator for some 0≤s1≤k10 \le s_1 \le k_1 and 0<ϵ<10 < \epsilon< 1. There exists a structure-preserving operator network Sθ:Uh→Vh⊂V\mathcal{S}_{\theta} : \mathcal{U}_{h} \to\mathcal{V}_{h} \subset\mathcal{V} with a number of parameters bounded by

∣θ∣<C1ϵ−C2/h1k1(log⁡(1/ϵ)+1),|\theta| < C_1\epsilon^{-C_2/h_1^{k_1}} \left(\log(1/\epsilon) + 1\right),

such that for all f∈Uf \in\mathcal{U} and 0≤s2≤k20 \le s_2 \le k_2,

∥(G−Sθ∘PU)(f)∥Hs2(Ω2)≤C3(h1k1−s1∥f∥Hk1(Ω1)+h2k2−s2∥u∥Hk2(Ω2))+ϵ(h),(5)\left\|(\mathcal{G} - \mathcal{S}_{\theta} \circ P_{\mathcal{U}})(f)\right\|_{H^{s_2}(\Omega_2)} \le C_3\left(h_1^{k_1-s_1}\|f\|_{H^{k_1}(\Omega_1)} + h_2^{k_2-s_2}\|u\|_{H^{k_2}(\Omega_2)}\right) + \epsilon(h), \tag*{(5)}

where u=G(f)u = \mathcal{G}(f), C1>0C_1 > 0 is a constant independent of ϵ\epsilon, and C2,C3>0C_2, C_3 > 0 do not depend on h1h_1, h2h_2, ϵ\epsilon.

The first two terms denote the finite element error on the input and output spaces with respect to h1h_1 and h2h_2, while the last term in Eq. (5) characterizes the quality of the neural network approximation as the number of parameters increases. The left-hand side in Eq. (5) can be seen as an operator aliasing error [4], which can be explicitly controlled by the mesh resolution and the discretization of the input-output spaces. The proof of Theorem 1 is deferred to Appendix § 10 and combines standard finite element bounds along with an error analysis similar to the ones derived by Kovachki et al. [37]; Lanthaler et al. [41] for FNOs and DeepONets.

Software

We release an open-source software, spon, for designing structure-preserving operator networks that interface with the Firedrake Ham et al., [30], PyTorch Paszke et al., [54], and physics-driven-ml Bouziani & Ham, [16] packages. The spon package leverages the Firedrake API for specifying meshes, as well as a rich collection of finite element spaces, including elements such as Lagrange, discontinuous Galerkin, and Raviart-Thomas. The Firedrake interface also allows for the specification of complex boundary conditions or input functions from observable point data, which is convenient for certain operator learning applications. Our software automates the encoder and decoder of SPON architectures, which includes the construction of the latent graphs and boundary conditions support. Additionally, our package facilitates the construction of the mesh and function space hierarchies for multigrid processors, along with the restriction, prolongation, and interpolation operators for nested and non-nested meshes. Listing 1 outlines how SPON models can be implemented using our interface. Moreover, recent advances in differentiable programming Bouziani et al., [17]; Bouziani & Ham [15] enable SPONs to be coupled with PDE constraints implemented in Firedrake.

Listing 1: Outline of the spon interface. A SPON model is defined in line 16, mapping from UhU_h to VhV_h, where UhU_h (resp. VhV_h) results from a continuous Lagrange discretization of degree 1 (resp. 2) as defined in line 6 (resp. line 7). Homogeneous Dirichlet boundary conditions are specified on the entire domain boundary in line 10.

1  import firedrake as fd
2  import spon
3  ...
4  # Define the discretized function spaces Uh and Vh
5  mesh = ...
6  U = fd.FunctionSpace(mesh, "CG", 1)
7  V = fd.FunctionSpace(mesh, "CG", 2)
8
9  # Define boundary conditions
10 bcs = fd.DirichletBC(V, Constant(0.), "on_boundary")
11
12 # Define the processor Pθ
13 processor = ...
14
15 # Define the structure-preserving operator network Sθ
16 S = spon.SPON(U, V, processor=processor, bcs=bcs)

Numerical experiments

We evaluate the performance of our framework on several numerical examples pertaining to different physics and discretizations. We begin with a standard Poisson problem in Section 4.1 to demonstrate key properties of our framework and compare it with state-of-the-art architectures. In Section 4.2, we learn the time-dependent incompressible Navier-Stokes solution operator in a quasi-turbulent regime, on a complex geometry and with boundary conditions using a highly refined mesh. Additional details are provided in Appendix D, and further experiments are conducted in Appendix E.1.

Poisson's equation with strong boundary conditions

We consider a 2D Poisson equation on the unit square Ω=[0,1]2\Omega= [0,1]^2 with a Dirichlet condition:

−∇2u=fin Ω,u=gon ΓD,(6)-\nabla^2 u = f \quad\text{in } \Omega,\qquad u = g \quad\text{on } \Gamma_D, \tag*{(6)}

where f∈H1(Ω)f \in H^1(\Omega), g∈C∞(ΓD)g \in C^\infty(\Gamma_D), and ΓD\Gamma_D is the top boundary of Ω\Omega. We aim to approximate the corresponding solution operator G:H1(Ω)→H2(Ω)∩H01(Ω)\mathcal{G}: H^1(\Omega) \to H^2(\Omega) \cap H^1_0(\Omega) that maps the source term ff to the solution uu of Eq. (6). The performance and efficiency of our approach are compared with FNO (Li et al., [45]) and DeepONet (Lu et al., 2021). Due to the requirements of FNO and DeepONet, we consider a uniform nx×nxn_x \times n_x grid, with different resolutions ranging from nx=16n_x = 16 to 256. To facilitate the comparison with benchmarks, we use the same discretization for the input and output spaces, namely a CG1 finite element space. We consider two types of structure-preserving operator networks for this experiment: one with the single-level processor ψ\psi (see Eq. (10)), which we simply refer to as SPON, and the multigrid processor introduced in § 3.2, which we denote by SPON-MG.

Our experiments demonstrate that our framework outperforms the benchmarks by a significant margin for both the single-level and multigrid architectures, as illustrated in Figure 3a and Table 1. We also display the preservation of the Dirichlet boundary condition from the continuous output space at the discrete level, resulting in exact matching of the boundary condition on ΓD\Gamma_D (cf. Table 1), unlike other methods. We evaluate the performance of our multigrid processor, which results in a model that surpasses the best benchmark we compare it with while having 3.5 times fewer parameters, illustrating the benefits of the multigrid FE structure we employed. The single-level SPON architecture stands out as the fastest approach. However, we note that SPON-MG is slower than SPON at this resolution due to the overhead induced by the mappings across levels. This is compensated at higher resolutions, where SPON-MG is significantly faster while massively reducing the number of parameters, as illustrated in Figure 11a. We emphasize that our framework primarily aims to provide a flexible way to preserve continuous structure at the discrete level while allowing for arbitrary geometries, rather than to outperform existing architectures.

Relative errors across epochs and resolutions for different benchmarks

Figure 3. Left: Relative errors across the epochs for different benchmarks (trained and tested at nx=64n_x = 64). Middle: Relative errors of SPON and SPON-MG trained at nx=64n_x = 64 and evaluated at different resolutions Right: Relative errors when trained and evaluated at different resolutions.

MethodParametersEpoch timeRelative L2L^2-error∥u−g∥L2(ΓD)/∥g∥L2(ΓD)\lVert u-g\rVert_{L^2(\Gamma_D)} / \lVert g\rVert_{L^2(\Gamma_D)}
FNO1, 200, 2251.49s1.71×10−21.71 \times10^{-2}1.85×10−11.85 \times10^{-1}
DeepONet3, 484, 4172.38s8.09×10−28.09 \times10^{-2}4.38×10−14.38 \times10^{-1}
SPON3, 568, 2701.19s4.52×10−34.52 \times10^{-3}0
SPON-MG338, 8272.25s3.21 \times 10^{-3}0

Table 1. Benchmarks on Poisson (64 × 64 resolution for both training and testing).

Zero-shot super resolution. We compare the performance of both SPON models trained at a fixed resolution and evaluated at different resolutions using the SPON interpolation operator, see Eq. (4). We observe in Figure 3b that both models exhibit mesh-invariance capabilities similar to FNO [45], i.e., their performance remains constant when evaluated on finer resolutions.

Discretization dependence. We compare the approximation error of both SPON models trained for different resolutions. We show that as the resolution becomes finer, the approximation error decreases, as outlined in Figure 3c, which is in agreement with Theorem 1 since resolution and polynomial order are two ways by which the approximation error of structure-preserving operator networks can be improved. Note that the CG1_1 approximation used to discretize the input-output spaces should result in a linear convergence rate with respect to hh (=1/nx=1/n_x). However, this is not directly observable from Figure 3c since the ϵ(h)\epsilon(h) term in Theorem 1 is different for each resolution nxn_x.

Multigrid processor. We compare the efficiency of SPON and SPON-MG at different resolutions in Appendix E.2. Our main finding is that SPON-MG achieves significantly higher efficiency above a certain resolution, while consistently requiring less parameters across all resolutions, thereby further improving latency (see Figure 11).

Epoch time plot comparing SPON and SPON-MG

(a) (a) Epoch time

Number of parameters plot comparing SPON and SPON-MG

(b) (b) Number of parameters

Figure 11. Comparison of the epoch time (left) and the number of parameters (right) for the single-level structure-preserving operator network (SPON) and the multigrid-based structure-preserving operator network (SPON-MG).

Fluid flow past a cylinder

In this example, we aim to learn the time-forward operator u(⋅,t)↦u(⋅,t+Δt)u(\cdot,t) \mapsto u(\cdot,t+\Delta t) associated with a classical cylinder flow benchmark [34], for t≥0t \ge0 and a timestep Δt>0\Delta t > 0. The fluid velocity u:Ω×[0,T]→R2u : \Omega\times[0,T] \to\mathbb{R}^2 is governed by the incompressible Navier–Stokes equations:

∂u∂t−∇⋅2Reε(u)+(u⋅∇)u+∇p=fin Ω×(0,T],∇⋅u=0in Ω×[0,T],(7)\frac{\partial u}{\partial t} - \nabla\cdot\frac{2}{\mathrm{Re}}\varepsilon(u) + (u \cdot\nabla)u + \nabla p = f \quad\text{in } \Omega\times(0,T], \qquad \nabla\cdot u = 0 \quad\text{in } \Omega\times[0,T], \tag*{(7)}

where pp is the pressure, Ω⊂R2\Omega\subset\mathbb{R}^2 is the computational domain, Re=200\mathrm{Re} = 200 is the Reynolds number, and ε(u)=(∇u+∇uT)/2\varepsilon(u) = (\nabla u + \nabla u^{\mathsf{T}})/2. We equip Eq. (7) with the following boundary conditions: u=(1,0)Tu = (1,0)^{\mathsf{T}} on the upper and lower sides of Ω\Omega, a homogeneous Dirichlet boundary condition on the obstacle, and a “do-nothing” condition on the right side of Ω\Omega (see Figure 4). We consider fluid velocities up to T=30T = 30 and Δt=2\Delta t = 2, which include laminar flow as well as vortex structures. The dataset is formed by considering different random inflow conditions on the left boundary. The discrete space Uh  (=Vh)\mathcal{U}_h \;(= V_h) is derived using the standard Taylor-Hood finite elements, a stable element for the Stokes equation [60]. We consider an unstructured mesh to represent the non-trivial geometry Ω\Omega. The discrete spaces result in a graph of about 40k40k nodes and 800k800k edges, highlighting the computational challenge of this example. Additionally, the different source terms, the autoregressive nature of the network, and the change in physics occurring along the way, from laminar to periodic with complex von Kármán vortex street patterns, present additional challenges for the network to capture. We consider the multigrid processor PθMG\mathcal{P}_{\theta}^{MG} for our SPON architecture

Figure 4. Left: Exact fluid velocity flow and magnitude from a random source in the test set (left boundary condition) at different time steps in the simulation. Right: Predicted velocity solution from the same initial condition at t=2.2t = 2.2. The bottom row (highlighted in blue) is an extrapolation test as the time step has not been observed in training.

and generate a hierarchy of non-uniform meshes, as shown in Figure 7. Our model inherently satisfies the strong boundary conditions imposed on the obstacle and on the upper and lower boundaries of Ω\Omega exactly. The one-step modeling of the velocity [56] allows generating long trajectories at inference time via iterative application of the model. However, it makes the task more difficult as such operators are prone to error accumulations. For training, we use a divergence-free regularization on the loss function [17], which can be interpreted as an incompressibility constraint.

Mesh hierarchy for the Navier-Stokes problem

Figure 7. Mesh hierarchy considered for the Navier-Stokes problem in § 4.2, where the coarse mesh (bottom), medium mesh (middle), and fine mesh (top) are composed of 1850, 3830, and 9069 triangles, respectively.

Despite a relatively large timestep that intentionally misses fast-scale dynamics, we observe a good agreement between the ground truth and predicted solution, even at the last time step corresponding to a high number of model rollouts, as illustrated in Figure 4. We achieve a relative L2L^2-error of 1.3×10−11.3 \times10^{-1} on the test set (7.1×10−27.1 \times10^{-2} on the training set). Moreover, we observe relatively robust extrapolation capabilities of the model, demonstrating low errors on time steps not seen during training, see Figure 8 and Figure 9. The training and inference time of our architecture is dominated by the message passing updates occurring at the finest level in PθMG\mathcal{P}_{\theta}^{MG}. This can be attributed to the high number of nodes at that level and can be mitigated by reducing the number of message-passing layers specifically for that level or by using standard approaches for handling GNNs on large graphs. We highlight that this problem is highly challenging for most operator learning architectures due to the complex geometry with unstructured data points and long-term integration.

Figure 8. Relative error in the prediction of the velocity field for the Navier–Stokes problem, with the same initial condition as in Fig. 4. The left side of the figure displays the generalization capabilities of the model to new initial conditions, while the right side shows the extrapolation capabilities to new time steps, unseen during training.

Figure 9. Left: Exact fluid velocity flow and magnitude from the random source (left boundary condition) in Figure 4 at different time steps in the simulation. Right: Predicted velocity solution from the same initial condition at t=2.2t = 2.2. The rows highlighted in blue are extrapolation tests, i.e., they correspond to time steps that have not been observed during training.

Conclusions

Structure-preserving operator networks provide a generic and flexible framework for learning operators modeling complex physical systems while preserving important properties of the system of interest at the discrete level. Our approach leverages the rich literature on finite element discretizations and can be tailored to specific physics. SPONs can operate on complex geometries and meshes, can be used in conjunction with time-dependent approaches, and demonstrate state-of-the-art performance with mesh-invariance capabilities. Structure-preserving operator networks come with theoretical guarantees and have an approximation error that can be explicitly reduced through discretization. Our multigrid processor facilitates scaling to larger problems and demonstrates greater performance with higher efficiency, while massively reducing the number of parameters. The FEM structure of our framework can be exploited to develop SPON variants that incorporate physical prior knowledge, which may improve generalization and increase accuracy.

Limitations. The finite element method is naturally suited for problems with spatial dimension d≤3d \le3. Our primary motivation is to address these problems. The application of SPONs to higher-dimensional problems can result in dense graphs, making the naive implementation of GNN-based processors prohibitively expensive. However, techniques such as sampling [31] or similar methods can be employed to mitigate this issue.

Acknowledgments

Nacime Bouziani was supported by the Eric and Wendy Schmidt AI in Science Postdoctoral Fellowship, a Schmidt Futures program. Nicolas Boullé was supported by the Office of Naval Research (ONR), under grant N00014-23-1-2729.

Appendix

Structure-preserving operator networks

Finite element discretization

The finite element method (FEM) is a numerical method to approximate the solution of partial differential equations. Given that PDE systems are naturally posed on infinite-dimensional spaces, discretization is required to solve these systems on a computer. Given that this work lies at the intersection between machine learning and numerical PDE solvers, we provide some background material on the finite element discretization for the readers not familiar with these methods.

Let V\mathcal{V} be an infinite-dimensional function space defined on a bounded domain Ω⊂Rd\Omega\subset\mathbb{R}^{d}. The conforming finite element approach aims to approximate a solution u∈Vu \in\mathcal{V} to a given PDE on a finite-dimensional subspace Vh⊂V\mathcal{V}_{h} \subset\mathcal{V}, characterized by a basis (ϕi)1≤i≤N(\phi_{i})_{1 \le i \le N}, where N=dim⁡(Vh)N = \dim(\mathcal{V}_{h}). The approximation uhu_{h} to uu can be written as uh=∑i=1Nuiϕiu_{h} = \sum_{i=1}^{N}u_{i}\phi_{i}, and is determined by the coefficients (ui)1≤i≤N∈RN(u_{i})_{1 \le i \le N} \in\mathbb{R}^{N}, referred to as degrees of freedom, for a given choice of basis functions.

In a classical FEM context, the degrees of freedom are computed by solving the finite-dimensional system resulting from the discretization of the PDE of interest. In the proposed structure-preserving operator learning framework, the degrees of freedom are obtained by a structure-preserving operator network Sθ\mathcal{S}_{\theta}. Notably, SPONs are a tool to approximate a solution uu defined as u=G(f)u = \mathcal{G}(f), for some operator G\mathcal{G} and parameter ff, and are therefore not limited to approximate the solutions of PDE systems. Additionally, in our case, the finite element discretization is used for uu, but also to discretize the input parameter ff.

The finite element method originates from the idea of partitioning the computational domain Ω\Omega, where the PDE is posed, into a collection of subdomains, or cells. More specifically, let Th\mathcal{T}_{h} be a tessellation of Ω\Omega into elements defined as Th:={Ki}i\mathcal{T}_{h} := \{K_{i}\}_{i} such that Ki⊂ΩK_{i} \subset\Omega and Ω‾=⋃iKi\overline{\Omega} = \bigcup_{i}K_{i}, the interiors of KiK_{i} and KjK_{j} are disjoint for all i≠ji \ne j. Th\mathcal{T}_{h} is referred to as a mesh of Ω\Omega as its edges and vertices form Listing 2: Outline of the Firedrake interface for defining a finite element space with continuous piecewise polynomial of degree one on a unit square.

import firedrake as fd

# Define the mesh
mesh = fd.UnitSquareMesh(50, 50)
# Lagrange discretization
family = "CG"
# Polynomial degree
degree = 1
# Define the finite element space V_h
V = fd.FunctionSpace(mesh, family, degree)

a mesh. The choice of basis functions is another crucial aspect in the finite element method and pertains to the chosen finite element discretization. The vast literature on the finite element method has led to a plethora of choices of discretization. In practice, numerical analysts consider tailored discretizations for each specific PDE system arising across science and engineering, such as those in electromagnetism, fluid dynamics, and elasticity. Structure-preserving discretizations have also been proposed to conserve mathematical and physical properties of certain PDEs, as outlined in Arnold & Logg [1].

Finite element discretizations rely on a choice of basis functions with a small support adapted to the tessellation of the domain. This idea, introduced in Courant [22], was motivated by the fact that the product of such a basis function with most of the other basis function vanishes, leading to sparse linear systems that can be solved efficiently. For more details about the historical developments of the finite element method, we refer to Gander & Wanner [28]. The small support assumption of the finite element discretization implies that each degree of freedom uiu_i interacts with the small number of degrees of freedom whose basis functions have overlapping supports with ϕi\phi_i. This leads to a sparse graph representation of the degrees of freedom that is leveraged by our framework (cf. Appendix § 8.§ 2 for more details).

A popular choice of discretization is the continuous Lagrange finite element discretization. In this case, the finite element space VhV_h, also referred to as CGk\mathrm{CG}_k, is chosen to be the space of continuous piecewise polynomials of degree kk defined as Vh={v∈C(Ω‾)∣v∣Ki∈Pk, ∀Ki∈Th}V_h = \{v \in C(\overline{\Omega}) \mid v|_{K_i} \in\mathbb{P}_k, \ \forall K_i \in\mathcal{T}_h\}. The polynomial degree kk of the polynomials offers a trade-off between the computational efficiency of the numerical method and its accuracy, higher-order polynomials leading to more accurate approximations at the detriment of a higher computational cost. However, the efficient implementation of arbitrary finite element spaces such as CGk\mathrm{CG}_k can be a tedious task. The spon package, released with this paper, interfaces with the Firedrake FEM software (Ham et al., [30]) to automate the construction of a rich set of finite element spaces, which relies on code generation for high performance. Listing 2 illustrates how Lagrange finite element spaces can be simply defined in Firedrake. Finally, other discretizations may be considered, such as discontinuous Galerkin (Arnold et al., [2]) or the Raviart-Thomas elements (Raviart & Thomas, [58]), by changing the family keyword argument in line 6 of Listing 2.

Latent graph representation

The SPON encoder maps finite element functions to their degrees of freedom and constructs a graph based on the sparsity of the underlying discretization. Let VhV_h be a finite element space discretizing an appropriate infinite-dimensional space V\mathcal{V} and let (ϕi)i(\phi_i)_i be the basis functions of VhV_h. For every u∈Vhu \in V_h, we have u=∑iuiϕiu = \sum_i u_i\phi_i, where (ui)(u_i) denote the degrees of freedom of $V_h. We define the SPON latent graph of VhV_h as GVh:=(V,E)G_{V_h} := (V,E), where VV is the set of vertices and EE denotes the edges of the graph. The nodes of GVhG_{V_h} correspond to the degrees of freedom of VhV_h and the edges are defined by the sparsity of the discretization, which results from the fact that the basis functions (ϕi)(\phi_i) all have a compact support. More specifically, we have:

V:={ui}1≤i≤∣Vh∣,V := \{u_i\}_{1 \le i \le|V_h|},
E:={(i,j), where i and j are such that ∫Ωϕiϕj dx≠0}.(8)E := \left\{(i,j),\ \text{where } i\text{ and }j\text{ are such that } \int_{\Omega}\phi_i\phi_j\,\mathrm{d}x \ne0\right\}. \tag*{(8)}

Here, two nodes are connected by an edge if the corresponding basis functions have a spatial overlap. However, other criteria may be considered, leading to denser or sparser graph representations.

The released spon package automates the construction of the graph defined in Eq. (8). The construction of the edges is achieved by performing the finite element assembly of the mass matrix MM, where

Mij=∫Ωϕiϕj dx.M_{ij} = \int_{\Omega} \phi_i\phi_j \,\mathrm{d}x.

We assemble MM and retrieves the column and row pairs of indices of its nonzero coefficients, which yields the adjacency list as defined in Eq. (8). The matrix assembly is achieved by the Firedrake finite element software [30], which relies on low-level generated code for high efficiency. We illustrate the latent graph induced by continuous Lagrange finite elements of order 1 and 2 on a square uniform and structured mesh with triangles in Figure 5.

Figure 5. Diagram illustrating the graph induced by a continuous Lagrange finite element of order 1 (CG1_1) (left) and order 2 (CG2_2) (right) on a square uniform and structured mesh with triangles. The neighbors (blue) of a given node (red) are illustrated. The latent graph’s edges can be derived by finding the neighbors of each DoF using Eq. (8). Note that the graph’s edges do not necessarily coincide with the mesh edges. In this example, for the CG1_1 (resp. CG2_2) discretization, each node has at most 7 (resp. 19) edges in the latent graph.

Our approach generalizes naturally for vector, tensor, and mixed function spaces. As an example, when solving the incompressible Navier–Stokes equations using the finite element method, a common discretization approach consists of solving for w=(ux,uy,p)\boldsymbol{w} = (u_x, u_y, p), where u=[ux,uy]\boldsymbol{u} = [u_x, u_y] is the velocity field and pp the pressure field. The velocity field is typically discretized using a continuous Galerkin finite element space of degree two, while the pressure field is discretized using a continuous Galerkin finite element space of degree one. Here, one would extract the sparsity pattern of the matrix MM defined as

Mij=∫ΩwiwjT dx=∫Ω[uxiuxjuxiuyjuxipjuyiuxjuyiuyjuyipjpiuxjpiuyjpipj] dx.M_{ij} = \int_{\Omega} \boldsymbol{w}_i \boldsymbol{w}_j^{\mathsf{T}} \,\mathrm{d}x = \int_{\Omega} \begin{bmatrix} u_{xi}u_{xj} & u_{xi}u_{yj} & u_{xi}p_j \\ u_{yi}u_{xj} & u_{yi}u_{yj} & u_{yi}p_j \\ p_i u_{xj} & p_i u_{yj} & p_i p_j \end{bmatrix} \,\mathrm{d}x.

In this case, the latent graph would have the degrees of freedom of pp connected to the degrees of freedom of u\boldsymbol{u} and the degrees of freedom of the horizontal component of the velocity connected to the degrees of freedom of the vertical component of the velocity.

Boundary conditions support

The decoder enforces Dirichlet boundary conditions strongly by assigning the degrees of freedom associated with the boundary to impose the boundary condition of interest exactly at the discrete level. Other boundary conditions, such as Neumann or Robin conditions, as well as global linear constraints, may be enforced similarly by solving a sparse system of equations on the boundary nodes to find the determine the values to enforce. This process may become computationally expensive in a training setting and may require a trade-off between accurately enforcing such boundary conditions and maintaining computational efficiency. In such cases, a penalty term can be added in the loss function for a weak imposition of boundary conditions. Such explorations are left for future work.

Multigrid-based model

This section provides more details on the multigrid-based processor architecture introduced in § 3.2. Another motivation for multigrid-based architectures comes from the recent theoretical studies on sample complexity for operator learning [12, 14, 23] (Boullé & Townsend 2023; Boullé et al. 2023), which exploit regularity structure of partial differential operators [7] to show that one does not need many training data in operator learning to achieve a good approximation error. These results rely on a hierarchical decomposition of the spatial domain (HH-matrix) [6] to approximate short and long range interactions at different scales, similarly to the multigrid method.

Function space hierarchy and mapping operators

Let M1,…,MN\mathcal{M}^{1},\ldots,\mathcal{M}^{N} be a hierarchy of meshes with varying resolution, and Uh1,…,UhN,Vh1,…,VhN\mathcal{U}_{h}^{1},\ldots,\mathcal{U}_{h}^{N},\mathcal{V}_{h}^{1},\ldots,\mathcal{V}_{h}^{N} be the corresponding input-output function spaces, where Uh1=Uh\mathcal{U}_{h}^{1}=\mathcal{U}_{h} and Vh1=Vh\mathcal{V}_{h}^{1}=\mathcal{V}_{h}. The mesh hierarchy typically assumes nested meshes but non-nested meshes are also supported by our framework. We construct the mesh hierarchy in Firedrake using the procedure given in Listing 3.

The multigrid processor PMG:Rn→Rm\mathcal{P}^{MG}: \mathbb{R}^{n}\to\mathbb{R}^{m}, introduced in § 3.2, where n=dim⁡(Uh)n=\dim(\mathcal{U}_{h}) and m=dim⁡(Vh)m=\dim(\mathcal{V}_{h}) involves the restriction Rhi:Uhi→Uhi+1\mathcal{R}_{h}^{i}: \mathcal{U}_{h}^{i}\to\mathcal{U}_{h}^{i+1} and prolongation Phi:Vhi+1→Vhi\mathcal{P}_{h}^{i}: \mathcal{V}_{h}^{i+1}\to\mathcal{V}_{h}^{i} operators, for 1≤i≤N−11\leq i\leq N-1, and the interpolation operator Ihi:Uhi→Vhi\mathcal{I}_{h}^{i}: \mathcal{U}_{h}^{i}\to\mathcal{V}_{h}^{i}, for 1≤i≤N1\leq i\leq N. These are the classical finite element multigrid operators, often referred to as the “fine-to-coarse” and “coarse-to-fine” operators [20 Sec. 6.3], along with a standard finite element interpolation operator. These operators map between finite element spaces and produce finite element functions as outputs. On the other hand, the multigrid processor only acts on the degrees of freedom. Given that these finite element operators are linear, they induce linear matrices that directly map the degrees of freedom in the input space to the degrees of freedom in the output space. More precisely, our processor uses the restriction, prolongation, and interpolation matrices Ri∈Rni+1×ni\mathrm{R}^{i}\in\mathbb{R}^{n_{i+1}\times n_{i}}, Pi∈Rmi×mi+1\mathrm{P}^{i}\in\mathbb{R}^{m_{i}\times m_{i+1}}, and Ii∈Rmi×ni\mathrm{I}^{i}\in\mathbb{R}^{m_{i}\times n_{i}}, where ni=dim⁡(Uhi)n_{i}=\dim(\mathcal{U}_{h}^{i}) and mi=dim⁡(Vhi)m_{i}=\dim(\mathcal{V}_{h}^{i}). Notably, these matrices are all sparse and defined as follows:

Rhi(uhi)=∑j=1ni+1(Riuji)ϕji+1∈Uhi+1,Phi(vhi+1)=∑j=1mi(Pivji+1)ψji∈Vhi,Ihi(uhi)=∑j=1mi(Iiuji)ψji∈Vhi,\begin{aligned} \mathcal{R}_{h}^{i}(u_{h}^{i})&=\sum_{j=1}^{n_{i+1}}(\mathrm{R}^{i}u_{j}^{i})\phi_{j}^{i+1}\in\mathcal{U}_{h}^{i+1}, \qquad\mathcal{P}_{h}^{i}(v_{h}^{i+1})=\sum_{j=1}^{m_{i}}(\mathrm{P}^{i}v_{j}^{i+1})\psi_{j}^{i}\in\mathcal{V}_{h}^{i},\\ \mathcal{I}_{h}^{i}(u_{h}^{i})&=\sum_{j=1}^{m_{i}}(\mathrm{I}^{i}u_{j}^{i})\psi_{j}^{i}\in\mathcal{V}_{h}^{i}, \end{aligned}

where (ϕji)j(\phi_{j}^{i})_{j} and (ψji)j(\psi_{j}^{i})_{j} are the basis functions of the spaces Uhi\mathcal{U}_{h}^{i} and Vhi\mathcal{V}_{h}^{i}, respectively, and (uji)j(u_{j}^{i})_{j} are the degrees of freedom of uhi∈Uhiu_{h}^{i}\in\mathcal{U}_{h}^{i} and (vji)j(v_{j}^{i})_{j} the degrees of freedom of vhi∈Vhiv_{h}^{i}\in\mathcal{V}_{h}^{i}.

Listing 3: Outline of the Firedrake interface for defining a mesh hierarchy M1,…,M4\mathcal{M}^{1},\ldots,\mathcal{M}^{4} and the corresponding corresponding finite element spaces Vhi\mathcal{V}_{h}^{i}, using a CG1 discretization and where the coarse mesh M1\mathcal{M}_{1} is a unit square with 32 cells in each direction.

import firedrake as fd

# Define a coarse mesh (M₁)
coarse_mesh = fd.UnitSquareMesh(32, 32)

# Define a hierarchy of meshes with 3 refinements (N = 4)
hierarchy = fd.MeshHierarchy(coarse_mesh, 3)

# Define the space hierarchy for CG1 elements
V_spaces = [fd.FunctionSpace(m, "CG", 1) for m in hierarchy]

To speed up training and inference, we assemble these sparse matrices in Firedrake offline, and convert them to PyTorch sparse tensors. The restriction, prolongation, and interpolation operations, occurring across the architecture of PθMG\mathcal{P}_{\theta}^{MG} during training and inference, are then performed via sparse matrix-vector products. Finally, it is worth noting that the restriction and prolongation operators facilitate the inference of structure-preserving operator networks at different mesh resolution than the training resolution, as illustrated in Eq. (4).

Message-passing models

The multigrid processor PθMG\mathcal{P}^{MG}_{\theta} mainly relies on the learnable (φi)1≤i≤N(\varphi_i)_{1\le i\le N} and ψ\psi message-passing-based architectures across the NN levels of the hierarchy. Each φi\varphi_i is a lightweight model associated with the ii-th level in the hierarchy and is used for both latent features on Uhi\mathcal{U}^{i}_{h} and Vhi\mathcal{V}^{i}_{h}. These models are agnostic of the number of degrees of freedom and consist of MM identical message passing blocks ϕ\phi with distinct sets of parameters. The MM blocks are combined in a pipeline manner with residual connections.

More precisely, let H0=[h1,…,hnDoFs]TH^0 = [h_1,\ldots,h_{n_{\mathrm{DoFs}}}]^{\mathsf{T}} be a global feature vector containing the node features (hi)1≤i≤nDoFs(h_i)_{1\le i\le n_{\mathrm{DoFs}}}, where nDoFsn_{\mathrm{DoFs}} is the number of nodes in the graph, i.e., the number of degrees of freedom of the finite element space associated with the latent graph. For 1≤i≤N1\le i\le N, we define φi(H0)=HM\varphi_i(H^0)=H^M, where HMH^M results from the following iterative procedure:

Hn+1=Hn+αϕ(Hn),H^{n+1}=H^n+\alpha\phi(H^n),

for 0≤n≤M−10\le n\le M-1 and α>0\alpha>0. For our experiments, we consider α=1M\alpha=\frac{1}{M}. Here, ϕ\phi is a message passing architecture defined by the following update.

Edge j→i message update:mij=ϕe(hi,hj−hi),Node i update:hi=ϕv(hi,1∣N(i)∣∑j∈N(i)mij).(9)\begin{aligned} \text{Edge }j\to i\text{ message update}: \quad&m_{ij}=\phi_e(h_i,h_j-h_i),\\ \text{Node }i\text{ update}: \quad&h_i=\phi_v\left(h_i,\frac{1}{|\mathcal{N}(i)|}\sum_{j\in\mathcal{N}(i)}m_{ij}\right). \tag*{(9)} \end{aligned}

where N(i)\mathcal{N}(i) denotes the neighborhood of the node feature hih_i, and ϕe\phi_e and ϕv\phi_v are MLPs with “swish” activation functions [57]. We consider 4 linear layers for both MLPs. It is worth noting that the number of parameters of the message passing architectures does not depend on the number of degrees of freedom and therefore remains constant across the different hierarchy levels.

The (φi)1≤i≤N(\varphi_i)_{1\le i\le N} models can only propagate information MM hops away at each level and are not enough to capture long-range dependencies. We employ a coarse model ψ\psi that allows efficient global exchange of information throughout the domain. ψ\psi operates at the coarser level and therefore enables node updates that are significantly cheaper than at the finer levels. The coarser model ψ\psi can also be seen as an encode-process-decode model with linear encoding and decoding that allow global exchange of information across all the DoFs. The processor of ψ\psi comprises message-passing updates via φN\varphi_N on the graph associated with UhN\mathcal{U}^{N}_{h}, followed by an interpolation to VhN\mathcal{V}^{N}_{h}, and finally message-passing layers via φN\varphi_N over the graph associated with VhN\mathcal{V}^{N}_{h}. More specifically, the coarse model can be defined as:

ψ:=WVhN∘φN∘IhN∘φN∘WUhN,(10)\psi:= W_{\mathcal{V}^{N}_{h}}\circ\varphi_N\circ\mathcal{I}^{N}_{h}\circ\varphi_N\circ W_{\mathcal{U}^{N}_{h}}, \tag*{(10)}

with φN\varphi_N the message-passing architecture defined in Eq. (9), IhN\mathcal{I}^{N}_{h} the interpolation operator between UhN\mathcal{U}^{N}_{h} and VhN\mathcal{V}^{N}_{h}, and where WUhN∈Rn×nW_{\mathcal{U}^{N}_{h}}\in\mathbb{R}^{n\times n} and WVhN∈Rm×mW_{\mathcal{V}^{N}_{h}}\in\mathbb{R}^{m\times m} are learnable matrices acting over the degrees of freedom of UhN\mathcal{U}^{N}_{h} and VhN\mathcal{V}^{N}_{h}, respectively, for n=dim⁡(UhN)n=\operatorname{dim}(\mathcal{U}^{N}_{h}) and m=dim⁡(VhN)m=\operatorname{dim}(\mathcal{V}^{N}_{h}). Notably, these linear layers dominate the number of parameters of the processor PθMG\mathcal{P}^{MG}_{\theta} with n2n^2 and m2m^2 parameters, respectively.

To cut down the number of parameters, reduce the memory footprint, and improve latency, we consider a low-rank approximation of the matrices WUhNW_{\mathcal{U}^{N}_{h}} and WVhNW_{\mathcal{V}^{N}_{h}} with a compression factor of kk. That is, we define

WUhN=WUhN↑WUhN↓,WVhN=WVhN↑WVhN↓,(11)W_{\mathcal{U}^{N}_{h}}=W_{\mathcal{U}^{N}_{h}}{}^{\uparrow}W_{\mathcal{U}^{N}_{h}}{}^{\downarrow},\qquad W_{\mathcal{V}^{N}_{h}}=W_{\mathcal{V}^{N}_{h}}{}^{\uparrow}W_{\mathcal{V}^{N}_{h}}{}^{\downarrow}, \tag*{(11)}

with WUhN↑∈Rn×n/kW_{\mathcal{U}^{N}_{h}}{}^{\uparrow}\in\mathbb{R}^{n\times n/k}, WUhN↓∈Rn/k×nW_{\mathcal{U}^{N}_{h}}{}^{\downarrow}\in\mathbb{R}^{n/k\times n}, reducing the number of parameters to 2n2/k2n^2/k, and WVhN↑∈Rm×m/kW_{\mathcal{V}^{N}_{h}}{}^{\uparrow}\in\mathbb{R}^{m\times m/k}, WVhN↓∈Rm/k×mW_{\mathcal{V}^{N}_{h}}{}^{\downarrow}\in\mathbb{R}^{m/k\times m}, reducing the number of parameters to 2m2/k2m^2/k.

Approximation theory proof

Before proving Theorem 1, we introduce some necessary notations. Following Figure 6, let PUP_U, PVP_V denote the Galerkin interpolation operators onto the finite-dimensional space Uh1U_{h_1} and Vh2V_{h_2}, IU:Uh1→UI_U: U_{h_1} \to U and IV:Vh2→VI_V: V_{h_2} \to V the injection operators. Let EUh1:Uh1→K⊂RnE_{U_{h_1}}: U_{h_1} \to K \subset\mathbb{R}^{n}, DVh2:Rm→Vh2D_{V_{h_2}}: \mathbb{R}^{m} \to V_{h_2} be the structure-preserving encoder and decoder, and Pθ:Rn→RmP_\theta: \mathbb{R}^{n} \to\mathbb{R}^{m} be the processor. Here, KK is the compact image of Uh1U_{h_1} under EUh1E_{U_{h_1}}, which is compact since UU is compact. Moreover, we denote by G^=PV∘G∘IU\widehat{G} = P_V \circ G \circ I_U the finite element approximation of GG. We recall the standard finite element error estimate ([20 Eq. 4.4.28]) that will be used extensively in the proof of Theorem 1:

Figure 6. Diagram describing the operators used in the proof of Theorem 1.

∥f−PU(f)∥Hs1(Ω1)≤C1hk1−s1∥f∥Hk1(Ω1),f∈U,∥u−PV(u)∥Hs2(Ω2)≤C2hk2−s2∥u∥Hk2(Ω2),u∈V,(12)\begin{aligned} \lVert f-P_U(f)\rVert_{H^{s_1}(\Omega_1)} &\le C_1 h^{k_1-s_1}\lVert f\rVert_{H^{k_1}(\Omega_1)}, \qquad f\in U,\\ \lVert u-P_V(u)\rVert_{H^{s_2}(\Omega_2)} &\le C_2 h^{k_2-s_2}\lVert u\rVert_{H^{k_2}(\Omega_2)}, \qquad u\in V, \tag*{(12)} \end{aligned}

for some constant C1,C2>0C_1,C_2>0, 0≤s1≤k10\le s_1\le k_1, and 0≤s2≤k20\le s_2\le k_2.

Proof of Theorem 1. First, we remark that the function EVh2∘G^∘DUh1:K⊂Rn→RmE_{V_{h_2}}\circ\widehat{G}\circ D_{U_{h_1}}: K\subset\mathbb{R}^{n}\to\mathbb{R}^{m} is Lipschitz continuous (a similar argument can be found in Lanthaler et al. [41]). Moreover, we can embed KK into the hypercube [−M,M]n[-M,M]^n for sufficiently large M>0M>0. Let ϵ∈(0,1)\epsilon\in(0,1), then by Yarotsky [63], there exists a ReLU neural network Pθ:Rm→RnP_\theta:\mathbb{R}^{m}\to\mathbb{R}^{n} with ∣θ∣≤C1ϵ−n(log⁡(1/ϵ)+1)|\theta|\le C_1\epsilon^{-n}(\log(1/\epsilon)+1) weights such that

sup⁡x∈K∥EVh2∘G^∘DUh1(x)−Pθ(x)∥ℓ2(Rm)≤ϵ,(13)\sup_{x\in K}\lVert E_{V_{h_2}}\circ\widehat{G}\circ D_{U_{h_1}}(x)-P_\theta(x)\rVert_{\ell_2(\mathbb{R}^{m})}\le\epsilon, \tag*{(13)}

where C1>0C_1>0 is a constant independent of ϵ\epsilon. Moreover, since Uh1U_{h_1} is a finite element space defined on a regular mesh, we have that n=dim⁡(Uh1)≤C2/hk1n1n=\dim(U_{h_1})\le C_2/h^{k_1}{}^{n_1} for some constant C2>0C_2>0.

Now, let f∈Uf\in U, s1≤k1s_1\le k_1, s2≤k2s_2\le k_2, and denote fh1=PU(f)f_{h_1}=P_U(f). Then, the approximation error ∥(G−Sθ∘PU)(f)∥Hs2(Ω2)\lVert(G-S_\theta\circ P_U)(f)\rVert_{H^{s_2}(\Omega_2)} can be expressed using triangular inequality in terms of the approximation error of the input-output spaces and the processor as

∥(G−Sθ∘PU)(f)∥Hs2(Ω2)≤∥G(f)−G(fh1)∥Hs2(Ω2)+∥G(fh1)−G^(fh1)∥Hs2(Ω2)+∥G^(fh1)−Sθ(fh1)∥Hs2(Ω2).\begin{aligned} \lVert(G-S_\theta\circ P_U)(f)\rVert_{H^{s_2}(\Omega_2)} \le{}&\lVert G(f)-G(f_{h_1})\rVert_{H^{s_2}(\Omega_2)} +\lVert G(f_{h_1})-\widehat{G}(f_{h_1})\rVert_{H^{s_2}(\Omega_2)}\\ &+\lVert\widehat{G}(f_{h_1})-S_\theta(f_{h_1})\rVert_{H^{s_2}(\Omega_2)}. \end{aligned}

where G^:=PV∘G∘IU\widehat{G}:=P_V\circ G\circ I_U is the finite element approximation of GG. Here, (A) (resp. (B)) corresponds to the finite element approximation error of the input space UU (resp. output space VV), and (C) is the neural approximation error. We bound the three terms independently.

The term (A) is bounded using the Lipschitz continuity of GG from Hs1(Ω1)→Hs2(Ω2)H^{s_1}(\Omega_1)\to H^{s_2}(\Omega_2) (as V⊂Hs2(Ω2)V\subset H^{s_2}(\Omega_2)) and the finite element approximation bound in the input space UU ([20 Eq. 4.4.28]) as

∥G(f)−G(fh1)∥≤Lip⁡(G)∥f−fh1∥Hs1(Ω1)≤Lip⁡(G)C1hk1−s1∥f∥Hk1(Ω1).\lVert G(f)-G(f_{h_1})\rVert \le\operatorname{Lip}(G)\lVert f-f_{h_1}\rVert_{H^{s_1}(\Omega_1)} \le\operatorname{Lip}(G)C_1h^{k_1-s_1}\lVert f\rVert_{H^{k_1}(\Omega_1)}.

We bound the term (B)(B) using the finite element approximation bound in the output space V\mathcal{V} [20 Eq. 4.4.28]. First, we remark that by definition of G^\widehat{\mathcal{G}} (see Figure 6) and since IU∘PU(f)=PU(f)I_{\mathcal{U}} \circ P_{\mathcal{U}}(f) = P_{\mathcal{U}}(f), we have

∥G(fh1)−G^(fh1)∥Hs2(Ω2)=∥G(fh1)−PV∘G(fh1)∥Hs2(Ω2).\left\|\mathcal{G}(f_{h_1})-\widehat{\mathcal{G}}(f_{h_1})\right\|_{H^{s_2}(\Omega_2)}=\left\|\mathcal{G}(f_{h_1})-P_{\mathcal{V}}\circ\mathcal{G}(f_{h_1})\right\|_{H^{s_2}(\Omega_2)}.

Then,

∥(Id−PV)∘G(fh1)∥Hs2(Ω2)≤∥(Id−PV)∘G(f)−(Id−PV)∘G(fh1)∥Hs2(Ω2)+∥(Id−PV)∘G(f)∥Hs2(Ω2).(14)\left\|(I_d-P_{\mathcal{V}})\circ\mathcal{G}(f_{h_1})\right\|_{H^{s_2}(\Omega_2)} \leq\left\|(I_d-P_{\mathcal{V}})\circ\mathcal{G}(f)-(I_d-P_{\mathcal{V}})\circ\mathcal{G}(f_{h_1})\right\|_{H^{s_2}(\Omega_2)} +\left\|(I_d-P_{\mathcal{V}})\circ\mathcal{G}(f)\right\|_{H^{s_2}(\Omega_2)}. \tag*{(14)}

Since Id−PVI_d-P_{\mathcal{V}} is linear and continuous, (Id−PV)∘G(I_d-P_{\mathcal{V}})\circ\mathcal{G} is Lipschitz from Hs1(Ω1)→Vh2H^{s_1}(\Omega_1)\to\mathcal{V}_{h_2}, therefore from Hs1(Ω1)→Hs2(Ω2)H^{s_1}(\Omega_1)\to H^{s_2}(\Omega_2) as Vh2⊂V⊂Hs2(Ω2)\mathcal{V}_{h_2}\subset\mathcal{V}\subset H^{s_2}(\Omega_2), and

Lip⁡((Id−PV)∘G)≤C2Lip⁡(G).\operatorname{Lip}((I_d-P_{\mathcal{V}})\circ\mathcal{G})\leq C_2\operatorname{Lip}(\mathcal{G}).

Then, the first term in the right-hand side of Eq. (14) is bounded as

∥(Id−PV)∘G(f)−(Id−PV)∘G(fh1)∥Hs2(Ω2)≤C2Lip⁡(G)∥f−fh1∥Hs1(Ω1)≤C1C2Lip⁡(G)h1k1−s1∥f∥Hk1(Ω1),\left\|(I_d-P_{\mathcal{V}})\circ\mathcal{G}(f)-(I_d-P_{\mathcal{V}})\circ\mathcal{G}(f_{h_1})\right\|_{H^{s_2}(\Omega_2)} \leq C_2\operatorname{Lip}(\mathcal{G})\|f-f_{h_1}\|_{H^{s_1}(\Omega_1)} \leq C_1C_2\operatorname{Lip}(\mathcal{G})h_1^{k_1-s_1}\|f\|_{H^{k_1}(\Omega_1)},

using [20 Eq. 4.4.28] on the input space Hs1(Ω1)H^{s_1}(\Omega_1). We now bound the second term in Eq. (14) using the same finite element estimate on the output space Hs2(Ω2)H^{s_2}(\Omega_2) as

∥(Id−PV)∘G(f)∥Hs2(Ω2)≤C2h2k2−s2∥G(f)∥Hk2(Ω2).\left\|(I_d-P_{\mathcal{V}})\circ\mathcal{G}(f)\right\|_{H^{s_2}(\Omega_2)} \leq C_2h_2^{k_2-s_2}\|\mathcal{G}(f)\|_{H^{k_2}(\Omega_2)}.

Finally, the term (C)(C) is bounded as follows:

∥G^(fh1)−Sθ(fh1)∥Hs2(Ω2)=∥G^(fh1)−DVh2∘Pθ∘EUh1(fh1)∥Hs2(Ω2)=∥DVh2∘(EVh2∘G^−Pθ∘EUh1)∘fh1∥Hs2(Ω2)≤∥DVh2∥Rm→Hs2(Ω2)∥(EVh2∘G^−Pθ∘EUh1)∘fh1∥ℓ2(Rm),\begin{aligned} \left\|\widehat{\mathcal{G}}(f_{h_1})-\mathcal{S}_{\theta}(f_{h_1})\right\|_{H^{s_2}(\Omega_2)} &=\left\|\widehat{\mathcal{G}}(f_{h_1})-\mathcal{D}_{\mathcal{V}_{h_2}}\circ P_{\theta}\circ\mathcal{E}_{\mathcal{U}_{h_1}}(f_{h_1})\right\|_{H^{s_2}(\Omega_2)}\\ &=\left\|\mathcal{D}_{\mathcal{V}_{h_2}}\circ\left(\mathcal{E}_{\mathcal{V}_{h_2}}\circ\widehat{\mathcal{G}}-P_{\theta}\circ\mathcal{E}_{\mathcal{U}_{h_1}}\right)\circ f_{h_1}\right\|_{H^{s_2}(\Omega_2)}\\ &\leq\left\|\mathcal{D}_{\mathcal{V}_{h_2}}\right\|_{\mathbb{R}^{m}\to H^{s_2}(\Omega_2)} \left\|\left(\mathcal{E}_{\mathcal{V}_{h_2}}\circ\widehat{\mathcal{G}}-P_{\theta}\circ\mathcal{E}_{\mathcal{U}_{h_1}}\right)\circ f_{h_1}\right\|_{\ell^2(\mathbb{R}^{m})}, \end{aligned}

as DVh2∘EVh2\mathcal{D}_{\mathcal{V}_{h_2}}\circ\mathcal{E}_{\mathcal{V}_{h_2}} is the identity operator on Vh2\mathcal{V}_{h_2} and using the submultiplicativity of the operator norm. First, we note that ∥DVh2∥Rm→Hs2(Ω2)≤∥DVh2∥Rm→V≤1\left\|\mathcal{D}_{\mathcal{V}_{h_2}}\right\|_{\mathbb{R}^{m}\to H^{s_2}(\Omega_2)}\leq\left\|\mathcal{D}_{\mathcal{V}_{h_2}}\right\|_{\mathbb{R}^{m}\to\mathcal{V}}\leq1. Additionally, using the fact that DUh1∘EUh1\mathcal{D}_{\mathcal{U}_{h_1}}\circ\mathcal{E}_{\mathcal{U}_{h_1}} is the identity operator on Uh1\mathcal{U}_{h_1}, we obtain

∥G^(fh1)−Sθ(fh1)∥Hs2(Ω2)≤∥(EVh2∘G^∘DUh1−Pθ)∘EUh1(fh1)∥ℓ2(Rm)≤sup⁡x∈K∥EVh2∘G^∘DUh1(x)−Pθ(x)∥ℓ2(Rm)≤ϵ,\left\|\widehat{\mathcal{G}}(f_{h_1})-\mathcal{S}_{\theta}(f_{h_1})\right\|_{H^{s_2}(\Omega_2)} \leq\left\|\left(\mathcal{E}_{\mathcal{V}_{h_2}}\circ\widehat{\mathcal{G}}\circ\mathcal{D}_{\mathcal{U}_{h_1}}-P_{\theta}\right)\circ\mathcal{E}_{\mathcal{U}_{h_1}}(f_{h_1})\right\|_{\ell^2(\mathbb{R}^{m})} \leq\sup_{x\in K}\left\|\mathcal{E}_{\mathcal{V}_{h_2}}\circ\widehat{\mathcal{G}}\circ\mathcal{D}_{\mathcal{U}_{h_1}}(x)-P_{\theta}(x)\right\|_{\ell^2(\mathbb{R}^{m})}\leq\epsilon,

where the last inequality is due to Eq. (13). Combining the bounds for (A)(A), (B)(B), and (C)(C), we obtain the desired result.

Additional details on the experiments

This section provides additional details on the numerical experiments presented in Section 4. All the experiments are conducted on a single RTX 4070 Ti GPU. For training, all the models are trained with the AdamW optimizer [49] and using an exponential learning rate decay from 10−410^{-4} to 10−610^{-6} at the last epoch.

We consider two types of structure-preserving operator networks in these experiments. A single-level architecture that uses the message-passing model ψ\psi, defined in Eq. (10), as processor, which we refer to as SPON, and a multigrid architecture that uses PθMGP_{\theta}^{MG} as processor, which we refer to as SPON-MG. The multigrid processor uses ψ\psi as its coarse model, as depicted in Figure 2. We use the same architecture for all the message passing GNN layers. More specifically, we use two MLPs to compute the messages on each edge (ϕe)(\phi_e) and to update the node features (ϕv)(\phi_v), see Appendix B.2 for further detail.

Poisson's equation with strong boundary conditions

For the Poisson experiment in § 4.1, we consider a nonhomogeneous Dirichlet condition on the top boundary ΓD\Gamma_D defined as g=10−2sin⁡(πx)g = 10^{-2}\sin(\pi x), for x∈ΓDx \in\Gamma_D. A CG1 discretization is used for the input and output spaces, see Appendix § 8.§ 1.

The source terms are generated using the Chebfun software system {[24, 25]} as random smooth functions f∼GP(0,K)f \sim\mathcal{GP}(0, K). Here, KK is a squared-exponential kernel with length-scale 0.4. The corresponding solutions are generated by solving the Poisson problem, see Eq. (6), in Firedrake {[30]} and using LU decomposition.

The single-level structure-preserving operator network SPON uses the message-passing model ψ\psi as processor (cf. Eq. (10)). We consider a total number of 4 layers of message-passing layers for ψ\psi, all with different parameters. For the multigrid architecture SPON-MG, we consider a hierarchy of N=3N = 3 levels, with resolutions nx/4n_x/4, nx/2n_x/2, and nxn_x. We employ 1 message-passing layer for each level except for the coarse model ψ\psi, for which we consider a total of 4 GNN layers. For this experiment, we consider a low-rank approximation of the weight matrices of ψ\psi for both SPON and SPON-MG, as detailed in Eq. (11). We use a compression factor kk of 20 for SPON and 1 for SPON-MG.

We train all the models for 500 epochs with a batch size of 4. For the loss, we use the L2L^2-relative error. It is worth noting that the computation of the L2L^2 norm is achieved using the finite element assembly, which is made possible by the fact that structure-preserving operator networks return finite element functions.

The Fourier neural operator is implemented using the neuraloperator {[47]} library with the default architecture settings for the Darcy flow datasets. Note that we do not use tensorization of the weights and add a zero padding to account for the non-periodic boundary conditions. The DeepONet implementation is based on the implementation of Lu et al. {(2022; 2024a)} and relies on the DeepXDE library {(Lu et al., 2024b)}. DeepONets usually require a much larger number of training epochs than FNO, which might explain the poor performance of DeepONet on the Poisson benchmark in Figure 3(a).

Fluid flow past a cylinder

We consider a rectangular domain [−5,20]×[−2.5,2.5][-5, 20] \times[-2.5, 2.5] with a circular obstacle of center (0,0)(0, 0) and radius 0.5. We consider an unstructured mesh discretization comprising 9069 triangles, see top figure in Figure 7.

To generate the dataset, we consider different inflow conditions on the left boundary. We generate 100 random source terms, sampled from a Gaussian process with periodic kernel (Williams & Rasmussen, 2006, Eq. 4.31), which we use for the left boundary. For each of them, we solve Eq. (7) using the finite element method to generate the corresponding trajectories, i.e. the fluid velocity u(⋅,t)u(\cdot,t) for t∈[0,T]t \in[0,T]. In addition to the Dirichlet condition on the obstacle and the upper and lower sides, we also consider a compatibility "do-nothing" condition on Γout\Gamma_{\mathrm{out}}, the right side of Ω\Omega, which is defined as pn⃗=1Re∇u⋅n⃗p\vec{n} = \frac{1}{\mathrm{Re}}\nabla u \cdot\vec{n}, where n⃗\vec{n} denotes the outward normal vector, and pp is the pressure. The velocity and pressure are discretized using the Taylor–Hood finite element, i.e., continuous piecewise quadratic for the velocity and piecewise linear for the pressure, which is a stable and standard element pair for the Stokes equations [60]. We solve Eq. (7) using a backward Euler method up to T=30T = 30 and with a timestep Δtsolve=0.1\Delta t^{solve} = 0.1. For each timestep, we solve the corresponding nonlinear problem using Newton’s method with LU decomposition for the linear solver. We record the solution every 2 timestep, which results in 150 samples for each trajectory. We form the training dataset by considering the trajectories associated with 80 inflow conditions. The validation and test splits are formed by considering 10 trajectories for each split.

We train the SPON-MG model to learn the one-step forward operator u(⋅,t)↦u(⋅,t+Δt)u(\cdot,t) \mapsto u(\cdot,t + \Delta t), for a relatively large prediction timestep Δt=2\Delta t = 2. For training and inference, the predicted velocities are obtained autoregressively, i.e., by rolling out the predictions until the final timestep. The prediction timestep is 20 times greater than the timestep Δtsolve\Delta t^{solve} used to solve Eq. (7) using FEM. To augment the amount of training data, for each trajectory (i.e. inflow condition), we also train our model using the subtrajectories between each prediction time step. More precisely, given that we generated the fluid velocities every 0.2s0.2s for each trajectory and that we consider a prediction timestep Δt=2s\Delta t = 2s, we have 10 subtrajectories for each inflow condition. For example, the two first trajectories comprise the fluid velocities u(⋅,t)u(\cdot,t) at times t=0,2,4,…t = 0, 2, 4, \ldots, and t=0.2,2.2,4.2,…t = 0.2, 2.2, 4.2, \ldots, respectively.

For the multigrid processor, we consider a non-nested hierarchy of 3 unstructured meshes comprising 1850, 3830, and 9069 triangles, see Figure 7. All the meshes are constructed using Gmsh (Geuzaine & Remacle, [29]). We consider 3 layers of message-passing GNNs for each level, all with different parameters. For the coarse model ψ\psi, we consider a low-rank approximation with a compression factor kk of 500 to reduce the computational cost (cf. Eq. (11)), since the latent graph associated with the finest level contains approximately 40k nodes and 800k edges. A summary of the model is outlined in Table 2.

Layer (type)Output Shape# Parameters
Encoder[batch_size, 36848, 1]–
MessagePassingMultiGridProcessor[batch_size, 36848, 1]–
3 x MessagePassingBlocks[batch_size, 36848, 1]3 x 337
RestrictionFEM[batch_size, 15692]–
3 x MessagePassingBlocks[batch_size, 15692, 1]3 x 337
RestrictionFEM[batch_size, 7656]–
Coarse Model
Linear[batch_size, 500]3,828,000
Linear[batch_size, 7656]3,828,000
3 x MessagePassingBlocks[batch_size, 7656, 1]3 x 337
Linear[batch_size, 500]3,828,000
Linear[batch_size, 7656]3,828,000
ProlongationFEM[batch_size, 15692]–
Linear (smoother)[batch_size, 15692, 1]3
3 x MessagePassingBlocks[batch_size, 15692, 1](recursive)
ProlongationFEM[batch_size, 36848]–
Linear (smoother)[batch_size, 36848, 1](recursive)
3 x MessagePassingBlocks[batch_size, 36848, 1](recursive)
Decoder––
Total parameters:15,315,036

Table 2. Summary of the model for Navier–Stokes equation.

We train our model for 15000 epochs with a batch size of 2. To improve generalization, we consider a divergence-free regularization on the loss function, as introduced by Bouziani et al. [17], to enforce flow incompressibility on the model’s predictions. Additionally, to reduce memory usage and training time, we unroll the full trajectory but only backpropagate through the last half of the predictions. This approach has similarities with the pushforward trick (Brandstetter et al., [19]), although the loss considered is different.

In Figure 8, we report the generalization error, i.e., error across time steps that have been observed during training for a new source term in the test dataset, of our model for the source displayed in Figure 4. Here, we observe a low L2L^2-relative error up to t≈15t \approx15, where the fluid flow transitions from laminar to periodic behavior (see Figure 9). Next in Figure 8 and Figure 9, we extrapolate the model further in time at time steps not seen during training (the model was trained up to t=30t = 30) and observe a low relative error between the predicted velocity and ground truth, which demonstrates the model’s ability to generalize to unseen time steps.

Additional experiments

Hyperelastic beam under compression

In this section, we consider a hyperelastic beam under compression on one of its boundary, and learn the mapping between the force applied on the boundary and the corresponding displacement field on the entire domain. This example illustrates how SPON architectures can be applied to problems where the input and output spaces are defined on different domains and with different spatial dimensions.

More specifically, we aim to approximate the solution operator G:H1(ΓR;R+)→H1(Ω;R2)\mathcal{G}: H^1(\Gamma_R;\mathbb{R}_+) \to H^1(\Omega;\mathbb{R}^2) that maps the compression load gg to the beam displacement field uu satisfying the following nonlinear elasticity equation:

−∇⋅P(u)=Bin Ω,(15)-\nabla\cdot P(u) = B \quad\text{in } \Omega, \tag*{(15)}

with u∣ΓR=(g,0)⊤u|_{\Gamma_R} = (g, 0)^\top, and where Ω=[0,1]×[0,0.1]\Omega= [0, 1] \times[0, 0.1] is the 2D computational domain, ΓR\Gamma_R the 1D right boundary, B=(0,−1000)⊤B = (0, -1000)^\top, and P(u)P(u) is the first Piola–Kirchhoff stress tensor given by

P(u)=μF(tr⁡(C)I)−μF⊤+λ2F⊤,F=I+∇u,C=F⊤F,J=det⁡(F),(16)P(u) = \mu F(\operatorname{tr}(C)I) - \mu F^\top+ \frac{\lambda}{2}F^\top,\qquad F = I + \nabla u,\qquad C = F^\top F,\qquad J = \det(F), \tag*{(16)}

where the Lamé parameters μ(E,ν)=E/(2(1+ν))\mu(E, \nu) = E/(2(1 + \nu)) and λ(E,ν)=Eν(1+ν)(1−2ν)\lambda(E, \nu) = E\nu(1 + \nu)(1 - 2\nu) are derived from Young’s modulus E=106E = 10^6 and Poisson ratio ν=0.3\nu= 0.3. We further complement Eq. (15) with the Dirichlet boundary condition u(0,⋅)=(0,0)⊤u(0, \cdot) = (0, 0)^\top, along with natural boundary conditions P(u)⋅n⃗=0P(u) \cdot\vec{n} = 0 on the top and bottom boundaries of Ω\Omega, where n⃗\vec{n} denotes the outward normal vector. The input and output spaces are discretized using CG2_2 scalar and vector elements, respectively. For sake of simplicity, we consider constant load compressions on ΓR\Gamma_R, i.e. g=−ϵg = -\epsilon, with ϵ∈R+\epsilon\in\mathbb{R}_+.

The dataset is formed of N=80N = 80 pairs of loads ϵ\epsilon, ranging from 0 to 0.2, and associated displacement uu, solution to Eq. (15). The numerical solver consists of Newton’s method with linesearch, along with GMRES for the linear solver. We use a continuation method and first solve for a load ϵ=0.1\epsilon= 0.1 and then use the solution as an initial guess for the next load. The dataset is split into 50 training samples, 20 validation samples, and 10 test samples.

We use a 40 ×\times 40 rectangular grid for the beam, and an interval mesh with 40 points for ΓR\Gamma_R. We consider the single-level message passing processor SPON (with processor ψ\psi). We train the model for 1000 epochs with a batch size of 4. The resulting SPON model achieves a relative L2L^2-error of 4.3×10−24.3 \times10^{-2} on the test split and is capable of reproducing the hyperelastic deformations on the beam for larger values of compression load ϵ\epsilon it was trained on, as shown in Figure 10.

Figure 10. Ground truth displacement at ϵ=0.1995\epsilon= 0.1995 against the prediction by the trained SPON.

Efficiency of the multigrid processor PθMGP_\theta^{MG}

We have seen in § 4.1 that the multigrid-based structure-preserving operator network (SPON-MG) surpasses the structure-preserving operator network that uses the single-level message-passing processor ψ\psi (SPON). In this section, we compare the efficiency of both architectures and compare how the number of parameters grow as we consider finer resolutions.

We consider the same setting as in § 4.1, i.e., a unit square mesh of resolution nxn_x, and learn the solution operator associated with Eq. (6). Both architectures rely on the model ψ\psi: SPON uses it at the finer level while it serves as the coarse model for SPON-MG, as illustrated in Figure 2. As discussed in Appendix § 9.§ 2, we employ a low-rank approximation for the weight matrices of ψ\psi (cf. Eq. (10)–Eq. (11)). For both SPONs, we use a compression factor kk of 5.

We report the evolution of the number of parameters and epoch time as we scale to higher resolution in Figure 11. We can see in Figure 11b that the number of parameters increases dramatically for the SPON model, reaching approximately 500M500M parameters for nx=160n_x = 160. This is due to the liner layers used in ψ\psi, see Eq. (10), that largely dominate the total number of parameters. These linear layers comprise O(4×nDoFs2/k)O(4 \times n_{\mathrm{DoFs}}^2/k) parameters, with kk the compression factor, which equates O(4×nx4/k)O(4 \times n_x^4/k) since the degrees of freedom correspond to the grid nodes for a CG1_1 discretization. For this experiment, we kept the same number of levels (N=3N = 3) for SPON-MG as we move to finer resolutions. This explains why the number of parameters in Figure 11b grows with the same rate but lead to significantly less parameters, since ψ\psi is, in this case, applied at the coarse level, i.e., at the resolution nx/4n_x/4. Alternatively, one could increase the number of levels for finer resolutions to reduce the number of parameters or even to keep it constant.

We also observe the substantial advantage of SPON-MG in terms of efficiency, as illustrated in Figure 11a, with an epoch time that almost remains constant as we move from nx=40n_x = 40 to 160, where the number of DoFs is 16 times greater. This contrasts with the single-level SPON architecture we considered for this experiment, for which the epoch time augments drastically, becoming 7 times slower than the SPON-MG. This speed-up results from delegating the computational load of message-passing updates and linear layers to the coarse level, where the number of degrees of freedom is lower.

References

  1. [1]Douglas N. Arnold and Anders Logg. Periodic table of the finite elements. SIAM News, 47(9):212, 2014.
  2. [2]Douglas N Arnold, Franco Brezzi, Bernardo Cockburn, and L Donatella Marini. Unified analysis of discontinuous Galerkin methods for elliptic problems. SIAM J. Numer. Anal., 39(5):1749–1779, 2002.DOI
  3. [3]Douglas N. Arnold, Richard S. Falk, and Ragnar Winther. Finite element exterior calculus, homological techniques, and applications. Acta Numer., 15:1–155, 2006.DOI
  4. [4]Francesca Bartolucci, Emmanuel de Bezenac, Bogdan Raonic, Roberto Molinaro, Siddhartha Mishra, and Rima Aliafari. Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learning. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2305.19913
  5. [5]Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018.arxiv.org/abs/1806.01261
  6. [6]Mario Bebendorf. Hierarchical matrices. Springer, 2008.DOI
  7. [7]Mario Bebendorf and Wolfgang Hackbusch. Existence of H-matrix approximants to the inverse FE-matrix of elliptic operators with L∞-coefficients. Numer. Math., 95:1–28, 2003.DOI
  8. [11]Nicolas Boullé, Christopher J. Earls, and Alex Townsend. Data-driven discovery of Green’s functions with human-understandable deep learning. Sci. Rep., 12(1):4824, 2022a.DOI
  9. [12]Nicolas Boullé, Seick Kim, Tianyi Shi, and Alex Townsend. Learning Green’s functions associated with time-dependent partial differential equations. J. Mach. Learn. Res., 23(218):1–34, 2022b.arxiv.org/abs/2204.12789
  10. [14]Nicolas Boullé, Diana Halikias, Samuel E. Otto, and Alex Townsend. Operator learning without the adjoint. arXiv preprint arXiv:2401.17739, 2024.arxiv.org/abs/2401.17739
  11. [15]Nacime Bouziani and David A. Ham. Escaping the abstraction: a foreign function interface for the Unified Form Language [UFL]. In NeurIPS Workshop on Differentiable Programming, 2021. doi: 10.48550/arXiv.2111.00945.DOI
  12. [16]Nacime Bouziani and David A. Ham. Physics-driven machine learning models coupling PyTorch and Firedrake. In ICLR Workshop on Physics for Machine Learning, 2023. doi: 10.48550/arXiv.2303.06871.DOI
  13. [17]Nacime Bouziani, David A. Ham, and Ado Farsi. Differentiable programming across the PDE and Machine Learning barrier. arXiv preprint arXiv:2409.06085, 2024.arxiv.org/abs/2409.06085
  14. [18]Dietrich Braess. Finite elements: Theory, fast solvers, and applications in solid mechanics. Cambridge University Press, 2001.
  15. [19]Johannes Brandstetter, Daniel E. Worrall, and Max Welling. Message Passing Neural PDE Solvers. In International Conference on Learning Representations, 2022.arxiv.org/abs/2202.03376
  16. [20]Susanne C Brenner and L. Ridgway Scott. The mathematical theory of finite element methods. Springer, 2008.
  17. [21]Shuhao Cao. Choose a Transformer: Fourier or Galerkin. In Advances in Neural Information Processing Systems, volume 34, pp. 24924–24940, 2021.arxiv.org/abs/2105.14995
  18. [22]R Courant. Variational methods for the solution of problems of equilibrium and vibrations. Bull. Amer. Math. Soc., 49(12):1–23, 1943.
  19. [23]Maarten V. de Hoop, Nikola B. Kovachki, Nicholas H. Nelsen, and Andrew M. Stuart. Convergence rates for learning linear operators from noisy data. SIAM-ASA J. Uncertain. Quantif., 11(2):480–513, 2023.
  20. [24]Tobin A. Driscoll, Nicholas Hale, and Lloyd N. Trefethen. Chebfun Guide. Pafnuty Publications, 2014.
  21. [25]Silviu Filip, Aurya Javeed, and Lloyd N. Trefethen. Smooth random functions, random ODEs, and Gaussian processes. SIAM Rev., 61(1):185–205, 2019.
  22. [26]Meire Fortunato, Tobias Pfaff, Peter Wirnsberger, Alexander Pritzel, and Peter Battaglia. Multiscale meshgraphnets. arXiv preprint arXiv:2210.00612, 2022.arxiv.org/abs/2210.00612
  23. [27]Nicola Rares Franco, Andrea Manzoni, and Paolo Zunino. Mesh-informed neural networks for operator learning in finite element spaces. J. Sci. Comput., 97(2):35, 2023.
  24. [28]Martin J Gander and Gerhard Wanner. From euler, ritz, and galerkin to modern computing. SIAM Rev., 54(4):627–666, 2012.
  25. [29]Christophe Geuzaine and Jean-François Remacle. Gmsh: A 3-D finite element mesh generator with built-in pre- and post-processing facilities. Int. J. Numer. Methods Eng., 79(11):1309–1331, 2009.
  26. [30]David A. Ham, Paul H. J. Kelly, Lawrence Mitchell, Colin J. Cotter, Robert C. Kirby, Koki Sagiyama, Nacime Bouziani, Sophia Vorderwuelbecke, Thomas J. Gregory, Jack Betteridge, Daniel R. Shapero, Reuben W. Nixon-Hill, Connor J. Ward, Patrick E. Farrel, Pablo D. Drubeck, India Marsden, Thomas H. Gibson, Miklós Homolya, Tianjiao Sun, Andrew T. T. McRae, Fabio Luporini, Alastair Gregory, Michael Lange, Simon W. Funke, Florian Rathgeber, Gheorghe-Teodor Bercea, and Graham R. Markall. Firedrake User Manual. Imperial College London and University of Oxford and Baylor University and University of Washington, 1st edition, 2023.DOI
  27. [31]Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1706.02216
  28. [32]Juncai He, Xinliang Liu, and Jinchao Xu. MgNO: Efficient Parameterization of Linear Operators via Multigrid. In International Conference on Learning Representations, 2024.
  29. [33]Quercus Hernández, Alberto Badías, David González, Francisco Chینesta, and Elías Cueto. Structure-preserving neural networks. J. Comput. Phys., 426, 2021.DOI
  30. [34]CP Jackson. A finite-element study of the onset of vortex shedding in flow past variously shaped bodies. J. Fluid Mech., 182:23–45, 1987.
  31. [35]Shuai Jiang, Jonas Actor, Scott Roberts, and Nathaniel Trask. A structure-preserving domain decomposition method for data-driven modeling. In Numerical Analysis Meets Machine Learning, volume 25 of Handbook of Numerical Analysis, pp. 469–514. Elsevier, 2024.arxiv.org/abs/2406.05571
  32. [36]Karthik Kashinath, M. Mustafa, Adrian Albert, et al. Physics-informed machine learning: case studies for weather and climate modelling. Phil. Trans. R. Soc. A, 379(2194):20200093, 2021.DOI
  33. [37]Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for Fourier neural operators. J. Mach. Learn. Res., 22(290):1–76, 2021.arxiv.org/abs/2107.07562
  34. [38]Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to PDEs. J. Mach. Learn. Res., 24(89):1–97, 2023.
  35. [39]Nikola B. Kovachki, Samuel Lanthaler, and Andrew M. Stuart. Operator learning: Algorithms and analysis. In Handbook of Numerical Analysis, volume 25, pp. 419–467. Elsevier, 2024.arxiv.org/abs/2402.15715
  36. [40]Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Wilson, et al. Learning skillful medium-range global weather forecasting. Science, 382(6677), 2023.DOI
  37. [41]Samuel Lanthaler, Siddhartha Mishra, and George E. Karniadakis. Error estimates for DeepONets: A deep learning framework in infinite dimensions. Trans. Math. Appl., 6(1), 2022.arxiv.org/abs/2102.09618
  38. [42]Jae Yong Lee, Seungchan Ko, and Youngjoon Hong. Finite element operator network for solving parametric pdes. arXiv preprint arXiv:2308.04690, 2023.arxiv.org/abs/2308.04690
  39. [43]Jae Yong Lee, Steffen Schotthöfer, Tianbai Xiao, Sebastian Krumscheid, and Martin Frank. Structure-Preserving Operator Learning: Modeling the Collision Operator of Kinetic Equations. arXiv preprint arXiv:2402.16613, 2024.arxiv.org/abs/2402.16613
  40. [44]Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems, 33:6755–6766, 2020.arxiv.org/abs/2006.09535
  41. [45]Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier Neural Operator for Parametric Partial Differential Equations. In International Conference on Learning Representations, 2021.
  42. [46]Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier Neural Operator with Learned Deformations for PDEs on General Geometries. J. Mach. Learn. Res., 24(388):1–26, 2023.DOI
  43. [47]Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, and Anima Anandkumar. neuraloperator GitHub repository. https://github.com/neuraloperator/neuraloperator, 2024a. Commit 8be96d0.
  44. [48]Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations. ACM/JMS J. Data Sci., 1(3):1–27, 2024b.arxiv.org/abs/2111.03794
  45. [49]Ilya Loshchilov, Frank Hutter, et al. Fixing weight decay regularization in Adam. arXiv preprint arXiv:1711.05101, 5, 2017.
  46. [54]Adam Paszke, Sam Gross, Francisco Massa, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32, 2019.arxiv.org/abs/1912.01703
  47. [55]Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizade nesheli, et al. Fourcastnet: A global data-driven high-resolution weather model using adaptive Fourier neural operators. arXiv preprint arXiv:2202.11214, 2022.arxiv.org/abs/2202.11214
  48. [56]Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter Battaglia. Learning Mesh-Based Simulation with Graph Networks. In International Conference on Learning Representations, 2021.arxiv.org/abs/2010.03409
  49. [57]Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.arxiv.org/abs/1710.05941
  50. [58]Pierre-Arnaud Raviart and Jean-Marie Thomas. A mixed finite element method for 2nd order elliptic problems. In Mathematical Aspects of Finite Element Methods: Proceedings of the Conference Held in Rome, December 10–12, 1975, pp. 292–315. Springer, 2006.
  51. [59]Nadim Saad, Gaurav Gupta, Shima Alizadeh, and Danielle Maddix Robinson. Guiding continuous operator learning through physics-based boundary constraints. In International Conference on Learning Representations, 2023.arxiv.org/abs/2212.07477
  52. [60]Cedric Taylor and Paul Hood. A numerical solution of the Navier-Stokes equations using the finite element technique. Comput. Fluids, 1(1):73–100, 1973.DOI
  53. [62]Tengfei Xu, Dachuan Liu, Peng Hao, and Bo Wang. Variational operator learning: A unified paradigm marrying training neural operators and solving partial differential equations. J. Mech. Phys. Solids, pp. 105714, 2024.DOI
  54. [63]Dmitry Yarotsky. Error bounds for approximations with deep ReLU networks. Neur. Netw., 94: 103–114, 2017.DOI
  55. [64]Hua qian You, Quinn Zhang, Colton J. Ross, Chung-Hao Lee, and Yue Yu. Learning deep implicit Fourier neural operators (IFNOs) with applications to heterogeneous material modeling. Comput. Methods Appl. Mech. Eng., 398:115296, 2022.DOI

Paper details

vlnc:W_4859v9rp
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents