betaFeedback

WorksPhysics and Astronomy

Operator Learning for Reconstructing Flow Fields from Sparse Measurements: an Energy Transformer Approach

Abstract

Machine learning methods have shown great success in various scientific areas, including fluid mechanics. However, reconstruction problems, where full velocity fields must be recovered from partial observations, remain challenging. In this paper, we propose a novel operator learning framework for solving reconstruction problems by using the Energy Transformer (ET), an architecture inspired by associative memory models. We formulate reconstruction as a mapping from incomplete observed data to full reconstructed fields. The method is validated on three fluid mechanics examples using diverse types of data: (1) unsteady 2D vortex street in flow past a cylinder using simulation data; (2) high-speed under-expanded impinging supersonic jets impingement using Schlieren imaging; and (3) 3D turbulent jet flow using particle tracking. The results demonstrate the ability of ET to accurately reconstruct complex flow fields from highly incomplete data (90% missing), even for noisy experimental measurements, with fast training and inference on a single GPU. This work provides a promising new direction for tackling reconstruction problems in fluid mechanics and other areas in mechanics, geophysics, weather prediction, and beyond.

Introduction

Machine learning methods have been widely applied in various scientific problems, e.g., physics [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], chemistry [12, 13, 14], biology [15, 16], epidemiology [17, 18], etc. This great success is due to the rapid simultaneous development of algorithms and computation power, which enable transforming the tasks that are difficult or expensive to solve by traditional methods into optimization tasks, which may be easier to tackle.

In this paradigm, operator learning is a prominent approach that has gained increasing interest recently [14, 19, 20, 21, 22]. In traditional methods, time-consuming and computationally expensive algorithms need to be run every time for a new input condition. On the other hand, operator learning aims to learn the mapping from general input conditions or distributions to solutions, i.e. the solution operator, by a neural network. Hence, neural operators can be used as surrogates to the traditional methods as the forward pass of a neural network usually is much faster than the conventional algorithms. Some recent progress in architectures like DeepONet [19], Fourier neural operator (FNO) [23], LNO [14] and ViTO [24] have demonstrated the great potential of operator learning in diverse applications. More specifically, DeepONet is a mesh-free method, allowing solutions to be evaluated at any point in the domain. FNO leverages the Fourier transform to efficiently model long-range dependencies and capture global structures in data. LNO is based on Laplace transform and excels in handling transient responses and non-periodic signals. ViTO utilizes the Vision Transformer to learn the operator from data on a regular mesh, offering a fast and efficient approach.

However, the situation is more challenging for reconstruction problems, i.e. full flow fields, from sparse and sporadic measurements. A common example is jet flow reconstruction, where measurement limitations often result in only partial access to the flow field. Although obtaining high-resolution flow fields is desirable, it can be costly. Therefore, the primary goal in these scenarios is to reconstruct the complete data from the observed, incomplete data. In addition, the objective of the operator learning method for reconstruction problems is to learn the mapping between the observed data and the full data with limited example pairs. Clearly, the incomplete and noisy nature of the data pose great challenges for the generalization ability of operator learning methods, because the test data is probably out of the distribution of the training data. For instance, the physics-informed diffusion model (PIDM) [25] effectively reconstructs flow fields from sparse measurements. However, this approach relies on measurements taken at regular mesh points, which limits its applicability to more general and irregular scenarios. Expanding operator learning methods to handle such cases remains an open area of research. For non-operator learning methods, Physics-Informed Neural Networks (PINNs) [26] have shown some success in reconstruction problems in fluid mechanics [4, 27, 28, 29, 30, 31, 32], because PINNs can incorporate the prior knowledge (physical laws) into the loss function. However, they are still limited by the requirement of the explicit form of the governing equations, relatively high computational cost and lack of ability to generalize.

Herein, we propose to use the Energy Transformer [33] for solving reconstruction problems in the operator learning framework. The Energy Transformer is inspired by recent developments in Dense Associative Memory models [34, 35], which are high memory storage capacity extensions of Hopfield networks [36]. Energy Transformer reconstructs the patterns from partial data and fill in the missing parts, which concurs with the objective of reconstruction problems. This makes Energy Transformer an ideal model for reconstruction operator. More specifically, the patterns are stored as the local minima of the energy function learned by the Energy Transformer. Then in reconstruction, the full data are inferred by minimizing the energy function starting from the observed data.

In this paper, we formulate the reconstruction problems in an operator learning framework with the Energy Transformer. We present three diverse examples in fluid mechanics using multimodal data, including real problems with noisy data, to demonstrate the effectiveness of the method.

Method

Operator learning framework

In this section, we formulate an operator learning framework for the reconstruction problem and introduce the proper notation. We denote a data sample as a collection of position-value pairs Pa={(xaA,vaA)}a=1Ma\mathcal{P}_a = \{(x^A_a,v^A_a)\}_{a=1}^{M_a}, where xaAx^A_a represents the position, vaAv^A_a denotes the data value at position xaAx^A_a, and MaM_a is the number of position-value pairs in sample aa. This collection Pa\mathcal{P}_a is referred to as a full data sample. Meanwhile, an observed data sample is defined as P~a={(xaA,vaA)}j∈Oa\tilde{\mathcal{P}}_a = \{(x^A_a,v^A_a)\}_{j\in\mathcal{O}_a}, where Oa\mathcal{O}_a indexes the observed position-value pairs. A dataset is defined as a collection of full data samples D={Pa}a=1M\mathcal{D} = \{\mathcal{P}_a\}_{a=1}^{M}, where MM is the number of samples. The goal of operator learning for the reconstruction problem is to learn a mapping R\mathcal{R} from the observed data sample to the full data sample, i.e.,

vaA=R(xaA,P~a),∀a′∉Oa.(1)v^A_a = \mathcal{R}(x^A_a,\tilde{\mathcal{P}}_a), \quad\forall a' \notin\mathcal{O}_a. \tag*{(1)}

In certain reconstruction problems, we have access to adequate full data samples, making a data-driven approach possible. This involves representing the reconstruction operator with a neural network Rθ\mathcal{R}_{\theta}. The observed data for training can be easily obtained by selecting some pairs from the full data samples. Furthermore, for new observed data, the neural network can predict the full data directly without running complex reconstruction algorithms, which is the advantage of operator learning compared to PINNs reconstruction or any other method [29].

We assume that the full data sample can be divided into patches of the same size, with the observed data consisting of some patches from the full data (i.e., the remaining patches are masked). Therefore, the positions xaAx^A_a become the patch indices. The model is expected to learn to fill in the masked patches based on the observed data. The key idea is to use the Energy Transformer (introduced in detail in the next section) to serve as a memory model to store the patterns of the full data, and retrieve the missing parts by querying the memory with the observed data.

The workflow of the proposed method is illustrated in Figure 1. The patcher and depatcher are fixed operations while the Energy Transformer, tokenizer and detokenizer are trainable neural networks. In the training stage, the full data are divided into patches and then tokenized, and the observed data are sampled from the full data patches and tokenized by the same tokenizer (masked patches are filled with fixed random noise and corresponding tokens are represented by dashed rectangles). Then, the Energy Transformer is trained on the tokens from the full data and observed data. In the evaluation stage, the observed data are patched and passed through the trained tokenizer, Energy Transformer and detokenizer sequentially, and then depatched to get the reconstructed data. The dashed arrows represent the operations that are only used in the training stage. In Figure 1(a), we include the patch and depatch operations. Given a dd-dimensional dataset (e.g., for images d=2d = 2) and patch sizes p1,…,pdp_1,\ldots,p_d, the patcher splits the input data of shape [L1,…,Ld][L_1,\ldots,L_d] into (L1/p1)×…×(Ld/pd)(L_1/p_1) \times\ldots\times(L_d/p_d) patches. For multi-modality data, the patches are stacked for further processing. The depatcher does the reverse. In (b) we include the tokenize (blue arrow) and detokenize (green arrow) operations. For the patched data, a neural network (tokenizer) is employed on each patch to generate vectors of length DD, hence the tokenized data has shape [N,D][N,D] where N=(L1/p1)×…×(Ld/pd)N = (L_1/p_1) \times\ldots\times(L_d/p_d). The detokenizer, which is also a neural network, takes the token vector of length DD and produces a patch of shape [p1,…,pd][p_1,\ldots,p_d]. In (c) we present the reconstruction operation based on the Energy Transformer. The Energy Transformer learns to fill in the tokens corresponding to the masked patches from the knowledge of the tokens generated by the observed data.

Figure 1. Overview of the proposed workflow. The dashed arrows represent the operations that are only used in the training stage. (a) Patch and depatch operations. (b) Tokenize (blue arrow) and detokenize (green arrow) operation. (c) Reconstruction operation based on the Energy Transformer.

Energy Transformer

We now introduce the structure of the Energy Transformer and how it is employed [33]. The Energy Transformer is built on the ET block (depicted in Figure 2), which consists of three components: layer normalization, energy attention layer, and Hopfield network. The energy attention layer can learn large-scale global patterns from the interaction between the input tokens, while the Hopfield network can learn small-scale local patterns within each token.

Figure 2. ET block. (a) The structure of the ET block. (b) The update rule in the forward pass of the ET block.

Layer Normalization. Each token x∈RDx \in\mathbb{R}^{D} passed to Energy Transformer is normalized by layer normalization. This enhances the model stability and generalization ability. The layer normalization is defined as:

g=γx−μσ2+ϵ+δ,(2)g = \gamma\frac{x-\mu}{\sqrt{\sigma^2+\epsilon}}+\delta, \tag*{(2)}

where μ\mu and σ\sigma are the mean and standard deviation of the input xx, γ\gamma and δ\delta are learnable parameters, and ϵ\epsilon is a small constant to avoid division by zero. Furthermore, the layer normalization can be defined as the partial derivative of a Lagrangian function [37, 38], which defines the energy function and the dynamical equations of temporal evolution leading to the decrease of the energy.

Multi-head Energy Attention. The main contribution of the Energy Transformer is the multi-head energy attention layer. Similar to the conventional attention mechanism, energy attention layer learns the interaction between tokens. However, in this case, each token generates query and key vectors, and there is no value vector. The output of the energy attention layer is a scalar energy value, where the minimization of this energy indicates that the queries of the masked tokens are aligned with the keys of the observed tokens. This alignment results in the output of the energy attention layer capturing the global structure of the full data with limited observations. The energy function of the multi-head energy attention layer is defined as:

EATT=−1β∑h=1H∑B=1Nlog⁡(∑B≠Cexp⁡(βAhBC)),(3)E^{ATT}=-\frac{1}{\beta}\sum_{h=1}^{H}\sum_{B=1}^{N}\log\left(\sum_{B\ne C}\exp(\beta A_{hBC})\right), \tag*{(3)}

where HH is the number of heads, NN is the number of tokens, β\beta is a constant, and AhBCA_{hBC} is the attention matrix between the BB-th and CC-th tokens in the hh-th head. More specifically, the attention matrix is defined as:

AhBC=∑α=1YKαhBQαhCA∈RH×N×N,KαhB=∑l=1DWαhlKgBlK∈RY×H×N,QαhC=∑l=1DWαhlQgClQ∈RY×H×N,(4)\begin{aligned} A_{hBC}&=\sum_{\alpha=1}^{Y}K_{\alpha hB}Q_{\alpha hC}\quad A\in\mathbb{R}^{H\times N\times N},\\ K_{\alpha hB}&=\sum_{l=1}^{D}W^{K}_{\alpha hl}g_{Bl}\quad K\in\mathbb{R}^{Y\times H\times N},\\ Q_{\alpha hC}&=\sum_{l=1}^{D}W^{Q}_{\alpha hl}g_{Cl}\quad Q\in\mathbb{R}^{Y\times H\times N}, \tag*{(4)} \end{aligned}

where WK,WQ∈RY×H×DW^{K},W^{Q}\in\mathbb{R}^{Y\times H\times D} are learnable parameters, gBg_{B} and gCg_{C} are the normalized vectors of the BB-th and CC-th tokens, DD is the dimension of each token, and YY is the internal dimension of the KK-QQ contraction.

Hopfield Network. The Hopfield network is employed to learn the local patterns within each token, ensuring that the token representations are consistent with real data. The energy function of the Hopfield network is defined as:

EHN=−∑B=1N∑μ=1KG(∑j=1DξμjgBj),(5)E^{HN}=-\sum_{B=1}^{N}\sum_{\mu=1}^{K}G\left(\sum_{j=1}^{D}\xi_{\mu j}g_{Bj}\right), \tag*{(5)}

where KK is the number of hidden units in the Hopfield network, ξ∈RK×D\xi\in\mathbb{R}^{K\times D} is a learnable matrix, which corresponds to “memories” in simple Hopfield networks, and G(⋅)G(\cdot) is the integral of the activation function. The Hopfield network functions analogously to the feedforward MLP in traditional transformer models, and it is applied to each token individually.

Forward Pass. The patches are initially tokenized by an embedding layer. The input tokens xx are then normalized by a layer normalization layer. Both the multi-head energy attention layer and the Hopfield network take the normalized gg and output a scalar energy. The energy of the ET block, denoted as EE, is the summation of these two energies. Then, the output of the ET block is the local minimizer of the energy function EE, which is obtained by iteratively updating the tokens according to the rule: xt+1=xt−α∇gE(g(xt))x^{t+1}=x^{t}-\alpha\nabla_{g}E(g(x^{t})), where α\alpha is the step size constant. The tokens are initialized as x0=xx^{0}=x and iterated until convergence. In practice, the convergence is achieved after a certain number of iterations. Finally, the tokens are detokenized to obtain the reconstructed data.

Results

In this section, we demonstrate the effectiveness of the proposed method for three different examples in fluid mechanics. These examples validate the ability of the Energy Transformer to learn the reconstruction operator for different types of data with small errors. In all these examples, two-layer MLPs with ReLU activation are used for the tokenizer and detokenizer. The patches are flattened to be tokenized. The hyperparameters for the model are in Table 1.

TypeHyperparameterValue
Patch and tokenizePatch size8×88 \times8
Patch and tokenizeToken dim DD256
Patch and tokenizeTokenizer hidden dim256
Patch and tokenizeDetokenizer hidden dim256
Energy TransformerHeads of energy attention layer8
Energy TransformerHopfield typesoftmax
Energy TransformerHopfield hidden dim2048
Update ruleIteration steps TT12
Update ruleα\alpha0.1
TrainingOptimizerAdam
TrainingBatch size16

Table 1. Model hyperparameters (the same for all three examples).

2D Vortex Street

In flow past bluff bodies the von Karman vortex street is formed in the wake above a certain Reynolds number, e.g. Re≈50Re \approx50 for flow past a circular cylinder [39]. Here, we will use simulated data to demonstrate the ET-based flow field reconstruction. The spatial domain is [−4,12]×[−4,4][-4, 12] \times[-4, 4], with the cylinder diameter D=1D = 1 and the Reynolds number is 400. The flow is simulated by a high-order spectral element method and we obtain 100 snapshots [39]. The training data is the first 80 time steps and the model is tested on the rest 20 time steps. The data shape of each snapshot is [128,256,4][128, 256, 4], which is the stack of the velocity field, pressure field and the temperature field. Each patch is of shape [8,8,4][8, 8, 4] due to the multi-modality of the data. The observed data is randomly masked with 90% of the patches, in other words, only 10% of the patches are observed. The model is trained based on the relative mean squared error (RMSE) loss for 10,000 epochs with learning rate 10−410^{-4}. The training loss history is depicted in Figure 4, which shows that the model converges well. On a single NVIDIA A6000 GPU, the training time is around 90 minutes and the inference time is around 0.0027 seconds per snapshot.

Figure 4. Training loss of the 2D vortex street example.

Reconstruction results at time step 90 are shown in Figure 5. The errors of reconstruction are shown in Figure 6.

Reconstruction results of a 2D vortex street flow showing full, observed, and reconstructed fields for velocity components $u$ and $v$, pressure $p$, and temperature $T$

Figure 5. (a) Velocity component uu (velocity along horizontal spatial coordinate). (b) Velocity component vv (velocity along vertical spatial coordinate). (c) Pressure field pp. (d) Temperature field TT. Reconstruction results of a 2D vortex street flow at time step 90. Each row in the figure displays a different physical field of the flow. In each panel, the x-axis represents the horizontal spatial coordinate, while the y-axis represents the vertical spatial coordinate, with values in physical units relevant to the flow domain. The columns show (from left to right): (1) the full field, representing the original, high-resolution simulation data; (2) the observed field, displaying sparse measurements of the field, typically available from limited sensors or measurements; and (3) the reconstructed field, where the sparse observations have been processed by an algorithm to estimate the full field. The color in each plot represents the magnitude of each physical field, with warmer colors (e.g., red) indicating higher values and cooler colors (e.g., blue) indicating lower values. Time step 90 is chosen because it is in the test dataset and it captures a characteristic snapshot of the vortex street pattern, where complex flow structures are fully developed, making it a representative example for illustrating the algorithm’s performance in reconstructing detailed flow features.

Errors of the 2D vortex street example at time step 90 for four physical components

Figure 6. Errors of the 2D vortex street example at time step 90 for four physical component. The error is defined as (x^−x)/∥x∥2(\hat{x}-x)/\lVert x\rVert_2, xx and x^\hat{x} are the ground truth and reconstructed results. Here the xx and yy axis are the same as in Figure 5. The colors close to red and blue indicate large positive and negative errors respectively. Errors of the vv component is most significant, which is a common phenomenon in vortex reconstruction [27].

The relative mean squared error (RMSE) and the relative error (defined as the RMSE divided by the l2l^2 norm of the full data) on the test set (the last 20 snapshots) are listed in Table 2. The errors are small, which indicates the reconstruction operator is learned effectively in this example.

RMSERelative error
u⃗\vec{u}0.02860.0405
pp0.00810.0798
TT0.00800.0667

Table 2. Reconstruction errors of the 2D vortex street example. u⃗=(u,v)\vec{u}=(u,v).

Ablation Study. To evaluate the data efficiency of our method, we train the model using reduced subsets of the training data, specifically the first 25%, 50%, and 75% of the full dataset. The results are presented in Table 3. The relative reconstruction error remains low even when only the first 50% of the training data is used. However, the error increases significantly when the training data is further reduced to the first 25% of the original dataset. Since the first 25% of the data does not contain a complete period (as illustrated in Figure 3), it is not surprising that the model fails to capture the periodicity, as it lacks sufficient information from the limited data.

Figure 3. Signature of the 2D vortex street dataset: values of uu, vv, pp, TT (horizontal velocity, vertical velocity, pressure and temperature) at the center of the domain versus the time. The red dashed line represents the split of training and test data. Periodicity can be observed in all components, which is characteristic of the vortex street.

100%75%50%25%
u⃗\vec{u}0.04050.04940.05950.1616
pp0.07980.12200.15260.3116
TT0.06670.09850.12180.2493

Table 3. Relative reconstruction errors with fewer training data.

2D Jet Flow

The jet impinging experiments provide detailed data about the interaction between fluid and surface. In this example, the dataset is captured using Schlieren imaging techniques, hence represents the gradient of the density of the fluid [40]. The experiments are conducted in the following way: the air is compressed and the temperature is approximately equal to ambient temperature, then air jet is utilized to impact the plate. At the same time, a conventional Z-type Schlieren system is adopted to visualize above under-expanded impinging jet flow. The Mach number of the flow at the nozzle exit is 1. The experiment setup is depicted in Figure 7. Further details can be found in [41].

Illustration of the supersonic jet impinging experiment setup

Figure 7. Illustration of the supersonic jet impinging experiment setup.

The dataset contains 1000 snapshots of the flow field, where each snapshot has shape [128,256][128, 256], and the spatial domain is [−0.8,5.6]×[−1.45,1.75][-0.8, 5.6] \times[-1.45, 1.75]. The values of the dataset are in the range [0,255][0, 255]. The observed data is randomly masked with 90% of the patches. The model is trained with RMSE loss for 10,000 epochs with learning rate 10−410^{-4}. The training loss is shown in Figure 9. On a single NVIDIA A6000 GPU, the training time is around 150 minutes and the inference time is around 0.0025 second per snapshot.

Figure 9. Training loss of the 2D jet flow example.

Unlike the vortex street example, the data in this example are collected from real fluid experiments, hence they contain large noise. The results and error of our methods are shown in Figure 10 and 11. The RMSE is 10.4124 and the relative error is 0.1313 on the test set (last 200 snapshots). Despite the noisy nature of the data (shown in the Figure 8), the ET method still achieves a good performance.

Figure 8. Signature of the 2D jet flow dataset: data values at the center of the domain versus the time. The red dashed line represents the split of training and test data.

Figure 10. Results of the jet impinging example at timestep 900. The x and y axis represent the horizontal and vertical spatial coordinate. Since data has only one component, this is equivalent to image reconstruction. Time step 900 is chosen because it is in the middle of the test dataset.

Errors of the jet impinging example at timestep 900

Figure 11. Errors of the jet impinging example at timestep 900. The x and y axis are the same as in Figure 10. The errors concentraded at small-scale fluctuations.

3D Turbulent Jet Flow

Turbulent jet flow experiments facilitate the understanding of the potential core of the jet and the breakdown to turbulence downstream. In this example, the data is collected from experiments conducted at TU Delft, where the velocity measurements of jet flow can be obtained by tomographic particle tracking velocimetry (PTV). The PTV data are then processed by PINNs [42] to obtain the pressure fields and continuous velocity field.

The dataset contains 200 snapshots of the flow field, each of which has shape [64,96,64,4][64, 96, 64, 4]. The spatial domain is rescaled into [−1,1]×[0,3]×[−1,1][-1, 1] \times[0, 3] \times[-1, 1]. Each patch is of shape [8,8,8,4][8, 8, 8, 4] due to the multi-modality of the data. The observed data is masked with a fixed pattern, where the last half of the jet flow center is observed and all the other patches are masked (illustrated in Figure 14). The mask rate is around 90%. The model is trained with RMSE loss for 1,000 epochs with learning rate 10−310^{-3}. The training loss is depicted in Figure 13. On a single NVIDIA A6000 GPU, the training time is around 70 minutes and the inference time is around 0.1 second per snapshot.

Figure 13. Training loss of the 3D turbulent jet example.

Figure 14. Illustration of the masks in the 3D turbulent jet flow example.

This example is even more difficult because the dataset is in 3D and also collected from noisy experiments. The results and error of our methods are shown in Figure 15 and 16.

Figure 15. Reconstruction results of a 3D turbulent jet flow at timestep 180, visualized on the slice at z=0z = 0. Each row represents a different physical field: (a) velocity component uu (horizontal velocity), (b) velocity component vv (vertical velocity), (c) velocity component ww (depth-wise velocity), and (d) pressure field pp. The columns show (from left to right): (1) the full field, representing the original, high-resolution simulation data; (2) the observed field, displaying sparse measurements of the field, typically available from limited sensors or measurements; and (3) the reconstructed field, where the sparse observations have been processed by an algorithm to estimate the full field. The color in each plot represents the magnitude of each physical field, with warmer colors (e.g., red) indicating higher values and cooler colors (e.g., blue) indicating lower values. Timestep 180 is selected to illustrate a moment in the jet flow where turbulent structures are well-developed and it is in the test dataset.

Errors of the 3D turbulent jet flow example at timestep 180 at the slice $z = 0$

Figure 16. Errors of the 3D turbulent jet flow example at timestep 180 at the slice z=0z = 0. The x and y axis are the same as in Figure 15. The errors concentrate at the upstream flow.

The RMSE and the relative error on the test set (the last 40 snapshots) are listed in Table 4. The relative error of pp is small, but the error for u⃗\vec{u} is much larger. However, the relative errors are still satisfactory, because the task is essentially hard due to the 3D nature, non-periodicity (shown in Figure 12) and the large experimental noise.

Figure 12. Signature of the 3D jet flow dataset: values of uu, vv, ww, pp (velocity along xx, yy, zz axis and the pressure) at the center of the domain versus the time. The red dashed line represents the split of training and test data. Some periodicity can be observed in vv, pp components but uu, ww components are not periodic.

RMSERelative error
u⃗\vec{u}0.02370.2749
pp0.00740.0993

Table 4. Reconstruction errors of the 3D turbulent jet flow example. u⃗=(u,v,w)\vec{u} = (u, v, w).

Comparison with Other Methods

The Energy Transformer presents a unique approach to reconstruction problems when compared to other operator learning methods like PINNs, DeepONet, and FNO. PINNs, which are used in Artificial Intelligent Velocimetry, can incorporate physical laws into reconstruction process. Due to their minimization nature, they can handle observed data at random positions. However, the inference cost is high as the process requires training the model to minimize the physics equation loss and data loss, which are highly non-convex. Furthermore, PINNs suffer from poor generalization ability for extrapolation tasks.

On the other hand, DeepONet and FNO are purely data-driven methods. Their inference cost is much lower because the inference does not include minimization and is just a feedforward pass. However, they cannot handle observed data at random positions as covering all possible configurations is prohibitively expensive.

The Energy Transformer integrates the benefits of these approaches while addressing their limitations. Unlike other neural operators, Energy Transformer learns a memory-like energy function, whose minimizer is the reconstruction result. The inference is a minimization process on a relatively simple function, which is more computationally efficient than PINNs. And due to the strong inductive bias of the simple form of energy function, it can handle random observed data positions effectively without exhaustive search over all possible configurations. This unique capability makes the Energy Transformer a powerful tool for reconstruction tasks, offering low computational costs with flexibility for irregular sparse data measurements.

Summary

In this paper, we have proposed an operator learning framework with the Energy Transformer for reconstruction problems in fluid mechanics, where the goal is to recover the full flow field from sparse observed data. We demonstrated the effectiveness of the proposed method by three examples in fluid mechanics. The results show that the Energy Transformer can learn the reconstruction operator for different types of data with tolerable errors, even for the real problem with noisy data. Furthermore, its training and inference costs are relatively low compared with PINNs and the artificial intelligent velocimetry (AIV) method. In practice, sparse flow measurements can be accomplished in different parts of the domain, e.g. with small windows of particle image velocimetry (PIV), to cover the spatially varying dynamics across the domain. It is also possible to use direct numerical simulation for the training stage and build foundation models based on the Energy Transformer framework we presented herein, then the observed data will be obtained using selective experiments.

As a memory model, the Energy Transformer performs well when the test data resembles the training data, a scenario often encountered in fluid mechanics. However, for general evolution problems, the Energy Transformer may not perform as effectively, as it learns to complete individual snapshots rather than understand the underlying evolution rules. But even in this case, since the energy attention mechanism can learn large-scale structures in flow field, the Energy Transformer can still provide an averaged representation of the training data, which can be useful for limited extrapolation. Our work provides a new perspective for solving reconstruction problems in fluid mechanics and offers a promising direction for other engineering and scientific applications, e.g., in solid mechanics using digital impage correlation (DIC), in geophysics, in weather prediction, and more.

Acknowledgements

The authors from Brown University acknowledge support of the DARPA-ABAQuS program.

References

  1. [1]Ilan Price, Alvaro Sanchez-Gonzalez, Ferran Alet, Tom R. Andersson, Andrew El-Kadi, Dominic Masters, Timo Ewalds, Jacklynn Stott, Shakir Mohamed, Peter Battaglia, Remi Lam, and Matthew Willson. Probabilistic weather forecasting with machine learning. Nature, Dec 2024.DOI
  2. [2]Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J. R. Ruiz, Julian Schrittwieser, Gregorz Swirszcz, David Silver, Demis Hassabis, and Pushmeet Kohli. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47–53, 2022.DOI
  3. [3]Georges Aad, Tatekiv Abajyan, Brad Abbott, Jalal Abdallah, S Abdel Khalek, Ahmed Ali Abdelalim, R Aben, B Abi, M Abolins, OS AbouZeid, et al. Observation of a new particle in the search for the standard model higgs boson with the atlas detector at the lhc. Physics Letters B, 716(1):1–29, 2012.DOI
  4. [4]Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis. Physics-informed neural networks (pinns) for fluid mechanics: A review. Acta Mechanica Sinica, 37(12):1727–1738, 2021.arxiv.org/abs/2105.09506
  5. [5]Giuseppe Carleo and Matthias Troyer. Solving the quantum many-body problem with artificial neural networks. Science, 355(6325):602–606, 2017.DOI
  6. [6]Pengzhan Jin, Zhen Zhang, Aiqing Zhu, Yifa Tang, and George Em Karniadakis. Sympnets: Intrinsic structure-preserving symplectic networks for identifying hamiltonian systems. Neural Networks, 132:166–179, 2020.arxiv.org/abs/2001.03750
  7. [7]Qian Zhang, Adar Kahana, George Em Karniadakis, and Panos Stinis. Sms: Spiking marching scheme for efficient long time integration of differential equations. Journal of Computational Physics, 516:113363, 2024.arxiv.org/abs/2211.09928
  8. [8]Qian Zhang, Chenxi Wu, Adar Kahana, Youngeun Kim, Yuhang Li, George Em Karniadakis, and Priyadarshini Panda. Artificial to spiking neural networks conversion for scientific machine learning. arXiv:2308.16372, 2023.arxiv.org/abs/2308.16372
  9. [9]Bradley H. Theilman, Qian Zhang, Adar Kahana, Eric C. Cyr, Nathaniel Trask, James B. Aimone, and George Em Karniadakis. Spiking physics-informed neural networks on loihi 2. 2024 Neuro Inspired Computational Elements Conference (NICE), 2024.
  10. [10]Chenxi Wu, Min Zhu, Qinyang Tan, Yadhu Kartha, and Lu Lu. A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 403:115671, 2023.DOI
  11. [11]Ruyin Wan, Qian Zhang, and George Em Karniadakis. Randomized forward mode gradient for spiking neural networks in scientific machine learning. arXiv:2411.07057, 2024.arxiv.org/abs/2411.07057
  12. [12]John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583–589, 2021.DOI
  13. [13]Amil Merchant, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. Scaling deep learning for materials discovery. Nature, 624(80):80–85, 2023.DOI
  14. [14]Qianying Cao, Somdatta Goswami, and George Em Karniadakis. Lno: Laplace neural operator for solving differential equations. arXiv:2303.10528, 2023.arxiv.org/abs/2303.10528
  15. [15]Mitchell Daneker, Zhen Zhang, George Em Karniadakis, and Lu Lu. Systems biology: Identifiability analysis and parameter identification via systems-biology informed neural networks. arXiv:2202.01723, 2022.arxiv.org/abs/2202.01723
  16. [16]Qian Zhang, Konstantina Sampani, Mengjia Xu, Shengze Cai, Yixiang Deng, He Li, Jennifer K. Sun, and George Em Karniadakis. AOSLO-net: A Deep Learning-Based Method for Automatic Segmentation of Retinal Microaneurysms From Adaptive Optics Scanning Laser Ophthalmoscopy Images. Translational Vision Science & Technology, 11(8):7–7, 08 2022.DOI
  17. [17]Sheng Zhang, Joan Ponce, Zhen Zhang, Guang Lin, and George Em Karniadakis. An integrated framework for building trustworthy data-driven epidemiological models: Application to the covid-19 outbreak in new york city. PLOS Computational Biology, 17(9):1–29, 09 2021.DOI
  18. [18]Ehsan Kharazmi, Min Cai, Xiaoning Zheng, Zhen Zhang, Guang Lin, and George Em Karniadakis. Identifiability and predictability of integer- and fractional-order epidemiological models using physics-informed neural networks. Nature Computational Science, 1(11):744–753, 2021.DOI
  19. [19]Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218–229, 2021.
  20. [20]Somdatta Goswami, Minglang Yin, Yue Yu, and George Em Karniadakis. A physics-informed variational deeponet for predicting crack path in quasi-brittle materials. Computer Methods in Applied Mechanics and Engineering, 391:114587, 2022.DOI
  21. [21]Adar Kahana, Qian Zhang, Leonard Gleyzer, and George Em Karniadakis. Spiking neural operators for scientific machine learning. arXiv:2205.10130, 2022.arxiv.org/abs/2205.10130
  22. [22]Enrui Zhang, Adar Kahana, Eli Turkel, Rishikesh Ranade, Jay Pathak, and George Em Karniadakis. A hybrid iterative numerical transferable solver (hints) for pdes based on deep operator network and relaxation methods. arXiv:2208.13273, 2022.
  23. [23]Zongyi Li, Nikola Kovachki, Kamyar Azizadehnesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv:2010.08895, 2020.arxiv.org/abs/2010.08895
  24. [24]Oded Ovadia, Adar Kahana, Panos Stinis, Eli Turkel, Dan Givoli, and George Em Karniadakis. Vito: Vision transformer-operator. Computer Methods in Applied Mechanics and Engineering, 428:117109, 2024.arxiv.org/abs/2303.08891
  25. [25]Dule Shu, Zijie Li, and Amir Barati Farimani. A physics-informed diffusion model for high-fidelity flow field reconstruction. Journal of Computational Physics, 478:111972, 2023.DOI
  26. [26]Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019.DOI
  27. [27]Xiaowei Jin, Shengze Cai, Hui Li, and George Em Karniadakis. Nsfnets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations. Journal of Computational Physics, 426:109951, 2021.DOI
  28. [28]Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science, 367(6481):1026–1030, 2020.DOI
  29. [29]Shengze Cai, He Li, Fuyin Zheng, Fang Kong, Ming Dao, George Em Karniadakis, and Subra Suresh. Artificial intelligence velocimetry and microaneurysm-on-a-chip for three-dimensional analysis of blood flow in physiology and disease. Proceedings of the National Academy of Sciences, 118(13):e2100697118, 2021.DOI
  30. [30]Shengze Cai, Callum Gray, and George Em Karniadakis. Physics-informed neural networks enhanced particle tracking velocimetry: An example for turbulent jet flow. IEEE Transactions on Instrumentation and Measurement, 2024.DOI
  31. [31]Kimberly AS Boster, Shengze Cai, Antonio Ladrón-de Guevara, Jiatong Sun, Xiaoning Zheng, Ting Du, John H Thomas, Maiken Nedergaard, George Em Karniadakis, and Douglas H Kelley. Artificial intelligence velocimetry reveals in vivo flow rates, pressure gradients, and shear stresses in murine perivascular flows. Proceedings of the National Academy of Sciences, 120(14):e2217744120, 2023.DOI
  32. [32]Juan Diego Toscano, Theo Käufer, Martin Maxey, Christian Cierpka, and George Em Karniadakis. Inferring turbulent velocity and temperature fields and their statistics from lagrangian velocity measurements using physics-informed Kolmogorov-Arnold Networks. arXiv:2407.15727, 2024.arxiv.org/abs/2407.15727
  33. [33]Benjamin Hoover, Yuchen Liang, Bao Pham, Rameswar Panda, Hendrik Strobel, Duan Horng Chau, Mohammed Zaki, and Dmitry Krotov. Energy transformer. Advances in Neural Information Processing Systems, 36, 2024.arxiv.org/abs/2302.07253
  34. [34]Dmitry Krotov and John J Hopfield. Dense associative memory for pattern recognition. Advances in Neural Information Processing Systems, 29, 2016.arxiv.org/abs/1606.01164
  35. [35]Dmitry Krotov. A new frontier for hopfield networks. Nature Reviews Physics, 5(7):366–367, 2023.DOI
  36. [36]John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554–2558, 1982.
  37. [37]Dmitry Krotov and John J Hopfield. Large associative memory problem in neurobiology and machine learning. In International Conference on Learning Representations, 2021.arxiv.org/abs/2008.06996
  38. [38]Dmitry Krotov. Hierarchical associative memory. arXiv preprint arXiv:2107.06446, 2021.arxiv.org/abs/2107.06446
  39. [39]Maziar Raissi, Zhicheng Wang, Michael S Triantafyllou, and George Em Karniadakis. Deep learning of vortex-induced vibrations. Journal of Fluid Mechanics, 861:119–137, 2019.
  40. [40]Gary S Settles and Michael J Hargather. A review of recent developments in schlieren and shadowgraph techniques. Measurement Science and Technology, 28(4):042001, 2017.DOI
  41. [41]Zhibo Wang, Xiangru Li, Luhan Liu, Xuecheng Wu, Pengfei Hao, Xiwen Zhang, and Feng He. Deep-learning-based super-resolution reconstruction of high-speed imaging in fluids. Physics of Fluids, 34(3), 2022.DOI
  42. [42]Shengze Cai, Callum Gray, and George Em Karniadakis. Physics-informed neural networks enhanced particle tracking velocimetry: An example for turbulent jet flow. IEEE Transactions on Instrumentation and Measurement, 2024.DOI

Paper details

vlnc:W_4ev4pery
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents