betaFeedback

WorksComputer Science

An Attention-based Spatio-Temporal Neural Operator for Evolving Physics

Abstract

In scientific machine learning (SciML), a key challenge is learning unknown, evolving physical processes and making predictions across spatio-temporal scales. For example, in real-world manufacturing problems like additive manufacturing, users adjust known machine settings while unknown environmental parameters simultaneously fluctuate. To make reliable predictions, it is desired for a model to not only capture long-range spatio-temporal interactions from data but also adapt to new and unknown environments; traditional machine learning models excel at the first task but often lack physical interpretability and struggle to generalize under varying environmental conditions. To tackle these challenges, we propose the Attention-based Spatio-Temporal Neural Operator (ASNO), a novel architecture that combines separable attention mechanisms for spatial and temporal interactions and adapts to unseen physical parameters. Inspired by the backward differentiation formula (BDF), ASNO learns a transformer for temporal prediction and extrapolation and an attention-based neural operator for handling varying external loads, enhancing interpretability by isolating historical state contributions and external forces, enabling the discovery of underlying physical laws and generalizability to unseen physical environments. Empirical results on SciML benchmarks demonstrate that ASNO outperforms over existing models, establishing its potential for engineering applications, physics discovery, and interpretable machine learning.

Introduction

The increasing complexity of spatio-temporal data in scientific fields such as fluid dynamics, material science, and manufacturing, particularly in the context of Digital Twin technologies, has made the interpretability and accuracy of machine learning models critically important. Digital Twins—virtual replicas of physical systems—are becoming integral to modern manufacturing processes, enabling real-time monitoring, simulation, and optimization of physical assets [12, 33, 13]. In such settings, the critical challenge lies in creating models that generalize to new physical parameters (e.g., PDE coefficients) while maintaining alignment with underlying physical principles to enable robust predictions and insights into system behavior. This dual capability enhances decision-making and facilitates system optimization by aligning predictions with real-world physics [32, 14, 5].

Existing methods often struggle to generalize across unseen physical environments, particularly when tasked with interpreting spatial and temporal dependencies in a separable yet synergistic manner. Neglecting this separability can obscure the physical mechanisms driving system behavior, resulting in less interpretable and reliable predictions [30, 27]. For example, in material modeling, observable phenomena such as deformation fields may provide limited information, and inferring hidden parameters such as material properties remains an ill-posed problem. Similarly, in physics-based applications like manufacturing, accurately capturing the interplay between spatial interactions and evolving temporal dynamics is essential for refining product quality and operational efficiency (Ko et al., 2022).

While existing data-driven surrogate architectures—such as transformer encoders, U-Net, DeepONet, and newer operator-based frameworks like the General Neural Operator Transformer (GNOT), Transolver, and Fourier Neural Operator (FNO) have been developed to capture spatio-temporal dependencies in physical systems [41, 11, 35] (Lu et al. 2021; Wu et al. 2024), they often lack inherent mechanisms for reliable generalization to unseen conditions or for disentangling spatial and temporal influences. Transformers, despite their strength in modeling long-range temporal correlations via self-attention, frequently underrepresent the spatial interactions essential for nonlinear phenomena such as turbulent flows, reaction–diffusion processes, and phase transitions. Moreover, many of these approaches struggle to separate the roles of temporal evolution and spatial coupling, which limits their applicability in PDE-governed settings. Finally, their adaptability to systems with varying PDE coefficients remains limited, reducing their effectiveness across diverse physical scenarios. While the specific implementations vary—from sequence-based encoders to spectral and attention-driven operators—all of these methods share the common challenge of balancing expressivity with robust, physics-informed generalization.

To address these limitations, we propose the Attention-based Spatio-Temporal Neural Operator (ASNO). In ASNO, a transformer encoder captures temporal dynamics in the solution field, while an attention-based nonlocal operator handles the long-range spatial dependencies and the interplay between solution and loading/environment fields. Inspired by the implicit-explicit (IMEX) scheme, ASNO leverages the backward differentiation formula (BDF) to separate temporal effects from spatial loading/solution interactions. This separable architecture enhances generalizability to unseen physical parameters (e.g., initial conditions, loadings, environments) while improving interpretability.

The key contributions of this paper are as follows:

  • We introduce ASNO, a novel architecture inspired by the IMEX scheme, which discovers separate mechanisms for capturing temporal effects and spatial loading/solution interactions.

  • ASNO discovers a spatio-temporal relationship from data, enabling zero-shot generalizability to PDEs with unseen initial conditions, loadings, and environments.

  • The attention mechanism provides insights into the separable contributions of temporal and spatial dynamics, overcoming the limitations of models focused on only one of the two.

  • We conduct experiments on examples ranging from chaotic systems and PDE solving problems to real-world applications. Results demonstrate the advantages of ASNO over baseline transformer models and neural operator models in terms of generalizability, long-term stability, and interpretability.

Background and Related Work

Transformers for Time Series Data

Traditional time series models such as AutoRegressive Integrated Moving Average (ARIMA) and Long Short-Term Memory (LSTM) have effectively captured short-term dependencies but face limitations when dealing with long-range dependencies and highly non-linear, multivariate data [28, 9]. Transformers, initially developed for natural language processing (NLP), revolutionized time series modeling by introducing self-attention mechanisms that capture dependencies across long sequences [34, 31, 40, 43]. This attention mechanism allows transformers to weigh different parts of the input sequence based on their relevance, making them highly suitable for long-sequence tasks [42].

Despite their strengths, transformers can be computationally expensive. Their complexity grows quadratically with sequence length, limiting scalability, especially when applied to large datasets [7]. This has driven the development of models like Informer [43] and Reformer [16], which employ sparse attention mechanisms to reduce complexity. Models such as the Temporal Fusion Transformer (TFT) have improved upon traditional transformers by using attention scores to enhance interpretability and effectively address practical challenges like variable input lengths and missing data [21]; these features are byproducts of TFT’s core innovations in handling complex temporal dependencies.

These developments underscore the adaptability of transformer architectures in handling diverse temporal tasks, though they remain limited in their ability to generalize to new physical parameters (e.g., PDE coefficients) and lack mechanisms that explicitly separate spatial and temporal interactions for interpretability. [6].

Neural Operators in Scientific Machine Learning

Neural operators have become essential in scientific machine learning, particularly for modeling mappings between function spaces (Li et al., 2020). These models are effective at solving forward problems in physics-based systems, such as those governed by PDEs, by serving as efficient surrogates for physical systems and offering black-box approximations without explicitly interpreting the underlying physical laws [18, 2, 3, 36]. For instance, DeepONet utilizes separate networks to encode input functions and evaluation points, thereby learning the nonlinear operator that maps function inputs to output values at arbitrary spatial locations. (Lu et al., 2021; 2019). However, as the problem’s dimensionality increases, DeepONet faces scalability issues (Mandal et al., 2024). Similarly, Fourier Neural Operator (FNO), which operates in the frequency domain to better capture global spatial dependencies, becomes computationally expensive either at higher resolutions or when handling non-smooth data (Li et al., 2020). Recent advancements, such as the GNOT and Transolver, have improved the flexibility of neural operators for handling complex, spatially distributed data. GNOT incorporates heterogeneous normalized attention layers and geometric gating mechanisms, enabling it to address multi-scale problems and integrate diverse data sources effectively [11]. Similarly, Transolver leverages transformer-based architectures to solve PDEs on irregular geometries, enhancing its adaptability to various applications. However, these models often struggle to generalize to unseen PDE parameters (e.g., new initial conditions or environments) and lack interpretability in disentangling spatial and temporal interactions [35] (Wu et al. 2024).

Most neural operators primarily address forward PDE problems—predicting future states from current conditions by capturing spatio-temporal dependencies—but struggle with inverse problems, like reconstructing unknown PDE parameters, which often require prior knowledge of governing equations and are generally ill-posed [26]. Integrating forward and inverse problem-solving capabilities into a single spatio-temporal framework enables the development of more versatile models that can not only predict system behaviors but also infer hidden states and causal mechanisms. This dual functionality enhances model robustness and adaptability—qualities vital for scientific applications where elucidating system dynamics, which may inherently encode underlying causes, contributes to more informed simulation, and potentially supports control and optimization efforts, even if these are ultimately executed through computational rather than interpretative means [15].

Model Architecture

This section presents ASNO, a spatio-temporal neural operator integrating the Transformer Encoder for temporal dependencies and the nonlocal attention mechanism for spatial interactions. We outline the Transformer Encoder, followed by the nonlocal attention operator model, and explain their combined role in the form of a backward differentiation formula (BDF) for complex spatio-temporal prediction.

Backward Differentiation Formula (BDF)

Backward differentiation formula (BDF) is a family of popular numerical schemes for stiff differential equations, thanks to its high-order accuracy and large region of absolute stability [8]. For a initial-value differential equation:

X˙=F(t,X),X(t0)=X0,\dot{X} = F(t, X), \quad X(t_{0}) = X_{0},

the general formula for a BDF can be written as

∑k=1nαkXm−k+1+Xm+1=ΔtβF((m+1)Δt,Xm+1).(1)\sum_{k=1}^{n} \alpha_{k}X_{m-k+1} + X_{m+1} = \Delta t \beta F((m+1)\Delta t, X_{m+1}). \tag*{(1)}

Here, Δt\Delta t denotes the time step size, and XmX_m is the approximated solution at time instance t0+mΔtt_0 + m\Delta t. Notice that (1) can be naturally decomposed as the two-phase implicit-explicit (IMEX) scheme [1]:

−∑k=1nαkXm−k+1=X~m+1,(2)-\sum_{k=1}^{n} \alpha_k X_{m-k+1} = \widetilde{X}_{m+1}, \tag*{(2)}
X~m+1−Xm+1+ΔtβF((m+1)Δt,Xm+1)=0.(3)\widetilde{X}_{m+1} - X_{m+1} + \Delta t \beta F((m+1)\Delta t, X_{m+1}) = 0. \tag*{(3)}

In the explicit step (2), nn historical steps are accumulated, hence it characterizes the temporal interactions. Notice that when the system is homogeneous, i.e., F≡0F \equiv0, we have X~m+1=Xm+1\widetilde{X}_{m+1} = X_{m+1}. That means, the explicit steps generate an estimated temporal extrapolated solution for a system without external loading FF. On the other hand, the implicit step (3) takes X~m+1\widetilde{X}_{m+1} as the input and tries to solve a (possibly nonlinear) static equation of Xm+1X_{m+1}. Hence, this step is characterized by capturing the spatial interaction between components of XX, as well as their interplay with the external loading FF.

Based on this idea, our key innovation in ASNO is to design a separable architecture that captures the temporal and spatial effects respectively. In particular, our architecture resembles the implicit-explicit decomposition of BDF. We notice that the external loading F~\widetilde{F} is non-autonomous, meaning that it depends not only on XX but also on another (hidden) time variant quantity, which can be seen as the hidden system state, Sm+1S_{m+1}, to be discovered through learning.

We now formally state the learning settings in ASNO: consider multiple physical systems which can be described by a series of initial-valued problems:

X˙(η)=F(S(η),X(η)),X(η)(t0)=X0(η),(4)\dot{X}^{(\eta)} = F(S^{(\eta)}, X^{(\eta)}), \quad X^{(\eta)}(t_0) = X_0^{(\eta)}, \tag*{(4)}

where η=1,⋯ ,S\eta= 1, \cdots, S is the sample/system index. Then, for each system, we assume a sequence of observations:

D(η)={X(η)(mΔt),F(η)(mΔt)}m=0T/Δt(5)\mathcal{D}^{(\eta)} = \left\{X^{(\eta)}(m\Delta t), F^{(\eta)}(m\Delta t)\right\}_{m=0}^{T/\Delta t} \tag*{(5)}

are provided. The goal of ASNO are three folds:

  1. Identify a temporal extrapolation rule (2) from data, for a stable and accurate long-term prediction of XX.

  1. For each system with hidden state S(η)S^{(\eta)}, infer the underlying context or structure of S(η)S^{(\eta)} directly from observational data, without relying on labeled supervision or prior knowledge of the state.

  1. For each new and unseen system with the first nn steps of (X,F)(X,F) given, provide a zero-shot prediction rule without training.

Explicit Step: Temporal Extrapolation

As the first component of ASNO, we employ a Transformer Encoder to resemble the explicit step (2). In particular, it processes time series data by capturing long-range temporal dependencies, and predicts future states of systems governed by high-dimensional, nonlinear dynamical equations.

In Transformer Encoder, the input to the encoder is a sequence of observations from previous time steps, denoted as {Xm,Xm−1,Xm−2,…,Xm−n+1}\{X_m, X_{m-1}, X_{m-2}, \ldots, X_{m-n+1}\}, where nn represents the number of past time steps considered. Each observation Xt∈RdX_t \in\mathbb{R}^{d} is projected into an embedding space of dimension dembedd_{\mathrm{embed}}, with positional encodings added to preserve temporal order:

Et=XtWE+Pt,(6)E_t = X_t W_E + P_t, \tag*{(6)}

where WE∈Rd×dembedW_E \in\mathbb{R}^{d \times d_{\mathrm{embed}}} is the embedding matrix, and Pt∈RdembedP_t \in\mathbb{R}^{d_{\mathrm{embed}}} is the positional encoding vector. The encoded sequence {Em,Em−1,…,Em−n+1}\{E_m, E_{m-1}, \ldots, E_{m-n+1}\} is then passed through multiple layers of the transformer encoder, each consisting of a multi-head self-attention mechanism and a feed-forward network, allowing the model to learn complex dependencies across time steps.

In the attention mechanism [29], the query, key, and value matrices are derived from the encoded input sequence using weights WTq,WTk,WTv∈Rdembed×dtW_{Tq}, W_{Tk}, W_{Tv} \in\mathbb{R}^{d_{\mathrm{embed}} \times d_t}, which are learnable parameters for the query, key, and value transformations, respectively.

Q=EtWTq,K=EtWTk,V=EtWTv,(7)Q = E_t W_{Tq}, \qquad K = E_t W_{Tk}, \qquad V = E_t W_{Tv}, \tag*{(7)}

The attention score between the query at the tt-th time step, QtQ_t, and the keys KK from previous steps is calculated as a weighted sum, capturing dependencies between the current time step and previous steps. Formally, for the mm-th time step, the future latent space Hm+1H_{m+1} is predicted using the attention-weighted sum of past values:

Hm+1=TE⁡(Xm,Xm−1,…,Xm−n+1)=∑i=1nαm,i⋅Vm,(8)H_{m+1} = \operatorname{TE}(X_m, X_{m-1}, \ldots, X_{m-n+1}) = \sum_{i=1}^{n} \alpha_{m,i} \cdot V_m, \tag*{(8)}

where αm,i\alpha_{m,i} represents the attention weights between QmQ_m and Km−i+1K_{m-i+1}, computed as

αm,i=exp⁡(Qm⋅Km−i+1Tdt)∑j=1nexp⁡(Qm⋅Km−j+1Tdt),(9)\alpha_{m,i} = \frac{\exp\left(\frac{Q_m \cdot K_{m-i+1}^{\mathsf{T}}}{\sqrt{d_t}}\right)}{\sum_{j=1}^{n} \exp\left(\frac{Q_m \cdot K_{m-j+1}^{\mathsf{T}}}{\sqrt{d_t}}\right)}, \tag*{(9)}

where dtd_t is the dimensionality of the key vectors. This enables the transformer to focus on the most relevant past steps when extrapolating the latent state.

The predicted latent variable Hm+1∈RdembedH_{m+1} \in\mathbb{R}^{d_{\mathrm{embed}}} corresponds to the homogeneous BDF extrapolation X~m+1\widetilde{X}_{m+1} and is passed into the NAO for the implicit spatial correction step (3).

Implicit Step: Static PDE Solver

For the implicit step (3), we take H~m+1\widetilde{H}_{m+1} and Fm+1F_{m+1} as inputs, and aim to approximate the solution Xm+1X_{m+1}. Since (3) can be seen as a nonlinear static PDE solver, we propose to learn a neural operator as the surrogate solution operator. However, due to the possible change of underlying system state, SS, generic neural operators (Lu et al., 2019; Li et al., 2020; You et al., [38]) would fail this task because they focus on the solution of one single PDE. Herein, we propose to employ the attention mechanism, in the form of the Nonlocal Attention Operator (NAO) (Yu et al., 2024), which combines attention and neural operator learning, offering a generalizable solution operator across different PDEs. In particular, NAO expands on traditional attention by effectively modeling long-range spatial dependencies crucial for capturing continuous, nonlocal interactions within physical systems (Yu et al., 2024). This approach allows NAO to aggregate global context, addressing limitations of standard attention in handling nonlocal, spatially distributed information, especially beneficial for high-dimensional data processing in fluid dynamics and thermodynamics.

Formally, NAO is designed to solve both forward and inverse problems, with the underlying system dynamics encapsulated by operators that map input functions h∈Hh \in\mathcal{H} to output functions g∈Fg \in\mathcal{F}. This operator is defined as:

LK[h]+ϵ=g,(10)L_K[h] + \epsilon= g, \tag*{(10)}

where H\mathcal{H} and F\mathcal{F} are Banach spaces, KK is the nonlocal interaction kernel, and ϵ\epsilon denotes additive noise accounting for model–system discrepancies. NAO introduces a critical component: a kernel map constructed via attention. This kernel map estimates the nonlocal kernel based on both input hh and output gg, enabling the model to capture global context across physical states.

Herein, we take hh as the latent variable field together with the loading fields, and gg as the prediction of the next time step. The latent variable and loading fields are discretized over the spatial mesh {yk}k=1N′\{y_k\}_{k=1}^{N'} as:

H1:d=(Hj(yk))1≤j≤d, 1≤k≤N′,H_{1:d} = \left(H_j(y_k)\right)_{1\leq j\leq d,\ 1\leq k\leq N'},
F1:d=(Fj(yk))1≤j≤d, 1≤k≤N′.(11)F_{1:d} = \left(F_j(y_k)\right)_{1\leq j\leq d,\ 1\leq k\leq N'}. \tag*{(11)}

with dd being the feature size, which is usually as dd steps from the same system.

The overall process in NAO iteratively transforms an initial state

J0=(H1:d,F1:d)J_0 = (H_{1:d}, F_{1:d})

through TT steps, each applying attention with residual connections:

Jt=Attn⁡(Jt−1;θt)Jt−1+Jt−1:=(Jt,Ft),1≤t≤T,(12)J_t = \operatorname{Attn}(J_{t-1};\theta_t)J_{t-1} + J_{t-1} := (J_t,F_t), \quad1 \leq t \leq T, \tag*{(12)}

where

Attn⁡[J;θt]=σ(1dkJWQt(WKtJ)TJT).(13)\operatorname{Attn}[J;\theta_t] = \sigma\left(\frac{1}{\sqrt{d_k}}JW_{Q_t}(W_{K_t}J)^{\mathsf{T}}J^{\mathsf{T}}\right). \tag*{(13)}

After TT steps, we form the learned kernel via a kernel map:

K[H1:d,F1:d;θ]=WP,hσ(1dk(JT)TWQT+1(WKT+1JT)T)+WP,fσ(1dk(FT)TWQT+1(WKT+1JT)T).(14)\begin{aligned} K[H_{1:d},F_{1:d};\theta] ={}& W_{P,h}\sigma\left(\frac{1}{\sqrt{d_k}}(J_T)^{\mathsf{T}}W_{Q_{T+1}}(W_{K_{T+1}}J_T)^{\mathsf{T}}\right) \\ &+ W_{P,f}\sigma\left(\frac{1}{\sqrt{d_k}}(F_T)^{\mathsf{T}}W_{Q_{T+1}}(W_{K_{T+1}}J_T)^{\mathsf{T}}\right). \tag*{(14)} \end{aligned}

and compute the output

X1:dout(y)=∫K[H1:d,F1:d](y,z)F1:d(z) dz.(15)X_{1:d}^{\mathrm{out}}(y) = \int K[H_{1:d},F_{1:d}](y,z)F_{1:d}(z)\,\mathrm{d}z. \tag*{(15)}

In this architecture, θ={WP,h,WP,f,WQt,WKt}\theta= \{W_{P,h}, W_{P,f}, W_{Q_t}, W_{K_t}\} are trainable, WQt,WKtW_{Q_t}, W_{K_t} are query/key matrices, σ\sigma is a linear activation function following the suggestions in (Yu et al., 2024; Cao, [4]; Lu and Yu, [22]). To optimize θ\theta, we minimize the training loss across all SS systems:

L=∑n=1S∥∫K[H1:d,F1:d](y,z)F1:d(z) dz−X1:dout,true(y)∥L22.(16)\mathcal{L} = \sum_{n=1}^{S}\left\|\int K[H_{1:d},F_{1:d}](y,z)F_{1:d}(z)\,\mathrm{d}z - X_{1:d}^{\mathrm{out,true}}(y)\right\|_{L_2}^{2}. \tag*{(16)}

The NAO thus models nonlocal spatial interactions via a data-dependent kernel on the latent space H\mathcal{H}, capturing complex system states SS across diverse PDEs and enhancing generalizability.

ASNO: Summary and Algorithm

The ASNO architecture combines the Transformer Encoder and the NAO to effectively capture both temporal and spatial dependencies in complex spatio-temporal data, as shown in Figure 1.

Figure 1. The ASNO framework with the Transformer Encoder for capturing temporal dependencies and the Non-Local Attention Operator (NAO) for modeling spatial interactions, following a Backward Differentiation Formula (BDF) approach.

As the Transformer Encoder processes the past observations and outputs the latent variable Hm+1H_{m+1} using Equation (8), Hm+1H_{m+1} is fed into the NAO to provide the temporal information. This process can be represented as:

Xm+1out=NAO⁡(Hm+1,Fm+1)=NAO⁡(TE⁡(Xm,Xm−1,…,Xm−n+1),Fm+1)=ASNO⁡(Xm,Xm−1,…,Xm−n+1,Fm+1).(17)\begin{aligned} X^{\mathrm{out}}_{m+1} &= \operatorname{NAO}(H_{m+1}, F_{m+1}) \\ &= \operatorname{NAO}(\operatorname{TE}(X_m, X_{m-1}, \ldots, X_{m-n+1}), F_{m+1}) \\ &= \operatorname{ASNO}(X_m, X_{m-1}, \ldots, X_{m-n+1}, F_{m+1}). \tag*{(17)} \end{aligned}

Since the temporal information is compressed into a latent representation by the Transformer Encoder, and the NAO leverages it to learn spatial relationships, the combination of the two modules allows ASNO to learn spatio-temporal dependencies. The Transformer Encoder and NAO are jointly trained with a loss function that optimizes the model’s ability to capture complex dependencies efficiently, making it adaptable to a wide range of scientific machine-learning tasks.

Experiments

In this section, we evaluate ASNO’s performance on case studies involving chaotic systems and spatio-temporal PDEs, each chosen to illustrate its strengths in capturing complex dynamical behaviors crucial to scientific machine learning.

Learning a PDE Solution Operator for Darcy Flow

We evaluate ASNO on a dynamic Darcy flow equation, which models fluid flow through porous media and exhibits complex spatio-temporal dynamics. On a square domain Ω:=[0,1]2\Omega:= [0, 1]^2, The governing equation writes:

∂p(t,x)∂t−∇⋅(b(x)∇p(t,x))=g(t,x),x∈Ω,p(x)=0,x∈∂Ω.(18)\frac{\partial p(t,x)}{\partial t} - \nabla\cdot\left(b(x)\nabla p(t,x)\right) = g(t,x), \qquad x \in\Omega,\qquad p(x) = 0,\qquad x \in\partial\Omega. \tag*{(18)}

In this context, we aim to learn the solution operator of Darcy’s equation and compute the pressure field p(t,x)p(t,x), given the source field g(t,x)g(t,x). The training dataset consists of 100 time-series profiles on a 21×2121 \times21 grid, each with 100 timesteps. Here, each profile is determined by a different hidden microstructure field b(x)b(x). Input–output pairs are generated via a sliding window of five consecutive timesteps (i.e., stride 1), yielding 96 samples per profile. We apply 20 random permutations per trajectory, resulting in 153,600 training samples and 38,400 test samples (80:20 split). All baseline models (FNO, U-Net, Transolver, GNOT, DeepONet, Transformer Encoder, Transformer Encoder + NAO) use the same inputs: the past five states Xm−4,…,XmX_{m-4}, \ldots, X_m and the current forcing field FmF_m, where the state XmX_m represents the pressure field at time step mm and the forcing field FmF_m encodes the source or sink terms influencing fluid flow at each spatial location.

Table 1 compares the number of trainable parameters, GPU memory usage, test loss on the standard test set, and out-of-distribution (OOD) test losses for two variants of OOD data. Here, OOD refers to inputs whose underlying distributions differ from those seen during training. Assessing OOD performance is important because it measures a model’s generalizability under distributional shifts. In our case, the OOD-f set uses time-varying source fields drawn from a different Gaussian random field (GRF) distribution than the training set. Specifically, while training samples use source functions g(t,x)g(t,x) drawn from a GRF with fixed hyperparameters α=2\alpha= 2 and τ=3\tau= 3, the OOD-f samples are generated by modulating these GRFs with a sinusoidal time component and scaling their amplitudes by a factor of 200. In particular, the source term evolves over time as g(t,x)=g~(x)⋅sin⁡(t)g(t,x) = \widetilde{g}(x) \cdot\sin(t), where g~(x)\widetilde{g}(x) is the spatial GRF realization and t=qΔtt = q\Delta t is the temporal index with Δt=5×10−5\Delta t = 5 \times10^{-5}. This construction introduces a dynamic temporal structure not present in the training distribution, thereby altering both the spatial and temporal characteristics of the source field. Meanwhile, the OOD-b set modifies the spatial distribution of the permeability field b(x)b(x), which is a crucial hidden input in Darcy flow. While the training set uses binary bi-phase microstructures generated from a GRF with parameters αx=4\alpha_x = 4 and τx=5\tau_x = 5, the OOD-b set uses a rougher, higher-frequency microstructure generated from a GRF with αχ=7\alpha_{\chi}=7 and τχ=6\tau_{\chi}=6. This change leads to sharper, more fragmented phase boundaries and higher heterogeneity in the underlying permeability, directly impacting the operator’s ability to generalize to previously unseen flow geometries. Generalizability is quantified by the extent to which the test loss increases on these OOD sets compared to the standard test set: smaller increases imply better robustness. ASNO achieves the lowest best and OOD test losses with a comparative number of trainable parameters among the neural-operator baselines, indicating the strongest OOD generalization with respect to both types of distributional shifts.

ModelTrainable ParamsGPU (MB)Best Test LossBest OOD-fBest OOD-b
ASNO760,2341810.03680.06730.0982
FNO900,2242140.07680.11290.1892
U-Net820,9941230.11500.15230.2224
Transolver810,5734220.04280.07210.1535
GNOT760,3492080.05160.08110.1729
DeepONet6,230,00021460.05370.08260.1826
Trans. Enc.1,620,3941730.05590.09270.1736
Linear Enc. + NAO720,3981650.054740.12450.1394

Table 1. Performance of different models on Darcy flow and Darcy OOD test sets.

ASNO not only provides a high-performing predictor but also reveals underlying physical mechanisms through its learned representations. Its separable design helps to learn the implicit–explicit decomposition of backward differentiation formulas. Moreover, its modular structure enhances interpretability by allowing independent analysis of temporal and spatial contributions to the final prediction. To illustrate this, we compared the temporal predictions of the Transformer Encoder component with a classical BDF5 numerical approximation:

X~m=12137Xm−5−75137Xm−4+200137Xm−3−300137Xm−2+300137Xm−1,(19)\widetilde{X}_m = \frac{12}{137}X_{m-5} - \frac{75}{137}X_{m-4} + \frac{200}{137}X_{m-3} - \frac{300}{137}X_{m-2} + \frac{300}{137}X_{m-1}, \tag*{(19)}

showing that the learned latent representation HmH_m closely matches this high-order temporal extrapolation (see Figure 2). For the spatial interactions captured by the NAO, we compared the learned kernel with a theoretical kernel derived from the Darcy discretization. Starting from

Comparison of the latent prediction from the Transformer Encoder, the BDF approximation, and the ASNO-predicted output at Timestep 10

Figure 2. Comparison of the latent prediction from the Transformer Encoder (Hm+1H_{m+1}), the BDF approximation (X~m+1\widetilde{X}_{m+1}), and the ASNO-predicted output at Timestep 10. The latent prediction and BDF approximation closely align, and the final ASNO output differs as the NAO adjusts the prediction using field-based corrections.

X~m=Xm+60137Δt(AXm+Fm),(20)\widetilde{X}_m = X_m + \frac{60}{137}\Delta t\left(A X_m + F_m\right), \tag*{(20)}

one obtains

Xm=(I+60137ΔtA)−1(X~m−60137ΔtFm),(21)X_m = \left(I+\frac{60}{137}\Delta t A\right)^{-1}\left(\widetilde{X}_m-\frac{60}{137}\Delta t F_m\right), \tag*{(21)}
Ktrue=−60137Δt(I+60137ΔtA)−1.(22)K_{\mathrm{true}}=-\frac{60}{137}\Delta t\left(I+\frac{60}{137}\Delta t A\right)^{-1}. \tag*{(22)}

Figure 3 compares the ground-truth kernel and the learned kernel from the NAO component, showing good agreement between the two. Additionally, the cumulative loss over time for various models illustrates that NAO outperforms existing methods in accuracy. When computing long-term time integration, each prediction XtpredX_t^{\mathrm{pred}} is fed back as an input for the next step, so errors accumulate over successive rollouts. The cumulative loss—also referred to as long-term time integration—is computed as

Figure 3. Left: Cumulative error over time for various models on the Darcy flow benchmark, illustrating ASNO’s long-term stability. Right: Comparison between the ground-truth Darcy kernel and the spatial kernel learned by the Nonlocal Attention Operator (NAO), demonstrating alignment with the underlying physics.

ET=∑t=1τ∥Xttrue−Xtpred∥L2,(23)E_T = \sum_{t=1}^{\tau}\left\lVert X_t^{\mathrm{true}}-X_t^{\mathrm{pred}}\right\rVert_{L^2}, \tag*{(23)}

where XttrueX_t^{\mathrm{true}} and XtpredX_t^{\mathrm{pred}} denote the ground-truth and predicted states at timestep tt. This confirms that the spatial mechanism captured by NAO effectively reflects the physical structure of the underlying PDE and maintains strong predictive performance across timesteps.

These results show that ASNO balances predictive accuracy, interpretability, and computational cost, and maintains stable performance even under variations in forcing and permeability.

Chaotic ODE: Lorenz System

In this section, we evaluate the performance of ASNO on the Lorenz system, a well-known nonlinear dynamical model featuring high sensitivity to initial conditions:

∂x∂t=σ(y(t)−x(t))+g1(t),(24)\frac{\partial x}{\partial t} = \sigma(y(t)-x(t)) + g_1(t), \tag*{(24)}
∂y∂t=x(t)(ρ−z(t))−y(t)+g2(t),(25)\frac{\partial y}{\partial t} = x(t)(\rho-z(t)) - y(t) + g_2(t), \tag*{(25)}
∂z∂t=x(t)y(t)−βz(t)+g3(t).(26)\frac{\partial z}{\partial t} = x(t)y(t) - \beta z(t) + g_3(t). \tag*{(26)}

The dataset consists of 2000 time-series profiles with 100 different coefficients combinations (σ\sigma, ρ\rho, β\beta), 20 different loading functions (g1g_1, g2g_2, g3g_3), and the same initial conditions (x(0),y(0),z(0))=(0,1,0)(x(0), y(0), z(0)) = (0, 1, 0). Each profile contains 1000 timesteps. 80% profiles are used for training and 20% are for testing. To construct input–output pairs, we apply a sliding window of length 5 that moves at every timestep, yielding 996 windows per profile (from t=1t = 1 through t=996t = 996); we discard the final partial window to obtain 995 samples per profile. To improve model generalization and robustness, we augment the data by permuting the order of entire 5-step windows within each trajectory, rather than shuffling individual timesteps, which preserves local temporal structure while increasing data diversity, and by randomly permuting the profile indices to avoid overlap between train and test sets. This results in a total of 80×995×20=1,592,00080 \times995 \times20 = 1,592,000 training samples and 20×995×20=398,00020 \times995 \times20 = 398,000 testing samples without leakage. ASNO is compared against five baselines (Transolver, DeepONet, Transformer Encoder, GNOT, and Linear Model + NAO), all trained under identical input formats: each model receives the five previous states {Xm−4,…,Xm}\{X_{m-4}, \ldots, X_m\} as well as the current forcing FmF_m. We did not include architectures such as U-Net, FNO, and GNOT in our Lorenz comparison due to architectural mismatch: U-Net is designed for high-dimensional spatial grids and relies on convolutional down-/up-sampling; applying it to a low-dimensional ODE time series forces unnatural reshaping and often leads to over-smoothing of the rapid state changes characteristic of chaos. FNO similarly assumes a spatial Fourier basis—its global spectral filters excel for PDEs on regular meshes but provide little benefit (and much overhead) when modeling three-dimensional trajectories in (x,y,z)(x, y, z). GNOT builds in geometric gating layers that require constructing a graph or manifold structure even for these 3-variable ODEs, adding implementation complexity and extra hyperparameters with little payoff. The results in Table 2 show that ASNO achieves the lowest test loss (0.000794), demonstrating its predictive accuracy on this chaotic system.

Model TypeTrainable ParamsGPU (MB)Best Test Loss
ASNO258,808760.000794
Transolver395,907950.000835
DeepONet265,512790.001750
Transformer Encoder257,647710.001821
GNOT401,1551060.002189
Linear Model + NAO305,776870.005298

Table 2. Comparison of ASNO and baselines models on the Lorenz system.

By emulating an implicit–explicit BDF scheme, ASNO first employs a Transformer encoder to extrapolate the homogeneous (linear) dynamics from the past nn states, thereby capturing dominant temporal modes and the system’s exponential divergence rate without interference from nonlinear feedback. The Nonlocal Attention Operator then applies a compact learned kernel to introduce the corrective coupling terms (e.g. the products xyxy, xzxz), focusing exclusively on rapid cross-variable interactions. This two-stage decomposition lightens the representational load on each module: the Transformer Encoder solves a multistep linear recurrence, while the NAO specializes in a small nonlinear correction. As a result, ASNO achieves more stable long-term rollouts in the chaotic regime—small errors in one stage do not immediately amplify in the other—and empirical trajectories adhere much more closely to the true Lorenz attractor, especially in the sensitive yy and zz directions, than do those from monolithic models. For long-term rollouts, each new prediction is generated by feeding ASNO’s previous output back into the model as the next input, mirroring an autoregressive integration scheme. Figure 4 shows ASNO’s predictions remaining near the ground truth over many steps, whereas the other baselines exhibit deviation as of earlier timesteps. By cleanly separating temporal forecasting from coupling correction, ASNO is able to achieve improved long-term stability and robustness—properties that are particularly desirable for chaotic forecasting tasks in climate science, physics simulations, and real-time control.

Long-time prediction comparison of predicted versus true time-series trajectories for the Lorenz system

Figure 4. Long Time prediction comparison of Predicted vs. true time-series trajectories (x,y,z)(x, y, z) for models listed in Table 2 on the Lorenz system.

Navier–Stokes Benchmark for PDE Testing

To assess ASNO’s efficacy on complex PDEs, we evaluate it on the two-dimensional (2D) incompressible Navier–Stokes (NVS) equations in vorticity form. The governing equation is given by

∂ω∂t+u⋅∇ω=νΔω+f,withΔψ=−ω,u=(∂ψ∂y,−∂ψ∂x),\frac{\partial\omega}{\partial t} + \mathbf{u} \cdot\nabla\omega= \nu\Delta\omega+ f,\qquad\text{with}\quad\Delta\psi= -\omega,\quad\mathbf{u} = \left(\frac{\partial\psi}{\partial y},-\frac{\partial\psi}{\partial x}\right),

where ω\omega is the scalar vorticity in 2D, u=(u,v)\mathbf{u} = (u,v) is the velocity field, ψ\psi is the streamfunction, ν\nu is the viscosity, and ff is the external forcing function. This benchmark tests ASNO’s ability to capture nonlinear interactions and multi-field dependencies, following the setup in Li et al. (2020). The dataset consists of 100 simulated time-series profiles, each with a spatial resolution of 30×3030 \times30, governed by varying viscosity coefficients ν∈[10−4,1]\nu\in[10^{-4},1], and subject to a forcing term–driven velocity field. Each profile contains 100 temporal snapshots with a time step size of Δt=10−2\Delta t = 10^{-2}. To construct training and testing data, a sliding window of length 5 and stride 1 is used, resulting in 96 samples per profile. Additionally, we apply 20 random permutations per trajectory to increase diversity and simulate realistic variations in flow dynamics. This yields 80×96×20=153,60080 \times96 \times20 = 153{,}600 training samples and 20×96×20=38,40020 \times96 \times20 = 38{,}400 testing samples. These settings create a comprehensive dataset for evaluating ASNO’s generalization and stability in modeling high-dimensional, nonlinear fluid systems.

Table 3 compares ASNO against state-of-the-art models in test accuracy, efficiency, and parameter count. ASNO achieves the lower test loss while maintaining computational efficiency. Figure 5 illustrates cumulative error trends, demonstrating ASNO’s stability over long time horizons. These results confirm ASNO’s effectiveness in handling high-dimensional, nonlinear PDEs, generalizing across diverse physical systems with minimal numerical dissipation. Crucially, ASNO’s separable architecture—where the Transformer Encoder approximates the homogeneous (viscous and pressure) evolution through a high-order multistep forecast and the Nonlocal Attention Operator captures the advective and coupling interactions spatially—reduces the representational burden on each component. By isolating the stiff linear dynamics from the convective nonlinearities, ASNO attains both greater numerical stability and sharper resolution of fine-scale flow structures than monolithic models. Building on ASNO’s success in modeling high-dimensional, nonlinear PDEs like the Navier–Stokes equations, we extend its application to real-world engineering challenges.

Figure 5. Left: Cumulative error comparison across models listed in Table 3 on the Navier–Stokes benchmark. ASNO demonstrates minimal error growth over time, indicating long-term stability. Right: Visualization of predicted versus ground-truth velocity fields, showcasing ASNO’s ability to recover fine-scale flow structures.

ModelTrainable Params (M)GPU (MB)Best Test Loss
ASNO4.668800.0213
Transolver4.149110.0234
GNOT5.2510240.0322
DeepONet5.1031000.0921
Trans. Enc.5.199610.0967
FNO4.108460.1186
U-Net5.029910.1940
Linear Encoder + NAO4.057910.0328

Table 3. Performance of models on Navier–Stokes Equations: Model Complexity, GPU Memory, and Best Test Loss.

Melt Pool Temperature Field Prediction in Additive Manufacturing

In this section we present a case study on Directed Energy Deposition (DED) AM processes where the goal is to predict the evolution of the melt pool temperature field [10]. Accurate melt pool prediction is crucial in real-world manufacturing scenarios as small deviations in melt pool temperature can significantly affect part quality. First, to collect data to train ASNO we employ GAMMA, an in-house developed GPU-accelerated Finite Element Analysis (FEA) code, for part-scale simulations [20]. The GAMMA simulation follows the transient heat conduction equation, incorporating the essential partial differential equations (PDEs) that underpin the DED process:

ρCp(T)∂T∂t+∇⋅q=0,(27)\rho C_p(T) \frac{\partial T}{\partial t} + \nabla\cdot q = 0, \tag*{(27)}

Where TT is the temperature (K), ρ\rho and CpC_p are the density (g/mm3g/mm^3) and the effective specific heat capacity (J/g/KJ/g/K) of the material, respectively. The heat flux qq is given by Fourier’s law:

q=−k∇T,(28)q = -k\nabla T, \tag*{(28)}

where kk is the thermal conductivity of the material. The boundary conditions in the DED process can be formulated as:

q⋅n=−2ηPπrbeam2exp⁡(−2d2rbeam2)+σϵ(T4−T04)+h(T−T0),(29)q \cdot n = \frac{-2\eta P}{\pi r_{beam}^{2}} \exp\left(\frac{-2d^{2}}{r_{beam}^{2}}\right) + \sigma\epsilon(T^{4} - T_{0}^{4}) + h(T - T_{0}), \tag*{(29)}

where η\eta is the absorption coefficient (%), PP is the laser power (W), rbeamr_{beam} is the beam radius (mm), and dd is the distance (mm) from the material point to the center of the laser. hh is the convection heat transfer coefficient (W/m2/KW/m^2/K), σ\sigma is the Stefan-Boltzmann constant (5.67×10−8 W/m2/K45.67 \times10^{-8}\ W/m^2/K^4), ϵ\epsilon is the material’s emissivity, and T0T_0 is the ambient temperature. In the simulation, elements are activated and incorporated into the mesh when the distance between the element’s center and the beam center is less than the beam’s size.

To generate training data, we first generated 100 different laser power profiles with various combinations of laser power Plaser(m)P_{laser}(m) and scanning rate Vscan(m)V_{scan}(m) as process parameters, and then simulate the printing process of a thin wall using the GAMMA simulation. The temperature field Tpool(m)T_{pool}(m) corresponding to the laser location Llaser(m)L_{laser}(m) at each time step mm was saved as the training output. Each temperature field is recorded on a 21×2121 \times21 spatial grid, and each profile consists of 730 time steps. To construct the training data, a sliding window of length 5 and stride 1 was used, resulting in 730−5+1=726730 - 5 + 1 = 726 samples per profile. A random permutation of 20 per trajectory was applied to enhance data diversity while preserving temporal locality, yielding a total of 80×726×20=1,161,60080 \times726 \times20 = 1,161,600 training samples and 20×726×20=290,40020 \times726 \times20 = 290,400 testing samples (based on an 80:20 split). With training data collected, the ASNO for predicting the melt pool temperature field can be trained by formulating the prediction based on the processing parameters and the history from the past five steps:

T^pool(m+1)=ASNO⁡(Plaser(m+1),Vscan(m+1),Llaser(m+1),Tpool(m−4:m)).(30)\begin{aligned} \widehat{T}_{pool}(m+1) &= \operatorname{ASNO}\left(P_{laser}(m+1), V_{scan}(m+1), \right.\\ &\qquad\left. L_{laser}(m+1), T_{pool}(m-4:m)\right). \tag*{(30)} \end{aligned}

The predicted melt pool temperature field from a selected frame is shown in Figure 6, compared with the ground truth result from the GAMMA simulation. As a result, the ASNO achieves a Mean Absolute Percentage Error (MAPE) of 2.50%. Additional prediction results of different process input scenarios for additive manufacturing are provided in Appendix § 8.

Prediction results on the DED AM case study with the ASNO prediction on the left, and target results on the right.

Figure 6. Prediction results on the DED AM case study with the ASNO prediction on the left, and target results on the right.

Conclusion

In this paper, we introduce ASNO, a spatio-temporal framework designed to enhance generalizability and interpretability in modeling time-dependent differential equations. ASNO leverages a separable kernel approach inspired by the implicit-explicit BDF, enabling to disentangle temporal and spatial interactions to provide insights on system dynamics. The Transformer Encoder estimates homogeneous system solutions from historical states, analogous to the explicit step in BDF, while a neural operator models spatial interactions and external load impacts. To ensure robust generalization across diverse physical systems, ASNO integrates the NAO, which facilitates adaptation to varying PDE parameters such as initial conditions, loadings, and environments. Evaluations on benchmarks including the Lorenz system, Darcy flow equation, Navier Stokes, and the DED application demonstrate that ASNO outperforms over the existing models in terms of accuracy, long-term stability, and zero-shot generalizability to unseen physical parameters. In addition, ASNO offers interpretability by isolating the influence of temporal and spatial dynamics, revealing how historical states and spatial loadings contribute to governing behaviors, and aligning predictions with underlying physical laws. This dual emphasis on generalizability and interpretability makes ASNO suitable for real-time decision-making in high-stakes environments, while its attention mechanisms uncover meaningful patterns for physics-informed discovery and decision-making. Future research can extend ASNO’s applications to foundational modeling for transfer learning across different physical systems, further advancing its role in uncovering and predicting complex physical phenomena in scientific and industrial domains.

Acknowledgements

We appreciate the grant support from the NSF HAMMER-ERC (Engineering Research Center for Hybrid Autonomous Manufacturing, Moving from Evolution to Revolution) under Award Number EEC-2133630, and the NSF MADE-PUBLIC Future Manufacturing Research Grant Program under Award Number CMMI-2037026. Wei Chen and Doksoo Lee acknowledge support from the NSF Boosting Research Ideas for Transformative and Equitable Advances in Engineering (BRITE) Fellow Program (CMMI 2227641). Yi-Ping Chen appreciates the Taiwan-Northwestern Doctoral Scholarship funded by the Ministry of Education in Taiwan, and the fellowship support from the Predictive Science and Engineering Design (PSED) cluster at Northwestern University. Yue Yu was funded by the National Institute of Health under Award 1R01GM157589-01 and the AFOSR under Grant FA9550-22-1-0197. Portions of this research were conducted on Lehigh University’s Research Computing infrastructure, partially supported by NSF Award 2019035.

Appendix

Melt Pool Temperature Field Prediction in Additive Manufacturing

In this section, we provide a comprehensive analysis of the melt pool characteristics predicted by ASNO and other comparative models. The melt pool in Directed Energy Deposition (DED) is a key factor in determining material properties. Accurate modeling of the melt pool helps assess temperature distribution, phase transformation, and thermal cycles, which directly affect the microstructure and mechanical performance of the final product. Additionally, understanding melt pool dynamics is essential for process control, defect mitigation, and ensuring part consistency in DED applications. By comparing ASNO’s predictive performance with other models, we aim to demonstrate how well it captures these critical characteristics in a manufacturing setting.

The simulations were conducted over a 21×2121 \times21 spatial grid, with temperature distributions captured at multiple timesteps throughout the deposition process. In each timestep, the model outputs a 2D temperature field that reflects the temperature variations across the substrate due to the moving heat source.

Figure 7 illustrates these results, showing a side-by-side comparison between ASNO’s model predictions and the target values across three distinct timesteps (10, 25, and 36). The color gradient, representing temperature from 2000 K to 5500 K, demonstrates the intensity of heat within the melt pool and the cooling pattern radiating outward from the laser’s path. In each row, the predicted temperature distribution is shown on the left, directly alongside the corresponding target temperature distribution on the right, offering a clear visual comparison of ASNO’s accuracy at various time steps. The ASNO results show good agreement with the target profiles.

Figure 7. Comparison of ASNO model output and target temperature distributions for the melt pool in Directed Energy Deposition at Timesteps 10, 25, and 36. Each row represents a different timestep, with the model output shown on the left and the target on the right. The color scale indicates temperature in Kelvin, highlighting ASNO’s accuracy in capturing the thermal profile of the melt pool.

These visualizations underscore ASNO’s ability to model high-temperature zones and predict the progression of thermal profiles over time accurately. Notably, the model predictions retain stability over longer timesteps, capturing the complexity of the transient heat flow in DED processes. The agreement between the predicted and target distributions suggests that ASNO is well-suited for high-fidelity simulations in additive manufacturing applications, where precise thermal predictions are crucial for quality control and optimization.

Nomenclature

SymbolMeaning
XX, xxState vector (general)
XmX_mState at timestep mm
X~m+1\tilde{X}_{m+1}Extrapolated (homogeneous) state
FFExternal forcing / loading field
SSHidden system (environment) state
ttContinuous time
mmTime-index
nnHistory length
Δt\Delta tTime step size
αk\alpha_k, β\betaBDF coefficients (see (1))
D(η)\mathcal{D}^{(\eta)}Dataset for η
F\mathcal{F}ASNO operator mapping
y^t+1\hat{y}_{t+1}Predicted output at time t+1t + 1
yty_t, y^t\hat{y}_tTrue and predicted values at time tt
ETE_TCumulative error over TT steps
ete_tInstantaneous error at timestep tt
QQ, KK, VVQuery, Key, and Value matrices used in attention mechanisms; used generically, with context-specific definitions below
WEW_EEmbedding matrix
WTqW_{Tq}, WTkW_{Tk}, WTvW_{Tv}Time-series Transformer Encoder attention weight matrices for Query (QQ), Key (KK), and Value (VV) used in the explicit extrapolation step
PkP_kPositional encoding at position kk
dembedd_{\mathrm{embed}}, dtd_t, ddEmbedding dimension, key/query dimension, and feature dimension
HtH_tPenultimate-step latent features used in computing the NAO kernel weights for hh and ff
JtJ_tIntermediate NAO state at attention step tt composed of latent features and forcing fields (HtH_t, FtF_t), iteratively updated across TT layers
WP,hW_{P,h}, WP,fW_{P,f}, WQtW_{Q_t}, WKtW_{K_t}Weight matrices in the Nonlocal Attention Operator (NAO) used for computing kernel projections; WQtW_{Q_t} and WKtW_{K_t} generate attention Query and Key vectors used in the implicit interaction modeling
κ[⋅]\kappa[\cdot]Learned nonlocal kernel operator acting over latent space in NAO
H\mathcal{H}, F\mathcal{F}Input and output Branch function spaces
TTNumber of attention steps in NAO

Table 4.

References

  1. [1]Ascher, U. M., Ruuth, S. J., and Wetton, B. T. (1995). Implicit-explicit methods for time-dependent partial differential equations. SIAM Journal on Numerical Analysis, 32(3):797–823.DOI
  2. [2]Azizzadenesheli, K., Kovachki, N., Li, Z., Liu-Schiaffini, M., Kossaifi, J., and Anandkumar, A. (2024). Neural operators for accelerating scientific simulations and design. Nature Reviews Physics, pages 1–9.arxiv.org/abs/2309.15325
  3. [3]Cao, Q., Goswami, S., and Kaniadakis, G. E. (2024). Laplace neural operator for solving differential equations. Nature Machine Intelligence, 6(6):631–640.DOI
  4. [4]Cao, S. (2021). Choose a transformer: Fourier or galerkin. Advances in neural information processing systems, 34:24924–24940.arxiv.org/abs/2105.14995
  5. [5]Chen, Y.-P., Karkaria, V., Tsai, Y.-K., Rolark, F., Quispe, D., Gao, R. X., Cao, J., and Chen, W. (2025). Real-time decision-making for digital twin in additive manufacturing with model predictive control using time-series deep neural networks. arXiv preprint arXiv:2501.07601.DOI
  6. [6]Cheng, C.-W., Huang, J., Zhang, Y., Yang, G., Schönlieb, C.-B., and Aviles-Rivero, A. I. (2024). Mamba neural operator: Who wins? transformers vs. state-space models for pdes. arXiv preprint arXiv:2410.02113.
  7. [7]Fournier, Q., Caron, G. M., and Aloise, D. (2023). A practical survey on faster and lighter transformers. ACM Computing Surveys, 55(14s):1–40.DOI
  8. [8]Fredebeul, C. (1998). A-bdf: a generalization of the backward differentiation formulae. SIAM journal on numerical analysis, 35(5):1917–1938.DOI
  9. [9]Graves, A. and Graves, A. (2012). Long short-term memory. Supervised sequence labelling with recurrent neural networks, pages 37–45.DOI
  10. [10]Gunasegaram, D., Barnard, A., Matthews, M., Jared, B., Andreaco, A., Bartsch, K., and Murphy, A. (2024). Machine learning-assisted in-situ adaptive strategies for the control of defects and anomalies in metal additive manufacturing. Additive Manufacturing, page 104013.DOI
  11. [11]Hao, Z., Wang, Z., Su, H., Ying, C., Dong, Y., Liu, S., Cheng, Z., Song, J., and Zhu, J. (2023). Gnot: A general neural operator transformer for operator learning. In International Conference on Machine Learning, pages 12556–12569. PMLR.arxiv.org/abs/2302.14376
  12. [12]Kapteyn, M. G., Pretorius, J. V., and Willcox, K. E. (2021). A probabilistic graphical model foundation for enabling predictive digital twins at scale. Nature Computational Science, 1(5):337–347.DOI
  13. [13]Karkaria, V., Goeckner, A., Zha, R., Chen, J., Zhang, J., Zhu, Q., Cao, J., Gao, R. X., and Chen, W. (2024a). Towards a digital twin framework in additive manufacturing: Machine learning and bayesian optimization for time series process optimization. Journal of Manufacturing Systems.arxiv.org/abs/2402.17718
  14. [14]Karkaria, V., Tsai, Y.-K., Chen, Y.-P., and Chen, W. (2024b). An optimization-centric review on integrating artificial intelligence and digital twin technologies in manufacturing. Engineering Optimization, pages 1–47.DOI
  15. [15]Kim, D. and Lee, J. (2024). A review of physics informed neural networks for multiscale analysis and inverse problems. Multiscale Science and Engineering, pages 1–11.DOI
  16. [16]Kitaev, N., Kaiser, Ł., and Levskaya, A. (2020). Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451.arxiv.org/abs/2001.04451
  17. [18]Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhattacharya, K., Stuart, A., and Anandkumar, A. (2023). Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research, 24(89):1–97.
  18. [20]Liao, S., Golgoon, A., Mozaffar, M., and Cao, J. (2023). Efficient gpu-accelerated thermomechanical solver for residual stress prediction in additive manufacturing. Computational Mechanics, 71(5):879–893.DOI
  19. [21]Lim, B., Arik, S. Ö., Loeff, N., and Pfister, T. (2021). Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4):1748–1764.arxiv.org/abs/1912.09363
  20. [22]Lu, F. and Yu, Y. (2025). Transformer learns the cross-task prior and regularization for in-context learning. arXiv preprint arXiv:2505.12138.arxiv.org/abs/2505.12138
  21. [26]Molinaro, R., Yang, Y., Engquist, B., and Mishra, S. (2023). Neural inverse operators for solving pde inverse problems. arXiv preprint arXiv:2301.11167.arxiv.org/abs/2301.11167
  22. [27]Molnar, C. (2020). Interpretable machine learning. Lulu. com.
  23. [28]Nelson, B. K. (1998). Time series analysis using autoregressive integrated moving average (arima) models. Academic emergency medicine, 5(7):739–744.pubmed.ncbi.nlm.nih.gov/9678399
  24. [29]Niu, Z., Zhong, G., and Yu, H. (2021). A review on the attention mechanism of deep learning. Neurocomputing, 452:48–62.DOI
  25. [30]Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., and Zhong, C. (2022). Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistic Surveys, 16:1–85.arxiv.org/abs/2103.11251
  26. [31]Tang, B. and Matteson, D. S. (2021). Probabilistic transformer for time series analysis. Advances in Neural Information Processing Systems, 34:23592–23608.
  27. [32]Thelen, A., Zhang, X., Fink, O., Lu, Y., Ghosh, S., Youn, B. D., Todd, M. D., Mahadevan, S., Hu, C., and Hu, Z. (2022). A comprehensive review of digital twin—part 1: modeling and twinning enabling technologies. Structural and Multidisciplinary Optimization, 65(12):354.arxiv.org/abs/2208.14197
  28. [33]van Beek, A., Nevile Karkaria, V., and Chen, W. (2023). Digital twins for the designs of systems: a perspective. Structural and Multidisciplinary Optimization, 66(3):49.DOI
  29. [34]Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.arxiv.org/abs/1706.03762
  30. [35]Wang, Q., Ren, P., Zhou, H., Liu, X.-Y., Deng, Z., Zhang, Y., Chengze, R., Liu, H., Wang, Z., Wang, J.-X., et al. (2024). P²c²net: Pde-preserved coarse correction network for efficient prediction of spatiotemporal dynamics. arXiv preprint arXiv:2411.00040.arxiv.org/abs/2411.00040
  31. [36]Wen, G., Li, Z., Azizadehnesheli, K., Anandkumar, A., and Benson, S. M. (2022). U-fno—an enhanced fourier neural operator-based deep-learning model for multiphase flow. Advances in Water Resources, 163:104180.arxiv.org/abs/2109.03697
  32. [38]You, H., Zhang, Q., Ross, C. J., Lee, C.-H., and Yu, Y. (2022). Learning deep implicit fourier neural operators (IFNOs) with applications to heterogeneous material modeling. Computer Methods in Applied Mechanics and Engineering, 398:115296.DOI
  33. [40]Zerveas, G., Jayaraman, S., Patel, D., Bhamidipaty, A., and Eickhoff, C. (2021). A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 2114–2124.arxiv.org/abs/2010.02803
  34. [41]Zhang, Y., Zhang, Y., Zhang, Z., Bao, J., and Song, Y. (2018). Human activity recognition based on time series analysis using u-net. arXiv preprint arXiv:1809.08113.arxiv.org/abs/1809.08113
  35. [42]Zhao, T., Fang, L., Ma, X., Li, X., and Zhang, C. (2024). Tffomer: A time-frequency domain bidirectional sequence-level attention based transformer for interpretable long-term sequence forecasting. Pattern Recognition, page 110994.DOI
  36. [43]Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. (2021). Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11106–11115.arxiv.org/abs/2012.07436

Paper details

vlnc:W_6n8qw66v
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents