betaFeedback

WorksComputer Science

Higher-order-ReLU-KANs (HRKANs) for solving physics-informed neural networks (PINNs) more accurately, robustly and faster

Abstract

Finding solutions to partial differential equations (PDEs) is an important and essential component in many scientific and engineering discoveries. One of the common approaches empowered by deep learning is Physics-informed Neural Networks (PINNs). Recently, a new type of fundamental neural network model, Kolmogorov-Arnold Networks (KANs), has been proposed as a substitute of Multilayer Perceptions (MLPs), and possesses trainable activation functions. To enhance KANs in fitting accuracy, a modification of KANs, so called ReLU-KANs, using "square of ReLU" as the basis of its activation functions, has been suggested. In this work, we propose another basis of activation functions, namely, Higherorder-ReLU (HR), which is simpler than the basis of activation functions used in KANs, namely, Bsplines; allows efficient KAN matrix operations; and possesses smooth and non-zero higher-order derivatives, essential to physicsinformed neural networks. We name such KANs with Higher-order-ReLU (HR) as their activations, HRKANs. Our detailed experiments on two famous and representative PDEs, namely, the linear Poisson equation and nonlinear Burgers' equation with viscosity, reveal that our proposed Higher-order-ReLU-KANs (HRKANs) achieve the highest fitting accuracy and training robustness and lowest training time significantly among KANs, ReLU-KANs and HRKANs. The codes to replicate our experiments are available at https://github.com/kelvinhkcs/HRKAN.

Introduction

Partial differential equations (PDEs) play a prominent role in many real-world scenarios for modelling the dynamics in the systems of interest. Solving PDEs is essential for us to learn the dynamics of many scientific and engineering phenomena. However, it is often difficult, if not impossible, to find an exact solution of PDEs, especially when the PDE systems or the boundary or initial conditions are too complex [1, 2]. A variety of numerical methods thus become desirable given their ease of use and wide applicability in finding approximate solutions for PDEs. Common examples include finite element methods [3], mesh-free methods [4], finite difference methods [5], finite volume methods [6], and boundary element methods [7].

"AI for Science" is a recently emerging field of AI, which comprises of inventing and using deep learning algorithms to solve PDEs. Among these algorithms, there are four major approaches:

  1. Physics-informed Neural Networks (PINNs) [8]: In this approach, neural networks are designed to approximate the solution of PDEs. The loss functions of neural networks are defined based on the PDE equations, with the aim to train the neural networks to become the solutions of the PDEs. PINNs have been widely adopted in solving PDEs [9], identifying PDEs from data [10], and obtaining optimal controls of PDEs [11, 12].

  1. Operator learning: In this approach, neural networks are designed to learn the operator mappings from PDE space to the solution space from a massive data-set [13], key representatives of which include DeepONet [14, 15] and Fourier Neural Operator (FNO) [16]. A key difference between PINNs and operator learning is that when the initial or boundary conditions change, PINNs need to be re-trained with a new loss function or additional data points, whereas the operators do not require any re-training. Instead, the operators can immediately provide solutions even when those conditions change [17].

  1. Physics-informed Neural Operators [18, 19, 20]: This approach combines PINNs with operator learning such that the operators learn the mappings from the PDE space to the solution space based on a physics-informed loss function, without the necessary need of big data-set.

  1. Neural Spatial Representations and PDE-Nets: These models are proposed to solve time-dependent PDEs. Neural Spatial Representations [21] use neural networks to represent spatially-dependent functions to map the solutions at a time instance to the next time instance. PDE-Nets [22, 23] design tailor-made convolution filters to act as differential operators and apply forward Euler method to map the solutions at a time instance to the next time instance.

Underlying all of these four approaches, the backbone of the neural network models is Multilayer perceptrons (MLPs) [24, 25], also known as fully-connected feedforward neural networks. The famous universal approximation power possessed by MLPs [26] makes it the most essential building block in deep learning applications, ranging from self-driving cars [27] to textual and video generations [28].

Recently, a new architecture of neural network building block, Kolmogorov-Arnold Networks (KANs) [29], has been proposed as a substitute of MLPs. Different from MLPs, KANs are not backed by the universal approximation theorem. Instead, KANs are supported by the Kolmogorov-Arnold representation theorem [30], which states that any multivariate function can be expressed by a finite combination of univariate functions [31]. Another key difference between MLPs and KANs lie on their parameter spaces. The parameter space of MLPs solely consists of weight matrices, whereas the parameter space of KANs also includes the activation functions. In other words, KANs need to learn not only the optimal weight matrices, but also the optimal activation functions. In the original invention of KANs [29], B-splines are used as the basis of activation functions to be learnt. Afterwards, suggestions of different bases of activation functions have been seen, including Chebyshev orthogonal polynomials [32], radial basis functions [33], wavelet transforms [34], Jacobi basis functions [35, 36], Fourier transform [37] and "square of ReLU" basis functions [38].

Some efforts have also been witnessed in trying to apply KANs to find solutions of PDEs, for example, by incorporating KANs in DeepONets [39] and putting KANs in PINNs [40, 41, 42, 43]. However, due to the complexity of KANs’ B-spline basis functions, the training speed of KANs is not comparable to MLPs. One possible remedy is the use of "square of ReLU" basis functions in ReLU-KANs, which optimize KAN operations for efficient GPU parallel computing. A significant drawback, nevertheless, is the discontinuity of higher-order derivatives of such "square of ReLU" basis functions.

In this paper, we introduce a more suitable basis function, Higher-order-ReLU (HR), which not only optimizes KAN operations for efficient GPU parallel computing but at the same time is well-suited for PINNs. We call such KANs using Higher-order-ReLU basis functions (HR) as HRKANs, and such PINNs using HRKANs as PI-HRKANs. We evaluated PI-HRKANs’ performance on two representative and famous PDEs, namely, the linear Poisson equation (the example used in the github codebase of the original KAN paper), and the nonlinear Burgers’ equation with viscosity. Compared against KANs and ReLU-KANs, HRKANs exhibit significant improvements in fitting accuracy, training robustness and convergence speed. The codes to replicate our experiments are available at https://github.com/kelvinhkcs/HRKAN.

This paper presents the following key contributions:

  • A simple activation basis function: We introduce a simple activation basis function, Higher-order-ReLU (HR), which inherits the fitting capability of the original KAN B-spline basis functions and the "square of ReLU" basis functions.

  • Matrix-based operations: Similar to the "square of ReLU" basis functions in ReLU-KAN, Higher-order-ReLU (HR) basis functions facilitate efficient matrix computations with GPUs, speeding up the training process.

  • Well-suited for Physics-informed problems: While enabling efficient matrix operations and being simpler than the B-spline basis functions, Higher-order-ReLU (HR) basis functions also possess smooth and non-zero higher-order derivatives (with order up to users’ choices), thus being suitable for Physics-informed problems. Such smooth and non-zero higher-order derivatives are missing in "square of ReLU" basis functions as we shall show in later sections.

  • Higher fitting accuracy, stronger robustness and faster convergence: HRKANs demonstrate highest fitting accuracy, strongest robustness and fastest convergence compared to KANs and ReLU-KANs in our detailed experiments.

In section 2, we introduce the B-spline activation basis functions in KANs and the "square of ReLU" activation basis functions in ReLU-KANs and their respective properties. In section 3, we introduce our proposed Higher-order-ReLU (HR) activation basis functions, and discuss how it is better than the B-splines and "square of ReLU". In section 4, we conduct detailed and comprehensive experiments to evaluate HRKANs against KANs and ReLU-KANs, in terms of convergence speed, robustness and fitting accuracy. Section 5 is our conclusion.

Basis of activation function in KANs and ReLU-KANs

In this section, we review the bases of activation functions used in KANs and ReLU-KANs, along with some comparisons.

KANs: B-spline basis function

In KANs, B-splines are used as the basis of activation functions. We use Bg,k={b0(x),b2(x),…,bn−1(x)}B_{g,k} = \{b_0(x), b_2(x), \ldots, b_{n-1}(x)\} to denote a set consisting of n=g+kn = g + k B-spline basis functions with gg grid points and order kk. Each bib_i is defined recursively from linear combinations of B-spline basis functions of lower orders, i.e. order less than kk [44]. B-Spline basis functions are of compact support and their shapes depend on kk and gg.

In figure 1, the left hand side shows the B-spline basis B5,3={b0(x),b1(x),…,b7(x)}B_{5,3} = \{b_0(x), b_1(x), \ldots, b_7(x)\} on the interval [−1,1][-1, 1] with g=5g = 5 and k=3k = 3, and the right hand side shows the b4b_4 in the B-spline basis on the interval [−1,1][-1, 1] with g=5g = 5 and k=3k = 3.

ReLU-KANs: "square of ReLU" basis function

In ReLU-KANs, the bi(x)b_i(x) with support on [si,ei][s_i,e_i] is replaced by the "square of ReLU" basis ri(x)r_i(x) defined by

ri(x)=[ReLU⁡(ei−x)×ReLU⁡(x−si)]2×c,(1)r_i(x) = [\operatorname{ReLU}(e_i-x) \times\operatorname{ReLU}(x-s_i)]^2 \times c, \tag*{(1)}

where c=16(ei−si)4c = \frac{16}{(e_i-s_i)^4} is a normalization constant for ensuring the height of each basis being 1. The set containing the n=g+kn=g+k "square of ReLU" basis functions is denoted R5,3R_{5,3}.

Similar to the B-spline basis functions, the ri(x)r_i(x) is of compact support. Their shapes depend on the choice of eie_i and sis_i. Figure 2 shows the "square of ReLU" basis functions and r4(x)r_4(x) on the interval [−1,1][-1,1] defined with the same support as the B-splines basis function example in Figure 1.

B-Spline basis functions and basis function $b_4(x)$

Figure 1. B-Spline basis B5,3B_{5,3} (left) and basis function b4(x)b_4(x) (right)

“Square of ReLU” basis functions and basis function $r_4(x)$

Figure 2. "Square of ReLU" basis R5,3R_{5,3} (left) and basis function r4(x)r_4(x) (right)

Our method: Higher-order-ReLU

In this section, we introduce our proposed basis function, namely, Higher-order-ReLU (HR), an improvement from the "square of ReLU". Higher-order-ReLU looks similar to "square of ReLU", but is better than "square of ReLU" in the following ways:

  1. Higher-order-ReLU offers smooth and non-zero higher-order derivatives, essential to Physics-informed problems, whereas "square of ReLU" has discontinuous derivatives.

  2. Higher-order-ReLU still inherits all advantages of “square of ReLU" on its simplicity and efficient matrix-based KAN operations over the B-splines.

The Higher-order-ReLU basis viv_i of order mm is defined as

vm,i(x)=ReLU⁡(ei−x)m×ReLU⁡(x−si)m×cm(2)v_{m,i}(x) = \operatorname{ReLU}(e_i-x)^m \times\operatorname{ReLU}(x-s_i)^m \times c_m \tag*{(2)}

where cm=(2ei−si)2mc_m = \left(\frac{2}{e_i-s_i}\right)^{2m} is the normalization constant for ensuring the height of each basis being 1.

A key difference between the ri(x)r_i(x) in "square of ReLU" and the vm,i(x)v_{m,i}(x) in Higher-order-ReLU lies on the order mm of vm,i(x)v_{m,i}(x), which determines the smoothness of higher-order derivatives of Higher-order-ReLU. Figure 3 compares the Higher-order-ReLU vm,i(x)v_{m,i}(x) with m=4m=4 and "square of ReLU" ri(x)r_i(x) and their first and second-order derivatives. It is obvious that the second-order derivative of "square of ReLU" jumps at the end points of its support, which hinders the learning of PDE solution. If a PDE equation possesses derivative terms of order higher than 4, for example, uxxxxxu_{xxxxx}, we can pick mm to be larger, like 6, in an adaptive manner, whereas its "square of ReLU" basis R5,3R_{5,3} counterpart will fail as its 5-th order derivatives are completely zero.

Plots of the square of ReLU and Higher-order-ReLU and their derivatives

Figure 3. The "square of ReLU" r4(x)r_4(x) and its first and second-order derivatives r4′(x)r'_4(x) and r4′′(x)r''_4(x) (left) and the Higher-order-ReLU v4,4(x)v_{4,4}(x) and its first and second-order derivatives v4,4′(x)v'_{4,4}(x) and v4,4′′(x)v''_{4,4}(x) (right).

Comparing with the B-splines in KANs, we can observe that the higher-order derivatives of the B-spline basis are smooth, but they vanish up to the order kk specified. So, again, if the PDE equation possesses derivative terms of order higher than 4, for example, uxxxxxu_{xxxxx}, then the kk must be set to be higher than 6 to work out successfully. In this case, as B-splines are essentially polynomial, having an order which is too high may lead to the Runge’s phenomenon [45], an obstacle in function approximation which is not negligible.

Experiments

In this section, we evaluate the performance of KANs, ReLU-KANs and HRKANs (with m=4m = 4) with two physics-informed problems. The first one is a linear PDE, the Poisson equation. The second one is a non-linear PDE, the Burgers’ equation with viscosity.

Poisson Equation

Problem Description. The 2D Poisson equation is

∇2u(x,y)=−2π2sin⁡(πx)sin⁡(πy)(3)\nabla^2 u(x,y) = -2\pi^2 \sin(\pi x)\sin(\pi y) \tag*{(3)}

for (x,y)∈[−1,1]×[−1,1](x,y) \in[-1,1] \times[-1,1], with boundary conditions

{u(−1,y)=0u(1,y)=0u(x,−1)=0u(x,1)=0\begin{cases} u(-1,y) = 0 \\ u(1,y) = 0 \\ u(x,-1) = 0 \\ u(x,1) = 0 \end{cases}

The ground-truth solution is

u(x,y)=sin⁡(πx)sin⁡(πy).u(x,y) = \sin(\pi x)\sin(\pi y).

In the train-set, the number of interior points are Npde=960N_{\mathrm{pde}} = 960, and number of boundary points on the 4 boundaries the same, namely, Nbc1=30N_{\mathrm{bc1}} = 30, Nbc2=30N_{\mathrm{bc2}} = 30, Nbc3=30N_{\mathrm{bc3}} = 30 and Nbc4=30N_{\mathrm{bc4}} = 30 respectively. The train-set data points on the interior are generated randomly with uniform distribution while those on the boundaries are generated uniformly. We generate test-set data with a grid of grid size 100×100100 \times100.

We trained a KAN, a ReLU-KAN and a HRKAN using the loss function

L=αLpde+Lbc(4)L = \alpha L_{\mathrm{pde}} + L_{\mathrm{bc}} \tag*{(4)}

where

Lpde=1Npde∑i=1Npde(∇2u(xi,yi)+2π2sin⁡(πxi)sin⁡(πyi))2L_{\mathrm{pde}} = \frac{1}{N_{\mathrm{pde}}} \sum_{i=1}^{N_{\mathrm{pde}}} \left(\nabla^2 u(x_i,y_i) + 2\pi^2 \sin(\pi x_i)\sin(\pi y_i)\right)^2
Lbc=∑i=1Nbc1(u(−1,yi))2+∑i=1Nbc2(u(1,yi))2+∑i=1Nbc3(u(xi,−1))2+∑i=1Nbc4(u(xi,1))2Nbc1+Nbc2+Nbc3+Nbc4L_{\mathrm{bc}} = \frac{\sum_{i=1}^{N_{\mathrm{bc1}}} \left(u(-1,y_i)\right)^2 + \sum_{i=1}^{N_{\mathrm{bc2}}} \left(u(1,y_i)\right)^2 + \sum_{i=1}^{N_{\mathrm{bc3}}} \left(u(x_i,-1)\right)^2 + \sum_{i=1}^{N_{\mathrm{bc4}}} \left(u(x_i,1)\right)^2}{N_{\mathrm{bc1}} + N_{\mathrm{bc2}} + N_{\mathrm{bc3}} + N_{\mathrm{bc4}}}

We set hyperparameter α=0.05\alpha= 0.05. The structure of the KAN, ReLU-KAN and HRKAN are all of [2,2,1][2, 2, 1] with same support for all the bases with g=5g = 5 and k=3k = 3. For all three of them, we use the Adam optimizer, set the number of epoch to 3000 and train on the same GPU. We run the same experiment for 10 times.

Two remarks have to be made here.

  1. In the default implementation of KAN, there is a call of the function update_grid_from_samples() every five epoch till epoch 50. We keep this implementation unchanged for KANs in both experiments of Poisson equation and Burgers’ equation with viscosity, so KAN can update its grid whereas ReLU-KANs and HRKANs cannot in our experiments.

  1. In the default implementation of KAN, there is a also a step of sparsification for pruncing. Again, we have not included this implementation in ReLU-KANs and HRKANs but keep the implementation in KANs unchanged.

Both of these features may lead to extra overhead in computation, but we aim to keep the code of KAN unchanged to be fair. We will discuss more about our consideration in the section of result discussion.

Results Discussion. Figure 4 shows the ground-truth solution and the solutions learnt by the KAN, ReLU-KAN and HRKAN and their residual difference compared against the ground-truth solution in one of the 10 runs with the Poisson equation.

The ground-truth solution and the solutions learnt by the KAN, ReLU-KAN and HRKAN (Top row) and their residual difference compared against the ground-truth solution (Bottom row)

Figure 4. The ground-truth solution and the solutions learnt by the KAN, ReLU-KAN and HRKAN (Top row) and their residual difference compared against the ground-truth solution (Bottom row) in one of the 10 runs with the Poisson equation.

Figure 5 reveals the median and max-min-band of training loss, test loss and test MSE (Mean square error between the test-set ground-truth solution and the learnt solutions) of 10 runs for the KAN,

Loss and accuracy (median and max-min-band of 10 runs)

Figure 5. The median and max-min-band of training loss, test loss and test MSE (MSE between the test-set ground-truth solution and the learnt solutions) of 10 runs for the KAN, ReLU-KAN and HRKAN with the Poisson equation. First column: all 3000 epoches; Second column: last 2000 epoches; Last column: last 1000 epoches.

ReLU-KANHRKANKAN
Mean of Test MSE in 10 runs (in %)2.180.04346.80
Std. of Test MSE in 10 runs (in %)4.280.1299.43
Mean of training time in 10 runs (in second)21.227.9109

caption: The means and standard deviations of test MSE and means of training time for the KAN, ReLU-KAN and HRKAN in the 10 runs with the Poisson equation.

:::

ReLU-KAN and HRKAN with the Poisson equation. For ease of reading, the first column shows the curves for the entire training process, while the second and last columns show those in the last 2000 and last 1000 epochs respectively.

tbl:1 exhibits the means and standard deviations of test MSE, and means of training time, for the KAN, ReLU-KAN and HRKAN in the 10 runs with the Poisson equation.

There are some key observations to make:

  • From Figure 4, the last row of Figure 5 and the first row of table 1, it can be easily observed that the solutions learnt by HRKANs achieve the highest fitting accuracy (test MSE), while both ReLU-KAN and KAN fail the learn the solution accurately.

  • The mean test MSEs attained by ReLU-KANs and KANs are 2.18% and 6.80% respectively, whereas the mean test MSE attained by HRKANs is 0.0434%, at a difference in the scale of 10000%. This demonstrates the strong fitting ability of HRKANs.

  • Also, the standard deviations of test MSEs over the 10 runs are the smallest for HRKANs, at the level of 0.129%, whereas those of ReLU-KANs and KANs are 4.28% and 9.43% respectively, at a difference in the scale of 1000%. This suggests stronger robustness in the learning of HRKANs.

  • As shown in Figure 5, the training loss, test loss and test MSE of ReLU-KANs keep fluctuating over the training process. One possible reason may be the discontinuity of the higher-order derivatives of activation functions in ReLU-KANs as discussed in § 3. Another possible explanation is the lack of regularization tools in ReLU-KANs compared to KANs, which deserve deeper investigation as potential further study.

  • The median and max-min-band of training loss, test loss and test MSE of HRKANs and KANs on the other hand show that HRKANs converge relatively rapidly, approaching optimum at around epoch 500, whereas KANs and ReLU-KANs seem to be still converging slowly at the end of the training process (as shown in the 2 plots at the right bottom in Figure 7). This demonstrates the fast convergence of HRKANs.

  • Lastly, we can see that the training time of both HRKANs and ReLU-KANs is much smaller than KANs, exhibiting the efficiency of matrix operations in HRKANs and ReLU-KANs. As mentioned before in the problem description, there is possibly extra overhead in computation for KANs because of its steps of grid extension and sparsification, but we may expect such steps won’t contribute to most of the computation time for KANs, so our judgement remains valid.

Loss and accuracy (median and max-min-band of 10 runs)

Figure 7. The median and max-min-band of training loss, test loss and test MSE (MSE between the test-set ground-truth solution and the learnt solutions) of 10 runs for the KAN, ReLU-KAN and HRKAN with the Burgers’ equation with viscosity. First column: all 3000 epoches; Second column: last 2000 epoches; Last column: last 1000 epoches.

Burgers' Equation with viscosity

The Burgers’ equation with viscosity is

ut+uux−νuxx=0(5)u_t + u u_x - \nu u_{xx} = 0 \tag*{(5)}

for (x,t)∈[−5,5]×[0,2.5](x,t) \in[-5,5] \times[0,2.5], with boundary and initial conditions

{u(−5,t)=0u(5,t)=0u(x,0)=1cosh⁡(x)\begin{cases} u(-5,t) = 0 \\ u(5,t) = 0 \\ u(x,0) = \frac{1}{\cosh(x)} \end{cases}

and ν=0.001\nu= 0.001.

We solve for the ground-truth solution using Fast Fourier Transform method [46]. As we have not included the implementation of grid extension in ReLU-KANs and HRKANs, to make the comparison between KANs, ReLU-KANs and HRKANs fair, we perform a change of variable y=x+54y = \frac{x+5}{4} to transform the domain into a square domain [0,2.5]×[0,2.5][0, 2.5] \times[0, 2.5]. The resulting PDE is

ut+14uux−ν16uxx=0(6)u_t + \frac{1}{4}u u_x - \frac{\nu}{16}u_{xx} = 0 \tag*{(6)}

for (x,t)∈[0,2.5]×[0,2.5](x,t) \in[0, 2.5] \times[0, 2.5], with boundary and initial conditions

{u(0,t)=0u(2.5,t)=0u(x,0)=1cosh⁡(4x−5)\begin{cases} u(0,t) = 0 \\ u(2.5,t) = 0 \\ u(x,0) = \frac{1}{\cosh(4x-5)} \end{cases}

In the train-set, the number of interior points is Npde=10000N_{\mathrm{pde}} = 10000, and the number of boundary points on the 2 boundaries and the number of points for the initial conditions are Nbc1=100N_{\mathrm{bc1}} = 100, Nbc2=100N_{\mathrm{bc2}} = 100 and Nic=100N_{\mathrm{ic}} = 100 respectively. The train-set data points on the interior are generated randomly with uniform distribution while those for the boundary and initial conditions are generated uniformly. We generate test-set data with a grid with grid size 200×200200 \times200.

We trained a KAN, a ReLU-KAN and a HRKAN using the loss function

L=αLpde+Lbc & ic(7)L = \alpha L_{\mathrm{pde}} + L_{\mathrm{bc\ \&\ ic}} \tag*{(7)}

where

Lpde=1Npde∑i=1Npde(ut(xi,ti)+14u(xi,ti)ux(xi,ti)−ν16uxx(xi,ti))2,L_{\mathrm{pde}} = \frac{1}{N_{\mathrm{pde}}} \sum_{i=1}^{N_{\mathrm{pde}}} \left(u_t(x_i,t_i) + \frac{1}{4}u(x_i,t_i)u_x(x_i,t_i) - \frac{\nu}{16}u_{xx}(x_i,t_i)\right)^2,
Lbc & ic=1Nbc1+Nbc2+Nic(∑i=1Nbc1(u(0,ti))2+∑i=1Nbc2(u(2.5,ti))2+∑i=1Nic(u(xi,0)−1cosh⁡(4xi−5))2).L_{\mathrm{bc\ \&\ ic}} = \frac{1}{N_{\mathrm{bc1}} + N_{\mathrm{bc2}} + N_{\mathrm{ic}}} \left(\sum_{i=1}^{N_{\mathrm{bc1}}}(u(0,t_i))^2 + \sum_{i=1}^{N_{\mathrm{bc2}}}(u(2.5,t_i))^2 + \sum_{i=1}^{N_{\mathrm{ic}}}\left(u(x_i,0)-\frac{1}{\cosh(4x_i-5)}\right)^2\right).

We set hyperparameter α=0.05\alpha= 0.05.

The structure of the KAN, ReLU-KAN and HRKAN are all of [2,3,3,3,1][2, 3, 3, 3, 1] with same support for all the bases as g=7g = 7 and k=3k = 3. For all three of them, we use the Adam optimizer, set the number of epoch to 3000 and train on the same GPU. We run the same experiment for 10 times. Same as in the previous experiment, we keep the implementation of grid extension and sparsification in KANs unchanged.

Results Discussion. Figure 6 shows the ground-truth solution and the solutions learnt by the KAN, ReLU-KAN and HRKAN and their residual difference compared against the ground-truth solution in one of the 10 runs with the Burgers’ equation with viscosity.

Solutions and their residual difference

Figure 6. The ground-truth solution and the solutions learnt by the KAN, ReLU-KAN and HRKAN (Top row) and their residual difference compared against the ground-truth solution (Bottom row) in one of the 10 runs with the Burgers’ equation with viscosity.

Figure 7 reveals the median and max-min-band of training loss, test loss and test MSE (Mean square error between the test-set ground-truth solution and the learnt solutions) of 10 runs for the KAN, ReLU-KAN and HRKAN with the Burgers’ equation with viscosity. For ease of reading, the first column shows the curves for the entire training process, while the second and last columns show those in the last 2000 and last 1000 epochs respectively.

Table 2 exhibits the means and standard deviations of test MSE, and means of training time for the KAN, ReLU-KAN and HRKAN in the 10 runs with the Burgers’ equation with viscosity.

ReLU-KANHRKANKAN
Mean of Test MSE in 10 runs (in %)0.5750.2291.48
Std. of Test MSE in 10 runs (in %)2.230.8615.85
Mean of training time in 10 runs (in second)61.675.0624

Table 2. The means and standard deviations of test MSE and means of training time for the KAN, ReLU-KAN and HRKAN in the 10 runs with the Burgers’ equation with viscosity.

There are some key observations to make:

  • From Figure 6 and Figure 7 and Table 2, it is obvious that the solutions learnt by HRKANs achieve the highest fitting accuracy (test MSE), while the accuracies of the solutions learnt by ReLU-KANs and KANs are relatively lower.

  • The largest difficulty for all the three models to learn the solution probably lies on the "discontinuity" of the ground-truth solution near t=2.5t = 2.5, as displayed in Figure 6. Although all three models fail to learn such discontinuity in certain degree, the error in fitting accuracy is the lowest for HRKANs, as shown in Figure 6, the last row of Figure 7 and first row of Table 2.

  • The mean test MSEs attained by ReLU-KANs and KANs are 0.575% and 1.48% respectively, whereas the mean test MSE attained by HRKANs is 0.229%, demonstrating a stronger fitting ability of HRKANs.

  • Also, the standard deviations of test MSEs over the 10 runs are the smallest for HRKAN, at the level of 0.861%, whereas those of ReLU-KAN and KAN are 2.23% and 5.85%. This suggests the stronger robustness in the learning of HRKANs.

  • In Figure 7, the test MSE converges at the fastest speed for HRKAN, approaching the minimum at around epoch 1500, whereas those of ReLU-KAN and KAN are still on the way of slowly converging at the end of the training process. This suggests the fast convergence of HRKANs. The fluctuation of the training and test loss of HRKANs after epoch 1250 may imply over-fitting of HRKANs after epoch 1250 to the "discontinuous" solution. The relatively more fluctuating training loss and test loss in ReLU-KANs and HRKANs may also due to the lack of regularization tools in these two models, compared to KANs.

  • Lastly, we can see that the training time of both HRKAsN and ReLU-KANs is much smaller than KAN, exhibiting the efficiency of matrix operations in HRKANs and ReLU-KANs. As mentioned before, there is possibly extra overhead in computation for KANs because of its steps of grid extension and sparsification, but we expect such steps won’t contribute to most of the computation time for KANs, so our judgement remains valid.

Conclusion

In this paper, we proposed a new basis of activation functions for KAN, namely, Higher-order-ReLU (HR). Such Higher-order-ReLU is (1) simpler than the B-Spline basis functions in the original KANs, (2) allows matrix-based operations for fast GPU computing, (3) possesses smooth and non-zero higher-order derivatives essential to Physics-informed neural networks (PINNs). We evaluated Higher-order-ReLU-KANs (HRKANs) on two representative and famous PDEs, namely, a linear Poisson equation, and a non-linear Burgers’ equation with viscosity. The detailed experiments exhibit the higher fitting accuracy, stronger robustness and faster convergence of HRKANs. Not only can HRKANs find solutions of PDEs, we expect HRKANs to have strong potential in directions like (1) identifying coefficients of PDEs, (2) finding the optimal control of PDEs, and (3) explaining a neural operator etc.

References

  1. [1]S. Larsson and V. Thomée, Partial differential equations with numerical methods. Springer, 2003, vol. 45.DOI
  2. [2]T. J. Hughes, J. A. Cottrell, and Y. Bazilevs, “Isogeometric analysis: Cad, finite elements, nurbs, exact geometry and mesh refinement,” Computer methods in applied mechanics and engineering, vol. 194, no. 39-41, pp. 4135–4195, 2005.DOI
  3. [3]T. J. Hughes, The finite element method: linear static and dynamic finite element analysis. Courier Corporation, 2012.
  4. [4]T. Rabczuk, J.-H. Song, X. Zhuang, and C. Anitescu, Extended finite element and meshfree methods. Academic Press, 2019.DOI
  5. [5]R. J. LeVeque, Finite difference methods for ordinary and partial differential equations: steady-state and time-dependent problems. SIAM, 2007.
  6. [6]F. M. L. M. M. Darwish, The finite volume method in computational fluid dynamics, 2016.DOI
  7. [7]C. A. Brebbia, J. C. F. Telles, and L. C. Wrobel, Boundary element techniques: theory and applications in engineering. Springer Science & Business Media, 2012.
  8. [8]M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations,” Journal of Computational physics, vol. 378, pp. 686–707, 2019.DOI
  9. [9]L. Lu, X. Meng, Z. Mao, and G. E. Karniadakis, “Deepxde: A deep learning library for solving differential equations,” SIAM review, vol. 63, no. 1, pp. 208–228, 2021.DOI
  10. [10]E. Haghighat, M. Raissi, A. Moure, H. Gomez, and R. Juanes, “A deep learning framework for solution and discovery in solid mechanics,” arXiv preprint arXiv:2003.02751, 2020.arxiv.org/abs/2003.02751
  11. [11]Y. Cao, C. C. So, J. Wang, and S. P. Yung, “System stabilization of pdes using physics-informed neural networks (pinns),” in Proceedings of the 43rd Chinese control conference. IEEE, 2024, pp. 8759–8764.DOI
  12. [12]S. Mowlavi and S. Nabi, “Optimal control of pdes using physics-informed neural networks,” Journal of Computational Physics, vol. 473, p. 111731, 2023.
  13. [13]L. Lu, X. Meng, S. Cai, Z. Mao, S. Goswami, Z. Zhang, and G. E. Karniadakis, “A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data,” Computer Methods in Applied Mechanics and Engineering, vol. 393, p. 114778, 2022.DOI
  14. [14]L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis, “Learning nonlinear operators via deeponet based on the universal approximation theorem of operators,” Nature machine intelligence, vol. 3, no. 3, pp. 218–229, 2021.
  15. [15]L. Lu, P. Jin, and G. E. Karniadakis, “Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators,” arXiv preprint arXiv:1910.03193, 2019.DOI
  16. [16]Z. Li, N. Kovachki, K. Azizadehnesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar, “Fourier neural operator for parametric partial differential equations, arXiv,” arXiv preprint arXiv:2010.08895, 2020.arxiv.org/abs/2010.08895
  17. [17]N. Kovachki, Z. Li, B. Liu, K. Azizadehnesheli, K. Bhattacharya, A. Stuart, and A. Anandkumar, “Neural operator: Learning maps between function spaces with applications to pdes,” Journal of Machine Learning Research, vol. 24, no. 89, pp. 1–97, 2023.
  18. [18]S. Wang, H. Wang, and P. Perdikaris, “Learning the solution operator of parametric partial differential equations with physics-informed deeponets,” Science advances, vol. 7, no. 40, p. eabi8605, 2021.
  19. [19]Z. Li, H. Zheng, N. Kovachki, D. Jin, H. Chen, B. Liu, K. Azizadehnesheli, and A. Anandkumar, “Physics-informed neural operator for learning partial differential equations,” ACM/JMS Journal of Data Science, vol. 1, no. 3, pp. 1–27, 2024.arxiv.org/abs/2111.03794
  20. [20]S. Goswami, A. Bora, Y. Yu, and G. E. Karniadakis, “Physics-informed deep neural operator networks,” in Machine Learning in Modeling and Simulation: Methods and Applications. Springer, 2023, pp. 219–254.arxiv.org/abs/2207.05748
  21. [21]H. Chen, R. Wu, E. Grinspun, C. Zheng, and P. Y. Chen, “Implicit neural spatial representations for time-dependent pdes,” in International Conference on Machine Learning. PMLR, 2023, pp. 5162–5177.arxiv.org/abs/2210.00124
  22. [22]Z. Long, Y. Lu, X. Ma, and B. Dong, “Pde-net: Learning pdes from data,” in International conference on machine learning. PMLR, 2018, pp. 3208–3216.arxiv.org/abs/1710.09668
  23. [23]C. C. So, T. O. Li, C. Wu, and S. P. Yung, “Differential spectral normalization (dsn) for pde discovery,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 11, 2021, pp. 9675–9684.
  24. [24]S. Haykin, Neural networks: a comprehensive foundation. Prentice Hall PTR, 1998.
  25. [25]G. Cybenko, “Approximation by superpositions of a sigmoidal function,” Mathematics of control, signals and systems, vol. 2, no. 4, pp. 303–314, 1989.
  26. [26]K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,” Neural networks, vol. 2, no. 5, pp. 359–366, 1989.DOI
  27. [27]M. R. Bachute and J. M. Subhedar, “Autonomous driving architectures: insights of machine learning and deep learning algorithms,” Machine Learning with Applications, vol. 6, p. 100164, 2021.DOI
  28. [28]T. Wu, S. He, J. Liu, S. Sun, K. Liu, Q.-L. Han, and Y. Tang, “A brief overview of chatgpt: The history, status quo and potential future development,” IEEE/CAA Journal of Automatica Sinica, vol. 10, no. 5, pp. 1122–1136, 2023.DOI
  29. [29]Z. Liu, Y. Wang, S. Vaidya, F. Ruehle, J. Halverson, M. Soljačić, T. Y. Hou, and M. Tegmark, “Kan: Kolmogorov-arnold networks,” arXiv preprint arXiv:2404.19756, 2024.arxiv.org/abs/2404.19756
  30. [30]A. N. Kolmogorov, On the representation of continuous functions of several variables by superpositions of continuous functions of a smaller number of variables. American Mathematical Society, 1961.DOI
  31. [31]J. Braun and M. Griebel, “On a constructive proof of kolmogorov’s superposition theorem,” Constructive approximation, vol. 30, pp. 653–675, 2009.
  32. [32]S. SS, “Chebyshev polynomial-based kolmogorov-arnold networks: An efficient architecture for nonlinear function approximation,” arXiv preprint arXiv:2405.07200, 2024.
  33. [33]Z. Li, “Kolmogorov-arnold networks are radial basis function networks,” arXiv preprint arXiv:2405.06721, 2024.arxiv.org/abs/2405.06721
  34. [34]Z. Bozorgasl and H. Chen, “Wav-kan: Wavelet kolmogorov-arnold networks,” arXiv preprint arXiv:2405.12832, 2024.arxiv.org/abs/2405.12832
  35. [35]A. A. Aghaei, “fkan: Fractional kolmogorov-arnold networks with trainable jacobi basis functions,” arXiv preprint arXiv:2406.07456, 2024.arxiv.org/abs/2406.07456
  36. [36]——, “rkan: Rational kolmogorov-arnold networks,” arXiv preprint arXiv:2406.14495, 2024.arxiv.org/abs/2406.14495
  37. [37]J. Xu, Z. Chen, J. Li, S. Yang, W. Wang, X. Hu, and E. C.-H. Ngai, “Fourierkan-gcf: Fourier kolmogorov-arnold network—an effective and efficient feature transformation for graph collaborative filtering,” arXiv preprint arXiv:2406.01034, 2024.
  38. [38]Q. Qiu, T. Zhu, H. Gong, L. Chen, and H. Ning, “Relu-kan: New kolmogorov-arnold networks that only need matrix addition, dot multiplication, and relu,” arXiv preprint arXiv:2406.02075, 2024.arxiv.org/abs/2406.02075
  39. [39]D. W. Abueidda, P. Pantidis, and M. E. Mobasher, “Deepokan: Deep operator network based on kolmogorov arnold networks for mechanics problems,” arXiv preprint arXiv:2405.19143, 2024.arxiv.org/abs/2405.19143
  40. [40]Y. Wang, J. Sun, J. Bai, C. Anitescu, M. S. Eshaghi, X. Zhuang, T. Rabczuk, and Y. Liu, “Kolmogorov arnold informed neural network: A physics-informed deep learning framework for solving pdes based on kolmogorov arnold networks,” arXiv preprint arXiv:2406.11045, 2024.DOI
  41. [41]H. Wu, H. Luo, Y. Ma, J. Wang, and M. Long, “Ropinn: Region optimized physics-informed neural networks,” arXiv preprint arXiv:2405.14369, 2024.arxiv.org/abs/2405.14369
  42. [42]A. A. Howard, B. Jacob, S. H. Murphy, A. Heinlein, and P. Stinis, “Finite basis kolmogorov-arnold networks: domain decomposition for data-driven and physics-informed problems,” arXiv preprint arXiv:2406.19662, 2024.arxiv.org/abs/2406.19662
  43. [43]M. Calafà, E. Hovad, A. P. Engsig-Karup, and T. Andriollo, “Physics-informed holomorphic neural networks (pihnns): Solving linear elasticity problems,” arXiv preprint arXiv:2407.01088, 2024.DOI
  44. [44]H. Prautzsch, W. Boehm, and M. Paluszny, Bézier and B-spline techniques. Springer, 2002, vol. 6.DOI
  45. [45]J. F. Epperson, “On the runge example,” The American Mathematical Monthly, vol. 94, no. 4, pp. 329–341, 1987.DOI
  46. [46]G. Chen, B. Cloutier, N. Li, B. Muite, and P. Rigge, “Parallel spectral numerical methods,” 2012.

Paper details

vlnc:W_xr8ut2x6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents