betaFeedback

WorksMathematics

Spectral methods for Neural Integral Equations

Abstract

Neural integral equations are deep learning models based on the theory of integral equations, where the model consists of an integral operator and the corresponding equation (of the second kind) which is learned through an optimization procedure. This approach allows to leverage the nonlocal properties of integral operators in machine learning, but it is computationally expensive. In this article, we introduce a framework for neural integral equations based on spectral methods that allows us to learn an operator in the spectral domain, resulting in a cheaper computational cost, as well as in high interpolation accuracy. We study the properties of our methods and show various theoretical guarantees regarding the approximation capabilities of the model, and convergence to solutions of the numerical methods. We provide numerical experiments to demonstrate the practical effectiveness of the resulting model.

Introduction

The theory of integral equations (IEs) has found important applications in several disciplines of science. In physics, for example, integral operators and integral equations are found in the theory of nonlocal gravity [10], plasma physics [34], fluid dynamics [3], and nuclear reactor physics [1]. Several applications are also found in engineering [33], brain dynamics [2, 35], and epidemiology [12]. Such a wide range of applications has therefore motivated the study of IE and integro-differential equation (IDE) models in machine learning, for operator learning tasks [39, 40, 41]. These models are called neural IEs (NIEs) and neural IDEs (NIDEs), respectively. They need to solve an IE or IDE at each step of the training (and for each data sample), therefore incurring in significant computational costs. While there is a benefit in modeling dynamics using neural IEs and IDEs due to to the nonlocality of the integral operators, it is of interest to obtain formulation of these models that are more computationally efficient.

Operator learning is a branch of machine learning where the training procedure aims at obtaining an operator acting in an infinite dimensional space of functions. The theoretical foundation of operator learning has been significantly developed in [30], based on results dating back to the 90’s, and more recently found examples in several other works, e.g. [15, 16, 21, 28]. Operator learning allows to learn the governing equations of a system from data alone, without further assumptions or knowledge on the properties of the system that generated the dataset. As such, it is greatly useful in studying complex systems whose theoretical properties are not completely understood.

The scope of this article is to introduce a framework for neural IEs, which is an operator learning problem, based on spectral methods. This allows to perform integration in the spectral domain, greatly simplifying the architecture of the models, but still retaining their great modeling capabilities of NIEs, especially in regards to long range dependencies (high nonlocality).

The convenience of this method with respect to the approach to neural network approaches for learning integral operators in IEs and IDEs found in [39, 40, 41] lies in the following facts:

  • A small (in terms of parameters) neural network GθG_{\theta} can achieve comparable expressivity and task accuracy to larger models;

  • Integration in this setting consists only of a matrix multiplication as opposed to quadrature rules or Monte Carlo methods, therefore improving computational speed and memory scalability of the model;

  • Th output of the spectral IE solver implemented in this article is a solution y\mathbf{y} which is expanded in the Chebyshev basis, and that can therefore be easily interpolated.

Overall, this method produces highly smooth outputs that can be interpreted as denoised approximations of the dataset. Throughout this article we will use Chebyshev polynomials of the first kind, referring to them simply as Chebyshev polynomials without further specification.

The codes implemented for this method, and for the experiments found in § 5, can be found at https://github.com/emazap7/Spectral_NIE.

Methods

In this section we introduce and discuss the spectral approaches used in this article for integral operator learning. We consider the one dimensional case, where our method is based on Chebyshev polynomials, and defer a generalization of this methods to higher dimensions to a later work.

We consider integral equations of Fredholm and Volterra types which take the form

y(t)=f(y,t)+λ∫−1α(t)G(y,t,s) ds(1)y(t)=f(\mathbf{y},t)+\lambda\int_{-1}^{\alpha(t)}G(\mathbf{y},t,s)\,ds \tag*{(1)}

where α(t)=t\alpha(t)=t for a Volterra equation, and α(t)=1\alpha(t)=1 for a Fredholm equation, G:R×[−1,1]×[−1,1]→RG:\mathbb{R}\times[-1,1]\times[-1,1]\to\mathbb{R} is potentially non-linear function in y\mathbf{y} called the kernel, and y:[−1,1]→R\mathbf{y}:[-1,1]\to\mathbb{R} is the unknown function. In this article we consider equations defined on the interval [−1,1][-1,1], which are sufficient for the scope of machine learning in the assumption that the time intervals have been normalized. In the context of traditional IEs (with no learning), spectral approaches for linear equations in y\mathbf{y}, where G(y,t,s)=K(t,s)yG(\mathbf{y},t,s)=K(t,s)\mathbf{y} have been treated in [29], while the nonlinear case where GG has the form G(y,t,s)=K(t,s)ymG(\mathbf{y},t,s)=K(t,s)y^{m} for some m>1m>1 has been considered in [38]. In the present work, we consider arbitrary functions GG where the learning process determines a neural network GθG_{\theta} which is obtained through optimization on a given dataset. Our setup is an operator learning problem where we want to learn the integral operator by learning the parameters θ\theta of GθG_\theta, therefore obtaining

y(t)=f(y,t)+λ∫−1α(t)Gθ(y,t,s) ds,(2)\mathbf{y}(t) = f(\mathbf{y},t) + \lambda\int_{-1}^{\alpha(t)} G_\theta(\mathbf{y},t,s)\,ds, \tag*{(2)}

Let us denote by Cj(t)C_j(t) the jthj^{\mathrm{th}} Chebyshev polynomial of the first kind, and let NN be a fixed natural number. We write the truncated Chebyshev series as

y(t)≈yN(t):=∑j=0NajCj(t),(3)\mathbf{y}(t) \approx\mathbf{y}_N(t) := \sum_{j=0}^{N} a_j C_j(t), \tag*{(3)}

where aj∈Ra_j \in\mathbb{R} are coefficients that determine the expansion of y\mathbf{y} in Cj(t)C_j(t). The collocation points {tk}k=0N⊂[−1,1]\{t_k\}_{k=0}^{N} \subset[-1,1] are optimally taken to be the Chebyshev collocation points tk=cos⁡(kπN)t_k = \cos\left(\frac{k\pi}{N}\right) for k=0,…,Nk = 0,\ldots,N, but we will experimentally show that the method is effective even when the points tkt_k are taken to be different from this choice, which is of importance since datasets might have arbitrary time stamps (including irregularly sampled time points).

Using Equation (3), from Equation (1) we obtain

y(ti)=f(y(ti),ti)+λ∫−1α(ti)G(∑j=0NajCj(s),ti,s) ds,(4)\mathbf{y}(t_i) = f(\mathbf{y}(t_i),t_i) + \lambda\int_{-1}^{\alpha(t_i)} G\left(\sum_{j=0}^{N} a_j C_j(s),t_i,s\right)\,ds, \tag*{(4)}

which we want to solve for all i=0,…,Ni = 0,\ldots,N. Applying this setup to the case of integral operators parameterized by neural networks, we find the equation corresponding to Equation (4)

y(ti)=f(y(ti),ti)+λ∫−1α(ti)Gθ(∑j=0NajCj(s),ti,s) ds.(5)\mathbf{y}(t_i) = f(\mathbf{y}(t_i),t_i) + \lambda\int_{-1}^{\alpha(t_i)} G_\theta\left(\sum_{j=0}^{N} a_j C_j(s),t_i,s\right)\,ds. \tag*{(5)}

In order to solve Equation (4) or Equation (5) we need an efficient way of computing the integrals ∫−1α(ti)G(∑j=0NajCj(s),ti,s)ds\int_{-1}^{\alpha(t_i)}G\left(\sum_{j=0}^{N}a_jC_j(s),t_i,s\right)ds, and ∫−1α(ti)Gθ(∑j=0NajCj(s),ti,s)ds\int_{-1}^{\alpha(t_i)}G_\theta\left(\sum_{j=0}^{N}a_jC_j(s),t_i,s\right)ds for all i=0,…,Ni = 0,\ldots,N. Let us first consider the former case, where θ\theta does not appear. For the moment we will only concern ourselves with being able to solve Equation (4), and Equation (5) for given parameters θ\theta, without concerning ourselves with optimizing the parameters θ\theta.

Observe that for each fixed spectral collocation point tit_i, and fixed y\mathbf{y} (hence fixed coefficients aka_k), the function GG induces a function G^i:[−1,1]⟶R\widehat{G}^i:[-1,1]\longrightarrow\mathbb{R} obtained through the correspondence s↦G(∑j=0NajCj(s),ti,s)s\mapsto G\left(\sum_{j=0}^{N}a_jC_j(s),t_i,s\right). Here we suppress the dependence of G^i\widehat{G}^i on y\mathbf{y} for notational simplicity. Similarly to how y\mathbf{y} has been expanded through a truncated Chebyshev series we have

G^(s)≈G^N(s):=∑r=0NbrCr(s),(6)\widehat{G}(s) \approx\widehat{G}_N(s) := \sum_{r=0}^{N} b_r C_r(s), \tag*{(6)}

for some coefficients brb_r. In the assumption that the brb_r are known, there is a computationally efficient method for computing the integral ∫−1α(ti)G^i(s)ds\int_{-1}^{\alpha(t_i)}\widehat{G}^i(s)ds through the coefficients of the Chebyshev expansion, given in [13, 14]. We recall these results for Fradholm and Volterra equations here.

First, let us denote by X\mathcal{X} the space of infinite sequences (a0,…,an,…)(a_0,\ldots,a_n,\ldots) of real numbers. We can express a function y:[−1,1]⟶R\mathbf{y}:[-1,1]\longrightarrow\mathbb{R} as such a sequence by associating to it its Chebyshev coefficients. In practice, we are interested in sequences that vanish after a certain N∈NN\in\mathbb{N}, since we consider in this article finite Chebyshev expansions. As such, we can express integral (Fredholm and Volterra) operators as mappings J:X⟶X\mathcal{J}:\mathcal{X}\longrightarrow\mathcal{X}. Let b∈X\mathbf{b}\in\mathcal{X}, and let b^\widehat{\mathbf{b}} denote the sequence obtained from b\mathbf{b} by defining b^0:=2b0\widehat{b}_0 := 2b_0, and setting b^n=bn\widehat{b}_n = b_n for all $n>0. Then, (see [13, 14]) we define d:=J(b)\mathbf{d} := \mathcal{J}(\mathbf{b}) by the formula

dk=b^k−1−b^k+12k,(7)d_k = \frac{\hat{b}_{k-1}-\hat{b}_{k+1}}{2k}, \tag*{(7)}

for all k>1k > 1, and

d0=∑i=1∞(−1)i+1di.(8)d_0 = \sum_{i=1}^{\infty}(-1)^{i+1}d_i. \tag*{(8)}

In this article, in practice, the latter is finite, as all the coefficients aka_k are eventually zero (we only consider finite Chenyshev expansions), and we therefore will start using finite sums up to NN, to explicitly take this fact into account. This mapping has been called spectral integration in [14], and we will follow this convention here. We can now define the Fredholm and Volterra integral operators in spectral form, JF\mathcal{J}_{F} and JV\mathcal{J}_{V} respectively, by the formulae

∫−11G^i(s) ds=2∑k=0Nd2k+1(9)\int_{-1}^{1} \hat{G}^{i}(s)\,ds = 2\sum_{k=0}^{N}d_{2k+1} \tag*{(9)}
∫−1α(ti)G^i(s) ds=12d0+∑k=1NdkCk(α(ti)),(10)\int_{-1}^{\alpha(t_i)} \hat{G}^{i}(s)\,ds = \frac{1}{2}d_0+\sum_{k=1}^{N}d_k C_k(\alpha(t_i)), \tag*{(10)}

for all spectral collocation points tit_i, cf. Lemma 2.1 in [14] and the Appendix to [13].

Solving the IE problem in Equation (4) presents now the complication that once we have the Chebyshev expansion of yy, i.e. its coefficients ana_n, n≤Nn \le N, we need to compute the Chebyshev coefficients bnb_n of GiG^i for each collocation point. This procedure requires an integration per collocation point in order to project the function GG onto the Chebyshev basis. However, when our task is to learn the integral operator as in the present article, we can bypass the issue altogether, and learn the neural network GθG_\theta as a mapping in spectral space, directly giving the correspondence a↦b\mathbf{a} \mapsto\mathbf{b}. Moreover, integration in this formulation simply accounts to performing a matrix multiplication, as seen in Equation (9) and Equation (10).

Theoretical framework and approximation capabilities

A fundamental reason to formulate an operator learning problem on a finite space, such as the space spanned by the Chebyshev polynomials of degree at most nn, is that it allows us to effectively perform the optimization process on a finite dimensional approximation of the original infinite dimensional space. In order to guarantee that the algorithm produces solutions close to the target ones, it is important to suitably choose the finite dimensional representation of the original space.

In this article, we use Chebyshev polynomials, which are known to provide stability and rapid convergence in approximation problems. For example, they are known to satisfy an optimality condition on the approximation of the degree n+1n+1 monomial via lower degree polynomials over the closed interval [−1,1][-1,1]. Moreover, the normalized polynomial CnC_n realizes the minimum uniform norm among the degree nn monic polynomials. As a result, when minimizing a function with uniformly bounded n+1n+1 derivative, Chebyshev interpolation at the Chebyshev spectral collocation points minimizes the error. These properties result in a decreased Runge phenomenon outside of the interpolation points, therefore reducing oscillations in interpolation problems.

After projecting a function on a truncated polynomial basis, an integral operator TT applied to this function does not generally produce another truncated polynomial, and the explicit form obtained needs to be determined based on the form of the kernel of the operator. However, we can subsequently project the output again to consider the restriction of the operator on the space of Chebyshev polynomials with fixed highest degree. This process effectively produces a representation of the operator on a finite dimensional space, which is the target of our learning algorithm. We will show that, under some relatively mild assumptions, we can learn a neural network that approximates the finite dimensional representation of an integral operator, and such that the solutions of the corresponding equation in finite space approximate the solutions of the integral equation in the infinite dimensional space.

We can reformulate the method as described in § 2 using a more general language. While this does not add any new information to the setup, it makes a general treatment simpler. In this regard, we let XX denote a Banach space of functions of interest (whose precise definition depends on the problem considered), and we indicate the integral operator of Equation (1) by T:X→XT:X\to X, where we assume that a choice of α\alpha is performed.

Definition 3.1. Let f:[a,b]→Rf:[a,b]\to\mathbb{R} be a function. Then, we say that ff is Hölder continuous if there exists a constant cc such that

∣f(x)−f(y)∣≤c∣x−y∣β,|f(x)-f(y)| \le c|x-y|^{\beta},

for all x,y∈[a,b]x,y\in[a,b]. The case where β=1\beta=1 is of particular importance, and an Hölder function of exponent β=1\beta=1 is called Lipschitz continuous. We define the set Ck,β([a,b])C^{k,\beta}([a,b]) of all Hölder functions whose kkth derivative is Hölder continuous, and define the norm

∥f∥k,β=∥f∥k+sup⁡{∣fk(x)−fk(y)∣∣x−y∣β∣x,y∈[a,b], x≠y},\|f\|_{k,\beta}=\|f\|_k+\sup\left\{\frac{|f^{k}(x)-f^{k}(y)|}{|x-y|^{\beta}}\mid x,y\in[a,b],\ x\ne y\right\},

where ∥f∥k:=max⁡r≤k∥fr∥∞\|f\|_k:=\max_{r\le k}\|f^{r}\|_\infty is the maximum among the uniform norms of the derivatives of ff. It is well known that for any 0<β≤10<\beta\le1, the space Ck,β([a,b])C^{k,\beta}([a,b]) is a Banach space with the norm ∥⋅∥k,β\|\cdot\|_{k,\beta}. Moreover, Ck,γ([a,b])C^{k,\gamma}([a,b]) compactly embeds into Ck,β([a,b])C^{k,\beta}([a,b]) whenever 1≥γ>β>01\ge\gamma>\beta>0.

In this article, the space XX is assumed to be a Hölder space, and we will consider some suitable subspace of XX for our theoretical guarantees. When the integral operator is parameterized by a neural network with parameters θ\theta, we indicate it by TθT_\theta when we want to explicitly make this fact clear.

We observe that the Chebyshev expansion considered in Equation (3) is the result of projecting a function y∈Xy\in X on the space XNX_N spanned by the first NN Chebyshev polynomials. So, we have a projector (or projection operator) PN:X→XNP_N:X\to X_N, the precise formulation of which will be given in the algorithm. From now on, we will consider a situation where the operator TT maps in a closed subspace RR of XX such that for each y∈Ry\in R the projections PNy=:yNP_Ny=:y_N satisfy ∥y−yN∥→0\|y-y_N\|\to0 as N→∞N\to\infty.

The method of Section 2 consists of approximating the integral equation (1−T)y=f(1-T)y=f through a function and a spectral integration on the Chebyshev space XNX_N, and solving the corresponding integral equation, which we denote by

(1−PNJTθ,N)yN=PNf,(11)(1-P_N\mathcal{J}T_{\theta,N})y_N=P_Nf, \tag*{(11)}

where J\mathcal{J} denotes the spectral integration (either Fredholm, or Volterra), and Tθ,NT_{\theta,N} indicates a neural network parameterized by θ\theta and that is defined over the space XNX_N. We consider now also the related problem of “projecting” the IE over XNX_N by means of the projection PN:X→XNP_N:X\to X_N, for a choice of N>0N>0

(1−PNT)yN=PNf(12)(1-P_NT)y_N=P_Nf \tag*{(12)}

in the space XNX_N. See also Chapter 12 in [5] for a similar procedure. The underlying idea of the method is that since the spaces XNX_N approximate increasingly well the given subspace of XX, solving Equation (12) for NN large enough produces an approximate solution yNy_N which is close enough to the solution yy of Equation (1). Analogously, when using a parameterized operator (as in Equation (2)), we can solve with high accuracy when NN is large enough.

Some questions now arise.

Question 3.2. Firstly, it is of interest to understand whether if uN∈XNu_N \in X_N is a solution of Equation (12) for each given NN, we have ∥y−uN∥⟶0\lVert y-u_N\rVert\longrightarrow0 as N⟶∞N \longrightarrow\infty, where yy is the real solution of the IE before projecting it. We observe that while this is naturally expected (cf. with the intuitive idea above), it is not necessarily true.

Question 3.3. Secondly, another important question regards the possibility of approximating projected operators PNT:XN⟶XNP_NT:X_N \longrightarrow X_N through the integral operators in spectral domain described in § 2. The importance of this question regards the fact that if TT is such that ∥y−uN∥⟶0\lVert y-u_N\rVert\longrightarrow0 as N⟶∞N \longrightarrow\infty, then we can approximate solutions of Equation (1) by knowing solutions of Equation (12), and approximating PNTP_NT would mean that we can approximate the original problem through a neural network in the spectral space with high accuracy (upon choosing NN large enough).

Question 3.4. Thirdly, it remains to be understood whether when the first and second problems have positive answer, it is possible to learn the operators PNTP_NT, e.g. through Stochastic Gradient Descent, or adjoint methods as in [41].

We start by considering Question 3.2 where we see that in some well-behaved situations solving the projected equation gives an approximate solution of the original equation. We recall that an operator T:X⟶XT:X \longrightarrow X is said to be bounded if ∥T(y)∥≤M∥y∥\lVert T(y)\rVert\le M\lVert y\rVert for all y∈Xy \in X and some MM. While boundedness and continuity coincide for linear operators, the two concepts are different when dealing with nonlinear operators, as in our situation.

We assume that the integral equation T(y)+f=yT(y)+f=y admits a solution uu that is isolated and of non-zero index for TT (see [24]), where uu is a Hölder continuous function that belongs to Ck,βC^{k,\beta} for some kk and β<12\beta< \frac{1}{2}. We assume that TT is completely continuous and Frechet differentiable, and that T′T' is bounded, with bound LL. These assumptions are not very restrictive, see for instance [24] for several families of integral operators satisfying them. Also, in what follows, we assume that TT maps the Hölder space Ck,βC^{k,\beta} into itself. This simply means that TT takes regular enough functions and does not make them less regular. Here we set DD to denote a closed bounded domain of Ck,βC^{k,\beta} containing uu, such that T(D)⊂Ck,γT(D) \subset C^{k,\gamma} for some γ>2β\gamma> 2\beta, if ∥T(D)∥>0\lVert T(D)\rVert> 0 and sup⁡∥T(D)∥γ<∞\sup\lVert T(D)\rVert_\gamma< \infty. We let RR denote the smallest closed subspace containing the image T(D)⊂RT(D) \subset R, and assume that the PNP_N's are uniformly bounded on RR. Lastly, we assume that 11 is not an eigenvalue of T′(u)T'(u), and therefore (1−T′(u))−1(\mathbb{1}-T'(u))^{-1} exists and it is bounded. The results found in [27, 32] give conditions under which examples of such operators can be given, and show that under relatively mild regularity assumptions on the kernel of the operator, a large family of examples arises. Carathéodory-type functions, for instance, produce a class of operators of interest.

Theorem 3.5. In the hypotheses above, the projected equation

PNT(yN)+f=yNP_NT(y_N)+f=y_N

admits at least a solution uNu_N for all NN large enough. Moreover,

∥u−uN∥⟶0\lVert u-u_N\rVert\longrightarrow0

as N⟶∞N \longrightarrow\infty, with the same rate as ∥PNu−u∥⟶0\lVert P_Nu-u\rVert\longrightarrow0.

Proof. For yy in Ck,γC^{k,\gamma}, Jackson’s Theorem gives us that

∥y−PNy∥∞≤Mk(y)nk+γ⟶0,\lVert y-P_Ny\rVert_\infty\le\frac{M_k(y)}{n^{k+\gamma}} \longrightarrow0,

where Mk(y)M_k(y) is a constant linearly related to the Hölder constant of the kthk^{\mathrm{th}} derivative. Applying Kalandiya’s Theorem [19, 20], the convergence in uniform norm implies convergence in Hölder norm. So, we find that

∥y−PNy∥⟶0,\lVert y-P_Ny\rVert\longrightarrow0,

for all y∈Ry \in R, using the fact that PNP_N are uniformly bounded on RR. From the fact that the operator TT has an isolated fixed point of non-zero index, we deduce that we can apply the framework of [24] Chapter 3 (see also [25]). This gives us that the projected equations eventually admit solutions uNu_N, and that the projected solutions converge to uu, the fixed point of the operator T+fT+f. This facts do not depend on the function ff appearing in the equation, as one can easily verify directly. Now, we want to show that the numerical scheme converges to uu uniformly, at the same rate as ∥PNu−u∥\lVert P_Nu-u\rVert. To do so, we proceed as in [6]. We write the equality

(1−T′(u))(uN−u)=(PN−1)T′(u)(uN−u)+(PN−1)u+PN[T(uN)−T(u)−T′(u)(uN−u)].(1-T'(u))(u_N-u)=(P_N-1)T'(u)(u_N-u)+(P_N-1)u+P_N[T(u_N)-T(u)-T'(u)(u_N-u)].

We then derive the inequalities

∥uN−u∥≤∥(1−T′(u))−1T′(u)(uN−u)∥+∥(1−T′(u))−1(PN−1)u∥+∥(1−T′(u))−1PN[T(uN)−T(u)−T′(u)(uN−u)]∥≤∥(1−T′(u))−1∥⋅∥T′(u)∥⋅∥uN−u∥+∥(1−T′(u))−1∥⋅∥PNu−1u∥+∥(1−T′(u))−1PN(T(uN)−T(u)−T′(u)(uN−u))∥∥uN−u∥⋅∥uN−u∥.\begin{aligned} \lVert u_N-u\rVert &\le\lVert(1-T'(u))^{-1}T'(u)(u_N-u)\rVert+\lVert(1-T'(u))^{-1}(P_N-1)u\rVert\\ &\quad+\lVert(1-T'(u))^{-1}P_N[T(u_N)-T(u)-T'(u)(u_N-u)]\rVert\\ &\le\lVert(1-T'(u))^{-1}\rVert\cdot\lVert T'(u)\rVert\cdot\lVert u_N-u\rVert+\lVert(1-T'(u))^{-1}\rVert\cdot\lVert P_Nu-1u\rVert\\ &\quad+\frac{\lVert(1-T'(u))^{-1}P_N(T(u_N)-T(u)-T'(u)(u_N-u))\rVert}{\lVert u_N-u\rVert}\cdot\lVert u_N-u\rVert. \end{aligned}

The fraction qN=∥(1−T′(u))−1PN(T(uN)−T(u)−T′(u)(uN−u))∥∥uN−u∥q_N=\dfrac{\lVert(1-T'(u))^{-1}P_N(T(u_N)-T(u)-T'(u)(u_N-u))\rVert}{\lVert u_N-u\rVert} goes to zero, as NN goes to ∞\infty from the results of [24, 25].

We have, as noted above, ∥PNu−u∥⟶0\lVert P_Nu-u\rVert\longrightarrow0 as N⟶0N \longrightarrow0. In addition we have ∥(PN−1)T′(u)∥⟶0\lVert(P_N-1)T'(u)\rVert\longrightarrow0 over DD and we can bound ∥PN∥\lVert P_N\rVert uniformly over DD. We therefore obtain

∥uN−u∥≤11−∥(PN−1)T′(u)∥−∥PN∥qN∥PNu−u∥,\lVert u_N-u\rVert\le\frac{1}{1-\lVert(P_N-1)T'(u)\rVert-\lVert P_N\rVert q_N}\lVert P_Nu-u\rVert,

where ∥(PN−1)T′(u)∥⟶0\lVert(P_N-1)T'(u)\rVert\longrightarrow0 and ∥PN∥qN⟶0\lVert P_N\rVert q_N \longrightarrow0, and therefore for NN large enough this inequality makes sense. We obtain that ∥uN−u∥\lVert u_N-u\rVert goes to zero with at least the same speed as ∥PNu−u∥\lVert P_Nu-u\rVert, whose convergence to zero depends on kk and β\beta, as shown above. We can also get a similar lower bound following an analogous procedure, showing that the speed with which ∥uN−u∥\lVert u_N-u\rVert converges to zero is the same as the speed with which ∥PNu−u∥\lVert P_Nu-u\rVert converges to zero. □

Remark 3.6. Observe that since ∥PN∥⟶∞\lVert P_N\rVert\longrightarrow\infty as N⟶∞N \longrightarrow\infty when PNP_N is applied on functions that are not regular enough, e.g. outside of RR, the previous result does not give that ∥u−uN∥⟶0\lVert u-u_N\rVert\longrightarrow0 always. In fact, from the Uniform Boundedness Principle, we know that there must be functions for which ∥u−PNu∥\lVert u-P_Nu\rVert does not converge to zero. In such cases, the same procedure above cannot be applied. This is somehow inconvenient, but not unexpected. In fact, the Chebyshev expansion is known to not always converge. However, it converges for functions that are regular enough, which is the most important case in applications, especially in physics and engineering.

Remark 3.7. The result above is local in nature, in the sense that we are considering a ball DD in the space Ck,βC^{k,\beta}. The ball has arbitrary radius, as long that it contains the solution uu to the integral equation, which is not a very strong condition. When DD contains more than a single fixed point, one can restrict the ball around a fixed point that does not contain other fixed points (using the assumption that a solution to the equation is isolated).

There is a consideration regarding the method of Section 2 in relation to the result of Theorem 3.5. In fact, as in any collocation method, the spectral approach followed in this article produces solutions at the collocation points {ti}\{t_i\}, while in Theorem 3.5 the functions uNu_N are assumed to be solutions for all tt (i.e. not only at the collocation points). We momentarily leave aside the further issue that we only obtain approximate solutions at the collocation points. As a consequence, we can consider that for each NN we obtain an interpolation of the function uNu_N. However, we observe that it is known that for a function ξ\xi which is N+1N+1 times continuously differentiable, the error for a Chebyshev interpolation ΠN\Pi_N of degree N+1N+1 is bounded as

∥ξ−ΠN∥∞≤KN2N(N+1)!,\lVert\xi-\Pi_N\rVert_\infty\leq\frac{K_N}{2^N(N+1)!},

where KN=max⁡[−1,1]∣ξN+1∣K_N=\max_{[-1,1]}|\xi^{N+1}|, and the points tit_i of interpolation are assumed to be the Chebyshev nodes. As a consequence, we can obtain solutions uNu_N with arbitrarily high precision through the described collocation method. We point out that, in general, a dataset might require points tit_i that do not coincide with the Chebyshev nodes, in which case the polynomial interpolation would not be “optimal”. We show that our algorithm still works in such situations with good accuracy by means of experimentation. See Section 5, where the data points are not assumed to be at the Chebyshev nodes.

We now turn to Question 3.3. We assume here that the operator T:X⟶XT:X\longrightarrow X is defined through a kernel GG that is continuous in all three variables yy, tt and ss, and it is therefore a continuous operator with respect to the uniform norm. Here XX is a space of continuous functions depending on the problem (e.g. continuous functions with some fixed boundary in [−1,1][-1,1]).

Lemma 3.8. With notation and assumptions as above, let TT denote a Fredholm or Volterra integral operator, and let PNT:XN⟶XNP_NT:X_N\longrightarrow X_N be its projection on XNX_N. Then, for any fixed ϵ>0\epsilon>0, there exist neural networks FN,gN:XN⟶XNF_N,g_N:X_N\longrightarrow X_N that satisfy the condition

∥PNT(y)−PNJFN(y)−gN(y)∥∞<ϵ,\lVert P_NT(y)-P_N\mathcal{J}F_N(y)-g_N(y)\rVert_\infty<\epsilon,

for all y∈XNy\in X_N, where J\mathcal{J} denotes the spectral integration in Fredholm or Volterra form (depending on TT). Moreover, PNJFNP_N\mathcal{J}F_N approximates with arbitrarily high precision the operator defined through:

PN∫−1α(t)PNG(y,t,s) ds.P_N\int_{-1}^{\alpha(t)}P_NG(y,t,s)\,ds.

Proof. We focus on the case of Fredholm operators, but the reasoning given here can be applied also to the case of Volterra operators with minor modifications.

Since y∈XNy\in X_N, it follows that any such function can be expressed as a linear combination y(s)=∑i=0NaiCi(s)y(s)=\sum_{i=0}^{N}a_iC_i(s). Therefore, the operator T(y)T(y) induces a function T:RN+1×[−1,1]⟶R\mathcal{T}:\mathbb{R}^{N+1}\times[-1,1]\longrightarrow\mathbb{R} through the assignment

(a0,…,aN,t)↦T(∑i=0NaiCi)(t).(a_0,\ldots,a_N,t)\mapsto T\left(\sum_{i=0}^{N}a_iC_i\right)(t).

Since TT is continuous with respect to the uniform norms, we can show that T\mathcal{T} is continuous as well, where the domain is endowed with the standard Euclidean norm. In fact, for a tuple (a0,1,…,aN,1,t1)∈RN+1×[−1,1](a_{0,1},\ldots,a_{N,1},t_1)\in\mathbb{R}^{N+1}\times[-1,1], and an arbitrary choice of ϵ>0\epsilon>0, from the uniform continuity of T\mathcal{T} we can find δ′\delta' such that max⁡s∣y1(s)−y2(s)∣<δ′\max_s |y_1(s)-y_2(s)| < \delta' implies that max⁡t∣T(y1)(t)−T(y2)(t)∣<ϵ/2\max_t |\mathcal{T}(y_1)(t)-\mathcal{T}(y_2)(t)| < \epsilon/2, where yk:=∑iai,kCiy_k := \sum_i a_{i,k}C_i, for k=1,2k=1,2. Then, taking a ball B=Bδ′N+1(a0,1,…,aN,1,t1)B = B_{\frac{\delta'}{N+1}}(a_{0,1},\ldots,a_{N,1},t_1) of radius δ′N+1\frac{\delta'}{N+1} centered in (a0,1,…,aN,1,t1)(a_{0,1},\ldots,a_{N,1},t_1) we have that ∣y1(s)−y2(s)∣=∣∑i(ai,1−ai,2)Ci(s)∣≤∑i∣ai,1−ai,2∣<δ′|y_1(s)-y_2(s)| = |\sum_i(a_{i,1}-a_{i,2})C_i(s)| \leq\sum_i|a_{i,1}-a_{i,2}| < \delta', whenever (a0,2,…,aN,2,t2)(a_{0,2},\ldots,a_{N,2},t_2) is taken in BB. This in turn implies that ∣T(y1)(t)−T(y2)(t)∣<ϵ/2|\mathcal{T}(y_1)(t)-\mathcal{T}(y_2)(t)| < \epsilon/2 for all t∈[−1,1]t \in[-1,1]. Since T(y1)\mathcal{T}(y_1) is a continuous function with respect to tt, we can find δ′′\delta'' such that ∣T(y1)(t1)−T(y1)(t2)∣<ϵ/2|\mathcal{T}(y_1)(t_1)-\mathcal{T}(y_1)(t_2)| < \epsilon/2 whenever ∣t1−t2∣<ϵ/2|t_1-t_2| < \epsilon/2. Choosing δ=min⁡{δ′N+1,δ′′}\delta= \min\{\frac{\delta'}{N+1},\delta''\}, we have

∣T(y1)(t1)−T(y2)(t2)∣=∣T(y1)(t1)−T(y1)(t2)+T(y1)(t2)−T(y2)(t2)∣≤∣T(y1)(t1)−T(y1)(t2)∣+∣T(y1)(t2)−T(y2)(t2)∣<ϵ/2+ϵ/2=ϵ,\begin{aligned} |\mathcal{T}(y_1)(t_1)-\mathcal{T}(y_2)(t_2)| &=|\mathcal{T}(y_1)(t_1)-\mathcal{T}(y_1)(t_2)+\mathcal{T}(y_1)(t_2)-\mathcal{T}(y_2)(t_2)|\\ &\leq|\mathcal{T}(y_1)(t_1)-\mathcal{T}(y_1)(t_2)|+|\mathcal{T}(y_1)(t_2)-\mathcal{T}(y_2)(t_2)|\\ &<\epsilon/2+\epsilon/2=\epsilon, \end{aligned}

whenever (a0,2,…,aN,2,t2)∈Bδ(a0,1,…,aN,1,t1)(a_{0,2},\ldots,a_{N,2},t_2) \in B_\delta(a_{0,1},\ldots,a_{N,1},t_1). Using the definition of T\mathcal{T}, this implies that T\mathcal{T} is continuous as claimed.

Now, using the spectral integration (Section 2), we can write T(y)(t)\mathcal{T}(y)(t) by first decomposing G(y,t,s)G(y,t,s) in a Chebyshev series ∑ibi(t)Ci\sum_i b_i(t)C_i for any tt, and then integrating it through the mapping I\mathcal{I}. Since GG depends both on y∈XNy \in X_N and tt, the coefficients bib_i are functions of the coefficients aia_i that uniquely determine yy, and the variable tt, i.e. they are of the form bi(a0,…,aN,t)b_i(a_0,\ldots,a_N,t). We want to show now that they are continuous functions RN+1×[−1,1]⟶R\mathbb{R}^{N+1}\times[-1,1]\longrightarrow\mathbb{R}. To do so, first recall that the Chebyshev coefficients of G(y,t,s)G(y,t,s) (for fixed tt) are found through the formula

bk=2π∫−11G(y,t,s)Ck(s)1−s2 ds.b_k=\frac{2}{\pi}\int_{-1}^{1}\frac{G(y,t,s)C_k(s)}{\sqrt{1-s^2}}\,\mathrm{d}s.

As seen above, for all s∈[−1,1]s \in[-1,1] we have ∣y1(s)−y2(s)∣≤∑i∣ai,1−ai,2∣|y_1(s)-y_2(s)|\leq\sum_i|a_{i,1}-a_{i,2}| for functions in XNX_N, from which we have that using the continuity of GG, for any choice of ϵ>0\epsilon>0, we can find δ>0\delta>0 small enough such that ∣G(y1(s),t1,s)−G(y2(s),t2,s)∣<ϵ/2|G(y_1(s),t_1,s)-G(y_2(s),t_2,s)|<\epsilon/2 whenever (a0,2,…,aN,2,t2)∈Bδ(a0,1,…,aN,1,t1)(a_{0,2},\ldots,a_{N,2},t_2)\in B_\delta(a_{0,1},\ldots,a_{N,1},t_1). Now, we have

∣bk(a0,1,…,aN,1,t1)−bk(a0,2,…,aN,2,t2)∣=∣2π∫−11{G(y1(s),t1,s)−G(y2(s),t2,s)}Ck(s)1−s2 ds∣≤2π∫−11∣G(y1(s),t1,s)−G(y2(s),t2,s)∣∣Ck(s)∣1−s2 ds<2πϵ2∫−1111−s2 ds=ϵ,\begin{aligned} |b_k(a_{0,1},\ldots,a_{N,1},t_1)-b_k(a_{0,2},\ldots,a_{N,2},t_2)| &=\left|\frac{2}{\pi}\int_{-1}^{1}\{G(y_1(s),t_1,s)-G(y_2(s),t_2,s)\}\frac{C_k(s)}{\sqrt{1-s^2}}\,\mathrm{d}s\right|\\ &\leq\frac{2}{\pi}\int_{-1}^{1}|G(y_1(s),t_1,s)-G(y_2(s),t_2,s)|\frac{|C_k(s)|}{\sqrt{1-s^2}}\,\mathrm{d}s\\ &<\frac{2}{\pi}\frac{\epsilon}{2}\int_{-1}^{1}\frac{1}{\sqrt{1-s^2}}\,\mathrm{d}s=\epsilon, \end{aligned}

whenever (a0,2,…,aN,2,t2)∈Bδ(a0,1,…,aN,1,t1)(a_{0,2},\ldots,a_{N,2},t_2)\in B_\delta(a_{0,1},\ldots,a_{N,1},t_1), showing the continuity of bk:RN+1×[−1,1]⟶Rb_k:\mathbb{R}^{N+1}\times[-1,1]\longrightarrow\mathbb{R} for any arbitrary kk.

Now, from the continuity of the coefficients bkb_k, it follows that we can find neural networks fkf_k of some depth and width [31] such that ∣bk−fk∣<ϵ2(N+1)|b_k-f_k|<\frac{\epsilon}{2(N+1)} on RN+1×[−1,1]\mathbb{R}^{N+1}\times[-1,1], for k=0,…,N+1k=0,\ldots,N+1. Since the spectral integration creates a linear combination of the coefficients bkb_k, we have that setting FN=∑ifiF_N=\sum_i f_i the inequality ∣IPNG−IFN∣<ϵ/2|\mathcal{I}P_NG-\mathcal{I}F_N|<\epsilon/2 holds on all RN+1×[−1,1]\mathbb{R}^{N+1}\times[-1,1], i.e. for all y∈XNy\in X_N. Since the bkb_k are continuous and T(y)(t)\mathcal{T}(y)(t) is continuous as well (and continuous with respect to aia_i and tt), we can find a neural network gN:RN+1×[−1,1]⟶Rg_N:\mathbb{R}^{N+1}\times[-1,1]\longrightarrow\mathbb{R} which satisfies ∣T(y)(t)−IPNG−gN∣<ϵ/2|\mathcal{T}(y)(t)-\mathcal{I}P_NG-g_N|<\epsilon/2 on RN+1×[−1,1]\mathbb{R}^{N+1}\times[-1,1] (again [31]). This means that on RN+1×[−1,1]\mathbb{R}^{N+1}\times[-1,1] we have ∥TN(y)−PNTFN(y)−gN(y)∥∞<ϵ\|\mathcal{T}N(y)-P_N\mathcal{T}F_N(y)-g_N(y)\|_\infty<\epsilon, and moreover FNF_N approximates the projected integral operator over XNX_N with arbitrary precision. To complete, now we apply the projector PNP_N and since this is continuous, the approximation properties found above are preserved (possibly upon choosing an ϵ′\epsilon' appropriately in the previous discussion). □\square

Remark 3.9. We can use other universal approximation results such as [18], instead of [31], to derive the approximation capabilities of spectral integral neural networks. Depending on the results used, some extra assumptions need to be imposed on FNF_N and gNg_N, but the gain is that simpler architectures might be used. For instance, in [18], one finds that a single layer neural network can locally (i.e. in some closed ball) approximate integral operators.

In the following we consider an operator TT which satisfies both the of Theorem 3.5, and of Lemma 3.8.

Theorem 3.10. Let T(y)+f=yT(y)+f=y be an integral equation that admits a unique solution u∈Ck,βu\in C^{k,\beta}. Then, for any ϵ>0\epsilon>0 we can find N>0N>0, and neural networks GG, gg and f~\tilde{f} such that if u~\tilde{u} is the solution of PNJG(y)+g(y)+f~=yP_N\mathcal{J}G(y)+g(y)+\tilde{f}=y which we assume to have solutions for NN large enough for any f~\tilde{f}, we have ∥u−u~∥∞<ϵ\lVert u-\tilde{u}\rVert_\infty<\epsilon.

Proof. For NN large enough, PNJG(y)+g(y)+f~=yP_N\mathcal{J}G(y)+g(y)+\tilde{f}=y has a solution u~N\tilde{u}_N and we can write

∥u−u~N∥≤∥u−uN∥+∥uN−u~N∥,\lVert u-\tilde{u}_N\rVert\leq\lVert u-u_N\rVert+\lVert u_N-\tilde{u}_N\rVert,

where uNu_N is the solution to the projected equation PNT(y)+PNf=yP_NT(y)+P_Nf=y as in the case of Theorem 3.5. Then, we can choose NN large enough such that ∥u−uN∥∞≤∥u−uN∥<ϵ/2\lVert u-u_N\rVert_\infty\leq\lVert u-u_N\rVert<\epsilon/2, since ∥u−uN∥⟶0\lVert u-u_N\rVert\longrightarrow0. From Lemma 3.8 we can find neural networks GG and gg such that

∥PNT(y)−PNJG(y)−g(y)∥∞<ϵ/4,\lVert P_NT(y)-P_N\mathcal{J}G(y)-g(y)\rVert_\infty<\epsilon/4,

for all yy in XNX_N, the projection space of continuous functions (in particular the projection of the Hölder space). Since ff is continuous, we can also choose (via the Universal Approximation Theorem) a neural network f~\tilde{f} such that

∥PNf−f~∥∞<ϵ/4.\lVert P_Nf-\tilde{f}\rVert_\infty<\epsilon/4.

Then, from the definition of uNu_N and u~N\tilde{u}_N we have

∥uN−u~N∥∞=∥PNT(uN)+PNf−PNJG(y)−g(y)−f~∥∞≤ϵ/4+ϵ/4.\lVert u_N-\tilde{u}_N\rVert_\infty=\lVert P_NT(u_N)+P_Nf-P_N\mathcal{J}G(y)-g(y)-\tilde{f}\rVert_\infty\leq\epsilon/4+\epsilon/4.

Therefore, we find that for NN large enough, ∥u−u~N∥∞<ϵ\lVert u-\tilde{u}_N\rVert_\infty<\epsilon, which completes the proof. □\square

We can also derive the following result, which shows that we can at least locally approximate continuous bounded operators in Hölder spaces through the methods considered in this article, where TT is as above.

Theorem 3.11. Let TT be an integral operator (Fredholm or Volterra) defined on the Hölder space Ck,βC^{k,\beta}. Then, for any choice of ϵ>0\epsilon>0, we can find N>0N>0 and neural networks GG and gg for which

∥T(y)−PNJG(y)−g(y)∥∞<ϵ,\lVert T(y)-P_N\mathcal{J}G(y)-g(y)\rVert_\infty<\epsilon,

for all y∈B(0,r)y\in B(0,r), where B(0,r)B(0,r) is an arbitrary radius-rr ball centered at zero in Ck,βC^{k,\beta}.

Proof. For any yy and any choice of neural networks GG and gg we use the projection PNP_N to obtain

∥T(y)−PNJG(y)−g(y)∥∞=∥T(y)−PNT(y)+PNT(y)−PNJG(y)−g(y)∥∞≤∥T(y)−PNT(y)∥∞+∥PNT(y)−PNJG(y)−g(y)∥∞≤Mk(y)Nk+β+∥PNT(y)−PNJG(y)−g(y)∥∞.\begin{aligned} \lVert T(y)-P_N\mathcal{J}G(y)-g(y)\rVert_\infty &=\lVert T(y)-P_NT(y)+P_NT(y)-P_N\mathcal{J}G(y)-g(y)\rVert_\infty\\ &\leq\lVert T(y)-P_NT(y)\rVert_\infty+\lVert P_NT(y)-P_N\mathcal{J}G(y)-g(y)\rVert_\infty\\ &\leq\frac{M_k(y)}{N^{k+\beta}}+\lVert P_NT(y)-P_N\mathcal{J}G(y)-g(y)\rVert_\infty. \end{aligned}

Let us now choose an arbitrary rr and select NN such that rNk+β<ϵ/2\frac{r}{N^{k+\beta}}<\epsilon/2. Now, from Lemma 3.8 we can find GG and gg such that ∥PNT(y)−PNJG(y)−g(y)∥∞<ϵ/2\left\|P_NT(y)-P_N\mathcal{J}G(y)-g(y)\right\|_\infty<\epsilon/2, which completes the proof. □\square

Remark 3.12. A generalization of the previous result to Sobolev spaces, using the results of [9], would be of theoretical interest.

Remark 3.13. Lastly, we observe that in several cases in practice, one is interested in studying the equation

λT(y)+f=y,\lambda T(y)+f=y,

where λ\lambda is a nonzero parameter which is useful to determine When λ\lambda is small enough, the existence and uniqueness of the solutions of the integral equation, e.g. through fixed point iterations, is guaranteed. See [24], Chapter 3, for examples. This is substantially equivalent to an eigenvalue problem for nonlinear operators (for nonzero eigenvalues), where the definition of eigenvalue of TT is defined in the “naive” way as in [24]. See [7] for an overview of spectral theory for nonlinear operators.

Regarding Question 3.4, an approach similar to the one pursued in [41] can be applied here to show that the gradient descent method can be applied to the architecture proposed in this article to obtain the neural networks GG and gg. However, we do not adapt those methods to our case, but rather content ourselves with showing that gradient descend is possible by means of experimentation.

Algorithm

The considerations up to now do not take into account the fact that the approximated (projected) equation needs to be solved using some numerical scheme. Therefore, the theoretical bounds obtained in § 3 assume an exact solution of the projected equation. This is obviously not the case in practice, and we should introduce some numerical solver scheme to obtain the solutions uNu_N.

Our algorithm, therefore, is based on projecting the equation as described in § 3, and then solving the corresponding equation. Gradient descent is performed on the error between the solution obtained, and the target function. The function ff is used for inizialization of the solver, and it is obtained from the available data. For instance, if the problem is to predict a dynamics from the initial 5 time points, these 5 points are used to obtain the function ff, and solving the projected integral equation gives the predicted (approximated) dynamics.

We use the Chebyshev coefficients as inputs of our neural networks, and peform integration on the output function (identified with its coefficients) through the spectral integration J\mathcal{J} described in § 2. To solve the projected equation, we use a fixed point iteration in the projected coefficient space. While the fixed point iteration is guaranteed to converge under the more restrictive assumptions of contractivity of the operator, we see that in practice such convergence is achieved in our experiments. In more delicate cases, one might introduce extra constraints to guarantee the convergence of the iterations. The method is schematically described in Algorithm 1.

It is possible to extend the algorithm to a higher dimensional version by considering a tensor product of Chebyshev grid points. This procedure has been widely employed in practice in engineering, see e.g. [17] for an example. However, the resulting product polynomials do not satisfy the optimality guarantees of the 1D Chebyshev polynomials. Therefore, while the algorithm extends, mutatis mutandis, to a higher dimensional version, the theoretical analysis does not proceed in a straightforward manner. A more formal approach follows the work of Kowalski [22, 23] and Xu [36, 37]. In fact, they provide a way of constructing families of multivariate orthogonal polynomials Algorithm 1 Algorithm for the spectral neural integral equation model.

Require: Integrand neural network GθG_\theta and ff ▹\triangleright Initial model and function ff obtained from the available data

Ensure: Trained neural network GθG_\theta ▹\triangleright Solution of (5) projected in NN-dimensional space

  1. Project dataset on NN-dimensional space XNX_N using PN=∑k=0N−12π∫−11F(s)Ck(s)1−s2 ds⋅CkP_N = \sum_{k=0}^{N-1} \frac{2}{\pi} \int_{-1}^{1} \frac{F(s)C_k(s)}{\sqrt{1-s^2}} \, \mathrm{d}s \cdot C_k

  2. Initialize iterations with function u0:=fu_0 := f

  3. while While ∥uk−uk+1∥>τ\lVert u_k-u_{k+1}\rVert> \tau do ▹\triangleright Solve (5) iteratively in XNX_N

  4. uk+1=IGθ(uk)+fu_{k+1} = \mathcal{I}G_\theta(u_k) + f

  5. end while

  6. Solution uNu_N ▹\triangleright Output of the iterative procedure

  7. Convert uNu_N to an approximate solution in original space, yNy_N ▹\triangleright Use Chebyshev interpolation

  8. Compute ∥yN−u∥\lVert y_N-u\rVert ▹\triangleright Evaluate the error between yNy_N and data uu

  9. Use gradients computed while solving (5) to perform gradient descent to optimize GθG_\theta

that can serve as the basis for projection, playing the same role as the Chebyshev polynomials in this article for an analogous algorithm in higher dimensions.

Remark 4.1. It is possible to choose the tolerance parameter τ\tau in Algorithm 1 based on the Chebyshev interpolation. The parameter τ\tau controls the error in the coefficients of the Chebyshev basis to represent a target function. Therefore, relating the error on the coefficients to the interpolation error allows to meaningfully fix τ\tau theoretically. However, we point out that it is usually enough to set τ\tau based on empirical observation, and fix it as a hyperparameter.

Experiments

We showcase the algorithm described in § 4 on two classes of experiments. We consider a dataset consisting of solutions of an integral equation solved numerically. The kernel of the integral operator consists of matrices with hyperbolic functions entries in the variables tt and ss, where the latter is the variable of integration.Moreover, in order to test the model’s stability with respect to noise, the original integral equation solutions are perturbed by adding gaussian noise. The second dataset is a simulated fMRI dataset generated by solving a delayed ODE system that describes the behavior of neurons in the brain.

We show that the model can predict the dynamics, and show that the memory footprint of the model is very low, considering that it is solving an integral equation for each epoch. Moreover, we perform interpolation experiments to demonstrate that the model is very stable with respect to variations in the temporal domain of train and evaluation. We have chosen datasets with nonlocal behaviors, like integral equations and delay differential equations, to explore the capability of the model to model long range temporal dependencies in the datasets.

Integral Equations Dataset

In this experiment, the model is initialized using two points for each dynamics. The task is to predict the whole noisy dynamics, which consists of 100 time points. To test the capability of the model to interpolate, we train our Spectral NIE on a downsampled dataset that contains half of the points of the curves. During evaluation, the model then outputs 100 points, even though it has been trained on 50 time points alone. Convergence of the model is considered to be 200 epochs with no improvement on the validation set. The walltime of the experiment is the time elapsed between the beginning of the training to the convergence, as per the aforementioned criterion. An upper bound of one hour training has been fixed, and models that did not converge within this time are reported as having walltime >3600> 3600.

In Table 1 we have reported two different Spectral NIE models, a large model with more than 100K parameters, and a small model with only 316 parameters. For both models, the MCMC represents the number of points used to perform the Monte Carlo integration used in the interpolation. We see that increasing MCMC by a 10-fold factor we obtain a faster convergence time, but higher memory footprint.

ModelsMemory (MiB)Walltime (sec)Error (MSE)Interp Error (MSE)Parameters
Spectral NIE (MC = 1K)74165.210.0022 ± 0.00160.0028 ± 0.0032126498
Spectral NIE (MC = 10K)324408.040.0032 ± 0.00220.0035 ± 0.0024316
Spectral NIE (MC = 1K)68546.500.0025 ± 0.00130.0026 ± 0.0014316
ANIE (Large)122564.940.0014 ± 0.00020.0015 ± 0.0003135363
ANIE (Small)441111.740.0062 ± 0.00090.01175 ± 0.0017531
ResNet42305.470.4269 ± 0.05880.2619 ± 0.034330802
Latent ODE40>3600> 36000.6342 ± 0.28380.6290 ± 0.28131428
LSTM (init 10)102262.900.1785 ± 0.0942NA41802
LSTM (init 20)122864.660.0492 ± 0.01100.1707 ± 0.128541802
FNO1D (init 5)258>3600> 36000.0292 ± 0.02850.0994 ± 0.1207287843
FNO1D (init 10)2503488.400.0286 ± 0.02680.1541 ± 0.2724288163

Table 1. Experimental results for the Integral Equations Dataset. Both errors are reported as mean squared errors.

For the other models, we have also tested ANIE (which is an implementation of the original NIE of [39, 40]) with two models, a large one and a small one, with number of parameters comparable to the corresponding Spectral NIE models. We find that for small models, the Spectral NIE model has a significantly better performance than ANIE, with smaller walltime and one order of magnitude better interpolation. For the larger models, we observe that Spectral NIE has better memory footprint and better walltime, but ANIE has a slightly better overall performance in terms of error and interpolation error. Moreover, we see that the memory footprint of the Spectral NIE grows slowly with respect to the size of the model, since most of the computational expense is due to the Monte Carlo integration used for interpolation, and therefore independent on the model itself. ANIE’s complexity, however, grows faster with respect to the size of the model.

Among the other models that we have tested for comparison, we also have ResNet, LSTM, Nerual Ordinary Differential Equation (its newer Latent ODE variant), and Fourier Neural Operator (FNO). No surprisingly, both the integral equation based models perform significantly better than these models, since they either do not implement nonlocal dynamics or, as in the case of LSTM, their nonlocality properties are limited.

In Table 1, Error refers to the mean squared error of the predicted dynamics with respect to the real dynamics when trained over the full time interval (no downsampling). Interpolation error refers to the setup where the dataset is downsampled for training, and the prediction of the model is performed over the full time interval. The experiment shows that the model is quite stable with respect to changes of time interval between train and evaluation. We think that such stability is due to the use of spectral methods, which automatically give a set of polynomials that can be evaluated on arbitrary points, resulting in good interpolation results. All experiments reported in Table 1 were performed on an Nvidia RTX 3090 GPU.

We report results from a more extensive study of the computational trade-offs between Monte Carlo sampling, convergence walltime, and model accuracy. The experiments were performed on a CPU using a medium-sized model with 67, 934 parameters, and the results are presented in Table 2. We observe that although larger Monte Carlo sampling initially improves accuracy, the gains quickly saturate as sampling increases, while memory usage continues to grow. Moreover, runtime convergence also increases with larger sampling, since each epoch requires more computation time.

MCMemory(MiB)Walltime (sec)Error (MSE)
500866650.0020±0.00120.0020 \pm0.0012
10001337440.0021±0.00120.0021 \pm0.0012
20002356010.0018±0.00100.0018 \pm0.0010
40004298490.0018±0.00090.0018 \pm0.0009
80008157390.0019±0.00100.0019 \pm0.0010

Table 2. Experimental results for the Integral Equations Dataset on computational trade-off.

To further determine the stability of the model with respect to irregularly sampled data, we consider an experimental setup where we irregularly sample the trajectories to 70%, 60%, 50%, 40%, 30%, and 20% of their original size during training, and predict the original curve during evaluation. Therefore, this is another interpolation test where the model accesses irregular time stamps of the data. The full dynamics is never seen during training by the model, as it is trained to fit only the available downsampled data. Effectively, the interpolation component of evaluation is a generalization task. When the percentage of sampled data becomes too low, the error increases significantly, which is an expected behavior considering that the task becomes increasingly harder, and for sufficiently low available data the theoretical optimal interpolation error bounds increase. The model shows good interpolation capabilities with respect to irregular downsampling of the data, until the downsampling grows excessively. The results of the experiment are reported in Table 3.

70%60%50%40%30%20%
0.0022±0.00180.0022 \pm0.00180.0022±0.00180.0022 \pm0.00180.022±0.00180.022 \pm0.00180.0022±0.00140.0022 \pm0.00140.0022±0.00150.0022 \pm0.00150.0167±0.00690.0167 \pm0.0069

Table 3. Experimental results for the Integral Equations Dataset on irregularly sampled interpolation task.

Simulated fMRI Dataset

We consider now a simulated fMRI dataset. This is generated using the package neurolib [8] through a system of delay differential equations that simulate the stimulus of brain regions. The dynamics is 80 dimensional, and each dimension refers to a brain region. Moreover, the system has a high degree of nonlocality.

We consider different types of initializations, where we assume that 3, 4 or 7 time points are available for initialization of the model. Clearly, as the number of available points for initialization increases, the models’ predictions become more accurate, since the task is simpler.

Table 4 shows the results of the experiment, where we have compared the Spectral NIE to ANIE, NODE, LSTM and FNO1D. In all experiments, the number of parameters of the model are comparable, to ensure a more meaningful comparison. Once again, since the dynamics is highly nonlocal, we expect that integral equation based models perform better. This is indeed the case, as shown in the table. Moreover, we see that Spectral NIE performs significantly better than ANIE in this experiment, and the gain in using our spectral approach is not only in computational cost and convergence time, but also in accuracy.

N initSpectral NIEANIENODELSTMFNO1D
30.0027 ±\pm 0.00060.0090 ±\pm 0.00920.1189 ±\pm 0.04370.0871 ±\pm 0.01460.0794 ±\pm 0.0150
40.0025 ±\pm 0.00070.0096 ±\pm 0.00970.1102 ±\pm 0.02430.0819 ±\pm 0.01450.0726 ±\pm 0.0130
70.0012 ±\pm 0.00060.0069 ±\pm 0.01130.1138 ±\pm 0.03020.0810 ±\pm 0.01360.0576 ±\pm 0.0100

Table 4. Experimental results for fMRI simulated dataset: all reported values refer to mean squared error. The models are initialized with varying amount of time points.

Example predictions for the Spectral NIE for 3, 4 and 7 initialization points are given in Figure 1, Figure 2 and Figure 3, respectively. In all figures, the top dynamics represents the prediction, and the bottom dynamics represents the ground truth. The x⃗\vec{x}-axis represents all the 80 brain locations, while the y⃗\vec{y}-axis refers to the 20 time points per dynamics.

Example prediction for Spectral NIE on fMRI Dataset with 7 time points used for initialization. Top: Prediction. Middle: Ground truth. Bottom: Error.

Figure 1. Example prediction for Spectral NIE on fMRI Dataset with 7 time points used for initialization. Top: Prediction. Middle: Ground truth. Bottom: Error.

Example prediction for Spectral NIE on fMRI Dataset with 4 time points used for initialization

Figure 2. Example prediction for Spectral NIE on fMRI Dataset with 4 time points used for initialization. Top: Prediction. Middle: Ground truth. Bottom: Error.

Example prediction for Spectral NIE on fMRI Dataset with 7 time points used for initialization

Figure 3. Example prediction for Spectral NIE on fMRI Dataset with 7 time points used for initialization. Top: Prediction. Middle: Ground truth. Bottom: Error.

We further investigate the runtime-accuracy trade-off on the fMRI dataset, with results summarized in Table 5. All experiments were run on a CPU using a model with 348, 648 parameters. We observe that walltime convergence decreases as the number of Monte Carlo samples increases, before stabilizing. This behavior appears to stem from improved model stability, which allows convergence in fewer training epochs. However, each epoch becomes slower due to the higher computational cost. Overall, accuracy improves markedly with larger Monte Carlo samples.

MCMemory(MiB)Walltime (sec)Error (MSE)
5004894310.0021 ±\pm 0.0009
10005034050.0017 ±\pm 0.0008
20005253690.0015 ±\pm 0.0009
40005913600.0013 ±\pm 0.0006
80006023690.0012 ±\pm 0.0005

Table 5. Experimental results for the fMRI Dataset on computational trade-off.

Conclusions

We conclude by providing some potential applications of the framework introduced in this article. A natural class of problems that this work can be applied to, regards neuroscience. EEG and fMRI have signals that are highly nonlocal, and therefore modeling the corresponding operators in a nonlocal manner, as with integral equations, is meaningful. While fMRI is a higher dimensional method, with 2 or 3 spatial dimensions and a temporal one, it is often the case that only the dynamics of certain brain areas are of interests, leading to a temporal dynamics with one channel per brain area, as in the fMRI dataset considered in this article. Similarly, problems in population dynamics where delay and long-range terms are not negligible lead to nonlocal equations that can be modeled via integral equations. Virus spread modling represents an example of such situation. Lastly, 1D Boltzmann equations, or nonlocal diffusion problems where one dimension dominates the equation term are a class of examples of problems in physics that can be treated using the framework in this article.

Acknowledgement

The author acknowledges support from the NIH under the grant R16GM154734.

Department of Mathematics and Statistics, Idaho State University, Physical Science Complex — 921 S. 8th Ave., Stop 8085 — Pocatello, ID 83209

Email address: [email protected]

References

  1. [1]FT Adler, Reactor kinetics: integral equation formulation, Journal of Nuclear Energy. Parts A/B. Reactor Science and Technology 15 (1961), no. 2-3, 81–85.DOI
  2. [2]Shun-ichi Amari, Dynamics of pattern formation in lateral-inhibition type neural fields, Biological cybernetics 27 (1977), no. 2, 77–87.pubmed.ncbi.nlm.nih.gov/911931
  3. [3]John D Anderson, Governing equations of fluid dynamics, Computational fluid dynamics: an introduction (1992), 15–51.
  4. [5]Kendall Atkinson and Weimin Han, Theoretical numerical analysis, Vol. 39, Springer, 2005.
  5. [6]Kendall E Atkinson and Florian A Potra, Projection and iterated projection methods for nonlinear integral equations, SIAM journal on numerical analysis 24 (1987), no. 6, 1352–1373.DOI
  6. [7]Alessandro Calamai, Massimo Furi, and Alfonso Vignoli, An overview on spectral theory for nonlinear operators, Communications in Applied Analysis 13 (2009), no. 4, 509–534.
  7. [8]Caglar Cakan, Nikola Jajcay, and Klaus Obermayer, neurolib: a simulation framework for whole-brain neural mass modeling, Cognitive Computation (2021), 1–21.DOI
  8. [9]Claudio Canuto and Alfio Quarteroni, Approximation results for orthogonal polynomials in Sobolev spaces, Mathematics of Computation 38 (1982), no. 157, 67–86.DOI
  9. [10]Salvatore Capozziello and Francesco Bajardi, Nonlocal gravity cosmology: An overview, International Journal of Modern Physics D 31 (2022), no. 06, 2320009.DOI
  10. [12]Odo Diekmann, On a nonlinear integral equation arising in mathematical epidemiology, North-Holland Mathematics Studies, 1978, pp. 133–140.
  11. [13]David Gottlieb and Steven A Orszag, Numerical analysis of spectral methods: theory and applications, SIAM, 1977.
  12. [14]Leslie Greengard, Spectral integration and two-point boundary value problems, SIAM Journal on Numerical Analysis 28 (1991), no. 4, 1071–1080.
  13. [15]Gaurav Gupta, Xiongye Xiao, and Paul Bogdan, Multiwavelet-based operator learning for differential equations, Advances in neural information processing systems 34 (2021), 24048–24062.arxiv.org/abs/2109.13459
  14. [16]Zhongkai Hao, Zhengyi Wang, Hang Su, Chengyang Ying, Yinpeng Dong, Songming Liu, Ze Cheng, Jian Song, and Jun Zhu, Gnot: A general neural operator transformer for operator learning, International Conference on Machine Learning, 2023, pp. 12556–12569.arxiv.org/abs/2302.14376
  15. [17]Keshan He, Pengbi Du, Hao Jiang, Chongwen Duan, Hongxia Wang, and Lizhi Cheng, Accurate and Efficient Evaluation of Chebyshev Tensor Product Surface, Mathematical Problems in Engineering 2017 (2017), no. 1, 1729287.DOI
  16. [18]Kurt Hornik, Maxwell Stinchcombe, and Halbert White, Multilayer feedforward networks are universal approximators, Neural networks 2 (1989), no. 5, 359–366.DOI
  17. [19]NI Ioakimidis, A simple proof of Kalandiya’s theorem in approximation theory, Serdica 9 (1983), 414–416.
  18. [20]Apollon Iosifovich Kalandiya, On a direct method of solution of an equation in wing theory with an application to the theory of elasticity, Matematicheskii Sbornik 84 (1957), no. 2, 249–272.
  19. [21]Nikola B Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew M Stuart, and Anima Anandkumar, Neural Operator: Learning Maps Between Function Spaces With Applications to PDEs., J. Mach. Learn. Res. 24 (2023), no. 89, 1–97.
  20. [22]MA Kowalski, Orthogonality and recursion formulas for polynomials in n variables, SIAM Journal on Mathematical Analysis 13 (1982), no. 2, 316–323.DOI
  21. [23]———, The recursion formulas for orthogonal polynomials in n variables, SIAM Journal on Mathematical Analysis 13 (1982), no. 2, 309–315.DOI
  22. [24]Yu P Krasnosel’skii, Topological methods in the theory of nonlinear integral equations, Pergamon Press (1964).
  23. [25]MA Krasnosel’skii and PP Zabreiko, Geometrical Methods of Nonlinear Analysis, Springer-Verlag, Berlin, 1984, MR 85b 47057.
  24. [27]Massimo Lanza de Cristoforis, Integral operators in Hölder spaces on upper Ahlfors regular sets, Atti Della Accademia Nazionale dei Lincei. Rendiconti Lincei. Matematica e Applicazioni 34 (2023), no. 1, 195–234.DOI
  25. [28]Chensen Lin, Zhen Li, Lu Lu, Shengze Cai, Martin Maxey, and George Em Karniadakis, Operator learning for predicting multiscale bubble growth dynamics, The Journal of Chemical Physics 154 (2021), no. 10.DOI
  26. [29]Yucheng Liu, Application of the Chebyshev polynomial in solving Fredholm integral equations, Mathematical and Computer Modelling 50 (2009), no. 3-4, 465–469.DOI
  27. [30]Lu Lu, Pengsheng Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis, Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, Nature Machine Intelligence 3 (2021), no. 3, 218–229.
  28. [31]Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang, The expressive power of neural networks: A view from the width, Advances in neural information processing systems 30 (2017).arxiv.org/abs/1709.02540
  29. [32]Christian Pötzsche, Urysohn and Hammerstein operators on Hölder spaces, Analysis 42 (2022), no. 4, 205–240.DOI
  30. [33]Peter Schiavone, Christian Constanda, and Andrew Mioduchowski, Integral methods in science and engineering, Springer Science & Business Media, 2002.DOI
  31. [34]Z Sedláček, Electrostatic oscillations in cold inhomogeneous plasma Part 2. Integral equation approach, Journal of Plasma Physics 6 (1971), no. 1, 187–199.DOI
  32. [35]Hugh R Wilson and Jack D Cowan, Excitatory and inhibitory interactions in localized populations of model neurons, Biophysical journal 12 (1972), no. 1, 1–24.pubmed.ncbi.nlm.nih.gov/4332108
  33. [36]Yuan Xu, On multivariate orthogonal polynomials, SIAM journal on mathematical analysis 24 (1993), no. 3, 783–794.DOI
  34. [37]———, Multivariate orthogonal polynomials and operator theory, Transactions of the American Mathematical Society 343 (1994), no. 1, 193–202.DOI
  35. [38]Changqing Yang, Chebyshev polynomial solution of nonlinear integral equations, Journal of the Franklin Institute 349 (2012), no. 3, 947–956.DOI
  36. [39]Emanuele Zappala, Antonio Henrique de Oliveira Fonseca, Josue Ortega Caro, and David van Dijk, Neural Integral Equations, arXiv:22109.15190 (2022).DOI
  37. [40]Emanuele and Fonseca Zappala Antonio Henrique de Oliveira and Caro, Learning integral operators via neural integral equations, Nature Machine Intelligence 6 (2024), no. 9, 1046–1062.DOI
  38. [41]Emanuele Zappala, Antonio H de O Fonseca, Andrew H Moberly, Michael J Higley, Chadi Abdallah, Jessica A Cardin, and David van Dijk, Neural integro-differential equations, Proceedings of the AAAI Conference on Artificial Intelligence, 2023, pp. 11104–11112.

Paper details

vlnc:W_eaj5meks
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents