Introduction
Neural operators, acting between infinite-dimensional function spaces, constitute a natural extension of classical finite neural networks. In recent years, they have attracted considerable attention in the field of machine learning due to their ability to enabling surrogate modeling of high-dimensional and complex physical systems. Such systems can be modeled by partial differential equations (PDEs) or controlled ordinary differential equations (CODEs), whose numerical solving may become time consuming, particularly when repeated evaluations are required. Using neural operators to replace the solution operator of a PDE or CODE, where the latter also acts as an operator between function spaces due to the presence of control functions, may work out this issue. Once a neural operator is learned, it allows for a faster computational inference time while also maintaining an architecture that enables the processing of batched data. This also makes them an encouraging tool in solving (Bayesian) inverse problems via the Ensemble Kalman Inversion, especially in cases where a large data set of multiple experiments is present.
Various different architectures to realize a neural operator were introduced over the last years. One of the most promising ones is the so-called Fourier neural operator (FNO) which employs the (inverse) Fourier transform inside the network’s layered structure. While Universal Approximation Theorems of the FNO are well studied and suggest that any continuous operator can be approximated, quantitative approximation studies show that FNO faces exponential growth in model complexity, the so-called curse of parametric complexity [34 ] , even when approximating C r C^r C r - or Lipschitz-regular operators. The curse of parametric complexity is an infinite-dimensional analogue to the well-known curse of dimensionality common in high-dimensional approximation problems.
Recent research suggests that this curse of parametric complexity can be overcome when approximating the solution operator of certain PDEs by emulating numerical methods like the Picard iteration. While current studies therefore expand the class of PDEs which allow for breaking the curse using a FNO, CODEs have not been studied in this context, although they play an important role in modeling physics while also being solvable through the Picard iteration. Therefore in this article, we focus on CODEs of form y ˙ = f ( t , y , u ) \dot{y}=f(t,y,u) y ˙ = f ( t , y , u ) , where f f f is a non-linear function and u u u are the Lebesgue-integrable control functions.
This article starts with introducing a system of CODEs under a Carathéodory setting, that is that the controls u u u do not need to be continuous, to determine appropriate mathematical assumptions to show the well-posedness of such a system while also deriving a continuous solution operator. This results in reformulating the CODE as a fixed point integral formula and thus, we follow the idea of emulating the Picard iteration by relying on the differentiation formula of the (discrete) Fourier transform to realize the integration of the controlled ordinary differential equation. Thus, the FNO constitutes a natural candidate for this approach. By sharpening our assumptions, which requires to assume more regularity on u u u and f f f , while for the latter also a smallness condition is needed, we break the curse by showing that the size of the FNO scales at most algebraically, and under a sufficiently smooth f f f even sub-quadratically up to logarithmic factors.
Our article is organized as follows:
In § 2 we introduce the controlled ordinary differential equation y ˙ = f ( t , y , u ) \dot{y}=f(t,y,u) y ˙ = f ( t , y , u ) under suitable assumptions, such that we obtain the existence of a continuous control-to-solution operator. As we are interested in deriving error bounds for approximating this solution operator with a Fourier neural operator, we start to sharpening the assumptions on the CODE such that we can apply tools from Fourier analysis.
The Fourier neural operator is being formally introduced and defined in § 3 . One essential key-point of the FNO is the usage of the (inverse) Fourier transform inside the network’s architecture. Since deriving error bounds requires to quantify the network’s size, we define the Ψ \Psi Ψ -FNO which relates to the numerical implementable version of a FNO. Afterwards, we state known quantitative approximation results regarding finite neural networks, as we need to approximate f f f by such networks. At the end of § 3 , we propose a Picard type fixed-point iteration scheme which produces an approximate output to the true solution of the CODE while also allowing to be emulated through the layers of Ψ \Psi Ψ -FNO.
In § 4 we establish the main result. Under the sharpened assumptions from § 2 , which include a smallness condition on f f f , we show that the size of the Ψ \Psi Ψ -FNO scales at most algebraically when approximating the control-to-solution operator. Thus, the size of the Ψ \Psi Ψ -FNO avoids the curse of parametric complexity.
To conclude the article, we present in § 5 the application of a learned FNO in an inverse parameter estimation problem motivated by a physical example. Due to the nature of the given example, it is the case that a large data set of multiple physical experiments is present. We therefore propose to utilize the learned FNO in an Ensemble Kalman Inversion which is extended to a Kaczmarz-like structure.
Related works
Neural operators [28 ] were first introduced in [3 ] by realizing them as graph kernel networks. By doing so, they offer a new approach to operator learning compared to other methods such as DeepONet [42 ] and PCA-Net [31 ] . Numerous neural operator architectures, such as the Fourier neural operator [39 ] , the convolutional neural operator [49 ] , the Laplace neural operator [11 ] , the orthogonal polynomial neural operator [41 ] and the Wavelet neural operators [63 ] , have been developed in recent years. Owing to their strong empirical approximation capabilities and fast inference times, they have established themselves as state-of-the-art tools for surrogate modeling physical simulations [64 ] , which also makes them interesting objects to study in the context of inverse problems [68 , 5 , 69 , 10 , 56 , 65 , 45 ] , due to the reliance on repeatedly evaluating the forward operator to numerically solve inverse problems. We refer the reader to [4 , 29 ] for an overview of neural operators and to [47 ] for neural operators in the context of inverse problems.
One of the most important theoretical foundations in the field of machine learning are the Universal Approximation Theorems, which state that for a given continuous operator and any accuracy level, there exists a neural operator capable of approximating the target operator over a compact set to any desired degree of accuracy. These theorems further justify the application of neural operators. Multiple Universal Approximation Theorems have been established for various kinds of neural operators [28 ] , including the FNO [27 , 32 ] , as well as for the DeepONet [33 ] and PCA-Net [31 ] .
Analogous to the case of finite-dimensional neural networks, neural operators suffer from the curse of parametric complexity, the infinite-dimensional analogue to the curse of dimensionality. This curse is an exponential growth in size with respect to an increasing accuracy, which is also present when approximating C r − C^{r-} C r − or Lipschitz-regular operators as discussed in [34 ] . In [27 ] , they overcome this curse when approximating the solution operator of the Darcy-type elliptic equation and the incompressible Navier–Stokes equations. The idea of the proof relies on recasting the underlying PDEs as fixed point problems which then allows for utilizing the Picard iteration. The latter is then realized by the layers of the investigated neural operator. The authors of [17 , 16 , 30 ] follow the same idea, leading to multiple neural operators that approximate solution operators of various PDEs that do not suffer from the curse of parametric complexity.
Although much encouraging work has been done to couple neural operators with ODEs [42 , 24 , 5 , 40 , 71 ] , no effort has yet been made to derive quantitative approximation rates for approximating the solution operator of a controlled ODE with a Fourier neural operator. Since neural operators usually act on function spaces, the object of interest in the case of ODEs is the setting when so-called control functions are present, which then act as the input functions for the neural operator. We therefore refer to such ODEs as controlled ordinary differential equations (CODEs) and highlight [59 , 12 ] for further reading on this topic. In this article, we derive error bounds for the Fourier neural operator when approximating the solution operator of CODEs. This is motivated by the fact that the differentiation formula of the Fourier transform suggests that the emulation of a CODE, when recast as a fixed point integral formula, is realizable through the layers of a FNO.
Such controlled systems can arise in practice, e.g., when controlling and modeling the process of electric arc furnaces [6 , 44 ] . To obtain meaningful predictions, simulations of these furnaces require inverse parameter estimation [50 , 43 , 2 ] . As electric arc furnaces are run under different control settings, their operation generates large data sets that can be used for parameter estimation. However, the long inference times of complex physical simulations can hinder a successful inversion. We therefore propose to apply the Ensemble Kalman Inversion (EKI) where one additionally replaces the simulation by a FNO surrogate. Interestingly the authors of [30 , 23 ] share the same idea of deploying the EKI with a learned neural operator. However, we extend this approach to a Kaczmarz structure, which is commonly encountered in deterministic inverse problems [36 , 51 , 70 , 37 ] . For further reading on EKI we refer the reader to [22 , 21 , 20 ] and for inverse problems we suggest [14 , 61 , 52 , 54 , 53 , 26 ] .
Our results and contributions
After introducing the CODE system in a Carathéodory setting, we give appropriate assumptions that lead to a well-defined operator acting between the spaces of Lebesgue integrable and continuous functions. The corresponding result is summarized in Theorem 2.1 . We then strengthen these assumptions to establish the setting for deriving error bounds when deploying the Fourier neural operator as a surrogate of the solution operator. Our main result Theorem 4.1 states that the size of the FNO scales at most algebraically and thus, exponential growth in the size of the FNO with respect to an increasing accuracy can be avoided. Under a sufficiently smooth f f f the size in fact grows at most sub-quadratically up to logarithmic factors. Proving this relies on a reformulation of the CODE as a fixed point integral formula, while utilizing numerous results from Fourier analysis. The non-linearity f f f in the CODE system can be efficiently approximated by a finite neural network. Consequently, the idea of our proof is similar to that of [17 ] , implying that other problems with a comparable structure can be resolved in the same manner. At the end of the paper, we introduce an inverse parameter estimation problem in which a set of measurement data from multiple experiments is present. We propose solving this problem through surrogate modeling and applying the Ensemble Kalman Inversion in a Kaczmarz-type fashion.
Notation
Let us briefly introduce the notation used throughout this paper.
We set N = { 1 , 2 , … } \mathbb{N}=\{1,2,\ldots\} N = { 1 , 2 , … } and N 0 = N ∪ { 0 } \mathbb{N}_0=\mathbb{N}\cup\{0\} N 0 = N ∪ { 0 } . Further we set R + = ( 0 , ∞ ) \mathbb{R}^{+}=(0,\infty) R + = ( 0 , ∞ ) .
Let T d = [ 0 , 2 π ] d \mathbb{T}^{d}=[0,2\pi]^{d} T d = [ 0 , 2 π ] d , d ∈ N d\in\mathbb{N} d ∈ N , be the periodic domain which we identify with the standard torus.
By ∥ ⋅ ∥ R n \|\cdot\|_{\mathbb{R}^{n}} ∥ ⋅ ∥ R n , n ∈ N n\in\mathbb{N} n ∈ N , we denote the euclidean norm.
For q ∈ [ 1 , ∞ ] q\in[1,\infty] q ∈ [ 1 , ∞ ] the Lebesgue space L q ( I ; R n ) L^{q}(I;\mathbb{R}^{n}) L q ( I ; R n ) consists of all measurable functions u : I → R n u:I\to\mathbb{R}^{n} u : I → R n such that
∥ u ∥ L q ( I ; R n ) : = { ( ∫ I ∥ u ( x ) ∥ R n q d x ) 1 / q , if q < ∞ , ess sup x ∈ I ∥ u ( x ) ∥ R n , if q = ∞ . \|u\|_{L^{q}(I;\mathbb{R}^{n})}:=
\begin{cases}
\left(\int_{I}\|u(x)\|_{\mathbb{R}^{n}}^{q}\,\mathrm{d}x\right)^{1/q}, & \text{if }q<\infty,\\
\underset{x\in I}{\operatorname{ess\,sup}}\|u(x)\|_{\mathbb{R}^{n}}, & \text{if }q=\infty.
\end{cases} ∥ u ∥ L q ( I ; R n ) := ⎩ ⎨ ⎧ ( ∫ I ∥ u ( x ) ∥ R n q d x ) 1/ q , x ∈ I ess sup ∥ u ( x ) ∥ R n , if q < ∞ , if q = ∞.
For s ∈ N 0 s \in\mathbb{N}_{0} s ∈ N 0 , p ∈ [ 1 , ∞ ] p \in[1,\infty] p ∈ [ 1 , ∞ ] the Sobolev space W s , p ( I ; R n ) W^{s,p}(I;\mathbb{R}^{n}) W s , p ( I ; R n ) is defined by
W s , p ( I ; R n ) : = { u ∈ L p ( I ; R n ) ∣ D α u ∈ L p ( I : R n ) for all α ∈ N 0 d with ∣ α ∣ ≤ s } , W^{s,p}(I;\mathbb{R}^{n}) := \{u \in L^{p}(I;\mathbb{R}^{n}) \mid D^{\alpha}u \in L^{p}(I:\mathbb{R}^{n}) \text{ for all } \alpha\in\mathbb{N}_{0}^{d} \text{ with } |\alpha| \le s\}, W s , p ( I ; R n ) := { u ∈ L p ( I ; R n ) ∣ D α u ∈ L p ( I : R n ) for all α ∈ N 0 d with ∣ α ∣ ≤ s } ,
where D α u D^{\alpha}u D α u denotes the weak derivative of order s s s and α \alpha α is a multi-index such that
α = ( α 1 , … , α d ) , D α u = ( ∂ α 1 ∂ α 1 x 1 ⋯ ∂ α d ∂ α d x d ) u , ∣ α ∣ = ∑ i = 1 d α i . \alpha= (\alpha_{1},\ldots,\alpha_{d}),\quad D^{\alpha}u = \left(\frac{\partial^{\alpha_{1}}}{\partial^{\alpha_{1}}x_{1}}\cdots\frac{\partial^{\alpha_{d}}}{\partial^{\alpha_{d}}x_{d}}\right)u,\quad|\alpha| = \sum_{i=1}^{d}\alpha_{i}. α = ( α 1 , … , α d ) , D α u = ( ∂ α 1 x 1 ∂ α 1 ⋯ ∂ α d x d ∂ α d ) u , ∣ α ∣ = i = 1 ∑ d α i .
The W s , p ( I ; R n ) W^{s,p}(I;\mathbb{R}^{n}) W s , p ( I ; R n ) -norm is given by
∥ u ∥ W s , p ( I ; R n ) : = { ( ∑ 0 ≤ ∣ α ∣ ≤ s ∥ D α u ∥ L p ( I ; R n ) p ) 1 / p , if p < ∞ , max 0 ≤ ∣ α ∣ ≤ s ∥ D α u ∥ L ∞ ( I ; R n ) , if p = ∞ . \lVert u\rVert_{W^{s,p}(I;\mathbb{R}^{n})}:=
\begin{cases}
\left(\sum_{0\le|\alpha|\le s}\lVert D^{\alpha}u\rVert_{L^{p}(I;\mathbb{R}^{n})}^{p}\right)^{1/p}, & \text{if }p<\infty,\\
\max_{0\le|\alpha|\le s}\lVert D^{\alpha}u\rVert_{L^{\infty}(I;\mathbb{R}^{n})}, & \text{if }p=\infty.
\end{cases} ∥ u ∥ W s , p ( I ; R n ) := ⎩ ⎨ ⎧ ( ∑ 0 ≤ ∣ α ∣ ≤ s ∥ D α u ∥ L p ( I ; R n ) p ) 1/ p , max 0 ≤ ∣ α ∣ ≤ s ∥ D α u ∥ L ∞ ( I ; R n ) , if p < ∞ , if p = ∞.
We define the notation H s ( I ; R n ) : = W s , 2 ( I ; R n ) H^{s}(I;\mathbb{R}^{n}) := W^{s,2}(I;\mathbb{R}^{n}) H s ( I ; R n ) := W s , 2 ( I ; R n ) .
All proofs of our results are found in the appendix of the respective chapters.
Well-posedness for controlled ordinary differential equations (CODE)
We are interested in systems of controlled ordinary differential equations, that is
y ˙ ( t ) = f ( t , y ( t ) , u ( t ) ) , t ∈ I : = ( t 1 , t 2 ) ⊂ R , − ∞ < t 1 < t 2 < ∞ , \dot{y}(t) = f(t,y(t),u(t)),\quad t \in I := (t_{1},t_{2}) \subset\mathbb{R},\quad-\infty< t_{1} < t_{2} < \infty, y ˙ ( t ) = f ( t , y ( t ) , u ( t )) , t ∈ I := ( t 1 , t 2 ) ⊂ R , − ∞ < t 1 < t 2 < ∞ ,
y ( t 1 ) = y 0 , (1) y(t_{1}) = y^{0}, \tag*{(1)} y ( t 1 ) = y 0 , (1)
where f : I × R n y × R n u → R n y f : I \times\mathbb{R}^{n_{y}} \times\mathbb{R}^{n_{u}} \to\mathbb{R}^{n_{y}} f : I × R n y × R n u → R n y , y : I → R n y y : I \to\mathbb{R}^{n_{y}} y : I → R n y , u : I → R n u u : I \to\mathbb{R}^{n_{u}} u : I → R n u , n y , n u ∈ N n_{y},n_{u} \in\mathbb{N} n y , n u ∈ N . Further we assume that the control function u u u does not necessarily need to be a continuous but a Lebesgue integrable function. Such setting is usually referred to as Carathéodory setting in literature. See [59 ] , [12 ] , [55 ] , [48 ] and [7 ] for the study of controlled ODEs. We introduce standard assumptions, under which the system is well posed . By well-posedness we mean that for every control function u u u and initial value y 0 y^{0} y 0 there exists a unique solution y y y of Equation (1) . By slightly extending the assumptions we obtain continuity of the solution operator, i.e. the mapping
( u , y 0 ) ↦ y (u,y^{0}) \mapsto y ( u , y 0 ) ↦ y
is continuous with respect to appropriately chosen function spaces and their respective norms. Note that the derivative and integration are understood component-wise throughout this paper.
Theorem 2.1 (Well-posedness of CODE).
Let n y , n u ∈ N n_{y},n_{u} \in\mathbb{N} n y , n u ∈ N and I ⊂ R I \subset\mathbb{R} I ⊂ R bounded and consider f = f ( t , y ( t ) , u ( t ) ) : I × R n y × R n u → R n y f = f(t,y(t),u(t)) : I \times\mathbb{R}^{n_{y}} \times\mathbb{R}^{n_{u}} \to\mathbb{R}^{n_{y}} f = f ( t , y ( t ) , u ( t )) : I × R n y × R n u → R n y . Assume the following properties, which we refer to as Carathéodory properties,
u ∈ L q ( I ; R n u ) u \in L^{q}(I;\mathbb{R}^{n_{u}}) u ∈ L q ( I ; R n u ) , 1 ≤ q ≤ ∞ 1 \le q \le\infty 1 ≤ q ≤ ∞ ,
f ( t , ⋅ , ⋅ ) f(t,\cdot,\cdot) f ( t , ⋅ , ⋅ ) is continuous for almost all t ∈ I t \in I t ∈ I ,
f ( ⋅ , y , u ) f(\cdot,y,u) f ( ⋅ , y , u ) is measurable for all y ∈ R n y y \in\mathbb{R}^{n_{y}} y ∈ R n y , u ∈ R n u u \in\mathbb{R}^{n_{u}} u ∈ R n u ,
there exists α ∈ L p ( I ; [ 0 , ∞ ) ) \alpha\in L^{p}(I;[0,\infty)) α ∈ L p ( I ; [ 0 , ∞ )) , 1 ≤ p ≤ ∞ 1 \le p \le\infty 1 ≤ p ≤ ∞ with ∥ f ( t , y , u ) ∥ R n y ≤ α ( t ) \lVert f(t,y,u)\rVert_{\mathbb{R}^{n_{y}}} \le\alpha(t) ∥ f ( t , y , u ) ∥ R n y ≤ α ( t ) for almost all t ∈ I t \in I t ∈ I , all y ∈ R n y y \in\mathbb{R}^{n_{y}} y ∈ R n y and u ∈ R n u u \in\mathbb{R}^{n_{u}} u ∈ R n u ,
there exists m ∈ L 1 ( I ; [ 0 , ∞ ) ) m \in L^{1}(I;[0,\infty)) m ∈ L 1 ( I ; [ 0 , ∞ )) with ∥ f ( t , y , u ) − f ( t , z , u ) ∥ R n y ≤ m ( t ) ∥ y − z ∥ R n y \lVert f(t,y,u)-f(t,z,u)\rVert_{\mathbb{R}^{n_{y}}} \le m(t)\lVert y-z\rVert_{\mathbb{R}^{n_{y}}} ∥ f ( t , y , u ) − f ( t , z , u ) ∥ R n y ≤ m ( t ) ∥ y − z ∥ R n y for almost every t ∈ I t \in I t ∈ I and all y , z ∈ R n y y,z \in\mathbb{R}^{n_{y}} y , z ∈ R n y , u ∈ R n u u \in\mathbb{R}^{n_{u}} u ∈ R n u ,
for every fixed t 1 ∈ I t_{1} \in I t 1 ∈ I and y ∈ C ( I ; R n y ) y \in C(I;\mathbb{R}^{n_{y}}) y ∈ C ( I ; R n y ) the mapping u ↦ ( t ↦ ∫ t 1 t f ( s , y ( s ) , u ( s ) ) d s ) u \mapsto\left(t \mapsto\int_{t_{1}}^{t} f(s,y(s),u(s))\,\mathrm{d}s\right) u ↦ ( t ↦ ∫ t 1 t f ( s , y ( s ) , u ( s )) d s ) is continuous from L q ( I ; R n u ) L^{q}(I;\mathbb{R}^{n_{u}}) L q ( I ; R n u ) to C ( I ; R n y ) C(I;\mathbb{R}^{n_{y}}) C ( I ; R n y ) .
Then, for every t 1 ∈ I t_1 \in I t 1 ∈ I , y 0 ∈ R n y y^0 \in\mathbb{R}^{n_y} y 0 ∈ R n y and u ∈ L q ( I ; R n u ) u \in L^q(I;\mathbb{R}^{n_u}) u ∈ L q ( I ; R n u ) , there exists a unique function y ∈ C a b s ( I ; R n y ) y \in C_{\mathrm{abs}}(I;\mathbb{R}^{n_y}) y ∈ C abs ( I ; R n y ) such that
y ˙ ( t ) = f ( t , y ( t ) , u ( t ) ) almost everywhere in I , \dot{y}(t)=f(t,y(t),u(t))\ \text{almost everywhere in }I, y ˙ ( t ) = f ( t , y ( t ) , u ( t )) almost everywhere in I ,
y ( t 1 ) = y 0 , y(t_1)=y^0, y ( t 1 ) = y 0 ,
so that y y y is a solution of the controlled ODE. Further the solution y y y depends continuously on y 0 y^0 y 0 and u u u , that is, continuous in the sense of
R n y × L q ( I ; R n u ) → C ( I ; R n y ) , ( y 0 , u ) ↦ y . \mathbb{R}^{n_y}\times L^q(I;\mathbb{R}^{n_u})\to C(I;\mathbb{R}^{n_y}),\quad(y^0,u)\mapsto y. R n y × L q ( I ; R n u ) → C ( I ; R n y ) , ( y 0 , u ) ↦ y .
To highlight the dependence on initial value y 0 y^0 y 0 and control u u u we write y ( t ; u , y 0 ) y(t;u,y^0) y ( t ; u , y 0 ) .
To obtain error rates for approximating above mapping with a Fourier neural operator, we need to introduce more structure regarding the regularity of admissible controls. We define following control set
U l , I , A : = { u ∈ H l ( I ; R n u ) ∣ ∥ u ∥ H l ( I ; R n u ) ≤ A ∈ R + } , U^{l,I,A}:=\{u\in H^l(I;\mathbb{R}^{n_u})\mid\|u\|_{H^l(I;\mathbb{R}^{n_u})}\le A\in\mathbb{R}^{+}\}, U l , I , A := { u ∈ H l ( I ; R n u ) ∣ ∥ u ∥ H l ( I ; R n u ) ≤ A ∈ R + } ,
where we for now assume l ≥ 1 l\ge1 l ≥ 1 . For initial values consider
y 0 ∈ Y 0 ⊂ R n y , y^0\in Y^0\subset\mathbb{R}^{n_y}, y 0 ∈ Y 0 ⊂ R n y ,
where Y 0 ⊂ R n y Y^0\subset\mathbb{R}^{n_y} Y 0 ⊂ R n y is bounded. These assumptions immediately lead to bounded CODE solutions under the Carathéodory properties.
Lemma 2.2. Consider f f f from Theorem 2.1 with the assumptions (2)-(6) and u ∈ U l , I , A u\in U^{l,I,A} u ∈ U l , I , A , l ≥ 1 l\ge1 l ≥ 1 , y 0 ∈ Y 0 ⊂ R n y y^0\in Y^0\subset\mathbb{R}^{n_y} y 0 ∈ Y 0 ⊂ R n y additionally. Then we find a Y ∈ R + Y\in\mathbb{R}^{+} Y ∈ R + such that for the solution y ( t ; u , y 0 ) ∈ C a b s ( I ; R n y ) y(t;u,y^0)\in C_{\mathrm{abs}}(I;\mathbb{R}^{n_y}) y ( t ; u , y 0 ) ∈ C abs ( I ; R n y ) of the CODE system it holds
y ( t ; y 0 , u ) ∈ ( − Y , Y ) n y y(t;y^0,u)\in(-Y,Y)^{n_y} y ( t ; y 0 , u ) ∈ ( − Y , Y ) n y for all $t\in I.
When working with FNOs, we usually study periodic functions on the domain T d = [ 0 , 2 π ] d \mathbb{T}^d=[0,2\pi]^d T d = [ 0 , 2 π ] d , d ∈ N d\in\mathbb{N} d ∈ N . Therefore, to make Fourier analysis in this setting possible, we focus on solutions that at least live on a smaller time domain
J : = ( t ^ 1 , t ^ 2 ) ⊂ ( 0 , 2 π ) = T , t ^ 1 < t ^ 2 . J:=(\hat{t}_1,\hat{t}_2)\subset(0,2\pi)=\mathbb{T},\quad\hat{t}_1<\hat{t}_2. J := ( t ^ 1 , t ^ 2 ) ⊂ ( 0 , 2 π ) = T , t ^ 1 < t ^ 2 .
However, for our CODE system, we still assume the system to be defined for an interval I = ( t 1 , t 2 ) I=(t_1,t_2) I = ( t 1 , t 2 ) with T ⊂ I \mathbb{T}\subset I T ⊂ I , while the Carathéodory properties are assumed to hold on I I I . The reason for such interval I I I is discussed at the end of Appendix § 8 .
By definition, the control functions do not need to be periodic but the following lemma shows that we are able to consider extended control functions such that they are periodic. The lemma can be found in the appendix of [27 , 32 ] where it is stated that the lemma relies on a result by [60 ] .
Lemma 2.3 ([27 , 32 , 60 ] ). Let D ⊂ R d D\subset\mathbb{R}^d D ⊂ R d , d ∈ N d\in\mathbb{N} d ∈ N be a bounded, Lipschitz domain [15 ] . There exists a continuous, linear operator E : W s , p ( D ) → W p e r s , p ( F ) \mathcal{E}:W^{s,p}(D)\to W_{\mathrm{per}}^{s,p}(F) E : W s , p ( D ) → W per s , p ( F ) for any s ≥ 0 s\ge0 s ≥ 0 and 1 ≤ p ≤ ∞ 1\le p\le\infty 1 ≤ p ≤ ∞ where F ⊂ R d F\subset\mathbb{R}^d F ⊂ R d is a bounded hypercube with D ⋐ F D\Subset F D ⋐ F such that, for any u ∈ W s , p ( D ) u\in W^{s,p}(D) u ∈ W s , p ( D ) ,
E ( u ) ∣ D = u \mathcal{E}(u)|_D=u E ( u ) ∣ D = u ,
E ( u ) ∈ W p e r s , p ( F ) \mathcal{E}(u)\in W_{\mathrm{per}}^{s,p}(F) E ( u ) ∈ W per s , p ( F ) is periodic on F F F (including its derivatives).
Furthermore, E \mathcal{E} E maps continuously differentiable functions to continuously differentiable functions, so
E ( C s ( D ‾ ) ) ⊂ C p e r s ( F ) , \mathcal{E}(C^s(\overline{D}))\subset C_{\mathrm{per}}^s(F), E ( C s ( D )) ⊂ C per s ( F ) , and hence defines a continuous mapping E : C s ( D ‾ ) → C p e r s ( F ) \mathcal{E}:C^s(\overline{D})\to C_{\mathrm{per}}^s(F) E : C s ( D ) → C per s ( F ) .
We fix this extension operator E \mathcal{E} E and introduce following control set to keep notation simple
U l , T , A : = { E u ∈ H l ( T ; R n u ) ∣ u ∈ U l , I , A } , (2) U^{l,\mathbb{T},A}:=\{\mathcal{E}u\in H^l(\mathbb{T};\mathbb{R}^{n_u})\mid u\in U^{l,I,A}\}, \tag*{(2)} U l , T , A := { E u ∈ H l ( T ; R n u ) ∣ u ∈ U l , I , A } , (2)
where we now assume to have at least l ≥ 2 l\ge2 l ≥ 2 regarding the regularity. The controls u ∈ U l , T , A u\in U^{l,\mathbb{T},A} u ∈ U l , T , A are the periodic versions of the control functions from U l , I , A U^{l,I,A} U l , I , A . We sometime omit to explicitly write E u \mathcal{E}u E u and by abuse of notation just write u ∈ U l , T , A u\in U^{l,\mathbb{T},A} u ∈ U l , T , A when it is obvious. Further we define the periodically extended time variable as
τ : T → R , t ↦ E id ( t ) . \tau:\mathbb{T}\to\mathbb{R},\ t\mapsto\mathcal{E}\operatorname{id}(t). τ : T → R , t ↦ E id ( t ) .
One of the reasons we introduce the periodic time variation τ \tau τ is displayed in the following. Ignoring the inner function y y y , the function
t ↦ f ( t , y ( t ) , u ( t ) ) , t ∈ T t \mapsto f(t,y(t),u(t)),\quad t \in\mathbb{T} t ↦ f ( t , y ( t ) , u ( t )) , t ∈ T
cannot be periodic alone due to the t t t -dependence. Thus we need to introduce such periodic time variable. As τ \tau τ is dependent on t t t and constructed the same way as the controls u ∈ U l , T , A u \in\mathcal{U}^{l,T,A} u ∈ U l , T , A , we may consider τ \tau τ as an additional control function. Therefore to simplify analysis and notation, we study for now the autonomous system
y ˙ ( t ) = f ( y ( t ) , u ( t ) ) , t ∈ T , y ( t ^ 1 ) = y 0 , (3) \begin{aligned}
\dot{y}(t)&=f(y(t),u(t)),\quad t\in\mathbb{T},\\
y(\hat{t}_1)&=y^0, \tag*{(3)}
\end{aligned} y ˙ ( t ) y ( t ^ 1 ) = f ( y ( t ) , u ( t )) , t ∈ T , = y 0 , (3)
where u ∈ U l , T , A u \in\mathcal{U}^{l,T,A} u ∈ U l , T , A , l ≥ 2 l \geq2 l ≥ 2 but with Im ( u ) ⊂ R 1 + n u \operatorname{Im}(u) \subset\mathbb{R}^{1+n_u} Im ( u ) ⊂ R 1 + n u . Looking at the Carathéodory assumptions, this means we require
t ↦ f ( t , y , u ) t \mapsto f(t,y,u) t ↦ f ( t , y , u )
to be continuous instead of measurable for every y ∈ R n y y \in\mathbb{R}^{n_y} y ∈ R n y , u ∈ R n u u \in\mathbb{R}^{n_u} u ∈ R n u .
The functions u u u from the introduced control set U l , T , A \mathcal{U}^{l,T,A} U l , T , A share the same bound with respect to the sup-norm. Since we discussed that the solutions are also bounded, this will allow us to study the non-linearity f f f on a bounded domain, which is essential if one is interested in approximating f f f by finite neural networks.
Lemma 2.4. For the control functions u ∈ U l , T , A u \in\mathcal{U}^{l,T,A} u ∈ U l , T , A , l ≥ 2 l \geq2 l ≥ 2 , with Im ( u ) ⊂ R 1 + n u \operatorname{Im}(u) \subset\mathbb{R}^{1+n_u} Im ( u ) ⊂ R 1 + n u there exists a U ′ ∈ R + U' \in\mathbb{R}^{+} U ′ ∈ R + such that
u ( t ) ∈ ( − U ′ , U ′ ) 1 + n u u(t) \in(-U',U')^{1+n_u} u ( t ) ∈ ( − U ′ , U ′ ) 1 + n u for all t ∈ T t \in\mathbb{T} t ∈ T .
Later, when reformulating CODE as Picard iteration, we have to recast it in a way such that it is applicable for emulation through the layers of a FNO. This leads us to replace f f f with an approximation by finite neural networks. An approximation like that eventually introduces an accuracy error inside the Picard iteration. Thus we have to enlarge the domain of interest.
Choose a γ ∈ R + \gamma\in\mathbb{R}^{+} γ ∈ R + such that
Y ′ : = Y + γ > 1 (4) Y' := Y+\gamma>1 \tag*{(4)} Y ′ := Y + γ > 1 (4)
and define the domain
D ′ : = ( − Y ′ − 1 , Y ′ + 1 ) n y × ( − U ′ − 1 , U ′ + 1 ) 1 + n u D' := (-Y'-1,Y'+1)^{n_y}\times(-U'-1,U'+1)^{1+n_u} D ′ := ( − Y ′ − 1 , Y ′ + 1 ) n y × ( − U ′ − 1 , U ′ + 1 ) 1 + n u
to study f f f on. The margin 1 ensures that the closed boxes [ − Y ′ , Y ′ ] n y [-Y',Y']^{n_y} [ − Y ′ , Y ′ ] n y and [ − U ′ , U ′ ] 1 + n u [-U',U']^{1+n_u} [ − U ′ , U ′ ] 1 + n u , which contain the values of all functions inserted into f f f below, lie in the interior of D ′ D' D ′ . Before introducing Fourier neural operators we summarize all final assumptions on the CODE system.
Assumption 2.5 (Final assumptions of CODE system).
{ J : = ( t ^ 1 , t ^ 2 ) ⊂ ( 0 , 2 π ) , y 0 ∈ Y 0 , u ∈ U l , J , A , l ≥ 2 , f as in x r e f ‘ T h e o r e m 2.1 ‘ with α ∈ L ∞ ( J ; [ 0 , ∞ ) ) , s ≥ 2 , D β f ∣ D ′ ∈ L ∞ ( D ′ ; R n y ) , ∀ β ∈ N 0 d ( d : = 1 + n y + n u ) , ∣ β ∣ ≤ s , ∥ f ∥ W s , ∞ ( D ′ ; R n y ) ≤ E ∈ R + . \left\{
\begin{aligned}
J&:=(\hat{t}_1,\hat{t}_2)\subset(0,2\pi),\\
y^0&\in Y^0,\\
u&\in\mathcal{U}^{l,J,A},\ l\geq2,\\
f&\text{ as in }{xref}`Theorem 2.1`\text{ with }\alpha\in L^\infty(J;[0,\infty)),\\
s&\geq2,\\
D^\beta f|_{D'}&\in L^\infty(D';\mathbb{R}^{n_y}),\ \forall\beta\in\mathbb{N}_0^d\ (d:=1+n_y+n_u),\ |\beta|\leq s,\\
\|f\|_{W^{s,\infty}(D';\mathbb{R}^{n_y})}&\leq E\in\mathbb{R}^{+}.
\end{aligned}
\right. ⎩ ⎨ ⎧ J y 0 u f s D β f ∣ D ′ ∥ f ∥ W s , ∞ ( D ′ ; R n y ) := ( t ^ 1 , t ^ 2 ) ⊂ ( 0 , 2 π ) , ∈ Y 0 , ∈ U l , J , A , l ≥ 2 , as in x r e f ‘ T h eor e m 2.1‘ with α ∈ L ∞ ( J ; [ 0 , ∞ )) , ≥ 2 , ∈ L ∞ ( D ′ ; R n y ) , ∀ β ∈ N 0 d ( d := 1 + n y + n u ) , ∣ β ∣ ≤ s , ≤ E ∈ R + .
Note that we assume that f ∈ W s , ∞ f \in W^{s,\infty} f ∈ W s , ∞ , s ≥ 2 s \geq2 s ≥ 2 with ∥ f ∥ W s , ∞ ( D ′ ; R n y ) ≤ E ∈ R + \|f\|_{W^{s,\infty}(D';\mathbb{R}^{n_y})} \leq E \in\mathbb{R}^{+} ∥ f ∥ W s , ∞ ( D ′ ; R n y ) ≤ E ∈ R + which is needed for the efficient approximation by a finite neural network, see Lemmata 3.11 and 3.13 . The assumption ∥ f ∥ W s , ∞ ( D ′ ; R n y ) ≤ E ∈ R + \|f\|_{W^{s,\infty}(D';\mathbb{R}^{n_y})} \leq E \in\mathbb{R}^{+} ∥ f ∥ W s , ∞ ( D ′ ; R n y ) ≤ E ∈ R + is a smallness condition where we need to choose E E E as discussed in Equation (C.1) to secure convergence of our proposed Picard type iteration.
Fourier neural operator for CODE
Fourier neural operators, first introduced in [39 ] , have strong approximation capabilities [27 , 32 ] while offering to apply methods from Fourier analysis including the differentiation formula. The latter formula allows us to rewrite an integral formula using the (inverse) Fourier transform, which makes FNOs an encouraging candidate for the emulation of the Picard iteration. Thus we begin by introducing all Fourier analysis related notations and definitions needed for understanding our main result Theorem 4.1. Most of those notations and definitions are from the paper [27 ] as our article is intended to build upon.
Definition
We start by introducing neural operator architecture in a broader sense. Let d ∈ N d \in\mathbb{N} d ∈ N and denote by D D D a domain in R d \mathbb{R}^{d} R d . Consider arbitrary Banach spaces of form A ( D ; R n a ) \mathcal{A}(D;\mathbb{R}^{n_a}) A ( D ; R n a ) and U ( D ; R n u ) \mathcal{U}(D;\mathbb{R}^{n_u}) U ( D ; R n u ) with d a , d u ∈ N d_a,d_u \in\mathbb{N} d a , d u ∈ N . A general neural operator is a mapping between such spaces denoted by
N : A ( D ; R n a ) → U ( D ; R n u ) , a ↦ N ( a ) = Q ∘ L L ∘ L L − 1 ∘ ⋯ ∘ L 1 ∘ R ( a ) \mathcal{N}:\mathcal{A}(D;\mathbb{R}^{n_a}) \to\mathcal{U}(D;\mathbb{R}^{n_u}),\quad a \mapsto\mathcal{N}(a)=Q\circ\mathcal{L}_{L}\circ\mathcal{L}_{L-1}\circ\cdots\circ\mathcal{L}_{1}\circ\mathcal{R}(a) N : A ( D ; R n a ) → U ( D ; R n u ) , a ↦ N ( a ) = Q ∘ L L ∘ L L − 1 ∘ ⋯ ∘ L 1 ∘ R ( a )
for a given depth L ∈ N L \in\mathbb{N} L ∈ N where R : A ( D ; R n a ) → U ( D ; R n v ) \mathcal{R}:\mathcal{A}(D;\mathbb{R}^{n_a})\to\mathcal{U}(D;\mathbb{R}^{n_v}) R : A ( D ; R n a ) → U ( D ; R n v ) , d v ≥ d u d_v\ge d_u d v ≥ d u is a lifting operator (acting locally) defined by
R ( a ) ( x ) = R a ( x ) , R ∈ R d v × d a , (5) \mathcal{R}(a)(x)=Ra(x),\quad R\in\mathbb{R}^{d_v\times d_a}, \tag*{(5)} R ( a ) ( x ) = R a ( x ) , R ∈ R d v × d a , (5)
and Q : U ( D ; R d v ) → U ( D ; R d u ) Q:\mathcal{U}(D;\mathbb{R}^{d_v})\to\mathcal{U}(D;\mathbb{R}^{d_u}) Q : U ( D ; R d v ) → U ( D ; R d u ) is a local projection operator of form
Q ( v ) ( x ) = Q v ( x ) , Q ∈ R d u × d v . (6) Q(v)(x)=Qv(x),\quad Q\in\mathbb{R}^{d_u\times d_v}. \tag*{(6)} Q ( v ) ( x ) = Q v ( x ) , Q ∈ R d u × d v . (6)
The non-linear operator layers L l : U ( D ; R n v ) → U ( D ; R n v ) \mathcal{L}_l:\mathcal{U}(D;\mathbb{R}^{n_v})\to\mathcal{U}(D;\mathbb{R}^{n_v}) L l : U ( D ; R n v ) → U ( D ; R n v ) , v ↦ L l ( v ) v\mapsto\mathcal{L}_l(v) v ↦ L l ( v ) have following structure
L l ( v ) ( x ) = σ ( W l v ( x ) + b l ( x ) + ( K ( a ; θ l ) v ) ( x ) ) , ∀ x ∈ D . \mathcal{L}_l(v)(x)=\sigma\left(W_lv(x)+b_l(x)+(\mathcal{K}(a;\theta_l)v)(x)\right),\quad\forall x\in D. L l ( v ) ( x ) = σ ( W l v ( x ) + b l ( x ) + ( K ( a ; θ l ) v ) ( x ) ) , ∀ x ∈ D .
The weight matrix W l ∈ R n v × n v W_l\in\mathbb{R}^{n_v\times n_v} W l ∈ R n v × n v and the bias b l ( x ) ∈ U ( D ; R n v ) b_l(x)\in\mathcal{U}(D;\mathbb{R}^{n_v}) b l ( x ) ∈ U ( D ; R n v ) define affine point-wise mappings W l v ( x ) + b l ( x ) W_lv(x)+b_l(x) W l v ( x ) + b l ( x ) , l = 1 , … , L l=1,\ldots,L l = 1 , … , L , while the linear operators K ( a , θ ) \mathcal{K}(a,\theta) K ( a , θ ) , acting non-local, are integral operators of form
( K ( a ; θ ) v ) ( x ) = ∫ D κ θ ( x , y ; a ( x ) , a ( y ) ) v ( y ) d y , ∀ x ∈ D , (\mathcal{K}(a;\theta)v)(x)=\int_D\kappa_\theta(x,y;a(x),a(y))v(y)\,\mathrm{d}y,\quad\forall x\in D, ( K ( a ; θ ) v ) ( x ) = ∫ D κ θ ( x , y ; a ( x ) , a ( y )) v ( y ) d y , ∀ x ∈ D ,
where the integral kernel κ θ : R 2 ( d + d a ) → R d v × d v \kappa_\theta:\mathbb{R}^{2(d+d_a)}\to\mathbb{R}^{d_v\times d_v} κ θ : R 2 ( d + d a ) → R d v × d v is a neural network parametrized by θ ∈ Θ \theta\in\Theta θ ∈ Θ . The non-linear activation function σ \sigma σ acts component-wise. Fourier neural operators are a special case of these general neural operators where the kernel is of form κ θ = κ θ ( x − y ) \kappa_\theta=\kappa_\theta(x-y) κ θ = κ θ ( x − y ) . In this case our linear operators are convolutions
( K ( θ ) v ) ( x ) = ∫ D κ θ ( x − y ) v ( y ) d y , ∀ x ∈ D . (\mathcal{K}(\theta)v)(x)=\int_D\kappa_\theta(x-y)v(y)\,\mathrm{d}y,\quad\forall x\in D. ( K ( θ ) v ) ( x ) = ∫ D κ θ ( x − y ) v ( y ) d y , ∀ x ∈ D .
The domain is now considered as D = [ 0 , 2 π ] d D=[0,2\pi]^d D = [ 0 , 2 π ] d . Concluding, FNOs are concatenations of
N ( a ) = Q ∘ L L ∘ L L − 1 ∘ ⋯ ∘ L 1 ∘ R ( a ) , \mathcal{N}(a)=Q\circ\mathcal{L}_{L}\circ\mathcal{L}_{L-1}\circ\cdots\circ\mathcal{L}_{1}\circ\mathcal{R}(a), N ( a ) = Q ∘ L L ∘ L L − 1 ∘ ⋯ ∘ L 1 ∘ R ( a ) ,
where
L l ( v ) ( x ) = σ ( W l v ( x ) + b l ( x ) + F − 1 ( P l ( k ) ⋅ F ( v ) ( k ) ) ( x ) ) , (7) \mathcal{L}_l(v)(x)=\sigma\left(W_lv(x)+b_l(x)+\mathcal{F}^{-1}\left(P_l(k)\cdot\mathcal{F}(v)(k)\right)(x)\right), \tag*{(7)} L l ( v ) ( x ) = σ ( W l v ( x ) + b l ( x ) + F − 1 ( P l ( k ) ⋅ F ( v ) ( k ) ) ( x ) ) , (7)
while F \mathcal{F} F (F − 1 \mathcal{F}^{-1} F − 1 ) denotes the (inverse) Fourier transform. The Fourier multiplier matrix P l : Z d → C d v × d v P_l:\mathbb{Z}^d\to\mathbb{C}^{d_v\times d_v} P l : Z d → C d v × d v defines the coefficients of a non-local, linear mapping via the Fourier transform. It stems from the integral kernel but we consider it directly parametrized in Fourier space as we have P θ ( k ) = F ( κ θ ) ( k ) P_\theta(k)=\mathcal{F}(\kappa_\theta)(k) P θ ( k ) = F ( κ θ ) ( k ) . We impose that P θ ( − k ) = P θ ( k ) ‾ P_\theta(-k)=\overline{P_\theta(k)} P θ ( − k ) = P θ ( k ) , where the bar denotes entry-wise complex conjugation, to ensure that ( K ( θ ) v ) ( x ) (\mathcal{K}(\theta)v)(x) ( K ( θ ) v ) ( x ) is a real-valued function for a real-valued input function v v v .
As we are interested in deriving approximation rates for the FNO, we need to quantify the size, i.e. the amount of trainable parameters of such architecture. Note that the layers of a FNO require to compute an integral exactly to perform the (inverse) Fourier transform. Thus, an implementable version of the FNO, the so-called Ψ \Psi Ψ -FNO, is introduced by utilizing the discrete (inverse) Fourier transform. For this, we need the Fourier wavenumbers first.
Definition 3.1 (Fourier wavenumbers). A set of Fourier wavenumbers k ∈ K N k\in\mathcal{K}_N k ∈ K N is given by
K N : = { k ∈ Z d ∣ ∣ k ∣ ∞ ≤ N } , \mathcal{K}_N:=\{k\in\mathbb{Z}^{d}\mid\lvert k\rvert_{\infty}\le N\}, K N := { k ∈ Z d ∣ ∣ k ∣ ∞ ≤ N } ,
where k = ( k 1 , … , k d ) ∈ Z d k=(k_1,\ldots,k_d)\in\mathbb{Z}^{d} k = ( k 1 , … , k d ) ∈ Z d and
∣ k ∣ ∞ : = max i = 1 , … , d ∣ k i ∣ ≤ N . \lvert k\rvert_{\infty}:=\max_{i=1,\ldots,d}\lvert k_i\rvert\le N. ∣ k ∣ ∞ := i = 1 , … , d max ∣ k i ∣ ≤ N .
In the following we introduce a uniform discretized grid of T d = [ 0 , 2 π ] d \mathbb{T}^{d}=[0,2\pi]^{d} T d = [ 0 , 2 π ] d . Set the index set as
J N : = { 0 , … , 2 N } d \mathcal{J}_{N}:=\{0,\ldots,2N\}^{d} J N := { 0 , … , 2 N } d
and the grid points with the corresponding function evaluations as
t j : = 2 π j 2 N + 1 , v j : = v ( t j ) , j ∈ J N . t_{j}:=\frac{2\pi j}{2N+1},\quad v_{j}:=v(t_{j}),\ j\in\mathcal{J}_{N}. t j := 2 N + 1 2 π j , v j := v ( t j ) , j ∈ J N .
Definition 3.2 (Discrete Fourier transform). The discrete Fourier transform F N : R J N → C K N \mathcal{F}_{N}:\mathbb{R}^{\mathcal{J}_{N}}\to\mathbb{C}^{K_{N}} F N : R J N → C K N is defined by
F N ( v ) ( k ) : = 1 ( 2 N + 1 ) d ∑ j ∈ J N v j e − 2 π i ⟨ j , k ⟩ / ( 2 N + 1 ) \mathcal{F}_{N}(v)(k):=\frac{1}{(2N+1)^{d}}\sum_{j\in\mathcal{J}_{N}}v_{j}e^{-2\pi i\langle j,k\rangle/(2N+1)} F N ( v ) ( k ) := ( 2 N + 1 ) d 1 j ∈ J N ∑ v j e − 2 π i ⟨ j , k ⟩ / ( 2 N + 1 ) and the inverse F N − 1 : C K N → R J N \mathcal{F}_{N}^{-1}:\mathbb{C}^{K_{N}}\to\mathbb{R}^{\mathcal{J}_{N}} F N − 1 : C K N → R J N ,
F N − 1 ( v ^ ) ( j ) : = ∑ k ∈ K N v ^ k e 2 π i ⟨ j , k ⟩ / ( 2 N + 1 ) . \mathcal{F}_{N}^{-1}(\hat{v})(j):=\sum_{k\in K_{N}}\hat{v}_{k}e^{2\pi i\langle j,k\rangle/(2N+1)}. F N − 1 ( v ^ ) ( j ) := k ∈ K N ∑ v ^ k e 2 π i ⟨ j , k ⟩ / ( 2 N + 1 ) .
When working with the discrete (inverse) Fourier transform, a natural function space to study are the trigonometric polynomials of corresponding degree N ∈ N N\in\mathbb{N} N ∈ N . Thus, we introduce the precise definition.
Definition 3.3 (Trigonometric polynomials). Let N ∈ N N\in\mathbb{N} N ∈ N , then denote by L N 2 ( T d ) L_{N}^{2}(\mathbb{T}^{d}) L N 2 ( T d ) the space of trigonometric polynomials v N : T d → R v_{N}:\mathbb{T}^{d}\to\mathbb{R} v N : T d → R defined by
v N ( t ) = ∑ ∣ k ∣ ∞ ≤ N c k e i ⟨ t , k ⟩ , v_{N}(t)=\sum_{|k|_{\infty}\leq N}c_{k}e^{i\langle t,k\rangle}, v N ( t ) = ∣ k ∣ ∞ ≤ N ∑ c k e i ⟨ t , k ⟩ , with c k ∈ C c_{k}\in\mathbb{C} c k ∈ C .
We recall that a trigonometric polynomial is uniquely defined by its values on the grid J N \mathcal{J}_{N} J N . With the following projection, the so-called pseudo-spectral projection, we are able to to project continuous function, and thus Sobolev function under certain assumptions, on the space of trigonometric polynomials.
Definition 3.4 (Pseudo-spectral projection). The pseudo-spectral projection is given by
I N : C ( T d ) → L N 2 ( T d ) , \mathcal{I}_{N}:C(\mathbb{T}^{d})\to L_{N}^{2}(\mathbb{T}^{d}), I N : C ( T d ) → L N 2 ( T d ) , where I N v \mathcal{I}_{N}v I N v of a continuous function v v v is defined as the unique trigonometric polynomial in L N 2 ( T d ) L_{N}^{2}(\mathbb{T}^{d}) L N 2 ( T d ) such that
I N v ( t j ) = v ( t j ) , ∀ j ∈ J N . \mathcal{I}_{N}v(t_{j})=v(t_{j}),\ \forall j\in\mathcal{J}_{N}. I N v ( t j ) = v ( t j ) , ∀ j ∈ J N .
This projection allows us, to give an analogue definition of the Ψ \Psi Ψ -FNO compared to the continuous FNO.
Definition 3.5 (Ψ \Psi Ψ -FNO). A Ψ \Psi Ψ -FNO is a mapping
N : A ( T d ; R d a ) → U ( T d ; R d u ) , a ↦ N ( a ) \mathcal{N}:A(\mathbb{T}^{d};\mathbb{R}^{d_{a}})\to U(\mathbb{T}^{d};\mathbb{R}^{d_{u}}),\ a\mapsto\mathcal{N}(a) N : A ( T d ; R d a ) → U ( T d ; R d u ) , a ↦ N ( a ) of form
N ( a ) = Q ∘ I N ∘ L L ∘ I N ∘ ⋯ ∘ L 1 ∘ I N ∘ R ( a ) , \mathcal{N}(a)=Q\circ\mathcal{I}_{N}\circ\mathcal{L}_{L}\circ\mathcal{I}_{N}\circ\cdots\circ\mathcal{L}_{1}\circ\mathcal{I}_{N}\circ R(a), N ( a ) = Q ∘ I N ∘ L L ∘ I N ∘ ⋯ ∘ L 1 ∘ I N ∘ R ( a ) , where I N \mathcal{I}_{N} I N is the pseudo-spectral Fourier projection. The lifting and projection layer are given again as before, see Equation (5) and Equation (6) . The non-linear layers L l \mathcal{L}_{l} L l , l = 1 , … , L l=1,\ldots,L l = 1 , … , L are given by Equation (7) , where we allow the Lipschitz continuous activation function σ = σ l \sigma=\sigma_{l} σ = σ l to depend on the layer l l l . Lemma D.4 and Lemma D.5 of [27 ] remain valid in this case, since their proofs only use the Lipschitz continuity of the activation functions.
A Ψ \Psi Ψ -FNO is uniquely defined as by its restriction on the finite-dimensional subspace L N 2 ( T d ; R d a ) ⊂ A ( T d ; R d a ) L_{N}^{2}(\mathbb{T}^{d};\mathbb{R}^{d_{a}})\subset A(\mathbb{T}^{d};\mathbb{R}^{d_{a}}) L N 2 ( T d ; R d a ) ⊂ A ( T d ; R d a ) . Further a Ψ \Psi Ψ -FNO can be naturally identified with a finite-dimensional mapping
N ^ : R d a × J N → R d u × J N , a ↦ N ^ ( a ) , \widehat{\mathcal{N}}:\mathbb{R}^{d_{a}\times\mathcal{J}_{N}}\to\mathbb{R}^{d_{u}\times\mathcal{J}_{N}},\ a\mapsto\widehat{\mathcal{N}}(a), N : R d a × J N → R d u × J N , a ↦ N ( a ) ,
with input a = { a j } j ∈ J N ∈ R d a × J N a=\{a_{j}\}_{j\in\mathcal{J}_{N}}\in\mathbb{R}^{d_{a}\times\mathcal{J}_{N}} a = { a j } j ∈ J N ∈ R d a × J N corresponding to the point-values a j = a ( t j ) a_{j}=a(t_{j}) a j = a ( t j ) on the grid { t j } j ∈ J N \{t_{j}\}_{j\in\mathcal{J}_{N}} { t j } j ∈ J N and
J N : = { 0 , … , 2 N } d . \mathcal{J}_{N}:=\{0,\ldots,2N\}^{d}. J N := { 0 , … , 2 N } d .
The non-linear layers L ^ l \widehat{\mathcal{L}}_{l} L l for l = 1 , … , L l=1,\ldots,L l = 1 , … , L are
L ^ ( v ) j = σ ( W l v j + b l , j + F N − 1 ( P l ( k ) ⋅ F N ( v ) ( k ) ) j ) \widehat{\mathcal{L}}(v)_{j}=\sigma\left(W_{l}v_{j}+b_{l,j}+\mathcal{F}_{N}^{-1}\left(P_{l}(k)\cdot\mathcal{F}_{N}(v)(k)\right)_{j}\right) L ( v ) j = σ ( W l v j + b l , j + F N − 1 ( P l ( k ) ⋅ F N ( v ) ( k ) ) j )
for j ∈ J N j\in\mathcal{J}_{N} j ∈ J N . To quantify the approximation error with respect to the network’s size, we give following definition.
Definition 3.6 (Size of Ψ \Psi Ψ -FNO). The depth and width of a Ψ \Psi Ψ -FNO N \mathcal{N} N are defined by
depth ( N ) : = L , width ( N ) : = d v ∣ J N ∣ = d v ∣ K N ∣ = ( 2 N + 1 ) d d v , lift ( N ) : = d v . \operatorname{depth}(\mathcal{N}) := L,\quad\operatorname{width}(\mathcal{N}) := d_v|\mathcal{J}_N| = d_v|\mathcal{K}_N| = (2N+1)^d d_v,\quad\operatorname{lift}(\mathcal{N}) := d_v. depth ( N ) := L , width ( N ) := d v ∣ J N ∣ = d v ∣ K N ∣ = ( 2 N + 1 ) d d v , lift ( N ) := d v . The precise size of Ψ \Psi Ψ -FNO is then given by
size ( N ) = d v d u + L ( d v 2 + d v ∣ J N ∣ + d v 2 ∣ J N ∣ ) + d a d v . \operatorname{size}(\mathcal{N}) = d_vd_u + L\left(d_v^2 + d_v|\mathcal{J}_N| + d_v^2|\mathcal{J}_N|\right) + d_ad_v. size ( N ) = d v d u + L ( d v 2 + d v ∣ J N ∣ + d v 2 ∣ J N ∣ ) + d a d v . We use the estimate
size ( N ) ≲ depth ( N ) width ( N ) lift ( N ) , (8) \operatorname{size}(\mathcal{N}) \lesssim\operatorname{depth}(\mathcal{N})\operatorname{width}(\mathcal{N})\operatorname{lift}(\mathcal{N}), \tag*{(8)} size ( N ) ≲ depth ( N ) width ( N ) lift ( N ) , (8) where we assume that max ( d a , d u ) ≤ d v \max(d_a,d_u) \leq d_v max ( d a , d u ) ≤ d v which implies that we have size ( N ) ∼ L d v 2 ∣ J N ∣ \operatorname{size}(\mathcal{N}) \sim Ld_v^2|\mathcal{J}_N| size ( N ) ∼ L d v 2 ∣ J N ∣ .
Besides the pseudo-spectral projection, we also use the L 2 L^2 L 2 -orthogonal projection to project the space of square-integrable functions onto the space of trigonometric polynomials.
Definition 3.7 (L 2 L^2 L 2 -orthogonal projection). The L 2 L^2 L 2 -orthogonal projection
P N : L 2 ( T d ) → L N 2 ( T d ) P_N : L^2(\mathbb{T}^d) \to L_N^2(\mathbb{T}^d) P N : L 2 ( T d ) → L N 2 ( T d ) is defined by
P N ( ∑ k ∈ Z d c k e i ⟨ t , k ⟩ ) = ∑ ∣ k ∣ ∞ ≤ N c k e i ⟨ t , k ⟩ . P_N\left(\sum_{k\in\mathbb{Z}^d}c_k e^{i\langle t,k\rangle}\right)=\sum_{|k|_\infty\leq N}c_k e^{i\langle t,k\rangle}. P N ( k ∈ Z d ∑ c k e i ⟨ t , k ⟩ ) = ∣ k ∣ ∞ ≤ N ∑ c k e i ⟨ t , k ⟩ .
Recall, that we need to reformulate the Picard iteration of the CODE in such a way that emulation through the FNO layers, or to be more precise through the Ψ \Psi Ψ -FNO layers, is feasible. Then, to secure well-definedness regarding the reformulated Picard iteration, we need following additional projection.
Definition 3.8 (Mean reducing L 2 L^2 L 2 -orthogonal projection). Define the projection
P ˙ N : L 2 ( T d ) → L ˙ N 2 ( T d ) \dot{P}_N : L^2(\mathbb{T}^d) \to\dot{L}_N^2(\mathbb{T}^d) P ˙ N : L 2 ( T d ) → L ˙ N 2 ( T d ) by removing the mean, i.e. P ˙ N v = P N v − ∫ T d v ( t ) d t \dot{P}_Nv=P_Nv-\int_{\mathbb{T}^d}v(t)\,dt P ˙ N v = P N v − ∫ T d v ( t ) d t or equivalently
P ˙ N ( ∑ k ∈ Z d c k e i ⟨ t , k ⟩ ) = ∑ 0 < ∣ k ∣ ∞ ≤ N c k e i ⟨ t , k ⟩ , ∀ ( c k ) k ∈ Z d ∈ ℓ 2 ( Z d ) . \dot{P}_N\left(\sum_{k\in\mathbb{Z}^d}c_k e^{i\langle t,k\rangle}\right)=\sum_{0<|k|_\infty\leq N}c_k e^{i\langle t,k\rangle},\quad\forall(c_k)_{k\in\mathbb{Z}^d}\in\ell^2(\mathbb{Z}^d). P ˙ N ( k ∈ Z d ∑ c k e i ⟨ t , k ⟩ ) = 0 < ∣ k ∣ ∞ ≤ N ∑ c k e i ⟨ t , k ⟩ , ∀ ( c k ) k ∈ Z d ∈ ℓ 2 ( Z d ) .
See also [58 ] and [38 ] for further error estimates of the L 2 L^2 L 2 -orthogonal and pseudo-spectral projection. We conclude this subsection by briefly defining the Dirichlet kernel as it is needed for the formulation of our main result Theorem 4.1.
Definition 3.9 (Dirichlet kernel). For N ∈ N N\in\mathbb{N} N ∈ N and t ∈ R t\in\mathbb{R} t ∈ R we denote by
D N ( t ) : = ∑ ∣ k ∣ ≤ N e i t k D_N(t):=\sum_{|k|\leq N}e^{itk} D N ( t ) := ∣ k ∣ ≤ N ∑ e i t k the Dirichlet kernel.
Further additional definitions and statements of Ψ \Psi Ψ -FNOs needed solely for proving Theorem 4.1 are found in Appendix § 9 .
Finite neural network approximation
As mentioned earlier, we need to find a way of dealing with the non-linearity f f f in the CODE system to derive error bounds. Fortunately, there are useful results which imply under certain assumptions the existence of finite neural networks capable of avoiding the curse of dimensionality when approximating other functions. These results mostly rely on which type of function is approximated and which kind of activation function is assumed inside the neural network’s architecture.
Following definition, presenting one of the most known activation functions, is from [18 ] . We also recommend said reference for a further reading regarding the precise definition of a neural network as well as regarding the quantities depth and width describing a finite network’s architecture.
Definition 3.10 (ReLU activation function). The function
σ : R → R , x ↦ max ( 0 , x ) \sigma:\mathbb{R}\to\mathbb{R},\ x\mapsto\max(0,x) σ : R → R , x ↦ max ( 0 , x ) is called ReLU (Rectified Linear Unit) activation function.
The next lemma is an extended version to the n n n -dimensional output case of Theorem 4.1 found in [18 ] by applying their Lemma 2.9 and Lemma 2.11, such that the obtained network is a standard neural network, which is a network without having skip connections. They show in their Theorem 4.1 the existence of a constant c c c and a finite neural network φ ε f \varphi_{\varepsilon}^{f} φ ε f with one-dimensional output such that the network’s layers L ( φ ε f ) L(\varphi_{\varepsilon}^{f}) L ( φ ε f ) are bounded by L ( φ ε f ) ≤ c ⋅ log ( ε − s / ( s − r ) ) L(\varphi_{\varepsilon}^{f})\leq c\cdot\log(\varepsilon^{-s/(s-r)}) L ( φ ε f ) ≤ c ⋅ log ( ε − s / ( s − r ) ) , while the amount of total neurons N ( φ ε f ) N(\varphi_{\varepsilon}^{f}) N ( φ ε f ) is bounded by N ( φ ε f ) ≤ c ⋅ ε − d / ( s − r ) log ( ε − s / ( s − r ) ) N(\varphi_{\varepsilon}^{f})\leq c\cdot\varepsilon^{-d/(s-r)}\log(\varepsilon^{-s/(s-r)}) N ( φ ε f ) ≤ c ⋅ ε − d / ( s − r ) log ( ε − s / ( s − r ) ) . Such network may allow for skip connections which is not aligning with our setting as we want to apply Lemma B.10 and the Ψ \Psi Ψ -FNO does not allow for such connections. Further, we want to quantify the width of the finite neural network and not the total amount of neurons due to Equation (8) and Lemma B.10 . In the proof of their Lemma 2.11 they construct a standard neural network φ s t , ε f \varphi_{\mathrm{st},\varepsilon}^{f} φ st , ε f such that the output aligns with the output from the network φ ε f \varphi_{\varepsilon}^{f} φ ε f having skip connections. They show, while having the same amount of layers, that the standard network’s neurons per layer N l s t N_{l}^{\mathrm{st}} N l st , l = 1 , … , L l=1,\ldots,L l = 1 , … , L are bounded by N l s t ≤ 2 N ( φ f ) N_{l}^{\mathrm{st}}\leq2N(\varphi^{f}) N l st ≤ 2 N ( φ f ) , l = 1 , … , L l=1,\ldots,L l = 1 , … , L . The n n n -dimensional output case follows by placing n n n such networks φ i , ε f \varphi_{i,\varepsilon}^{f} φ i , ε f , i = 1 , … , n i=1,\ldots,n i = 1 , … , n with possible skip connections in parallel (cf. their Lemma 2.9) implying the existence of a single network φ ε f \varphi_{\varepsilon}^{f} φ ε f with n n n -dimensional output and
(i) L ( φ ε f ) ≤ max i = 1 , … , n L ( φ i , ε f ) ≤ max i = 1 , … , n { c i } log ( ε − s / ( s − r ) ) L(\varphi_{\varepsilon}^{f})\leq\max_{i=1,\ldots,n}L(\varphi_{i,\varepsilon}^{f})\leq\max_{i=1,\ldots,n}\{c_i\}\log(\varepsilon^{-s/(s-r)}) L ( φ ε f ) ≤ max i = 1 , … , n L ( φ i , ε f ) ≤ max i = 1 , … , n { c i } log ( ε − s / ( s − r ) ) ,
(ii) N ( φ ε f ) ≤ ∑ i = 1 n N ( φ i , ε f ) − ( n − 1 ) d ≤ n ⋅ max i = 1 , … , n { c i } ε − d / ( s − r ) log ( ε − s / ( s − r ) ) . N(\varphi_{\varepsilon}^{f})\leq\sum_{i=1}^{n}N(\varphi_{i,\varepsilon}^{f})-(n-1)d\leq n\cdot\max_{i=1,\ldots,n}\{c_i\}\varepsilon^{-d/(s-r)}\log(\varepsilon^{-s/(s-r)}). N ( φ ε f ) ≤ ∑ i = 1 n N ( φ i , ε f ) − ( n − 1 ) d ≤ n ⋅ max i = 1 , … , n { c i } ε − d / ( s − r ) log ( ε − s / ( s − r ) ) .
Using their Lemma 2.11 on φ ε f \varphi_{\varepsilon}^{f} φ ε f we obtain a standard neural network φ s t , ε f \varphi_{\mathrm{st},\varepsilon}^{f} φ st , ε f with n n n -dimensional output such that
L ( φ s t , ε f ) ≤ max i = 1 , … , n { c i } log ( ε − s / ( s − r ) ) L(\varphi_{\mathrm{st},\varepsilon}^{f})\leq\max_{i=1,\ldots,n}\{c_i\}\log(\varepsilon^{-s/(s-r)}) L ( φ st , ε f ) ≤ max i = 1 , … , n { c i } log ( ε − s / ( s − r ) ) ,
N l ( φ s t , ε f ) ≤ 2 N ( φ ε f ) ≤ 2 ⋅ n ⋅ max i = 1 , … , n { c i } ε − d / ( s − r ) log ( ε − s / ( s − r ) ) N_l(\varphi_{\mathrm{st},\varepsilon}^{f})\leq2N(\varphi_{\varepsilon}^{f})\leq2\cdot n\cdot\max_{i=1,\ldots,n}\{c_i\}\varepsilon^{-d/(s-r)}\log(\varepsilon^{-s/(s-r)}) N l ( φ st , ε f ) ≤ 2 N ( φ ε f ) ≤ 2 ⋅ n ⋅ max i = 1 , … , n { c i } ε − d / ( s − r ) log ( ε − s / ( s − r ) ) .
Denoting L ( φ s t , ε f ) L(\varphi_{\mathrm{st},\varepsilon}^{f}) L ( φ st , ε f ) by depth ( φ s t , ε f ) \operatorname{depth}(\varphi_{\mathrm{st},\varepsilon}^{f}) depth ( φ st , ε f ) and the maximum amount of neurons in a layer by width ( φ s t , ε f ) \operatorname{width}(\varphi_{\mathrm{st},\varepsilon}^{f}) width ( φ st , ε f ) , we end up with following result.
Lemma 3.11 (Quantitative approximation of ReLU neural network). Let d , n , s ∈ N d,n,s\in\mathbb{N} d , n , s ∈ N where s ≥ 2 s\geq2 s ≥ 2 with 1 ≤ p ≤ ∞ 1\leq p\leq\infty 1 ≤ p ≤ ∞ , E > 0 E>0 E > 0 , and 0 ≤ r ≤ 1 0\leq r\leq1 0 ≤ r ≤ 1 . Then, there exists a constant c R e L U = c R e L U ( d , n , s , p , E , r ) > 0 c_{\mathrm{ReLU}}=c_{\mathrm{ReLU}}(d,n,s,p,E,r)>0 c ReLU = c ReLU ( d , n , s , p , E , r ) > 0 with the following properties:
For any ε ∈ ( 0 , 1 / 2 ) \varepsilon\in(0,1/2) ε ∈ ( 0 , 1/2 ) there is a standard neural network architecture with d d d -dimensional input and n n n -dimensional output such that for any f ∈ { g ∈ W s , p ( ( 0 , 1 ) d ) : ∥ g ∥ W s , p ( ( 0 , 1 ) d ) ≤ E } f\in\{g\in W^{s,p}((0,1)^d):\|g\|_{W^{s,p}((0,1)^d)}\leq E\} f ∈ { g ∈ W s , p (( 0 , 1 ) d ) : ∥ g ∥ W s , p (( 0 , 1 ) d ) ≤ E } , there is a standard neural network φ s t , ε f \varphi_{\mathrm{st},\varepsilon}^{f} φ st , ε f that has said architecture such that
∥ φ s t , ε f − f ∥ W r , p ( ( 0 , 1 ) d ; R n ) ≤ ε , \|\varphi_{\mathrm{st},\varepsilon}^{f}-f\|_{W^{r,p}((0,1)^d;\mathbb{R}^{n})}\leq\varepsilon, ∥ φ st , ε f − f ∥ W r , p (( 0 , 1 ) d ; R n ) ≤ ε , where σ \sigma σ is the ReLU activation function and (i) depth ( φ s t , ε f ) ≤ c R e L U ⋅ log ( ε − s / ( s − r ) ) \operatorname{depth}(\varphi_{\mathrm{st},\varepsilon}^{f})\leq c_{\mathrm{ReLU}}\cdot\log(\varepsilon^{-s/(s-r)}) depth ( φ st , ε f ) ≤ c ReLU ⋅ log ( ε − s / ( s − r ) ) ,
(ii) width ( φ s t , ε f ) ≤ c R e L U ⋅ ε − d / ( s − r ) ⋅ log ( ε − s / ( s − r ) ) \operatorname{width}(\varphi_{\mathrm{st},\varepsilon}^{f})\leq c_{\mathrm{ReLU}}\cdot\varepsilon^{-d/(s-r)}\cdot\log(\varepsilon^{-s/(s-r)}) width ( φ st , ε f ) ≤ c ReLU ⋅ ε − d / ( s − r ) ⋅ log ( ε − s / ( s − r ) ) .
To conclude, finite neural networks using the ReLU activation function have under certain assumptions quantitative approximation results, which avoid the curse of dimensionality. However, in practice one commonly uses “smoother” versions of the ReLU activation. Thus, we consider an additional activation function to study how the approximation rates are affected. The activation function that we also consider is the Gaussian Error Linear Unit, or GELU for short, which was first introduced in [19 ] and is given here in its standard form as it can also be found in [67 ] .
Definition 3.12 (GELU activation function). We define
GELU ( x ) : = x 2 π ∫ − ∞ x e − t 2 / 2 d t . \operatorname{GELU}(x):=\frac{x}{\sqrt{2\pi}}\int_{-\infty}^{x}e^{-t^{2}/2}\,\mathrm{d}t. GELU ( x ) := 2 π x ∫ − ∞ x e − t 2 /2 d t .
The GELU activation function is smooth, meaning it is a C ∞ C^{\infty} C ∞ -function and therefore a standard neural network using the GELU function is also smooth. Analogue to the quantitative approximation statement using the ReLU function, we can find in the literature a similar result. Following theorem is from [67 ] where it is found as Theorem 3.
Lemma 3.13 (Quantitative approximation of GELU neural networks [67 ] ). Let d ∈ N d \in\mathbb{N} d ∈ N . For any f ∈ W s , ∞ ( [ 0 , 1 ] d ) f \in W^{s,\infty}([0,1]^d) f ∈ W s , ∞ ([ 0 , 1 ] d ) with 0 ≤ r < s 0 \le r < s 0 ≤ r < s , and for any M , L ∈ N M,L \in\mathbb{N} M , L ∈ N , there exists a GELU-neural network ϕ \phi ϕ with depth c 1 , G E L U L log L c_{1,\mathrm{GELU}}L \log L c 1 , GELU L log L and width c 2 , G E L U M log M c_{2,\mathrm{GELU}}M \log M c 2 , GELU M log M such that
∥ f − ϕ ∥ W r , ∞ ( [ 0 , 1 ] d ) ≤ c 0 , G E L U ∥ f ∥ W s , ∞ ( [ 0 , 1 ] d ) M − 2 ( s − r ) / d L − 2 ( s − r ) / d , \|f-\phi\|_{W^{r,\infty}([0,1]^d)} \le c_{0,\mathrm{GELU}}\|f\|_{W^{s,\infty}([0,1]^d)}M^{-2(s-r)/d}L^{-2(s-r)/d}, ∥ f − ϕ ∥ W r , ∞ ([ 0 , 1 ] d ) ≤ c 0 , GELU ∥ f ∥ W s , ∞ ([ 0 , 1 ] d ) M − 2 ( s − r ) / d L − 2 ( s − r ) / d , where the constants c 0 , G E L U , c 1 , G E L U , c 2 , G E L U > 0 c_{0,\mathrm{GELU}},c_{1,\mathrm{GELU}},c_{2,\mathrm{GELU}}>0 c 0 , GELU , c 1 , GELU , c 2 , GELU > 0 are independent of M M M and L L L .
To apply the last lemmata on our case of interest, consider f f f from Assumption 2.5 with d = 1 + n y + n u d=1+n_y+n_u d = 1 + n y + n u , and specify r = 1 r=1 r = 1 in Lemma 3.11 while choosing a ε R e L U > 0 \varepsilon_{\mathrm{ReLU}}>0 ε ReLU > 0 according to Equation (C.2). Thus, we obtain the existence of a standard ReLU neural network f R e L U : R d → R n y f_{\mathrm{ReLU}}:\mathbb{R}^{d}\to\mathbb{R}^{n_y} f ReLU : R d → R n y with
∥ f R e L U − f ∥ L ∞ ( D ′ ; R n y ) ≤ ∥ f R e L U − f ∥ W 1 , ∞ ( D ′ ; R n y ) ≤ ε R e L U , \|f_{\mathrm{ReLU}}-f\|_{L^\infty(D';\mathbb{R}^{n_y})}\le\|f_{\mathrm{ReLU}}-f\|_{W^{1,\infty}(D';\mathbb{R}^{n_y})}\le\varepsilon_{\mathrm{ReLU}}, ∥ f ReLU − f ∥ L ∞ ( D ′ ; R n y ) ≤ ∥ f ReLU − f ∥ W 1 , ∞ ( D ′ ; R n y ) ≤ ε ReLU ,
and
depth ( f R e L U ) ≤ c R e L U ⋅ log ( ε R e L U − s / ( s − 1 ) ) , \operatorname{depth}(f_{\mathrm{ReLU}})\le c_{\mathrm{ReLU}}\cdot\log\left(\varepsilon_{\mathrm{ReLU}}^{-s/(s-1)}\right), depth ( f ReLU ) ≤ c ReLU ⋅ log ( ε ReLU − s / ( s − 1 ) ) ,
width ( f R e L U ) ≤ c R e L U ⋅ ε R e L U − d / ( s − 1 ) ⋅ log ( ε R e L U − s / ( s − 1 ) ) , \operatorname{width}(f_{\mathrm{ReLU}})\le c_{\mathrm{ReLU}}\cdot\varepsilon_{\mathrm{ReLU}}^{-d/(s-1)}\cdot\log\left(\varepsilon_{\mathrm{ReLU}}^{-s/(s-1)}\right), width ( f ReLU ) ≤ c ReLU ⋅ ε ReLU − d / ( s − 1 ) ⋅ log ( ε ReLU − s / ( s − 1 ) ) ,
where the constant c R e L U > 0 c_{\mathrm{ReLU}}>0 c ReLU > 0 is now further dependent on the domain D ′ D' D ′ . Now for the GELU case, we again choose r = 1 r=1 r = 1 and apply Lemma 3.13 . To simplify the numerical analysis, we set the width of the GELU network as fixed with value M 0 ∈ N M_0\in\mathbb{N} M 0 ∈ N and study the approximation rates under the influence regarding the amount of layers. We observe that there then exists a constant c 0 , G E L U = c 0 , G E L U ( M 0 , E ) > 0 c_{0,\mathrm{GELU}}=c_{0,\mathrm{GELU}}(M_0,E)>0 c 0 , GELU = c 0 , GELU ( M 0 , E ) > 0 such that
∥ f G E L U − f ∥ L ∞ ( D ′ ; R n y ) ≤ ∥ f G E L U − f ∥ W 1 , ∞ ( D ′ ; R n y ) ≤ c 0 , G E L U L − 2 ( s − 1 ) / d , \|f_{\mathrm{GELU}}-f\|_{L^\infty(D';\mathbb{R}^{n_y})}\le\|f_{\mathrm{GELU}}-f\|_{W^{1,\infty}(D';\mathbb{R}^{n_y})}\le c_{0,\mathrm{GELU}}L^{-2(s-1)/d}, ∥ f GELU − f ∥ L ∞ ( D ′ ; R n y ) ≤ ∥ f GELU − f ∥ W 1 , ∞ ( D ′ ; R n y ) ≤ c 0 , GELU L − 2 ( s − 1 ) / d ,
where we set ε G E L U : = c 0 , G E L U L − 2 ( s − 1 ) / d \varepsilon_{\mathrm{GELU}}:=c_{0,\mathrm{GELU}}L^{-2(s-1)/d} ε GELU := c 0 , GELU L − 2 ( s − 1 ) / d and L ∈ N L\in\mathbb{N} L ∈ N chosen as in Equation (C.3) for fixed s ≥ 2 s\ge2 s ≥ 2 and d = 1 + n y + n u d=1+n_y+n_u d = 1 + n y + n u . In the following we will start to use the notation of
f n e t ∈ { f R e L U , f G E L U } , ε n e t ∈ { ε R e L U , ε G E L U } , (9) f_{\mathrm{net}}\in\{f_{\mathrm{ReLU}},f_{\mathrm{GELU}}\},\quad\varepsilon_{\mathrm{net}}\in\{\varepsilon_{\mathrm{ReLU}},\varepsilon_{\mathrm{GELU}}\}, \tag*{(9)} f net ∈ { f ReLU , f GELU } , ε net ∈ { ε ReLU , ε GELU } , (9)
which refers to above obtained ReLU, as well as GELU neural network, for the approximation of f f f .
Approximate Picard iteration of Equation (3)
As earlier mentioned, and as one can observe in the proof of Theorem 2.1 , the CODE can be solved via Picard iteration by introducing the so-called Picard operator. Therefore, let us formally introduce said operator.
Note that the CODE is well defined on the time interval T = [ 0 , 2 π ] \mathbb{T}=[0,2\pi] T = [ 0 , 2 π ] . This allows us to define the Picard operator also on T \mathbb{T} T , as we see in the following. Define
Φ : C ( T ; R n y ) → C ( T ; R n y ) , [ Φ ( y ; u , y 0 ) ] ( t ) : = y 0 + ∫ t ^ 1 t f ( y , u ) d s , (10) \Phi:C(\mathbb{T};\mathbb{R}^{n_y})\to C(\mathbb{T};\mathbb{R}^{n_y}),\quad[\Phi(y;u,y^0)](t):=y^0+\int_{\hat{t}_1}^{t}f(y,u)\,\mathrm{d}s, \tag*{(10)} Φ : C ( T ; R n y ) → C ( T ; R n y ) , [ Φ ( y ; u , y 0 )] ( t ) := y 0 + ∫ t ^ 1 t f ( y , u ) d s , (10)
where u ∈ U l , T , A u\in\mathcal{U}^{l,\mathbb{T},A} u ∈ U l , T , A , l ≥ 2 l\ge2 l ≥ 2 and t ^ 1 ∈ T \hat{t}_1\in\mathbb{T} t ^ 1 ∈ T with J = ( t ^ 1 , t ^ 2 ) J=(\hat{t}_1,\hat{t}_2) J = ( t ^ 1 , t ^ 2 ) . To keep notation simple we sometimes write v : = ( u , y 0 ) v:=(u,y^0) v := ( u , y 0 ) and start to use either Φ ( y ; v ) \Phi(y;v) Φ ( y ; v ) or Φ ( y ; u , y 0 ) \Phi(y;u,y^0) Φ ( y ; u , y 0 ) . Further in the appendix, the Corollary A.3 establishes that Φ \Phi Φ defines a κ \kappa κ -contraction with κ < 1 / 2 \kappa<1/2 κ < 1/2 .
To conclude this section, we state the aforementioned Picard type fixed point iteration scheme that produces an approximate solution to the true solution of the CODE. This iteration scheme has the advantage to be emulated through the Ψ \Psi Ψ -FNO layers introduced earlier. To study the convergence of the proposed algorithm, we define the corresponding Picard operator. For N ∈ N N\in\mathbb{N} N ∈ N we define the operator Φ N , n e t : L N 2 ( T ; R n y ) → L N 2 ( T ; R n y ) \Phi_{N,\mathrm{net}}:L_N^2(\mathbb{T};\mathbb{R}^{n_y})\to L_N^2(\mathbb{T};\mathbb{R}^{n_y}) Φ N , net : L N 2 ( T ; R n y ) → L N 2 ( T ; R n y ) by
[ Φ N , n e t ( y N , n e t ; v N ) ] ( t ) : = y 0 + ∫ t ^ 1 t P ˙ N L N f n e t ( y N , n e t , u N ) d s + c 0 ( τ N ( t ) − t ^ 1 ) (11) [\Phi_{N,\mathrm{net}}(y_{N,\mathrm{net}};v_N)](t):=y^0+\int_{\hat{t}_1}^{t}\dot{P}_N L_N f_{\mathrm{net}}(y_{N,\mathrm{net}},u_N)\,\mathrm{d}s+c_0(\tau_N(t)-\hat{t}_1) \tag*{(11)} [ Φ N , net ( y N , net ; v N )] ( t ) := y 0 + ∫ t ^ 1 t P ˙ N L N f net ( y N , net , u N ) d s + c 0 ( τ N ( t ) − t ^ 1 ) (11)
for u ∈ U l , T , A u\in\mathcal{U}^{l,\mathbb{T},A} u ∈ U l , T , A and v N : = ( u N , y 0 , τ N − t ^ 1 ) v_N:=(u_N,y^0,\tau_N-\hat{t}_1) v N := ( u N , y 0 , τ N − t ^ 1 ) , u N : = P N u u_N:=P_Nu u N := P N u , τ N : = P N τ \tau_N:=P_N\tau τ N := P N τ , c 0 : = ∫ T L N f n e t ( y N , n e t , u N ) d s c_0:=\int_{\mathbb{T}}L_Nf_{\mathrm{net}}(y_{N,\mathrm{net}},u_N)\,\mathrm{d}s c 0 := ∫ T L N f net ( y N , net , u N ) d s . We use following notation to state the K K K -times composition of the above operators
Φ [ K ] = Φ ∘ Φ ∘ … ∘ Φ ⏟ K -times , \Phi^{[K]}=\underbrace{\Phi\circ\Phi\circ\ldots\circ\Phi}_{K\text{-times}}, Φ [ K ] = K -times Φ ∘ Φ ∘ … ∘ Φ ,
analogue for Φ N , n e t [ K ] \Phi^{[K]}_{N,\mathrm{net}} Φ N , net [ K ] .
We introduce, as the domain to study the convergence on, the set
B N : = { z ∈ L N 2 ( J ; R n y ) ∣ ∥ z ∥ W 1 , ∞ ( T ; R n y ) ≤ Y ′ } . \mathcal{B}_N := \left\{z \in L_N^2(J;\mathbb{R}^{n_y}) \mid\|z\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})} \le Y'\right\}. B N := { z ∈ L N 2 ( J ; R n y ) ∣ ∥ z ∥ W 1 , ∞ ( T ; R n y ) ≤ Y ′ } .
meaning that we consider (restrictions to J J J of) trigonometric polynomials whose W 1 , ∞ ( T ; R n y ) W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y}) W 1 , ∞ ( T ; R n y ) -norm is at most Y ′ Y' Y ′ . Here L N 2 ( J ; R n y ) : = { z ∣ J ∣ z ∈ L N 2 ( T ; R n y ) } L_N^2(J;\mathbb{R}^{n_y}) := \left\{z|_J \mid z \in L_N^2(\mathbb{T};\mathbb{R}^{n_y})\right\} L N 2 ( J ; R n y ) := { z ∣ J ∣ z ∈ L N 2 ( T ; R n y ) } ; since trigonometric polynomials are real-analytic, z z z is uniquely determined by its restriction z ∣ J z|_J z ∣ J . We identify both, so that B N ⊂ C ( J ; R n y ) \mathcal{B}_N \subset C(J;\mathbb{R}^{n_y}) B N ⊂ C ( J ; R n y ) and operators defined on L N 2 ( T ; R n y ) L_N^2(\mathbb{T};\mathbb{R}^{n_y}) L N 2 ( T ; R n y ) act on B N \mathcal{B}_N B N .
Following algorithm is the Picard reformulation of CODE while allowing for being emulated through the layers of a Ψ \Psi Ψ -FNO.
Algorithm 3.14 (Approximate solution of CODE). Let N ∈ N N \in\mathbb{N} N ∈ N .
Input: control u ∈ U l , J , A u \in\mathcal{U}^{l,J,A} u ∈ U l , J , A , l ≥ 2 l \ge2 l ≥ 2 , initial value y 0 = y ( t ^ 1 ) ∈ Y 0 y^0 = y(\hat{t}_1) \in Y^0 y 0 = y ( t ^ 1 ) ∈ Y 0 , shifted identity approximation τ N − t ^ 1 \tau_N-\hat{t}_1 τ N − t ^ 1 .
Output: y N , n e t ∈ L N 2 ( T ; R n y ) y_{N,\mathrm{net}} \in L_N^2(\mathbb{T};\mathbb{R}^{n_y}) y N , net ∈ L N 2 ( T ; R n y ) such that y N , n e t ∣ J ≈ y ^ y_{N,\mathrm{net}}|_J \approx\hat{y} y N , net ∣ J ≈ y ^ where y ^ \hat{y} y ^ solves y ^ = Φ ( y ^ ; u , y 0 ) ∣ J \hat{y}=\Phi(\hat{y};u,y^0)|_J y ^ = Φ ( y ^ ; u , y 0 ) ∣ J .
(1) Compute u N : = P N ( τ , E u ) ∈ L N 2 ( T ; R 1 + n u ) u_N := P_N(\tau,\mathcal{E}u) \in L_N^2(\mathbb{T};\mathbb{R}^{1+n_u}) u N := P N ( τ , E u ) ∈ L N 2 ( T ; R 1 + n u ) .
(2) Set
K : = ⌈ log ( Y ′ N ) log ( 1 / κ ) ⌉ . K := \left\lceil\frac{\log(Y'N)}{\log(1/\kappa)} \right\rceil. K := ⌈ log ( 1/ κ ) log ( Y ′ N ) ⌉ . (3) Initial value y N , n e t ( 0 ) ( t ) : = 0 ∈ B N ⊂ L N 2 ( T ; R n y ) y_{N,\mathrm{net}}^{(0)}(t) := 0 \in\mathcal{B}_N \subset L_N^2(\mathbb{T};\mathbb{R}^{n_y}) y N , net ( 0 ) ( t ) := 0 ∈ B N ⊂ L N 2 ( T ; R n y ) .
(4) For k = 1 , … , K k=1,\ldots,K k = 1 , … , K calculate
y N , n e t ( k ) ( t ) = y 0 + ∫ t ^ 1 t P N I N f n e t ( y N , n e t ( k − 1 ) , u N ) d s + c 0 , k ( τ N ( t ) − t ^ 1 ) ∈ L N 2 ( T ; R n y ) , y_{N,\mathrm{net}}^{(k)}(t) = y^0 + \int_{\hat{t}_1}^{t} P_N I_N f_{\mathrm{net}}\left(y_{N,\mathrm{net}}^{(k-1)},u_N\right) \, \mathrm{d}s + c_{0,k}\left(\tau_N(t)-\hat{t}_1\right) \in L_N^2(\mathbb{T};\mathbb{R}^{n_y}), y N , net ( k ) ( t ) = y 0 + ∫ t ^ 1 t P N I N f net ( y N , net ( k − 1 ) , u N ) d s + c 0 , k ( τ N ( t ) − t ^ 1 ) ∈ L N 2 ( T ; R n y ) , where
c 0 , k : = ∫ 0 2 π I N f n e t ( y N , n e t ( k − 1 ) , u N ) d s . c_{0,k} := \int_0^{2\pi} I_N f_{\mathrm{net}}\left(y_{N,\mathrm{net}}^{(k-1)},u_N\right) \, \mathrm{d}s. c 0 , k := ∫ 0 2 π I N f net ( y N , net ( k − 1 ) , u N ) d s . (5) Set y N , n e t : = y N , n e t ( K ) y_{N,\mathrm{net}} := y_{N,\mathrm{net}}^{(K)} y N , net := y N , net ( K ) .
The well-definedness of above algorithm is proven in the beginning of Appendix § 10 . Further, note that the algorithm produces an approximate solution of the CODE. That this approximate converges to the true solution is shown in Theorem C.20 .
Quantitative approximation theorem
The following quantitative approximation theorem for a Ψ \Psi Ψ -FNO approximating the solution operator of a CODE is our main result.
Theorem 4.1. Let Assumption 2.5 hold, including the smallness condition Equation (C.1), and let d = 1 + n y + n u d=1+n_y+n_u d = 1 + n y + n u . There are constants ε 0 ∈ ( 0 , 1 / 2 ] \varepsilon_0 \in(0,1/2] ε 0 ∈ ( 0 , 1/2 ] and C > 0 C>0 C > 0 depending on n y n_y n y , n u n_u n u , E E E , α \alpha α , m m m , s s s , l l l , A A A , Y 0 Y^0 Y 0 , γ \gamma γ , J J J , on the fixed extension operator E \mathcal{E} E and on the GELU width parameter M 0 M_0 M 0 , but not on ε \varepsilon ε , such that for every ε ∈ ( 0 , ε 0 ) \varepsilon\in(0,\varepsilon_0) ε ∈ ( 0 , ε 0 ) there exist a N = N ( ε ) ≤ C ε − 1 N=N(\varepsilon) \le C\varepsilon^{-1} N = N ( ε ) ≤ C ε − 1 and two Ψ \Psi Ψ -FNOs N ^ ∈ { N ^ R e L U , N ^ G E L U } \widehat{\mathcal{N}} \in\{\widehat{\mathcal{N}}_{\mathrm{ReLU}},\widehat{\mathcal{N}}_{\mathrm{GELU}}\} N ∈ { N ReLU , N GELU } , N ^ : L N 2 ( T ; R ( 1 + n u ) + n y + 2 ) → L N 2 ( T ; R n y ) \widehat{\mathcal{N}}:L_N^2(\mathbb{T};\mathbb{R}^{(1+n_u)+n_y+2}) \to L_N^2(\mathbb{T};\mathbb{R}^{n_y}) N : L N 2 ( T ; R ( 1 + n u ) + n y + 2 ) → L N 2 ( T ; R n y ) with
sup y 0 ∈ Y 0 , u ∈ U l , J , A ∥ y ^ ( ⋅ ; u , y 0 ) − N ^ ( u N , y 0 , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) ∥ C ( J ; R n y ) ≤ ε , \sup_{\substack{y^0 \in Y^0,u \in\mathcal{U}^{l,J,A}}} \left\|\hat{y}(\cdot;u,y^0)-\widehat{\mathcal{N}}\left(u_N,y^0,\tau_N-\hat{t}_1,D_N(\hat{t}_1-\cdot)\right)\right\|_{C(J;\mathbb{R}^{n_y})} \le\varepsilon, y 0 ∈ Y 0 , u ∈ U l , J , A sup y ^ ( ⋅ ; u , y 0 ) − N ( u N , y 0 , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) C ( J ; R n y ) ≤ ε , where y ^ ( ⋅ ; u , y 0 ) \hat{y}(\cdot;u,y^0) y ^ ( ⋅ ; u , y 0 ) solves y ˙ = f ( y , u ) \dot{y}=f(y,u) y ˙ = f ( y , u ) on J J J with y ( t ^ 1 ) = y 0 y(\hat{t}_1)=y^0 y ( t ^ 1 ) = y 0 , u N : = P N ( τ , E u ) ∈ L N 2 ( T ; R 1 + n u ) u_N:=P_N(\tau,\mathcal{E}u)\in L_N^2(\mathbb{T};\mathbb{R}^{1+n_u}) u N := P N ( τ , E u ) ∈ L N 2 ( T ; R 1 + n u ) is the projected periodically extended control with first component τ N = P N τ \tau_N=P_N\tau τ N = P N τ , D N D_N D N is the Dirichlet kernel and y 0 y^0 y 0 enters as a constant function. Moreover,
size ( N ^ R e L U ) ≤ C ( 1 ε ) 2 d s − 1 + 1 ( log ( 1 ε ) ) 4 , size ( N ^ G E L U ) ≤ C ( 1 ε ) d 2 ( s − 1 ) + 1 ( log ( 1 ε ) ) 2 , \operatorname{size}(\widehat{\mathcal{N}}_{\mathrm{ReLU}}) \le C\left(\frac{1}{\varepsilon}\right)^{\frac{2d}{s-1}+1}\left(\log\left(\frac{1}{\varepsilon}\right)\right)^4,\qquad
\operatorname{size}(\widehat{\mathcal{N}}_{\mathrm{GELU}}) \le C\left(\frac{1}{\varepsilon}\right)^{\frac{d}{2(s-1)}+1}\left(\log\left(\frac{1}{\varepsilon}\right)\right)^2, size ( N ReLU ) ≤ C ( ε 1 ) s − 1 2 d + 1 ( log ( ε 1 ) ) 4 , size ( N GELU ) ≤ C ( ε 1 ) 2 ( s − 1 ) d + 1 ( log ( ε 1 ) ) 2 , where subscript indicates whether f f f is approximated by ReLU or a GELU network. In the case of ReLU, the remaining layers of N ^ \widehat{\mathcal{N}} N use the GELU activation function, see Remark C.21.
We see that we obtain similar results as in [27 ] for their approximation regarding the Navier-Stokes equation, however, the log-terms here are taken to the power of four when using a ReLU network to approximate f f f , while they are only squared when using the GELU activation function. Neglecting the log-terms in both cases, we see that the size of the Ψ \Psi Ψ -FNO grows at most algebraically in 1 / ε 1/\varepsilon 1/ ε and that the size of N ^ R e L U \widehat{\mathcal{N}}_{\mathrm{ReLU}} N ReLU scales even sub-quadratically for the case s > 2 d + 1 s > 2d+1 s > 2 d + 1 , while the size of N ^ G E L U \widehat{\mathcal{N}}_{\mathrm{GELU}} N GELU scales at most sub-quadratically already when s > d / 2 + 1 s > d/2+1 s > d /2 + 1 . The latter requirement is favorable compared to s > 2 d + 1 s > 2d+1 s > 2 d + 1 , due to the fact that d d d is the amount of control and solution functions of a CODE system. In complex physical simulations the amount of control and solution functions can be in the range of hundreds. Nonetheless, the curse of parametric complexity is avoided in Theorem 4.1.
Application to inverse parameter estimation
Motivated by the efficient approximation of the constructed FNO, we show how one can apply a FNO surrogate model in an inverse parameter estimation problem where a large data set of measurement data is available for inversion.
Consider for an interval J ⊂ ( 0 , 2 π ) J \subset(0,2\pi) J ⊂ ( 0 , 2 π ) the case where one has the solution operator
S : U l , J , A × Y 0 × ∏ i = 1 n x [ x ^ 1 , i , x ^ 2 , i ] → C ( J ; R n y ) S : U^{l,J,A} \times Y^0 \times\prod_{i=1}^{n_x} [\hat{x}_{1,i},\hat{x}_{2,i}] \to C(J;\mathbb{R}^{n_y}) S : U l , J , A × Y 0 × i = 1 ∏ n x [ x ^ 1 , i , x ^ 2 , i ] → C ( J ; R n y )
coming from a CODE system as described in the previous sections. Here, we introduce an additional parameter dependence on x ∈ ∏ i = 1 n x [ x ^ 1 , i , x ^ 2 , i ] ⊂ R n x x \in\prod_{i=1}^{n_x}[\hat{x}_{1,i},\hat{x}_{2,i}] \subset\mathbb{R}^{n_x} x ∈ ∏ i = 1 n x [ x ^ 1 , i , x ^ 2 , i ] ⊂ R n x . One can think of S S S being implemented by a complex computer simulation. Then, in the presence of measurement data, it is common to infer an inverse parameter estimation such that the computer simulation provides accurate predictions describing the real underlying physical process [9 ] . One possible bottleneck for the parameter inversion are long inference times. Ensemble methods are derivative-free and only require evaluations of the forward operator, which the surrogate provides at low cost. Therefore, we propose the following method, which combines the EKI introduced in [22 ] , in the form used in [23 ] , with the Kaczmarz method from [36 ] .
Consider a physical process described by S S S under a fixed x † ∈ ∏ i = 1 n x [ x ^ 1 , i , x ^ 2 , i ] ⊂ R n x x^\dagger\in\prod_{i=1}^{n_x}[\hat{x}_{1,i},\hat{x}_{2,i}] \subset\mathbb{R}^{n_x} x † ∈ ∏ i = 1 n x [ x ^ 1 , i , x ^ 2 , i ] ⊂ R n x , n x ∈ N n_x \in\mathbb{N} n x ∈ N . If said process is run under different control functions and initial values, we have a data set consisting of N e x p ∈ N N_{\mathrm{exp}} \in\mathbb{N} N exp ∈ N real experiments
D e x p : = { ( u l , y l 0 , y l δ ) } l = 1 , … , N e x p , D_{\mathrm{exp}} := \{(u_l,y_l^0,y_l^\delta)\}_{l=1,\ldots,N_{\mathrm{exp}}}, D exp := {( u l , y l 0 , y l δ ) } l = 1 , … , N exp ,
while an observation operator
O : C ( J ; R n y ) → R n m e a s , n m e a s ∈ N , y ↦ ( y c r ( t r m ) ) r = 1 n m e a s \mathcal{O}: C(J;\mathbb{R}^{n_y}) \to\mathbb{R}^{n_{\mathrm{meas}}},\quad n_{\mathrm{meas}} \in\mathbb{N},\quad y \mapsto(y_{c_r}(t_r^m))_{r=1}^{n_{\mathrm{meas}}} O : C ( J ; R n y ) → R n meas , n meas ∈ N , y ↦ ( y c r ( t r m ) ) r = 1 n meas
maps the continuous simulation output to the discrete measurements per experiment where y c r y_{c_r} y c r denotes the measurement of component c r c_r c r at the time point t r m ∈ J t_r^m \in J t r m ∈ J . Varying the parameters x x x , the controls and initial values y l 0 y_l^0 y l 0 , leads to a data set
D s i m : = { ( u l , y l 0 , x , y l ) } l = 1 , … , N s i m , N s i m ∈ N D_{\mathrm{sim}} := \{(u_l,y_l^0,x,y_l)\}_{l=1,\ldots,N_{\mathrm{sim}}},\quad N_{\mathrm{sim}} \in\mathbb{N} D sim := {( u l , y l 0 , x , y l ) } l = 1 , … , N sim , N sim ∈ N
generated by the simulation. This data set is used for a FNO N ^ \widehat{\mathcal{N}} N that serves as a surrogate for the simulation. For doing the approximation with a FNO, one treats the parameters x x x as additional but constant controls and thus they serve as input functions to the FNO. This allows us to introduce a forward operator per experiment, i.e. for i = 1 , … , N e x p i=1,\ldots,N_{\mathrm{exp}} i = 1 , … , N exp ,
F i : ∏ l = 1 n x [ x ^ 1 , i , x ^ 2 , i ] ⊂ R n x → R n m e a s , x ↦ O ∘ N ^ ( ( u i , y i 0 , x ) ) , F_i : \prod_{l=1}^{n_x}[\hat{x}_{1,i},\hat{x}_{2,i}] \subset\mathbb{R}^{n_x} \to\mathbb{R}^{n_{\mathrm{meas}}},\quad x \mapsto\mathcal{O}\circ\widehat{\mathcal{N}}((u_i,y_i^0,x)), F i : l = 1 ∏ n x [ x ^ 1 , i , x ^ 2 , i ] ⊂ R n x → R n meas , x ↦ O ∘ N (( u i , y i 0 , x )) ,
where the influence of the parameters x x x is studied under fixed control and initial value. To formulate several multiple forward operators is inspired by [36 ] . We also can define one single forward operator
F : ∏ i = 1 n x [ x ^ 1 , i , x ^ 2 , i ] ⊂ R n x → R N e x p × n m e a s , x ↦ ( O ∘ N ^ ( ( u 1 , y 1 0 , x ) ) , … , O ∘ N ^ ( ( u N e x p , y N e x p 0 , x ) ) ) \mathbf{F} : \prod_{i=1}^{n_x}[\hat{x}_{1,i},\hat{x}_{2,i}] \subset\mathbb{R}^{n_x} \to\mathbb{R}^{N_{\mathrm{exp}}\times n_{\mathrm{meas}}},\quad x \mapsto\left(\mathcal{O}\circ\widehat{\mathcal{N}}((u_1,y_1^0,x)),\ldots,\mathcal{O}\circ\widehat{\mathcal{N}}((u_{N_{\mathrm{exp}}},y_{N_{\mathrm{exp}}}^0,x))\right) F : i = 1 ∏ n x [ x ^ 1 , i , x ^ 2 , i ] ⊂ R n x → R N exp × n meas , x ↦ ( O ∘ N (( u 1 , y 1 0 , x )) , … , O ∘ N (( u N exp , y N exp 0 , x )) )
for a more concise notation. From a mathematical point of view, we are interested in solving the inverse problem
F ( x ) = y δ ∈ R N e x p × n m e a s , \mathbf{F}(x)=y^\delta\in\mathbb{R}^{N_{\mathrm{exp}}\times n_{\mathrm{meas}}}, F ( x ) = y δ ∈ R N exp × n meas ,
where y δ y^\delta y δ denotes the set of all real measurements with
y δ : = ( y 1 δ , … , y N e x p δ ) , y i δ ∈ R n m e a s , y^\delta:= (y_1^\delta,\ldots,y_{N_{\mathrm{exp}}}^\delta),\quad y_i^\delta\in\mathbb{R}^{n_{\mathrm{meas}}}, y δ := ( y 1 δ , … , y N exp δ ) , y i δ ∈ R n meas ,
where it is already hinted at the presence of noise in the data measurements. Depending on the amount of discrete measurements done per experiment, the number N e x p ⋅ n m e a s N_{\mathrm{exp}} \cdot n_{\mathrm{meas}} N exp ⋅ n meas already depicts that a parameter inversion may not be feasible when done under a solution operator with long computation time. For each measurement per experiment, we assume following relationship
y i δ = F i ( x † ) + ε i , a p p r o x + ε i , s i m + ε i , m e a s , i = 1 , … , N e x p , y_i^\delta= F_i(x^\dagger) + \varepsilon_{i,\mathrm{approx}} + \varepsilon_{i,\mathrm{sim}} + \varepsilon_{i,\mathrm{meas}},\quad i=1,\ldots,N_{\mathrm{exp}}, y i δ = F i ( x † ) + ε i , approx + ε i , sim + ε i , meas , i = 1 , … , N exp ,
where
ε i , a p p r o x ∼ N ( ε ‾ a p p r o x , Γ a p p r o x ) , ε i , s i m ∼ N ( ε ‾ s i m , Γ s i m ) , ε i , m e a s ∼ N ( 0 , Γ m e a s ) \varepsilon_{i,\mathrm{approx}} \sim N(\overline{\varepsilon}_{\mathrm{approx}},\Gamma_{\mathrm{approx}}),\quad\varepsilon_{i,\mathrm{sim}} \sim N(\overline{\varepsilon}_{\mathrm{sim}},\Gamma_{\mathrm{sim}}),\quad\varepsilon_{i,\mathrm{meas}} \sim N(0,\Gamma_{\mathrm{meas}}) ε i , approx ∼ N ( ε approx , Γ approx ) , ε i , sim ∼ N ( ε sim , Γ sim ) , ε i , meas ∼ N ( 0 , Γ meas )
are respectively the approximation-, simulation- and measurement error. We assume that the three errors are Gaussian, mutually independent and independent of the inputs, so that their covariances add up as below. Note that we assume a simulation error, which is not uncommon in the theory of computer model calibration [25 ] . We can now consider the mean values ε ‾ a p p r o x , ε ‾ s i m \overline{\varepsilon}_{\mathrm{approx}},\overline{\varepsilon}_{\mathrm{sim}} ε approx , ε sim and rewrite the relation as
y i δ − ε ‾ a p p r o x − ε ‾ s i m = F i ( x † ) + ζ i , i = 1 , … , N e x p , y_i^\delta-\overline{\varepsilon}_{\mathrm{approx}}-\overline{\varepsilon}_{\mathrm{sim}}=F_i(x^\dagger)+\zeta_i,\quad i=1,\ldots,N_{\mathrm{exp}}, y i δ − ε approx − ε sim = F i ( x † ) + ζ i , i = 1 , … , N exp ,
where
ε i , a p p r o x − ε ‾ a p p r o x + ε i , s i m − ε ‾ s i m + ε i , m e a s = : ζ i ∼ N ( 0 , Γ y ) \varepsilon_{i,\mathrm{approx}}-\overline{\varepsilon}_{\mathrm{approx}}+\varepsilon_{i,\mathrm{sim}}-\overline{\varepsilon}_{\mathrm{sim}}+\varepsilon_{i,\mathrm{meas}}=:\zeta_i\sim N(0,\Gamma_y) ε i , approx − ε approx + ε i , sim − ε sim + ε i , meas =: ζ i ∼ N ( 0 , Γ y )
with an inflated noise covariance matrix defined as
Γ a p p r o x + Γ s i m + Γ m e a s = : Γ y ∈ R n m e a s × n m e a s . \Gamma_{\mathrm{approx}}+\Gamma_{\mathrm{sim}}+\Gamma_{\mathrm{meas}}=:\Gamma_y\in\mathbb{R}^{n_{\mathrm{meas}}\times n_{\mathrm{meas}}}. Γ approx + Γ sim + Γ meas =: Γ y ∈ R n meas × n meas .
We end up with shifted measurement data set denoted by y ζ ∈ R N e x p × n m e a s y^\zeta\in\mathbb{R}^{N_{\mathrm{exp}}\times n_{\mathrm{meas}}} y ζ ∈ R N exp × n meas . By y i ζ ∈ R n m e a s y_i^\zeta\in\mathbb{R}^{n_{\mathrm{meas}}} y i ζ ∈ R n meas , i = 1 , … , N e x p i=1,\ldots,N_{\mathrm{exp}} i = 1 , … , N exp , we denote the shifted data measured per experiment for which we assume, a common noise level
∥ Γ y − 1 / 2 ( y i ζ − F i ( x † ) ) ∥ ≤ η ∈ R + . \left\|\Gamma_y^{-1/2}(y_i^\zeta-F_i(x^\dagger))\right\|\leq\eta\in\mathbb{R}^{+}. Γ y − 1/2 ( y i ζ − F i ( x † )) ≤ η ∈ R + .
Since ∥ Γ y − 1 / 2 ζ i ∥ 2 \|\Gamma_y^{-1/2}\zeta_i\|^2 ∥ Γ y − 1/2 ζ i ∥ 2 is χ 2 \chi^2 χ 2 -distributed with n m e a s n_{\mathrm{meas}} n meas degrees of freedom, such a bound can only hold with a prescribed probability. For δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , the choice η 2 : = n m e a s + 2 n m e a s log ( N e x p / δ ) + 2 log ( N e x p / δ ) \eta^2:=n_{\mathrm{meas}}+2\sqrt{n_{\mathrm{meas}}\log(N_{\mathrm{exp}}/\delta)}+2\log(N_{\mathrm{exp}}/\delta) η 2 := n meas + 2 n meas log ( N exp / δ ) + 2 log ( N exp / δ ) guarantees, by the tail bound in [35 Lemma 1] and a union bound over the experiments, that the noise level holds for all i = 1 , … , N e x p i=1,\ldots,N_{\mathrm{exp}} i = 1 , … , N exp simultaneously with probability at least 1 − δ 1-\delta 1 − δ . Note that E ∥ Γ y − 1 / 2 ζ i ∥ 2 = n m e a s \mathbb{E}\|\Gamma_y^{-1/2}\zeta_i\|^2=n_{\mathrm{meas}} E ∥ Γ y − 1/2 ζ i ∥ 2 = n meas , so the naive choice η = n m e a s \eta=\sqrt{n_{\mathrm{meas}}} η = n meas is exceeded with a probability of roughly 1 / 2 1/2 1/2 .
The above problem is being solved using the Ensemble Kalman Inversion in which we take presence of a measurement data set into account. The idea is to iterate through each experiment of the data set. We do so by extending the Ensemble Kalman Inversion in a Kaczmarz-like style. Before, we introduce a component-wise transformation on parameters coming from [23 , 13 ]
T ( x ) i = log ( x ^ 2 , i − x i x i − x ^ 1 , i ) , T − 1 ( θ ) i = x ^ 2 , i + e θ i x ^ 1 , i 1 + e θ i , i = 1 , … , n x , \mathcal{T}(x)_i=\log\left(\frac{\hat{x}_{2,i}-x_i}{x_i-\hat{x}_{1,i}}\right),\qquad\mathcal{T}^{-1}(\theta)_i=\frac{\hat{x}_{2,i}+e^{\theta_i}\hat{x}_{1,i}}{1+e^{\theta_i}},\quad i=1,\ldots,n_x, T ( x ) i = log ( x i − x ^ 1 , i x ^ 2 , i − x i ) , T − 1 ( θ ) i = 1 + e θ i x ^ 2 , i + e θ i x ^ 1 , i , i = 1 , … , n x ,
and generate a prior ensemble x 0 ( j ) ∼ U ( ∏ i = 1 n x [ x ^ 1 , i , x ^ 2 , i ] ) x_0^{(j)}\sim U\left(\prod_{i=1}^{n_x}[\hat{x}_{1,i},\hat{x}_{2,i}]\right) x 0 ( j ) ∼ U ( ∏ i = 1 n x [ x ^ 1 , i , x ^ 2 , i ] ) , j = 1 , … , N e n s j=1,\ldots,N_{\mathrm{ens}} j = 1 , … , N ens , with N e n s ≥ 2 N_{\mathrm{ens}}\geq2 N ens ≥ 2 using the uniform distribution. Then for every iteration step we will have an updated ensemble x k ( j ) x_k^{(j)} x k ( j ) at iteration k k k .
The Ensemble Kalman Inversion is performed in the following way.
Algorithm 5.1 (Kaczmarz-EKI).
Set ρ > 1 \rho>1 ρ > 1 and choose a maximal number of iterations k max ∈ N k_{\max}\in\mathbb{N} k m a x ∈ N .
Input: forward operators F i F_i F i , input and shifted measurement data { ( u i , y i 0 , y i ζ ) } \{(u_i,y_i^0,y_i^\zeta)\} {( u i , y i 0 , y i ζ )} , i = 1 , … , N e x p i=1,\ldots,N_{\mathrm{exp}} i = 1 , … , N exp , prior ensemble { x 0 ( j ) } j = 1 , … , N e n s \{x_0^{(j)}\}_{j=1,\ldots,N_{\mathrm{ens}}} { x 0 ( j ) } j = 1 , … , N ens , symmetric positive definite inflated noise covariance matrix Γ y ∈ R n m e a s × n m e a s \Gamma_y\in\mathbb{R}^{n_{\mathrm{meas}}\times n_{\mathrm{meas}}} Γ y ∈ R n meas × n meas , noise level η > 0 \eta>0 η > 0 , and h h h with 0 < h < n x / 2 0<h<n_x/2 0 < h < n x /2 .
For k = 0 , 1 , … , k max k=0,1,\ldots,k_{\max} k = 0 , 1 , … , k m a x and [ k ] : = ( k m o d N e x p ) + 1 [k]:=(k\bmod N_{\mathrm{exp}})+1 [ k ] := ( k mod N exp ) + 1
(1) Apply metric projection onto ∏ i = 1 n x [ x ^ 1 , i + h , x ^ 2 , i − h ] \prod_{i=1}^{n_x}[\hat{x}_{1,i}+h,\hat{x}_{2,i}-h] ∏ i = 1 n x [ x ^ 1 , i + h , x ^ 2 , i − h ] , for a small h > 0 h>0 h > 0 , denoted by P h P_h P h ,
x k ( j ) ← P h x k ( j ) for j = 1 , … , N e n s . x_k^{(j)}\leftarrow P_hx_k^{(j)}\text{ for }j=1,\ldots,N_{\mathrm{ens}}. x k ( j ) ← P h x k ( j ) for j = 1 , … , N ens . (2) Evaluate forward maps F i , k ( j ) ← F i ( x k ( j ) ) F_{i,k}^{(j)}\leftarrow F_i(x_k^{(j)}) F i , k ( j ) ← F i ( x k ( j ) ) for j = 1 , … , N e n s j=1,\ldots,N_{\mathrm{ens}} j = 1 , … , N ens and i = 1 , … , N e x p i=1,\ldots,N_{\mathrm{exp}} i = 1 , … , N exp .
(3) Compute
ω k : = { 1 ∥ Γ y − 1 / 2 ( F ‾ [ k ] , k − y [ k ] ) ∥ > ρ η , 0 otherwise , \omega_k :=
\begin{cases}
1 & \lVert\Gamma_y^{-1/2}(\overline{F}_{[k],k}-y_{[k]})\rVert> \rho\eta,\\
0 & \text{otherwise},
\end{cases} ω k := { 1 0 ∥ Γ y − 1/2 ( F [ k ] , k − y [ k ] )∥ > ρ η , otherwise ,
where F ‾ i , k : = 1 N e n s ∑ j = 1 N e n s F i , k ( j ) \overline{F}_{i,k}:=\frac{1}{N_{\mathrm{ens}}}\sum_{j=1}^{N_{\mathrm{ens}}}F_{i,k}^{(j)} F i , k := N ens 1 ∑ j = 1 N ens F i , k ( j ) for i = 1 , … , N e x p i=1,\ldots,N_{\mathrm{exp}} i = 1 , … , N exp .
(4) Stop and return { x k ( j ) } j = 1 , … , N e n s \{x_k^{(j)}\}_{j=1,\ldots,N_{\mathrm{ens}}} { x k ( j ) } j = 1 , … , N ens if ∥ Γ y − 1 / 2 ( F ‾ [ k − i + 1 ] , k − i + 1 − y [ k − i + 1 ] ) ∥ ≤ ρ η \lVert\Gamma_y^{-1/2}(\overline{F}_{[k-i+1],k-i+1}-y_{[k-i+1]})\rVert\leq\rho\eta ∥ Γ y − 1/2 ( F [ k − i + 1 ] , k − i + 1 − y [ k − i + 1 ] )∥ ≤ ρ η for all i = 1 , … , N e x p i=1,\ldots,N_{\mathrm{exp}} i = 1 , … , N exp .
(5) Perform component-wise transformation
θ k ( j ) ← T ( x k ( j ) ) for j = 1 , … , N e n s . \theta_k^{(j)}\leftarrow\mathcal{T}(x_k^{(j)})\text{ for }j=1,\ldots,N_{\mathrm{ens}}. θ k ( j ) ← T ( x k ( j ) ) for j = 1 , … , N ens .
(6) Compute with
C k θ F = 1 N e n s − 1 ∑ j = 1 N e n s [ ( θ k ( j ) − θ ‾ k ) ⊗ ( F [ k ] , k ( j ) − F ‾ [ k ] , k ) ] , θ ‾ k : = 1 N e n s ∑ j = 1 N e n s θ k ( j ) , C_k^{\theta F}=\frac{1}{N_{\mathrm{ens}}-1}\sum_{j=1}^{N_{\mathrm{ens}}}\left[(\theta_k^{(j)}-\overline{\theta}_k)\otimes(F_{[k],k}^{(j)}-\overline{F}_{[k],k})\right],\quad\overline{\theta}_k:=\frac{1}{N_{\mathrm{ens}}}\sum_{j=1}^{N_{\mathrm{ens}}}\theta_k^{(j)}, C k θ F = N ens − 1 1 j = 1 ∑ N ens [ ( θ k ( j ) − θ k ) ⊗ ( F [ k ] , k ( j ) − F [ k ] , k ) ] , θ k := N ens 1 j = 1 ∑ N ens θ k ( j ) ,
C k F F = 1 N e n s − 1 ∑ j = 1 N e n s [ ( F [ k ] , k ( j ) − F ‾ [ k ] , k ) ⊗ ( F [ k ] , k ( j ) − F ‾ [ k ] , k ) ] . C_k^{FF}=\frac{1}{N_{\mathrm{ens}}-1}\sum_{j=1}^{N_{\mathrm{ens}}}\left[(F_{[k],k}^{(j)}-\overline{F}_{[k],k})\otimes(F_{[k],k}^{(j)}-\overline{F}_{[k],k})\right]. C k F F = N ens − 1 1 j = 1 ∑ N ens [ ( F [ k ] , k ( j ) − F [ k ] , k ) ⊗ ( F [ k ] , k ( j ) − F [ k ] , k ) ] .
(7) Sample ξ k ( j ) ∼ N ( 0 , Γ y ) \xi_k^{(j)}\sim\mathcal{N}(0,\Gamma_y) ξ k ( j ) ∼ N ( 0 , Γ y ) .
(8) Calculate update
θ k + 1 ( j ) = θ k ( j ) + ω k C k θ F ( Γ y + C k F F ) − 1 ( y [ k ] + ξ k ( j ) − F ‾ [ k ] , k ) , j = 1 , … , N e n s . \theta_{k+1}^{(j)}=\theta_k^{(j)}+\omega_k C_k^{\theta F}(\Gamma_y+C_k^{FF})^{-1}(y_{[k]}+\xi_k^{(j)}-\overline{F}_{[k],k}),\quad j=1,\ldots,N_{\mathrm{ens}}. θ k + 1 ( j ) = θ k ( j ) + ω k C k θ F ( Γ y + C k F F ) − 1 ( y [ k ] + ξ k ( j ) − F [ k ] , k ) , j = 1 , … , N ens .
(9) Transform back
x k + 1 ( j ) ← T − 1 ( θ k + 1 ( j ) ) . x_{k+1}^{(j)}\leftarrow\mathcal{T}^{-1}(\theta_{k+1}^{(j)}). x k + 1 ( j ) ← T − 1 ( θ k + 1 ( j ) ) .
Output: final ensemble { x k ( j ) } j = 1 , … , N e n s \{x_k^{(j)}\}_{j=1,\ldots,N_{\mathrm{ens}}} { x k ( j ) } j = 1 , … , N ens at the iteration k k k at which the algorithm stops in step (4), or { x k max + 1 ( j ) } j = 1 , … , N e n s \{x_{k_{\max}+1}^{(j)}\}_{j=1,\ldots,N_{\mathrm{ens}}} { x k m a x + 1 ( j ) } j = 1 , … , N ens if the stopping criterion is not met for k ≤ k max k\leq k_{\max} k ≤ k m a x .
As aforementioned the notation using multiple forward operators is inspired by [36 ] where they also use the parameter ω k \omega_k ω k combined with a projection step. Here, the metric projection step helps avoiding numerical issues while the transformation on parameters helps avoiding to leave the parameter space used for training the surrogate model as it is stated in [23 ] . Note that neither the noise level η \eta η nor the covariance matrix Γ y \Gamma_y Γ y are dependent on the experiments. This is just due to the fact that we want to keep the presentation simple but an extension to the experiment dependent case is favorable, especially in such physical settings where the time length of an experiment can vary. Further, recall that in the beginning of this paper it was mentioned, that neural operators maintain an architecture to handle batched data, that is the simultaneous but fast evaluation on a set of data. Thus, we highlight the strength of a FNO in the above algorithm due to the requirement of performing step (2).
Summary and future work
We derived quantitative rates for approximating the solution operator of controlled ordinary differential equations using the implementable version of the Fourier neural operator. These results show that we overcome the curse of parametric complexity by stating that the size of the FNO grows most algebraically in 1 / ε 1/\varepsilon 1/ ε , and even sub-quadratically if s > d / 2 + 1 s>d/2+1 s > d /2 + 1 (GELU) or s > 2 d + 1 s>2d+1 s > 2 d + 1 (ReLU), suggesting the Fourier neural operator as a suitable surrogate model for physical systems described by CODES. Examples of such systems include physical simulations of electric arc furnaces. Those simulations require inverse parameter estimation based on data sets consisting of multiple experiments data. This is not feasible when relying on long computational times that come with high dimensional systems. We suggest mitigating this issue by deploying a learned FNO surrogate within an Ensemble Kalman Inversion. Our proposed EKI method takes multiple measurement data into account by allowing to iterate through the data set, which we refer to as the Kaczmarz structure. Future work will demonstrate the practical effectiveness of the described method by presenting numerical results on an electric arc furnace simulation.
Acknowledgements
Dean Zenner was supported by the “AG der Dillinger Hüttenwerke”.
Proof of well-posedness of CODE
Let d , n , n y , n u ∈ N d,n,n_y,n_u \in\mathbb{N} d , n , n y , n u ∈ N . Before proving Theorem 2.1 we state the well known fundamental theorem of calculus giving us a concise result of ODEs as definite integrals.
Theorem A.1 (Fundamental theorem of calculus). Let I ⊂ R I \subset\mathbb{R} I ⊂ R be open. If y ∈ C a b s ( I ; R n ) y \in C_{\mathrm{abs}}(I;\mathbb{R}^n) y ∈ C abs ( I ; R n ) , then the derivative y ˙ \dot{y} y ˙ exists almost everywhere in I I I , is finite and in L 1 ( I ; R n ) L^1(I;\mathbb{R}^n) L 1 ( I ; R n ) . Additionally we have
y ( t 2 ) − y ( t 1 ) = ∫ t 1 t 2 y ˙ ( t ) d t ∀ t 1 , t 2 ∈ I . y(t_2)-y(t_1)=\int_{t_1}^{t_2}\dot{y}(t)\mathrm{d}t \quad\forall t_1,t_2 \in I. y ( t 2 ) − y ( t 1 ) = ∫ t 1 t 2 y ˙ ( t ) d t ∀ t 1 , t 2 ∈ I . Conversely, if h ∈ L 1 ( I ; R n ) h \in L^1(I;\mathbb{R}^n) h ∈ L 1 ( I ; R n ) and t 1 ∈ I t_1 \in I t 1 ∈ I fixed, then the function
y ( t ) : = ∫ t 1 t h ( s ) d s y(t):=\int_{t_1}^{t}h(s)\mathrm{d}s y ( t ) := ∫ t 1 t h ( s ) d s defines an absolutely continuous function y ∈ C a b s ( I ; R n ) y \in C_{\mathrm{abs}}(I;\mathbb{R}^n) y ∈ C abs ( I ; R n ) with y ˙ ( t ) = h ( t ) \dot{y}(t)=h(t) y ˙ ( t ) = h ( t ) almost everywhere in I I I .
Proof. See E3.6 in [1 ] .
Following extension of the Banach fixed point theorem allows us to derive the continuous dependence on the control functions in Theorem 2.1 .
Theorem A.2 (Parameter-dependent Banach fixed point theorem). Let X X X be a (non-empty) Banach space and Λ \Lambda Λ a metric space. Let Φ : X × Λ → X \Phi: X \times\Lambda\to X Φ : X × Λ → X such that for every fixed x ∈ X x \in X x ∈ X the mapping λ ↦ Φ ( x , λ ) \lambda\mapsto\Phi(x,\lambda) λ ↦ Φ ( x , λ ) is continuous. Assume there exists a κ ∈ ( 0 , 1 ) \kappa\in(0,1) κ ∈ ( 0 , 1 ) , with
∥ Φ ( x , λ ) − Φ ( y , λ ) ∥ X ≤ κ ∥ x − y ∥ X for all λ ∈ Λ , x , y ∈ X . \lVert\Phi(x,\lambda)-\Phi(y,\lambda)\rVert_X \leq\kappa\lVert x-y\rVert_X \quad\text{for all }\lambda\in\Lambda,x,y \in X. ∥ Φ ( x , λ ) − Φ ( y , λ ) ∥ X ≤ κ ∥ x − y ∥ X for all λ ∈ Λ , x , y ∈ X . Then
for every λ ∈ Λ \lambda\in\Lambda λ ∈ Λ exists a unique fixed point x ( λ ) ∈ X x(\lambda) \in X x ( λ ) ∈ X with Φ ( x ( λ ) , λ ) = x ( λ ) \Phi(x(\lambda),\lambda)=x(\lambda) Φ ( x ( λ ) , λ ) = x ( λ ) . Further for every y ∈ X y \in X y ∈ X
∥ y − x ( λ ) ∥ X ≤ 1 1 − κ ∥ y − Φ ( y , λ ) ∥ X , \lVert y-x(\lambda)\rVert_X \leq\frac{1}{1-\kappa}\lVert y-\Phi(y,\lambda)\rVert_X, ∥ y − x ( λ ) ∥ X ≤ 1 − κ 1 ∥ y − Φ ( y , λ ) ∥ X ,
Proof. First point is just applying the Banach fixed point theorem for each fixed λ ∈ Λ \lambda\in\Lambda λ ∈ Λ . For the second point consider a sequence ( λ n ) n ∈ N (\lambda_n)_{n\in\mathbb{N}} ( λ n ) n ∈ N with λ n → λ 0 ∈ Λ \lambda_n \to\lambda_0 \in\Lambda λ n → λ 0 ∈ Λ . For y = x ( λ 0 ) y=x(\lambda_0) y = x ( λ 0 )
∥ x ( λ 0 ) − x ( λ n ) ∥ X ≤ 1 1 − κ ∥ Φ ( x ( λ 0 ) , λ 0 ) − Φ ( x ( λ 0 ) , λ n ) ∥ X , \lVert x(\lambda_0)-x(\lambda_n)\rVert_X \leq\frac{1}{1-\kappa}\lVert\Phi(x(\lambda_0),\lambda_0)-\Phi(x(\lambda_0),\lambda_n)\rVert_X, ∥ x ( λ 0 ) − x ( λ n ) ∥ X ≤ 1 − κ 1 ∥ Φ ( x ( λ 0 ) , λ 0 ) − Φ ( x ( λ 0 ) , λ n ) ∥ X ,
where the right hand side vanishes due to the assumed continuity.
With both established results we can now give the proof of Theorem 2.1 .
Proof. We want to apply the parameterized Banach fixed point theorem from before. Define for X : = C ( I ; R n y ) X:=C(I;\mathbb{R}^{n_y}) X := C ( I ; R n y ) , Λ : = L q ( I ; R n u ) \Lambda:=L^q(I;\mathbb{R}^{n_u}) Λ := L q ( I ; R n u ) , the operator
Φ : X × Λ × R n y → X , [ Φ ( y , u , y 0 ) ] ( t ) : = y 0 + ∫ t 1 t f ( s , y ( s ) , u ( s ) ) d s . \Phi:X\times\Lambda\times\mathbb{R}^{n_y}\to X,\quad[\Phi(y,u,y^0)](t):=y^0+\int_{t_1}^{t}f(s,y(s),u(s))\mathrm{d}s. Φ : X × Λ × R n y → X , [ Φ ( y , u , y 0 )] ( t ) := y 0 + ∫ t 1 t f ( s , y ( s ) , u ( s )) d s .
First we have to argue that Φ \Phi Φ is well defined. For this, the integral has to exist. The first three assumptions ensure that f f f is measurable with respect to t t t while the fourth assumption shows that the integral is finite, since for 1 ≤ p < ∞ 1\leq p<\infty 1 ≤ p < ∞ it is
∥ f ( ⋅ , y ( ⋅ ) , u ( ⋅ ) ) ∥ L p ( I ; R n y ) = ( ∫ I ∥ f ( s , y ( s ) , u ( s ) ) ∥ R n y p d s ) 1 / p ≤ ( ∫ I α ( s ) p d s ) 1 / p = ( ∫ I ∣ α ( s ) ∣ p d s ) 1 / p = ∥ α ∥ L p ( I ) < ∞ \begin{aligned}
\lVert f(\cdot,y(\cdot),u(\cdot))\rVert_{L^p(I;\mathbb{R}^{n_y})}
&=\left(\int_I\lVert f(s,y(s),u(s))\rVert_{\mathbb{R}^{n_y}}^p\mathrm{d}s\right)^{1/p}\\
&\leq\left(\int_I\alpha(s)^p\mathrm{d}s\right)^{1/p}\\
&=\left(\int_I\lvert\alpha(s)\rvert^p\mathrm{d}s\right)^{1/p}\\
&=\lVert\alpha\rVert_{L^p(I)}<\infty
\end{aligned} ∥ f ( ⋅ , y ( ⋅ ) , u ( ⋅ )) ∥ L p ( I ; R n y ) = ( ∫ I ∥ f ( s , y ( s ) , u ( s )) ∥ R n y p d s ) 1/ p ≤ ( ∫ I α ( s ) p d s ) 1/ p = ( ∫ I ∣ α ( s ) ∣ p d s ) 1/ p = ∥ α ∥ L p ( I ) < ∞
and for p = ∞ p=\infty p = ∞ we have
∥ f ( t , y ( t ) , u ( t ) ) ∥ R n y ≤ α ( t ) ≤ ∥ α ∥ L ∞ < ∞ , \lVert f(t,y(t),u(t))\rVert_{\mathbb{R}^{n_y}}\leq\alpha(t)\leq\lVert\alpha\rVert_{L^\infty}<\infty, ∥ f ( t , y ( t ) , u ( t )) ∥ R n y ≤ α ( t ) ≤ ∥ α ∥ L ∞ < ∞ ,
for almost all t ∈ I t\in I t ∈ I . Thus it is ∥ f ( t , y , u ) ∥ L ∞ ( I ; R n y ) = ess sup t ∈ I ∥ f ( t , y ( t ) , u ( t ) ) ∥ R n y < ∞ \lVert f(t,y,u)\rVert_{L^\infty(I;\mathbb{R}^{n_y})}=\operatorname{ess\,sup}_{t\in I}\lVert f(t,y(t),u(t))\rVert_{\mathbb{R}^{n_y}}<\infty ∥ f ( t , y , u ) ∥ L ∞ ( I ; R n y ) = ess sup t ∈ I ∥ f ( t , y ( t ) , u ( t )) ∥ R n y < ∞ . By that we conclude y ˙ ( ⋅ ) = f ( ⋅ , y ( ⋅ ) , u ( ⋅ ) ) ∈ L p ( I ; R n y ) ⊆ L 1 ( I ; R n y ) \dot{y}(\cdot)=f(\cdot,y(\cdot),u(\cdot))\in L^p(I;\mathbb{R}^{n_y})\subseteq L^1(I;\mathbb{R}^{n_y}) y ˙ ( ⋅ ) = f ( ⋅ , y ( ⋅ ) , u ( ⋅ )) ∈ L p ( I ; R n y ) ⊆ L 1 ( I ; R n y ) , 1 ≤ p ≤ ∞ 1\leq p\leq\infty 1 ≤ p ≤ ∞ for finite domains. In the next step, we need to show that Φ \Phi Φ defines a contraction. We show that Φ \Phi Φ is a contraction with respect to the equivalent Bielecki norm
∥ y ∥ X : = sup t ∈ I ∥ y ( t ) ∥ R n y e − λ ∣ t − t 1 ∣ , \lVert y\rVert_X:=\sup_{t\in I}\lVert y(t)\rVert_{\mathbb{R}^{n_y}}e^{-\lambda\lvert t-t_1\rvert}, ∥ y ∥ X := t ∈ I sup ∥ y ( t ) ∥ R n y e − λ ∣ t − t 1 ∣ ,
where λ > 0 \lambda>0 λ > 0 is chosen later. Fix y 0 = y ( t 1 ) y^0=y(t_1) y 0 = y ( t 1 ) and u 0 ∈ Λ u^0\in\Lambda u 0 ∈ Λ . For y ^ = Φ ( y , u 0 , y 0 ) \hat{y}=\Phi(y,u^0,y^0) y ^ = Φ ( y , u 0 , y 0 ) and z ^ = Φ ( z , u 0 , y 0 ) \hat{z}=\Phi(z,u^0,y^0) z ^ = Φ ( z , u 0 , y 0 ) it is
∥ y ^ ( t ) − z ^ ( t ) ∥ ≤ ∣ ∫ t 1 t ∥ f ( s , y ( s ) , u 0 ) − f ( s , z ( s ) , u 0 ) ∥ R n y d s ∣ ≤ ∥ y − z ∥ X ∣ ∫ t 1 t m ( s ) e λ ∣ s − t 1 ∣ d s ∣ . \begin{aligned}
\lVert\hat{y}(t)-\hat{z}(t)\rVert\leq\left\lvert\int_{t_1}^{t}\lVert f(s,y(s),u^0)-f(s,z(s),u^0)\rVert_{\mathbb{R}^{n_y}}\mathrm{d}s\right\rvert\\
&\leq\lVert y-z\rVert_X\left\lvert\int_{t_1}^{t}m(s)e^{\lambda\lvert s-t_1\rvert}\mathrm{d}s\right\rvert.
\end{aligned} ∥ y ^ ( t ) − z ^ ( t )∥ ≤ ∫ t 1 t ∥ f ( s , y ( s ) , u 0 ) − f ( s , z ( s ) , u 0 ) ∥ R n y d s ≤ ∥ y − z ∥ X ∫ t 1 t m ( s ) e λ ∣ s − t 1 ∣ d s .
Hence it is ∥ y ^ − z ^ ∥ X ≤ κ ∥ y − z ∥ X \|\hat{y}-\hat{z}\|_{X} \le\kappa\|y-z\|_{X} ∥ y ^ − z ^ ∥ X ≤ κ ∥ y − z ∥ X , for κ : = sup t ∈ I ∣ ∫ t 1 t m ( s ) e − λ ∣ t − s ∣ d s ∣ \kappa:= \sup_{t\in I}\left|\int_{t_1}^{t}m(s)e^{-\lambda|t-s|}\,ds\right| κ := sup t ∈ I ∫ t 1 t m ( s ) e − λ ∣ t − s ∣ d s , where we used ∣ t − t 1 ∣ − ∣ s − t 1 ∣ = ∣ t − s ∣ |t-t_1|-|s-t_1|=|t-s| ∣ t − t 1 ∣ − ∣ s − t 1 ∣ = ∣ t − s ∣ for s s s between t 1 t_1 t 1 and t t t . We write m = m 1 + m 2 m=m_1+m_2 m = m 1 + m 2 with m 1 : = m χ { m > M } m_1:=m\chi_{\{m>M\}} m 1 := m χ { m > M } and m 2 : = m χ { m ≤ M } m_2:=m\chi_{\{m\le M\}} m 2 := m χ { m ≤ M } . Choose M > 0 M>0 M > 0 large enough such that ∫ I m 1 ( s ) d s < 1 / 4 \int_I m_1(s)\,ds<1/4 ∫ I m 1 ( s ) d s < 1/4 . Then for all t ∈ I t\in I t ∈ I it is
∣ ∫ t 1 t m 1 ( s ) e − λ ∣ t − s ∣ d s ∣ ≤ ∣ ∫ I m 1 ( s ) d s ∣ < 1 4 . \left|\int_{t_1}^{t}m_1(s)e^{-\lambda|t-s|}\,ds\right|\le\left|\int_I m_1(s)\,ds\right|<\frac{1}{4}. ∫ t 1 t m 1 ( s ) e − λ ∣ t − s ∣ d s ≤ ∫ I m 1 ( s ) d s < 4 1 .
Further we have by substitution
∣ ∫ t 1 t m 2 ( s ) e − λ ∣ t − s ∣ d s ∣ ≤ M ∣ ∫ t 1 t e − λ ∣ t − s ∣ d s ∣ ≤ M ∫ 0 ∞ e − λ s d s = M λ . \left|\int_{t_1}^{t}m_2(s)e^{-\lambda|t-s|}\,ds\right|\le M\left|\int_{t_1}^{t}e^{-\lambda|t-s|}\,ds\right|\le M\int_{0}^{\infty}e^{-\lambda s}\,ds=\frac{M}{\lambda}. ∫ t 1 t m 2 ( s ) e − λ ∣ t − s ∣ d s ≤ M ∫ t 1 t e − λ ∣ t − s ∣ d s ≤ M ∫ 0 ∞ e − λ s d s = λ M .
When we choose λ > 4 M \lambda>4M λ > 4 M , we then have κ < 1 / 2 \kappa<1/2 κ < 1/2 and thus Φ \Phi Φ is contraction measured in the Bielecki norm. We still need continuity of u ↦ Φ ( y , u , y 0 ) u\mapsto\Phi(y,u,y^0) u ↦ Φ ( y , u , y 0 ) considering the L q L^q L q norm and the sup-norm. Note that we can now endow the space of continuous functions with the sup-norm since it is equivalent to the Bielecki norm. For u , v ∈ Λ u,v\in\Lambda u , v ∈ Λ it is
sup t ∈ I ∥ [ Φ ( y , u , y 0 ) ] ( t ) − [ Φ ( y , v , y 0 ) ] ( t ) ∥ R n y = sup t ∈ I ∥ ∫ t 1 t f ( s , y ( s ) , u ( s ) ) d s − ∫ t 1 t f ( s , y ( s ) , v ( s ) ) d s ∥ R n y → 0 \sup_{t\in I}\left\|[\Phi(y,u,y^0)](t)-[\Phi(y,v,y^0)](t)\right\|_{\mathbb{R}^{n_y}}=\sup_{t\in I}\left\|\int_{t_1}^{t}f(s,y(s),u(s))\,ds-\int_{t_1}^{t}f(s,y(s),v(s))\,ds\right\|_{\mathbb{R}^{n_y}}\to0 t ∈ I sup [ Φ ( y , u , y 0 )] ( t ) − [ Φ ( y , v , y 0 )] ( t ) R n y = t ∈ I sup ∫ t 1 t f ( s , y ( s ) , u ( s )) d s − ∫ t 1 t f ( s , y ( s ) , v ( s )) d s R n y → 0
for ∥ u − v ∥ L q → 0 \|u-v\|_{L^q}\to0 ∥ u − v ∥ L q → 0 , due to the last assumption in the statement. Hence the parameterized Banach fixed point states that for y 0 ∈ R n y y^0\in\mathbb{R}^{n_y} y 0 ∈ R n y and u ∈ Λ u\in\Lambda u ∈ Λ there exists a unique fixed point y ∈ C ( I ; R n y ) y\in C(I;\mathbb{R}^{n_y}) y ∈ C ( I ; R n y ) such that
Φ ( y , y 0 , u ) = y . \Phi(y,y^0,u)=y. Φ ( y , y 0 , u ) = y .
This implies
y ( t ) = [ Φ ( y , u , y 0 ) ] ( t ) = y 0 + ∫ t 1 t f ( s , y ( s ) , u ( s ) ) d s for all t ∈ I , y(t)=[\Phi(y,u,y^0)](t)=y^0+\int_{t_1}^{t}f(s,y(s),u(s))\,ds\quad\text{for all }t\in I, y ( t ) = [ Φ ( y , u , y 0 )] ( t ) = y 0 + ∫ t 1 t f ( s , y ( s ) , u ( s )) d s for all t ∈ I ,
which is being equivalent to y ∈ C a b s ( I ; R n y ) y\in C_{\mathrm{abs}}(I;\mathbb{R}^{n_y}) y ∈ C abs ( I ; R n y ) with y ˙ ( t ) = f ( t , y ( t ) , u ( t ) ) \dot{y}(t)=f(t,y(t),u(t)) y ˙ ( t ) = f ( t , y ( t ) , u ( t )) for almost all t ∈ I t\in I t ∈ I , due to the Theorem A.1 . Further the mapping
R n y × L q ( I ; R n u ) → C ( I ; R n y ) , ( y 0 , u ) ↦ y = Φ ( y , u , y 0 ) \mathbb{R}^{n_y}\times L^q(I;\mathbb{R}^{n_u})\to C(I;\mathbb{R}^{n_y}),\quad(y^0,u)\mapsto y=\Phi(y,u,y^0) R n y × L q ( I ; R n u ) → C ( I ; R n y ) , ( y 0 , u ) ↦ y = Φ ( y , u , y 0 )
is continuous.
Later, we study Φ ( y ; u , y 0 ) \Phi(y;u,y^0) Φ ( y ; u , y 0 ) parameterized by ( u , y 0 ) (u,y^0) ( u , y 0 ) . Then, looking at the above proof, Φ \Phi Φ is remains a contraction with respect to y y y which we summarize in following corollary.
Corollary A.3. Fix t 1 ∈ I t_1\in I t 1 ∈ I and y ( t 1 ) = y 0 ∈ R n y y(t_1)=y^0\in\mathbb{R}^{n_y} y ( t 1 ) = y 0 ∈ R n y and u ∈ L q ( I ; R n u ) u\in L^q(I;\mathbb{R}^{n_u}) u ∈ L q ( I ; R n u ) , 1 ≤ q ≤ ∞ 1\le q\le\infty 1 ≤ q ≤ ∞ . Consider Φ ( y ; u , y 0 ) : C ( I ; R n y ) → C ( I ; R n y ) \Phi(y;u,y^0):C(I;\mathbb{R}^{n_y})\to C(I;\mathbb{R}^{n_y}) Φ ( y ; u , y 0 ) : C ( I ; R n y ) → C ( I ; R n y ) , y ↦ y 0 + ∫ t 1 t f ( s , y ( s ) , u ( s ) ) y\mapsto y^0+\int_{t_1}^{t}f(s,y(s),u(s)) y ↦ y 0 + ∫ t 1 t f ( s , y ( s ) , u ( s )) . Then there exists a κ < 1 / 2 \kappa<1/2 κ < 1/2 such that Φ \Phi Φ is a κ \kappa κ -contraction in the Bielecki norm, meaning
∥ Φ ( y ; u , y 0 ) − Φ ( z ; u , y 0 ) ∥ X ≤ κ ∥ y − z ∥ X \|\Phi(y;u,y^0)-\Phi(z;u,y^0)\|_X\le\kappa\|y-z\|_X ∥Φ ( y ; u , y 0 ) − Φ ( z ; u , y 0 ) ∥ X ≤ κ ∥ y − z ∥ X for all $y,z\in C(I;\mathbb{R}^{n_y}).
With this, we have proven the well-posedness of our CODE system. The following proof of Lemma 2.2 shows that the assumptions of Theorem 2.1 together with the bounded initial values y 0 ∈ Y 0 ⊂ R n y y^0\in Y^0\subset\mathbb{R}^{n_y} y 0 ∈ Y 0 ⊂ R n y are enough to show that solutions of the CODE system are also bounded, while not relying on properties of u ∈ L q ( I ; R n u ) u\in L^q(I;\mathbb{R}^{n_u}) u ∈ L q ( I ; R n u ) . This is due to the assumption (4) in the statement as we can observe in the proof of Lemma 2.2 .
Proof. From y 0 ∈ Y 0 ⊂ R n y y^0\in Y^0\subset\mathbb{R}^{n_y} y 0 ∈ Y 0 ⊂ R n y bounded there exists a S > 0 S>0 S > 0 such that we have ∥ y 0 ∥ R n y ≤ S \|y^0\|_{\mathbb{R}^{n_y}}\le S ∥ y 0 ∥ R n y ≤ S for all y 0 ∈ Y 0 ⊂ R n y y^0\in Y^0\subset\mathbb{R}^{n_y} y 0 ∈ Y 0 ⊂ R n y . From Theorem A.1 and Assumption (4) in Theorem 2.1 we can show that it holds for the solution that
∥ y ( t ) ∥ R n y ≤ ∥ y 0 ∥ R n y + ∫ t 1 t ∥ f ( s , y ( s ) , u ( s ) ) ∥ R n y d s < S + ∫ t 1 t 2 α ( s ) d s ≤ S + C ∥ α ∥ L p = : Y < ∞ , \begin{aligned}
\|y(t)\|_{\mathbb{R}^{n_y}}\le\|y^0\|_{\mathbb{R}^{n_y}}+\int_{t_1}^{t}\|f(s,y(s),u(s))\|_{\mathbb{R}^{n_y}}\,ds \\
&<S+\int_{t_1}^{t_2}\alpha(s)\,ds \\
&\le S+C\|\alpha\|_{L^p}=:Y<\infty,
\end{aligned} ∥ y ( t ) ∥ R n y ≤ ∥ y 0 ∥ R n y + ∫ t 1 t ∥ f ( s , y ( s ) , u ( s )) ∥ R n y d s < S + ∫ t 1 t 2 α ( s ) d s ≤ S + C ∥ α ∥ L p =: Y < ∞ ,
for all t ∈ I t\in I t ∈ I and all 1 ≤ p ≤ ∞ 1\le p\le\infty 1 ≤ p ≤ ∞ where C = C ( p , I ) > 0 C=C(p,I)>0 C = C ( p , I ) > 0 is the constant obtained by estimating the L 1 L^1 L 1 -norm upwards with the L p L^p L p -norm, $1\le p\le\infty.
Next we want to prove the boundedness of controls u ∈ U l , T , A u \in\mathcal{U}^{l,T,A} u ∈ U l , T , A . As they are Sobolev functions, we need a well-known Sobolev embedding statement, which will help us several times throughout this paper.
Definition A.4. Consider the Sobolev space W s , p ( I ; R n ) W^{s,p}(I;\mathbb{R}^{n}) W s , p ( I ; R n ) , I ⊂ R d I \subset\mathbb{R}^{d} I ⊂ R d . We call the number
γ : = s − d p \gamma:= s - \frac{d}{p} γ := s − p d the Sobolev number.
Lemma A.5 (Sobolev embedding). Let I ⊂ R d I \subset\mathbb{R}^{d} I ⊂ R d be a bounded, open subset, 1 ≤ p < ∞ 1 \le p < \infty 1 ≤ p < ∞ , s ∈ N 0 s \in\mathbb{N}_{0} s ∈ N 0 and γ \gamma γ the Sobolev number to W s , p ( I ; R n ) W^{s,p}(I;\mathbb{R}^{n}) W s , p ( I ; R n ) . Then for γ > k \gamma> k γ > k we have the continuous embedding
W s , p ( I ) ↪ C k ( I ; R n ) ⊂ C ( I ; R n ) , W^{s,p}(I) \hookrightarrow C^{k}(I;\mathbb{R}^{n}) \subset C(I;\mathbb{R}^{n}), W s , p ( I ) ↪ C k ( I ; R n ) ⊂ C ( I ; R n ) , where the latter are endowed with the sup norm. In particular by setting p = 2 p = 2 p = 2 , k = 0 k = 0 k = 0 and then s > d / 2 s > d/2 s > d /2 , we have a continuous embedding
H s ( I ; R n ) ↪ C ( I ; R n ) . H^{s}(I;\mathbb{R}^{n}) \hookrightarrow C(I;\mathbb{R}^{n}). H s ( I ; R n ) ↪ C ( I ; R n ) . Thus, there exists a constant C = C ( s , d ) > 0 C = C(s,d) > 0 C = C ( s , d ) > 0 such that
∥ v ∥ L ∞ ( I ; R n ) ≤ C ∥ v ∥ H s ( I ; R n ) , ∀ v ∈ H s ( I ; R n ) . \|v\|_{L^{\infty}(I;\mathbb{R}^{n})} \le C\|v\|_{H^{s}(I;\mathbb{R}^{n})}, \quad\forall v \in H^{s}(I;\mathbb{R}^{n}). ∥ v ∥ L ∞ ( I ; R n ) ≤ C ∥ v ∥ H s ( I ; R n ) , ∀ v ∈ H s ( I ; R n ) . Further, when p = 2 p = 2 p = 2 , k = 1 k = 1 k = 1 and then s > d / 2 + 1 s > d/2 + 1 s > d /2 + 1 , we have the continuous embedding
H s ( I ; R n ) ↪ C 1 ( I ; R n ) ⊂ W 1 , ∞ ( I ; R n ) , H^{s}(I;\mathbb{R}^{n}) \hookrightarrow C^{1}(I;\mathbb{R}^{n}) \subset W^{1,\infty}(I;\mathbb{R}^{n}), H s ( I ; R n ) ↪ C 1 ( I ; R n ) ⊂ W 1 , ∞ ( I ; R n ) , so there exists a constant C = C ( s , d ) > 0 C = C(s,d) > 0 C = C ( s , d ) > 0 such that
∥ v ∥ W 1 , ∞ ( I ; R n ) ≤ C ∥ v ∥ H s ( I ; R n ) , ∀ v ∈ H s ( I ; R n ) . \|v\|_{W^{1,\infty}(I;\mathbb{R}^{n})} \le C\|v\|_{H^{s}(I;\mathbb{R}^{n})}, \quad\forall v \in H^{s}(I;\mathbb{R}^{n}). ∥ v ∥ W 1 , ∞ ( I ; R n ) ≤ C ∥ v ∥ H s ( I ; R n ) , ∀ v ∈ H s ( I ; R n ) .
Proof. See Corollary 9.13 in [8 ] .
Before we now prove Lemma 2.4 , we first investigate the boundedness of the control functions, when they are not periodically extended using the linear periodic extension operator.
Lemma A.6. For l > 1 / 2 l > 1/2 l > 1/2 we have
∥ u ∥ L ∞ ( I ; R n u ) ≤ C ∥ u ∥ H l ( I ; R n u ) ≤ C A , ∀ u ∈ U l , I , A , (12) \|u\|_{L^{\infty}(I;\mathbb{R}^{n_u})} \le C\|u\|_{H^{l}(I;\mathbb{R}^{n_u})} \le CA, \quad\forall u \in\mathcal{U}^{l,I,A}, \tag*{(12)} ∥ u ∥ L ∞ ( I ; R n u ) ≤ C ∥ u ∥ H l ( I ; R n u ) ≤ C A , ∀ u ∈ U l , I , A , (12) where C = C ( H l ( I ; R n u ) , L ∞ ( I ; R n u ) ) > 0 C = C(H^{l}(I;\mathbb{R}^{n_u}),L^{\infty}(I;\mathbb{R}^{n_u})) > 0 C = C ( H l ( I ; R n u ) , L ∞ ( I ; R n u )) > 0 is a constant depending on l > 1 / 2 l > 1/2 l > 1/2 .
Proof. This is immediate by the Sobolev embedding Lemma A.5 and the definition of U l , I , A \mathcal{U}^{l,I,A} U l , I , A .
This will benefit us at several places but for now it is important to note that we can conclude from it, that there exists a U ∈ R + U \in\mathbb{R}^{+} U ∈ R + such that
u ( t ) ∈ ( − U , U ) n u u(t) \in(-U,U)^{n_u} u ( t ) ∈ ( − U , U ) n u
for all t ∈ I t \in I t ∈ I , u ∈ U l , I , A u \in\mathcal{U}^{l,I,A} u ∈ U l , I , A since u u u has a continuous representative.
We prove Lemma 2.4 next.
Proof. Due to the Sobolev embedding, Lemma A.5 , and the continuity of E \mathcal{E} E we have
∥ E u ∥ C ( T ) ≤ C 1 ∥ E u ∥ H l ( T ) ≤ C 1 C E u ∥ u ∥ H l ( J ) ≤ C 1 C E u A ≤ C u C E u A , \|\mathcal{E}u\|_{C(\mathbb{T})} \le C_1\|\mathcal{E}u\|_{H^{l}(\mathbb{T})} \le C_1C_{\mathcal{E}u}\|u\|_{H^{l}(J)} \le C_1C_{\mathcal{E}u}A \le C_uC_{\mathcal{E}u}A, ∥ E u ∥ C ( T ) ≤ C 1 ∥ E u ∥ H l ( T ) ≤ C 1 C E u ∥ u ∥ H l ( J ) ≤ C 1 C E u A ≤ C u C E u A ,
with C 1 = C 1 ( H l ( T ) , C ( T ) ) > 0 C_1 = C_1(H^{l}(\mathbb{T}),C(\mathbb{T})) > 0 C 1 = C 1 ( H l ( T ) , C ( T )) > 0 and C > 0 C > 0 C > 0 defined as
C u : = max { C 1 , C 2 } . (13) C_u := \max\{C_1,C_2\}. \tag*{(13)} C u := max { C 1 , C 2 } . (13)
Here, C 2 = C 2 ( H l ( T ; R n u ) , W 1 , ∞ ( T ; R n u ) ) > 0 C_2 = C_2(H^{l}(\mathbb{T};\mathbb{R}^{n_u}),W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_u})) > 0 C 2 = C 2 ( H l ( T ; R n u ) , W 1 , ∞ ( T ; R n u )) > 0 is the constant from the embedding H l ( T ; R n u ) ↪ W 1 , ∞ ( T ; R n u ) H^{l}(\mathbb{T};\mathbb{R}^{n_u}) \hookrightarrow W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_u}) H l ( T ; R n u ) ↪ W 1 , ∞ ( T ; R n u ) when l ≥ 2 l \ge2 l ≥ 2 .
Consider the identity function on our time interval
id : [ t ^ 1 , t ^ 2 ] → [ t ^ 1 , t ^ 2 ] , t ↦ t , \operatorname{id} : [\hat{t}_1,\hat{t}_2] \to[\hat{t}_1,\hat{t}_2],\ t \mapsto t, id : [ t ^ 1 , t ^ 2 ] → [ t ^ 1 , t ^ 2 ] , t ↦ t ,
which is a ∞ \infty ∞ -times differentiable continuous function. Define the (up to its derivative) periodically extended time variation as
τ : T → R , t ↦ E id ( t ) . \tau: \mathbb{T} \to\mathbb{R},\ t \mapsto\mathcal{E}\operatorname{id}(t). τ : T → R , t ↦ E id ( t ) .
With the properties of the periodic extension operator we obtain
∥ τ ∥ C r ( T ) ≤ C E id ∥ id ∥ C r ( [ t ^ 1 , t ^ 2 ] ) ≤ C E id max { 1 , t ^ 2 } \|\tau\|_{C^r(\mathbb{T})} \le C_{\mathcal{E}\operatorname{id}}\|\operatorname{id}\|_{C^r([\hat{t}_1,\hat{t}_2])} \le C_{\mathcal{E}\operatorname{id}}\max\{1,\hat{t}_2\} ∥ τ ∥ C r ( T ) ≤ C E id ∥ id ∥ C r ([ t ^ 1 , t ^ 2 ]) ≤ C E id max { 1 , t ^ 2 }
for any r ≥ 0 r \ge0 r ≥ 0 and where C E id = C E id ( r , t ^ 1 , t ^ 2 ) > 0 C_{\mathcal{E}\operatorname{id}}=C_{\mathcal{E}\operatorname{id}}(r,\hat{t}_1,\hat{t}_2)>0 C E id = C E id ( r , t ^ 1 , t ^ 2 ) > 0 . Further, it is τ ∣ J = id ∣ J \tau|_J=\operatorname{id}|_J τ ∣ J = id ∣ J . We obtain
τ ∈ C r ( T ; R ) (14) \tau\in C^r(\mathbb{T};\mathbb{R}) \tag*{(14)} τ ∈ C r ( T ; R ) (14)
for any r ≥ 0 r \ge0 r ≥ 0 . In the same way, we also have
∥ τ ∥ C ( T ) ≤ C 1 ∥ τ ∥ H 1 ( T ) ≤ C 1 C E u ∥ id ∥ H 1 ( J ) ≤ C u C E u t ^ 2 3 − t ^ 1 3 3 + t ^ 2 − t ^ 1 . \|\tau\|_{C(\mathbb{T})} \le C_1\|\tau\|_{H^1(\mathbb{T})} \le C_1 C_{\mathcal{E}u}\|\operatorname{id}\|_{H^1(J)} \le C_u C_{\mathcal{E}u}\sqrt{\frac{\hat{t}_2^3-\hat{t}_1^3}{3}+\hat{t}_2-\hat{t}_1}. ∥ τ ∥ C ( T ) ≤ C 1 ∥ τ ∥ H 1 ( T ) ≤ C 1 C E u ∥ id ∥ H 1 ( J ) ≤ C u C E u 3 t ^ 2 3 − t ^ 1 3 + t ^ 2 − t ^ 1 .
We now set
U ′ = C u C E u ( A + t ^ 2 3 − t ^ 1 3 3 + t ^ 2 − t ^ 1 ) + 2 π U'=C_u C_{\mathcal{E}u}\left(A+\sqrt{\frac{\hat{t}_2^3-\hat{t}_1^3}{3}+\hat{t}_2-\hat{t}_1}\right)+2\pi U ′ = C u C E u A + 3 t ^ 2 3 − t ^ 1 3 + t ^ 2 − t ^ 1 + 2 π
to be able to handle the control ( τ , u ) (\tau,u) ( τ , u ) simultaneously. Note that we then have
( τ , u ) ∈ ( − U ′ , U ′ ) 1 + n u . (\tau,u)\in(-U',U')^{1+n_u}. ( τ , u ) ∈ ( − U ′ , U ′ ) 1 + n u .
Further notice, that by the notation we introduced, it is ∥ u ∥ C ( T ; R 1 + n u ) ≤ U ′ \|u\|_{C(\mathbb{T};\mathbb{R}^{1+n_u})}\le U' ∥ u ∥ C ( T ; R 1 + n u ) ≤ U ′ .
In the paper we mentioned that for the interval I I I we have to at least assume T ⊂ I \mathbb{T}\subset I T ⊂ I for the Carathéodory assumptions to hold. This is due to the fact that the image of τ \tau τ is not necessarily in T \mathbb{T} T . We even can specify a concrete interval I I I . Set
I : = ( − U ′ − 1 , U ′ + 1 ) , (15) I:=(-U'-1,U'+1), \tag*{(15)} I := ( − U ′ − 1 , U ′ + 1 ) , (15)
which contains [ 0 , 2 π ] [0,2\pi] [ 0 , 2 π ] as well as the ranges of τ \tau τ and τ N \tau_N τ N , since U ′ > 2 π U'>2\pi U ′ > 2 π and both functions are bounded by U ′ U' U ′ . It coincides with the time interval of the domain D ′ D' D ′ .
Fourier analysis
Let d d d , d v d_v d v , d y d_y d y , N ∈ N N\in\mathbb{N} N ∈ N . Define the grid { t j } j ∈ J N \{t_j\}_{j\in J_N} { t j } j ∈ J N and J N J_N J N as in § 3 . As we use the projection P ˙ N \dot{P}_N P ˙ N in Algorithm 3.14 , we make use of following estimation, when showing the convergence of the algorithm.
Lemma B.1. Let s ∈ N s\in\mathbb{N} s ∈ N . For a function v ∈ H s ( T d ) v\in H^s(\mathbb{T}^d) v ∈ H s ( T d ) we have ∥ P ˙ N v ∥ H s ( T d ) ≤ ∥ v ∥ H s ( T d ) \|\dot{P}_N v\|_{H^s(\mathbb{T}^d)}\le\|v\|_{H^s(\mathbb{T}^d)} ∥ P ˙ N v ∥ H s ( T d ) ≤ ∥ v ∥ H s ( T d ) .
Proof. To see this, we work with the equivalent definition of the H s H^s H s -norm, see [62 ] , that is
∥ v ∥ H s ( T d ) : = ( ( 2 π ) d 2 ∑ k ∈ Z d ( 1 + ∣ k ∣ 2 s ) ∣ v ^ k ∣ 2 ) 1 / 2 , \|v\|_{H^s(\mathbb{T}^d)}:=\left(\frac{(2\pi)^d}{2}\sum_{k\in\mathbb{Z}^d}(1+|k|^{2s})|\hat{v}_k|^2\right)^{1/2}, ∥ v ∥ H s ( T d ) := ( 2 ( 2 π ) d k ∈ Z d ∑ ( 1 + ∣ k ∣ 2 s ) ∣ v ^ k ∣ 2 ) 1/2 , where v ^ k = F ( v ) ( k ) \hat{v}_k=\mathcal{F}(v)(k) v ^ k = F ( v ) ( k ) are the Fourier coefficients. With this, it is easy to verify that
∥ P ˙ N v ∥ H s ( T d ) = ( ( 2 π ) d 2 ∑ 0 < ∣ k ∣ ∞ ≤ N ( 1 + ∣ k ∣ 2 s ) ∣ v ^ k ∣ 2 ) 1 / 2 ≤ ( ( 2 π ) d 2 ∑ k ∈ Z d ( 1 + ∣ k ∣ 2 s ) ∣ v ^ k ∣ 2 ) 1 / 2 = ∥ v ∥ H s ( T d ) . \|\dot{P}_N v\|_{H^s(\mathbb{T}^d)}
=\left(\frac{(2\pi)^d}{2}\sum_{0<|k|_\infty\le N}(1+|k|^{2s})|\hat{v}_k|^2\right)^{1/2}
\le\left(\frac{(2\pi)^d}{2}\sum_{k\in\mathbb{Z}^d}(1+|k|^{2s})|\hat{v}_k|^2\right)^{1/2}
=\|v\|_{H^s(\mathbb{T}^d)}. ∥ P ˙ N v ∥ H s ( T d ) = 2 ( 2 π ) d 0 < ∣ k ∣ ∞ ≤ N ∑ ( 1 + ∣ k ∣ 2 s ) ∣ v ^ k ∣ 2 1/2 ≤ ( 2 ( 2 π ) d k ∈ Z d ∑ ( 1 + ∣ k ∣ 2 s ) ∣ v ^ k ∣ 2 ) 1/2 = ∥ v ∥ H s ( T d ) .
Following spectral estimate for the pseudo-spectral projection is also needed for the proof of convergence for Algorithm 3.14 .
Lemma B.2 (Pseudo-spectral approximation estimate). For any s > d / 2 s > d/2 s > d /2 the spectral interpolation operator I N : H s ( T d ) → L N 2 ( T d ) \mathcal{I}_N : H^s(\mathbb{T}^d) \to L_N^2(\mathbb{T}^d) I N : H s ( T d ) → L N 2 ( T d ) is well defined. There exists a constant C = C ( s , d ) > 0 C = C(s,d) > 0 C = C ( s , d ) > 0 such that
∥ ( 1 − I N ) v ∥ H ζ ≤ C N − ( s − ζ ) ∥ v ∥ H s \left\|(1-\mathcal{I}_N)v\right\|_{H^\zeta} \le CN^{-(s-\zeta)}\|v\|_{H^s} ∥ ( 1 − I N ) v ∥ H ζ ≤ C N − ( s − ζ ) ∥ v ∥ H s for all v ∈ H s ( T d ) v \in H^s(\mathbb{T}^d) v ∈ H s ( T d ) and for any ζ ∈ [ 0 , s ] \zeta\in[0,s] ζ ∈ [ 0 , s ] .
Proof. See Theorem 2.3 in [58 ] .
As a corollary, we state the boundedness of the interpolation operator I N \mathcal{I}_N I N .
Lemma B.3. For any s > d / 2 s > d/2 s > d /2 the spectral interpolation operator I N : H s ( T d ) → L N 2 ( T d ) \mathcal{I}_N : H^s(\mathbb{T}^d) \to L_N^2(\mathbb{T}^d) I N : H s ( T d ) → L N 2 ( T d ) is bounded in the following sense
∥ I N v ∥ H s ≤ C ∥ v ∥ H s , ∀ v ∈ H s ( T d ) \|\mathcal{I}_Nv\|_{H^s} \le C\|v\|_{H^s}, \quad\forall v \in H^s(\mathbb{T}^d) ∥ I N v ∥ H s ≤ C ∥ v ∥ H s , ∀ v ∈ H s ( T d ) for s > 1 / 2 s > 1/2 s > 1/2 and C = C ( s , d ) > 0 C = C(s,d) > 0 C = C ( s , d ) > 0 .
Proof. For the one-dimensional case one can see [38 ] but one can also use Lemma B.2 with ζ = s \zeta= s ζ = s to see that
∥ I N v ∥ H s ≤ ∥ I N v − v ∥ H s + ∥ v ∥ H s ≤ C ∥ v ∥ H s + ∥ v ∥ H s ≤ ( C + 1 ) ∥ v ∥ H s . \|\mathcal{I}_Nv\|_{H^s} \le\|\mathcal{I}_Nv-v\|_{H^s}+\|v\|_{H^s} \le C\|v\|_{H^s}+\|v\|_{H^s} \le(C+1)\|v\|_{H^s}. ∥ I N v ∥ H s ≤ ∥ I N v − v ∥ H s + ∥ v ∥ H s ≤ C ∥ v ∥ H s + ∥ v ∥ H s ≤ ( C + 1 ) ∥ v ∥ H s .
We give a spectral estimate for the L 2 L2 L 2 -orthogonal projection.
Lemma B.4 (Spectral approximation estimate). Let s > 0 s > 0 s > 0 . There exists a constant C = C ( s , d ) > 0 C = C(s,d) > 0 C = C ( s , d ) > 0 such that for any v ∈ H s ( T d ) v \in H^s(\mathbb{T}^d) v ∈ H s ( T d ) we have
∥ v − P N v ∥ H ζ ≤ C N − ( s − ζ ) ∥ v ∥ H s \|v-P_Nv\|_{H^\zeta} \le CN^{-(s-\zeta)}\|v\|_{H^s} ∥ v − P N v ∥ H ζ ≤ C N − ( s − ζ ) ∥ v ∥ H s for all v ∈ H s ( T d ) v \in H^s(\mathbb{T}^d) v ∈ H s ( T d ) and for any ζ ∈ [ 0 , s ] \zeta\in[0,s] ζ ∈ [ 0 , s ] .
Proof. See Theorem 2.1 in [58 ] .
In § 3 we introduced the architecture of FNOs. In the next, we present that we can decompose that architecture.
Definition B.5 (σ \sigma σ -layer). A σ \sigma σ -layer L = L σ \mathcal{L} = \mathcal{L}_\sigma L = L σ of a Ψ \Psi Ψ -FNO is a local, non-linear layer of the form L σ ( v ) ( x ) = I N σ ( W I N v ( x ) + b ) \mathcal{L}_\sigma(v)(x) = \mathcal{I}_N\sigma(W\mathcal{I}_Nv(x)+b) L σ ( v ) ( x ) = I N σ ( W I N v ( x ) + b ) , or, in the numerical implementation
L σ ( v ) j = σ ( W v j + b j ) , ∀ j ∈ J N , \mathcal{L}_\sigma(v)_j = \sigma(Wv_j+b_j), \quad\forall j \in\mathcal{J}_N, L σ ( v ) j = σ ( W v j + b j ) , ∀ j ∈ J N , with W ∈ R d v × d v W \in\mathbb{R}^{d_v \times d_v} W ∈ R d v × d v and b j ∈ R J N × d v b_j \in\mathbb{R}^{\mathcal{J}_N \times d_v} b j ∈ R J N × d v defining an affine mapping.
Definition B.6 (F \mathcal{F} F -layer). A F \mathcal{F} F -layer L = L F \mathcal{L} = \mathcal{L}_{\mathcal{F}} L = L F of a Ψ \Psi Ψ -FNO is a non-local, linear layer of the form L F ( v ) ( x ) = F − 1 ( P ( k ) ⋅ F ( I N v ) ( k ) ) ( x ) \mathcal{L}_{\mathcal{F}}(v)(x) = \mathcal{F}^{-1}(P(k)\cdot\mathcal{F}(\mathcal{I}_Nv)(k))(x) L F ( v ) ( x ) = F − 1 ( P ( k ) ⋅ F ( I N v ) ( k )) ( x ) , which corresponds in implementation to
L F ( v ) j = F N − 1 ( P ( k ) ⋅ F N ( v ) ( k ) ) j , ∀ j ∈ J N , \mathcal{L}_{\mathcal{F}}(v)_j = \mathcal{F}_N^{-1}\left(P(k)\cdot\mathcal{F}_N(v)(k)\right)_j, \quad\forall j \in\mathcal{J}_N, L F ( v ) j = F N − 1 ( P ( k ) ⋅ F N ( v ) ( k ) ) j , ∀ j ∈ J N , where P : K N → C d v × d v P : \mathcal{K}_N \to\mathbb{C}^{d_v \times d_v} P : K N → C d v × d v is a collection of complex weights, with P ( − k ) = P ( k ) ‾ P(-k)=\overline{P(k)} P ( − k ) = P ( k ) (entry-wise complex conjugation), and F N \mathcal{F}_N F N (F N − 1 \mathcal{F}_N^{-1} F N − 1 ) denotes the discrete (inverse) Fourier transform.
Each Ψ \Psi Ψ -FNO can be decomposed into a finite number of σ \sigma σ -layers and F \mathcal{F} F -layers. The converse is also true. This is proven in the Appendix D.7 of [27 ] by proving following lemmata which are also useful for proving our result.
Lemma B.7 (Composition Lemma). For p ∈ N p \in\mathbb{N} p ∈ N , let N 1 , … , N p \mathcal{N}_1,\ldots,\mathcal{N}_p N 1 , … , N p be Ψ \Psi Ψ -FNOs, defined with respect to the same grid { t j } j ∈ J N \{t_j\}_{j\in\mathcal{J}_N} { t j } j ∈ J N , and defining operators N k : L N 2 ( T d ; R d k − 1 ) → L N 2 ( T d ; R d k ) \mathcal{N}_k : L_N^2(\mathbb{T}^d;\mathbb{R}^{d_{k-1}}) \to L_N^2(\mathbb{T}^d;\mathbb{R}^{d_k}) N k : L N 2 ( T d ; R d k − 1 ) → L N 2 ( T d ; R d k ) for d 0 , … , d p ∈ N d_0,\ldots,d_p \in\mathbb{N} d 0 , … , d p ∈ N , such that the composition N p ∘ ⋯ ∘ N 1 : L N 2 ( T d ; R d 0 ) → L N 2 ( T d ; R d p ) \mathcal{N}_p \circ\cdots\circ\mathcal{N}_1 : L_N^2(\mathbb{T}^d;\mathbb{R}^{d_0}) \to L_N^2(\mathbb{T}^d;\mathbb{R}^{d_p}) N p ∘ ⋯ ∘ N 1 : L N 2 ( T d ; R d 0 ) → L N 2 ( T d ; R d p ) is well defined. Then there exists a Ψ \Psi Ψ -FNO N : L N 2 ( T d ; R d 0 ) → L N 2 ( T d ; R d p ) \mathcal{N} : L_N^2(\mathbb{T}^d;\mathbb{R}^{d_0}) \to L_N^2(\mathbb{T}^d;\mathbb{R}^{d_p}) N : L N 2 ( T d ; R d 0 ) → L N 2 ( T d ; R d p ) , such that
N = N p ∘ ⋯ ∘ N 1 , \mathcal{N}=\mathcal{N}_p\circ\cdots\circ\mathcal{N}_1, N = N p ∘ ⋯ ∘ N 1 , and such that depth ( N ) ≤ ∑ k = 1 p depth ( N k ) \operatorname{depth}(\mathcal{N}) \le\sum_{k=1}^{p}\operatorname{depth}(\mathcal{N}_k) depth ( N ) ≤ ∑ k = 1 p depth ( N k ) , width ( N ) ≤ max k = 1 , … , p width ( N k ) \operatorname{width}(\mathcal{N}) \le\max_{k=1,\ldots,p}\operatorname{width}(\mathcal{N}_k) width ( N ) ≤ max k = 1 , … , p width ( N k ) , lift ( N ) ≤ max k = 1 , … , p lift ( N k ) \operatorname{lift}(\mathcal{N}) \le\max_{k=1,\ldots,p}\operatorname{lift}(\mathcal{N}_k) lift ( N ) ≤ max k = 1 , … , p lift ( N k ) .
Proof. See Lemma D.4 in [27 ] .
Lemma B.8 (Linear approximation lemma). Let σ ∈ C 2 \sigma\in C^{2} σ ∈ C 2 be a twice continuously differentiable and non-constant activation function. Consider a linear Ψ \Psi Ψ -FNO layer L \mathcal{L} L of form
L : L N 2 ( T d ; R d v ) → L N 2 ( T d ; R d v ) , v N ↦ W v N ( t j ) + b j + F N − 1 ( P F N v N ) j . \mathcal{L}: L_{N}^{2}(\mathbb{T}^{d};\mathbb{R}^{d_{v}}) \to L_{N}^{2}(\mathbb{T}^{d};\mathbb{R}^{d_{v}}), \quad v_{N} \mapsto Wv_{N}(t_{j}) + b_{j} + \mathcal{F}_{N}^{-1}(\mathcal{P}\mathcal{F}_{N}v_{N})_{j}. L : L N 2 ( T d ; R d v ) → L N 2 ( T d ; R d v ) , v N ↦ W v N ( t j ) + b j + F N − 1 ( P F N v N ) j . Then there exists a constant C > 0 C > 0 C > 0 , such that for any ε , B > 0 \varepsilon, B > 0 ε , B > 0 , there exists a Ψ \Psi Ψ -FNO N : L N 2 ( T d ; R d v ) → L N 2 ( T ; R d v ) \mathcal{N}: L_{N}^{2}(\mathbb{T}^{d};\mathbb{R}^{d_{v}}) \to L_{N}^{2}(\mathbb{T};\mathbb{R}^{d_{v}}) N : L N 2 ( T d ; R d v ) → L N 2 ( T ; R d v ) , such that
sup ∥ v N ∥ L 2 ≤ B ∥ N ( v N ) − L ( v N ) ∥ L 2 ≤ ε , \sup_{\lVert v_{N}\rVert_{L^{2}}\le B} \lVert\mathcal{N}(v_{N})-\mathcal{L}(v_{N})\rVert_{L^{2}} \le\varepsilon, ∥ v N ∥ L 2 ≤ B sup ∥ N ( v N ) − L ( v N ) ∥ L 2 ≤ ε , and width ( N ) ≤ C N d \operatorname{width}(\mathcal{N}) \le CN^{d} width ( N ) ≤ C N d , depth ( N ) ≤ C \operatorname{depth}(\mathcal{N}) \le C depth ( N ) ≤ C , lift ( N ) ≤ C \operatorname{lift}(\mathcal{N}) \le C lift ( N ) ≤ C .
Proof. See Lemma D.3 in [27 ] .
Lemma B.9 (Replacement lemma). Assume that σ \sigma σ is Lipschitz continuous. For p ∈ N p \in\mathbb{N} p ∈ N , let N ^ 1 , … , N ^ p \widehat{\mathcal{N}}_{1},\ldots,\widehat{\mathcal{N}}_{p} N 1 , … , N p be continuous operators N ^ k : L N 2 ( T d ; R d k − 1 ) → L N 2 ( T d ; R d k ) \widehat{\mathcal{N}}_{k}: L_{N}^{2}(\mathbb{T}^{d};\mathbb{R}^{d_{k-1}}) \to L_{N}^{2}(\mathbb{T}^{d};\mathbb{R}^{d_{k}}) N k : L N 2 ( T d ; R d k − 1 ) → L N 2 ( T d ; R d k ) for d 0 , … , d p ∈ N d_{0},\ldots,d_{p} \in\mathbb{N} d 0 , … , d p ∈ N , such that the composition N ^ : = N ^ p ∘ ⋯ ∘ N ^ 1 : L N 2 ( T d ; R d 0 ) → L N 2 ( T d ; R d p ) \widehat{\mathcal{N}} := \widehat{\mathcal{N}}_{p} \circ\cdots\circ\widehat{\mathcal{N}}_{1}: L_{N}^{2}(\mathbb{T}^{d};\mathbb{R}^{d_{0}}) \to L_{N}^{2}(\mathbb{T}^{d};\mathbb{R}^{d_{p}}) N := N p ∘ ⋯ ∘ N 1 : L N 2 ( T d ; R d 0 ) → L N 2 ( T d ; R d p ) is well-defined. Assume that there exist constants D k , W k , L k > 0 D_{k}, W_{k}, L_{k} > 0 D k , W k , L k > 0 , k = 1 , … , p k=1,\ldots,p k = 1 , … , p , such that for any ε , M > 0 \varepsilon, M > 0 ε , M > 0 , there exists a Ψ \Psi Ψ -FNO N k : L N 2 ( T d ; R d k − 1 ) → L N 2 ( T d ; R d k ) \mathcal{N}_{k}: L_{N}^{2}(\mathbb{T}^{d};\mathbb{R}^{d_{k-1}}) \to L_{N}^{2}(\mathbb{T}^{d};\mathbb{R}^{d_{k}}) N k : L N 2 ( T d ; R d k − 1 ) → L N 2 ( T d ; R d k ) , such that
sup ∥ u ∥ L 2 ≤ M ∥ N ^ k ( u ) − N k ( u ) ∥ L 2 ≤ ε , \sup_{\lVert u\rVert_{L^{2}}\le M} \lVert\widehat{\mathcal{N}}_{k}(u)-\mathcal{N}_{k}(u)\rVert_{L^{2}} \le\varepsilon, ∥ u ∥ L 2 ≤ M sup ∥ N k ( u ) − N k ( u ) ∥ L 2 ≤ ε , and depth ( N k ) ≤ D k \operatorname{depth}(\mathcal{N}_{k}) \le D_{k} depth ( N k ) ≤ D k , width ( N k ) ≤ W k \operatorname{width}(\mathcal{N}_{k}) \le W_{k} width ( N k ) ≤ W k , lift ( N k ) ≤ L k \operatorname{lift}(\mathcal{N}_{k}) \le L_{k} lift ( N k ) ≤ L k . Then for any ε > 0 \varepsilon> 0 ε > 0 and M > 0 M > 0 M > 0 , there exists a Ψ \Psi Ψ -FNO N : L N 2 ( T d ; R d 0 ) → L N 2 ( T d ; R d p ) \mathcal{N}: L_{N}^{2}(\mathbb{T}^{d};\mathbb{R}^{d_{0}}) \to L_{N}^{2}(\mathbb{T}^{d};\mathbb{R}^{d_{p}}) N : L N 2 ( T d ; R d 0 ) → L N 2 ( T d ; R d p ) , such that
sup ∥ u ∥ L 2 ≤ M ∥ N ^ ( u ) − N ( u ) ∥ L 2 ≤ ε , \sup_{\lVert u\rVert_{L^{2}}\le M} \lVert\widehat{\mathcal{N}}(u)-\mathcal{N}(u)\rVert_{L^{2}} \le\varepsilon, ∥ u ∥ L 2 ≤ M sup ∥ N ( u ) − N ( u ) ∥ L 2 ≤ ε , and depth ( N ) ≤ ∑ k = 1 p D k \operatorname{depth}(\mathcal{N}) \le\sum_{k=1}^{p} D_{k} depth ( N ) ≤ ∑ k = 1 p D k , width ( N ) ≤ max k = 1 , … , p W k \operatorname{width}(\mathcal{N}) \le\max_{k=1,\ldots,p} W_{k} width ( N ) ≤ max k = 1 , … , p W k , lift ( N ) ≤ max k = 1 , … , p L k \operatorname{lift}(\mathcal{N}) \le\max_{k=1,\ldots,p} L_{k} lift ( N ) ≤ max k = 1 , … , p L k .
Proof. See Lemma D.5 in [27 ] .
Lemma B.10. Let ϕ : R d v → R d y \phi: \mathbb{R}^{d_{v}} \to\mathbb{R}^{d_{y}} ϕ : R d v → R d y be a (ordinary) neural network with activation function σ \sigma σ . For any N ∈ N N \in\mathbb{N} N ∈ N , the mapping
N : L N 2 ( T d ; R d v ) → L N 2 ( T d ; R d y ) , \mathcal{N}: L_{N}^{2}(\mathbb{T}^{d};\mathbb{R}^{d_{v}}) \to L_{N}^{2}(\mathbb{T}^{d};\mathbb{R}^{d_{y}}), N : L N 2 ( T d ; R d v ) → L N 2 ( T d ; R d y ) , v ( x ) ↦ N ( v ) ( x ) : = I N ϕ ( v ( x ) ) v(x) \mapsto\mathcal{N}(v)(x) := I_{N}\phi(v(x)) v ( x ) ↦ N ( v ) ( x ) := I N ϕ ( v ( x )) can be represented by a Ψ \Psi Ψ -FNO, with
width ( N ) = ∣ J N ∣ width ( ϕ ) , depth ( N ) = depth ( ϕ ) , lift ( N ) = width ( ϕ ) . \operatorname{width}(\mathcal{N}) = |\mathcal{J}_{N}|\operatorname{width}(\phi),\ \operatorname{depth}(\mathcal{N}) = \operatorname{depth}(\phi),\ \operatorname{lift}(\mathcal{N}) = \operatorname{width}(\phi). width ( N ) = ∣ J N ∣ width ( ϕ ) , depth ( N ) = depth ( ϕ ) , lift ( N ) = width ( ϕ ) .
Proof. See Lemma D.2 in [27 ] .
Lemma B.11. Consider an activation function σ \sigma σ with σ ∈ C 3 \sigma\in C^{3} σ ∈ C 3 . Then there exists a constant c ϕ = c ϕ ( d ) > 0 c_{\phi}=c_{\phi}(d)>0 c ϕ = c ϕ ( d ) > 0 , such that for any ε > 0 \varepsilon>0 ε > 0 and B > 0 B>0 B > 0 , there exists
a neural network ϕ 1 : R 2 → R \phi_{1}: \mathbb{R}^{2} \to\mathbb{R} ϕ 1 : R 2 → R , such that
sup ∣ a ∣ , ∣ b ∣ ≤ B ∣ ϕ 1 ( a , b ) − a b ∣ ≤ ε , \sup_{|a|,|b|\le B} |\phi_{1}(a,b)-ab| \le\varepsilon, ∣ a ∣ , ∣ b ∣ ≤ B sup ∣ ϕ 1 ( a , b ) − ab ∣ ≤ ε , and width ( ϕ 1 ) ≤ c ϕ \operatorname{width}(\phi_{1}) \le c_{\phi} width ( ϕ 1 ) ≤ c ϕ and depth ( ϕ 1 ) ≤ c ϕ \operatorname{depth}(\phi_{1}) \le c_{\phi} depth ( ϕ 1 ) ≤ c ϕ ,
a neural network ϕ 2 : R × R d → R d \phi_{2}: \mathbb{R} \times\mathbb{R}^{d} \to\mathbb{R}^{d} ϕ 2 : R × R d → R d , such that
sup ∣ a ∣ , ∥ v ∥ R d ≤ B ∣ ϕ 2 ( a , v ) − a v ∣ ≤ ε , \sup_{|a|,\lVert v\rVert_{\mathbb{R}^{d}}\le B} |\phi_{2}(a,v)-av| \le\varepsilon, ∣ a ∣ , ∥ v ∥ R d ≤ B sup ∣ ϕ 2 ( a , v ) − a v ∣ ≤ ε , and width ( ϕ 2 ) ≤ c ϕ \operatorname{width}(\phi_{2}) \le c_{\phi} width ( ϕ 2 ) ≤ c ϕ and depth ( ϕ 2 ) ≤ c ϕ \operatorname{depth}(\phi_{2}) \le c_{\phi} depth ( ϕ 2 ) ≤ c ϕ .
:::`
Proof. See Lemma E.2 in [27 ] .
Following results are needed for the emulation of Algorithm 3.14 through Ψ \Psi Ψ -FNO layers. First, we clarify that the coefficients of the discrete Fourier transform are in this setting exactly the coefficients of the continuous transform, i.e. the coefficients of a trigonometric polynomial.
Lemma B.12. For k ∈ Z k \in\mathbb{Z} k ∈ Z with ∣ k ∣ ≤ N |k| \le N ∣ k ∣ ≤ N and v N ∈ L N 2 ( T ) v_N \in L_N^2(\mathbb{T}) v N ∈ L N 2 ( T ) with v N ( t ) : = ∑ ∣ k ∣ ∞ ≤ N c k e i ⟨ t , k ⟩ v_N(t) := \sum_{|k|_\infty\le N} c_k e^{i\langle t,k\rangle} v N ( t ) := ∑ ∣ k ∣ ∞ ≤ N c k e i ⟨ t , k ⟩ we have
F N ( v N ) ( k ) = c k = F ( v N ) ( k ) . \mathcal{F}_N(v_N)(k)=c_k=\mathcal{F}(v_N)(k). F N ( v N ) ( k ) = c k = F ( v N ) ( k ) .
Proof. We have
F N ( v N ) ( k ) = 1 2 N + 1 ∑ j ∈ J N v N , j e − 2 π i j k / ( 2 N + 1 ) = 1 2 N + 1 ∑ j ∈ J N v N ( t j ) e − 2 π i j k / ( 2 N + 1 ) = 1 2 N + 1 ∑ j ∈ J N [ ∑ ∣ l ∣ ≤ N c l e i t j l ] e − 2 π i j k / ( 2 N + 1 ) = 1 2 N + 1 ∑ j ∈ J N [ ∑ ∣ l ∣ ≤ N c l e 2 π i j l / ( 2 N + 1 ) ] e − 2 π i j k / ( 2 N + 1 ) = 1 2 N + 1 ∑ j ∈ J N ∑ ∣ l ∣ ≤ N c l e 2 π i j ( l − k ) / ( 2 N + 1 ) = 1 2 N + 1 ∑ ∣ l ∣ ≤ N c l ∑ j ∈ J N e 2 π i j ( l − k ) / ( 2 N + 1 ) = 1 2 N + 1 ∑ ∣ l ∣ ≤ N c l ( 2 N + 1 ) δ l k = c k , \begin{aligned}
\mathcal{F}_N(v_N)(k) &= \frac{1}{2N+1}\sum_{j\in\mathcal{J}_N}v_{N,j}e^{-2\pi i j k/(2N+1)}\\
&= \frac{1}{2N+1}\sum_{j\in\mathcal{J}_N}v_N(t_j)e^{-2\pi i j k/(2N+1)}\\
&= \frac{1}{2N+1}\sum_{j\in\mathcal{J}_N}\left[\sum_{|l|\le N}c_l e^{i t_j l}\right]e^{-2\pi i j k/(2N+1)}\\
&= \frac{1}{2N+1}\sum_{j\in\mathcal{J}_N}\left[\sum_{|l|\le N}c_l e^{2\pi i j l/(2N+1)}\right]e^{-2\pi i j k/(2N+1)}\\
&= \frac{1}{2N+1}\sum_{j\in\mathcal{J}_N}\sum_{|l|\le N}c_l e^{2\pi i j(l-k)/(2N+1)}\\
&= \frac{1}{2N+1}\sum_{|l|\le N}c_l\sum_{j\in\mathcal{J}_N}e^{2\pi i j(l-k)/(2N+1)}\\
&= \frac{1}{2N+1}\sum_{|l|\le N}c_l(2N+1)\delta_{lk}=c_k,
\end{aligned} F N ( v N ) ( k ) = 2 N + 1 1 j ∈ J N ∑ v N , j e − 2 π ij k / ( 2 N + 1 ) = 2 N + 1 1 j ∈ J N ∑ v N ( t j ) e − 2 π ij k / ( 2 N + 1 ) = 2 N + 1 1 j ∈ J N ∑ ∣ l ∣ ≤ N ∑ c l e i t j l e − 2 π ij k / ( 2 N + 1 ) = 2 N + 1 1 j ∈ J N ∑ ∣ l ∣ ≤ N ∑ c l e 2 π ij l / ( 2 N + 1 ) e − 2 π ij k / ( 2 N + 1 ) = 2 N + 1 1 j ∈ J N ∑ ∣ l ∣ ≤ N ∑ c l e 2 π ij ( l − k ) / ( 2 N + 1 ) = 2 N + 1 1 ∣ l ∣ ≤ N ∑ c l j ∈ J N ∑ e 2 π ij ( l − k ) / ( 2 N + 1 ) = 2 N + 1 1 ∣ l ∣ ≤ N ∑ c l ( 2 N + 1 ) δ l k = c k ,
where δ l k \delta_{lk} δ l k defines the Kronecker delta and note, that we used the summation property of the root of unity.
Next result is based on the aforementioned differentiation formula of the Fourier transform. Here, it shows how one can calculate the integral of a trigonometric polynomial with zero mean using the discrete (inverse) Fourier transform.
Lemma B.13. Let v N ∈ L N 2 ( T ) v_N \in L_N^2(\mathbb{T}) v N ∈ L N 2 ( T ) with Fourier coefficients c k = F N ( v N ) ( k ) c_k=\mathcal{F}_N(v_N)(k) c k = F N ( v N ) ( k ) , ∣ k ∣ ≤ N |k|\le N ∣ k ∣ ≤ N , let δ 0 ( k ) : = 1 i k \delta_0(k):=\frac{1}{ik} δ 0 ( k ) := ik 1 for k ≠ 0 k\ne0 k = 0 and δ 0 ( 0 ) : = 0 \delta_0(0):=0 δ 0 ( 0 ) := 0 , and define the trigonometric polynomial
V N ( t ) : = ∑ ∣ k ∣ ≤ N δ 0 ( k ) c k e i k t , t ∈ T . V_N(t):=\sum_{|k|\le N}\delta_0(k)c_k e^{ikt},\qquad t\in\mathbb{T}. V N ( t ) := ∣ k ∣ ≤ N ∑ δ 0 ( k ) c k e ik t , t ∈ T . Then V N ∈ L N 2 ( T ) V_N\in L_N^2(\mathbb{T}) V N ∈ L N 2 ( T ) , V N ′ = P ˙ N v N V_N'=\dot{P}_Nv_N V N ′ = P ˙ N v N , and for all a , t ∈ T a,t\in\mathbb{T} a , t ∈ T
∫ a t ( P ˙ N v N ) ( s ) d s = V N ( t ) − V N ( a ) . \int_a^t(\dot{P}_Nv_N)(s)\,\mathrm{d}s=V_N(t)-V_N(a). ∫ a t ( P ˙ N v N ) ( s ) d s = V N ( t ) − V N ( a ) . On the grid, V N ( t j ) = F N − 1 ( δ 0 ( k ) F N ( v N ) ( k ) ) ( j ) V_N(t_j)=\mathcal{F}_N^{-1}(\delta_0(k)\mathcal{F}_N(v_N)(k))(j) V N ( t j ) = F N − 1 ( δ 0 ( k ) F N ( v N ) ( k )) ( j ) for j ∈ J N j\in\mathcal{J}_N j ∈ J N , i.e. V N V_N V N is obtained from v N v_N v N by a Fourier multiplier.
Proof. Term-wise differentiation gives V N ′ ( t ) = ∑ 0 < ∣ k ∣ ≤ N c k e i k t = ( P ˙ N v N ) ( t ) V_N'(t)=\sum_{0<|k|\le N}c_ke^{ikt}=(\dot{P}_Nv_N)(t) V N ′ ( t ) = ∑ 0 < ∣ k ∣ ≤ N c k e ik t = ( P ˙ N v N ) ( t ) . The claim follows from the fundamental theorem of calculus while the statement on the grid is immediately obtained due to Lemma B.12 and by the definition of the discrete (inverse) Fourier transform.
To calculate the mean over a function inside Algorithm 3.14 , we make use of the following result.
Lemma B.14. For the mean value of v N ∈ L N 2 ( T ) v_N \in L_N^2(\mathbb{T}) v N ∈ L N 2 ( T ) it is
∫ T v N ( t ) d t = F N − 1 ( δ 1 ( k ) F N ( v N ) ( k ) ) ( j ) \int_{\mathbb{T}} v_N(t)\,\mathrm{d}t = F_N^{-1}\bigl(\delta_1(k)F_N(v_N)(k)\bigr)(j) ∫ T v N ( t ) d t = F N − 1 ( δ 1 ( k ) F N ( v N ) ( k ) ) ( j ) for all j ∈ J N j \in\mathcal{J}_N j ∈ J N with
δ 1 ( k ) = { 0 , k ≠ 0 , 1 , k = 0. \delta_1(k) =
\begin{cases}
0, & k \ne0,\\
1, & k = 0.
\end{cases} δ 1 ( k ) = { 0 , 1 , k = 0 , k = 0. Proof. It is
∫ T v N ( t ) d t = c 0 = ∑ ∣ k ∣ ≤ N δ 1 ( k ) c k e i t j k = F N − 1 ( δ 1 ( k ) F N ( v N ) ( k ) ) ( j ) \int_{\mathbb{T}} v_N(t)\,\mathrm{d}t = c_0 = \sum_{|k|\le N} \delta_1(k)c_k e^{\mathrm{i}t_j k} = F_N^{-1}\bigl(\delta_1(k)F_N(v_N)(k)\bigr)(j) ∫ T v N ( t ) d t = c 0 = ∣ k ∣ ≤ N ∑ δ 1 ( k ) c k e i t j k = F N − 1 ( δ 1 ( k ) F N ( v N ) ( k ) ) ( j ) for all j ∈ J N j \in\mathcal{J}_N j ∈ J N with corresponding t j ∈ T t_j \in\mathbb{T} t j ∈ T .
In Algorithm 3.14 , it is also required to do a point-wise evaluation with the output function from the FNO layer before. Using the Dirichlet-kernel, which serves then as an additional input to the FNO, allows us to obtain such evaluations inside the FNO’s architecture.
Lemma B.15. Let v N ∈ L N 2 ( T ) v_N \in L_N^2(\mathbb{T}) v N ∈ L N 2 ( T ) and a ∈ T a \in\mathbb{T} a ∈ T , then it is
v N ( a ) = ∫ T v N ( t ) D N ( a − t ) d t , v_N(a) = \int_{\mathbb{T}} v_N(t)D_N(a-t)\,\mathrm{d}t, v N ( a ) = ∫ T v N ( t ) D N ( a − t ) d t , where D N D_N D N defines the Dirichlet kernel.
Proof. It is
∫ T v N ( t ) D N ( a − t ) d t = ∑ ∣ k ∣ ≤ N ∫ T v N ( t ) e i k ( a − t ) d t = ∑ ∣ k ∣ ≤ N F ( v N ) ( k ) e i k a = v N ( a ) . \int_{\mathbb{T}} v_N(t)D_N(a-t)\,\mathrm{d}t = \sum_{|k|\le N}\int_{\mathbb{T}}v_N(t)e^{\mathrm{i}k(a-t)}\,\mathrm{d}t = \sum_{|k|\le N}\mathcal{F}(v_N)(k)e^{\mathrm{i}ka}=v_N(a). ∫ T v N ( t ) D N ( a − t ) d t = ∣ k ∣ ≤ N ∑ ∫ T v N ( t ) e i k ( a − t ) d t = ∣ k ∣ ≤ N ∑ F ( v N ) ( k ) e i k a = v N ( a ) .
As a Ψ \Psi Ψ -FNO N : L N 2 ( T ; R n a ) → L N 2 ( T ; R n a ) \mathcal{N}:L_N^2(\mathbb{T};\mathbb{R}^{n_a})\to L_N^2(\mathbb{T};\mathbb{R}^{n_a}) N : L N 2 ( T ; R n a ) → L N 2 ( T ; R n a ) outputs functions of form I N y \mathcal{I}_N y I N y in each layer and the multiplication of two L N 2 L_N^2 L N 2 functions lies in L 2 N 2 L_{2N}^2 L 2 N 2 , we need following lemma.
Lemma B.16. Let v 2 N ∈ L 2 N 2 ( T ) v_{2N} \in L_{2N}^2(\mathbb{T}) v 2 N ∈ L 2 N 2 ( T ) , then we have
∫ T I N v 2 N ( s ) d s = ∫ T v 2 N ( s ) d s . \int_{\mathbb{T}}\mathcal{I}_N v_{2N}(s)\,\mathrm{d}s=\int_{\mathbb{T}}v_{2N}(s)\,\mathrm{d}s. ∫ T I N v 2 N ( s ) d s = ∫ T v 2 N ( s ) d s . Proof. Consider v 2 N ∈ L 2 N 2 ( T ) v_{2N}\in L_{2N}^2(\mathbb{T}) v 2 N ∈ L 2 N 2 ( T ) with Fourier coefficients c k c_k c k . Since 1 2 N + 1 ∑ j = 0 2 N e i k t j = 0 \frac{1}{2N+1}\sum_{j=0}^{2N}e^{\mathrm{i}kt_j}=0 2 N + 1 1 ∑ j = 0 2 N e i k t j = 0 for 0 < ∣ k ∣ ≤ 2 N 0<|k|\le2N 0 < ∣ k ∣ ≤ 2 N , it is
1 2 N + 1 ∑ j = 0 2 N v 2 N ( t j ) = 1 2 N + 1 ∑ j = 0 2 N ∑ ∣ k ∣ ≤ 2 N c k e i k t j = ∑ ∣ k ∣ ≤ 2 N c k 1 2 N + 1 ∑ j = 0 2 N e i k t j = c 0 = ∫ T v 2 N ( s ) d s . \frac{1}{2N+1}\sum_{j=0}^{2N}v_{2N}(t_j)=\frac{1}{2N+1}\sum_{j=0}^{2N}\sum_{|k|\le2N}c_ke^{\mathrm{i}kt_j}=\sum_{|k|\le2N}c_k\frac{1}{2N+1}\sum_{j=0}^{2N}e^{\mathrm{i}kt_j}=c_0=\int_{\mathbb{T}}v_{2N}(s)\,\mathrm{d}s. 2 N + 1 1 j = 0 ∑ 2 N v 2 N ( t j ) = 2 N + 1 1 j = 0 ∑ 2 N ∣ k ∣ ≤ 2 N ∑ c k e i k t j = ∣ k ∣ ≤ 2 N ∑ c k 2 N + 1 1 j = 0 ∑ 2 N e i k t j = c 0 = ∫ T v 2 N ( s ) d s . The same computation applied to the trigonometric polynomial I N v 2 N \mathcal{I}_Nv_{2N} I N v 2 N of degree at most N N N gives ∫ T I N v 2 N ( s ) d s = 1 2 N + 1 ∑ j = 0 2 N I N v 2 N ( t j ) \int_{\mathbb{T}}\mathcal{I}_Nv_{2N}(s)\,\mathrm{d}s=\frac{1}{2N+1}\sum_{j=0}^{2N}\mathcal{I}_Nv_{2N}(t_j) ∫ T I N v 2 N ( s ) d s = 2 N + 1 1 ∑ j = 0 2 N I N v 2 N ( t j ) . We conclude by noting that I N v 2 N ( t j ) = v 2 N ( t j ) \mathcal{I}_Nv_{2N}(t_j)=v_{2N}(t_j) I N v 2 N ( t j ) = v 2 N ( t j ) for j ∈ J N j\in\mathcal{J}_N j ∈ J N .
Proof of the main theorem
Let n , n y , N ∈ N n,n_y,N\in\mathbb{N} n , n y , N ∈ N . First we clarify the well-definedness of our proposed Algorithm 3.14 . In the algorithm constant values appear which we consider also as trigonometric polynomials due to the next lemma. That this is well-defined shows following statement.
Lemma C.1. For y ∈ R n y \in\mathbb{R}^{n} y ∈ R n we have that the function give by y ( t ) : = y y(t) := y y ( t ) := y is in L N 2 ( T ; R n ) L^{2}_{N}(\mathbb{T};\mathbb{R}^{n}) L N 2 ( T ; R n ) .
Proof. It is
y ( t ) = ∑ ∣ k ∣ ≤ N c k e i k t = y y(t) = \sum_{|k|\le N} c_{k}e^{ikt} = y y ( t ) = ∣ k ∣ ≤ N ∑ c k e ik t = y
with c k = 0 c_{k}=0 c k = 0 , k ≠ 0 k\ne0 k = 0 and c 0 = y c_{0}=y c 0 = y .
To make sense of the expression under the integral, we define
g n e t ( t ) : = f n e t ( y N , n e t ( t ) , u N ( t ) ) , g_{\mathrm{net}}(t) := f_{\mathrm{net}}(y_{N,\mathrm{net}}(t),u_{N}(t)), g net ( t ) := f net ( y N , net ( t ) , u N ( t )) ,
where y N , n e t ∈ L N 2 ( T ; R n y ) y_{N,\mathrm{net}} \in L^{2}_{N}(\mathbb{T};\mathbb{R}^{n_y}) y N , net ∈ L N 2 ( T ; R n y ) is any trigonometric polynomial of degree N N N . Then, the expression under the integral is understood as a function
t ↦ [ P ˙ N ∘ I N ] ( g n e t ( t ) ) , t ∈ T , t \mapsto[\dot{P}_{N}\circ I_{N}](g_{\mathrm{net}}(t)),\ t\in\mathbb{T}, t ↦ [ P ˙ N ∘ I N ] ( g net ( t )) , t ∈ T ,
meaning that the Fourier projection and interpolation are applied with respect to the time variable. To apply I N I_{N} I N on g n e t g_{\mathrm{net}} g net , the function g n e t g_{\mathrm{net}} g net needs to be at least continuous on T \mathbb{T} T .
Lemma C.3. The function g n e t g_{\mathrm{net}} g net is a continuous function.
Proof. Since we know that f n e t f_{\mathrm{net}} f net is continuous as function
R n y × R 1 + n u → R n y , ( y , u ) ↦ f n e t ( y , u ) , \mathbb{R}^{n_y}\times\mathbb{R}^{1+n_u}\to\mathbb{R}^{n_y},\quad(y,u)\mapsto f_{\mathrm{net}}(y,u), R n y × R 1 + n u → R n y , ( y , u ) ↦ f net ( y , u ) ,
we know that g n e t g_{\mathrm{net}} g net is a continuous function as composition of such, as we have
u N ∈ L N 2 ( T ; R 1 + n u ) , l ≥ 2 , and y N , n e t ∈ L N 2 ( T ; R n y ) . u_{N}\in L^{2}_{N}(\mathbb{T};\mathbb{R}^{1+n_u}), l\ge2,\ \text{and}\ y_{N,\mathrm{net}}\in L^{2}_{N}(\mathbb{T};\mathbb{R}^{n_y}). u N ∈ L N 2 ( T ; R 1 + n u ) , l ≥ 2 , and y N , net ∈ L N 2 ( T ; R n y ) .
Therefore we can apply I N I_{N} I N on g n e t g_{\mathrm{net}} g net and thus apply P ˙ N \dot{P}_{N} P ˙ N which then solely subtracts the mean of I N f n e t ( y , u ) I_{N}f_{\mathrm{net}}(y,u) I N f net ( y , u ) . Following lemma resolves the integrability of the studied function.
Lemma C.4. We have that t ↦ P ˙ N I N g n e t ( t ) , t ∈ T t\mapsto\dot{P}_{N}I_{N}g_{\mathrm{net}}(t),\ t\in\mathbb{T} t ↦ P ˙ N I N g net ( t ) , t ∈ T , is integrable.
Proof. After applying P ˙ N \dot{P}_{N} P ˙ N we end up with a trigonometric polynomial of zero mean. As trigonometric polynomials are continuous, we can integrate the expression P ˙ N I N g n e t ( t ) \dot{P}_{N}I_{N}g_{\mathrm{net}}(t) P ˙ N I N g net ( t ) over compact sets. Alternatively, we have
L ˙ N 2 ( T ; R n y ) ⊂ L N 2 ( T ; R n y ) ⊂ L 2 ( T ; R n y ) ⊂ L 1 ( T ; R n y ) . \dot{L}^{2}_{N}(\mathbb{T};\mathbb{R}^{n_y})\subset L^{2}_{N}(\mathbb{T};\mathbb{R}^{n_y})\subset L^{2}(\mathbb{T};\mathbb{R}^{n_y})\subset L^{1}(\mathbb{T};\mathbb{R}^{n_y}). L ˙ N 2 ( T ; R n y ) ⊂ L N 2 ( T ; R n y ) ⊂ L 2 ( T ; R n y ) ⊂ L 1 ( T ; R n y ) .
It remains the question whether the fixed point formulation of our proposed algorithm is a trigonometric polynomial.
Lemma C.5. When we have v N ∈ L N 2 ( T ) v_{N}\in L^{2}_{N}(\mathbb{T}) v N ∈ L N 2 ( T ) with ∫ T v N ( s ) d s = 0 \int_{\mathbb{T}}v_{N}(s)ds=0 ∫ T v N ( s ) d s = 0 , then also
w ( t ) : = ∫ a t v N ( s ) d s ∈ L N 2 ( T ) , w(t):=\int_{a}^{t}v_{N}(s)ds\in L^{2}_{N}(\mathbb{T}), w ( t ) := ∫ a t v N ( s ) d s ∈ L N 2 ( T ) , where a ∈ T a\in\mathbb{T} a ∈ T .
Proof. We have
w ( t ) = ∫ a t v N ( s ) d s = ∫ a t ∑ ∣ k ∣ ≤ N c k e i k s d s = ∑ ∣ k ∣ ≤ N ∫ a t c k e i k s d s = ∫ a t c 0 e 0 d s + ∑ 0 < ∣ k ∣ ≤ N ∫ a t c k e i k s d s = ( t − a ) c 0 + ∑ 0 < ∣ k ∣ ≤ N c k i k ( e i k t − e i k a ) = ( t − a ) c 0 + ∑ 0 < ∣ k ∣ ≤ N c k i k e i k t − ∑ 0 < ∣ k ∣ ≤ N c k i k e i k a = ∑ 0 < ∣ k ∣ ≤ N c k i k e i k t − ∑ 0 < ∣ k ∣ ≤ N c k i k e i k a , \begin{aligned}
w(t) &= \int_a^t v_N(s)\,\mathrm{d}s \\
&= \int_a^t \sum_{|k|\le N} c_k e^{iks}\,\mathrm{d}s \\
&= \sum_{|k|\le N} \int_a^t c_k e^{iks}\,\mathrm{d}s \\
&= \int_a^t c_0 e^0\,\mathrm{d}s + \sum_{0<|k|\le N}\int_a^t c_k e^{iks}\,\mathrm{d}s \\
&= (t-a)c_0+\sum_{0<|k|\le N}\frac{c_k}{ik}\left(e^{ikt}-e^{ika}\right) \\
&= (t-a)c_0+\sum_{0<|k|\le N}\frac{c_k}{ik}e^{ikt}-\sum_{0<|k|\le N}\frac{c_k}{ik}e^{ika} \\
&= \sum_{0<|k|\le N}\frac{c_k}{ik}e^{ikt}-\sum_{0<|k|\le N}\frac{c_k}{ik}e^{ika},
\end{aligned} w ( t ) = ∫ a t v N ( s ) d s = ∫ a t ∣ k ∣ ≤ N ∑ c k e ik s d s = ∣ k ∣ ≤ N ∑ ∫ a t c k e ik s d s = ∫ a t c 0 e 0 d s + 0 < ∣ k ∣ ≤ N ∑ ∫ a t c k e ik s d s = ( t − a ) c 0 + 0 < ∣ k ∣ ≤ N ∑ ik c k ( e ik t − e ik a ) = ( t − a ) c 0 + 0 < ∣ k ∣ ≤ N ∑ ik c k e ik t − 0 < ∣ k ∣ ≤ N ∑ ik c k e ik a = 0 < ∣ k ∣ ≤ N ∑ ik c k e ik t − 0 < ∣ k ∣ ≤ N ∑ ik c k e ik a ,
where we notice that the last summand does not depend on t t t and is therefore just a constant in L N 2 L_N^2 L N 2 . Further, we know that c 0 = 0 c_0=0 c 0 = 0 due to the assumption. So by setting d 0 : = − ∑ 0 < ∣ k ∣ ≤ N c k i k e i k a d_0:=-\sum_{0<|k|\le N}\frac{c_k}{ik}e^{ika} d 0 := − ∑ 0 < ∣ k ∣ ≤ N ik c k e ik a and d k = c k i k d_k=\frac{c_k}{ik} d k = ik c k , k ≠ 0 k\ne0 k = 0 , we have that w w w is of form
w ( t ) = ∑ ∣ k ∣ ≤ N d k e i k t ∈ L N 2 ( T ) . w(t)=\sum_{|k|\le N}d_k e^{ikt}\in L_N^2(\mathbb{T}). w ( t ) = ∣ k ∣ ≤ N ∑ d k e ik t ∈ L N 2 ( T ) .
We conclude the well-definedness of Algorithm 3.14 with following lemma.
Lemma C.6. For the iterates y N , n e t ( k ) y_{N,\mathrm{net}}^{(k)} y N , net ( k ) , k = 0 , … , K k=0,\ldots,K k = 0 , … , K it is
y N , n e t ( k ) ∈ L N 2 ( T ; R n y ) ⊂ C ∞ ( T ; R n y ) . y_{N,\mathrm{net}}^{(k)}\in L_N^2(\mathbb{T};\mathbb{R}^{n_y})\subset C^\infty(\mathbb{T};\mathbb{R}^{n_y}). y N , net ( k ) ∈ L N 2 ( T ; R n y ) ⊂ C ∞ ( T ; R n y ) .
Proof. We have that
t ↦ y 0 + ∫ t ^ 1 t P ˙ N I N g n e t ( s ) d s + c 0 ( τ N ( t ) − t ^ 1 ) , t ∈ T , t\mapsto y^0+\int_{\hat t_1}^{t}\dot P_N I_N g_{\mathrm{net}}(s)\,\mathrm{d}s+c_0(\tau_N(t)-\hat t_1),\quad t\in\mathbb{T}, t ↦ y 0 + ∫ t ^ 1 t P ˙ N I N g net ( s ) d s + c 0 ( τ N ( t ) − t ^ 1 ) , t ∈ T ,
is a function in L N 2 ( T ; R n y ) L_N^2(\mathbb{T};\mathbb{R}^{n_y}) L N 2 ( T ; R n y ) . As the function we integrate is a trigonometric polynomial with zero mean, we again end up with a trigonometric polynomial in L N 2 ( T ; R n y ) L_N^2(\mathbb{T};\mathbb{R}^{n_y}) L N 2 ( T ; R n y ) , see Lemma C.5 . Since constants are in L N 2 ( T ) L_N^2(\mathbb{T}) L N 2 ( T ) , see Lemma C.1 , and since τ N ∈ L N 2 ( T ) \tau_N\in L_N^2(\mathbb{T}) τ N ∈ L N 2 ( T ) is also a trigonometric polynomial we are done, due to that L N 2 ( T ) L_N^2(\mathbb{T}) L N 2 ( T ) defines a vector space. For the constant function y N , n e t ( 0 ) ( t ) = 0 y_{N,\mathrm{net}}^{(0)}(t)=0 y N , net ( 0 ) ( t ) = 0 it is obvious. Inductively, it follows
y N , n e t ( k ) ( t ) = y 0 + ∫ t ^ 1 t P ˙ N I N f n e t ( y N , n e t ( k − 1 ) ( s ) , u N ( s ) ) d s + c 0 , k ( τ N ( t ) − t ^ 1 ) ∈ L N 2 ( T ; R n y ) y_{N,\mathrm{net}}^{(k)}(t)=y^0+\int_{\hat t_1}^{t}\dot P_N I_N f_{\mathrm{net}}\left(y_{N,\mathrm{net}}^{(k-1)}(s),u_N(s)\right)\,\mathrm{d}s+c_{0,k}(\tau_N(t)-\hat t_1)\in L_N^2(\mathbb{T};\mathbb{R}^{n_y}) y N , net ( k ) ( t ) = y 0 + ∫ t ^ 1 t P ˙ N I N f net ( y N , net ( k − 1 ) ( s ) , u N ( s ) ) d s + c 0 , k ( τ N ( t ) − t ^ 1 ) ∈ L N 2 ( T ; R n y )
for k = 1 , … , K k=1,\ldots,K k = 1 , … , K .
With this, we have proved that the presented algorithm is well defined. We dedicate the rest of this section, to that the proposed algorithm converges as well as that we can emulate that algorithm using layers from the Ψ \Psi Ψ -FNO. We start by choosing certain quantities in a suitable way.
Under Assumption 2.5 we consider the following selection of values. Choose
∥ f ∥ W s , ∞ ( D ′ ; R n y ) ≤ E : = γ 4 K ^ max { U ′ , Y ′ } ∈ R + , (16) \|f\|_{W^{s,\infty}(D';\mathbb{R}^{n_y})}\le E:=\frac{\gamma}{4\widehat K\max\{U',Y'\}}\in\mathbb{R}^{+}, \tag*{(16)} ∥ f ∥ W s , ∞ ( D ′ ; R n y ) ≤ E := 4 K max { U ′ , Y ′ } γ ∈ R + , (16)
where we recall that γ > 0 \gamma> 0 γ > 0 was chosen such that Y ′ = Y + γ > 1 Y' = Y + \gamma> 1 Y ′ = Y + γ > 1 . We refer to Equation (C.1) as smallness condition. The constant K ^ > 0 \widehat{K} > 0 K > 0 is defined by
K ^ : = ( 1 + U ′ ) 2 π K 1 K 2 K 3 K 4 , \widehat{K} := (1+U')^{2\pi}K_{1}K_{2}K_{3}K_{4}, K := ( 1 + U ′ ) 2 π K 1 K 2 K 3 K 4 ,
where K 1 > 0 K_{1} > 0 K 1 > 0 is the constant from the embedding
H 1 ( T ; R n y ) ↪ L ∞ ( T ; R n y ) , H^{1}(\mathbb{T};\mathbb{R}^{n_y}) \hookrightarrow L^{\infty}(\mathbb{T};\mathbb{R}^{n_y}), H 1 ( T ; R n y ) ↪ L ∞ ( T ; R n y ) ,
K 2 > 0 K_{2} > 0 K 2 > 0 is the constant from Lemma B.3 with s = 1 s=1 s = 1 , so
∥ I N v ∥ H 1 ( T ) ≤ K 2 ∥ v ∥ H 1 ( T ) , ∀ v ∈ H 1 ( T ) , \|I_{N}v\|_{H^{1}(\mathbb{T})} \leq K_{2}\|v\|_{H^{1}(\mathbb{T})}, \quad\forall v \in H^{1}(\mathbb{T}), ∥ I N v ∥ H 1 ( T ) ≤ K 2 ∥ v ∥ H 1 ( T ) , ∀ v ∈ H 1 ( T ) ,
K 3 > 0 K_{3} > 0 K 3 > 0 is the constant from the embedding
W 1 , ∞ ( T ; R n y ) ↪ H 1 ( T ; R n y ) , W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y}) \hookrightarrow H^{1}(\mathbb{T};\mathbb{R}^{n_y}), W 1 , ∞ ( T ; R n y ) ↪ H 1 ( T ; R n y ) ,
see Lemma A.5 and K 4 > 0 K_{4} > 0 K 4 > 0 is a constant from Corollary C.11 , which gives us an estimate for the composition of Sobolev functions. The accuracy levels for the network are determined in the following. For N ∈ N N \in\mathbb{N} N ∈ N choose ε R e L U \varepsilon_{\mathrm{ReLU}} ε ReLU with
ε R e L U = N − 1 , (17) \varepsilon_{\mathrm{ReLU}} = N^{-1}, \tag*{(17)} ε ReLU = N − 1 , (17)
and the number L ∈ N L \in\mathbb{N} L ∈ N of the GELU network as
L = ⌈ ( c 0 , G E L U N ) d / ( 2 ( s − 1 ) ) ⌉ ∈ N , (18) L = \left\lceil(c_{0,\mathrm{GELU}}N)^{d/(2(s-1))}\right\rceil\in\mathbb{N}, \tag*{(18)} L = ⌈ ( c 0 , GELU N ) d / ( 2 ( s − 1 )) ⌉ ∈ N , (18)
implying for ε G E L U = c 0 , G E L U L − 2 ( s − 1 ) / d ≤ N − 1 \varepsilon_{\mathrm{GELU}} = c_{0,\mathrm{GELU}}L^{-2(s-1)/d} \leq N^{-1} ε GELU = c 0 , GELU L − 2 ( s − 1 ) / d ≤ N − 1 . Recall that the amount of iterations in Algorithm 3.14 was chosen as K = ⌈ log ( Y N ) / log ( 1 / κ ) ⌉ K = \lceil\log(Y N)/\log(1/\kappa)\rceil K = ⌈ log ( Y N ) / log ( 1/ κ )⌉ . Together with Lemma C.9 and Lemma C.19 we obtain the existence of a constant C ∗ = C ∗ ( n y , n u , E , α , m , s , l , E , A , Y 0 , γ , M 0 , J ) > 0 C^{*} = C^{*}(n_y,n_u,E,\alpha,m,s,l,\mathcal{E},A,Y^{0},\gamma,M_{0},J) > 0 C ∗ = C ∗ ( n y , n u , E , α , m , s , l , E , A , Y 0 , γ , M 0 , J ) > 0 . Since it can be the case that 3 C ∗ min { 1 / 2 , E } > 1 3C^{*}\min\{1/2,E\} > 1 3 C ∗ min { 1/2 , E } > 1 , we set for the main result
ε ∈ ( 0 , min { 1 / 2 , 3 C ∗ min { 1 / 2 , E } } ) , \varepsilon\in(0,\min\{1/2,3C^{*}\min\{1/2,E\}\}), ε ∈ ( 0 , min { 1/2 , 3 C ∗ min { 1/2 , E }}) ,
define ε 0 : = min { 1 / 2 , 3 C ∗ min { 1 / 2 , E } } \varepsilon_{0} := \min\{1/2,3C^{*}\min\{1/2,E\}\} ε 0 := min { 1/2 , 3 C ∗ min { 1/2 , E }} and choose N ∈ N N \in\mathbb{N} N ∈ N by
N = ⌈ 3 C ∗ ε ⌉ . N = \left\lceil\frac{3C^{*}}{\varepsilon}\right\rceil. N = ⌈ ε 3 C ∗ ⌉ .
Otherwise, we would allow for log ( ε − 1 ) < 0 \log(\varepsilon^{-1}) < 0 log ( ε − 1 ) < 0 to happen. Then, for both accuracy levels of the finite neural networks, we have
ε n e t < min { 1 / 2 , E } , \varepsilon_{\mathrm{net}} < \min\{1/2,E\}, ε net < min { 1/2 , E } ,
which is needed for the existence of the network, as well it is needed for the proof of Lemma C.12 . It is important to note that no circular reasoning is present since C ∗ C^{*} C ∗ is not dependent on the constructed networks f n e t f_{\mathrm{net}} f net . The constant is dependent on the width number M 0 M_{0} M 0 that we fix for the GELU network architecture beforehand. The proof of Lemma C.12 could also be done by just requiring ε n e t < min { 1 / 2 , E } \varepsilon_{\mathrm{net}} < \min\{1/2,E\} ε net < min { 1/2 , E } .
Following lemma helps us proving the convergence of Algorithm 3.14 to the true solution of the CODE by solely utilizing the fact that the Picard operator Φ \Phi Φ already defines a κ \kappa κ -contraction.
Lemma C.7 (Disturbance lemma).
Let B ⊂ X B \subset\mathcal{X} B ⊂ X be a closed set in a Banach space X \mathcal{X} X . Further, let Φ : X → X \Phi: \mathcal{X} \to\mathcal{X} Φ : X → X be a κ \kappa κ -contraction. Then assume we have an operator Φ ^ \widehat{\Phi} Φ with properties
Φ ^ ( B ) ⊂ B \widehat{\Phi}(B) \subset B Φ ( B ) ⊂ B ,
sup y ∈ B ∥ Φ ( y ) − Φ ^ ( y ) ∥ X ≤ η \sup_{y \in B}\|\Phi(y)-\widehat{\Phi}(y)\|_{\mathcal{X}} \leq\eta sup y ∈ B ∥Φ ( y ) − Φ ( y ) ∥ X ≤ η .
Then, we have
∥ Φ [ K ] ( y 0 ) − Φ ^ [ K ] ( y 0 ) ∥ X ≤ η 1 − κ \|\Phi^{[K]}(y^{0})-\widehat{\Phi}^{[K]}(y^{0})\|_{\mathcal{X}} \leq\frac{\eta}{1-\kappa} ∥ Φ [ K ] ( y 0 ) − Φ [ K ] ( y 0 ) ∥ X ≤ 1 − κ η for any $y^{0} \in B.
Proof. Define
e k : = ∥ Φ [ k ] ( y 0 ) − Φ ^ [ k ] ( y 0 ) ∥ X . e_k := \left\|\Phi^{[k]}(y^0)-\widehat{\Phi}^{[k]}(y^0)\right\|_{\mathcal{X}}. e k := Φ [ k ] ( y 0 ) − Φ [ k ] ( y 0 ) X .
Then recursively we obtain
e K = ∥ Φ [ K ] ( y 0 ) − Φ ^ [ K ] ( y 0 ) ∥ X = ∥ Φ ( Φ [ K − 1 ] ( y 0 ) ) − Φ ^ ( Φ ^ [ K − 1 ] ( y 0 ) ) ∥ X ≤ ∥ Φ ( Φ [ K − 1 ] ( y 0 ) ) − Φ ( Φ ^ [ K − 1 ] ( y 0 ) ) ∥ X + ∥ Φ ( Φ ^ [ K − 1 ] ( y 0 ) ) − Φ ^ ( Φ ^ [ K − 1 ] ( y 0 ) ) ∥ X ≤ κ e K − 1 + η ≤ κ ( κ e K − 2 + η ) + η ≤ ⋯ ≤ ∑ k = 0 K − 1 κ k η , \begin{aligned}
e_K &= \left\|\Phi^{[K]}(y^0)-\widehat{\Phi}^{[K]}(y^0)\right\|_{\mathcal{X}} \\
&= \left\|\Phi\left(\Phi^{[K-1]}(y^0)\right)-\widehat{\Phi}\left(\widehat{\Phi}^{[K-1]}(y^0)\right)\right\|_{\mathcal{X}} \\
&\le\left\|\Phi\left(\Phi^{[K-1]}(y^0)\right)-\Phi\left(\widehat{\Phi}^{[K-1]}(y^0)\right)\right\|_{\mathcal{X}}+\left\|\Phi\left(\widehat{\Phi}^{[K-1]}(y^0)\right)-\widehat{\Phi}\left(\widehat{\Phi}^{[K-1]}(y^0)\right)\right\|_{\mathcal{X}} \\
&\le\kappa e_{K-1}+\eta\\
&\le\kappa(\kappa e_{K-2}+\eta)+\eta\\
&\le\cdots\\
&\le\sum_{k=0}^{K-1}\kappa^k\eta,
\end{aligned} e K = Φ [ K ] ( y 0 ) − Φ [ K ] ( y 0 ) X = Φ ( Φ [ K − 1 ] ( y 0 ) ) − Φ ( Φ [ K − 1 ] ( y 0 ) ) X ≤ Φ ( Φ [ K − 1 ] ( y 0 ) ) − Φ ( Φ [ K − 1 ] ( y 0 ) ) X + Φ ( Φ [ K − 1 ] ( y 0 ) ) − Φ ( Φ [ K − 1 ] ( y 0 ) ) X ≤ κ e K − 1 + η ≤ κ ( κ e K − 2 + η ) + η ≤ ⋯ ≤ k = 0 ∑ K − 1 κ k η ,
since e 0 = 0 e_0=0 e 0 = 0 . We use the geometric series to obtain ∥ Φ [ K ] ( y 0 ) − Φ ^ [ K ] ( y 0 ) ∥ X ≤ η 1 − κ \left\|\Phi^{[K]}(y^0)-\widehat{\Phi}^{[K]}(y^0)\right\|_{\mathcal{X}}\le\frac{\eta}{1-\kappa} Φ [ K ] ( y 0 ) − Φ [ K ] ( y 0 ) X ≤ 1 − κ η .
To apply the Disturbance lemma, we need to consider a suitable closed subset while also verifying the assumptions regarding the operators Φ \Phi Φ and Φ N , n e t \Phi_{N,\mathrm{net}} Φ N , net . We begin with showing that the set B N \mathcal{B}_N B N , presented earlier, is a closed subset. We recall that J ⊂ T J\subset\mathbb{T} J ⊂ T and Y ′ ∈ R + Y'\in\mathbb{R}^{+} Y ′ ∈ R + .
Lemma C.8. For N ∈ N N\in\mathbb{N} N ∈ N , the set
B N = { z ∈ L N 2 ( J ; R n y ) ∣ ∥ z ∥ W 1 , ∞ ( T ; R n y ) ≤ Y ′ } ⊂ C ( J ; R n y ) \mathcal{B}_N=\left\{z\in L_N^2(J;\mathbb{R}^{n_y})\mathrel{\Big|}\|z\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})}\le Y'\right\}\subset C(J;\mathbb{R}^{n_y}) B N = { z ∈ L N 2 ( J ; R n y ) ∥ z ∥ W 1 , ∞ ( T ; R n y ) ≤ Y ′ } ⊂ C ( J ; R n y ) is closed.
Proof. The operator
V : L N 2 ( J ; R n y ) → L N 2 ( T ; R n y ) , z ↦ z V:L_N^2(J;\mathbb{R}^{n_y})\to L_N^2(\mathbb{T};\mathbb{R}^{n_y}),\quad z\mapsto z V : L N 2 ( J ; R n y ) → L N 2 ( T ; R n y ) , z ↦ z is linear and thus continuous due to L N 2 ( J ; R n y ) L_N^2(J;\mathbb{R}^{n_y}) L N 2 ( J ; R n y ) and L N 2 ( T ; R n y ) L_N^2(\mathbb{T};\mathbb{R}^{n_y}) L N 2 ( T ; R n y ) being finite-dimensional. Also, we have that L N 2 ( T ; R n y ) ⊂ C ( T ; R n y ) L_N^2(\mathbb{T};\mathbb{R}^{n_y})\subset C(\mathbb{T};\mathbb{R}^{n_y}) L N 2 ( T ; R n y ) ⊂ C ( T ; R n y ) is closed due to the finite-dimensionality. Further, the set
A : = { z ∈ C ( T ; R n y ) ∣ ∥ z ∥ W 1 , ∞ ( T ; R n y ) ≤ Y ′ } ⊂ C ( T ; R n y ) A:=\left\{z\in C(\mathbb{T};\mathbb{R}^{n_y})\mathrel{\Big|}\|z\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})}\le Y'\right\}\subset C(\mathbb{T};\mathbb{R}^{n_y}) A := { z ∈ C ( T ; R n y ) ∥ z ∥ W 1 , ∞ ( T ; R n y ) ≤ Y ′ } ⊂ C ( T ; R n y ) is closed. Thus, we have that
B 0 : = L N 2 ( T ; R n y ) ∩ A ⊂ C ( T ; R n y ) \mathcal{B}_0:=L_N^2(\mathbb{T};\mathbb{R}^{n_y})\cap A\subset C(\mathbb{T};\mathbb{R}^{n_y}) B 0 := L N 2 ( T ; R n y ) ∩ A ⊂ C ( T ; R n y ) is closed. We conclude that
B N = V − 1 ( B 0 ) ⊂ C ( J ; R n y ) \mathcal{B}_N=V^{-1}(\mathcal{B}_0)\subset C(J;\mathbb{R}^{n_y}) B N = V − 1 ( B 0 ) ⊂ C ( J ; R n y ) is closed.
The idea now is to apply the disturbance Lemma C.7 on our setting due to the following.
Lemma C.9. Let Assumption 2.5 hold and consider Φ \Phi Φ and Φ N , n e t \Phi_{N,\mathrm{net}} Φ N , net defined as in Equation (10) and Equation (11) , while assuming that Φ N , n e t ∣ J \Phi_{N,\mathrm{net}}|_J Φ N , net ∣ J fulfills the assumption in Lemma C.7 . Let K ∈ N K\in\mathbb{N} K ∈ N and y N , n e t 0 ≡ 0 ∈ L N 2 ( T ; R n y ) y_{N,\mathrm{net}}^0\equiv0\in L_N^2(\mathbb{T};\mathbb{R}^{n_y}) y N , net 0 ≡ 0 ∈ L N 2 ( T ; R n y ) . Then we have
sup y 0 ∈ Y 0 , u ∈ U I , J , A ∥ y ^ ( ⋅ ; u , y 0 ) − Φ N , n e t [ K ] ( y N , n e t 0 ; u N , y 0 , τ N − t ^ 1 ) ∣ J ∥ C ( J ; R n y ) ≤ e λ ∣ J ∣ ( κ K Y + 2 η ) , \sup_{\substack{y^0\in Y^0,\ u\in U^{I,J,A}}}\left\|\widehat{y}(\cdot;u,y^0)-\Phi_{N,\mathrm{net}}^{[K]}\left(y_{N,\mathrm{net}}^0;u_N,y^0,\tau_N-\widehat{t}_1\right)\big|_J\right\|_{C(J;\mathbb{R}^{n_y})}\le e^{\lambda|J|}(\kappa^K Y+2\eta), y 0 ∈ Y 0 , u ∈ U I , J , A sup y ( ⋅ ; u , y 0 ) − Φ N , net [ K ] ( y N , net 0 ; u N , y 0 , τ N − t 1 ) J C ( J ; R n y ) ≤ e λ ∣ J ∣ ( κ K Y + 2 η ) , where y ^ ( ⋅ ; u , y 0 ) \widehat{y}(\cdot;u,y^0) y ( ⋅ ; u , y 0 ) solves y ˙ = f ( y , u ) \dot{y}=f(y,u) y ˙ = f ( y , u ) on J J J with y ( t ^ 1 ) = y 0 y(\widehat{t}_1)=y^0 y ( t 1 ) = y 0 , u N : = P N ( τ , E u ) ∈ L N 2 ( T ; R 1 + n u ) u_N:=P_N(\tau,\mathcal{E}u)\in L_N^2(\mathbb{T};\mathbb{R}^{1+n_u}) u N := P N ( τ , E u ) ∈ L N 2 ( T ; R 1 + n u ) is the projected periodically control with first component τ N = P N τ \tau_N=P_N\tau τ N = P N τ . The value e λ ∣ J ∣ e^{\lambda|J|} e λ ∣ J ∣ is to calculate between the sup-norm and the Bielecki norm.
Proof. Let y N , n e t ( 0 ) ≡ 0 ∈ L N 2 ( T ; R n y ) y_{N,\mathrm{net}}^{(0)} \equiv0 \in L_N^2(\mathbb{T};\mathbb{R}^{n_y}) y N , net ( 0 ) ≡ 0 ∈ L N 2 ( T ; R n y ) . Define v : = ( u , y 0 ) v := (u,y^0) v := ( u , y 0 ) and v N : = ( u N , y 0 , τ N − t ^ 1 ) v_N := (u_N,y^0,\tau_N-\hat{t}_1) v N := ( u N , y 0 , τ N − t ^ 1 ) . Note that Φ ∣ J \Phi|_J Φ ∣ J defines a κ \kappa κ -contraction in the Bielecki norm. Thus in the Bielecki norm, denoted by ∥ ⋅ ∥ X \|\cdot\|_X ∥ ⋅ ∥ X we obtain
∥ y ^ − Φ N , n e t [ K ] ( y N , n e t ( 0 ) ; v N ) ∣ J ∥ X ≤ ∥ y ^ − Φ N , n e t [ K ] ( y N , n e t ( 0 ) ; v ) ∣ J ∥ X + ∥ Φ N , n e t [ K ] ( y N , n e t ( 0 ) ; v ) ∣ J − Φ N , n e t [ K ] ( y N , n e t ( 0 ) ; v N ) ∣ J ∥ X ≤ κ K ∥ y ^ − y N , n e t ( 0 ) ∥ X + η 1 − κ ≤ κ K ∥ y ^ ∥ X + 2 η ≤ κ K ∥ y ^ ∥ C ( J ; R n y ) + 2 η ≤ κ K Y + 2 η , \begin{aligned}
\|\hat{y}-\Phi_{N,\mathrm{net}}^{[K]}(y_{N,\mathrm{net}}^{(0)};v_N)|_J\|_X
&\le\|\hat{y}-\Phi_{N,\mathrm{net}}^{[K]}(y_{N,\mathrm{net}}^{(0)};v)|_J\|_X+\|\Phi_{N,\mathrm{net}}^{[K]}(y_{N,\mathrm{net}}^{(0)};v)|_J-\Phi_{N,\mathrm{net}}^{[K]}(y_{N,\mathrm{net}}^{(0)};v_N)|_J\|_X\\
&\le\kappa^K\|\hat{y}-y_{N,\mathrm{net}}^{(0)}\|_X+\frac{\eta}{1-\kappa}\\
&\le\kappa^K\|\hat{y}\|_X+2\eta\\
&\le\kappa^K\|\hat{y}\|_{C(J;\mathbb{R}^{n_y})}+2\eta\\
&\le\kappa^K Y+2\eta,
\end{aligned} ∥ y ^ − Φ N , net [ K ] ( y N , net ( 0 ) ; v N ) ∣ J ∥ X ≤ ∥ y ^ − Φ N , net [ K ] ( y N , net ( 0 ) ; v ) ∣ J ∥ X + ∥ Φ N , net [ K ] ( y N , net ( 0 ) ; v ) ∣ J − Φ N , net [ K ] ( y N , net ( 0 ) ; v N ) ∣ J ∥ X ≤ κ K ∥ y ^ − y N , net ( 0 ) ∥ X + 1 − κ η ≤ κ K ∥ y ^ ∥ X + 2 η ≤ κ K ∥ y ^ ∥ C ( J ; R n y ) + 2 η ≤ κ K Y + 2 η ,
since κ < 1 / 2 \kappa<1/2 κ < 1/2 .
Before we verify Φ N , n e t ∣ J ( B N ) ⊂ B N \Phi_{N,\mathrm{net}}|_J(\mathcal{B}_N)\subset\mathcal{B}_N Φ N , net ∣ J ( B N ) ⊂ B N , we need to establish the bounds for the control functions, when they are not only periodically extended but also projected using the L 2 L^2 L 2 -orthogonal projection.
Lemma C.10. For a function u ∈ U l , J , A u\in U^{l,J,A} u ∈ U l , J , A , l ≥ 2 l\ge2 l ≥ 2 with u N : = P N E u u_N:=P_N\mathcal{E}u u N := P N E u it is
∥ u N ∥ C ( T ; R 1 + n u ) ≤ ∥ u N ∥ W 1 , ∞ ( T ; R 1 + n u ) ≤ U ′ . \|u_N\|_{C(\mathbb{T};\mathbb{R}^{1+n_u})}\le\|u_N\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{1+n_u})}\le U'. ∥ u N ∥ C ( T ; R 1 + n u ) ≤ ∥ u N ∥ W 1 , ∞ ( T ; R 1 + n u ) ≤ U ′ . Proof. It is
∥ u N ∥ C ( T ; R 1 + n u ) ≤ ∥ u N ∥ W 1 , ∞ ( T ; R 1 + n u ) = ∥ P N u ∥ W 1 , ∞ ( T ; R 1 + n u ) ≤ C 2 ∥ P N u ∥ H l ( T ; R 1 + n u ) ≤ C 2 ∥ u ∥ H l ( T ; R 1 + n u ) ≤ C 2 C E u ∥ u ∥ H l ( J ; R 1 + n u ) ≤ C 2 C E u ( A + t ^ 2 3 − t ^ 1 3 3 + t ^ 2 − t ^ 1 ) ≤ C u C E u ( A + t ^ 2 3 − t ^ 1 3 3 + t ^ 2 − t ^ 1 ) + 2 π = U ′ , \begin{aligned}
\|u_N\|_{C(\mathbb{T};\mathbb{R}^{1+n_u})}
&\le\|u_N\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{1+n_u})}\\
&= \|P_Nu\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{1+n_u})}\\
&\le C_2\|P_Nu\|_{H^l(\mathbb{T};\mathbb{R}^{1+n_u})}\\
&\le C_2\|u\|_{H^l(\mathbb{T};\mathbb{R}^{1+n_u})}\\
&\le C_2C_{\mathcal{E}u}\|u\|_{H^l(J;\mathbb{R}^{1+n_u})}\\
&\le C_2C_{\mathcal{E}u}\left(A+\sqrt{\frac{\hat{t}_2^3-\hat{t}_1^3}{3}+\hat{t}_2-\hat{t}_1}\right)\\
&\le C_uC_{\mathcal{E}u}\left(A+\sqrt{\frac{\hat{t}_2^3-\hat{t}_1^3}{3}+\hat{t}_2-\hat{t}_1}\right)+2\pi=U',
\end{aligned} ∥ u N ∥ C ( T ; R 1 + n u ) ≤ ∥ u N ∥ W 1 , ∞ ( T ; R 1 + n u ) = ∥ P N u ∥ W 1 , ∞ ( T ; R 1 + n u ) ≤ C 2 ∥ P N u ∥ H l ( T ; R 1 + n u ) ≤ C 2 ∥ u ∥ H l ( T ; R 1 + n u ) ≤ C 2 C E u ∥ u ∥ H l ( J ; R 1 + n u ) ≤ C 2 C E u A + 3 t ^ 2 3 − t ^ 1 3 + t ^ 2 − t ^ 1 ≤ C u C E u A + 3 t ^ 2 3 − t ^ 1 3 + t ^ 2 − t ^ 1 + 2 π = U ′ , where the first constant C 2 = C 2 ( W 1 , ∞ ( T ; R 1 + n u ) , l ) > 0 C_2=C_2(W^{1,\infty}(\mathbb{T};\mathbb{R}^{1+n_u}),l)>0 C 2 = C 2 ( W 1 , ∞ ( T ; R 1 + n u ) , l ) > 0 comes from the Sobolev embedding Lemma A.5 and the second constant C E u > 0 C_{\mathcal{E}u}>0 C E u > 0 comes from the continuous, linear periodic extension operator, see Lemma 2.3. Recalling Equation (A.2) and Lemma 2.4 and we see that we are done. Note that we used that P N P_N P N defines a projection. □
To show the aforementioned invariance of Φ N , n e t ∣ J \Phi_{N,\mathrm{net}}|_J Φ N , net ∣ J , we need to use an estimate for compositions of Sobolev functions, presented as next.
Corollary C.11. Let u N u_N u N with u N : = P N u u_N:=P_Nu u N := P N u for u ∈ U l , T , A u\in U^{l,T,A} u ∈ U l , T , A , l ≥ 2 l\ge2 l ≥ 2 and y N , n e t ∈ L N 2 ( T ; R n y ) y_{N,\mathrm{net}}\in L_N^2(\mathbb{T};\mathbb{R}^{n_y}) y N , net ∈ L N 2 ( T ; R n y ) such that
Im ( y N , n e t ) ⊂ ( − Y ′ , Y ′ ) n y . \operatorname{Im}(y_{N,\mathrm{net}})\subset(-Y',Y')^{n_y}. Im ( y N , net ) ⊂ ( − Y ′ , Y ′ ) n y . Consider f n e t : D ′ ⊂ R d → R n y f_{\mathrm{net}}:D'\subset\mathbb{R}^d\to\mathbb{R}^{n_y} f net : D ′ ⊂ R d → R n y , d = 1 + n y + n u d=1+n_y+n_u d = 1 + n y + n u , to be a ReLU or GELU neural network, i.e. f n e t ∈ W 1 , ∞ ( D ′ ; R n y ) f_{\mathrm{net}}\in W^{1,\infty}(D';\mathbb{R}^{n_y}) f net ∈ W 1 , ∞ ( D ′ ; R n y ) . Then, there exists a constant, which we denote by K 4 = K 4 ( d , T ) > 0 K_4=K_4(d,\mathbb{T})>0 K 4 = K 4 ( d , T ) > 0 , such that:
f n e t ( y N , n e t , u N ) ∈ W 1 , ∞ ( T ; R n y ) ⊂ H 1 ( T ; R n y ) f_{\mathrm{net}}(y_{N,\mathrm{net}},u_N)\in W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})\subset H^1(\mathbb{T};\mathbb{R}^{n_y}) f net ( y N , net , u N ) ∈ W 1 , ∞ ( T ; R n y ) ⊂ H 1 ( T ; R n y ) and
∥ f n e t ( y N , n e t , u N ) ∥ W 1 , ∞ ( T ; R n y ) ≤ K 4 max { ∥ f n e t ∥ L ∞ ( D ′ ; R n y ) , ∥ f n e t ∥ W 1 , ∞ ( D ′ ; R n y ) ∥ h ∥ W 1 , ∞ ( T ; R d ) } , \|f_{\mathrm{net}}(y_{N,\mathrm{net}},u_N)\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})}\le K_4\max\{\|f_{\mathrm{net}}\|_{L^\infty(D';\mathbb{R}^{n_y})},\|f_{\mathrm{net}}\|_{W^{1,\infty}(D';\mathbb{R}^{n_y})}\|h\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^d)}\}, ∥ f net ( y N , net , u N ) ∥ W 1 , ∞ ( T ; R n y ) ≤ K 4 max { ∥ f net ∥ L ∞ ( D ′ ; R n y ) , ∥ f net ∥ W 1 , ∞ ( D ′ ; R n y ) ∥ h ∥ W 1 , ∞ ( T ; R d ) } , where
h : T → R d , t ↦ ( y N , n e t ( t ) , u N ( t ) ) . h:\mathbb{T}\to\mathbb{R}^d,\quad t\mapsto(y_{N,\mathrm{net}}(t),u_N(t)). h : T → R d , t ↦ ( y N , net ( t ) , u N ( t )) .
Proof. Use Corollary B.5 from [18 ] and consider there the mentioned domain Ω 1 \Omega_1 Ω 1 as Ω 1 = T ∖ ∂ T \Omega_1 = \mathbb{T}\setminus\partial\mathbb{T} Ω 1 = T ∖ ∂ T . Then, the norms coincide
∥ f n e t ( y N , n e t , u N ) ∥ W 1 , ∞ ( T ∖ ∂ T ; R n y ) = ∥ f n e t ( y N , n e t , u N ) ∥ W 1 , ∞ ( T ; R n y ) . \|f_{\mathrm{net}}(y_{N,\mathrm{net}},u_N)\|_{W^{1,\infty}(\mathbb{T}\setminus\partial\mathbb{T};\mathbb{R}^{n_y})} = \|f_{\mathrm{net}}(y_{N,\mathrm{net}},u_N)\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})}. ∥ f net ( y N , net , u N ) ∥ W 1 , ∞ ( T ∖ ∂ T ; R n y ) = ∥ f net ( y N , net , u N ) ∥ W 1 , ∞ ( T ; R n y ) .
The inner functions are in W 1 , ∞ ( T ; R d ) W^{1,\infty}(\mathbb{T};\mathbb{R}^{d}) W 1 , ∞ ( T ; R d ) since they are in L N 2 ( T ; R d ) ⊂ C ∞ ( T ; R d ) L_N^2(\mathbb{T};\mathbb{R}^{d}) \subset C^\infty(\mathbb{T};\mathbb{R}^{d}) L N 2 ( T ; R d ) ⊂ C ∞ ( T ; R d ) . Then use that the Sobolev semi-norm is smaller than the Sobolev norm.
Last lemma allows us to give an estimate on the bound of the iterates, which then will lead automatically to the invariance of the operator Φ N , n e t ∣ J \Phi_{N,\mathrm{net}}|_J Φ N , net ∣ J .
Lemma C.12. Consider u ∈ U l , T , A u \in\mathcal{U}^{l,T,A} u ∈ U l , T , A , u N : = P N u u_N := P_Nu u N := P N u , y 0 ∈ Y 0 y^0 \in Y^0 y 0 ∈ Y 0 , y N , n e t ( 0 ) ∈ B N y_{N,\mathrm{net}}^{(0)} \in\mathcal{B}_N y N , net ( 0 ) ∈ B N and f : D ′ ⊂ R d → R n y f : D' \subset\mathbb{R}^{d} \to\mathbb{R}^{n_y} f : D ′ ⊂ R d → R n y , d = 1 + n y + n u d = 1+n_y+n_u d = 1 + n y + n u , f ∈ W s , ∞ ( D ′ ; R n y ) f \in W^{s,\infty}(D';\mathbb{R}^{n_y}) f ∈ W s , ∞ ( D ′ ; R n y ) , ∥ f ∥ W s , ∞ ( D ′ ; R n y ) ≤ E \|f\|_{W^{s,\infty}(D';\mathbb{R}^{n_y})} \le E ∥ f ∥ W s , ∞ ( D ′ ; R n y ) ≤ E , s ≥ 2 s \ge2 s ≥ 2 , with f n e t , ε n e t f_{\mathrm{net}},\varepsilon_{\mathrm{net}} f net , ε net as in Equation (9) . For the iterates of Algorithm 3.14 it holds that
∥ y N , n e t ( k ) ∥ W 1 , ∞ ( T ; R n y ) ≤ Y ′ , \|y_{N,\mathrm{net}}^{(k)}\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})} \le Y', ∥ y N , net ( k ) ∥ W 1 , ∞ ( T ; R n y ) ≤ Y ′ , and thus
y N , n e t ( k ) ( t ) ∈ ( − Y ′ , Y ′ ) n y y_{N,\mathrm{net}}^{(k)}(t) \in(-Y',Y')^{n_y} y N , net ( k ) ( t ) ∈ ( − Y ′ , Y ′ ) n y for all t ∈ T t \in\mathbb{T} t ∈ T and k = 0 , … , K k=0,\ldots,K k = 0 , … , K . Recall that Y ′ = Y + γ > 1 Y' = Y+\gamma> 1 Y ′ = Y + γ > 1 .
Proof. It is y N , n e t ( 0 ) ∈ B N y_{N,\mathrm{net}}^{(0)} \in\mathcal{B}_N y N , net ( 0 ) ∈ B N and thus ∥ y N , n e t ( 0 ) ∥ W 1 , ∞ ( T ; R n y ) ≤ Y ′ \|y_{N,\mathrm{net}}^{(0)}\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})} \le Y' ∥ y N , net ( 0 ) ∥ W 1 , ∞ ( T ; R n y ) ≤ Y ′ . So by induction we assume that ∥ y N , n e t ( k − 1 ) ∥ W 1 , ∞ ( T ; R n y ) ≤ Y ′ \|y_{N,\mathrm{net}}^{(k-1)}\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})} \le Y' ∥ y N , net ( k − 1 ) ∥ W 1 , ∞ ( T ; R n y ) ≤ Y ′ , which implies
y N , n e t ( k − 1 ) ( t ) ≤ ( − Y ′ , Y ′ ) n y y_{N,\mathrm{net}}^{(k-1)}(t) \le(-Y',Y')^{n_y} y N , net ( k − 1 ) ( t ) ≤ ( − Y ′ , Y ′ ) n y for all t ∈ T t \in\mathbb{T} t ∈ T . Therefore we can apply Corollary C.11 and obtain
∥ y N , n e t ( k ) ∥ W 1 , ∞ ( T ; R n y ) ≤ ∥ y 0 ∥ R n y + ∥ ∫ t ^ 1 t P ˙ N I N f n e t ( y N , n e t ( k − 1 ) , u N ) d s ∥ W 1 , ∞ ( T ; R n y ) + ∣ c 0 , k ∣ ⋅ ∥ τ N − t ^ 1 ∥ W 1 , ∞ ( T ) , \|y_{N,\mathrm{net}}^{(k)}\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})} \le\|y^0\|_{\mathbb{R}^{n_y}} + \left\|\int_{\hat{t}_1}^{t} \dot{P}_N I_N f_{\mathrm{net}}(y_{N,\mathrm{net}}^{(k-1)},u_N)\,ds\right\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})} + |c_{0,k}|\cdot\|\tau_N-\hat{t}_1\|_{W^{1,\infty}(\mathbb{T})}, ∥ y N , net ( k ) ∥ W 1 , ∞ ( T ; R n y ) ≤ ∥ y 0 ∥ R n y + ∫ t ^ 1 t P ˙ N I N f net ( y N , net ( k − 1 ) , u N ) d s W 1 , ∞ ( T ; R n y ) + ∣ c 0 , k ∣ ⋅ ∥ τ N − t ^ 1 ∥ W 1 , ∞ ( T ) , where the norm is taking over the function [ t ↦ ∫ t ^ 1 t P ˙ N I N f n e t ( y N , n e t ( k − 1 ) , u N ) d s ] \left[t\mapsto\int_{\hat{t}_1}^{t}\dot{P}_N I_N f_{\mathrm{net}}(y_{N,\mathrm{net}}^{(k-1)},u_N)\,ds\right] [ t ↦ ∫ t ^ 1 t P ˙ N I N f net ( y N , net ( k − 1 ) , u N ) d s ] . We can further estimate
∥ ∫ t ^ 1 t P ˙ N I N f n e t ( y N , n e t ( k − 1 ) , u N ) d s ∥ W 1 , ∞ ( T ; R n y ) ≤ 2 π ∥ P ˙ N I N f n e t ( y N , n e t ( k − 1 ) , u N ) ∥ L ∞ ( T ; R n y ) ≤ 2 π K 1 ∥ P ˙ N I N f n e t ( y N , n e t ( k − 1 ) , u N ) ∥ H 1 ( T ; R n y ) ≤ 2 π K 1 ∥ I N f n e t ( y N , n e t ( k − 1 ) , u N ) ∥ H 1 ( T ; R n y ) , \begin{aligned}
\left\|\int_{\hat{t}_1}^{t}\dot{P}_N I_N f_{\mathrm{net}}(y_{N,\mathrm{net}}^{(k-1)},u_N)\,ds\right\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})}
&\le2\pi\|\dot{P}_N I_N f_{\mathrm{net}}(y_{N,\mathrm{net}}^{(k-1)},u_N)\|_{L^\infty(\mathbb{T};\mathbb{R}^{n_y})}\\
&\le2\pi K_1\|\dot{P}_N I_N f_{\mathrm{net}}(y_{N,\mathrm{net}}^{(k-1)},u_N)\|_{H^1(\mathbb{T};\mathbb{R}^{n_y})}\\
&\le2\pi K_1\|I_N f_{\mathrm{net}}(y_{N,\mathrm{net}}^{(k-1)},u_N)\|_{H^1(\mathbb{T};\mathbb{R}^{n_y})},
\end{aligned} ∫ t ^ 1 t P ˙ N I N f net ( y N , net ( k − 1 ) , u N ) d s W 1 , ∞ ( T ; R n y ) ≤ 2 π ∥ P ˙ N I N f net ( y N , net ( k − 1 ) , u N ) ∥ L ∞ ( T ; R n y ) ≤ 2 π K 1 ∥ P ˙ N I N f net ( y N , net ( k − 1 ) , u N ) ∥ H 1 ( T ; R n y ) ≤ 2 π K 1 ∥ I N f net ( y N , net ( k − 1 ) , u N ) ∥ H 1 ( T ; R n y ) , and
∣ c 0 , k ∣ ⋅ ∥ τ N − t ^ 1 ∥ W 1 , ∞ ( T ) ≤ U ′ ∫ T ∣ I N f n e t ( y N , n e t ( k − 1 ) , u N ) ∣ d s ≤ U ′ 2 π ∥ I N f n e t ( y N , n e t ( k − 1 ) , u N ) ∥ L ∞ ( T ; R n y ) ≤ U ′ 2 π K 1 ∥ I N f n e t ( y N , n e t ( k − 1 ) , u N ) ∥ H 1 ( T ; R n y ) , \begin{aligned}
|c_{0,k}|\cdot\|\tau_N-\hat{t}_1\|_{W^{1,\infty}(\mathbb{T})}
&\le U'\int_{\mathbb{T}}|I_N f_{\mathrm{net}}(y_{N,\mathrm{net}}^{(k-1)},u_N)|\,ds\\
&\le U'2\pi\|I_N f_{\mathrm{net}}(y_{N,\mathrm{net}}^{(k-1)},u_N)\|_{L^\infty(\mathbb{T};\mathbb{R}^{n_y})}\\
&\le U'2\pi K_1\|I_N f_{\mathrm{net}}(y_{N,\mathrm{net}}^{(k-1)},u_N)\|_{H^1(\mathbb{T};\mathbb{R}^{n_y})},
\end{aligned} ∣ c 0 , k ∣ ⋅ ∥ τ N − t ^ 1 ∥ W 1 , ∞ ( T ) ≤ U ′ ∫ T ∣ I N f net ( y N , net ( k − 1 ) , u N ) ∣ d s ≤ U ′ 2 π ∥ I N f net ( y N , net ( k − 1 ) , u N ) ∥ L ∞ ( T ; R n y ) ≤ U ′ 2 π K 1 ∥ I N f net ( y N , net ( k − 1 ) , u N ) ∥ H 1 ( T ; R n y ) , where we used that we can estimate ∥ τ N − t ^ 1 ∥ W 1 , ∞ ( T ) \|\tau_N-\hat{t}_1\|_{W^{1,\infty}(\mathbb{T})} ∥ τ N − t ^ 1 ∥ W 1 , ∞ ( T ) also with U ′ ∈ R + U' \in\mathbb{R}_+ U ′ ∈ R + . Combining the estimates, we obtain
∥ y N , n e t ( k ) ∥ W 1 , ∞ ( T ; R n y ) ≤ ∥ y 0 ∥ R n y + ( 1 + U ′ ) 2 π K 1 ∥ I N f n e t ( y N , n e t ( k − 1 ) , u N ) ∥ H 1 ( T ; R n y ) ≤ ∥ y 0 ∥ R n y + 2 π ( 1 + U ′ ) K 1 K 2 ∥ f n e t ( y N , n e t ( k − 1 ) , u N ) ∥ H 1 ( T ; R n y ) ≤ ∥ y 0 ∥ R n y + 2 π ( 1 + U ′ ) K 1 K 2 K 3 ∥ f n e t ( y N , n e t ( k − 1 ) , u N ) ∥ W 1 , ∞ ( T ; R n y ) \begin{aligned}
\|y_{N,\mathrm{net}}^{(k)}\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})}
&\le\|y^0\|_{\mathbb{R}^{n_y}} +(1+U')2\pi K_1\|I_N f_{\mathrm{net}}(y_{N,\mathrm{net}}^{(k-1)},u_N)\|_{H^1(\mathbb{T};\mathbb{R}^{n_y})}\\
&\le\|y^0\|_{\mathbb{R}^{n_y}} +2\pi(1+U')K_1K_2\|f_{\mathrm{net}}(y_{N,\mathrm{net}}^{(k-1)},u_N)\|_{H^1(\mathbb{T};\mathbb{R}^{n_y})}\\
&\le\|y^0\|_{\mathbb{R}^{n_y}} +2\pi(1+U')K_1K_2K_3\|f_{\mathrm{net}}(y_{N,\mathrm{net}}^{(k-1)},u_N)\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})}
\end{aligned} ∥ y N , net ( k ) ∥ W 1 , ∞ ( T ; R n y ) ≤ ∥ y 0 ∥ R n y + ( 1 + U ′ ) 2 π K 1 ∥ I N f net ( y N , net ( k − 1 ) , u N ) ∥ H 1 ( T ; R n y ) ≤ ∥ y 0 ∥ R n y + 2 π ( 1 + U ′ ) K 1 K 2 ∥ f net ( y N , net ( k − 1 ) , u N ) ∥ H 1 ( T ; R n y ) ≤ ∥ y 0 ∥ R n y + 2 π ( 1 + U ′ ) K 1 K 2 K 3 ∥ f net ( y N , net ( k − 1 ) , u N ) ∥ W 1 , ∞ ( T ; R n y )
We do a case distinction due to two possible upper bounds in Corollary C.11 . For the first case we have
2 π ( 1 + U ′ ) K 1 K 2 K 3 ∥ f n e t ( y N , n e t ( k − 1 ) , u N ) ∥ W 1 , ∞ ( T ; R n y ) ≤ 2 π ( 1 + U ′ ) K 1 K 2 K 3 K 4 ∥ f n e t ∥ W 1 , ∞ ( D ′ ; R n y ) ∥ h ∥ W 1 , ∞ ( T ; R d ) = K ^ ∥ f n e t ∥ W 1 , ∞ ( D ′ ; R n y ) ∥ h ∥ W 1 , ∞ ( T ; R d ) ≤ K ^ ∥ f n e t ∥ W 1 , ∞ ( D ′ ; R n y ) 2 max { U ′ , Y ′ } ≤ K ^ ( ∥ f n e t − f ∥ W 1 , ∞ ( D ′ ; R n y ) + ∥ f ∥ W 1 , ∞ ( D ′ ; R n y ) ) 2 max { U ′ , Y ′ } ≤ K ^ ( ∥ f n e t − f ∥ W 1 , ∞ ( D ′ ; R n y ) + ∥ f ∥ W s , ∞ ( D ′ ; R n y ) ) 2 max { U ′ , Y ′ } ≤ K ^ ( ∥ f n e t − f ∥ W 1 , ∞ ( D ′ ; R n y ) + E ) 2 max { U ′ , Y ′ } ≤ K ^ ( γ 4 K ^ max { U ′ , Y ′ } + γ 4 K ^ max { U ′ , Y ′ } ) 2 max { U ′ , Y ′ } = γ , \begin{aligned}
2\pi(1+U')K_1K_2K_3\lVert f_{\mathrm{net}}(y_{N,\mathrm{net}}^{(k-1)},u_N)\rVert_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})}
&\le2\pi(1+U')K_1K_2K_3K_4\lVert f_{\mathrm{net}}\rVert_{W^{1,\infty}(D';\mathbb{R}^{n_y})}\lVert h\rVert_{W^{1,\infty}(\mathbb{T};\mathbb{R}^d)}\\
&=\widehat{K}\lVert f_{\mathrm{net}}\rVert_{W^{1,\infty}(D';\mathbb{R}^{n_y})}\lVert h\rVert_{W^{1,\infty}(\mathbb{T};\mathbb{R}^d)}\\
&\le\widehat{K}\lVert f_{\mathrm{net}}\rVert_{W^{1,\infty}(D';\mathbb{R}^{n_y})}2\max\{U',Y'\}\\
&\le\widehat{K}\left(\lVert f_{\mathrm{net}}-f\rVert_{W^{1,\infty}(D';\mathbb{R}^{n_y})}+\lVert f\rVert_{W^{1,\infty}(D';\mathbb{R}^{n_y})}\right)2\max\{U',Y'\}\\
&\le\widehat{K}\left(\lVert f_{\mathrm{net}}-f\rVert_{W^{1,\infty}(D';\mathbb{R}^{n_y})}+\lVert f\rVert_{W^{s,\infty}(D';\mathbb{R}^{n_y})}\right)2\max\{U',Y'\}\\
&\le\widehat{K}\left(\lVert f_{\mathrm{net}}-f\rVert_{W^{1,\infty}(D';\mathbb{R}^{n_y})}+E\right)2\max\{U',Y'\}\\
&\le\widehat{K}\left(\frac{\gamma}{4\widehat{K}\max\{U',Y'\}}+\frac{\gamma}{4\widehat{K}\max\{U',Y'\}}\right)2\max\{U',Y'\}\\
&=\gamma,
\end{aligned} 2 π ( 1 + U ′ ) K 1 K 2 K 3 ∥ f net ( y N , net ( k − 1 ) , u N ) ∥ W 1 , ∞ ( T ; R n y ) ≤ 2 π ( 1 + U ′ ) K 1 K 2 K 3 K 4 ∥ f net ∥ W 1 , ∞ ( D ′ ; R n y ) ∥ h ∥ W 1 , ∞ ( T ; R d ) = K ∥ f net ∥ W 1 , ∞ ( D ′ ; R n y ) ∥ h ∥ W 1 , ∞ ( T ; R d ) ≤ K ∥ f net ∥ W 1 , ∞ ( D ′ ; R n y ) 2 max { U ′ , Y ′ } ≤ K ( ∥ f net − f ∥ W 1 , ∞ ( D ′ ; R n y ) + ∥ f ∥ W 1 , ∞ ( D ′ ; R n y ) ) 2 max { U ′ , Y ′ } ≤ K ( ∥ f net − f ∥ W 1 , ∞ ( D ′ ; R n y ) + ∥ f ∥ W s , ∞ ( D ′ ; R n y ) ) 2 max { U ′ , Y ′ } ≤ K ( ∥ f net − f ∥ W 1 , ∞ ( D ′ ; R n y ) + E ) 2 max { U ′ , Y ′ } ≤ K ( 4 K max { U ′ , Y ′ } γ + 4 K max { U ′ , Y ′ } γ ) 2 max { U ′ , Y ′ } = γ ,
where h h h is given again by h : T → R d h:\mathbb{T}\to\mathbb{R}^{d} h : T → R d , t ↦ ( y N , n e t ( t ) , u N ( t ) ) t\mapsto(y_{N,\mathrm{net}}(t),u_N(t)) t ↦ ( y N , net ( t ) , u N ( t )) . For the other case we have
2 π ( 1 + U ′ ) K 1 K 2 K 3 ∥ f n e t ( y N , n e t ( k − 1 ) , u N ) ∥ W 1 , ∞ ( T ; R n y ) ≤ 2 π ( 1 + U ′ ) K 1 K 2 K 3 K 4 ∥ f n e t ∥ L ∞ ( D ′ ; R n y ) = K ^ ∥ f n e t ∥ L ∞ ( D ′ ; R n y ) ≤ K ^ ∥ f n e t ∥ W 1 , ∞ ( D ′ ; R n y ) ≤ K ^ ( ∥ f n e t − f ∥ W 1 , ∞ ( D ′ ; R n y ) + ∥ f ∥ W 1 , ∞ ( D ′ ; R n y ) ) ≤ K ^ ( ∥ f n e t − f ∥ W 1 , ∞ ( D ′ ; R n y ) + E ) ≤ γ 2 max { U ′ , Y ′ } 2 max { U ′ , Y ′ } ≤ γ Y ′ ≤ γ , \begin{aligned}
2\pi(1+U')K_1K_2K_3\lVert f_{\mathrm{net}}(y_{N,\mathrm{net}}^{(k-1)},u_N)\rVert_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})}
&\le2\pi(1+U')K_1K_2K_3K_4\lVert f_{\mathrm{net}}\rVert_{L^\infty(D';\mathbb{R}^{n_y})}\\
&=\widehat{K}\lVert f_{\mathrm{net}}\rVert_{L^\infty(D';\mathbb{R}^{n_y})}\\
&\le\widehat{K}\lVert f_{\mathrm{net}}\rVert_{W^{1,\infty}(D';\mathbb{R}^{n_y})}\\
&\le\widehat{K}\left(\lVert f_{\mathrm{net}}-f\rVert_{W^{1,\infty}(D';\mathbb{R}^{n_y})}+\lVert f\rVert_{W^{1,\infty}(D';\mathbb{R}^{n_y})}\right)\\
&\le\widehat{K}\left(\lVert f_{\mathrm{net}}-f\rVert_{W^{1,\infty}(D';\mathbb{R}^{n_y})}+E\right)\\
&\le\frac{\gamma}{2\max\{U',Y'\}}2\max\{U',Y'\}\\
&\le\frac{\gamma}{Y'}\\
&\le\gamma,
\end{aligned} 2 π ( 1 + U ′ ) K 1 K 2 K 3 ∥ f net ( y N , net ( k − 1 ) , u N ) ∥ W 1 , ∞ ( T ; R n y ) ≤ 2 π ( 1 + U ′ ) K 1 K 2 K 3 K 4 ∥ f net ∥ L ∞ ( D ′ ; R n y ) = K ∥ f net ∥ L ∞ ( D ′ ; R n y ) ≤ K ∥ f net ∥ W 1 , ∞ ( D ′ ; R n y ) ≤ K ( ∥ f net − f ∥ W 1 , ∞ ( D ′ ; R n y ) + ∥ f ∥ W 1 , ∞ ( D ′ ; R n y ) ) ≤ K ( ∥ f net − f ∥ W 1 , ∞ ( D ′ ; R n y ) + E ) ≤ 2 max { U ′ , Y ′ } γ 2 max { U ′ , Y ′ } ≤ Y ′ γ ≤ γ ,
since we choose γ \gamma γ exactly like that, see Equation (4) . Besides, we used that P ˙ N \dot{P}_N P ˙ N is a projection in the sense of Lemma B.1. We also used that I N I_N I N is bounded, see Lemma B.3 . Further we used the different Sobolev embeddings. Now since y 0 ∈ ( − Y , Y ) n y y^0\in(-Y,Y)^{n_y} y 0 ∈ ( − Y , Y ) n y and Y ′ = Y + γ Y'=Y+\gamma Y ′ = Y + γ , we are done.
In the proof, we also showed the boundedness for the composition f n e t ( y , u N ) f_{\mathrm{net}}(y,u_N) f net ( y , u N ) regarding ∥ ⋅ ∥ H 1 ( T ; R n y ) \lVert\cdot\rVert_{H^1(\mathbb{T};\mathbb{R}^{n_y})} ∥ ⋅ ∥ H 1 ( T ; R n y ) -norm, which is stated next as a corollary.
Corollary C.13. For f n e t f_{\mathrm{net}} f net , y y y and u N u_N u N as defined in Lemma C.12 , we have that f n e t ( y , u N ) f_{\mathrm{net}}(y,u_N) f net ( y , u N ) is bounded with respect to the ∥ ⋅ ∥ H 1 ( T ; R n y ) \lVert\cdot\rVert_{H^1(\mathbb{T};\mathbb{R}^{n_y})} ∥ ⋅ ∥ H 1 ( T ; R n y ) -norm for every y ∈ B N y\in\mathcal{B}_N y ∈ B N and u ∈ U l , T , A u\in\mathcal{U}^{l,T,A} u ∈ U l , T , A , u N : = P N u u_N:=P_Nu u N := P N u . The upper bound, denoted by D ˉ > 0 \bar{D}>0 D ˉ > 0 , is dependent on n y , n u , E , α , m , s , l , E , A , Y 0 , γ n_y,n_u,E,\alpha,m,s,l,\mathcal{E},A,Y^0,\gamma n y , n u , E , α , m , s , l , E , A , Y 0 , γ and on the GELU network width M 0 M_0 M 0 .
Proof. See proof of Lemma C.12 . There, one can see that the upper bound depends on K 4 K_4 K 4 , which depends on d = 1 + n y + n u d=1+n_y+n_u d = 1 + n y + n u , and f n e t f_{\mathrm{net}} f net , E E E and h h h . The network depends solely on f f f while h h h is dependent on U ′ U' U ′ and Y ′ Y' Y ′ . The remaining dependence of f f f is α \alpha α , m m m and s s s , see Assumption 2.5 . The domain U ′ U' U ′ is dependent on l l l , E E E and A A A , see Lemma C.10 for this dependence. The remaining dependence of the domain Y ′ Y' Y ′ is on Y 0 Y^0 Y 0 and γ \gamma γ , see Equation (4) . When f n e t = f G E L U f_{\mathrm{net}}=f_{\mathrm{GELU}} f net = f GELU , then we also have dependence on the fixed width M 0 M_0 M 0 . Thus, we choose the constant D ˉ > 0 \bar{D}>0 D ˉ > 0 as the maximum over both constants obtained either by f R e L U f_{\mathrm{ReLU}} f ReLU or f G E L U f_{\mathrm{GELU}} f GELU .
We can now show that Φ N , n e t ∣ J \Phi_{N,\mathrm{net}}|_J Φ N , net ∣ J verifies the first assumption from Lemma C.7 .
Corollary C.14. Consider the closed subset
B N = { z ∈ L N 2 ( J ; R n y ) ∣ ∥ z ∥ W 1 , ∞ ( T ; R n y ) ≤ Y ′ } ⊂ C ( J ; R n y ) \mathcal{B}_N=\left\{z\in L_N^2(J;\mathbb{R}^{n_y})\mid\lVert z\rVert_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})}\le Y'\right\}\subset C(J;\mathbb{R}^{n_y}) B N = { z ∈ L N 2 ( J ; R n y ) ∣ ∥ z ∥ W 1 , ∞ ( T ; R n y ) ≤ Y ′ } ⊂ C ( J ; R n y ) and Φ N , n e t \Phi_{N,\mathrm{net}} Φ N , net defined as in Equation (11) under the setting of Lemma C.12 , we then have
Φ N , n e t ∣ J ( B N ) ⊂ B N . \Phi_{N,\mathrm{net}}|_J(\mathcal{B}_N)\subset\mathcal{B}_N. Φ N , net ∣ J ( B N ) ⊂ B N .
Proof. Let z ∈ B N z \in\mathcal{B}_{N} z ∈ B N , then Φ N , n e t ∣ J ( z ) \Phi_{N,\mathrm{net}}|_{J}(z) Φ N , net ∣ J ( z ) is a trigonometric polynomial on J J J with norm on T \mathbb{T} T such that
∥ Φ N , n e t ( z ) ∥ W 1 , ∞ ( T ; R n y ) ≤ Y ′ , \left\|\Phi_{N,\mathrm{net}}(z)\right\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})} \leq Y', ∥ Φ N , net ( z ) ∥ W 1 , ∞ ( T ; R n y ) ≤ Y ′ ,
due to Lemma C.12 .
Following useful lemma can be found in the appendix of [18 ] and will help us to show the second assumption in Lemma C.7 .
Lemma C.15 (Lipschitz characterization of Sobolev functions [18 ] ). Let D ⊂ R d D \subset\mathbb{R}^{d} D ⊂ R d , d ∈ N d \in\mathbb{N} d ∈ N be open and convex. If f ∈ W 1 , ∞ ( D ) f \in W^{1,\infty}(D) f ∈ W 1 , ∞ ( D ) , then f f f is ∥ ∥ ∇ f ( x ) ∥ ∥ L ∞ ( D ) \|\|\nabla f(x)\|\|_{L^\infty(D)} ∥∥∇ f ( x ) ∥ ∥ L ∞ ( D ) -Lipschitz. Conversely, if f f f is bounded and L L L -Lipschitz on D D D , then f ∈ W 1 , ∞ ( D ) f \in W^{1,\infty}(D) f ∈ W 1 , ∞ ( D ) .
Proof. See [18 ] , Theorem B.3.
Corollary C.17. Consider f f f under Assumption 2.5 . There exists a Lipschitz-constant L u L_u L u such that
∥ f ( y , u ) − f ( y , v ) ∥ R n y ≤ L u ∥ u − v ∥ R 1 + n u \|f(y,u)-f(y,v)\|_{\mathbb{R}^{n_y}} \leq L_u\|u-v\|_{\mathbb{R}^{1+n_u}} ∥ f ( y , u ) − f ( y , v ) ∥ R n y ≤ L u ∥ u − v ∥ R 1 + n u for all u , v ∈ ( − U ′ , U ′ ) 1 + n u u,v \in(-U',U')^{1+n_u} u , v ∈ ( − U ′ , U ′ ) 1 + n u .
Proof. This is simply due to the fact that now, due to the finite neural network approximation, we assume f ∈ W s , ∞ ( D ′ ; R n y ) f \in W^{s,\infty}(D';\mathbb{R}^{n_y}) f ∈ W s , ∞ ( D ′ ; R n y ) . Then using Lemma C.15 gives us the statement.
We are left with proving the second assumption.
Lemma C.19. Consider the closed subset
B N = { z ∈ L N 2 ( J ; R n y ) ∣ ∥ z ∥ W 1 , ∞ ( T ; R n y ) ≤ Y ′ } ⊂ C ( J ; R n y ) . \mathcal{B}_{N}=\left\{z\in L_{N}^{2}(J;\mathbb{R}^{n_y})\mid\|z\|_{W^{1,\infty}(\mathbb{T};\mathbb{R}^{n_y})}\leq Y'\right\}\subset C(J;\mathbb{R}^{n_y}). B N = { z ∈ L N 2 ( J ; R n y ) ∣ ∥ z ∥ W 1 , ∞ ( T ; R n y ) ≤ Y ′ } ⊂ C ( J ; R n y ) . Let J = ( t ^ 1 , t ^ 2 ) ⊂ T J=(\hat{t}_1,\hat{t}_2)\subset\mathbb{T} J = ( t ^ 1 , t ^ 2 ) ⊂ T , N ∈ N N\in\mathbb{N} N ∈ N and f n e t , ε n e t f_{\mathrm{net}},\varepsilon_{\mathrm{net}} f net , ε net as in Equation (9) . For Φ \Phi Φ and Φ N , n e t \Phi_{N,\mathrm{net}} Φ N , net defined as in Equation (10) and Equation (11) there exists a constant C ′ = C ′ ( n y , n u , E , α , m , s , l , E , A , Y 0 , γ , M 0 , J ) > 0 C'=C'(n_y,n_u,E,\alpha,m,s,l,\mathcal{E},A,Y^0,\gamma,M_0,J)>0 C ′ = C ′ ( n y , n u , E , α , m , s , l , E , A , Y 0 , γ , M 0 , J ) > 0 such that
sup z ∈ B N ∥ Φ ( z ; u , y 0 ) ∣ J − Φ N , n e t ( z ; u N , y 0 , τ N − t ^ 1 ) ∣ J ∥ C ( J ; R n y ) ≤ C ′ ( N − 1 + ε n e t ) , \sup_{z\in\mathcal{B}_{N}}\left\|\Phi(z;u,y^0)|_{J}-\Phi_{N,\mathrm{net}}(z;u_N,y^0,\tau_N-\hat{t}_1)|_{J}\right\|_{C(J;\mathbb{R}^{n_y})}\leq C'(N^{-1}+\varepsilon_{\mathrm{net}}), z ∈ B N sup Φ ( z ; u , y 0 ) ∣ J − Φ N , net ( z ; u N , y 0 , τ N − t ^ 1 ) ∣ J C ( J ; R n y ) ≤ C ′ ( N − 1 + ε net ) , where y 0 ∈ Y 0 y^0\in\mathcal{Y}^0 y 0 ∈ Y 0 , u ∈ U l , J , A u\in\mathcal{U}^{l,J,A} u ∈ U l , J , A , u N : = P N ( τ , E u ) ∈ L N 2 ( T ; R 1 + n u ) u_N:=P_N(\tau,\mathcal{E}u)\in L_N^2(\mathbb{T};\mathbb{R}^{1+n_u}) u N := P N ( τ , E u ) ∈ L N 2 ( T ; R 1 + n u ) is the projected periodically extended control with first component τ N = P N τ \tau_N=P_N\tau τ N = P N τ .
Proof. Define v : = ( u , y 0 ) v:=(u,y^0) v := ( u , y 0 ) and v N : = ( u N , y 0 , τ N − t ^ 1 ) v_N:=(u_N,y^0,\tau_N-\hat{t}_1) v N := ( u N , y 0 , τ N − t ^ 1 ) . For z ∈ B N z\in\mathcal{B}_N z ∈ B N and t ∈ J = ( t ^ 1 , t ^ 2 ) ⊂ T t\in J=(\hat{t}_1,\hat{t}_2)\subset\mathbb{T} t ∈ J = ( t ^ 1 , t ^ 2 ) ⊂ T it is
∥ Φ ( z ; v ) [ t ] − Φ N , n e t ( z ; v N ) [ t ] ∥ R n y \left\|\Phi(z;v)[t]-\Phi_{N,\mathrm{net}}(z;v_N)[t]\right\|_{\mathbb{R}^{n_y}} ∥ Φ ( z ; v ) [ t ] − Φ N , net ( z ; v N ) [ t ] ∥ R n y = ∥ ∫ t ^ 1 t [ f ( z , u ) − P ˙ N I N f n e t ( z , u N ) ] d s − c 0 ( τ N ( t ) − t ^ 1 ) ∥ R n y =\left\|\int_{\hat{t}_1}^{t}\left[f(z,u)-\dot{P}_N I_Nf_{\mathrm{net}}(z,u_N)\right]\,ds-c_0(\tau_N(t)-\hat{t}_1)\right\|_{\mathbb{R}^{n_y}} = ∫ t ^ 1 t [ f ( z , u ) − P ˙ N I N f net ( z , u N ) ] d s − c 0 ( τ N ( t ) − t ^ 1 ) R n y = ∥ ∫ t ^ 1 t [ f ( z , u ) − I N f n e t ( z , u N ) + ( ∫ T I N f n e t ( z , u N ) d s 2 ) ] d s 1 − c 0 ( τ N ( t ) − t ^ 1 ) ∥ R n y =\left\|\int_{\hat{t}_1}^{t}\left[f(z,u)-I_Nf_{\mathrm{net}}(z,u_N)+\left(\int_{\mathbb{T}}I_Nf_{\mathrm{net}}(z,u_N)\,ds_2\right)\right]\,ds_1-c_0(\tau_N(t)-\hat{t}_1)\right\|_{\mathbb{R}^{n_y}} = ∫ t ^ 1 t [ f ( z , u ) − I N f net ( z , u N ) + ( ∫ T I N f net ( z , u N ) d s 2 ) ] d s 1 − c 0 ( τ N ( t ) − t ^ 1 ) R n y ≤ ∫ t ^ 1 t ∥ f ( z , u ) − I N f n e t ( z , u N ) ∥ R n y d s + ∥ ( t ^ 1 − t ) ∫ T I N f n e t ( z , u N ) d s − c 0 ( τ N ( t ) − t ^ 1 ) ∥ R n y \leq\int_{\hat{t}_1}^{t}\left\|f(z,u)-I_Nf_{\mathrm{net}}(z,u_N)\right\|_{\mathbb{R}^{n_y}}\,ds+\left\|(\hat{t}_1-t)\int_{\mathbb{T}}I_Nf_{\mathrm{net}}(z,u_N)\,ds-c_0(\tau_N(t)-\hat{t}_1)\right\|_{\mathbb{R}^{n_y}} ≤ ∫ t ^ 1 t ∥ f ( z , u ) − I N f net ( z , u N ) ∥ R n y d s + ( t ^ 1 − t ) ∫ T I N f net ( z , u N ) d s − c 0 ( τ N ( t ) − t ^ 1 ) R n y ≤ ∫ t ^ 1 t ∥ f ( z , u ) − I N f n e t ( z , u N ) ∥ R n y d s + ∣ c 0 ∣ ∣ t − τ N ( t ) ∣ , \leq\int_{\hat{t}_1}^{t}\left\|f(z,u)-I_Nf_{\mathrm{net}}(z,u_N)\right\|_{\mathbb{R}^{n_y}}\,ds+|c_0|\,|t-\tau_N(t)|, ≤ ∫ t ^ 1 t ∥ f ( z , u ) − I N f net ( z , u N ) ∥ R n y d s + ∣ c 0 ∣ ∣ t − τ N ( t ) ∣ , where the first term is (I) and the second term is (II).
where we already know that c 0 c_0 c 0 is bounded independent of z ∈ B N z \in\mathcal{B}_N z ∈ B N and u N u_N u N as seen in the proof of Lemma C.12 . For (I) we get
∫ t ^ 1 t ∥ f ( z , u ) − I N f n e t ( z , u N ) ∥ R n y d s ≤ ∫ 0 2 π ∥ f ( z , u ) − f ( z , u N ) ∥ R n y d s + ∫ t ^ 1 t ^ 2 ∥ f ( z , u N ) − f n e t ( z , u N ) ∥ R n y d s + ∫ 0 2 π ∥ f n e t ( z , u N ) − I N f n e t ( z , u N ) ∥ R n y d s ≤ ∫ 0 2 π L u ∥ u − u N ∥ R n u d s + ∣ J ∣ ⋅ ∥ f − f n e t ∥ L ∞ ( D ′ ) + 2 π ∥ f n e t ( z , u N ) − I N f n e t ( z , u N ) ∥ L 2 ( T ; R n y ) ≤ 2 π L u ∥ u − u N ∥ L 2 ( T ; R 1 + n u ) + ∣ J ∣ ε n e t + 2 π ∥ f n e t ( z , u N ) − I N f n e t ( z , u N ) ∥ L 2 ( T ; R n y ) ≤ 2 π L u C ~ N − l ∥ u ∥ H l ( T ; R 1 + n u ) + ∣ J ∣ ε n e t + 2 π ∥ f n e t ( z , u N ) − I N f n e t ( z , u N ) ∥ L 2 ( T ; R n y ) ≤ 2 π L u C ~ N − l C τ , u + ∣ J ∣ ε n e t + 2 π ∥ f n e t ( z , u N ) − I N f n e t ( z , u N ) ∥ L 2 ( T ; R n y ) , \begin{aligned}
\int_{\hat{t}_1}^{t} \|f(z,u)-I_Nf_{\mathrm{net}}(z,u_N)\|_{\mathbb{R}^{n_y}}\,\mathrm{d}s \\
&\leq\int_{0}^{2\pi}\|f(z,u)-f(z,u_N)\|_{\mathbb{R}^{n_y}}\,\mathrm{d}s+\int_{\hat{t}_1}^{\hat{t}_2}\|f(z,u_N)-f_{\mathrm{net}}(z,u_N)\|_{\mathbb{R}^{n_y}}\,\mathrm{d}s+\int_{0}^{2\pi}\|f_{\mathrm{net}}(z,u_N)-I_Nf_{\mathrm{net}}(z,u_N)\|_{\mathbb{R}^{n_y}}\,\mathrm{d}s \\
&\leq\int_{0}^{2\pi}L_u\|u-u_N\|_{\mathbb{R}^{n_u}}\,\mathrm{d}s+|J|\cdot\|f-f_{\mathrm{net}}\|_{L^\infty(D')}+\sqrt{2\pi}\|f_{\mathrm{net}}(z,u_N)-I_Nf_{\mathrm{net}}(z,u_N)\|_{L^2(\mathbb{T};\mathbb{R}^{n_y})} \\
&\leq\sqrt{2\pi}L_u\|u-u_N\|_{L^2(\mathbb{T};\mathbb{R}^{1+n_u})}+|J|\varepsilon_{\mathrm{net}}+\sqrt{2\pi}\|f_{\mathrm{net}}(z,u_N)-I_Nf_{\mathrm{net}}(z,u_N)\|_{L^2(\mathbb{T};\mathbb{R}^{n_y})} \\
&\leq\sqrt{2\pi}L_u\widetilde{C}N^{-l}\|u\|_{H^l(\mathbb{T};\mathbb{R}^{1+n_u})}+|J|\varepsilon_{\mathrm{net}}+\sqrt{2\pi}\|f_{\mathrm{net}}(z,u_N)-I_Nf_{\mathrm{net}}(z,u_N)\|_{L^2(\mathbb{T};\mathbb{R}^{n_y})} \\
&\leq\sqrt{2\pi}L_u\widetilde{C}N^{-l}C_{\tau,u}+|J|\varepsilon_{\mathrm{net}}+\sqrt{2\pi}\|f_{\mathrm{net}}(z,u_N)-I_Nf_{\mathrm{net}}(z,u_N)\|_{L^2(\mathbb{T};\mathbb{R}^{n_y})},
\end{aligned} ∫ t ^ 1 t ∥ f ( z , u ) − I N f net ( z , u N ) ∥ R n y d s ≤ ∫ 0 2 π ∥ f ( z , u ) − f ( z , u N ) ∥ R n y d s + ∫ t ^ 1 t ^ 2 ∥ f ( z , u N ) − f net ( z , u N ) ∥ R n y d s + ∫ 0 2 π ∥ f net ( z , u N ) − I N f net ( z , u N ) ∥ R n y d s ≤ ∫ 0 2 π L u ∥ u − u N ∥ R n u d s + ∣ J ∣ ⋅ ∥ f − f net ∥ L ∞ ( D ′ ) + 2 π ∥ f net ( z , u N ) − I N f net ( z , u N ) ∥ L 2 ( T ; R n y ) ≤ 2 π L u ∥ u − u N ∥ L 2 ( T ; R 1 + n u ) + ∣ J ∣ ε net + 2 π ∥ f net ( z , u N ) − I N f net ( z , u N ) ∥ L 2 ( T ; R n y ) ≤ 2 π L u C N − l ∥ u ∥ H l ( T ; R 1 + n u ) + ∣ J ∣ ε net + 2 π ∥ f net ( z , u N ) − I N f net ( z , u N ) ∥ L 2 ( T ; R n y ) ≤ 2 π L u C N − l C τ , u + ∣ J ∣ ε net + 2 π ∥ f net ( z , u N ) − I N f net ( z , u N ) ∥ L 2 ( T ; R n y ) ,
with constant C ~ = C ~ ( l ) > 0 \widetilde{C}=\widetilde{C}(l)>0 C = C ( l ) > 0 coming from Lemma B.4 and where C τ , u > 0 C_{\tau,u}>0 C τ , u > 0 is the bound of the periodically extended controls with respect to the included time variation τ \tau τ , for which showed boundedness in ∥ ⋅ ∥ H l ( T ) \|\cdot\|_{H^l(\mathbb{T})} ∥ ⋅ ∥ H l ( T ) at the end of Appendix § 8 . For the further estimation we do
2 π L u C ~ N − l C τ , u + ∣ J ∣ ε n e t + 2 π ∥ f n e t ( z , u N ) − I N f n e t ( z , u N ) ∥ L 2 ( T ; R n y ) ≤ 2 π L u C ~ N − l C τ , u + ∣ J ∣ ε n e t + 2 π D ~ N − 1 ∥ f n e t ( z , u N ) ∥ H 1 ( T ; R n y ) ≤ 2 π L u C ~ N − l C τ , u + ∣ J ∣ ε n e t + 2 π D ~ D ^ N − 1 , \begin{aligned}
\sqrt{2\pi}L_u\widetilde{C}N^{-l}C_{\tau,u}+|J|\varepsilon_{\mathrm{net}}+\sqrt{2\pi}\|f_{\mathrm{net}}(z,u_N)-I_Nf_{\mathrm{net}}(z,u_N)\|_{L^2(\mathbb{T};\mathbb{R}^{n_y})} \\
&\leq\sqrt{2\pi}L_u\widetilde{C}N^{-l}C_{\tau,u}+|J|\varepsilon_{\mathrm{net}}+\sqrt{2\pi}\widetilde{D}N^{-1}\|f_{\mathrm{net}}(z,u_N)\|_{H^1(\mathbb{T};\mathbb{R}^{n_y})} \\
&\leq\sqrt{2\pi}L_u\widetilde{C}N^{-l}C_{\tau,u}+|J|\varepsilon_{\mathrm{net}}+\sqrt{2\pi}\widetilde{D}\widehat{D}N^{-1},
\end{aligned} 2 π L u C N − l C τ , u + ∣ J ∣ ε net + 2 π ∥ f net ( z , u N ) − I N f net ( z , u N ) ∥ L 2 ( T ; R n y ) ≤ 2 π L u C N − l C τ , u + ∣ J ∣ ε net + 2 π D N − 1 ∥ f net ( z , u N ) ∥ H 1 ( T ; R n y ) ≤ 2 π L u C N − l C τ , u + ∣ J ∣ ε net + 2 π D D N − 1 ,
where the constant D ~ > 0 \widetilde{D}>0 D > 0 comes from the Lemma B.2 and D ^ = D ^ ( n y , n u , E , α , m , s , l , E , A , Y 0 , γ , M 0 ) > 0 \widehat{D}=\widehat{D}(n_y,n_u,E,\alpha,m,s,l,\mathcal{E},A,Y^0,\gamma,M_0)>0 D = D ( n y , n u , E , α , m , s , l , E , A , Y 0 , γ , M 0 ) > 0 comes from the fact that the composition f n e t ( z , u N ) f_{\mathrm{net}}(z,u_N) f net ( z , u N ) is bounded as stated in Corollary C.13 . For (II), we get for t ∈ J t\in J t ∈ J and any r ≥ 1 r\geq1 r ≥ 1 that
∣ t − τ N ( t ) ∣ ≤ ∥ id − τ N ∥ C ( J ) ≤ ∥ τ − τ N ∥ C ( T ) ≤ C 1 ∥ τ − τ N ∥ H 1 ( T ) ≤ C 1 ∥ τ − P N τ ∥ H 1 ( T ) ≤ C 1 D 1 N − ( r − 1 ) ∥ τ ∥ H r ( T ) , \begin{aligned}
|t-\tau_N(t)|&\leq\|\operatorname{id}-\tau_N\|_{C(J)}\\
&\leq\|\tau-\tau_N\|_{C(\mathbb{T})}\\
&\leq C_1\|\tau-\tau_N\|_{H^1(\mathbb{T})}\\
&\leq C_1\|\tau-P_N\tau\|_{H^1(\mathbb{T})}\\
&\leq C_1D_1N^{-(r-1)}\|\tau\|_{H^r(\mathbb{T})},
\end{aligned} ∣ t − τ N ( t ) ∣ ≤ ∥ id − τ N ∥ C ( J ) ≤ ∥ τ − τ N ∥ C ( T ) ≤ C 1 ∥ τ − τ N ∥ H 1 ( T ) ≤ C 1 ∥ τ − P N τ ∥ H 1 ( T ) ≤ C 1 D 1 N − ( r − 1 ) ∥ τ ∥ H r ( T ) ,
where D 1 = D 1 ( r ) > 0 D_1=D_1(r)>0 D 1 = D 1 ( r ) > 0 is the constant from Lemma B.4 and the last term ∥ τ ∥ H r ( T ) \|\tau\|_{H^r(\mathbb{T})} ∥ τ ∥ H r ( T ) is also bounded as shown in Equation (A.3). Note that the Lipschitz constant L u L_u L u , in the same way as D ^ \widehat{D} D , is dependent on f f f , thus we can drop the explicit dependency on L u L_u L u , when stating the existence of a constant C ′ > 0 C'>0 C ′ > 0 . Since we can choose r r r arbitrary, we also drop that dependency. In total, we can conclude that there exists a constant C ′ = C ′ ( n y , n u , E , α , m , s , l , E , A , Y 0 , γ , M 0 , J ) > 0 C'=C'(n_y,n_u,E,\alpha,m,s,l,\mathcal{E},A,Y^0,\gamma,M_0,J)>0 C ′ = C ′ ( n y , n u , E , α , m , s , l , E , A , Y 0 , γ , M 0 , J ) > 0 such that
sup z ∈ B N ∥ Φ ( z ; u , y 0 ) ∣ J − Φ N , n e t ( z ; u N , y 0 , τ N − t ^ 1 ) ∣ J ∥ C ( J ; R n y ) ≤ C ′ ( N − 1 + ε n e t ) . \sup_{z\in\mathcal{B}_N}\|\Phi(z;u,y^0)|_J-\Phi_{N,\mathrm{net}}(z;u_N,y^0,\tau_N-\hat{t}_1)|_J\|_{C(J;\mathbb{R}^{n_y})}\leq C'(N^{-1}+\varepsilon_{\mathrm{net}}). z ∈ B N sup ∥Φ ( z ; u , y 0 ) ∣ J − Φ N , net ( z ; u N , y 0 , τ N − t ^ 1 ) ∣ J ∥ C ( J ; R n y ) ≤ C ′ ( N − 1 + ε net ) .
Combining the last lemma with Lemma C.9 gives us a convergence result for our proposed algorithm.
Theorem C.20. Let Assumption 2.5 hold and consider y N , n e t ( 0 ) ≡ 0 ∈ B N y_{N,\mathrm{net}}^{(0)}\equiv0\in\mathcal{B}_N y N , net ( 0 ) ≡ 0 ∈ B N . For Φ N , n e t \Phi_{N,\mathrm{net}} Φ N , net defined as in Equation (11) with K = ⌈ log ( Y N ) / log ( 1 / κ ) ⌉ K=\lceil\log(Y^N)/\log(1/\kappa)\rceil K = ⌈ log ( Y N ) / log ( 1/ κ )⌉ there are constants C ∗ > 0 C^*>0 C ∗ > 0 and C ^ > 0 \widehat{C}>0 C > 0 depending on n y , n u , E , α , m , s , l , A , Y 0 , γ , J n_y,n_u,E,\alpha,m,s,l,A,Y^0,\gamma,J n y , n u , E , α , m , s , l , A , Y 0 , γ , J , on the fixed extension operator E \mathcal{E} E and on the GELU width parameter M 0 M_0 M 0 , but not on ε \varepsilon ε , such that for every ε ∈ ( 0 , min { 1 / 2 , 3 C ∗ min { 1 / 2 , E } } ) \varepsilon\in(0,\min\{1/2,3C^*\min\{1/2,E\}\}) ε ∈ ( 0 , min { 1/2 , 3 C ∗ min { 1/2 , E }}) there exists a N = N ( ε ) ≤ C ^ ε − 1 N=N(\varepsilon)\leq\widehat{C}\varepsilon^{-1} N = N ( ε ) ≤ C ε − 1 with
sup y 0 ∈ Y 0 , u ∈ U l , J , A ∥ y ^ ( ⋅ ; u , y 0 ) − Φ N , n e t [ K ] ( y N , n e t ( 0 ) ; u N , y 0 , τ N − t ^ 1 ) ∣ J ∥ C ( J ; R n y ) ≤ ε , \sup_{y^0\in Y^0,u\in\mathcal{U}_l,J,A}\|\hat{y}(\cdot;u,y^0)-\Phi_{N,\mathrm{net}}^{[K]}(y_{N,\mathrm{net}}^{(0)};u_N,y^0,\tau_N-\hat{t}_1)|_J\|_{C(J;\mathbb{R}^{n_y})}\leq\varepsilon, y 0 ∈ Y 0 , u ∈ U l , J , A sup ∥ y ^ ( ⋅ ; u , y 0 ) − Φ N , net [ K ] ( y N , net ( 0 ) ; u N , y 0 , τ N − t ^ 1 ) ∣ J ∥ C ( J ; R n y ) ≤ ε , where y ^ ( ⋅ ; u , y 0 ) \hat{y}(\cdot;u,y^0) y ^ ( ⋅ ; u , y 0 ) solves y ˙ = f ( y , u ) \dot{y}=f(y,u) y ˙ = f ( y , u ) on J J J with y ( t ^ 1 ) = y 0 y(\hat{t}_1)=y^0 y ( t ^ 1 ) = y 0 , u N : = P N ( τ , E u ) ∈ L N 2 ( T ; R 1 + n u ) u_N:=P_N(\tau,\mathcal{E}u)\in L_N^2(\mathbb{T};\mathbb{R}^{1+n_u}) u N := P N ( τ , E u ) ∈ L N 2 ( T ; R 1 + n u ) is the projected periodically extended control with first component τ N = P N τ \tau_N=P_N\tau τ N = P N τ . Moreover, for the obtained networks in this case, it holds
depth ( f R e L U ) ≤ C ^ log ( ε − s / ( s − 1 ) ) , width ( f R e L U ) ≤ C ^ ε − d / ( s − 1 ) log ( ε − s / ( s − 1 ) ) , \operatorname{depth}(f_{\mathrm{ReLU}})\leq\widehat{C}\log(\varepsilon^{-s/(s-1)}),\quad\operatorname{width}(f_{\mathrm{ReLU}})\leq\widehat{C}\varepsilon^{-d/(s-1)}\log(\varepsilon^{-s/(s-1)}), depth ( f ReLU ) ≤ C log ( ε − s / ( s − 1 ) ) , width ( f ReLU ) ≤ C ε − d / ( s − 1 ) log ( ε − s / ( s − 1 ) ) , depth ( f G E L U ) ≤ C ^ ε − d / ( 2 ( s − 1 ) ) log ( ε − d / ( 2 ( s − 1 ) ) ) , \operatorname{depth}(f_{\mathrm{GELU}})\leq\widehat{C}\varepsilon^{-d/(2(s-1))}\log(\varepsilon^{-d/(2(s-1))}), depth ( f GELU ) ≤ C ε − d / ( 2 ( s − 1 )) log ( ε − d / ( 2 ( s − 1 )) ) , where s s s is the regularity of f ∈ W s , ∞ ( D ′ ; R n y ) f\in W^{s,\infty}(D';\mathbb{R}^{n_y}) f ∈ W s , ∞ ( D ′ ; R n y ) and d = 1 + n y + n u d=1+n_y+n_u d = 1 + n y + n u .
Proof. In Algorithm 3.14 we choose
K = ⌈ log ( Y N ) log ( 1 / κ ) ⌉ , K = \left\lceil\frac{\log(Y N)}{\log(1/\kappa)} \right\rceil, K = ⌈ log ( 1/ κ ) log ( Y N ) ⌉ ,
which implicates κ K ≤ 1 / Y N \kappa^K \le1/Y N κ K ≤ 1/ Y N for 0 < κ < 1 0 < \kappa< 1 0 < κ < 1 . Recall that we chose ε n e t \varepsilon_{\mathrm{net}} ε net (see Equations (C.2) and (C.3)) such that ε n e t ≤ N − 1 \varepsilon_{\mathrm{net}} \le N^{-1} ε net ≤ N − 1 . The Lemmata C.9 and C.19 immediately give us the existence of a constant C ∗ = C ∗ ( n y , n u , E , α , m , s , l , E , A , Y 0 , γ , M 0 , J ) > 0 C^* = C^*(n_y,n_u,\mathcal{E},\alpha,m,s,l,\mathcal{E},A,Y^0,\gamma,M_0,J) > 0 C ∗ = C ∗ ( n y , n u , E , α , m , s , l , E , A , Y 0 , γ , M 0 , J ) > 0 by defining C ^ ∗ = e λ ∣ J ∣ max { 1 , 2 C ′ } \widehat{C}^* = e^{\lambda|J|}\max\{1,2C'\} C ∗ = e λ ∣ J ∣ max { 1 , 2 C ′ } , such that
sup y 0 ∈ Y 0 , u ∈ U l , J , A ∥ y ^ ( ⋅ ; u , y 0 ) − Φ N , n e t [ K ] ( y N , n e t ( 0 ) ; u N , y 0 , τ N − t ^ 1 ) ∥ C ( J ; R n y ) ≤ C ∗ ( κ K Y + N − 1 + ε n e t ) . \sup_{\substack{y^0\in Y^0,u\in U^{l,J,A}}}\left\|\widehat{y}(\cdot;u,y^0)-\Phi_{N,\mathrm{net}}^{[K]}\left(y_{N,\mathrm{net}}^{(0)};u_N,y^0,\tau_N-\widehat{t}_1\right)\right\|_{C(J;\mathbb{R}^{n_y})}\le C^*\left(\kappa^K Y+N^{-1}+\varepsilon_{\mathrm{net}}\right). y 0 ∈ Y 0 , u ∈ U l , J , A sup y ( ⋅ ; u , y 0 ) − Φ N , net [ K ] ( y N , net ( 0 ) ; u N , y 0 , τ N − t 1 ) C ( J ; R n y ) ≤ C ∗ ( κ K Y + N − 1 + ε net ) .
Now choosing N = ⌈ 3 C ∗ ε ⌉ N=\left\lceil\frac{3C^*}{\varepsilon}\right\rceil N = ⌈ ε 3 C ∗ ⌉ implicates 3 C ∗ ε ≤ N ≤ 3 C ∗ ε + 1 \frac{3C^*}{\varepsilon}\le N\le\frac{3C^*}{\varepsilon}+1 ε 3 C ∗ ≤ N ≤ ε 3 C ∗ + 1 and
C ∗ ( κ K Y + N − 1 + ε n e t ) ≤ C ∗ ( Y Y N + N − 1 + N − 1 ) ≤ C ∗ 3 N − 1 ≤ ε . C^*\left(\kappa^K Y+N^{-1}+\varepsilon_{\mathrm{net}}\right)\le C^*\left(\frac{Y}{Y N}+N^{-1}+N^{-1}\right)\le C^*3N^{-1}\le\varepsilon. C ∗ ( κ K Y + N − 1 + ε net ) ≤ C ∗ ( Y N Y + N − 1 + N − 1 ) ≤ C ∗ 3 N − 1 ≤ ε .
For the size of the ReLU network with ε R e L U = N − 1 \varepsilon_{\mathrm{ReLU}}=N^{-1} ε ReLU = N − 1 ,
depth ( f R e L U ) ≤ c R e L U ⋅ log ( ε R e L U − s / ( s − 1 ) ) , width ( f R e L U ) ≤ c R e L U ⋅ ε R e L U − d / ( s − 1 ) ⋅ log ( ε R e L U − s / ( s − 1 ) ) , \operatorname{depth}(f_{\mathrm{ReLU}})\le c_{\mathrm{ReLU}}\cdot\log\left(\varepsilon_{\mathrm{ReLU}}^{-s/(s-1)}\right),\quad\operatorname{width}(f_{\mathrm{ReLU}})\le c_{\mathrm{ReLU}}\cdot\varepsilon_{\mathrm{ReLU}}^{-d/(s-1)}\cdot\log\left(\varepsilon_{\mathrm{ReLU}}^{-s/(s-1)}\right), depth ( f ReLU ) ≤ c ReLU ⋅ log ( ε ReLU − s / ( s − 1 ) ) , width ( f ReLU ) ≤ c ReLU ⋅ ε ReLU − d / ( s − 1 ) ⋅ log ( ε ReLU − s / ( s − 1 ) ) ,
it therefore follows for a constant D 1 D_1 D 1 depending on C ∗ C^* C ∗ that
depth ( f R e L U ) ≤ D 1 ⋅ log ( ε − s / ( s − 1 ) ) , width ( f R e L U ) ≤ D 1 ⋅ ε − d / ( s − 1 ) ⋅ log ( ε − s / ( s − 1 ) ) . \operatorname{depth}(f_{\mathrm{ReLU}})\le D_1\cdot\log\left(\varepsilon^{-s/(s-1)}\right),\quad\operatorname{width}(f_{\mathrm{ReLU}})\le D_1\cdot\varepsilon^{-d/(s-1)}\cdot\log\left(\varepsilon^{-s/(s-1)}\right). depth ( f ReLU ) ≤ D 1 ⋅ log ( ε − s / ( s − 1 ) ) , width ( f ReLU ) ≤ D 1 ⋅ ε − d / ( s − 1 ) ⋅ log ( ε − s / ( s − 1 ) ) .
Recall that for the GELU network we set L = ⌈ ( c 0 , G E L U N ) d / ( 2 ( s − 1 ) ) ⌉ L=\left\lceil(c_{0,\mathrm{GELU}}N)^{d/(2(s-1))}\right\rceil L = ⌈ ( c 0 , GELU N ) d / ( 2 ( s − 1 )) ⌉ implying
L ≤ ( c 0 , G E L U N ) d / ( 2 ( s − 1 ) ) + 1 ≤ ( c 0 , G E L U ( 3 C ∗ ε + 1 ) ) d / ( 2 ( s − 1 ) ) + 1 , L\le(c_{0,\mathrm{GELU}}N)^{d/(2(s-1))}+1\le\left(c_{0,\mathrm{GELU}}\left(\frac{3C^*}{\varepsilon}+1\right)\right)^{d/(2(s-1))}+1, L ≤ ( c 0 , GELU N ) d / ( 2 ( s − 1 )) + 1 ≤ ( c 0 , GELU ( ε 3 C ∗ + 1 ) ) d / ( 2 ( s − 1 )) + 1 ,
so that we can find a constant D 2 D_2 D 2 depending on C ∗ C^* C ∗ , and c 1 , G E L U c_{1,\mathrm{GELU}} c 1 , GELU , such that
depth ( f G E L U ) ≤ D 2 ⋅ ε − d / ( 2 ( s − 1 ) ) log ( ε − d / ( 2 ( s − 1 ) ) ) , \operatorname{depth}(f_{\mathrm{GELU}})\le D_2\cdot\varepsilon^{-d/(2(s-1))}\log\left(\varepsilon^{-d/(2(s-1))}\right), depth ( f GELU ) ≤ D 2 ⋅ ε − d / ( 2 ( s − 1 )) log ( ε − d / ( 2 ( s − 1 )) ) ,
since we chose the GELU neural network with constant width M 0 M_0 M 0 . To conclude, we can now choose a constant C ^ = C ^ ( n y , n u , E , α , m , s , l , E , A , Y 0 , γ , M 0 , J ) > 0 \widehat{C}=\widehat{C}(n_y,n_u,\mathcal{E},\alpha,m,s,l,\mathcal{E},A,Y^0,\gamma,M_0,J)>0 C = C ( n y , n u , E , α , m , s , l , E , A , Y 0 , γ , M 0 , J ) > 0 such that C ^ ≥ max { D 1 , D 2 } \widehat{C}\ge\max\{D_1,D_2\} C ≥ max { D 1 , D 2 } and also N ≤ C ^ ε − 1 N\le\widehat{C}\varepsilon^{-1} N ≤ C ε − 1 .
We define the constant ε 0 : = min { 1 / 2 , 3 C ∗ min { 1 / 2 , E } } \varepsilon_0:=\min\{1/2,3C^*\min\{1/2,\mathcal{E}\}\} ε 0 := min { 1/2 , 3 C ∗ min { 1/2 , E }} where ε 0 \varepsilon_0 ε 0 has the same dependencies as C ∗ > 0 C^*>0 C ∗ > 0 .
Remark C.21. To show Theorem 4.1, we use the linear approximation Lemma B.8 which is stated in the L 2 L^2 L 2 -norm, but calculated for trigonometric polynomials. Due to the finite dimensionality of the space of trigonometric polynomials, we can reformulate the statement using the sup-norm but with a factor D ( N ) > 0 D(N)>0 D ( N ) > 0 depending on N N N and of order D ( N ) = O ( N 1 / 2 ) D(N)=O(N^{1/2}) D ( N ) = O ( N 1/2 ) . The lemmas from [27 ] that we use require smooth activation functions: Lemma B.8 assumes σ ∈ C 2 \sigma\in C^2 σ ∈ C 2 and Lemma B.11 assumes σ ∈ C 3 \sigma\in C^3 σ ∈ C 3 . We therefore use the GELU activation function in all layers that do not belong to f n e t f_{\mathrm{net}} f net . In particular, if f f f is approximated by a GELU network, the whole Ψ \Psi Ψ -FNO uses a single activation function.
It remains to show, that we can emulate Algorithm 3.14 using Ψ \Psi Ψ -FNO layers, which ultimately leads to proving Theorem 4.1.
Proof. Consider u N : = P N E u u_N:=P_N\mathcal{E}u u N := P N E u for u ∈ U l , J , A u\in U^{l,J,A} u ∈ U l , J , A , l ≥ 2 l\ge2 l ≥ 2 , y 0 ∈ Y 0 y^0\in Y^0 y 0 ∈ Y 0 and the projected periodically extended time variable τ N = P N τ \tau_N=P_N\tau τ N = P N τ . Then, under Assumption 2.5 we have the existence of a solution y ^ ( ⋅ ; u , y 0 ) \widehat{y}(\cdot;u,y^0) y ( ⋅ ; u , y 0 ) which solves y ˙ = f ( y , u ) \dot{y}=f(y,u) y ˙ = f ( y , u ) on J J J with y ^ ( t ^ 1 ) = y 0 \widehat{y}(\widehat{t}_1)=y^0 y ( t 1 ) = y 0 . For the approximation by our proposed Algorithm 3.14 , we choose ε / 2 > 0 \varepsilon/2>0 ε /2 > 0 and y N , n e t ( 0 ) : = 0 ∈ B N y_{N,\mathrm{net}}^{(0)}:=0\in\mathcal{B}_N y N , net ( 0 ) := 0 ∈ B N to apply Theorem C.20 , which results in the existence of constants C ∗ C^* C ∗ and C ^ \widehat{C} C depending on n y , n u , E , α , m , s , l , E , A , Y 0 , γ , M 0 n_y,n_u,\mathcal{E},\alpha,m,s,l,\mathcal{E},A,Y^0,\gamma,M_0 n y , n u , E , α , m , s , l , E , A , Y 0 , γ , M 0 and J J J while ⌈ 6 C ∗ / ε ⌉ = N = N ( ε / 2 ) ≤ C ^ ( ε / 2 ) − 1 \left\lceil6C^*/\varepsilon\right\rceil=N=N(\varepsilon/2)\le\widehat{C}(\varepsilon/2)^{-1} ⌈ 6 C ∗ / ε ⌉ = N = N ( ε /2 ) ≤ C ( ε /2 ) − 1 with
sup y 0 ∈ Y 0 , u ∈ U l , J , A ∥ y ^ ( ⋅ ; u , y 0 ) − Φ N , n e t [ K ] ( y N , n e t ( 0 ) ; u N , y 0 , τ N − t ^ 1 ) ∥ C ( J ; R n y ) ≤ ε / 2 , \sup_{\substack{y^0\in Y^0,u\in U^{l,J,A}}}\left\|\widehat{y}(\cdot;u,y^0)-\Phi_{N,\mathrm{net}}^{[K]}\left(y_{N,\mathrm{net}}^{(0)};u_N,y^0,\tau_N-\widehat{t}_1\right)\right\|_{C(J;\mathbb{R}^{n_y})}\le\varepsilon/2, y 0 ∈ Y 0 , u ∈ U l , J , A sup y ( ⋅ ; u , y 0 ) − Φ N , net [ K ] ( y N , net ( 0 ) ; u N , y 0 , τ N − t 1 ) C ( J ; R n y ) ≤ ε /2 ,
and
depth ( f R e L U ) ≤ C ^ log ( ( ε / 2 ) − s / ( s − 1 ) ) , width ( f R e L U ) ≤ C ^ ( ε / 2 ) − d / ( s − 1 ) log ( ( ε / 2 ) − s / ( s − 1 ) ) , \operatorname{depth}(f_{\mathrm{ReLU}})\le\widehat{C}\log\left((\varepsilon/2)^{-s/(s-1)}\right),\quad\operatorname{width}(f_{\mathrm{ReLU}})\le\widehat{C}(\varepsilon/2)^{-d/(s-1)}\log\left((\varepsilon/2)^{-s/(s-1)}\right), depth ( f ReLU ) ≤ C log ( ( ε /2 ) − s / ( s − 1 ) ) , width ( f ReLU ) ≤ C ( ε /2 ) − d / ( s − 1 ) log ( ( ε /2 ) − s / ( s − 1 ) ) ,
depth ( f G E L U ) ≤ C ^ ( ε / 2 ) − d / ( 2 ( s − 1 ) ) log ( ( ε / 2 ) − d / ( 2 ( s − 1 ) ) ) . \operatorname{depth}(f_{\mathrm{GELU}})\le\widehat{C}(\varepsilon/2)^{-d/(2(s-1))}\log\left((\varepsilon/2)^{-d/(2(s-1))}\right). depth ( f GELU ) ≤ C ( ε /2 ) − d / ( 2 ( s − 1 )) log ( ( ε /2 ) − d / ( 2 ( s − 1 )) ) .
By triangle inequality, we want to prove the existence of a Ψ \Psi Ψ -FNO N ^ : L N 2 ( T ; R ( 1 + n u ) + n y + 2 ) → L N 2 ( T ; R n y ) \widehat{\mathcal{N}}: L_N^2(\mathbb{T};\mathbb{R}^{(1+n_u)+n_y+2}) \to L_N^2(\mathbb{T};\mathbb{R}^{n_y}) N : L N 2 ( T ; R ( 1 + n u ) + n y + 2 ) → L N 2 ( T ; R n y ) such that
sup y 0 ∈ Y 0 , u ∈ U L , J , A ∥ Φ N , n e t [ K ] ( y N , n e t ( 0 ) ; u N , y 0 , τ N − t ^ 1 ) − N ^ ( u N , y 0 , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) ∥ C ( T ; R n y ) ≤ ε / 2. \sup_{y^0\in Y^0,u\in U_L,J,A}\left\|\Phi_{\mathrm{N,net}}^{[K]}\left(y_{\mathrm{N,net}}^{(0)};u_N,y^0,\tau_N-\hat{t}_1\right)-\widehat{\mathcal{N}}\left(u_N,y^0,\tau_N-\hat{t}_1,D_N(\hat{t}_1-\cdot)\right)\right\|_{C(\mathbb{T};\mathbb{R}^{n_y})}\leq\varepsilon/2. y 0 ∈ Y 0 , u ∈ U L , J , A sup Φ N , net [ K ] ( y N , net ( 0 ) ; u N , y 0 , τ N − t ^ 1 ) − N ( u N , y 0 , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) C ( T ; R n y ) ≤ ε /2.
Note that we show the estimation on T \mathbb{T} T . To prove this estimation, we replace the steps of Algorithm 3.14 with Ψ \Psi Ψ -FNO layers using either exact representation or the linear approximation Lemma B.8 . Due to Remark C.21, we set ε N ^ : = 3 C ∗ / D ( N ) N > 0 \varepsilon_{\widehat{\mathcal{N}}}:=3C^*/D(N)N>0 ε N := 3 C ∗ / D ( N ) N > 0 , when using Lemma B.8 . Define the lifting operator
R : L N 2 ( T ; R ( 1 + n u ) + n y + 2 ) → L N 2 ( T ; R ( 1 + n u ) + 2 n y + 2 ) \mathcal{R}:L_N^2(\mathbb{T};\mathbb{R}^{(1+n_u)+n_y+2})\to L_N^2(\mathbb{T};\mathbb{R}^{(1+n_u)+2n_y+2}) R : L N 2 ( T ; R ( 1 + n u ) + n y + 2 ) → L N 2 ( T ; R ( 1 + n u ) + 2 n y + 2 )
( u N , y 0 , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) ↦ ( u N , y 0 , 0 , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) , (u_N,y^0,\tau_N-\hat{t}_1,D_N(\hat{t}_1-\cdot))\mapsto(u_N,y^0,0,\tau_N-\hat{t}_1,D_N(\hat{t}_1-\cdot)), ( u N , y 0 , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ )) ↦ ( u N , y 0 , 0 , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ )) ,
which corresponds to the input step and step (3) in Algorithm 3.14 . Step (1) is already fulfilled due to the assumption and step (2) will be resolved by following construction. Define the operators
N 1 : L N 2 ( T ; R ( 1 + n u ) + 2 n y + 2 ) → L N 2 ( T ; R ( 1 + n u ) + 2 n y + 2 ) \mathcal{N}_1:L_N^2(\mathbb{T};\mathbb{R}^{(1+n_u)+2n_y+2})\to L_N^2(\mathbb{T};\mathbb{R}^{(1+n_u)+2n_y+2}) N 1 : L N 2 ( T ; R ( 1 + n u ) + 2 n y + 2 ) → L N 2 ( T ; R ( 1 + n u ) + 2 n y + 2 )
( u N , y 0 , y , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) ↦ ( u N , y 0 , I N f n e t ( y , u N ) , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) , (u_N,y^0,y,\tau_N-\hat{t}_1,D_N(\hat{t}_1-\cdot))\mapsto(u_N,y^0,\mathcal{I}_Nf_{\mathrm{net}}(y,u_N),\tau_N-\hat{t}_1,D_N(\hat{t}_1-\cdot)), ( u N , y 0 , y , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ )) ↦ ( u N , y 0 , I N f net ( y , u N ) , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ )) ,
N 2 : L N 2 ( T ; R ( 1 + n u ) + 2 n y + 2 ) → L N 2 ( T ; R ( 1 + n u ) + 3 n y + 2 ) \mathcal{N}_2:L_N^2(\mathbb{T};\mathbb{R}^{(1+n_u)+2n_y+2})\to L_N^2(\mathbb{T};\mathbb{R}^{(1+n_u)+3n_y+2}) N 2 : L N 2 ( T ; R ( 1 + n u ) + 2 n y + 2 ) → L N 2 ( T ; R ( 1 + n u ) + 3 n y + 2 )
( u N , y 0 , y , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) ↦ ( u N , y 0 , ∑ ∣ k ∣ ≤ N δ 0 ( k ) F N ( y ) ( k ) e i k ⋅ ∫ T y ( s ) d s , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) , (u_N,y^0,y,\tau_N-\hat{t}_1,D_N(\hat{t}_1-\cdot))\mapsto\left(u_N,y^0,\sum_{|k|\leq N}\delta_0(k)F_N(y)(k)e^{ik\cdot}\int_{\mathbb{T}}y(s)\,ds,\tau_N-\hat{t}_1,D_N(\hat{t}_1-\cdot)\right), ( u N , y 0 , y , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ )) ↦ u N , y 0 , ∣ k ∣ ≤ N ∑ δ 0 ( k ) F N ( y ) ( k ) e ik ⋅ ∫ T y ( s ) d s , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ,
N 3 : L N 2 ( T ; R ( 1 + n u ) + 3 n y + 2 ) → L N 2 ( T ; R ( 1 + n u ) + 4 n y + 2 ) \mathcal{N}_3:L_N^2(\mathbb{T};\mathbb{R}^{(1+n_u)+3n_y+2})\to L_N^2(\mathbb{T};\mathbb{R}^{(1+n_u)+4n_y+2}) N 3 : L N 2 ( T ; R ( 1 + n u ) + 3 n y + 2 ) → L N 2 ( T ; R ( 1 + n u ) + 4 n y + 2 )
( u N , y 0 , y , c 0 , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) ↦ ( u N , y 0 , y , I N ( y D N ( t ^ 1 − ⋅ ) ) , c 0 ( τ N − t ^ 1 ) , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) , (u_N,y^0,y,c_0,\tau_N-\hat{t}_1,D_N(\hat{t}_1-\cdot))\mapsto(u_N,y^0,y,\mathcal{I}_N(yD_N(\hat{t}_1-\cdot)),c_0(\tau_N-\hat{t}_1),\tau_N-\hat{t}_1,D_N(\hat{t}_1-\cdot)), ( u N , y 0 , y , c 0 , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ )) ↦ ( u N , y 0 , y , I N ( y D N ( t ^ 1 − ⋅ )) , c 0 ( τ N − t ^ 1 ) , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ )) ,
N 4 : L N 2 ( T ; R ( 1 + n u ) + 4 n y + 2 ) → L N 2 ( T ; R ( 1 + n u ) + 4 n y + 2 ) \mathcal{N}_4:L_N^2(\mathbb{T};\mathbb{R}^{(1+n_u)+4n_y+2})\to L_N^2(\mathbb{T};\mathbb{R}^{(1+n_u)+4n_y+2}) N 4 : L N 2 ( T ; R ( 1 + n u ) + 4 n y + 2 ) → L N 2 ( T ; R ( 1 + n u ) + 4 n y + 2 )
( u N , y 0 , y , y 1 , c 0 t , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) ↦ ( u N , y 0 , y , ∫ T y 1 ( s ) d s , c 0 t , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) , (u_N,y^0,y,y_1,c_0t,\tau_N-\hat{t}_1,D_N(\hat{t}_1-\cdot))\mapsto\left(u_N,y^0,y,\int_{\mathbb{T}}y_1(s)\,ds,c_0t,\tau_N-\hat{t}_1,D_N(\hat{t}_1-\cdot)\right), ( u N , y 0 , y , y 1 , c 0 t , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ )) ↦ ( u N , y 0 , y , ∫ T y 1 ( s ) d s , c 0 t , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) ,
and
N 5 : L N 2 ( T ; R ( 1 + n u ) + 4 n y + 2 ) → L N 2 ( T ; R ( 1 + n u ) + 2 n y + 2 ) \mathcal{N}_5:L_N^2(\mathbb{T};\mathbb{R}^{(1+n_u)+4n_y+2})\to L_N^2(\mathbb{T};\mathbb{R}^{(1+n_u)+2n_y+2}) N 5 : L N 2 ( T ; R ( 1 + n u ) + 4 n y + 2 ) → L N 2 ( T ; R ( 1 + n u ) + 2 n y + 2 )
( u N , y 0 , y , y 1 , c 0 t , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) ↦ ( u N , y 0 , y , y 0 + y − y 1 + c 0 t , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) . (u_N,y^0,y,y_1,c_0t,\tau_N-\hat{t}_1,D_N(\hat{t}_1-\cdot))\mapsto(u_N,y^0,y,y^0+y-y_1+c_0t,\tau_N-\hat{t}_1,D_N(\hat{t}_1-\cdot)). ( u N , y 0 , y , y 1 , c 0 t , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ )) ↦ ( u N , y 0 , y , y 0 + y − y 1 + c 0 t , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ )) .
Define the projection operator
Q : L N 2 ( T ; R ( 1 + n u ) + 2 n y + 2 ) → L N 2 ( T ; R n y ) Q:L_N^2(\mathbb{T};\mathbb{R}^{(1+n_u)+2n_y+2})\to L_N^2(\mathbb{T};\mathbb{R}^{n_y}) Q : L N 2 ( T ; R ( 1 + n u ) + 2 n y + 2 ) → L N 2 ( T ; R n y )
( u N , y 0 , y , τ N − t ^ 1 , D N ) ↦ y , (u_N,y^0,y,\tau_N-\hat{t}_1,D_N)\mapsto y, ( u N , y 0 , y , τ N − t ^ 1 , D N ) ↦ y ,
to see that our proposed Algorithm 3.14 can now be written as
N : L N 2 ( T ; R ( 1 + n u ) + n y + 2 ) → L N 2 ( T ; R n y ) \mathcal{N}:L_N^2(\mathbb{T};\mathbb{R}^{(1+n_u)+n_y+2})\to L_N^2(\mathbb{T};\mathbb{R}^{n_y}) N : L N 2 ( T ; R ( 1 + n u ) + n y + 2 ) → L N 2 ( T ; R n y )
v N ↦ [ Q ∘ N ′ ∘ ⋯ ∘ N ′ ] ( u N , y 0 , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) , v_N\mapsto\left[Q\circ\mathcal{N}'\circ\cdots\circ\mathcal{N}'\right]\left(u_N,y^0,\tau_N-\hat{t}_1,D_N(\hat{t}_1-\cdot)\right), v N ↦ [ Q ∘ N ′ ∘ ⋯ ∘ N ′ ] ( u N , y 0 , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) ,
where
N ′ : = N 5 ∘ N 4 ∘ N 3 ∘ N 2 ∘ N 1 , K = ⌈ log ( Y N ) log ( 1 / κ ) ⌉ . \mathcal{N}':=\mathcal{N}_5\circ\mathcal{N}_4\circ\mathcal{N}_3\circ\mathcal{N}_2\circ\mathcal{N}_1,\quad K=\left\lceil\frac{\log(YN)}{\log(1/\kappa)}\right\rceil. N ′ := N 5 ∘ N 4 ∘ N 3 ∘ N 2 ∘ N 1 , K = ⌈ log ( 1/ κ ) log ( Y N ) ⌉ .
For each operator N i \mathcal{N}_i N i , i = 1 , … , 5 i=1,\ldots,5 i = 1 , … , 5 , we argue that we can represent it exactly or can at least find an approximation by a Ψ \Psi Ψ -FNO layer. Note that when approximating one of the operators N i \mathcal{N}_i N i , i = 1 , … , 5 i=1,\ldots,5 i = 1 , … , 5 we introduce an error which propagates through the iteration. This causes no issue, as we can simply apply the Replacement Lemma B.9 . According to Lemma B.10 the operator N 1 \mathcal{N}_1 N 1 can be represented by a Ψ \Psi Ψ -FNO N ^ 1 \widehat{\mathcal{N}}_1 N 1 with
width ( N ^ 1 ) = ∣ J N ∣ width ( f n e t ) , depth ( N ^ 1 ) = depth ( f n e t ) , lift ( N ^ 1 ) = width ( f n e t ) . \operatorname{width}(\widehat{\mathcal{N}}_1)=|\mathcal{J}_N|\operatorname{width}(f_{\mathrm{net}}),\quad\operatorname{depth}(\widehat{\mathcal{N}}_1)=\operatorname{depth}(f_{\mathrm{net}}),\quad\operatorname{lift}(\widehat{\mathcal{N}}_1)=\operatorname{width}(f_{\mathrm{net}}). width ( N 1 ) = ∣ J N ∣ width ( f net ) , depth ( N 1 ) = depth ( f net ) , lift ( N 1 ) = width ( f net ) .
The channels u N u_N u N , y 0 y^0 y 0 , τ N − t ^ 1 \tau_N-\hat{t}_1 τ N − t ^ 1 and D N ( t ^ 1 − ⋅ ) D_N(\hat{t}_1-\cdot) D N ( t ^ 1 − ⋅ ) are passed through the depth ( f n e t ) \operatorname{depth}(f_{\mathrm{net}}) depth ( f net ) layers of N ^ 1 \widehat{\mathcal{N}}_1 N 1 exactly, since σ ( x ) − σ ( − x ) = x \sigma(x)-\sigma(-x)=x σ ( x ) − σ ( − x ) = x holds for the ReLU and the (standard) GELU activation function. This changes width and lift of N ^ 1 \widehat{\mathcal{N}}_1 N 1 only by a constant factor. To see that we can represent N 2 \mathcal{N}_2 N 2 using Ψ \Psi Ψ -FNO structure, recall that Lemma B.13 gives us
∑ ∣ k ∣ ≤ N δ 0 ( k ) F N ( y ) ( k ) e i t j k = F N − 1 ( δ 0 ( k ) F N ( y ) ( k ) ) ( j ) , ∀ j ∈ J N , \sum_{|k|\le N}\delta_0(k)F_N(y)(k)e^{\mathrm{i}t_jk}=F_N^{-1}\left(\delta_0(k)F_N(y)(k)\right)(j),\quad\forall j\in\mathcal{J}_N, ∣ k ∣ ≤ N ∑ δ 0 ( k ) F N ( y ) ( k ) e i t j k = F N − 1 ( δ 0 ( k ) F N ( y ) ( k ) ) ( j ) , ∀ j ∈ J N ,
and, implied by Lemma B.14 , that
∫ 0 2 π y ( s ) d s = F N − 1 ( δ 1 ( k ) F N ( y ) ( k ) ) ( j ) , ∀ j ∈ J N , \int_0^{2\pi}y(s)\,\mathrm{d}s=F_N^{-1}\left(\delta_1(k)F_N(y)(k)\right)(j),\quad\forall j\in\mathcal{J}_N, ∫ 0 2 π y ( s ) d s = F N − 1 ( δ 1 ( k ) F N ( y ) ( k ) ) ( j ) , ∀ j ∈ J N ,
where
δ 0 ( k ) : = { 1 i k , k ≠ 0 , 0 , k = 0 , and δ 1 ( k ) : = { 0 , k ≠ 0 , 1 , k = 0. \delta_0(k):=\begin{cases}\frac{1}{\mathrm{i}k},&k\ne0,\\0,&k=0,\end{cases}
\qquad\text{and}\qquad
\delta_1(k):=\begin{cases}0,&k\ne0,\\1,&k=0.\end{cases} δ 0 ( k ) := { i k 1 , 0 , k = 0 , k = 0 , and δ 1 ( k ) := { 0 , 1 , k = 0 , k = 0.
Both right-hand side already have a similar structure of a linear Ψ \Psi Ψ -FNO layer, or also called F \mathcal{F} F -layer, as defined in Definition B.6 . We now introduce the precise weight matrix, bias and Fourier multiplier matrix as to show that we can write N 2 \mathcal{N}_2 N 2 as a single linear F \mathcal{F} F -layer. Set the weight matrix, bias and Fourier multiplier matrix as
W 2 : = ( 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 ) , b 2 j : = ( 0 0 0 0 0 0 ) , P 2 ( k ) : = ( 0 0 0 0 0 0 0 0 0 0 0 0 δ 0 ( k ) 0 0 0 0 δ 1 ( k ) 0 0 0 0 0 0 0 0 0 0 0 0 ) , W_2:=\begin{pmatrix}1&0&0&0&0\\0&1&0&0&0\\0&0&0&0&0\\0&0&0&0&0\\0&0&0&1&0\\0&0&0&0&1\end{pmatrix},\quad
b_{2j}:=\begin{pmatrix}0\\0\\0\\0\\0\\0\end{pmatrix},\quad
P_2(k):=\begin{pmatrix}0&0&0&0&0\\0&0&0&0&0\\0&0&\delta_0(k)&0&0\\0&0&\delta_1(k)&0&0\\0&0&0&0&0\\0&0&0&0&0\end{pmatrix}, W 2 := 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 , b 2 j := 0 0 0 0 0 0 , P 2 ( k ) := 0 0 0 0 0 0 0 0 0 0 0 0 0 0 δ 0 ( k ) δ 1 ( k ) 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ,
then
N 2 ( v N ) j = W 2 v N j + b 2 j + F N − 1 ( P 2 F N v ) j , ∀ j ∈ J N , \mathcal{N}_2(v_N)_j=W_2v_{Nj}+b_{2j}+F_N^{-1}\left(P_2F_Nv\right)_j,\quad\forall j\in\mathcal{J}_N, N 2 ( v N ) j = W 2 v N j + b 2 j + F N − 1 ( P 2 F N v ) j , ∀ j ∈ J N ,
where v N : = ( u N , y 0 , y , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) ∈ L N 2 ( T ; R ( 1 + n u ) + 2 n y + 2 ) v_N:=(u_N,y^0,y,\tau_N-\hat{t}_1,D_N(\hat{t}_1-\cdot))\in L_N^2(\mathbb{T};\mathbb{R}^{(1+n_u)+2n_y+2}) v N := ( u N , y 0 , y , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ )) ∈ L N 2 ( T ; R ( 1 + n u ) + 2 n y + 2 ) . To apply the linear approximation Lemma B.8 , we choose ε N ^ > 0 \varepsilon_{\widehat{N}}>0 ε N > 0 as mentioned and we choose as activation function the GELU activation function, see Remark C.21. Further to apply the lemma, we need to show that ∥ v N ∥ L 2 ≤ B \lVert v_N\rVert_{L^2}\le B ∥ v N ∥ L 2 ≤ B for some B > 0 B>0 B > 0 . This follows easily since u N u_N u N is bounded in the L 2 L^2 L 2 -norm due to Lemma C.10 . Further we have y 0 ∈ Y 0 y^0\in Y^0 y 0 ∈ Y 0 , y ∈ B N y\in B_N y ∈ B N , τ N − t ^ 1 ∈ C ( T ) \tau_N-\hat{t}_1\in C(\mathbb{T}) τ N − t ^ 1 ∈ C ( T ) and D N ∈ L N 2 ( T ) D_N\in L_N^2(\mathbb{T}) D N ∈ L N 2 ( T ) . Thus, we can find a common L 2 L^2 L 2 -bound B > 0 B>0 B > 0 . By the linear approximation lemma, one obtains a constant C 1 > 0 C_1>0 C 1 > 0 and Ψ \Psi Ψ -FNO N ^ 2 \widehat{\mathcal{N}}_2 N 2 with
width ( N ^ 2 ) ≤ C 1 N , depth ( N ^ 2 ) ≤ C 1 , lift ( N ^ 2 ) ≤ C 1 \operatorname{width}(\widehat{\mathcal{N}}_2)\le C_1N,\quad\operatorname{depth}(\widehat{\mathcal{N}}_2)\le C_1,\quad\operatorname{lift}(\widehat{\mathcal{N}}_2)\le C_1 width ( N 2 ) ≤ C 1 N , depth ( N 2 ) ≤ C 1 , lift ( N 2 ) ≤ C 1
and
sup ∥ v N ∥ L 2 ≤ B ∥ N 2 ( v N ) − N ^ 2 ( v N ) ∥ L 2 ≤ ε N ^ . \sup_{\lVert v_N\rVert_{L^2}\le B}\lVert\mathcal{N}_2(v_N)-\widehat{\mathcal{N}}_2(v_N)\rVert_{L^2}\le\varepsilon_{\widehat{N}}. ∥ v N ∥ L 2 ≤ B sup ∥ N 2 ( v N ) − N 2 ( v N ) ∥ L 2 ≤ ε N .
For N 3 \mathcal{N}_3 N 3 we use that the multiplication can be approximated by finite neural networks due to Lemma B.11 . As an activation function, we again choose the GELU and set ε N ^ / 4 π > 0 \varepsilon_{\widehat{N}}/\sqrt{4\pi}>0 ε N / 4 π > 0 . Notice that we need to apply the lemma two times. Set z ( t ) : = ∑ ∣ k ∣ ≤ N δ 0 ( k ) F N ( y ) ( k ) e i k t z(t):=\sum_{|k|\le N}\delta_0(k)F_N(y)(k)e^{\mathrm{i}kt} z ( t ) := ∑ ∣ k ∣ ≤ N δ 0 ( k ) F N ( y ) ( k ) e i k t , t ∈ T t\in\mathbb{T} t ∈ T , and c 0 : = ∫ T y ( s ) d s ∈ R n y c_0:=\int_{\mathbb{T}}y(s)\,\mathrm{d}s\in\mathbb{R}^{n_y} c 0 := ∫ T y ( s ) d s ∈ R n y . Due to the finite dimensionality of the space of trigonometric polynomials it is
sup t ∈ T ∣ D N ( t ^ 1 − t ) ∣ ≤ B ′ ∥ D N ( t ^ 1 − ⋅ ) ∥ L 2 ≤ B ′ ′ , \sup_{t\in\mathbb{T}}|D_N(\hat{t}_1-t)|\le B'\lVert D_N(\hat{t}_1-\cdot)\rVert_{L^2}\le B'', t ∈ T sup ∣ D N ( t ^ 1 − t ) ∣ ≤ B ′ ∥ D N ( t ^ 1 − ⋅ ) ∥ L 2 ≤ B ′′ ,
where B ′ = B ′ ( N ) > 0 B'=B'(N)>0 B ′ = B ′ ( N ) > 0 and B ′ ′ = B ′ ′ ( N ) > 0 B''=B''(N)>0 B ′′ = B ′′ ( N ) > 0 . Then for t ∈ T t\in\mathbb{T} t ∈ T we have
∥ z ( t ) ∥ R n y ≤ Y ′ ∑ 0 < ∣ k ∣ ≤ N ∣ e i k t ∣ ∣ k ∣ ≤ 2 Y ′ ( 1 + log ( N ) ) ⇒ sup t ∈ T ∥ z ( t ) ∥ R n y ≤ 2 Y ′ ( 1 + log ( N ) ) . \begin{aligned}
\lVert z(t)\rVert_{\mathbb{R}^{n_y}}\le Y'\sum_{0<|k|\le N}\frac{|e^{\mathrm{i}kt}|}{|k|}\le2Y'(1+\log(N)) \\
&\Rightarrow\sup_{t\in\mathbb{T}}\lVert z(t)\rVert_{\mathbb{R}^{n_y}}\le2Y'(1+\log(N)).
\end{aligned} ∥ z ( t ) ∥ R n y ≤ Y ′ 0 < ∣ k ∣ ≤ N ∑ ∣ k ∣ ∣ e i k t ∣ ≤ 2 Y ′ ( 1 + log ( N )) ⇒ t ∈ T sup ∥ z ( t ) ∥ R n y ≤ 2 Y ′ ( 1 + log ( N )) .
It also is
sup t ∈ T ∣ τ N ( t ) − t ^ 1 ∣ ≤ B ′ ′ ′ , sup t ∈ T ∥ c 0 ∥ R n y ≤ 2 π Y ′ , \sup_{t\in\mathbb{T}}|\tau_N(t)-\hat{t}_1|\le B''',\quad\sup_{t\in\mathbb{T}}\lVert c_0\rVert_{\mathbb{R}^{n_y}}\le2\pi Y', t ∈ T sup ∣ τ N ( t ) − t ^ 1 ∣ ≤ B ′′′ , t ∈ T sup ∥ c 0 ∥ R n y ≤ 2 π Y ′ ,
where B ′ ′ ′ = B ′ ′ ′ ( N ) > 0 B^{\prime\prime\prime}=B^{\prime\prime\prime}(N)>0 B ′′′ = B ′′′ ( N ) > 0 . By choosing the maximum out of all bounds we have a bound B ^ = B ^ ( N ) > 0 \widehat{B}=\widehat{B}(N)>0 B = B ( N ) > 0 such that we obtain the existence of constants C 2 = C 2 ( n y ) > 0 C_{2}=C_{2}(n_y)>0 C 2 = C 2 ( n y ) > 0 and C 3 = C 3 ( n y ) > 0 C_{3}=C_{3}(n_y)>0 C 3 = C 3 ( n y ) > 0 and finite GELU neural networks ϕ 1 , ϕ 2 : R × R n y → R n y \phi_{1},\phi_{2}:\mathbb{R}\times\mathbb{R}^{n_y}\to\mathbb{R}^{n_y} ϕ 1 , ϕ 2 : R × R n y → R n y for the multiplication of a scalar and a vector, see Lemma B.11 (2), such that
sup ∣ D N ∣ , ∥ z ∥ R n y ≤ B ^ ∥ ϕ 1 ( D N , z ) − D N z ∥ R n y ≤ ε N 4 π \sup_{\lvert D_N\rvert,\lVert z\rVert_{R^{n_y}}\leq\widehat{B}}\lVert\phi_{1}(D_N,z)-D_Nz\rVert_{R^{n_y}}\leq\frac{\varepsilon_N}{\sqrt{4\pi}} ∣ D N ∣ , ∥ z ∥ R n y ≤ B sup ∥ ϕ 1 ( D N , z ) − D N z ∥ R n y ≤ 4 π ε N
sup ∣ τ N − t ^ 1 ∣ , ∥ c 0 ∥ R n y ≤ B ^ ∥ ϕ 2 ( τ N − t ^ 1 , c 0 ) − c 0 ( τ N − t ^ 1 ) ∥ R n y ≤ ε N 4 π . \sup_{\lvert\tau_N-\hat{t}_1\rvert,\lVert c_0\rVert_{R^{n_y}}\leq\widehat{B}}\lVert\phi_{2}(\tau_N-\hat{t}_1,c_0)-c_0(\tau_N-\hat{t}_1)\rVert_{R^{n_y}}\leq\frac{\varepsilon_N}{\sqrt{4\pi}}. ∣ τ N − t ^ 1 ∣ , ∥ c 0 ∥ R n y ≤ B sup ∥ ϕ 2 ( τ N − t ^ 1 , c 0 ) − c 0 ( τ N − t ^ 1 ) ∥ R n y ≤ 4 π ε N .
where the size is constant with width ( ϕ 1 ) , depth ( ϕ 1 ) ≤ C 2 \operatorname{width}(\phi_1),\operatorname{depth}(\phi_1)\leq C_2 width ( ϕ 1 ) , depth ( ϕ 1 ) ≤ C 2 and width ( ϕ 2 ) , depth ( ϕ 2 ) ≤ C 3 \operatorname{width}(\phi_2),\operatorname{depth}(\phi_2)\leq C_3 width ( ϕ 2 ) , depth ( ϕ 2 ) ≤ C 3 . Then, we again use Lemma B.10 just like for the operator N 1 \mathcal{N}_1 N 1 to obtain the existence of Ψ \Psi Ψ -FNOs N ^ 3 , i \widehat{\mathcal{N}}_{3,i} N 3 , i , i = 1 , 2 i=1,2 i = 1 , 2 , of constant size with
max j ∈ J N ∥ N ^ 3 , 1 ( z , D N ( t ^ 1 − ⋅ ) ) ( t j ) − z ( t j ) D N ( t ^ 1 − t j ) ∥ R n y ≤ ε N 4 π , \max_{j\in\mathcal{J}_N}\left\lVert\widehat{\mathcal{N}}_{3,1}\left(z,D_N(\hat{t}_1-\cdot)\right)(t_j)-z(t_j)D_N(\hat{t}_1-t_j)\right\rVert_{\mathbb{R}^{n_y}}\leq\frac{\varepsilon_N}{\sqrt{4\pi}}, j ∈ J N max N 3 , 1 ( z , D N ( t ^ 1 − ⋅ ) ) ( t j ) − z ( t j ) D N ( t ^ 1 − t j ) R n y ≤ 4 π ε N ,
max j ∈ J N ∥ N ^ 3 , 2 ( c 0 , τ N − t ^ 1 ) ( t j ) − c 0 ( τ N ( t j ) − t ^ 1 ) ∥ R n y ≤ ε N 4 π . \max_{j\in\mathcal{J}_N}\left\lVert\widehat{\mathcal{N}}_{3,2}(c_0,\tau_N-\hat{t}_1)(t_j)-c_0(\tau_N(t_j)-\hat{t}_1)\right\rVert_{\mathbb{R}^{n_y}}\leq\frac{\varepsilon_N}{\sqrt{4\pi}}. j ∈ J N max N 3 , 2 ( c 0 , τ N − t ^ 1 ) ( t j ) − c 0 ( τ N ( t j ) − t ^ 1 ) R n y ≤ 4 π ε N .
To obtain one single Ψ \Psi Ψ -FNO N ^ 3 \widehat{\mathcal{N}}_3 N 3 we just choose again a weight matrix, bias and Fourier multiplier matrix appropriately. Both estimates hold at the grid points. Since N ^ 3 , 1 ( z , D N ( t ^ 1 − ⋅ ) ) \widehat{\mathcal{N}}_{3,1}(z,D_N(\hat{t}_1-\cdot)) N 3 , 1 ( z , D N ( t ^ 1 − ⋅ )) and I N ( z D N ( t ^ 1 − ⋅ ) ) \mathcal{I}_N(zD_N(\hat{t}_1-\cdot)) I N ( z D N ( t ^ 1 − ⋅ )) , as well as N ^ 3 , 2 ( c 0 , τ N − t ^ 1 ) \widehat{\mathcal{N}}_{3,2}(c_0,\tau_N-\hat{t}_1) N 3 , 2 ( c 0 , τ N − t ^ 1 ) and c 0 ( τ N − t ^ 1 ) c_0(\tau_N-\hat{t}_1) c 0 ( τ N − t ^ 1 ) , are trigonometric polynomials of degree at most N N N , the discrete Parseval identity ∥ p ∥ L 2 ( T ) 2 = 2 π 2 N + 1 ∑ j ∈ J N ∥ p ( t j ) ∥ 2 \lVert p\rVert_{L^2(\mathbb{T})}^2=\frac{2\pi}{2N+1}\sum_{j\in\mathcal{J}_N}\lVert p(t_j)\rVert^2 ∥ p ∥ L 2 ( T ) 2 = 2 N + 1 2 π ∑ j ∈ J N ∥ p ( t j ) ∥ 2 yields L 2 L^2 L 2 -estimates with upper bound ε N / 2 \varepsilon_N/\sqrt{2} ε N / 2 for each of the two output channels and hence with upper bound ε N \varepsilon_N ε N for N ^ 3 \widehat{\mathcal{N}}_3 N 3 , as needed for the Replacement lemma. The bounds B B B and B ^ \widehat{B} B used for N 2 \mathcal{N}_2 N 2 , N 3 \mathcal{N}_3 N 3 and N 4 \mathcal{N}_4 N 4 are not restricted to the exact Picard iterates: for every M > 0 M>0 M > 0 and every input v ∈ L N 2 ( T ; R m ) v\in L_N^2(\mathbb{T};\mathbb{R}^m) v ∈ L N 2 ( T ; R m ) with ∥ v ∥ L 2 ≤ M \lVert v\rVert_{L^2}\leq M ∥ v ∥ L 2 ≤ M , the inverse estimate ∥ v ∥ L ∞ ≤ ( 2 N + 1 ) / ( 2 π ) ∥ v ∥ L 2 \lVert v\rVert_{L^\infty}\leq\sqrt{(2N+1)/(2\pi)}\lVert v\rVert_{L^2} ∥ v ∥ L ∞ ≤ ( 2 N + 1 ) / ( 2 π ) ∥ v ∥ L 2 yields point-wise bounds depending only on M M M and N N N . Since the sizes of the networks in Lemma B.8 and Lemma B.11 do not depend on these bounds, every operator N i \mathcal{N}_i N i can be approximated uniformly on L 2 L^2 L 2 -bounded sets by Ψ \Psi Ψ -FNOs of fixed size, as required in Lemma B.9 . In particular, this covers inputs that are perturbed by the approximation errors of preceding layers. The operator N 4 \mathcal{N}_4 N 4 is resolved exactly the same way as operator N 2 \mathcal{N}_2 N 2 . Note that the product of z z z and D N ( t ^ 1 − ⋅ ) D_N(\hat{t}_1-\cdot) D N ( t ^ 1 − ⋅ ) lies in the space L 2 N 2 L^2_{2N} L 2 N 2 while the Ψ \Psi Ψ -FNO calculates ∫ T I N ( z ( s ) D N ( t ^ 1 − s ) ) d s \int_{\mathbb{T}}\mathcal{I}_N(z(s)D_N(\hat{t}_1-s))\,\mathrm{d}s ∫ T I N ( z ( s ) D N ( t ^ 1 − s )) d s . This causes no issue as we have ∫ T I N ( z ( s ) D N ( t ^ 1 − s ) ) d s = ∫ T z ( s ) D N ( t ^ 1 − s ) d s \int_{\mathbb{T}}\mathcal{I}_N(z(s)D_N(\hat{t}_1-s))\,\mathrm{d}s=\int_{\mathbb{T}}z(s)D_N(\hat{t}_1-s)\,\mathrm{d}s ∫ T I N ( z ( s ) D N ( t ^ 1 − s )) d s = ∫ T z ( s ) D N ( t ^ 1 − s ) d s , due to Lemma B.16 . Thus, the result from Lemma B.15 implies
z ^ ( t ^ 1 ) = ∫ T z ( s ) D N ( t ^ 1 − s ) d s , \hat{z}(\hat{t}_1)=\int_{\mathbb{T}}z(s)D_N(\hat{t}_1-s)\,\mathrm{d}s, z ^ ( t ^ 1 ) = ∫ T z ( s ) D N ( t ^ 1 − s ) d s ,
where here it is z ^ ( t ^ 1 ) = ∑ ∣ k ∣ ≤ N δ 0 ( k ) F N ( y ) ( k ) e i k t ^ 1 \hat{z}(\hat{t}_1)=\sum_{\lvert k\rvert\leq N}\delta_0(k)F_N(y)(k)e^{ik\hat{t}_1} z ^ ( t ^ 1 ) = ∑ ∣ k ∣ ≤ N δ 0 ( k ) F N ( y ) ( k ) e ik t ^ 1 . To apply again the linear approximation lemma, we need to show that I N ( z D N ( t ^ 1 − ⋅ ) ) \mathcal{I}_N(zD_N(\hat{t}_1-\cdot)) I N ( z D N ( t ^ 1 − ⋅ )) and c 0 ( τ N − t ^ 1 ) c_0(\tau_N-\hat{t}_1) c 0 ( τ N − t ^ 1 ) are bounded in L 2 L^2 L 2 -norm, which is immediate due to the previous arguments. Also, the size of the obtained neural operator N ^ 4 \widehat{\mathcal{N}}_4 N 4 only differs by a positive constant compared to the size of N ^ 2 \widehat{\mathcal{N}}_2 N 2 . The mapping N ^ 5 \widehat{\mathcal{N}}_5 N 5 can be written as a linear Ψ \Psi Ψ -FNO layer with weight matrix, bias and Fourier multiplier
W 5 : = ( 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 1 − 1 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 ) , b 5 j : = ( 0 0 0 0 0 ) , P 5 ( k ) : = 0. W_5:=\begin{pmatrix}1&0&0&0&0&0&0\\0&1&0&0&0&0&0\\0&1&1&-1&1&0&0\\0&0&0&0&0&1&0\\0&0&0&0&0&0&1\end{pmatrix},\quad b_{5j}:=\begin{pmatrix}0\\0\\0\\0\\0\end{pmatrix},\quad P_5(k):=0. W 5 := 1 0 0 0 0 0 1 1 0 0 0 0 1 0 0 0 0 − 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 , b 5 j := 0 0 0 0 0 , P 5 ( k ) := 0.
Remark that the calculated function y − y 1 y-y_1 y − y 1 in N ^ 5 \widehat{\mathcal{N}}_5 N 5 is ∑ ∣ k ∣ ≤ N δ 0 ( k ) F N ( y ~ ) ( k ) ( e i k t − e i k t ^ 1 ) \sum_{\lvert k\rvert\leq N}\delta_0(k)F_N(\tilde{y})(k)(e^{ikt}-e^{ik\hat{t}_1}) ∑ ∣ k ∣ ≤ N δ 0 ( k ) F N ( y ~ ) ( k ) ( e ik t − e ik t ^ 1 ) , where it is y ~ : = I N f n e t ( 0 , u N ) \tilde{y}:=\mathcal{I}_N f_{\mathrm{net}}(0,u_N) y ~ := I N f net ( 0 , u N ) for the first iteration. Therefore, due to Lemma B.13 we calculate ∫ t ^ 1 t ( P ^ N I N f n e t ( 0 , u N ) ) ( s ) d s \int_{\hat{t}_1}^{t}(\widehat{P}_N\mathcal{I}_Nf_{\mathrm{net}}(0,u_N))(s)\,\mathrm{d}s ∫ t ^ 1 t ( P N I N f net ( 0 , u N )) ( s ) d s as required for the Picard type iteration. Again, to apply Lemma B.8 we argue that the input is bounded which is immediate. The obtained neural operator N ^ 5 \widehat{\mathcal{N}}_5 N 5 is of same size up to a constant as N ^ 2 \widehat{\mathcal{N}}_2 N 2 . Define
N † = N 5 ∘ N 4 ∘ N 3 ∘ N 2 ∘ N 1 ∘ ⋯ ∘ N 5 ∘ N 4 ∘ N 3 ∘ N 2 ∘ N 1 ⏟ K -fold composition \mathcal{N}^{\dagger}=\underbrace{\mathcal{N}_5\circ\mathcal{N}_4\circ\mathcal{N}_3\circ\mathcal{N}_2\circ\mathcal{N}_1\circ\cdots\circ\mathcal{N}_5\circ\mathcal{N}_4\circ\mathcal{N}_3\circ\mathcal{N}_2\circ\mathcal{N}_1}_{K\text{-fold composition}} N † = K -fold composition N 5 ∘ N 4 ∘ N 3 ∘ N 2 ∘ N 1 ∘ ⋯ ∘ N 5 ∘ N 4 ∘ N 3 ∘ N 2 ∘ N 1
to perform K K K -times iteration as defined in Algorithm 3.14 . While not all operators N i \mathcal{N}_i N i , i = 1 , … , 5 i=1,\ldots,5 i = 1 , … , 5 can be represented exactly by a Ψ \Psi Ψ -FNO but rather rely on approximation, we apply the Replacement Lemma B.9 on N † \mathcal{N}^{\dagger} N † to obtain a single Ψ \Psi Ψ -FNO N ^ † \widehat{\mathcal{N}}^{\dagger} N † with
sup ∥ v N ∥ L 2 ≤ B ^ ∥ N † ( v N ) − N ^ † ( v N ) ∥ L 2 ≤ ε N , \sup_{\lVert v_N\rVert_{L^2}\leq\widehat{B}}\lVert\mathcal{N}^{\dagger}(v_N)-\widehat{\mathcal{N}}^{\dagger}(v_N)\rVert_{L^2}\leq\varepsilon_N, ∥ v N ∥ L 2 ≤ B sup ∥ N † ( v N ) − N † ( v N ) ∥ L 2 ≤ ε N ,
where B † B^{\dagger} B † is chosen as the maximum out of all occurring L 2 L^{2} L 2 -input bounds. For the size of N ^ \widehat{\mathcal{N}} N it is depth ( N ^ ) ≤ 5 K ⋅ max i = 1 , … , 5 { depth ( N ^ i ) } \operatorname{depth}(\widehat{\mathcal{N}}) \le5K \cdot\max_{i=1,\ldots,5}\{\operatorname{depth}(\widehat{\mathcal{N}}_{i})\} depth ( N ) ≤ 5 K ⋅ max i = 1 , … , 5 { depth ( N i )} , width ( N ^ ) ≤ max i = 1 , … , 5 { width ( N ^ i ) } \operatorname{width}(\widehat{\mathcal{N}}) \le\max_{i=1,\ldots,5}\{\operatorname{width}(\widehat{\mathcal{N}}_{i})\} width ( N ) ≤ max i = 1 , … , 5 { width ( N i )} and lift ( N ^ ) ≤ max i = 1 , … , 5 { lift ( N ^ i ) } \operatorname{lift}(\widehat{\mathcal{N}}) \le\max_{i=1,\ldots,5}\{\operatorname{lift}(\widehat{\mathcal{N}}_{i})\} lift ( N ) ≤ max i = 1 , … , 5 { lift ( N i )} . We define
N ^ ( v N ) : = ( Q ∘ N ^ † ∘ R ) ( v N ) \widehat{\mathcal{N}}(v_N) := (Q \circ\widehat{\mathcal{N}}^{\dagger} \circ R)(v_N) N ( v N ) := ( Q ∘ N † ∘ R ) ( v N )
and since
Φ N , n e t [ K ] ( y N , n e t ( 0 ) ; u N , y 0 , τ N − t ^ 1 ) = N ( v N ) = ( Q ∘ N † ∘ R ) ( v N ) , \Phi^{[K]}_{N,\mathrm{net}}(y^{(0)}_{N,\mathrm{net}};u_N,y^0,\tau_N-\hat{t}_1) = \mathcal{N}(v_N) = (Q \circ\mathcal{N}^{\dagger} \circ R)(v_N), Φ N , net [ K ] ( y N , net ( 0 ) ; u N , y 0 , τ N − t ^ 1 ) = N ( v N ) = ( Q ∘ N † ∘ R ) ( v N ) ,
we conclude
sup ∥ v N ∥ L 2 ≤ B † ∥ Φ N , n e t [ K ] ( y N , n e t ( 0 ) ; u N , y 0 , τ N − t ^ 1 ) − N ^ ( v N ) ∥ L 2 ( T ; R n y ) ≤ ε N ^ . \sup_{\lVert v_N\rVert_{L^2}\le B^{\dagger}}\left\lVert\Phi^{[K]}_{N,\mathrm{net}}(y^{(0)}_{N,\mathrm{net}};u_N,y^0,\tau_N-\hat{t}_1)-\widehat{\mathcal{N}}(v_N)\right\rVert_{L^2(\mathbb{T};\mathbb{R}^{n_y})}\le\varepsilon_{\widehat{\mathcal{N}}}. ∥ v N ∥ L 2 ≤ B † sup Φ N , net [ K ] ( y N , net ( 0 ) ; u N , y 0 , τ N − t ^ 1 ) − N ( v N ) L 2 ( T ; R n y ) ≤ ε N .
where we recall that v N : = ( u N , y 0 , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ ) ) v_N := (u_N,y^0,\tau_N-\hat{t}_1,D_N(\hat{t}_1-\cdot)) v N := ( u N , y 0 , τ N − t ^ 1 , D N ( t ^ 1 − ⋅ )) . Due to the finite dimensionality of L N 2 L^2_N L N 2 , see also Remark C.21, we obtain the existence of a constant D = D ( N ) > 0 D=D(N)>0 D = D ( N ) > 0 such that
sup ∥ v N ∥ L 2 ≤ B † ∥ Φ N , n e t [ K ] ( y N , n e t ( 0 ) ; u N , y 0 , τ N − t ^ 1 ) − N ^ ( v N ) ∥ C ( T ; R n y ) ≤ D ( N ) ε N ^ = D ( N ) 3 C ∗ D ( N ) N = 3 C ∗ N − 1 ≤ ε / 2 , \sup_{\lVert v_N\rVert_{L^2}\le B^{\dagger}}\left\lVert\Phi^{[K]}_{N,\mathrm{net}}(y^{(0)}_{N,\mathrm{net}};u_N,y^0,\tau_N-\hat{t}_1)-\widehat{\mathcal{N}}(v_N)\right\rVert_{C(\mathbb{T};\mathbb{R}^{n_y})}\le D(N)\varepsilon_{\widehat{\mathcal{N}}}=D(N)\frac{3C^*}{D(N)N}=3C^*N^{-1}\le\varepsilon/2, ∥ v N ∥ L 2 ≤ B † sup Φ N , net [ K ] ( y N , net ( 0 ) ; u N , y 0 , τ N − t ^ 1 ) − N ( v N ) C ( T ; R n y ) ≤ D ( N ) ε N = D ( N ) D ( N ) N 3 C ∗ = 3 C ∗ N − 1 ≤ ε /2 ,
where N = ⌈ 6 C ∗ ε ⌉ N=\left\lceil\frac{6C^*}{\varepsilon}\right\rceil N = ⌈ ε 6 C ∗ ⌉ and ε N ^ = 3 C ∗ / D ( N ) N \varepsilon_{\widehat{\mathcal{N}}}=3C^*/D(N)N ε N = 3 C ∗ / D ( N ) N . Since our input always fulfills ∥ v N ∥ L 2 ≤ B † \lVert v_N\rVert_{L^2}\le B^{\dagger} ∥ v N ∥ L 2 ≤ B † , we have shown the estimate formulated in the statement. For the size of N ^ \widehat{\mathcal{N}} N we do following estimations first
depth ( N ^ ) ≤ 5 K ⋅ max i = 1 , … , 5 { depth ( N ^ i ) } ≲ K ⋅ depth ( f n e t ) , \operatorname{depth}(\widehat{\mathcal{N}})\le5K\cdot\max_{i=1,\ldots,5}\{\operatorname{depth}(\widehat{\mathcal{N}}_i)\}\lesssim K\cdot\operatorname{depth}(f_{\mathrm{net}}), depth ( N ) ≤ 5 K ⋅ i = 1 , … , 5 max { depth ( N i )} ≲ K ⋅ depth ( f net ) ,
width ( N ^ ) ≤ max i = 1 , … , 5 { width ( N ^ i ) } , \operatorname{width}(\widehat{\mathcal{N}})\le\max_{i=1,\ldots,5}\{\operatorname{width}(\widehat{\mathcal{N}}_i)\}, width ( N ) ≤ i = 1 , … , 5 max { width ( N i )} ,
lift ( N ^ ) ≤ max i = 1 , … , 5 { lift ( N ^ i ) } ≲ width ( f n e t ) . \operatorname{lift}(\widehat{\mathcal{N}})\le\max_{i=1,\ldots,5}\{\operatorname{lift}(\widehat{\mathcal{N}}_i)\}\lesssim\operatorname{width}(f_{\mathrm{net}}). lift ( N ) ≤ i = 1 , … , 5 max { lift ( N i )} ≲ width ( f net ) .
In case of f n e t = f R e L U f_{\mathrm{net}}=f_{\mathrm{ReLU}} f net = f ReLU , it is then
depth ( N ^ R e L U ) ≲ log ( 1 ε ) log ( ε − s / ( s − 1 ) ) , \operatorname{depth}(\widehat{\mathcal{N}}_{\mathrm{ReLU}})\lesssim\log\left(\frac{1}{\varepsilon}\right)\log\left(\varepsilon^{-s/(s-1)}\right), depth ( N ReLU ) ≲ log ( ε 1 ) log ( ε − s / ( s − 1 ) ) ,
lift ( N ^ R e L U ) ≤ max i = 1 , … , 5 { lift ( N ^ i ) } ≲ width ( f R e L U ) ≲ ε − d / ( s − 1 ) ⋅ log ( ε − s / ( s − 1 ) ) , \operatorname{lift}(\widehat{\mathcal{N}}_{\mathrm{ReLU}})\le\max_{i=1,\ldots,5}\{\operatorname{lift}(\widehat{\mathcal{N}}_i)\}\lesssim\operatorname{width}(f_{\mathrm{ReLU}})\lesssim\varepsilon^{-d/(s-1)}\cdot\log\left(\varepsilon^{-s/(s-1)}\right), lift ( N ReLU ) ≤ i = 1 , … , 5 max { lift ( N i )} ≲ width ( f ReLU ) ≲ ε − d / ( s − 1 ) ⋅ log ( ε − s / ( s − 1 ) ) ,
width ( N ^ R e L U ) ≲ max { width ( N ^ 1 ) , width ( N ^ 2 ) } ≤ max { ∣ J N ∣ width ( f R e L U ) , C 1 N } ≲ ε − 1 − d / ( s − 1 ) ⋅ log ( ε − s / ( s − 1 ) ) , \operatorname{width}(\widehat{\mathcal{N}}_{\mathrm{ReLU}})\lesssim\max\{\operatorname{width}(\widehat{\mathcal{N}}_1),\operatorname{width}(\widehat{\mathcal{N}}_2)\}
\le\max\{|J_N|\operatorname{width}(f_{\mathrm{ReLU}}),C_1N\}\lesssim\varepsilon^{-1-d/(s-1)}\cdot\log\left(\varepsilon^{-s/(s-1)}\right), width ( N ReLU ) ≲ max { width ( N 1 ) , width ( N 2 )} ≤ max { ∣ J N ∣ width ( f ReLU ) , C 1 N } ≲ ε − 1 − d / ( s − 1 ) ⋅ log ( ε − s / ( s − 1 ) ) ,
and in case of f n e t = f G E L U f_{\mathrm{net}}=f_{\mathrm{GELU}} f net = f GELU , we have
depth ( N ^ G E L U ) ≲ log ( 1 ε ) ε − d / ( 2 ( s − 1 ) ) log ( ε − d / ( 2 ( s − 1 ) ) ) , \operatorname{depth}(\widehat{\mathcal{N}}_{\mathrm{GELU}})\lesssim\log\left(\frac{1}{\varepsilon}\right)\varepsilon^{-d/(2(s-1))}\log\left(\varepsilon^{-d/(2(s-1))}\right), depth ( N GELU ) ≲ log ( ε 1 ) ε − d / ( 2 ( s − 1 )) log ( ε − d / ( 2 ( s − 1 )) ) ,
width ( N ^ G E L U ) ≲ N ≲ ( 1 ε ) , \operatorname{width}(\widehat{\mathcal{N}}_{\mathrm{GELU}})\lesssim N\lesssim\left(\frac{1}{\varepsilon}\right), width ( N GELU ) ≲ N ≲ ( ε 1 ) ,
lift ( N ^ G E L U ) ≤ max i = 1 , … , 5 { lift ( N ^ i ) } ≲ width ( f G E L U ) , \operatorname{lift}(\widehat{\mathcal{N}}_{\mathrm{GELU}})\le\max_{i=1,\ldots,5}\{\operatorname{lift}(\widehat{\mathcal{N}}_i)\}\lesssim\operatorname{width}(f_{\mathrm{GELU}}), lift ( N GELU ) ≤ i = 1 , … , 5 max { lift ( N i )} ≲ width ( f GELU ) ,
where width ( f G E L U ) = c 2 , G E L U M 0 log M 0 \operatorname{width}(f_{\mathrm{GELU}})=c_{2,\mathrm{GELU}}M_0\log M_0 width ( f GELU ) = c 2 , GELU M 0 log M 0 is constant, since the width parameter M 0 ∈ N M_0\in\mathbb{N} M 0 ∈ N was fixed beforehand. Recalling Equation (8) and choosing a constant C > 0 C>0 C > 0 in a similar fashion as in the proof of Theorem C.20 we obtain
size ( N ^ R e L U ) ≤ C ⋅ ( 1 ε ) 2 d s − 1 + 1 ( log ( 1 ε ) ) 4 , size ( N ^ G E L U ) ≤ C ⋅ ( 1 ε ) d 2 ( s − 1 ) + 1 ( log ( 1 ε ) ) 2 , \operatorname{size}(\widehat{\mathcal{N}}_{\mathrm{ReLU}})\le C\cdot\left(\frac{1}{\varepsilon}\right)^{\frac{2d}{s-1}+1}\left(\log\left(\frac{1}{\varepsilon}\right)\right)^4,\qquad\operatorname{size}(\widehat{\mathcal{N}}_{\mathrm{GELU}})\le C\cdot\left(\frac{1}{\varepsilon}\right)^{\frac{d}{2(s-1)}+1}\left(\log\left(\frac{1}{\varepsilon}\right)\right)^2, size ( N ReLU ) ≤ C ⋅ ( ε 1 ) s − 1 2 d + 1 ( log ( ε 1 ) ) 4 , size ( N GELU ) ≤ C ⋅ ( ε 1 ) 2 ( s − 1 ) d + 1 ( log ( ε 1 ) ) 2 ,
and N ≤ C ε − 1 N\le C\varepsilon^{-1} N ≤ C ε − 1 . Since C C C depends on C ∗ C^* C ∗ and C ^ \widehat{C} C , it has the same dependencies and thus we have proven our main result.