Introduction
Many computational tasks involving parametric partial differential equations (PDEs) require repeated evaluation of the same parameter-to-solution or parameter-to-observable map. Examples include uncertainty quantification, PDE-constrained optimisation and control, experimental design and parameter identification. When of a large-scale PDE, the cost can become prohibitive, particularly when sensitivities of the forward map are needed as well as its values.
Neural operators offer one way of amortising this cost. A neural operator is trained offline to approximate a map between function spaces and is then evaluated at comparatively low online cost [23 ] (Lu et al. 2021; Li et al. 2021). Learned operators have been embedded in PDE-constrained optimisation and control [17 ] , Bayesian experimental design (Go and Chen, 2025) and Bayesian parameter identification, where posterior exploration requires many forward and derivative evaluations [5 ] . Derivative-informed surrogates developed for such many-query settings learn the forward map together with its derivative [30 , 31 ] . Neural-operator surrogates have also been used in ensemble-based inversion of moving-boundary problems [18 ] .
Neural operators can also be trained to map observations directly to the unknown parameters. Neural Inverse Operators, introduced by Molinaro et al. [28 ] , learn operator-to-function maps for PDE inverse problems with operator-valued measurements. For the Calderón problem, de Hoop et al. [8 ] construct a noise-aware neural-operator approximation using an extension of the inverse conductivity map that preserves its stability properties. In inverse imaging, Auras et al. [3 ] interpret U-Nets, which are widely used to post-process a crude reconstruction such as the backprojected data, as neural operators, and study how such networks behave when the discretisation is refined, with limited-angle computed tomography as the application. These approaches rely on structure, or on a regularised input, that keeps the learned inverse map stable on the class of interest. Without such structure, the inverse map of an ill-posed problem is unstable, and this has a direct consequence for neural approximation (Proposition 2.2 in § 2 ). No continuous network can approximate a discontinuous inverse map uniformly near a point of discontinuity to within half the size of the jump, and networks that are accurate on training data resolving the discontinuity must have Lipschitz constants that grow without bound. For Fourier neural operators of fixed depth, some weights must then diverge. In this work we therefore learn the forward map, which in PDE parameter identification is typically continuously differentiable, and control the instability of the inversion by regularisation.
On the other hand, the use of learned forward maps within classical regularisation of ill-posed inverse problems has received less attention. In this setting the learned operator is part of the regularisation method itself, and replacing the forward map by an approximation may alter stability, convergence and the criterion by which the iteration should be stopped. Scherzer et al. [36 ] study Tikhonov regularisation with an approximate neural forward operator and show how observational and operator errors must be balanced with the regularisation parameter. Landweber-type iterations with perturbed or noisy forward operators have also been analysed (e.g. Gu et al., 2025), and learned components have been combined with Landweber iteration [2 ] . To our knowledge, however, the use of neural-operator surrogates within iterative regularisation, where the derivatives that determine the search directions are also approximate, remains largely unexplored.
Iterative regularisation methods evaluate the forward map and, for derivative-based schemes, its linearisation or adjoint at every step. When these operations are expensive, replacing them by a differentiable learned operator could reduce the online cost substantially once the offline training cost has been amortised. This is most relevant when many related inverse problems must be solved, or when reconstructions are needed quickly, for example in real-time monitoring. We study this question for nonlinear Landweber iteration. Landweber is among the simplest derivative-based iterative regularisation methods and is typically slower than Newton, Gauss–Newton or Levenberg–Marquardt-type schemes. Inexpensive surrogate evaluations change the balance between iteration count and cost per iteration, which makes this slower method more attractive. Landweber iteration also provides a transparent setting in which the effect of approximating both the forward map and its derivative can be analysed. The analysis is formulated for nonlinear maps between Hilbert spaces, and PDE parameter identification provides the computational motivation and the numerical examples.
Let M \mathcal{M} M and Y \mathcal{Y} Y be real separable Hilbert spaces and let F : dom ( F ) ⊆ M ⟶ Y \mathcal{F} : \operatorname{dom}(\mathcal{F}) \subseteq\mathcal{M} \longrightarrow\mathcal{Y} F : dom ( F ) ⊆ M ⟶ Y denote the forward operator. Given exact data y † = F ( m † ) y^\dagger= \mathcal{F}(m^\dagger) y † = F ( m † ) and noisy observations y δ y^\delta y δ satisfying
∥ y δ − y † ∥ Y ≤ δ , (1) \lVert y^\delta- y^\dagger\rVert_{\mathcal{Y}} \le\delta, \tag*{(1)} ∥ y δ − y † ∥ Y ≤ δ , (1)
the nonlinear Landweber iteration takes the form
m k + 1 = m k − ω D F ( m k ) ∗ ( F ( m k ) − y δ ) , (2) m_{k+1} = m_k - \omega D\mathcal{F}(m_k)^*(\mathcal{F}(m_k)-y^\delta), \tag*{(2)} m k + 1 = m k − ω D F ( m k ) ∗ ( F ( m k ) − y δ ) , (2)
together with a stopping rule. Under standard local assumptions, including a tangential cone condition, Landweber iteration stopped by the discrepancy principle is a regularisation method for nonlinear ill-posed problems [15 , 11 , 19 ] . As the iteration formula (2) shows, the Landweber iteration is an explicit regularisation method that involves only the forward mapping and its linearisation, both of which can be well approximated by suitable learned neural operators. Replacing F \mathcal{F} F by a differentiable learned operator F θ \mathcal{F}_\theta F θ , and differentiating the same surrogate, gives
m k + 1 = m k − ω D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) . (3) m_{k+1} = m_k - \omega D\mathcal{F}_\theta(m_k)^*(\mathcal{F}_\theta(m_k)-y^\delta). \tag*{(3)} m k + 1 = m k − ω D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) . (3)
The central question of this paper is whether this substitution preserves the regularising properties of the original method.
Two approximation errors enter the iteration. Error in F θ \mathcal{F}_\theta F θ perturbs the residual against which the data are fitted, and error in D F θ D\mathcal{F}_\theta D F θ perturbs the direction in which the parameter is updated. Accurate forward prediction alone is therefore not sufficient, because a surrogate with inaccurate derivatives may produce poor Landweber directions. The usual discrepancy principle also cannot be applied unchanged, since a small residual for F θ \mathcal{F}_\theta F θ need not imply that the physical forward equation is satisfied to the same accuracy.
Our contributions are as follows.
We show that near a discontinuity, no continuous neural network can uniformly approximate a discontinuous inverse map, which renders neural operator surrogates for the inverse map infeasible (§ 2 ). Meanwhile, we identify encoder–network–decoder neural operators that approximate a nonlinear forward map and its derivative simultaneously, first along compact sets of directions and then, for compact derivatives, in operator norm (§ 3 ). Generalised DeepONets and fully coupled Fourier operators realise this class exactly (§ 4 ).
For a surrogate with separate forward and derivative error bounds, we introduce an effective discrepancy level that accounts for observational, forward and derivative errors,
and prove descent before stopping and finite termination of the resulting discrepancy principle (§ 5 ).
m k + 1 = m k − ω C D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) m_{k+1} = m_k - \omega C D F_\theta(m_k)^* \left(F_\theta(m_k) - y^\delta\right) m k + 1 = m k − ω C D F θ ( m k ) ∗ ( F θ ( m k ) − y δ )
with a compact operator C C C (Theorem 5.15 ). Closed balls of the Hilbert space M C \mathcal{M}_C M C induced by C C C are compact in M \mathcal{M} M , descent in the M C \mathcal{M}_C M C norm keeps the iterates in such a ball, and the approximation theory supplies surrogates that are accurate on it. The price is the regularity condition m † − m 0 ∈ M C m^\dagger- m_0 \in\mathcal{M}_C m † − m 0 ∈ M C . Without a compact preconditioner, the descent argument still applies wherever the error bounds hold, but the iterates are no longer confined to a compact set and convergence as the errors vanish is not established.
We relate these requirements to practical derivative-informed training (§ 6 ). Supervision along a finite set of directions controls a finite-rank part of the derivative error, whereas the preconditioned iteration depends on the weighted error ( D F θ − D F ) C 1 / 2 (D F_\theta- D F)C^{1/2} ( D F θ − D F ) C 1/2 . Smoothing can attenuate poorly learned components of the search direction, but it also suppresses genuine fine-scale features and restricts the truths covered by the theory.
We study these effects numerically for the identification of a reaction coefficient in an elliptic PDE (§ 7 ), comparing standard and generalised architectures, training with and without the derivative loss, and different parameter-space geometries. The networks are trained with derivatives along a fixed set of empirical Karhunen–Loève (KL) directions, which is one practical choice among several covariance-, sensitivity- or random-projection-based options ([31 , 34 , 39 , 27 ] ). Transient Darcy-flow inversion with sparse space–time data (§ 7.12 ) illustrates the approach in a time-dependent setting with a standard architecture.
Conclusions and directions for further work are given in § 8 . The appendices contain the approximation proofs, the standard Fourier Neural Operator (FNO) calculus and the proofs for the preconditioned geometry.
Learning a discontinuous inverse map
This section formalizes the statement made in the introduction that directly learning a discontinuous inverse map is unstable, thereby establishing the necessity of our proposed approach, which uses a neural-network-surrogated Landweber iteration to solve inverse problems stably.
Let R : dom ( R ) ⊆ Y → M \mathcal{R} : \operatorname{dom}(\mathcal{R}) \subseteq\mathcal{Y} \to\mathcal{M} R : dom ( R ) ⊆ Y → M be an inversion or reconstruction map, for example, R = F − 1 \mathcal{R} = \mathcal{F}^{-1} R = F − 1 on the range of an injective forward operator. The following is a formal definition of such a discontinuous operator.
Definition 2.1 (Discontinuous map). The map R \mathcal{R} R is discontinuous at y 0 ∈ dom ( R ) y_0 \in\operatorname{dom}(\mathcal{R}) y 0 ∈ dom ( R ) if there exist a constant C > 0 C > 0 C > 0 and a sequence { y n } ⊂ dom ( R ) \{y_n\} \subset\operatorname{dom}(\mathcal{R}) { y n } ⊂ dom ( R ) such that y n → y 0 y_n \to y_0 y n → y 0 in Y \mathcal{Y} Y , y n ≠ y 0 y_n \ne y_0 y n = y 0 for all n n n , and
∥ R ( y n ) − R ( y 0 ) ∥ M ≥ C for all n . \lVert\mathcal{R}(y_n)-\mathcal{R}(y_0)\rVert_{\mathcal{M}} \ge C \quad\text{for all } n. ∥ R ( y n ) − R ( y 0 ) ∥ M ≥ C for all n .
Consequently, the following proposition shows that no continuous map can accurately approximate a discontinuous map, including Lipschitz continuous maps and generic neural operators. For simplicity, we prove the result only for FNO, but the argument holds for other architectures such as DeepONet.
Proposition 2.2 (Weight Blow-Up in Approximation of Discontinuous Maps). Let R \mathcal{R} R be discontinuous at y 0 y_0 y 0 , with C C C and { y n } \{y_n\} { y n } as in Definition 2.1 . Then the following hold.
(i) Let G : dom ( R ) → M G : \operatorname{dom}(\mathcal{R}) \to\mathcal{M} G : dom ( R ) → M be continuous at y 0 y_0 y 0 . If ∥ G ( y ) − R ( y ) ∥ M ≤ ϵ \lVert G(y)-\mathcal{R}(y)\rVert_{\mathcal{M}} \le\epsilon ∥ G ( y ) − R ( y ) ∥ M ≤ ϵ for y = y 0 y=y_0 y = y 0 and for all y n y_n y n , then there holds ϵ ≥ C / 2 > 0 \epsilon\ge C/2 > 0 ϵ ≥ C /2 > 0 .
(ii) Let 0 ≤ ϵ 1 < C / 2 0 \le\epsilon_1 < C/2 0 ≤ ϵ 1 < C /2 , and for each n n n let G n : dom ( R ) → M G_n : \operatorname{dom}(\mathcal{R}) \to\mathcal{M} G n : dom ( R ) → M be Lipschitz continuous with ∥ G n ( y ) − R ( y ) ∥ M ≤ ϵ 1 \lVert G_n(y)-\mathcal{R}(y)\rVert_{\mathcal{M}} \le\epsilon_1 ∥ G n ( y ) − R ( y ) ∥ M ≤ ϵ 1 for y ∈ { y 0 , y n } y \in\{y_0,y_n\} y ∈ { y 0 , y n } . Then
Lip ( G n ) ≥ C − 2 ϵ 1 ∥ y n − y 0 ∥ Y ⟶ ∞ . \operatorname{Lip}(G_n) \ge\frac{C-2\epsilon_1}{\lVert y_n-y_0\rVert_{\mathcal{Y}}} \longrightarrow\infty. Lip ( G n ) ≥ ∥ y n − y 0 ∥ Y C − 2 ϵ 1 ⟶ ∞. (iii) Suppose, in (ii), that each G n G_n G n is a standard FNO with fixed depth and a globally Lipschitz activation, and that its lifting and projection networks have fixed depth. Let Y \mathcal{Y} Y and M \mathcal{M} M carry the L 2 L^2 L 2 or grid norms in which (51) in Appendix § 12 holds. Then at least one of the weight-matrix or Fourier-multiplier norms in (51) is unbounded as n → ∞ n \to\infty n → ∞ .
Proof. The first two arguments are well known in inverse problems, and we briefly outline the proof below. For any map G G G and any n n n , the following inequality holds:
C ≤ ∥ R ( y n ) − R ( y 0 ) ∥ M ≤ ∥ R ( y n ) − G ( y n ) ∥ M + ∥ G ( y n ) − G ( y 0 ) ∥ M + ∥ G ( y 0 ) − R ( y 0 ) ∥ M . C \le\lVert\mathcal{R}(y_n)-\mathcal{R}(y_0)\rVert_{\mathcal{M}} \le\lVert\mathcal{R}(y_n)-G(y_n)\rVert_{\mathcal{M}}+\lVert G(y_n)-G(y_0)\rVert_{\mathcal{M}}+\lVert G(y_0)-\mathcal{R}(y_0)\rVert_{\mathcal{M}}. C ≤ ∥ R ( y n ) − R ( y 0 ) ∥ M ≤ ∥ R ( y n ) − G ( y n ) ∥ M + ∥ G ( y n ) − G ( y 0 ) ∥ M + ∥ G ( y 0 ) − R ( y 0 ) ∥ M .
Then we can prove: (i) the right-hand side is at most 2 ϵ + ∥ G ( y n ) − G ( y 0 ) ∥ M 2\epsilon+\lVert G(y_n)-G(y_0)\rVert_{\mathcal{M}} 2 ϵ + ∥ G ( y n ) − G ( y 0 ) ∥ M , which tends to 2 ϵ 2\epsilon 2 ϵ as n → ∞ n \to\infty n → ∞ by continuity of G G G at y 0 y_0 y 0 . Hence ϵ ≥ C / 2 > 0 \epsilon\ge C/2 > 0 ϵ ≥ C /2 > 0 .
(ii) Choosing G = G n G=G_n G = G n , we see that the right-hand side of the above inequality is at most 2 ϵ 1 + Lip ( G n ) ∥ y n − y 0 ∥ Y 2\epsilon_1+\operatorname{Lip}(G_n)\lVert y_n-y_0\rVert_{\mathcal{Y}} 2 ϵ 1 + Lip ( G n ) ∥ y n − y 0 ∥ Y . Rearranging yields a lower bound on Lip ( G n ) \operatorname{Lip}(G_n) Lip ( G n ) whose right-hand side diverges as y n → y 0 y_n \to y_0 y n → y 0 , since C − 2 ϵ 1 > 0 C-2\epsilon_1 > 0 C − 2 ϵ 1 > 0 .
(iii) By deriving the Lipschitz constant for FNO, as shown in the right-hand side of (51) in Appendix § 12 , it is a product of the listed norms and fixed constants. If all these norms were bounded in n n n , so would be Lip ( G n ) \operatorname{Lip}(G_n) Lip ( G n ) , which contradicts the result of (ii).
As we argued in the introduction and above, a neural operator cannot accurately approximate a discontinuous map, which in inverse problems is typically the inversion map. Nevertheless, we prove in the next section that if the forward map is well-posed, then generic neural operators can approximate the forward map and its linearisation to a certain accuracy.
Simultaneous approximation of the forward map and its derivative
Our proposed surrogate Landweber iteration (3) uses both the forward map and the adjoint of its derivative, so we need neural operators that approximate both at once. We work with a reduced-basis class built from a bounded linear encoder, a finite-dimensional neural network and a bounded linear decoder. We first approximate derivative actions on compact sets of directions, and then, under additional assumptions, the derivative in operator norm, which is the bound used in § 5 .
In this section, we allow M \mathcal{M} M and Y \mathcal{Y} Y to be Banach spaces. The error-bound analysis of the surrogate Landweber iteration (3) in § 5 is carried out in Hilbert spaces, but it can be extended to Banach spaces by adapting the convergence analysis of Schuster et al. [37 ] . An operator in the reduced-basis class has the form
F θ ( m ) = Φ f θ ( Ψ m ) , Ψ ∈ L ( M , R r ) , Φ ∈ L ( R s , Y ) . (4) F_{\theta}(m) = \Phi f_{\theta}(\Psi m), \qquad\Psi\in\mathcal{L}(\mathcal{M},\mathbb{R}^{r}), \qquad\Phi\in\mathcal{L}(\mathbb{R}^{s},\mathcal{Y}). \tag*{(4)} F θ ( m ) = Φ f θ ( Ψ m ) , Ψ ∈ L ( M , R r ) , Φ ∈ L ( R s , Y ) . (4)
Here, the bounded linear encoder Ψ \Psi Ψ maps the input m m m to r r r coordinates, the finite neural network f θ : R r → R s f_{\theta} : \mathbb{R}^{r} \to\mathbb{R}^{s} f θ : R r → R s maps these to s s s output coefficients, and the bounded linear decoder Φ \Phi Φ reconstructs an element of Y \mathcal{Y} Y from those coefficients. In the approximation results, the encoder, the decoder and their dimensions are chosen together with the network, so greater accuracy may require richer input and output representations as well as a larger network.
For the finite-dimensional network f θ f_{\theta} f θ , we use a nonpolynomial activation σ ∈ C 1 ( R ) \sigma\in C^{1}(\mathbb{R}) σ ∈ C 1 ( R ) , such as GELU or tanh. Networks with this activation can approximate a finite-dimensional C 1 C^{1} C 1 map and its first derivative uniformly on compact sets ([16 , 33 ] ). This is the finite-dimensional step in the constructions below, and Lemma A.4 in Appendix § 11 states the version used here.
Compact direction sets
de Feo [7 ] shows that encoder–decoder neural operators, a class that includes DeepONets, approximate a globally defined operator in his class C B k C^{k}_{B} C B k and its first k k k derivatives simultaneously on compact sets of inputs and directions. Theorem 3.1 gives a local, first-derivative version of this result for forward maps defined on an open subset of the input space. Appendix § 11 provides a self-contained proof and the corresponding encoder–network–decoder construction.
A Banach space has the approximation property if its identity can be approximated uniformly on every compact set by bounded finite-rank maps. In the theorem below, this property provides finite-dimensional representations of the inputs and directions in M \mathcal{M} M and of the corresponding values and derivative outputs in Y \mathcal{Y} Y .
For standard FNOs, simultaneous approximation of an operator and its derivative is established by Yao et al. [39 ] . § 4.3 compares that result with the ones below.
Theorem 3.1 (Approximation on compact inputs and directions). Suppose M \mathcal{M} M and Y \mathcal{Y} Y have the approximation property as defined in Definition A.1 , U ⊂ M U \subset\mathcal{M} U ⊂ M is open, and F ∈ C 1 ( U , Y ) \mathcal{F} \in C^{1}(U,\mathcal{Y}) F ∈ C 1 ( U , Y ) . Let K ⊂ U \mathcal{K} \subset U K ⊂ U and H ⊂ M H \subset\mathcal{M} H ⊂ M be compact. For every ϵ > 0 \epsilon> 0 ϵ > 0 , there exists an operator of the form (4) such that
sup m ∈ K ∥ F ( m ) − F θ ( m ) ∥ Y < ϵ . (5) \sup_{m \in\mathcal{K}} \lVert\mathcal{F}(m)-\mathcal{F}_{\theta}(m)\rVert_{\mathcal{Y}} < \epsilon. \tag*{(5)} m ∈ K sup ∥ F ( m ) − F θ ( m ) ∥ Y < ϵ . (5) sup m ∈ K h ∈ H ∥ D F ( m ) h − D F θ ( m ) h ∥ Y < ϵ . (6) \sup_{\substack{m \in\mathcal{K}\\h\in H}} \lVert D\mathcal{F}(m)h-D\mathcal{F}_{\theta}(m)h\rVert_{\mathcal{Y}} < \epsilon. \tag*{(6)} m ∈ K h ∈ H sup ∥ D F ( m ) h − D F θ ( m ) h ∥ Y < ϵ . (6)
Proof. See Appendix A.2.
Approximation of derivatives in operator norm
Theorem 3.1 controls the derivative error along a prescribed compact set of directions. The Landweber analysis in § 5 requires a bound over all directions of norm at most one, that is, in the operator norm
∥ D F ( m ) − D F θ ( m ) ∥ L ( M , Y ) = sup ∥ h ∥ M ≤ 1 ∥ ( D F ( m ) − D F θ ( m ) ) h ∥ Y . \lVert D\mathcal{F}(m)-D\mathcal{F}_{\theta}(m)\rVert_{\mathcal{L}(\mathcal{M},\mathcal{Y})}=\sup_{\lVert h\rVert_{\mathcal{M}}\leq1}\lVert(D\mathcal{F}(m)-D\mathcal{F}_{\theta}(m))h\rVert_{\mathcal{Y}}. ∥ D F ( m ) − D F θ ( m ) ∥ L ( M , Y ) = ∥ h ∥ M ≤ 1 sup ∥( D F ( m ) − D F θ ( m )) h ∥ Y .
Because the unit ball is not compact in infinite dimensions, Theorem 3.1 does not give this bound. The following theorem does, under additional assumptions that include compactness of the derivative.
Theorem 3.2 (Uniform operator-norm approximation). Let M \mathcal{M} M be a reflexive Banach space with a Schauder basis and let Y \mathcal{Y} Y have a Schauder basis. Suppose U ⊂ M U \subset\mathcal{M} U ⊂ M is open, F ∈ C 1 ( U , Y ) \mathcal{F} \in C^{1}(U,\mathcal{Y}) F ∈ C 1 ( U , Y ) , and K ⊂ U \mathcal{K} \subset U K ⊂ U is compact. If D F ( m ) D\mathcal{F}(m) D F ( m ) is compact for every m ∈ K m \in\mathcal{K} m ∈ K , then for every ϵ > 0 \epsilon> 0 ϵ > 0 there is an operator of the form (4) such that
sup m ∈ K ∥ F ( m ) − F θ ( m ) ∥ Y < ϵ , sup m ∈ K ∥ D F ( m ) − D F θ ( m ) ∥ L ( M , Y ) < ϵ . \sup_{m \in\mathcal{K}}\lVert\mathcal{F}(m)-\mathcal{F}_{\theta}(m)\rVert_{\mathcal{Y}} < \epsilon,\qquad\sup_{m \in\mathcal{K}}\lVert D\mathcal{F}(m)-D\mathcal{F}_{\theta}(m)\rVert_{\mathcal{L}(\mathcal{M},\mathcal{Y})} < \epsilon. m ∈ K sup ∥ F ( m ) − F θ ( m ) ∥ Y < ϵ , m ∈ K sup ∥ D F ( m ) − D F θ ( m ) ∥ L ( M , Y ) < ϵ .
Proof. See Appendix A.3.
The compactness assumption in Theorem 3.2 cannot be removed for the class (4) . Its derivatives have the form D F θ ( m ) = Φ D f θ ( Ψ m ) Ψ D\mathcal{F}_{\theta}(m)=\Phi Df_{\theta}(\Psi m)\Psi D F θ ( m ) = Φ D f θ ( Ψ m ) Ψ and therefore have finite rank, since their range lies in the range of Φ \Phi Φ . An operator-norm limit of finite-rank operators is compact, so approximating D F ( m ) D\mathcal{F}(m) D F ( m ) arbitrarily well in operator norm requires D F ( m ) D\mathcal{F}(m) D F ( m ) to be compact. More generally, de Feo [7 ] Theorem 4.2 shows that encoder–decoder architectures are not dense in C k C^{k} C k with derivatives measured in operator norm whenever some bounded operator from the input to the output space cannot be approximated by finite-rank operators.
Corollary 3.3 (Approximation of weighted derivatives). Suppose the assumptions of Theorem 3.1 hold on a compact input set K ⊂ U \mathcal{K} \subset U K ⊂ U , and let T ∈ L ( M ) T \in\mathcal{L}(\mathcal{M}) T ∈ L ( M ) be compact. Then , for every ϵ > 0 \epsilon> 0 ϵ > 0 , there exists an operator of the form (4) such that
sup m ∈ K ∥ F ( m ) − F θ ( m ) ∥ Y < ϵ , sup m ∈ K ∥ ( D F ( m ) − D F θ ( m ) ) T ∥ L ( M , Y ) < ϵ . \sup_{m \in\mathcal{K}}\lVert\mathcal{F}(m)-\mathcal{F}_{\theta}(m)\rVert_{\mathcal{Y}} < \epsilon,\qquad\sup_{m \in\mathcal{K}}\lVert(D\mathcal{F}(m)-D\mathcal{F}_{\theta}(m))T\rVert_{\mathcal{L}(\mathcal{M},\mathcal{Y})} < \epsilon. m ∈ K sup ∥ F ( m ) − F θ ( m ) ∥ Y < ϵ , m ∈ K sup ∥( D F ( m ) − D F θ ( m )) T ∥ L ( M , Y ) < ϵ .
Proof. The set H T = T B M 1 / 2 H_T = T B\mathcal{M}^{1/2} H T = T B M 1/2 is compact in M \mathcal{M} M . Apply Theorem 3.1 with direction set H T H_T H T and note that ∥ ( D F ( m ) − D F θ ( m ) ) T ∥ ≤ sup h ∈ H T ∥ ( D F ( m ) − D F θ ( m ) ) h ∥ Y \lVert(DF(m)-DF_\theta(m))T\rVert\le\sup_{h\in H_T}\lVert(DF(m)-DF_\theta(m))h\rVert_Y ∥( D F ( m ) − D F θ ( m )) T ∥ ≤ sup h ∈ H T ∥( D F ( m ) − D F θ ( m )) h ∥ Y .
Unlike Theorem 3.2, this result does not require D F ( m ) DF(m) D F ( m ) to be compact, because the weight T T T supplies the compactness. It is used with T = C 1 / 2 T=C^{1/2} T = C 1/2 in Section 5.1.
Section 4 writes the reduced-basis class as two neural-operator architectures, so the results above apply to both.
Generalised DeepONet and generalised Fourier architectures
This section writes the reduced-basis class (4) as two neural-operator architectures. In a generalised DeepONet, the encoder extracts scalar features from the input, the network predicts output coefficients, and the decoder combines fixed output elements with those coefficients. In a generalised Fourier operator, the same class is expressed in Fourier coordinates.
Generalised DeepONet
Definition 4.1 (Generalised DeepONet). Let ℓ 1 , … , ℓ r ∈ M ∗ \ell_1,\ldots,\ell_r\in\mathcal{M}^* ℓ 1 , … , ℓ r ∈ M ∗ be bounded linear sensors, β 1 , … , β s ∈ Y \beta_1,\ldots,\beta_s\in\mathcal{Y} β 1 , … , β s ∈ Y fixed output elements, and f θ : R r → R s f_\theta:\mathbb{R}^r\to\mathbb{R}^s f θ : R r → R s a neural network. Set
F θ G D ( m ) = ∑ j = 1 s [ f θ ( ℓ 1 ( m ) , … , ℓ r ( m ) ) ] j β j . (7) \mathcal{F}^{\mathrm{GD}}_\theta(m)=\sum_{j=1}^{s}[f_\theta(\ell_1(m),\ldots,\ell_r(m))]_j\beta_j. \tag*{(7)} F θ GD ( m ) = j = 1 ∑ s [ f θ ( ℓ 1 ( m ) , … , ℓ r ( m )) ] j β j . (7)
The generalised DeepONet formula is another way of writing the operators (4), with the same derivative when f θ f_\theta f θ is continuously differentiable.
Proposition 4.2 (Exact DeepONet representation). The class (7) equals the class (4). If f θ ∈ C 1 f_\theta\in C^1 f θ ∈ C 1 , the identification also preserves the Fréchet derivative:
D F θ G D ( m ) h = ∑ j = 1 s [ D f θ ( Ψ m ) Ψ h ] j β j = Φ D f θ ( Ψ m ) Ψ h . (8) D\mathcal{F}^{\mathrm{GD}}_\theta(m)h=\sum_{j=1}^{s}[Df_\theta(\Psi m)\Psi h]_j\beta_j=\Phi Df_\theta(\Psi m)\Psi h. \tag*{(8)} D F θ GD ( m ) h = j = 1 ∑ s [ D f θ ( Ψ m ) Ψ h ] j β j = Φ D f θ ( Ψ m ) Ψ h . (8)
Proof. For (4), take ℓ i = e i ∗ Ψ \ell_i=e_i^*\Psi ℓ i = e i ∗ Ψ and β j = Φ e j \beta_j=\Phi e_j β j = Φ e j . Expanding the vector f θ ( Ψ m ) f_\theta(\Psi m) f θ ( Ψ m ) in the standard basis yields (7). Conversely, the finitely many bounded sensors define a bounded encoder, and c ↦ ∑ j c j β j c\mapsto\sum_j c_j\beta_j c ↦ ∑ j c j β j is a bounded decoder. The chain rule gives (8).
The generalised formulation builds on several constructions in the operator-learning literature. In the original DeepONet, a branch network maps sampled input values to coefficients, while a trunk network represents the output functions combined with those coefficients (Lu et al., 2021). In (7), f θ f_\theta f θ plays the role of the branch network and the fixed elements β j \beta_j β j play the role of the trunk functions. The input features are specified more generally by bounded linear functionals ℓ i \ell_i ℓ i .
Choosing fixed output functions follows the approach of POD-DeepONet, where the trunk is constructed from modes obtained by proper orthogonal decomposition (POD) of training outputs (Lu et al., 2022). A related construction is PCA-Net, which uses principal component analysis (PCA) to obtain finite-dimensional representations of both inputs and outputs, and trains a neural network to map between their coordinates Bhattacharya et al. [4 ] . These constructions provide concrete choices of the input measurements and output elements in (7) .
The specific architecture used in our numerical experiments, denoted by GenDeepONet , adopts empirical PCA/POD representations within this class. Its encoder extracts coordinates in an empirical input subspace, f θ f_\theta f θ predicts output coefficients, and the decoder reconstructs the output using empirical output modes fixed before network training. Fixed means and affine normalisations can be absorbed into network biases and an additional constant output element, and they do not change the derivative formula for the linear parts.
Remark 4.3 (Simultaneous approximation by standard DeepONets). In the original DeepONet, the output elements are produced by a trained trunk network and the sensors are point evaluations. The approximation results extend to this setting under two conditions. First, the sensors must be bounded on M \mathcal{M} M , which holds for point evaluations on C ( Ω ‾ ) C(\overline{\Omega}) C ( Ω ) but not on L 2 ( Ω ) L^2(\Omega) L 2 ( Ω ) . Second, the trunk must approximate the fixed output elements of (7) in Y \mathcal{Y} Y , which is possible whenever network functions of the output variable are dense in Y \mathcal{Y} Y , as for L 2 ( Ω ) L^2(\Omega) L 2 ( Ω ) or C ( Ω ‾ ) C(\overline{\Omega}) C ( Ω ) .
Under the respective hypotheses of § 3 , the approximation results extend to standard DeepONets with the additional conditions above.
Fix F θ = Φ f θ Ψ F_\theta= \Phi f_\theta\Psi F θ = Φ f θ Ψ of the form (4) , where Ψ = ( ℓ 1 , … , ℓ r ) \Psi= (\ell_1,\ldots,\ell_r) Ψ = ( ℓ 1 , … , ℓ r ) . For M = C ( X ) \mathcal{M} = C(X) M = C ( X ) with X = Ω ‾ X = \overline{\Omega} X = Ω compact, take finite partitions of unity ( φ n , i ) i (\varphi_{n,i})_i ( φ n , i ) i subordinate to covers of vanishing diameter, and choose sample points x n , i x_{n,i} x n , i in the corresponding cover sets. Equicontinuity yields
S n m : = ∑ i m ( x n , i ) φ n , i ⟶ m uniformly for m ∈ K ∪ H . S_n m := \sum_i m(x_{n,i})\varphi_{n,i} \longrightarrow m \quad\text{uniformly for } m \in\mathcal{K} \cup H. S n m := i ∑ m ( x n , i ) φ n , i ⟶ m uniformly for m ∈ K ∪ H .
Thus, setting E n m = ( m ( x n , i ) ) i E_n m = (m(x_{n,i}))_i E n m = ( m ( x n , i ) ) i , the encoders
Ψ n = Ψ S n = A n E n , ( A n ) j i = ℓ j ( φ n , i ) , \Psi_n = \Psi S_n = A_n E_n, \qquad(A_n)_{ji} = \ell_j(\varphi_{n,i}), Ψ n = Ψ S n = A n E n , ( A n ) j i = ℓ j ( φ n , i ) ,
approximate Ψ \Psi Ψ uniformly on K ∪ H \mathcal{K} \cup H K ∪ H . This covers Theorem 3.1 and Corollary 3.3, taking H = T B 1 M ‾ H = \overline{T B_1^{\mathcal{M}}} H = T B 1 M in the latter.
For Theorem 3.2 , additionally assume that M \mathcal{M} M is a function space with bounded point evaluations separating its elements. The Hahn–Banach theorem and reflexivity give
span { δ x : x ∈ X } ‾ ∥ ⋅ ∥ M ∗ = M ∗ . \overline{\operatorname{span}\{\delta_x:x\in X\}}^{\|\cdot\|_{\mathcal{M}^*}} = \mathcal{M}^*. span { δ x : x ∈ X } ∥ ⋅ ∥ M ∗ = M ∗ .
Approximating each ℓ j \ell_j ℓ j by finite linear combinations of point evaluations gives Ψ n = A n E n → Ψ \Psi_n = A_n E_n \to\Psi Ψ n = A n E n → Ψ in operator norm.
In both cases, continuity of f θ f_\theta f θ , D f θ Df_\theta D f θ and
D F θ , n ( m ) = Φ D f θ ( Ψ n m ) Ψ n , F θ , n = Φ f θ Ψ n , DF_{\theta,n}(m) = \Phi Df_\theta(\Psi_n m)\Psi_n,\qquad F_{\theta,n} = \Phi f_\theta\Psi_n, D F θ , n ( m ) = Φ D f θ ( Ψ n m ) Ψ n , F θ , n = Φ f θ Ψ n ,
give the corresponding value and derivative approximation; A n A_n A n is absorbed into the first branch layer. Finally, replacing β j = Φ e j \beta_j = \Phi e_j β j = Φ e j by trunk functions satisfying ∥ β ~ j − β j ∥ Y ≤ η \|\widetilde{\beta}_j-\beta_j\|_{\mathcal{Y}}\leq\eta ∥ β j − β j ∥ Y ≤ η changes values and directional derivatives by at most
η ∥ f θ ( Ψ n m ) ∥ 1 , η ∥ D f θ ( Ψ n m ) Ψ n h ∥ 1 , \eta\|f_\theta(\Psi_n m)\|_1,\qquad\eta\|Df_\theta(\Psi_n m)\Psi_n h\|_1, η ∥ f θ ( Ψ n m ) ∥ 1 , η ∥ D f θ ( Ψ n m ) Ψ n h ∥ 1 ,
respectively. These factors are uniformly bounded for m ∈ K m \in K m ∈ K and h h h in the prescribed compact direction set, or in B 1 M B_1^M B 1 M for Theorem 3.2. Choosing the sensor and trunk errors successively small completes the approximation.
The input measurements must be bounded, so that small changes of the input produce small changes in the encoded coordinates. For L 2 ( Ω ) L^2(\Omega) L 2 ( Ω ) inputs, suitable measurements include inner products with fixed L 2 ( Ω ) L^2(\Omega) L 2 ( Ω ) functions, such as the coordinates used by the PCA encoder. Point samples can also be used when the input space controls pointwise values, as in C ( Ω ‾ ) C(\overline{\Omega}) C ( Ω ) equipped with the supremum norm.
Generalised fully coupled Fourier operators
Let V r , V s V_r,V_s V r , V s be finite-dimensional real Fourier spaces. We use a real trigonometric basis for each space, consisting of sine and cosine functions and the constant function when that mode is retained. The coordinate map C j : V j → R j \mathcal{C}_j: V_j \to\mathbb{R}^{j} C j : V j → R j extracts the real coefficients of an element of V j V_j V j , while S j : R j → V j \mathcal{S}_j: \mathbb{R}^{j} \to V_j S j : R j → V j reconstructs that element from its coefficients, so that C j S j = I \mathcal{C}_j\mathcal{S}_j=I C j S j = I and S j C j = I V j \mathcal{S}_j\mathcal{C}_j=I_{V_j} S j C j = I V j . This allows the real-valued networks of Section 3 to act on Fourier coefficients.
Definition 4.4 (Generalised fully coupled Fourier operator). Given bounded linear maps P : M → V r P:\mathcal{M}\to V_r P : M → V r , Q : V s → Y Q:V_s\to\mathcal{Y} Q : V s → Y , define
F θ G F = Q S s f θ C r P . (9) \mathcal{F}^{\mathrm{GF}}_{\theta}=Q\mathcal{S}_s f_{\theta}\mathcal{C}_rP. \tag*{(9)} F θ GF = Q S s f θ C r P . (9) Here f θ : R r → R s f_{\theta}:\mathbb{R}^{r}\to\mathbb{R}^{s} f θ : R r → R s is the finite-dimensional neural network introduced in Section 3. It maps the Fourier-coordinate vector C r P m \mathcal{C}_rPm C r P m to output coefficients, which S s \mathcal{S}_s S s uses to reconstruct an element of V s V_s V s . We allow fully connected layers in f θ f_{\theta} f θ , so each output coefficient may depend on all retained input coordinates.
We note that we avoid the name “generalised FNO”, which has been used for a different Fourier-based architecture (Ke et al., 2026).
Proposition 4.5 (Exact generalised Fourier representation). If the bounded outer maps P , Q P,Q P , Q may be chosen freely, the class (9) equals (4). The operators and their derivatives agree exactly.
Proof. For Φ f θ Ψ \Phi f_{\theta}\Psi Φ f θ Ψ , choose P = S r Ψ P=\mathcal{S}_r\Psi P = S r Ψ and Q = Φ C s Q=\Phi\mathcal{C}_s Q = Φ C s . Then
Q S s f θ C r P = Φ ( C s S s ) f θ ( C r S r ) Ψ = Φ f θ Ψ . Q\mathcal{S}_s f_{\theta}\mathcal{C}_rP=\Phi(\mathcal{C}_s\mathcal{S}_s)f_{\theta}(\mathcal{C}_r\mathcal{S}_r)\Psi=\Phi f_{\theta}\Psi. Q S s f θ C r P = Φ ( C s S s ) f θ ( C r S r ) Ψ = Φ f θ Ψ. Conversely, set Ψ = C r P \Psi=\mathcal{C}_rP Ψ = C r P and Φ = Q S s \Phi=Q\mathcal{S}_s Φ = Q S s . These are bounded linear maps, so the chain rule again gives (8) with these Ψ , Φ \Psi,\Phi Ψ , Φ .
The freedom in P , Q P,Q P , Q is essential here, since the Fourier maps then act as coordinate changes around an arbitrary reduced-basis operator. The statement does not identify every such operator with a model whose encoder is a fixed Fourier truncation and whose decoder is a fixed Fourier synthesis.
The specific architecture used in our numerical experiments, denoted by G e n F N O \mathit{GenFNO} GenFNO , uses fixed Fourier truncation and synthesis maps. Its finite-dimensional network f θ f_{\theta} f θ is a residual multilayer perceptron that maps the retained input Fourier coefficients to output Fourier coefficients. Learning a map between spectral coefficients follows the Fourier spectral neural operator approach, or SNO(F), of Fanaskov and Oseledets [12 ] , and the explicit mode mixing of the higher-order FNO of Colagrande et al. [6 ] is a related construction.
Corollary 4.6 (Transfer of simultaneous approximation). The conclusions of Theorems 3.1 and 3.2 , and of Corollary 3.3, hold with generalised DeepONets (7) and with the broad generalised Fourier class (9), under their respective hypotheses. If M = H s ( T d ) \mathcal{M}=H^{s}(\mathbb{T}^{d}) M = H s ( T d ) and Y = H s ′ ( T d ) \mathcal{Y}=H^{s'}(\mathbb{T}^{d}) Y = H s ′ ( T d ) , the approximants in these results can be chosen in (9) with P P P the orthogonal Fourier truncation onto V r V_{r} V r and Q Q Q the inclusion of V s V_{s} V s into Y \mathcal{Y} Y .
Proof. Let F θ = Φ f θ Ψ \mathcal{F}_{\theta}=\Phi f_{\theta}\Psi F θ = Φ f θ Ψ be an approximant from Theorem 3.1 , Theorem 3.2 or Corollary 3.3. By Proposition 4.2 it equals the generalised DeepONet with ℓ i = e i ∗ Ψ \ell_i=e_i^{*}\Psi ℓ i = e i ∗ Ψ and β j = Φ e j \beta_j=\Phi e_j β j = Φ e j . By Proposition 4.5 it equals the element of (9) with P = S r Ψ P=\mathcal{S}_{r}\Psi P = S r Ψ and Q = Φ C s Q=\Phi\mathcal{C}_{s} Q = Φ C s . In both cases the derivatives coincide by (8), so the errors are unchanged.
For the periodic case, let { φ k } \{\varphi_k\} { φ k } be the real trigonometric basis, normalised to be orthonormal in H s ( T d ) H^{s}(\mathbb{T}^{d}) H s ( T d ) , and let Π n \Pi_n Π n be the orthogonal projection onto span { φ k : ∣ k ∣ ∞ ≤ n } \operatorname{span}\{\varphi_k:|k|_{\infty}\le n\} span { φ k : ∣ k ∣ ∞ ≤ n } ; define Π N \Pi_N Π N similarly in Y \mathcal{Y} Y . Ordering the frequencies by ∣ k ∣ ∞ |k|_{\infty} ∣ k ∣ ∞ makes ( Π n ) (\Pi_n) ( Π n ) a subsequence of the partial-sum projections of an orthonormal basis. Hence ∥ Π n ∥ = 1 \|\Pi_n\|=1 ∥ Π n ∥ = 1 , Π n → I \Pi_n\to I Π n → I strongly and uniformly on compact sets, and, since Π n ∗ = Π n \Pi_n^{*}=\Pi_n Π n ∗ = Π n , also Π n ∗ → I \Pi_n^{*}\to I Π n ∗ → I strongly. These are the only properties of the projections used in Appendices § 11 .§ 2 and § 11 .§ 3 . Those proofs therefore apply with P = Π n P=\Pi_n P = Π n and Q = Π N Q=\Pi_N Q = Π N in the first, and with P n = Π n P_n=\Pi_n P n = Π n and Q N = Π N Q_N=\Pi_N Q N = Π N in the second. The encoder Ψ \Psi Ψ is then the vector of retained Fourier coefficients and the decoder Φ \Phi Φ is Fourier synthesis, which is (9) with P = Π n : M → V r P=\Pi_n:\mathcal{M}\to V_r P = Π n : M → V r and Q Q Q the inclusion V s ↪ Y V_s\hookrightarrow\mathcal{Y} V s ↪ Y . □
The corollary allows truncations and network sizes to grow and says nothing about the particular mode counts used in the experiments. The reaction benchmark of § 7.1 uses finite-grid Fourier coordinates on a bounded square, and this periodic representation is a computational choice that does not satisfy the periodic continuum hypotheses above.
Comparison with standard FNOs
Standard FNOs, which also appear in our numerical comparisons, arrange their layers differently from the classes above.
In the Fourier coefficient architecture of § 4.2 , the finite network f θ f_{\theta} f θ maps retained input Fourier coordinates to output coefficients, from which the decoder reconstructs the output. Its fully connected layers allow each predicted coefficient to depend on all retained input coordinates. A standard FNO instead applies successive layers to intermediate functions, combining Fourier multipliers with nonlinear activations in physical space (Li et al. (2021)). Each multiplier acts separately at each retained frequency, while the physical-space nonlinearities allow interactions between frequencies. Appendix § 12 gives the standard-FNO layer formulas and their derivative and norm calculations.
Remark 4.7 (Frequency coupling in standard FNO layers). Write a standard Fourier layer as v t + 1 = σ ( W t v t + F − 1 ( R t ⋅ F v t ) + b t ) v_{t+1}=\sigma(W_t v_t+\mathcal{F}^{-1}(R_t\cdot\mathcal{F}v_t)+b_t) v t + 1 = σ ( W t v t + F − 1 ( R t ⋅ F v t ) + b t ) , where W t W_t W t acts pointwise on the channels and R t ( k ) R_t(k) R t ( k ) is the multiplier at the retained frequency k k k . In Fourier coordinates, the argument of σ \sigma σ is ( W t + R t ( k ) ) v ^ t ( k ) + b ^ t ( k ) (W_t+R_t(k))\widehat{v}_t(k)+\widehat{b}_t(k) ( W t + R t ( k )) v t ( k ) + b t ( k ) at each retained frequency, and W t v ^ t ( k ) + b ^ t ( k ) W_t\widehat{v}_t(k)+\widehat{b}_t(k) W t v t ( k ) + b t ( k ) at the others. The linear part is therefore block diagonal in k k k . It couples channels but not frequencies, and coupling between frequencies arises only through the pointwise nonlinearity in physical space. The generalised Fourier class (9) replaces this block-diagonal structure by a dense map acting on all retained coefficients at once, which is what allows it to represent every operator of the form (4).
Approximation theory for standard FNOs includes universality for continuous operators [24 ] and simultaneous approximation of values and derivatives [39 Theorem 3.2] . The latter result concerns FNOs between periodic Sobolev spaces, gives uniform approximation on compact input sets, and assumes that the derivative depends continuously on the input in a Hilbert–Schmidt norm. The space of derivative directions may be the original input space or a smoother Sobolev space. On the same pair of spaces, a Hilbert–Schmidt bound also controls the operator norm, but when the direction space is smoother it controls only directions measured in the stronger norm, not the full unit ball of the original input space.
In their terminology, a standard FNO trained with derivative information is a derivative-informed Fourier neural operator (DIFNO). The distinction concerns the training, not the architecture.
Theorem 3.2, by contrast, controls the derivative error of the reduced-basis class over all unit directions of the original input space, without Hilbert–Schmidt derivatives, and in Hilbert spaces this also controls the adjoint error in the Landweber update. Yao et al. [39 ] use derivative approximation to analyse solutions of strongly convex regularised optimisation problems, whereas we use it for surrogate Landweber iteration and its stopping rule. The two sets of results are complementary, and we make no priority claim for simultaneous approximation of an operator and its derivative.
Preconditioned surrogate Landweber regularisation
In this section, we study preconditioned surrogate Landweber regularisation (PSLR) for the problem
F ( m ) = y . (10) F(m) = y . \tag*{(10)} F ( m ) = y . (10)
Based on the available data y δ y^{\delta} y δ satisfying (1), this corresponds to the iteration
m k + 1 = m k − ω C D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) , k = 0 , 1 , 2 , … , (11) m_{k+1} = m_k - \omega C D F_{\theta}(m_k)^{*}\left(F_{\theta}(m_k)-y^{\delta}\right), \qquad k = 0,1,2,\ldots, \tag*{(11)} m k + 1 = m k − ω C D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) , k = 0 , 1 , 2 , … , (11)
with initial guess m 0 m_0 m 0 , step size ω > 0 \omega> 0 ω > 0 and a self-adjoint, positive preconditioning operator C C C on the parameter space M \mathcal{M} M . The employed operator F θ F_{\theta} F θ is a surrogate for the original model or forward operator F F F in (10), e.g. a generalised DeepONet or a member of the generalised Fourier class as discussed before.
Preconditioned Landweber regularisation with the exact forward map and preconditioners given by negative powers of a Hilbert-scale generator was studied, e.g. by Neubauer [29 ] and Egger and Neubauer [10 ] .
Employing a preconditioner C C C in (11) offers several advantages. Most importantly, in practice the training of the surrogate is based on finitely many directional derivatives D F ( m ) h DF(m)h D F ( m ) h only, and the directions h h h are typically sampled from a distribution of smooth functions. A trained network therefore cannot be expected to reproduce D F ( m ) DF(m) D F ( m ) in every direction of the parameter space: in rough directions, which are essentially absent from the training data, the derivative of the surrogate may be inaccurate however small the training loss is. If the directions are sampled from a centred Gaussian measure with trace-class covariance C ′ C' C ′ , the expected squared derivative error is
E h ∼ N ( 0 , C ′ ) ∥ ( D F θ ( m ) − D F ( m ) ) h ∥ Y 2 = ∥ ( D F θ ( m ) − D F ( m ) ) ( C ′ ) 1 / 2 ∥ H S 2 ≥ ∥ ( D F θ ( m ) − D F ( m ) ) ( C ′ ) 1 / 2 ∥ L ( M , Y ) 2 . (12) \mathbb{E}_{h\sim\mathcal{N}(0,C')}\left\|\left(D\mathcal{F}_{\theta}(m)-D\mathcal{F}(m)\right)h\right\|_{Y}^{2}=\left\|\left(D\mathcal{F}_{\theta}(m)-D\mathcal{F}(m)\right)(C')^{1/2}\right\|_{\mathrm{HS}}^{2}
\geq\left\|\left(D\mathcal{F}_{\theta}(m)-D\mathcal{F}(m)\right)(C')^{1/2}\right\|_{\mathcal{L}(M,Y)}^{2}. \tag*{(12)} E h ∼ N ( 0 , C ′ ) ∥ ( D F θ ( m ) − D F ( m ) ) h ∥ Y 2 = ( D F θ ( m ) − D F ( m ) ) ( C ′ ) 1/2 HS 2 ≥ ( D F θ ( m ) − D F ( m ) ) ( C ′ ) 1/2 L ( M , Y ) 2 . (12)
The training thus controls the derivative error weighted by ( C ′ ) 1 / 2 (C')^{1/2} ( C ′ ) 1/2 , i.e. the derivative in the directions that are smooth with respect to C ′ C' C ′ , but not the unweighted error ∥ D F θ ( m ) − D F ( m ) ∥ L ( M , Y ) \|D\mathcal{F}_{\theta}(m)-D\mathcal{F}(m)\|_{\mathcal{L}(M,Y)} ∥ D F θ ( m ) − D F ( m ) ∥ L ( M , Y ) . A training loss averaged over parameters m m m controls this weighted error only on average, whereas the analysis below requires a bound that is uniform in m m m .
Assume now that the preconditioner C C C in (11) is at least as smoothing as C ′ C' C ′ , in the sense that
⟨ C h , h ⟩ M ≤ λ ⟨ C ′ h , h ⟩ M for all h ∈ M (13) \langle Ch,h\rangle_{M}\leq\lambda\langle C'h,h\rangle_{M}\qquad\text{for all }h\in\mathcal{M} \tag*{(13)} ⟨ C h , h ⟩ M ≤ λ ⟨ C ′ h , h ⟩ M for all h ∈ M (13)
for some λ > 0 \lambda>0 λ > 0 . Applying (13) to h = ( D F θ ( m ) − D F ( m ) ) ∗ y h=(D\mathcal{F}_{\theta}(m)-D\mathcal{F}(m))^{*}y h = ( D F θ ( m ) − D F ( m ) ) ∗ y , y ∈ Y y\in\mathcal{Y} y ∈ Y , yields
∥ ( D F θ ( m ) − D F ( m ) ) C 1 / 2 ∥ L ( M , Y ) ≤ λ 1 / 2 ∥ ( D F θ ( m ) − D F ( m ) ) ( C ′ ) 1 / 2 ∥ L ( M , Y ) . \left\|\left(D\mathcal{F}_{\theta}(m)-D\mathcal{F}(m)\right)C^{1/2}\right\|_{\mathcal{L}(M,Y)}
\leq\lambda^{1/2}\left\|\left(D\mathcal{F}_{\theta}(m)-D\mathcal{F}(m)\right)(C')^{1/2}\right\|_{\mathcal{L}(M,Y)}. ( D F θ ( m ) − D F ( m ) ) C 1/2 L ( M , Y ) ≤ λ 1/2 ( D F θ ( m ) − D F ( m ) ) ( C ′ ) 1/2 L ( M , Y ) .
The training therefore also controls the derivative error weighted by C 1 / 2 C^{1/2} C 1/2 , which is the only form in which the derivative error enters the analysis below. In practice, however, the directions are often taken from a fixed reduced basis h 1 , … , h r h_1,\ldots,h_r h 1 , … , h r , as in Section 6.1 and in our experiments. Then C ′ = ∑ j = 1 r h j ⊗ h j C'=\sum_{j=1}^{r}h_j\otimes h_j C ′ = ∑ j = 1 r h j ⊗ h j has finite rank, and (13) cannot hold for an injective C C C : for every h h h orthogonal to h 1 , … , h r h_1,\ldots,h_r h 1 , … , h r it forces ⟨ C h , h ⟩ M = 0 \langle Ch,h\rangle_{M}=0 ⟨ C h , h ⟩ M = 0 , that is, C h = 0 Ch=0 C h = 0 . Training then controls the derivative error only on span { h 1 , … , h r } \operatorname{span}\{h_1,\ldots,h_r\} span { h 1 , … , h r } , while a smoothing C C C attenuates, but does not bound, the error in the remaining directions; see (36) in § 6.3 . The analysis below therefore assumes a bound on the weighted derivative error rather than deriving it from the training loss. At the same time, every increment of (11) lies in ran C \operatorname{ran} C ran C , so a smoothing C C C keeps the iteration away from the rough directions in which the learned derivative is least reliable.
After introducing the PSLR (11), our aim is now to show that it is in fact a regularisation method in the classical sense:
It is well-defined, i.e. the iterates m k m_k m k are computable.
It is stable with respect to the data y δ y^\delta y δ .
It converges towards an exact solution when the noise level δ \delta δ and the approximation error of F θ \mathcal{F}_{\theta} F θ for the true model F \mathcal{F} F tend to 0.
Note that stability is a standard result for (exact) Landweber regularisation, see, e.g., Hanke et al. [15 ] ; Kaltenbacher et al. [19 ] . There it is shown that, under continuity assumptions on F \mathcal{F} F , for each fixed k k k the iterate m k m_k m k depends continuously on y δ y^\delta y δ . This result directly carries over to (11) provided that F θ \mathcal{F}_{\theta} F θ and the adjoint of its derivative are continuous.
Of further interest are results on convergence rates, which require source conditions on m † − m 0 m^\dagger-m_0 m † − m 0 ; for Hilbert-scale preconditioners and the exact forward map, their dependence on C C C is analysed by Neubauer [29 ] and Egger and Neubauer [10 ] . However, such results for the surrogate case are beyond the scope of the manuscript at hand.
Standing assumptions and linearisation error
In this subsection, we make the setting precise and bound the linearisation error of the surrogate. Let M \mathcal{M} M and Y \mathcal{Y} Y be the separable Hilbert spaces of § 1 . All adjoints are taken with respect to ⟨ ⋅ , ⋅ ⟩ M \langle\cdot,\cdot\rangle_{\mathcal{M}} ⟨ ⋅ , ⋅ ⟩ M . We group the hypotheses into assumptions on the forward problem and the surrogate, on the preconditioner and the initial guess, on the nonlinearity, and on the parameters of the method, and discuss each group directly after it.
Assumption 5.1 (Forward problem and surrogate). The forward operator and its surrogate satisfy F , F θ ∈ C 1 ( U , Y ) \mathcal{F},\mathcal{F}_{\theta}\in C^{1}(U,\mathcal{Y}) F , F θ ∈ C 1 ( U , Y ) for an open set U ⊆ M U\subseteq\mathcal{M} U ⊆ M , there exist exact data y † ∈ Y y^{\dagger}\in\mathcal{Y} y † ∈ Y and an exact solution m † ∈ U m^{\dagger}\in U m † ∈ U of (10) , i.e. F ( m † ) = y † \mathcal{F}(m^{\dagger})=y^{\dagger} F ( m † ) = y † , and the available data y δ ∈ Y y^{\delta}\in\mathcal{Y} y δ ∈ Y satisfy ∥ y δ − y † ∥ Y ≤ δ \lVert y^{\delta}-y^{\dagger}\rVert_{\mathcal{Y}}\leq\delta ∥ y δ − y † ∥ Y ≤ δ for a noise level δ ≥ 0 \delta\geq0 δ ≥ 0 .
Remark 5.2. Differentiability is a classical assumption to ensure well-definedness of Landweber iteration; in (11) it is required for the surrogate F θ \mathcal{F}_{\theta} F θ , since every step evaluates D F θ ( m k ) ∗ D\mathcal{F}_{\theta}(m_k)^* D F θ ( m k ) ∗ . Even though we use a surrogate in (11) , it is reasonable to assume differentiability of F \mathcal{F} F itself to ensure that F \mathcal{F} F and its derivative can be approximated by a differentiable surrogate.
For a self-adjoint, positive and injective operator C : M → M C:\mathcal{M}\to\mathcal{M} C : M → M , we denote by
M C : = ran ( C 1 2 ) , ⟨ h , v ⟩ M C : = ⟨ C − 1 2 h , C − 1 2 v ⟩ M , (14) \mathcal{M}_{C}:=\operatorname{ran}\left(C^{\frac{1}{2}}\right),\qquad\langle h,v\rangle_{\mathcal{M}_{C}}:=\left\langle C^{-\frac{1}{2}}h,C^{-\frac{1}{2}}v\right\rangle_{\mathcal{M}}, \tag*{(14)} M C := ran ( C 2 1 ) , ⟨ h , v ⟩ M C := ⟨ C − 2 1 h , C − 2 1 v ⟩ M , (14)
the space of directions that are smooth with respect to C C C , equipped with the induced inner product.
Assumption 5.3 (Preconditioner and initial guess). The preconditioning operator C : M → M C:\mathcal{M}\to\mathcal{M} C : M → M is self-adjoint, positive, injective and compact. For some radius ρ > 0 \rho>0 ρ > 0 , the closed ball
B ρ C ( m † ) : = { m ∈ m † + M C : ∥ m − m † ∥ M C ≤ ρ } (15) B_{\rho}^{C}(m^{\dagger}):=\{m\in m^{\dagger}+\mathcal{M}_{C}:\lVert m-m^{\dagger}\rVert_{\mathcal{M}_{C}}\leq\rho\} \tag*{(15)} B ρ C ( m † ) := { m ∈ m † + M C : ∥ m − m † ∥ M C ≤ ρ } (15)
around the exact solution with respect to the norm of M C \mathcal{M}_{C} M C obeys B ρ C ( m † ) ⊂ U B_{\rho}^{C}(m^{\dagger})\subset U B ρ C ( m † ) ⊂ U , and the initial guess satisfies m 0 ∈ B ρ C ( m † ) m_{0}\in B_{\rho}^{C}(m^{\dagger}) m 0 ∈ B ρ C ( m † ) , i.e. m † − m 0 ∈ M C m^{\dagger}-m_{0}\in\mathcal{M}_{C} m † − m 0 ∈ M C and ∥ m 0 − m † ∥ M C ≤ ρ \lVert m_{0}-m^{\dagger}\rVert_{\mathcal{M}_{C}}\leq\rho ∥ m 0 − m † ∥ M C ≤ ρ .
Remark 5.4. Self-adjointness, positivity and injectivity of C C C make M C \mathcal{M}_{C} M C in (14) a Hilbert space, in which C A ∗ CA^{*} C A ∗ is the adjoint of A ∣ M C A|_{\mathcal{M}_{C}} A ∣ M C for every A ∈ L ( M , Y ) A\in\mathcal{L}(\mathcal{M},\mathcal{Y}) A ∈ L ( M , Y ) . Hence (11) is the classical Landweber iteration, carried out in M C \mathcal{M}_{C} M C . Compactness of C C C implies that B ρ C ( m † ) B_{\rho}^{C}(m^{\dagger}) B ρ C ( m † ) in (15) is compact in M \mathcal{M} M . This will allow us to employ the previous results on uniform approximation of F \mathcal{F} F and D F D\mathcal{F} D F by the surrogate on B ρ C ( m † ) B_{\rho}^{C}(m^{\dagger}) B ρ C ( m † ) , the latter in the weighted norm ∥ ( ⋅ ) C 1 2 ∥ L ( M , Y ) \lVert(\cdot)C^{\frac{1}{2}}\rVert_{\mathcal{L}(\mathcal{M},\mathcal{Y})} ∥( ⋅ ) C 2 1 ∥ L ( M , Y ) (Corollary 3.3). In view of (12) , C C C should be at least as smoothing as the covariance C ′ C^{\prime} C ′ of the training directions in the sense of (13) .
The condition m 0 ∈ B ρ C ( m † ) m_{0}\in B_{\rho}^{C}(m^{\dagger}) m 0 ∈ B ρ C ( m † ) requires the initial error to be smooth with respect to C C C ; this is the price of preconditioning. For M = L 2 ( Ω ) \mathcal{M}=L^{2}(\Omega) M = L 2 ( Ω ) on a bounded Lipschitz domain Ω ⊂ R d \Omega\subset\mathbb{R}^{d} Ω ⊂ R d , a typical choice is the covariance operator ( C f ) ( x ) = ∫ Ω k ( x − y ) f ( y ) d y (Cf)(x)=\int_{\Omega}k(x-y)f(y)\,\mathrm{d}y ( C f ) ( x ) = ∫ Ω k ( x − y ) f ( y ) d y of a Matérn kernel k k k with smoothness ν C \nu_{C} ν C . Then M C \mathcal{M}_{C} M C coincides with H ν C + d / 2 ( Ω ) H^{\nu_{C}+d/2}(\Omega) H ν C + d /2 ( Ω ) , with equivalent norms and without boundary conditions (Wendland, [38 ] ). Hence a discontinuous initial error is excluded for every ν C > 0 \nu_{C}>0 ν C > 0 , whereas an initial error that is a Matérn sample of smoothness ν > ν C + d / 2 \nu>\nu_{C}+d/2 ν > ν C + d /2 lies in M C \mathcal{M}_{C} M C almost surely (Kanagawa et al., [20 ] ). The set B ρ C ( m † ) B_{\rho}^{C}(m^{\dagger}) B ρ C ( m † ) contains m † m^{\dagger} m † and depends only on m † m^{\dagger} m † , ρ \rho ρ and C C C , but not on the initial guess.
Assumption 5.5 (Nonlinearity). For some 0 ≤ η < 1 2 0 \le\eta< \frac{1}{2} 0 ≤ η < 2 1 , either the forward operator satisfies the local tangential cone condition
∥ F ( m ) − F ( m † ) − D F ( m ) ( m − m † ) ∥ Y ≤ η ∥ F ( m ) − F ( m † ) ∥ Y \lVert\mathcal{F}(m)-\mathcal{F}(m^\dagger)-D\mathcal{F}(m)(m-m^\dagger)\rVert_Y \le\eta\lVert\mathcal{F}(m)-\mathcal{F}(m^\dagger)\rVert_Y ∥ F ( m ) − F ( m † ) − D F ( m ) ( m − m † ) ∥ Y ≤ η ∥ F ( m ) − F ( m † ) ∥ Y (TCC)
for all m ∈ B ρ C ( m † ) m \in B_\rho^C(m^\dagger) m ∈ B ρ C ( m † ) , or the surrogate satisfies
∥ F θ ( m ) − F θ ( m † ) − D F θ ( m ) ( m − m † ) ∥ Y ≤ η ∥ F θ ( m ) − F θ ( m † ) ∥ Y \lVert\mathcal{F}_\theta(m)-\mathcal{F}_\theta(m^\dagger)-D\mathcal{F}_\theta(m)(m-m^\dagger)\rVert_Y \le\eta\lVert\mathcal{F}_\theta(m)-\mathcal{F}_\theta(m^\dagger)\rVert_Y ∥ F θ ( m ) − F θ ( m † ) − D F θ ( m ) ( m − m † ) ∥ Y ≤ η ∥ F θ ( m ) − F θ ( m † ) ∥ Y (TCCθ _\theta θ )
for all m ∈ B ρ C ( m † ) m \in B_\rho^C(m^\dagger) m ∈ B ρ C ( m † ) .
Remark 5.6. The tangential cone condition is a standard assumption for nonlinear Landweber iteration (see, e.g., Hanke et al., 1995; Kaltenbacher et al., 2008). Condition (TCC) is a property of the physical problem, whereas (TCCθ _\theta θ ) is a property of the trained network. Note that the latter does not follow from (TCC) and small forward or derivative training losses, since (TCC) is a relative condition whose right-hand side η ∥ F ( m ) − F ( m † ) ∥ Y \eta\lVert\mathcal{F}(m)-\mathcal{F}(m^\dagger)\rVert_Y η ∥ F ( m ) − F ( m † ) ∥ Y tends to zero as m → m † m \to m^\dagger m → m † , whereas absolute approximation errors do not. However, if (TCCθ _\theta θ ) holds for surrogates F n \mathcal{F}_n F n with a common η \eta η , and if F n ( m ) → F ( m ) \mathcal{F}_n(m) \to\mathcal{F}(m) F n ( m ) → F ( m ) and D F n ( m ) ( m − m † ) → D F ( m ) ( m − m † ) D\mathcal{F}_n(m)(m-m^\dagger) \to D\mathcal{F}(m)(m-m^\dagger) D F n ( m ) ( m − m † ) → D F ( m ) ( m − m † ) for all m ∈ B ρ C ( m † ) m \in B_\rho^C(m^\dagger) m ∈ B ρ C ( m † ) as n → ∞ n \to\infty n → ∞ , then letting n → ∞ n \to\infty n → ∞ in (TCCθ _\theta θ ) shows that F \mathcal{F} F satisfies (TCC) with the same η \eta η . That is, when the surrogates and their derivatives converge, the surrogate cone condition (TCCθ _\theta θ ) is not weaker than the physical one (TCC) in the limit.
Now, we quantify the approximation errors of the surrogate, which act as additional noise levels, by
ε o p : = sup m ∈ B ρ C ( m † ) ∥ F θ ( m ) − F ( m ) ∥ Y , ε D , C : = sup m ∈ B ρ C ( m † ) ∥ ( D F θ ( m ) − D F ( m ) ) C 1 / 2 ∥ L ( M , Y ) . (16) \varepsilon_{\mathrm{op}} := \sup_{m\in B_\rho^C(m^\dagger)} \lVert\mathcal{F}_\theta(m)-\mathcal{F}(m)\rVert_Y,
\qquad
\varepsilon_{D,C} := \sup_{m\in B_\rho^C(m^\dagger)} \lVert(D\mathcal{F}_\theta(m)-D\mathcal{F}(m))C^{1/2}\rVert_{\mathcal{L}(\mathcal{M},Y)}. \tag*{(16)} ε op := m ∈ B ρ C ( m † ) sup ∥ F θ ( m ) − F ( m ) ∥ Y , ε D , C := m ∈ B ρ C ( m † ) sup ∥( D F θ ( m ) − D F ( m )) C 1/2 ∥ L ( M , Y ) . (16)
Here ε o p \varepsilon_{\mathrm{op}} ε op is the forward error of the surrogate on B ρ C ( m † ) B_\rho^C(m^\dagger) B ρ C ( m † ) , and ε D , C \varepsilon_{D,C} ε D , C is its derivative error weighted by C 1 / 2 C^{1/2} C 1/2 . Both suprema are finite, since F , F θ ∈ C 1 ( U , Y ) \mathcal{F},\mathcal{F}_\theta\in C^1(U,Y) F , F θ ∈ C 1 ( U , Y ) and B ρ C ( m † ) B_\rho^C(m^\dagger) B ρ C ( m † ) is compact by Remark 5.4. Under Assumptions 5.1 and 5.3, F ∈ C 1 ( U , Y ) \mathcal{F} \in C^1(U,Y) F ∈ C 1 ( U , Y ) and B ρ C ( m † ) B_\rho^C(m^\dagger) B ρ C ( m † ) is compact, and hence Corollary 3.3 with T = C 1 / 2 T=C^{1/2} T = C 1/2 provides, for every ε > 0 \varepsilon> 0 ε > 0 , a surrogate of the form (4) with ε o p < ε \varepsilon_{\mathrm{op}} < \varepsilon ε op < ε and ε D , C < ε \varepsilon_{D,C} < \varepsilon ε D , C < ε . According to Corollary 4.6, this can also be realized as a generalised DeepONet or as a member of the generalised Fourier class.
The derivative error also controls the size of the weighted surrogate derivative. With
L C : = sup m ∈ B ρ C ( m † ) ∥ D F ( m ) C 1 / 2 ∥ L ( M , Y ) < ∞ , L_C := \sup_{m\in B_\rho^C(m^\dagger)} \lVert D\mathcal{F}(m)C^{1/2}\rVert_{\mathcal{L}(\mathcal{M},Y)} < \infty, L C := m ∈ B ρ C ( m † ) sup ∥ D F ( m ) C 1/2 ∥ L ( M , Y ) < ∞ ,
the triangle inequality gives
sup m ∈ B ρ C ( m † ) ∥ D F θ ( m ) C 1 / 2 ∥ L ( M , Y ) ≤ L C + ε D , C . (17) \sup_{m\in B_\rho^C(m^\dagger)} \lVert D\mathcal{F}_\theta(m)C^{1/2}\rVert_{\mathcal{L}(\mathcal{M},Y)} \le L_C+\varepsilon_{D,C}. \tag*{(17)} m ∈ B ρ C ( m † ) sup ∥ D F θ ( m ) C 1/2 ∥ L ( M , Y ) ≤ L C + ε D , C . (17)
This will be useful for the choice of the step size ω \omega ω in (11) :
Assumption 5.7 (Parameters of the method). Choose a constant C 0 > 0 C_0 > 0 C 0 > 0 such that sup m ∈ B ρ C ( m † ) ∥ D F θ ( m ) C 1 / 2 ∥ L ( M , Y ) ≤ C 0 \sup_{m\in B_\rho^C(m^\dagger)} \lVert D\mathcal{F}_\theta(m)C^{1/2}\rVert_{\mathcal{L}(\mathcal{M},Y)} \le C_0 sup m ∈ B ρ C ( m † ) ∥ D F θ ( m ) C 1/2 ∥ L ( M , Y ) ≤ C 0 and choose the step size ω \omega ω such that
0 < ω ≤ C 0 − 2 . (18) 0 < \omega\le C_0^{-2}. \tag*{(18)} 0 < ω ≤ C 0 − 2 . (18)
Remark 5.8. Under Assumptions 5.1 and 5.3, the supremum in Assumption 5.7 is finite, so an admissible constant C 0 C_0 C 0 always exists. Any upper bound may be used; by (17), C 0 = 1 + L C C_0 = 1+L_C C 0 = 1 + L C is admissible for every surrogate with ε D , C ≤ 1 \varepsilon_{D,C} \le1 ε D , C ≤ 1 , independently of the particular surrogate.
Lemma 5.9 (Linearisation error of the surrogate). Let Assumptions 5.1, 5.3 and 5.5 hold, and let
γ : = { ρ ε D , C under (TCC) , 0 under (TCC θ ) , (19) \gamma:= \begin{cases}
\rho\varepsilon_{D,C} & \text{under (TCC)},\\
0 & \text{under (TCC}_{\theta}\text{)},
\end{cases} \tag*{(19)} γ := { ρ ε D , C 0 under (TCC) , under (TCC θ ) , (19) where γ = 0 \gamma= 0 γ = 0 may be used if both conditions hold. Then, for every m ∈ B ρ C ( m † ) m \in B_{\rho}^{C}(m^{\dagger}) m ∈ B ρ C ( m † ) ,
∥ D F θ ( m ) ( m − m † ) − ( F θ ( m ) − y δ ) ∥ Y ≤ η ∥ F θ ( m ) − y δ ∥ Y + ( 1 + η ) ( δ + ε o p ) + γ . (20) \left\|DF_{\theta}(m)(m-m^{\dagger})-\left(F_{\theta}(m)-y^{\delta}\right)\right\|_{Y}
\le\eta\left\|F_{\theta}(m)-y^{\delta}\right\|_{Y}+(1+\eta)(\delta+\varepsilon_{\mathrm{op}})+\gamma. \tag*{(20)} D F θ ( m ) ( m − m † ) − ( F θ ( m ) − y δ ) Y ≤ η F θ ( m ) − y δ Y + ( 1 + η ) ( δ + ε op ) + γ . (20)
Proof. Let m ∈ B ρ C ( m † ) m \in B_{\rho}^{C}(m^{\dagger}) m ∈ B ρ C ( m † ) and e : = m − m † e := m-m^{\dagger} e := m − m † , so that e ∈ M C e \in\mathcal{M}_{C} e ∈ M C and ∥ e ∥ M C ≤ ρ \|e\|_{\mathcal{M}_{C}} \le\rho ∥ e ∥ M C ≤ ρ . Moreover,
∥ y δ − F θ ( m † ) ∥ Y ≤ ∥ y δ − y † ∥ Y + ∥ F ( m † ) − F θ ( m † ) ∥ Y ≤ δ + ε o p . \left\|y^{\delta}-F_{\theta}(m^{\dagger})\right\|_{Y}
\le\left\|y^{\delta}-y^{\dagger}\right\|_{Y}
+\left\|\mathcal{F}(m^{\dagger})-F_{\theta}(m^{\dagger})\right\|_{Y}
\le\delta+\varepsilon_{\mathrm{op}}. y δ − F θ ( m † ) Y ≤ y δ − y † Y + F ( m † ) − F θ ( m † ) Y ≤ δ + ε op .
If the surrogate satisfies (TCCθ _{\theta} θ ), define the nonlinear remainder r r r of the surrogate by
D F θ ( m ) e = F θ ( m ) − F θ ( m † ) + r . DF_{\theta}(m)e=F_{\theta}(m)-F_{\theta}(m^{\dagger})+r. D F θ ( m ) e = F θ ( m ) − F θ ( m † ) + r .
Then directly ∥ r ∥ Y ≤ η ∥ F θ ( m ) − F θ ( m † ) ∥ Y \|r\|_{Y}\le\eta\|F_{\theta}(m)-F_{\theta}(m^{\dagger})\|_{Y} ∥ r ∥ Y ≤ η ∥ F θ ( m ) − F θ ( m † ) ∥ Y by (TCCθ _{\theta} θ ), and hence
∥ D F θ ( m ) e − ( F θ ( m ) − y δ ) ∥ Y = ∥ r + ( y δ − F θ ( m † ) ) ∥ Y ≤ η ∥ F θ ( m ) − F θ ( m † ) ∥ Y + ∥ y δ − F θ ( m † ) ∥ Y ≤ η ( ∥ F θ ( m ) − y δ ∥ Y + ∥ y δ − F θ ( m † ) ∥ Y ) + ∥ y δ − F θ ( m † ) ∥ Y ≤ η ∥ F θ ( m ) − y δ ∥ Y + ( 1 + η ) ( δ + ε o p ) . \begin{aligned}
\left\|DF_{\theta}(m)e-\left(F_{\theta}(m)-y^{\delta}\right)\right\|_{Y}
&=\left\|r+\left(y^{\delta}-F_{\theta}(m^{\dagger})\right)\right\|_{Y}\\
&\le\eta\left\|F_{\theta}(m)-F_{\theta}(m^{\dagger})\right\|_{Y}
+\left\|y^{\delta}-F_{\theta}(m^{\dagger})\right\|_{Y}\\
&\le\eta\left(\left\|F_{\theta}(m)-y^{\delta}\right\|_{Y}
+\left\|y^{\delta}-F_{\theta}(m^{\dagger})\right\|_{Y}\right)
+\left\|y^{\delta}-F_{\theta}(m^{\dagger})\right\|_{Y}\\
&\le\eta\left\|F_{\theta}(m)-y^{\delta}\right\|_{Y}
+(1+\eta)(\delta+\varepsilon_{\mathrm{op}}).
\end{aligned} D F θ ( m ) e − ( F θ ( m ) − y δ ) Y = r + ( y δ − F θ ( m † ) ) Y ≤ η F θ ( m ) − F θ ( m † ) Y + y δ − F θ ( m † ) Y ≤ η ( F θ ( m ) − y δ Y + y δ − F θ ( m † ) Y ) + y δ − F θ ( m † ) Y ≤ η F θ ( m ) − y δ Y + ( 1 + η ) ( δ + ε op ) .
If the forward operator satisfies (TCC), substituting F θ = F + ( F θ − F ) F_{\theta}=\mathcal{F}+(F_{\theta}-\mathcal{F}) F θ = F + ( F θ − F ) , D F θ = D F + ( D F θ − D F ) DF_{\theta}=D\mathcal{F}+(DF_{\theta}-D\mathcal{F}) D F θ = D F + ( D F θ − D F ) and y δ = F ( m † ) + ( y δ − y † ) y^{\delta}=\mathcal{F}(m^{\dagger})+(y^{\delta}-y^{\dagger}) y δ = F ( m † ) + ( y δ − y † ) and using the triangle inequality gives
∥ D F θ ( m ) e − ( F θ ( m ) − y δ ) ∥ Y = ∥ ( D F θ ( m ) − D F ( m ) ) e + D F ( m ) e − ( F ( m ) − F ( m † ) ) + ( F ( m ) − F θ ( m ) ) + ( y δ − y † ) ∥ Y ≤ ∥ ( D F θ ( m ) − D F ( m ) ) e ∥ Y + ∥ D F ( m ) e − ( F ( m ) − F ( m † ) ) ∥ Y + ∥ F ( m ) − F θ ( m ) ∥ Y + ∥ y δ − y † ∥ Y ≤ ρ ε D , C + η ∥ F ( m ) − F ( m † ) ∥ Y + ε o p + δ ≤ ρ ε D , C + η ( ∥ F ( m ) − F θ ( m ) ∥ Y + ∥ F θ ( m ) − y δ ∥ Y + ∥ y δ − y † ∥ Y ) + ε o p + δ ≤ η ∥ F θ ( m ) − y δ ∥ Y + ( 1 + η ) ( δ + ε o p ) + ρ ε D , C . \begin{aligned}
\left\|DF_{\theta}(m)e-\left(F_{\theta}(m)-y^{\delta}\right)\right\|_{Y}
&=\left\|(DF_{\theta}(m)-D\mathcal{F}(m))e+D\mathcal{F}(m)e-\left(\mathcal{F}(m)-\mathcal{F}(m^{\dagger})\right)\right.\\
&\qquad\left.+\left(\mathcal{F}(m)-F_{\theta}(m)\right)+\left(y^{\delta}-y^{\dagger}\right)\right\|_{Y}\\
&\le\left\|(DF_{\theta}(m)-D\mathcal{F}(m))e\right\|_{Y}
+\left\|D\mathcal{F}(m)e-\left(\mathcal{F}(m)-\mathcal{F}(m^{\dagger})\right)\right\|_{Y}\\
&\qquad+\left\|\mathcal{F}(m)-F_{\theta}(m)\right\|_{Y}
+\left\|y^{\delta}-y^{\dagger}\right\|_{Y}\\
&\le\rho\varepsilon_{D,C}+\eta\left\|\mathcal{F}(m)-\mathcal{F}(m^{\dagger})\right\|_{Y}
+\varepsilon_{\mathrm{op}}+\delta\\
&\le\rho\varepsilon_{D,C}
+\eta\left(\left\|\mathcal{F}(m)-F_{\theta}(m)\right\|_{Y}
+\left\|F_{\theta}(m)-y^{\delta}\right\|_{Y}
+\left\|y^{\delta}-y^{\dagger}\right\|_{Y}\right)
+\varepsilon_{\mathrm{op}}+\delta\\
&\le\eta\left\|F_{\theta}(m)-y^{\delta}\right\|_{Y}
+(1+\eta)(\delta+\varepsilon_{\mathrm{op}})
+\rho\varepsilon_{D,C}.
\end{aligned} D F θ ( m ) e − ( F θ ( m ) − y δ ) Y = ( D F θ ( m ) − D F ( m )) e + D F ( m ) e − ( F ( m ) − F ( m † ) ) + ( F ( m ) − F θ ( m ) ) + ( y δ − y † ) Y ≤ ∥ ( D F θ ( m ) − D F ( m )) e ∥ Y + D F ( m ) e − ( F ( m ) − F ( m † ) ) Y + ∥ F ( m ) − F θ ( m ) ∥ Y + y δ − y † Y ≤ ρ ε D , C + η F ( m ) − F ( m † ) Y + ε op + δ ≤ ρ ε D , C + η ( ∥ F ( m ) − F θ ( m ) ∥ Y + F θ ( m ) − y δ Y + y δ − y † Y ) + ε op + δ ≤ η F θ ( m ) − y δ Y + ( 1 + η ) ( δ + ε op ) + ρ ε D , C .
Here the first term was bounded by the weighted derivative error,
∥ ( D F θ ( m ) − D F ( m ) ) e ∥ Y = ∥ ( D F θ ( m ) − D F ( m ) ) C 1 / 2 C − 1 / 2 e ∥ Y ≤ ε D , C ∥ e ∥ M C ≤ ρ ε D , C , \left\|(DF_{\theta}(m)-D\mathcal{F}(m))e\right\|_{Y}
=\left\|(DF_{\theta}(m)-D\mathcal{F}(m))C^{1/2}C^{-1/2}e\right\|_{Y}
\le\varepsilon_{D,C}\|e\|_{\mathcal{M}_{C}}
\le\rho\varepsilon_{D,C}, ∥ ( D F θ ( m ) − D F ( m )) e ∥ Y = ( D F θ ( m ) − D F ( m )) C 1/2 C − 1/2 e Y ≤ ε D , C ∥ e ∥ M C ≤ ρ ε D , C ,
the second by (TCC), and the last two by the definition of ε o p \varepsilon_{\mathrm{op}} ε op and by Assumption 5.1. □
Lemma 5.9 is the only consequence of (TCC) or (TCCθ _{\theta} θ ) used in the sequel, so that both cases are treated simultaneously from now on.
Stopping and monotonicity
Let us now derive an appropriate discrepancy principle for stopping the iteration (11). Afterwards, we will prove that all iterates up to the corresponding stopping index lie in B ρ C ( m † ) B_\rho^C(m^\dagger) B ρ C ( m † ) and are hence well-defined.
As a rationale for the stopping criterion, note that
∥ F θ ( m † ) − y δ ∥ Y ≤ ∥ F θ ( m † ) − F ( m † ) ∥ Y + ∥ y † − y δ ∥ Y ≤ ε o p + δ . \lVert\mathcal{F}_\theta(m^\dagger)-y^\delta\rVert_Y \le\lVert\mathcal{F}_\theta(m^\dagger)-\mathcal{F}(m^\dagger)\rVert_Y+\lVert y^\dagger-y^\delta\rVert_Y \le\varepsilon_{\mathrm{op}}+\delta. ∥ F θ ( m † ) − y δ ∥ Y ≤ ∥ F θ ( m † ) − F ( m † ) ∥ Y + ∥ y † − y δ ∥ Y ≤ ε op + δ .
From this point of view, the effective noise level in the surrogate equation F θ ( m ) = y δ \mathcal{F}_\theta(m)=y^\delta F θ ( m ) = y δ is ε o p + δ \varepsilon_{\mathrm{op}}+\delta ε op + δ . However, the approximation of D F D\mathcal{F} D F may cause further errors, as seen in Lemma 5.9. Therefore, we define
Δ : = δ + ε o p + γ 1 + η (21) \Delta:= \delta+\varepsilon_{\mathrm{op}}+\frac{\gamma}{1+\eta} \tag*{(21)} Δ := δ + ε op + 1 + η γ (21)
with γ \gamma γ as in Lemma 5.9, and stop the iteration using the discrepancy principle
k ∗ : = inf { k ≥ 0 : ∥ F θ ( m k ) − y δ ∥ Y ≤ τ Δ } (22) k_*:=\inf\{k\ge0:\ \lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y\le\tau\Delta\} \tag*{(22)} k ∗ := inf { k ≥ 0 : ∥ F θ ( m k ) − y δ ∥ Y ≤ τ Δ } (22)
with the convention inf ∅ = ∞ \inf\varnothing=\infty inf ∅ = ∞ and a parameter τ > 0 \tau>0 τ > 0 satisfying
τ > 2 ( 1 + η ) 1 − 2 η . (23) \tau>\frac{2(1+\eta)}{1-2\eta}. \tag*{(23)} τ > 1 − 2 η 2 ( 1 + η ) . (23)
The lower bound on τ \tau τ in (23) is the usual condition of the discrepancy principle for nonlinear (exact) Landweber iteration (see, e.g., Hanke et al., [15 ] ). For the discrepancy principle to work, we have to assume that Δ > 0 \Delta>0 Δ > 0 . Note that Δ = 0 \Delta=0 Δ = 0 corresponds to exact data and a surrogate that coincides with F \mathcal{F} F on B ρ C ( m † ) B_\rho^C(m^\dagger) B ρ C ( m † ) , i.e. (11) reduces to the preconditioned Landweber iteration with exact forward operator and exact data, which is not stopped. Moreover, the approximation error ε D , C \varepsilon_{D,C} ε D , C of the derivative enters the discrepancy principle only through γ \gamma γ , i.e. only if (TCC) is used; under (TCCθ _\theta θ ), Δ = δ + ε o p \Delta=\delta+\varepsilon_{\mathrm{op}} Δ = δ + ε op .
In the following, we denote the error of the k k k -th iterate by e k : = m k − m † e_k:=m_k-m^\dagger e k := m k − m † . Following the standard analysis of Landweber iteration, we first show that a single step from an iterate in B ρ C ( m † ) B_\rho^C(m^\dagger) B ρ C ( m † ) decreases the error in the norm of M C \mathcal{M}_C M C as long as the discrepancy principle has not stopped the iteration. Since Lemma 5.9 and the step-size condition (18) are only available on B ρ C ( m † ) B_\rho^C(m^\dagger) B ρ C ( m † ) , we then show by induction that the iteration does not leave B ρ C ( m † ) B_\rho^C(m^\dagger) B ρ C ( m † ) before it is stopped.
Lemma 5.10 (Recursive error bound, monotonicity). Let Assumptions 5.1–5.7 hold, let Δ \Delta Δ be given by (21) and let τ \tau τ satisfy (22b). If m k ∈ B ρ C ( m † ) m_k\in B_\rho^C(m^\dagger) m k ∈ B ρ C ( m † ) and ∥ F θ ( m k ) − y δ ∥ Y > τ Δ \lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y>\tau\Delta ∥ F θ ( m k ) − y δ ∥ Y > τ Δ , then m k + 1 m_{k+1} m k + 1 from (11) satisfies e k + 1 ∈ M C e_{k+1}\in\mathcal{M}_C e k + 1 ∈ M C and
∥ e k + 1 ∥ M C 2 − ∥ e k ∥ M C 2 ≤ − c τ ω ∥ F θ ( m k ) − y δ ∥ Y 2 < 0 (24) \lVert e_{k+1}\rVert_{\mathcal{M}_C}^2-\lVert e_k\rVert_{\mathcal{M}_C}^2\le-c_\tau\omega\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y^2<0 \tag*{(24)} ∥ e k + 1 ∥ M C 2 − ∥ e k ∥ M C 2 ≤ − c τ ω ∥ F θ ( m k ) − y δ ∥ Y 2 < 0 (24) with c τ : = 1 − 2 η − 2 ( 1 + η ) / τ > 0 c_\tau:=1-2\eta-2(1+\eta)/\tau>0 c τ := 1 − 2 η − 2 ( 1 + η ) / τ > 0 .
Proof. Since m k ∈ B ρ C ( m † ) ⊆ U m_k\in B_\rho^C(m^\dagger)\subseteq U m k ∈ B ρ C ( m † ) ⊆ U , the iterate m k + 1 m_{k+1} m k + 1 is defined, and e k ∈ M C e_k\in\mathcal{M}_C e k ∈ M C . Moreover, C z = C 1 / 2 ( C 1 / 2 z ) ∈ ran ( C 1 / 2 ) = M C Cz=C^{1/2}(C^{1/2}z)\in\operatorname{ran}(C^{1/2})=\mathcal{M}_C C z = C 1/2 ( C 1/2 z ) ∈ ran ( C 1/2 ) = M C for every z ∈ M z\in\mathcal{M} z ∈ M , so that also
e k + 1 = e k − ω C D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) ∈ M C . e_{k+1}=e_k-\omega C D\mathcal{F}_\theta(m_k)^*(\mathcal{F}_\theta(m_k)-y^\delta)\in\mathcal{M}_C. e k + 1 = e k − ω C D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) ∈ M C .
Expanding the squared error in the norm of M C \mathcal{M}_{C} M C gives
∥ e k + 1 ∥ M C 2 = ∥ e k ∥ M C 2 − 2 ω ⟨ e k , C D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) ⟩ M C + ω 2 ∥ C D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) ∥ M C 2 . (25) \lVert e_{k+1}\rVert_{\mathcal{M}_{C}}^{2} = \lVert e_k\rVert_{\mathcal{M}_{C}}^{2} - 2\omega\langle e_k,CD F_\theta(m_k)^*(\mathcal{F}_\theta(m_k)-y^\delta)\rangle_{\mathcal{M}_{C}} + \omega^2\lVert CD F_\theta(m_k)^*(\mathcal{F}_\theta(m_k)-y^\delta)\rVert_{\mathcal{M}_{C}}^{2}. \tag*{(25)} ∥ e k + 1 ∥ M C 2 = ∥ e k ∥ M C 2 − 2 ω ⟨ e k , C D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) ⟩ M C + ω 2 ∥ C D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) ∥ M C 2 . (25)
As noted in Remark 5.4, C D F θ ( m k ) ∗ CD F_\theta(m_k)^* C D F θ ( m k ) ∗ is the adjoint of D F θ ( m k ) ∣ M C D F_\theta(m_k)|_{\mathcal{M}_{C}} D F θ ( m k ) ∣ M C in M C \mathcal{M}_{C} M C . For the norm, we use C − 1 / 2 C = C 1 / 2 C^{-1/2}C=C^{1/2} C − 1/2 C = C 1/2 and, since C 1 / 2 C^{1/2} C 1/2 is self-adjoint,
∥ C 1 / 2 D F θ ( m k ) ∗ ∥ L ( Y , M ) = ∥ ( D F θ ( m k ) C 1 / 2 ) ∗ ∥ L ( Y , M ) = ∥ D F θ ( m k ) C 1 / 2 ∥ L ( M , Y ) . \lVert C^{1/2}D F_\theta(m_k)^*\rVert_{\mathcal{L}(Y,\mathcal{M})}=\lVert(D F_\theta(m_k)C^{1/2})^*\rVert_{\mathcal{L}(Y,\mathcal{M})}=\lVert D F_\theta(m_k)C^{1/2}\rVert_{\mathcal{L}(\mathcal{M},Y)}. ∥ C 1/2 D F θ ( m k ) ∗ ∥ L ( Y , M ) = ∥( D F θ ( m k ) C 1/2 ) ∗ ∥ L ( Y , M ) = ∥ D F θ ( m k ) C 1/2 ∥ L ( M , Y ) .
Hence
⟨ e k , C D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) ⟩ M C = ⟨ D F θ ( m k ) e k , F θ ( m k ) − y δ ⟩ Y , \langle e_k,CD F_\theta(m_k)^*(\mathcal{F}_\theta(m_k)-y^\delta)\rangle_{\mathcal{M}_{C}}=\langle D F_\theta(m_k)e_k,\mathcal{F}_\theta(m_k)-y^\delta\rangle_Y, ⟨ e k , C D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) ⟩ M C = ⟨ D F θ ( m k ) e k , F θ ( m k ) − y δ ⟩ Y ,
∥ C D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) ∥ M C = ∥ C 1 / 2 D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) ∥ M ≤ ∥ D F θ ( m k ) C 1 / 2 ∥ L ( M , Y ) ∥ F θ ( m k ) − y δ ∥ Y ≤ C 0 ∥ F θ ( m k ) − y δ ∥ Y . \begin{aligned}
\lVert CD F_\theta(m_k)^*(\mathcal{F}_\theta(m_k)-y^\delta)\rVert_{\mathcal{M}_{C}}
&=\lVert C^{1/2}D F_\theta(m_k)^*(\mathcal{F}_\theta(m_k)-y^\delta)\rVert_{\mathcal{M}}\\
&\le\lVert D F_\theta(m_k)C^{1/2}\rVert_{\mathcal{L}(\mathcal{M},Y)}\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y\\
&\le C_0\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y.
\end{aligned} ∥ C D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) ∥ M C = ∥ C 1/2 D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) ∥ M ≤ ∥ D F θ ( m k ) C 1/2 ∥ L ( M , Y ) ∥ F θ ( m k ) − y δ ∥ Y ≤ C 0 ∥ F θ ( m k ) − y δ ∥ Y .
where the last step uses Assumption 5.7 and m k ∈ B ρ C ( m † ) m_k\in B_\rho^C(m^\dagger) m k ∈ B ρ C ( m † ) . Together with ω C 0 2 ≤ 1 \omega C_0^2\le1 ω C 0 2 ≤ 1 from (18) , we obtain
∥ e k + 1 ∥ M C 2 − ∥ e k ∥ M C 2 ≤ − 2 ω ⟨ D F θ ( m k ) e k , F θ ( m k ) − y δ ⟩ Y + ω ∥ F θ ( m k ) − y δ ∥ Y 2 . \lVert e_{k+1}\rVert_{\mathcal{M}_{C}}^2-\lVert e_k\rVert_{\mathcal{M}_{C}}^2\le-2\omega\langle D F_\theta(m_k)e_k,\mathcal{F}_\theta(m_k)-y^\delta\rangle_Y+\omega\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y^2. ∥ e k + 1 ∥ M C 2 − ∥ e k ∥ M C 2 ≤ − 2 ω ⟨ D F θ ( m k ) e k , F θ ( m k ) − y δ ⟩ Y + ω ∥ F θ ( m k ) − y δ ∥ Y 2 .
Furthermore, the Cauchy–Schwarz inequality and Lemma 5.9 with m = m k m=m_k m = m k show that the inner-product term satisfies
− 2 ω ⟨ D F θ ( m k ) e k , F θ ( m k ) − y δ ⟩ Y = − 2 ω ∥ F θ ( m k ) − y δ ∥ Y 2 − 2 ω ⟨ D F θ ( m k ) e k − ( F θ ( m k ) − y δ ) , F θ ( m k ) − y δ ⟩ Y ≤ − 2 ω ∥ F θ ( m k ) − y δ ∥ Y 2 + 2 ω ∥ D F θ ( m k ) e k − ( F θ ( m k ) − y δ ) ∥ Y ∥ F θ ( m k ) − y δ ∥ Y ≤ − 2 ω ∥ F θ ( m k ) − y δ ∥ Y 2 + 2 ω ( η ∥ F θ ( m k ) − y δ ∥ Y + ( 1 + η ) ( δ + ε o p ) + γ ) ∥ F θ ( m k ) − y δ ∥ Y . \begin{aligned}
-2\omega\langle D F_\theta(m_k)e_k,\mathcal{F}_\theta(m_k)-y^\delta\rangle_Y
&=-2\omega\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y^2\\
&\quad-2\omega\langle D F_\theta(m_k)e_k-(\mathcal{F}_\theta(m_k)-y^\delta),\mathcal{F}_\theta(m_k)-y^\delta\rangle_Y\\
&\le-2\omega\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y^2\\
&\quad+2\omega\lVert D F_\theta(m_k)e_k-(\mathcal{F}_\theta(m_k)-y^\delta)\rVert_Y\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y\\
&\le-2\omega\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y^2\\
&\quad+2\omega\bigl(\eta\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y+(1+\eta)(\delta+\varepsilon_{\mathrm{op}})+\gamma\bigr)\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y.
\end{aligned} − 2 ω ⟨ D F θ ( m k ) e k , F θ ( m k ) − y δ ⟩ Y = − 2 ω ∥ F θ ( m k ) − y δ ∥ Y 2 − 2 ω ⟨ D F θ ( m k ) e k − ( F θ ( m k ) − y δ ) , F θ ( m k ) − y δ ⟩ Y ≤ − 2 ω ∥ F θ ( m k ) − y δ ∥ Y 2 + 2 ω ∥ D F θ ( m k ) e k − ( F θ ( m k ) − y δ ) ∥ Y ∥ F θ ( m k ) − y δ ∥ Y ≤ − 2 ω ∥ F θ ( m k ) − y δ ∥ Y 2 + 2 ω ( η ∥ F θ ( m k ) − y δ ∥ Y + ( 1 + η ) ( δ + ε op ) + γ ) ∥ F θ ( m k ) − y δ ∥ Y .
Substituting this bound and using ( 1 + η ) ( δ + ε o p ) + γ = ( 1 + η ) Δ (1+\eta)(\delta+\varepsilon_{\mathrm{op}})+\gamma=(1+\eta)\Delta ( 1 + η ) ( δ + ε op ) + γ = ( 1 + η ) Δ by (21) yields
∥ e k + 1 ∥ M C 2 − ∥ e k ∥ M C 2 ≤ − ω ∥ F θ ( m k ) − y δ ∥ Y 2 + 2 ω ( η ∥ F θ ( m k ) − y δ ∥ Y + ( 1 + η ) Δ ) ∥ F θ ( m k ) − y δ ∥ Y = − ω ∥ F θ ( m k ) − y δ ∥ Y [ ( 1 − 2 η ) ∥ F θ ( m k ) − y δ ∥ Y − 2 ( 1 + η ) Δ ] . \begin{aligned}
\lVert e_{k+1}\rVert_{\mathcal{M}_{C}}^2-\lVert e_k\rVert_{\mathcal{M}_{C}}^2
&\le-\omega\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y^2\\
&\quad+2\omega\bigl(\eta\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y+(1+\eta)\Delta\bigr)\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y\\
&=-\omega\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y\bigl[(1-2\eta)\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y-2(1+\eta)\Delta\bigr].
\end{aligned} ∥ e k + 1 ∥ M C 2 − ∥ e k ∥ M C 2 ≤ − ω ∥ F θ ( m k ) − y δ ∥ Y 2 + 2 ω ( η ∥ F θ ( m k ) − y δ ∥ Y + ( 1 + η ) Δ ) ∥ F θ ( m k ) − y δ ∥ Y = − ω ∥ F θ ( m k ) − y δ ∥ Y [ ( 1 − 2 η ) ∥ F θ ( m k ) − y δ ∥ Y − 2 ( 1 + η ) Δ ] .
By assumption, Δ < 1 τ ∥ F θ ( m k ) − y δ ∥ Y \Delta<\frac{1}{\tau}\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y Δ < τ 1 ∥ F θ ( m k ) − y δ ∥ Y , and hence
( 1 − 2 η ) ∥ F θ ( m k ) − y δ ∥ Y − 2 ( 1 + η ) Δ > ( 1 − 2 η − 2 ( 1 + η ) τ ) ∥ F θ ( m k ) − y δ ∥ Y = c τ ∥ F θ ( m k ) − y δ ∥ Y , \begin{aligned}
(1-2\eta)\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y-2(1+\eta)\Delta>\left(1-2\eta-\frac{2(1+\eta)}{\tau}\right)\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y \\
&=c_\tau\lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y,
\end{aligned} ( 1 − 2 η ) ∥ F θ ( m k ) − y δ ∥ Y − 2 ( 1 + η ) Δ > ( 1 − 2 η − τ 2 ( 1 + η ) ) ∥ F θ ( m k ) − y δ ∥ Y = c τ ∥ F θ ( m k ) − y δ ∥ Y ,
where c τ > 0 c_\tau>0 c τ > 0 is equivalent to (23) , since 1 − 2 η > 0 1-2\eta>0 1 − 2 η > 0 . Finally, γ ≥ 0 \gamma\ge0 γ ≥ 0 implies Δ ≥ 0 \Delta\ge0 Δ ≥ 0 , so that ∥ F θ ( m k ) − y δ ∥ Y > τ Δ ≥ 0 \lVert\mathcal{F}_\theta(m_k)-y^\delta\rVert_Y>\tau\Delta\ge0 ∥ F θ ( m k ) − y δ ∥ Y > τ Δ ≥ 0 , which yields (24) including the strict inequality.
Theorem 5.11 (Well-definedness and finite stopping). Let Assumptions 5.1–5.7 hold and let k ∗ k_\ast k ∗ be given by the discrepancy principle (22). Then the iterates of (11) satisfy
m k ∈ B ρ C ( m † ) ⊆ U for all integers 0 ≤ k ≤ k ∗ , (26) m_k \in B_\rho^C(m^\dagger) \subseteq U \qquad\text{for all integers } 0 \leq k \leq k_\ast, \tag*{(26)} m k ∈ B ρ C ( m † ) ⊆ U for all integers 0 ≤ k ≤ k ∗ , (26) i.e. the iteration is well defined up to the stopping index, and (23) holds for all k < k ∗ k < k_\ast k < k ∗ . If, in addition, Δ > 0 \Delta> 0 Δ > 0 , then the stopping index is finite.
Proof. We argue by induction. By Assumption 5.3, m 0 ∈ B ρ C ( m † ) m_0 \in B_\rho^C(m^\dagger) m 0 ∈ B ρ C ( m † ) . Let k < k ∗ k < k_\ast k < k ∗ with m k ∈ B ρ C ( m † ) m_k \in B_\rho^C(m^\dagger) m k ∈ B ρ C ( m † ) . By (22a), ∥ F θ ( m k ) − y δ ∥ Y > τ Δ \lVert F_\theta(m_k) - y^\delta\rVert_Y > \tau\Delta ∥ F θ ( m k ) − y δ ∥ Y > τ Δ , so Lemma 5.10 applies and yields e k + 1 ∈ M C e_{k+1} \in M_C e k + 1 ∈ M C with ∥ e k + 1 ∥ M C < ∥ e k ∥ M C ≤ ρ \lVert e_{k+1}\rVert_{M_C} < \lVert e_k\rVert_{M_C} \leq\rho ∥ e k + 1 ∥ M C < ∥ e k ∥ M C ≤ ρ , that is, m k + 1 ∈ B ρ C ( m † ) m_{k+1} \in B_\rho^C(m^\dagger) m k + 1 ∈ B ρ C ( m † ) . This proves (25), and Lemma 5.10 gives (23) for every k < k ∗ k < k_\ast k < k ∗ .
Now let Δ > 0 \Delta> 0 Δ > 0 and let N ≥ 1 N \geq1 N ≥ 1 be an integer with N ≤ k ∗ N \leq k_\ast N ≤ k ∗ . Summing (23) from k = 0 k = 0 k = 0 to k = N − 1 k = N - 1 k = N − 1 gives
c τ ω ∑ k = 0 N − 1 ∥ F θ ( m k ) − y δ ∥ Y 2 ≤ ∥ e 0 ∥ M C 2 − ∥ e N ∥ M C 2 ≤ ∥ e 0 ∥ M C 2 . c_{\tau}\omega\sum_{k=0}^{N-1} \lVert F_\theta(m_k) - y^\delta\rVert_Y^2 \leq\lVert e_0\rVert_{M_C}^2 - \lVert e_N\rVert_{M_C}^2 \leq\lVert e_0\rVert_{M_C}^2. c τ ω k = 0 ∑ N − 1 ∥ F θ ( m k ) − y δ ∥ Y 2 ≤ ∥ e 0 ∥ M C 2 − ∥ e N ∥ M C 2 ≤ ∥ e 0 ∥ M C 2 . Since every residual before stopping satisfies ∥ F θ ( m k ) − y δ ∥ Y 2 > τ 2 Δ 2 \lVert F_\theta(m_k) - y^\delta\rVert_Y^2 > \tau^2\Delta^2 ∥ F θ ( m k ) − y δ ∥ Y 2 > τ 2 Δ 2 by (22a),
N c τ ω τ 2 Δ 2 < ∥ e 0 ∥ M C 2 . Nc_{\tau}\omega\tau^2\Delta^2 < \lVert e_0\rVert_{M_C}^2. N c τ ω τ 2 Δ 2 < ∥ e 0 ∥ M C 2 . Hence every integer 1 ≤ N ≤ k ∗ 1 \leq N \leq k_\ast 1 ≤ N ≤ k ∗ is bounded by ∥ e 0 ∥ M C 2 / ( c τ ω τ 2 Δ 2 ) \lVert e_0\rVert_{M_C}^2/(c_{\tau}\omega\tau^2\Delta^2) ∥ e 0 ∥ M C 2 / ( c τ ω τ 2 Δ 2 ) , which proves finiteness of k ∗ k_\ast k ∗ .
Remark 5.12 (Variable step sizes). The step size enters the proof of Lemma 5.10 only through
ω ∥ C 1 / 2 D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) ∥ M 2 ≤ ∥ F θ ( m k ) − y δ ∥ Y 2 , \omega\lVert C^{1/2}D F_\theta(m_k)^*(F_\theta(m_k)-y^\delta)\rVert_M^2 \leq\lVert F_\theta(m_k)-y^\delta\rVert_Y^2, ω ∥ C 1/2 D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) ∥ M 2 ≤ ∥ F θ ( m k ) − y δ ∥ Y 2 ,
which is
ω 2 ∥ C D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) ∥ M C 2 ≤ ω ∥ F θ ( m k ) − y δ ∥ Y 2 , \omega^2\lVert C D F_\theta(m_k)^*(F_\theta(m_k)-y^\delta)\rVert_{M_C}^2 \leq\omega\lVert F_\theta(m_k)-y^\delta\rVert_Y^2, ω 2 ∥ C D F θ ( m k ) ∗ ( F θ ( m k ) − y δ ) ∥ M C 2 ≤ ω ∥ F θ ( m k ) − y δ ∥ Y 2 ,
and, for finite stopping, through a positive lower bound. Lemma 5.10 and Theorem 5.11 therefore remain valid for variable step sizes ω k ∈ [ ω min , C 0 − 2 ] \omega_k \in[\omega_{\min}, C_0^{-2}] ω k ∈ [ ω m i n , C 0 − 2 ] with some fixed ω min > 0 \omega_{\min} > 0 ω m i n > 0 .
Regularisation convergence as the noise and surrogate errors vanish
Theorem 5.11 shows that, for a fixed noise level and a fixed surrogate, the iteration is well defined up to the stopping index, the error decreases monotonically before stopping, and the stopping index is finite. We now investigate the limiting behaviour of the stopped iterates as both the noise level and the surrogate errors tend to zero. To this end, we consider sequences of data and surrogates.
Assumption 5.13 (Sequences of data and surrogates). For every n ∈ N n \in\mathbb{N} n ∈ N , data y δ n y^{\delta_n} y δ n and a surrogate F n : = F θ n \mathcal{F}_n := \mathcal{F}_{\theta_n} F n := F θ n are given such that Assumptions 5.1–5.7 hold for ( F n , y δ n ) (\mathcal{F}_n, y^{\delta_n}) ( F n , y δ n ) with noise level δ n \delta_n δ n , step size ω n \omega_n ω n and constant C 0 ( n ) C_0^{(n)} C 0 ( n ) , where C C C , m † m^\dagger m † , ρ \rho ρ , m 0 m_0 m 0 and η \eta η , and hence B ρ C ( m † ) B_\rho^C(m^\dagger) B ρ C ( m † ) , are the same for all n n n . For each n n n , either (TCC) or (TCCθ _\theta θ ) may be used.
Remark 5.14. Apart from C C C , m † m^\dagger m † , ρ \rho ρ and m 0 m_0 m 0 , only η \eta η has to be the same for all n n n . In particular, the step sizes ω n \omega_n ω n and the constants C 0 ( n ) C_0^{(n)} C 0 ( n ) may change with n n n . Under (TCC), a common step size can nevertheless be chosen for all surrogates with ε D , C ≤ 1 \varepsilon_{D,C} \le1 ε D , C ≤ 1 , cf. Remark 5.8.
For each n n n , we denote by ε o p ( n ) \varepsilon_{\mathrm{op}}^{(n)} ε op ( n ) , ε D , C ( n ) \varepsilon_{D,C}^{(n)} ε D , C ( n ) , γ n \gamma_n γ n and Δ n \Delta_n Δ n the quantities (16), (19) and (21) for F n \mathcal{F}_n F n and y δ n y^{\delta_n} y δ n , by m k ( n ) m_k^{(n)} m k ( n ) the iterates of (11) with F n \mathcal{F}_n F n , y δ n y^{\delta_n} y δ n and ω n \omega_n ω n , and by m ^ n : = m k n ( n ) \widehat{m}_n := m_{k_n}^{(n)} m n := m k n ( n ) the iterate stopped by the discrepancy principle (22) at index k n k_n k n .
Theorem 5.15 (Regularisation property of PSLR). Let Assumption 5.13 hold, let the iterations be stopped by the discrepancy principle (22) with a common τ \tau τ satisfying (22b), and assume Δ n > 0 \Delta_n > 0 Δ n > 0 for all $n as well as
δ n → 0 , ε o p ( n ) → 0 , γ n → 0. \delta_n \to0, \qquad\varepsilon_{\mathrm{op}}^{(n)} \to0, \qquad\gamma_n \to0. δ n → 0 , ε op ( n ) → 0 , γ n → 0. Then every subsequence of ( m ^ n ) (\widehat{m}_n) ( m n ) has a further subsequence ( m ^ n j ) (\widehat{m}_{n_j}) ( m n j ) for which there is a solution m ‾ ∈ B ρ C ( m † ) \overline{m} \in B_\rho^C(m^\dagger) m ∈ B ρ C ( m † ) of (10) such that
m ^ n j → m ‾ in M \widehat{m}_{n_j} \to\overline{m} \quad\text{in } \mathcal{M} m n j → m in M If m † m^\dagger m † is the only solution of F ( m ) = y † \mathcal{F}(m) = y^\dagger F ( m ) = y † in B ρ C ( m † ) B_\rho^C(m^\dagger) B ρ C ( m † ) , then the whole sequence converges, i.e.
m ^ n → m † in M . \widehat{m}_n \to m^\dagger\quad\text{in } \mathcal{M}. m n → m † in M . Consequently, the PSLR (11) together with the discrepancy principle (22) is a regularisation method with respect to the combined data and surrogate errors.
Proof. For every n n n , Theorem 5.11 ensures that k n < ∞ k_n < \infty k n < ∞ and that all iterates up to and including k n k_n k n lie in B ρ C ( m † ) B_\rho^C(m^\dagger) B ρ C ( m † ) ; in particular, m ^ n ∈ B ρ C ( m † ) \widehat{m}_n \in B_\rho^C(m^\dagger) m n ∈ B ρ C ( m † ) . At the stopping index, the discrepancy principle (22a) gives
∥ F n ( m ^ n ) − y δ n ∥ Y ≤ τ Δ n . \lVert\mathcal{F}_n(\widehat{m}_n) - y^{\delta_n} \rVert_Y \le\tau\Delta_n. ∥ F n ( m n ) − y δ n ∥ Y ≤ τ Δ n .
The error of the stopped iterate with respect to the exact forward equation can be estimated by inserting the surrogate prediction and the noisy datum:
∥ F ( m ^ n ) − y † ∥ Y ≤ ∥ F ( m ^ n ) − F n ( m ^ n ) ∥ Y + ∥ F n ( m ^ n ) − y δ n ∥ Y + ∥ y δ n − y † ∥ Y ≤ ε o p ( n ) + τ Δ n + δ n = ( 1 + τ ) ( δ n + ε o p ( n ) ) + τ γ n 1 + η . (27) \begin{aligned}
\lVert\mathcal{F}(\widehat{m}_n) - y^\dagger\rVert_Y
&\le\lVert\mathcal{F}(\widehat{m}_n) - \mathcal{F}_n(\widehat{m}_n) \rVert_Y + \lVert\mathcal{F}_n(\widehat{m}_n) - y^{\delta_n} \rVert_Y + \lVert y^{\delta_n} - y^\dagger\rVert_Y \\
&\le\varepsilon_{\mathrm{op}}^{(n)} + \tau\Delta_n + \delta_n
= (1+\tau)(\delta_n+\varepsilon_{\mathrm{op}}^{(n)}) + \frac{\tau\gamma_n}{1+\eta}. \tag*{(27)}
\end{aligned} ∥ F ( m n ) − y † ∥ Y ≤ ∥ F ( m n ) − F n ( m n ) ∥ Y + ∥ F n ( m n ) − y δ n ∥ Y + ∥ y δ n − y † ∥ Y ≤ ε op ( n ) + τ Δ n + δ n = ( 1 + τ ) ( δ n + ε op ( n ) ) + 1 + η τ γ n . (27)
where the first term is bounded by ε o p ( n ) \varepsilon_{\mathrm{op}}^{(n)} ε op ( n ) because m ^ n ∈ B ρ C ( m † ) \widehat{m}_n \in B_\rho^C(m^\dagger) m n ∈ B ρ C ( m † ) , and the last one by Assumption 5.1. Since δ n → 0 \delta_n \to0 δ n → 0 , ε o p ( n ) → 0 \varepsilon_{\mathrm{op}}^{(n)} \to0 ε op ( n ) → 0 and γ n → 0 \gamma_n \to0 γ n → 0 , estimate (27) shows that ∥ F ( m ^ n ) − y † ∥ Y → 0 \lVert\mathcal{F}(\widehat{m}_n) - y^\dagger\rVert_Y \to0 ∥ F ( m n ) − y † ∥ Y → 0 . Thus, although the discrepancy principle is formulated using the learned surrogate F n \mathcal{F}_n F n and the noisy data y δ n y^{\delta_n} y δ n , the corresponding stopped iterates asymptotically satisfy the exact forward equation.
It remains to relate this vanishing residual to convergence of the unknowns. Since every m ^ n \widehat{m}_n m n belongs to B ρ C ( m † ) B_\rho^C(m^\dagger) B ρ C ( m † ) , the sequence ( m ^ n − m † ) (\widehat{m}_n - m^\dagger) ( m n − m † ) is bounded by ρ \rho ρ in the Hilbert space M C \mathcal{M}_C M C . Consider an arbitrary subsequence of ( m ^ n ) (\widehat{m}_n) ( m n ) . It possesses a further subsequence, still denoted for convenience by ( m ^ n j ) (\widehat{m}_{n_j}) ( m n j ) , such that m ^ n j − m † ⇀ m ‾ − m † \widehat{m}_{n_j} - m^\dagger\rightharpoonup\overline{m} - m^\dagger m n j − m † ⇀ m − m † in M C \mathcal{M}_C M C for some m ‾ ∈ m † + M C \overline{m} \in m^\dagger+ \mathcal{M}_C m ∈ m † + M C . By weak lower semicontinuity of the norm,
∥ m ‾ − m † ∥ M C ≤ lim inf j → ∞ ∥ m ^ n j − m † ∥ M C ≤ ρ , \lVert\overline{m} - m^\dagger\rVert_{\mathcal{M}_C} \le\liminf_{j \to\infty} \lVert\widehat{m}_{n_j} - m^\dagger\rVert_{\mathcal{M}_C} \le\rho, ∥ m − m † ∥ M C ≤ j → ∞ lim inf ∥ m n j − m † ∥ M C ≤ ρ ,
that is, m ˉ ∈ B ρ C ( m † ) \bar{m} \in B_{\rho}^{C}(m^{\dagger}) m ˉ ∈ B ρ C ( m † ) . Since B ρ C ( m † ) B_{\rho}^{C}(m^{\dagger}) B ρ C ( m † ) is compact in M \mathcal{M} M (Remark 5.4), the embedding M C ↪ M \mathcal{M}_{C} \hookrightarrow\mathcal{M} M C ↪ M is compact and hence maps weakly convergent sequences to strongly convergent ones, so that m ^ n j → m ˉ \widehat{m}_{n_j} \to\bar{m} m n j → m ˉ in M \mathcal{M} M . The assumption F ∈ C 1 ( U , Y ) \mathcal{F} \in C^{1}(\mathcal{U},\mathcal{Y}) F ∈ C 1 ( U , Y ) implies continuity of F \mathcal{F} F , and therefore F ( m ^ n j ) → F ( m ˉ ) \mathcal{F}(\widehat{m}_{n_j}) \to\mathcal{F}(\bar{m}) F ( m n j ) → F ( m ˉ ) . On the other hand, (26) gives F ( m ^ n j ) → y † \mathcal{F}(\widehat{m}_{n_j}) \to y^{\dagger} F ( m n j ) → y † , i.e. m ˉ ∈ B ρ C ( m † ) \bar{m} \in B_{\rho}^{C}(m^{\dagger}) m ˉ ∈ B ρ C ( m † ) is a solution of F ( m ) = y † \mathcal{F}(m)=y^{\dagger} F ( m ) = y † .
Finally, suppose that m † m^{\dagger} m † is the only solution of F ( m ) = y † \mathcal{F}(m)=y^{\dagger} F ( m ) = y † in B ρ C ( m † ) B_{\rho}^{C}(m^{\dagger}) B ρ C ( m † ) . Then m ˉ = m † \bar{m}=m^{\dagger} m ˉ = m † for every subsequence considered above, i.e. every subsequence of ( m ^ n ) (\widehat{m}_{n}) ( m n ) has a further subsequence that converges to m † m^{\dagger} m † in M \mathcal{M} M . Since the limit does not depend on the subsequence, the whole sequence converges, m ^ n → m † \widehat{m}_{n} \to m^{\dagger} m n → m † in M \mathcal{M} M . □
Remark 5.16. Note that convergence lim n → ∞ ε D , C ( n ) = 0 \lim_{n\to\infty}\varepsilon_{D,C}^{(n)}=0 lim n → ∞ ε D , C ( n ) = 0 of the derivative approximation error ε D , C ( n ) \varepsilon_{D,C}^{(n)} ε D , C ( n ) from (16) is required only if the tangential cone condition is assumed for the physical model (TCC). If the surrogates F n \mathcal{F}_{n} F n themselves satisfy (TCCθ _{\theta} θ ), the derivative approximation errors ε D , C ( n ) \varepsilon_{D,C}^{(n)} ε D , C ( n ) do not need to converge to 0.
Derivative-informed training and parameter-space geometry
The approximation results of § 3 and § 4 are existence results and do not say how the derivative accuracy required in § 5 can be obtained from finite training data. In practice, derivative information is supplied through finitely many directional derivatives. The directions used in training determine which parts of the derivative are controlled, whereas the parameter-space geometry used when the surrogate is employed in the inversion determines how the remaining errors are weighted in the Landweber update. This section makes the distinction precise and connects it with the choices made in the experiments.
Population and reduced derivative-informed training
Derivative-informed operator learning augments observations of the forward map with information about its Fréchet derivative. Following the standard DINO formulation, let μ \mu μ be a probability measure on the parameter space M \mathcal{M} M describing the distribution of inputs used for surrogate construction, and let H ↪ M \mathcal{H}\hookrightarrow\mathcal{M} H ↪ M be a Hilbert space of derivative directions. A natural population objective is the weighted Sobolev risk
R μ , H ( θ ) = E m ∼ μ [ ∥ F θ ( m ) − F ( m ) ∥ Y 2 + λ J ∥ D F θ ( m ) − D F ( m ) ∥ H S ( H , Y ) 2 ] . (28) \mathcal{R}_{\mu,\mathcal{H}}(\theta)=\mathbb{E}_{m\sim\mu}\left[\left\|\mathcal{F}_{\theta}(m)-\mathcal{F}(m)\right\|_{\mathcal{Y}}^{2}+\lambda_{J}\left\|D\mathcal{F}_{\theta}(m)-D\mathcal{F}(m)\right\|_{\mathrm{HS}(\mathcal{H},\mathcal{Y})}^{2}\right]. \tag*{(28)} R μ , H ( θ ) = E m ∼ μ [ ∥ F θ ( m ) − F ( m ) ∥ Y 2 + λ J ∥ D F θ ( m ) − D F ( m ) ∥ HS ( H , Y ) 2 ] . (28)
where λ J ≥ 0 \lambda_{J}\ge0 λ J ≥ 0 controls the relative contribution of derivative accuracy, and ∥ A ∥ H S ( H , Y ) 2 = ∑ j ∥ A e j ∥ Y 2 \lVert A\rVert_{\mathrm{HS}(\mathcal{H},\mathcal{Y})}^{2}=\sum_{j}\lVert Ae_{j}\rVert_{\mathcal{Y}}^{2} ∥ A ∥ HS ( H , Y ) 2 = ∑ j ∥ A e j ∥ Y 2 , for any orthonormal basis ( e j ) (e_{j}) ( e j ) of H \mathcal{H} H , is the Hilbert–Schmidt norm of a linear operator A : H → Y A:\mathcal{H}\to\mathcal{Y} A : H → Y . We write ∥ ⋅ ∥ H S \lVert\cdot\rVert_{\mathrm{HS}} ∥ ⋅ ∥ HS for ∥ ⋅ ∥ H S ( M , Y ) \lVert\cdot\rVert_{\mathrm{HS}(\mathcal{M},\mathcal{Y})} ∥ ⋅ ∥ HS ( M , Y ) . The risk (27) is the first-order weighted Sobolev (W μ , δ 1 , 2 W_{\mu,\delta}^{1,2} W μ , δ 1 , 2 ) formulation used in derivative-informed operator learning [31 , 39 ] . In particular, Yao et al. (2025, Theorem 4.3) establish universal approximation of FNOs in the corresponding W μ , δ 1 , 2 W_{\mu,\delta}^{1,2} W μ , δ 1 , 2 norm under suitable assumptions, and their § 5.3 compresses the Hilbert–Schmidt derivative term using reduced bases. The formulation below follows this population-to-reduced-loss viewpoint and adapts it to the finite set of physical derivative directions used in our experiments. The measure μ \mu μ determines the region of parameter space over which approximation accuracy is emphasised. In particular, H \mathcal{H} H determines the directions in which derivative accuracy is measured.
A particularly useful choice for covariance-based dimension reduction is obtained from the covariance of the data-generating measure. Assuming μ \mu μ has finite second moment, let
m ˉ = E μ [ m ] , Γ μ v = E μ [ ⟨ v , m − m ˉ ⟩ M ( m − m ˉ ) ] . (29) \bar{m} = \mathbb{E}_{\mu}[m], \qquad\Gamma_{\mu}v = \mathbb{E}_{\mu}\left[\langle v,m-\bar{m}\rangle_{\mathcal{M}}(m-\bar{m})\right]. \tag*{(29)} m ˉ = E μ [ m ] , Γ μ v = E μ [ ⟨ v , m − m ˉ ⟩ M ( m − m ˉ ) ] . (29)
Let Γ μ ψ j = κ j ψ j \Gamma_{\mu}\psi_j=\kappa_j\psi_j Γ μ ψ j = κ j ψ j with ⟨ ψ j , ψ k ⟩ M = δ j k \langle\psi_j,\psi_k\rangle_{\mathcal{M}}=\delta_{jk} ⟨ ψ j , ψ k ⟩ M = δ j k , and define the covariance-induced direction space M Γ μ : = ran ( Γ μ 1 / 2 ) \mathcal{M}_{\Gamma_{\mu}}:=\operatorname{ran}(\Gamma_{\mu}^{1/2}) M Γ μ := ran ( Γ μ 1/2 ) with its induced Hilbert norm. The vectors h j = κ j ψ j h_j=\sqrt{\kappa_j}\,\psi_j h j = κ j ψ j form an orthonormal basis of M Γ μ \mathcal{M}_{\Gamma_{\mu}} M Γ μ on the support of Γ μ \Gamma_{\mu} Γ μ . Consequently, writing
B θ ( m ) : = D F θ ( m ) − D F ( m ) (30) B_{\theta}(m):=DF_{\theta}(m)-DF(m) \tag*{(30)} B θ ( m ) := D F θ ( m ) − D F ( m ) (30)
for the derivative error, the derivative term in (27) can be written as
∥ B θ ( m ) ∥ H S ( M Γ μ , Y ) 2 = ∑ j ≥ 1 ∥ B θ ( m ) h j ∥ Y 2 = ∑ j ≥ 1 κ j ∥ B θ ( m ) ψ j ∥ Y 2 = ∥ B θ ( m ) Γ μ 1 / 2 ∥ H S ( M , Y ) 2 . (31) \lVert B_{\theta}(m)\rVert_{\mathrm{HS}(\mathcal{M}_{\Gamma_{\mu}},\mathcal{Y})}^{2}
=\sum_{j\geq1}\lVert B_{\theta}(m)h_j\rVert_{\mathcal{Y}}^{2}
=\sum_{j\geq1}\kappa_j\lVert B_{\theta}(m)\psi_j\rVert_{\mathcal{Y}}^{2}
=\lVert B_{\theta}(m)\Gamma_{\mu}^{1/2}\rVert_{\mathrm{HS}(\mathcal{M},\mathcal{Y})}^{2}. \tag*{(31)} ∥ B θ ( m ) ∥ HS ( M Γ μ , Y ) 2 = j ≥ 1 ∑ ∥ B θ ( m ) h j ∥ Y 2 = j ≥ 1 ∑ κ j ∥ B θ ( m ) ψ j ∥ Y 2 = ∥ B θ ( m ) Γ μ 1/2 ∥ HS ( M , Y ) 2 . (31)
When μ \mu μ is Gaussian, M Γ μ \mathcal{M}_{\Gamma_{\mu}} M Γ μ is the corresponding Cameron–Martin space, although no Gaussian assumption is required for the covariance-weighted representation above.
Computing the full Hilbert–Schmidt term in (30) is generally infeasible after high-dimensional discretisation. A standard approximation is therefore to retain a finite-dimensional subspace. For the leading r r r covariance modes, define
Γ μ , r = ∑ j = 1 r κ j ψ j ⊗ ψ j . (32) \Gamma_{\mu,r}=\sum_{j=1}^{r}\kappa_j\psi_j\otimes\psi_j. \tag*{(32)} Γ μ , r = j = 1 ∑ r κ j ψ j ⊗ ψ j . (32)
The corresponding reduced derivative error is
∥ B θ ( m ) Γ μ , r 1 / 2 ∥ H S 2 = ∑ j = 1 r ∥ B θ ( m ) h j ∥ Y 2 . (33) \lVert B_{\theta}(m)\Gamma_{\mu,r}^{1/2}\rVert_{\mathrm{HS}}^{2}
=\sum_{j=1}^{r}\lVert B_{\theta}(m)h_j\rVert_{\mathcal{Y}}^{2}. \tag*{(33)} ∥ B θ ( m ) Γ μ , r 1/2 ∥ HS 2 = j = 1 ∑ r ∥ B θ ( m ) h j ∥ Y 2 . (33)
This is the input-reduced Hilbert–Schmidt derivative loss used in reduced-basis derivative-informed learning. The discarded part of (30) lies between 0 and ∥ B θ ( m ) ∥ L ( M , Y ) 2 ∑ j > r κ j \lVert B_{\theta}(m)\rVert_{\mathcal{L}(\mathcal{M},\mathcal{Y})}^{2}\sum_{j>r}\kappa_j ∥ B θ ( m ) ∥ L ( M , Y ) 2 ∑ j > r κ j , so it is small when the eigenvalues of Γ μ \Gamma_{\mu} Γ μ decay quickly.
In practice, both the truncated covariance Γ μ , r \Gamma_{\mu,r} Γ μ , r in (31) and the expectation over μ \mu μ in (27) must be estimated from finite samples. Let m 1 , … , m N m_1,\ldots,m_N m 1 , … , m N be independent draws from μ \mu μ , and let h 1 , … , h r h_1,\ldots,h_r h 1 , … , h r denote the retained physical derivative directions obtained from a covariance approximation. The empirical objective takes the form
L N , r ( θ ) = 1 N ∑ i = 1 N [ ∥ F θ ( m i ) − F ( m i ) ∥ Y 2 + λ J r ∑ j = 1 r ∥ B θ ( m i ) h j ∥ Y 2 ] . (34) \mathcal{L}_{N,r}(\theta)=\frac{1}{N}\sum_{i=1}^{N}\left[\lVert F_{\theta}(m_i)-F(m_i)\rVert_{\mathcal{Y}}^{2}+\frac{\lambda_J}{r}\sum_{j=1}^{r}\lVert B_{\theta}(m_i)h_j\rVert_{\mathcal{Y}}^{2}\right]. \tag*{(34)} L N , r ( θ ) = N 1 i = 1 ∑ N [ ∥ F θ ( m i ) − F ( m i ) ∥ Y 2 + r λ J j = 1 ∑ r ∥ B θ ( m i ) h j ∥ Y 2 ] . (34)
The derivative term averages over the r r r directions instead of summing, so that the derivative error in a single direction drawn uniformly at random is an unbiased estimate of it. Our implementation uses one such direction for each occurrence of a training input (§ 7.3 ). Compared with (27), the average only rescales λ J \lambda_J λ J by 1 / r 1/r 1/ r .
With h j = κ j ψ j h_j = \sqrt{\kappa_j}\,\psi_j h j = κ j ψ j , the derivative term is (32) scaled by 1 / r 1/r 1/ r ; the eigenvalues κ j \kappa_j κ j are carried by the lengths of the h j h_j h j , since Γ μ , r = ∑ j = 1 r h j ⊗ h j \Gamma_{\mu,r} = \sum_{j=1}^{r} h_j \otimes h_j Γ μ , r = ∑ j = 1 r h j ⊗ h j . The same holds for any fixed directions h 1 , … , h r h_1,\ldots,h_r h 1 , … , h r , orthogonal or not: ∑ j = 1 r ∥ B θ ( m ) h j ∥ Y 2 = ∥ B θ ( m ) Γ ^ r 1 / 2 ∥ H S 2 \sum_{j=1}^{r} \lVert B_\theta(m)h_j\rVert_Y^2 = \lVert B_\theta(m)\widehat{\Gamma}_r^{1/2}\rVert_{\mathrm{HS}}^2 ∑ j = 1 r ∥ B θ ( m ) h j ∥ Y 2 = ∥ B θ ( m ) Γ r 1/2 ∥ HS 2 with Γ ^ r = ∑ j = 1 r h j ⊗ h j \widehat{\Gamma}_r = \sum_{j=1}^{r} h_j \otimes h_j Γ r = ∑ j = 1 r h j ⊗ h j , which plays the role of Γ μ , r \Gamma_{\mu,r} Γ μ , r . Finite directional supervision is therefore a finite-rank approximation of a covariance-weighted Hilbert–Schmidt derivative loss.
A small value of (27) gives mean-square control with respect to μ \mu μ , and at each parameter its Hilbert–Schmidt term controls the operator norm on H \mathcal{H} H . Neither the population risk nor its finite-sample version gives, without further assumptions, the uniform derivative bound over the admissible set required in § 5 .
Choosing the preconditioner after training
§ 5 introduced the preconditioned iteration (11) as the setting in which the approximation and regularisation results combine into Theorem 5.15. Here we consider its practical role. The geometry of the parameter space is part of the formulation of the inverse method, not a property of the forward problem, and preconditioned Landweber methods have a long history in inverse problems [29 , 10 , 10 ] . Our interest is in how this choice interacts with the approximation error of a learned forward operator.
Equivalently, (11) is obtained by applying Landweber iteration in M \mathcal{M} M to z ↦ F θ ( m 0 + C 1 / 2 z ) z \mapsto\mathcal{F}_\theta(m_0 + C^{1/2}z) z ↦ F θ ( m 0 + C 1/2 z ) and transforming back through m = m 0 + C 1 / 2 z m = m_0 + C^{1/2}z m = m 0 + C 1/2 z . The operator C C C is therefore more than a device to stabilise a learned model, since it specifies the geometry in which increments are measured. When C C C is smoothing, M C \mathcal{M}_C M C carries additional regularity relative to M \mathcal{M} M . The numerical experiments use a Matérn operator for this purpose. It is chosen after training and is not used to construct the derivative-supervision directions, although its hyperparameters lie within the range of the Matérn family used to generate training parameters.
Working in this stronger space entails the regularity condition m † − m 0 ∈ M C m^\dagger- m_0 \in\mathcal{M}_C m † − m 0 ∈ M C of Assumption 5.3. For a truth outside m 0 + M C m_0 + \mathcal{M}_C m 0 + M C , the iteration (11) is still well defined, but Theorem 5.15 does not apply. Remark 5.4 describes the condition for operators of Matérn type.
Which derivative errors matter under preconditioning
Changing the parameter-space geometry changes the derivative approximation that enters the Landweber iteration. With B θ ( m ) B_\theta(m) B θ ( m ) as in (29), we write ε D , C ( m ) : = ∥ B θ ( m ) C 1 / 2 ∥ L ( M , Y ) \varepsilon_{D,C}(m) := \lVert B_\theta(m)C^{1/2}\rVert_{\mathcal{L}(\mathcal{M},Y)} ε D , C ( m ) := ∥ B θ ( m ) C 1/2 ∥ L ( M , Y ) for the pointwise weighted error, whose supremum over B ρ C ( m † ) B_\rho^C(m^\dagger) B ρ C ( m † ) is ε D , C \varepsilon_{D,C} ε D , C in (16); it satisfies ε D , C ( m ) = ∥ C 1 / 2 B θ ( m ) ∗ ∥ L ( Y , M ) \varepsilon_{D,C}(m) = \lVert C^{1/2}B_\theta(m)^*\rVert_{\mathcal{L}(Y,\mathcal{M})} ε D , C ( m ) = ∥ C 1/2 B θ ( m ) ∗ ∥ L ( Y , M ) . For any residual R ∈ Y R \in Y R ∈ Y ,
∥ C B θ ( m ) ∗ R ∥ M C ≤ ε D , C ( m ) ∥ R ∥ Y . (35) \lVert C B_\theta(m)^*R\rVert_{\mathcal{M}_C} \le\varepsilon_{D,C}(m)\lVert R\rVert_Y. \tag*{(35)} ∥ C B θ ( m ) ∗ R ∥ M C ≤ ε D , C ( m ) ∥ R ∥ Y . (35)
The derivative error that matters for (11) is therefore the weighted error ε D , C ( m ) \varepsilon_{D,C}(m) ε D , C ( m ) , whatever the training. Smoother direction spaces also appear in the DIFNO theory of Yao et al. [39 ] , but there they are part of the approximation result, whereas here the stronger space is chosen when the trained surrogate is used for inversion.
The weighted error connects the inversion geometry with finite directional supervision. Let P r P_r P r be the orthogonal projection onto the supervised subspace H r = span { h 1 , … , h r } H_r = \operatorname{span}\{h_1,\ldots,h_r\} H r = span { h 1 , … , h r } .
Decomposing with respect to this subspace gives
∥ B θ ( m ) C 1 / 2 ∥ ≤ ∥ B θ ( m ) P r ∥ ∥ P r C 1 / 2 ∥ + ∥ B θ ( m ) ( I − P r ) ∥ ∥ ( I − P r ) C 1 / 2 ∥ . (36) \lVert B_\theta(m)C^{1/2}\rVert\le\lVert B_\theta(m)P_r\rVert\lVert P_rC^{1/2}\rVert+\lVert B_\theta(m)(I-P_r)\rVert\lVert(I-P_r)C^{1/2}\rVert. \tag*{(36)} ∥ B θ ( m ) C 1/2 ∥ ≤ ∥ B θ ( m ) P r ∥ ∥ P r C 1/2 ∥ + ∥ B θ ( m ) ( I − P r )∥ ∥( I − P r ) C 1/2 ∥ . (36)
The first term involves only the supervised directions and is therefore related to the error measured by the directional training objective. The second term contains derivative components not controlled by the prescribed directions, weighted by the extent to which the chosen parameter-space geometry retains them.
For a smoothing C C C , rough parameter directions are attenuated by C 1 / 2 C^{1/2} C 1/2 , which makes the iteration less sensitive to poorly approximated derivative components. It does not recover information that was never learned. The same operator acts on the exact gradient and attenuates genuine fine-scale features as well as surrogate error, so smoothing can improve robustness to continued iteration while slowing the recovery of fine-scale structure.
Forward error remains a separate contribution. With b = F θ ( m ) − F ( m ) b=F_\theta(m)-F(m) b = F θ ( m ) − F ( m ) , the learned and exact preconditioned search directions differ by C ( D F θ ( m ) − D F ( m ) ) ∗ R θ + C D F ( m ) ∗ b C(DF_\theta(m)-DF(m))^*R_\theta+CDF(m)^*b C ( D F θ ( m ) − D F ( m ) ) ∗ R θ + C D F ( m ) ∗ b , where R θ = F θ ( m ) − y δ R_\theta=F_\theta(m)-y^\delta R θ = F θ ( m ) − y δ ; in the norm of M C \mathcal{M}_C M C the first term is at most ε D , C ( m ) ∥ R θ ∥ Y \varepsilon_{D,C}(m)\lVert R_\theta\rVert_Y ε D , C ( m ) ∥ R θ ∥ Y , whereas the second is at most ∥ D F ( m ) C 1 / 2 ∥ ∥ b ∥ Y \lVert DF(m)C^{1/2}\rVert\lVert b\rVert_Y ∥ D F ( m ) C 1/2 ∥ ∥ b ∥ Y and is controlled only by the forward error. Changing the geometry can therefore reduce the influence of derivative error, but it does not remove the forward error and does not guarantee a more accurate search direction in relative terms.
Relation to the regularisation analysis
For compact C C C , Lemma 5.10 and Theorem 5.15 show that the effective error level in the discrepancy principle is Δ \Delta Δ in (21) , with γ = ρ ε D , C \gamma=\rho\varepsilon_{D,C} γ = ρ ε D , C under (TCC) and γ = 0 \gamma=0 γ = 0 under (TCCθ _\theta θ ). The derivative error therefore enters only through its weighted form, and not at all when the surrogate itself satisfies the cone condition. The scale of C C C plays no role: replacing C C C by s C sC s C multiplies ε D , C \varepsilon_{D,C} ε D , C by s 1 / 2 s^{1/2} s 1/2 and the radius ρ \rho ρ in M C \mathcal{M}_C M C by s − 1 / 2 s^{-1/2} s − 1/2 , so that only the relative weighting of parameter directions matters. A smoothing C C C reduces ε D , C \varepsilon_{D,C} ε D , C relative to the unweighted derivative error, but it also increases ρ \rho ρ , so the product ρ ε D , C \rho\varepsilon_{D,C} ρ ε D , C can decrease or increase.
The existence of surrogates with small ε o p \varepsilon_{\mathrm{op}} ε op and ε D , C \varepsilon_{D,C} ε D , C on B ρ C ( m † ) B_\rho^C(m^\dagger) B ρ C ( m † ) , noted after (16) , is an existence result. Minimising the derivative term in (34) on finitely many parameters and directions does not establish a uniform bound on ε D , C \varepsilon_{D,C} ε D , C . The numerical experiments keep the derivative-supervision protocol fixed while varying the parameter-space geometry used in the inversion. This separates the information used to construct the learned operator from the geometry in which it is used.
Numerical experiments
Implementation. All finite-element computations, in this section and in § 7.12 , use FEniCSx/DOLFINx 0.10.0 with PETSc 3.25.0, and exact derivatives are computed from the discrete tangent and adjoint equations of the same discretisation. The neural operators are implemented in PyTorch 2.4.0 with CUDA 12.4. GPU computations use the default, non-deterministic reductions of PyTorch, so repeated surrogate runs are not reproducible bit for bit, and rounding differences can grow along an iteration. Finite-element solves run on CPU cores, eight per exact inversion. For the reaction benchmark, networks are trained on one NVIDIA A100 GPU and surrogate inversions run on one NVIDIA L40 GPU. All runs were performed on the Sulis Tier 2 HPC platform.
Reaction benchmark: forward model and discretisation
The principal benchmark is the recovery of a spatially varying reaction coefficient in an elliptic PDE, simple enough that forward, derivative and adjoint computations can be evaluated accurately by finite elements, while retaining the nonlinear parameter-to-state structure relevant to the analysis. The unknown is a reaction coefficient m m m on Ω = ( 0 , 1 ) 2 \Omega= (0,1)^2 Ω = ( 0 , 1 ) 2 , with state p p p satisfying
− Δ p + m p = 1 in Ω , p = 0 on ∂ Ω , (37) -\Delta p + mp = 1 \quad\text{in } \Omega,\qquad p = 0 \quad\text{on } \partial\Omega, \tag*{(37)} − Δ p + m p = 1 in Ω , p = 0 on ∂ Ω , (37)
and F ( m ) = p ( m ) F(m) = p(m) F ( m ) = p ( m ) denotes the parameter-to-state map, with parameter and data spaces L 2 ( Ω ) L^2(\Omega) L 2 ( Ω ) . The admissible coefficients satisfy 0 ≤ m ≤ 200 0 \le m \le200 0 ≤ m ≤ 200 , so that the weak problem is coercive. For a perturbation h h h , the directional derivative v = D F ( m ) h v = DF(m)h v = D F ( m ) h satisfies
− Δ v + m v = − h p in Ω , v = 0 on ∂ Ω , (38) -\Delta v + mv = -hp \quad\text{in } \Omega,\qquad v = 0 \quad\text{on } \partial\Omega, \tag*{(38)} − Δ v + m v = − h p in Ω , v = 0 on ∂ Ω , (38)
and, for z ∈ L 2 ( Ω ) z \in L^2(\Omega) z ∈ L 2 ( Ω ) , the adjoint action is D F ( m ) ∗ z = − p w DF(m)^*z = -pw D F ( m ) ∗ z = − pw , where
− Δ w + m w = z in Ω , w = 0 on ∂ Ω , (39) -\Delta w + mw = z \quad\text{in } \Omega,\qquad w = 0 \quad\text{on } \partial\Omega, \tag*{(39)} − Δ w + m w = z in Ω , w = 0 on ∂ Ω , (39)
so that both the derivative labels used in training and the exact Landweber directions require only elliptic solves.
Discretisation. The surrogate training and inverse calculations use continuous piecewise-linear finite elements on a uniform triangulation obtained from 119 × 119 square cells, giving a 120 × 120 nodal representation of the coefficient. The state is observed at the interior mesh vertices. To keep the notation light, we write F F F and D F DF D F also for the resulting discrete forward map and its derivative; all computations below use the discrete model.
Training measure
The measure μ \mu μ of § 6.1 is a heterogeneous family of stationary Matérn fields: training coefficients are m = a + σ z ν , ℓ m = a + \sigma z_{\nu,\ell} m = a + σ z ν , ℓ , where z ν , ℓ z_{\nu,\ell} z ν , ℓ has zero mean and unit marginal variance, a ∼ U ( 95 , 105 ) a \sim\mathcal{U}(95,105) a ∼ U ( 95 , 105 ) , σ ∼ U ( 10 , 30 ) \sigma\sim\mathcal{U}(10,30) σ ∼ U ( 10 , 30 ) , ν ∈ { 1 , 1.5 , 2 , 2.5 } \nu\in\{1,1.5,2,2.5\} ν ∈ { 1 , 1.5 , 2 , 2.5 } and ℓ ∈ { 0.075 , 0.10 , 0.15 , 0.20 , 0.25 } \ell\in\{0.075,0.10,0.15,0.20,0.25\} ℓ ∈ { 0.075 , 0.10 , 0.15 , 0.20 , 0.25 } . When necessary, the fluctuation about a a a is rescaled to enforce 0 ≤ m ≤ 200 0 \le m \le200 0 ≤ m ≤ 200 , so μ \mu μ is a non-Gaussian mixture over amplitudes, regularities and correlation lengths. It is the measure in the risk (28) ; the inversion itself uses no prior. We draw 10 000 coefficients for training, 2 500 for validation and 2 500 for final testing, independently, and a further independent ensemble of 10 000 fields to construct the derivative directions.
Learning the forward map and derivative actions
We compare four learned forward operators, each mapping the nodal coefficient to the nodal state on the 120 × 120 grid. All are trained with AdamW for 2500 epochs (learning rate 10 − 3 10^{-3} 1 0 − 3 for the FNO and 10 − 4 10^{-4} 1 0 − 4 otherwise), with the learning rate reduced on plateaus of the validation forward error. Their parameter counts are listed in Table 1 .
Table 1. Independent-test accuracy of the four validation-selected surrogates.
FNO. A standard FNO (Li et al., 2021) whose lifting receives the normalised coefficient and the coordinates ( x , y ) (x,y) ( x , y ) , followed by four Fourier layers of width 24 with 10 × 10 10 \times10 10 × 10 retained modes, padding 8, GELU activations and a projection of width 128.
DeepONet. A standard DeepONet (Lu et al., 2021). The branch is a convolutional network with GroupNorm, ReLU and max pooling acting on the normalised coefficient and the coordinates, and returns 400 coefficients. The trunk has six GELU layers of width 400 acting on Fourier features of ( x , y ) (x,y) ( x , y ) with frequencies 1 , 2 , … , 32 1,2,\ldots,32 1 , 2 , … , 32 . The output is the scaled inner product of branch and trunk plus a bias.
GenDeepONet. A generalised DeepONet (7) . The encoder subtracts the mean field and returns the standardised coordinates of m m m in the leading 400 input PCA modes, and the decoder combines the leading 300 output POD modes and adds the mean state. Both bases use trapezoidal quadrature weights and are computed from the 10 000 training fields before training. The network f θ f_\theta f θ is a residual multilayer perceptron of width 1024 with four GELU residual blocks.
GenFNO. A generalised Fourier operator (9) . The encoder applies a discrete Fourier transform to the normalised coefficient and keeps a rectangular block of low frequencies, r = 1024 r=1024 r = 1024 real coordinates; f θ f_\theta f θ is a residual multilayer perceptron of width 512 with four GELU residual blocks, returning s = 2304 s=2304 s = 2304 real Fourier coefficients from which the decoder synthesises the state. The Dirichlet condition is not imposed but learned from the data, and the inversion uses only interior observations.
All derivatives of the surrogates are exact derivatives of the complete networks: Jacobian–vector products in the derivative loss and vector–Jacobian products, with the residual held fixed, in the Landweber update. They are computed by automatic differentiation of the nonlinear part, with the fixed linear encoders and decoders either included in the differentiation or applied analytically; no separate derivative network is trained. GenDeepONet and GenFNO are instances of the encoder–network–decoder class Φ f θ Ψ \Phi f_\theta\Psi Φ f θ Ψ of (4) , with fixed Ψ \Psi Ψ and Φ \Phi Φ given by PCA/POD and by Fourier coordinates, respectively (the mean fields and coordinate scalings are affine and can be absorbed as in § 4.1 ). The results of Theorems 3.1 and 3.2 therefore apply to their classes through Corollary 4.6, since GELU is C 1 C^1 C 1 and nonpolynomial, although they concern the classes, not the sizes used here, and for GenFNO the Fourier coordinates on the square lie outside the periodic setting of the corollary. The standard FNO and DeepONet are not of this form: the FNO couples frequencies only through pointwise nonlinearities (Remark 4.7), and the DeepONet learns its output functions through the trunk (Remark 4.3) and, with its ReLU branch, is only piecewise smooth in m m m . The purpose of the comparison is not to rank architectures, but to examine whether learned operators with different representations can supply sufficiently accurate values and derivatives for iterative regularisation.
Derivative directions. The r = 16 r=16 r = 16 training directions of § 6.1 are built from the independent ensemble. We separate the spatial mean ⟨ m ⟩ Ω \langle m\rangle_{\Omega} ⟨ m ⟩ Ω of each field from its fluctuation m ′ = m − ⟨ m ⟩ Ω 1 m'=m-\langle m\rangle_{\Omega}1 m ′ = m − ⟨ m ⟩ Ω 1 , apply PCA to the fluctuations, and take
h 1 = σ m e a n 1 , h j + 1 = κ j ψ j , j = 1 , … , r − 1 , (40) h_1=\sigma_{\mathrm{mean}}1,\qquad h_{j+1}=\sqrt{\kappa_j}\psi_j,\qquad j=1,\ldots,r-1, \tag*{(40)} h 1 = σ mean 1 , h j + 1 = κ j ψ j , j = 1 , … , r − 1 , (40)
where σ m e a n \sigma_{\mathrm{mean}} σ mean is the standard deviation of the spatial mean and ( κ j , ψ j ) (\kappa_j,\psi_j) ( κ j , ψ j ) are the leading empirical variances and modes of the fluctuations. Because the mean is split off first, these directions differ from the r r r leading eigenvectors of the empirical covariance of m m m . The derivative loss therefore weights derivative errors by Γ ^ r = ∑ j = 1 r h j ⊗ h j \widehat{\Gamma}_r = \sum_{j=1}^{r} h_j \otimes h_j Γ r = ∑ j = 1 r h j ⊗ h j , in the notation of Section 6.1, and not by a truncated covariance Γ μ , r \Gamma_{\mu,r} Γ μ , r .
Training and evaluation. For every training coefficient m i m_i m i , the finite-element model supplies F ( m i ) \mathcal{F}(m_i) F ( m i ) and the derivative actions D F ( m i ) h j D\mathcal{F}(m_i)h_j D F ( m i ) h j , j = 1 , … , r j=1,\ldots,r j = 1 , … , r , from (37), and the same data are given to all four architectures. During derivative-informed training, one direction is drawn uniformly at random for each occurrence of a training coefficient, as in Section 6.1. Since each architecture computes its loss in its own normalised or reduced coordinates, all comparisons use common physical-space metrics. On the independent test set, with N F = 2500 N_F=2500 N F = 2500 fields, a fixed subset I J I_J I J of 256 fields for derivatives and the full-grid trapezoidal weights W p W_p W p , we report
E F 2 = ∑ i = 1 N F ∥ F θ ( m i ) − F ( m i ) ∥ W p 2 ∑ i = 1 N F ∥ F ( m i ) ∥ W p 2 , E J 2 = ∑ i ∈ I J ∑ j = 1 r ∥ B θ ( m i ) h j ∥ W p 2 ∑ i ∈ I J ∑ j = 1 r ∥ D F ( m i ) h j ∥ W p 2 , (41) E_F^2 = \frac{\sum_{i=1}^{N_F}\left\| \mathcal{F}_\theta(m_i)-\mathcal{F}(m_i)\right\|_{W_p}^2}{\sum_{i=1}^{N_F}\left\|\mathcal{F}(m_i)\right\|_{W_p}^2}, \qquad E_J^2 = \frac{\sum_{i\in I_J}\sum_{j=1}^{r}\left\|B_\theta(m_i)h_j\right\|_{W_p}^2}{\sum_{i\in I_J}\sum_{j=1}^{r}\left\|D\mathcal{F}(m_i)h_j\right\|_{W_p}^2}, \tag*{(41)} E F 2 = ∑ i = 1 N F ∥ F ( m i ) ∥ W p 2 ∑ i = 1 N F ∥ F θ ( m i ) − F ( m i ) ∥ W p 2 , E J 2 = ∑ i ∈ I J ∑ j = 1 r ∥ D F ( m i ) h j ∥ W p 2 ∑ i ∈ I J ∑ j = 1 r ∥ B θ ( m i ) h j ∥ W p 2 , (41)
where B θ B_\theta B θ is the derivative error (29). E J E_J E J measures generalisation to new coefficient fields along the training directions, that is, in the Γ ^ r \widehat{\Gamma}_r Γ r geometry; it is not the full Hilbert–Schmidt or operator-norm error, nor a bound along an inverse trajectory.
For each architecture we considered λ J ∈ { 0 , 0.1 , 1 , 10 , 100 , 300 , 1000 , 3000 } \lambda_J \in\{0, 0.1, 1, 10, 100, 300, 1000, 3000\} λ J ∈ { 0 , 0.1 , 1 , 10 , 100 , 300 , 1000 , 3000 } and selected the model and checkpoint on the validation set only, by the score S v a l = [ ( E F v a l ) 2 + ( E J v a l ) 2 ] 1 / 2 S_{\mathrm{val}} = [(E_F^{\mathrm{val}})^2 + (E_J^{\mathrm{val}})^2]^{1/2} S val = [( E F val ) 2 + ( E J val ) 2 ] 1/2 . Table 1 gives the selected weights and epochs and the test errors: all four surrogates have forward errors below 0.5% and derivative errors below 0.8%. For each architecture we also retain the model trained without the derivative loss (λ J = 0 \lambda_J=0 λ J = 0 ), whose checkpoint is selected by the same score; this gives it derivative information its training did not use, so the comparison in Section 7.6, if anything, favours it.
Inversion setup
Inverse truths and data. We consider two prescribed inverse truths. T1 is an independent draw from the training family, with ν = 3 / 2 \nu=3/2 ν = 3/2 and ℓ = 0.075 \ell=0.075 ℓ = 0.075 , and provides an in-distribution problem. T2 is a deterministic coefficient with background value 100, a rectangular inclusion of value 150 and two circular inclusions of values 20 and 180; it lies in the admissible range but has sharp geometric structure absent from the training family. The observations are computed on a finer 952 × 952 952 \times952 952 × 952 -cell mesh and restricted to the observation grid, so that the data are not generated with the discretisation used for inversion. The comparisons below are noiseless until § 7.11 .
Landweber implementation. The baseline inverse calculations use the Matérn parameter-space geometry introduced in § 5.1 , with ν C = 3 / 2 \nu_C = 3/2 ν C = 3/2 , ℓ C = 0.1 \ell_C = 0.1 ℓ C = 0.1 and σ C 2 = 1 \sigma_C^2 = 1 σ C 2 = 1 , and start from m 0 ≡ 100 m_0 \equiv100 m 0 ≡ 100 . Let W y W_y W y denote the quadrature weights in the discrete observation norm, R k = F θ ( m k ) − y h d R_k = \mathcal{F}_{\theta}(m_k) - y_h^d R k = F θ ( m k ) − y h d the residual and J k = D F θ ( m k ) J_k = D\mathcal{F}_{\theta}(m_k) J k = D F θ ( m k ) the derivative of the learned map (or F \mathcal{F} F and D F D\mathcal{F} D F for the exact iteration). We form q k = J k T W y R k q_k = J_k^{\mathsf{T}}W_yR_k q k = J k T W y R k and d k = C h q k d_k = C_hq_k d k = C h q k , where C h C_h C h is the discrete action of the chosen parameter-space operator, and use the adaptive step
α k = min { 0.95 q k T d k ∥ J k d k ∥ W y 2 , 5 ∥ d k ∥ ∞ } . (42) \alpha_k = \min\left\{0.95\frac{q_k^{\mathsf{T}}d_k}{\lVert J_kd_k\rVert_{W_y}^2},\frac{5}{\lVert d_k\rVert_\infty}\right\}. \tag*{(42)} α k = min { 0.95 ∥ J k d k ∥ W y 2 q k T d k , ∥ d k ∥ ∞ 5 } . (42)
The proposed update m k − α k d k m_k-\alpha_kd_k m k − α k d k is projected componentwise onto [ 0 , 200 ] [0,200] [ 0 , 200 ] , without backtracking, and the exact-FEM reference uses the same protocol with finite-element evaluations. The first term in (42) is 0.95 times the steepest-descent step in the geometry induced by C h C_h C h , which the descent argument of Lemma 5.10 covers (Remark 5.12 on variable step sizes). The ℓ ∞ \ell^\infty ℓ ∞ cap and the projection are not covered by the analysis; see [32 ] for projected Landweber iteration in the linear case.
The matrix C h C_h C h is the nodal Matérn covariance matrix ( C h ) i j = k ( ∥ x i − x j ∥ ) (C_h)_{ij} = k(\lVert x_i-x_j\rVert) ( C h ) ij = k (∥ x i − x j ∥) for grid nodes x i , x j x_i,x_j x i , x j , with the Matérn kernel ([35 ] , Section 4.2)
k ( r ) = σ C 2 2 1 − ν C Γ ( ν C ) ( 2 ν C r ℓ C ) ν C K ν C ( 2 ν C r ℓ C ) , (43) k(r) = \sigma_C^2\frac{2^{1-\nu_C}}{\Gamma(\nu_C)}\left(\frac{\sqrt{2\nu_C}r}{\ell_C}\right)^{\nu_C}K_{\nu_C}\left(\frac{\sqrt{2\nu_C}r}{\ell_C}\right), \tag*{(43)} k ( r ) = σ C 2 Γ ( ν C ) 2 1 − ν C ( ℓ C 2 ν C r ) ν C K ν C ( ℓ C 2 ν C r ) , (43)
where K ν K_\nu K ν is the modified Bessel function of the second kind. It is constructed from the grid geometry alone, independently of the training data, and applied exactly, up to rounding, by zero-padded FFT convolution, so that opposite boundaries are not identified. It discretises the covariance operator ( C f ) ( x ) = ∫ Ω k ( ∥ x − y ∥ ) f ( y ) d y (Cf)(x) = \int_\Omega k(\lVert x-y\rVert)f(y)\,\mathrm{d}y ( C f ) ( x ) = ∫ Ω k (∥ x − y ∥) f ( y ) d y on L 2 ( Ω ) L^2(\Omega) L 2 ( Ω ) , the quadrature factor being absorbed by the adaptive step, so by Remark 5.4 the baseline geometry has M C = H 5 / 2 ( Ω ) \mathcal{M}_C = H^{5/2}(\Omega) M C = H 5/2 ( Ω ) . Neither truth satisfies m † − m 0 ∈ M C m^\dagger-m_0 \in\mathcal{M}_C m † − m 0 ∈ M C : T2 is discontinuous, and T1, a Matérn sample with the same smoothness ν = 3 / 2 \nu= 3/2 ν = 3/2 as C, lies outside its Cameron–Martin space almost surely. The baseline runs therefore lie outside the hypotheses of Theorem 5.15; § 7.9 repeats the noiseless experiments with geometries (ν C < 1 / 2 \nu_C < 1/2 ν C < 1/2 ) for which the regularity condition holds for T1.
With m h † m_h^\dagger m h † the target coefficient on the inversion grid and W m W_m W m the corresponding quadrature weights, we report the relative coefficient error and the oracle iteration
e m ( k ) = ∥ m k − m h † ∥ W m ∥ m h † ∥ W m , k o r = argmin 0 ≤ k ≤ 10000 e m ( k ) . (44) e_m(k) = \frac{\lVert m_k-m_h^\dagger\rVert_{W_m}}{\lVert m_h^\dagger\rVert_{W_m}}, \qquad k_{\mathrm{or}} = \underset{0\leq k\leq10000}{\operatorname{argmin}}\ e_m(k). \tag*{(44)} e m ( k ) = ∥ m h † ∥ W m ∥ m k − m h † ∥ W m , k or = 0 ≤ k ≤ 10000 argmin e m ( k ) . (44)
The oracle uses the known truth; it serves only to compare the forward models and is not a stopping rule. Many oracle iterations coincide with the end of the 10 000-iteration budget, and the corresponding errors are then finite-budget values rather than minima.
Noiseless reconstruction with learned operators
Table 2 summarises the noiseless reconstructions for T1 and T2 with the four selected surrogates and the baseline geometry. The exact-FEM iteration reaches oracle errors of 4.12% on T1 and 6.94% on T2. The corresponding surrogate oracle errors range from 5.83% to 7.65% on T1 and from 8.57% to 9.70% on T2. All four learned operators therefore support meaningful reconstruction on both the in-distribution and structured truths, although with an accuracy loss relative to the finite-element reference.
The last two columns compare the misfit seen by each surrogate with the misfit of the same reconstruction under the finite-element model. For FNO and DeepONet the two agree closely. For GenFNO on T1 the surrogate reports 0.46% against a finite-element misfit of 0.10%, whereas for GenDeepONet on T2 it reports 0.22% against 0.58%. The residual of the surrogate therefore need not reflect how well its reconstruction fits the data: it can overstate the true misfit, as for GenFNO on T1, or understate it, as for GenDeepONet on T2.
Figure 1 shows the complete noiseless trajectories. With exact data the exact iteration keeps reducing the coefficient error over all 10 000 steps, as expected for noise-free data, and so do most surrogate iterations. The exceptions are GenFNO and GenDeepONet on T2, whose error is smallest before the end of the run (Table 2 ) and then grows slightly while their residual keeps decreasing. Here the surrogate error plays the role of data noise, and this is the onset of the semiconvergence studied in § 7.8 .
Table 2. Noiseless reaction-coefficient inversion for the held-out in-distribution truth T1 and the structured truth T2. Here e m ∗ = e m ( k o r ) e_m^* = e_m(k_{\mathrm{or}}) e m ∗ = e m ( k or ) and e m K e_m^K e m K is the error at the final iterate K = 10 000 K = 10\,000 K = 10 000 . The residuals are relative data misfits at the oracle iterate, 100 ∥ G ( m k o r ) − y h ∥ W y / ∥ y h ∥ W y 100 \lVert G(m_{\mathrm{kor}}) - y_h \rVert_{W_y} / \lVert y_h \rVert_{W_y} 100 ∥ G ( m kor ) − y h ∥ W y / ∥ y h ∥ W y , with G = F θ G = \mathcal{F}_{\theta} G = F θ for the surrogate residual and G = F G = \mathcal{F} G = F for the FEM residual. The latter re-evaluates the surrogate reconstruction with the finite-element model. For exact FEM only the FEM residual applies.
Figure 1. Noiseless data misfit and relative coefficient error for T1 (top) and T2 (bottom), for the exact finite-element iteration and the four selected learned operators. Stars mark the smallest coefficient error over the 10 000 iterations. Each data misfit is evaluated with the forward model used in that iteration.
Figure 2 shows the oracle reconstructions: every method reproduces the large-scale variation of T1 and, although the training distribution contains no such shapes, the locations and relative contrasts of the three inclusions of T2. Even the exact-FEM reconstruction smooths the interfaces of T2, so the loss of sharpness is due partly to the baseline geometry, which § 7.8 varies.
Figure 2. Noiseless T1 (top) and T2 (bottom) truths and oracle reconstructions m k o r m_{\mathrm{kor}} m kor with the baseline Matérn geometry. Each row has its own colour scale, from the minimum to the maximum over its six fields. The inclusions of T2 are not drawn from the random-field morphology used for training.
Effect of derivative-informed training
To isolate the effect of derivative supervision, we compare each selected surrogate with the architecture-matched model trained without the derivative loss (λ J = 0 \lambda_J = 0 λ J = 0 ), with everything else unchanged. Derivative supervision reduces the test derivative error E J E_J E J by 45% for FNO, 72% for DeepONet, 63% for GenFNO and 94% for GenDeepONet, but hardly changes the best reconstruction: the oracle errors change by between −1.1 and +0.4 percentage points, with mixed sign, which is of the size to be expected between training seeds for models trained once. Its clearer effect is on the late iterations. Without the derivative loss, GenFNO and GenDeepONet reach their smallest errors early and then deteriorate, on T2 at iterations 7185 and 1286; with it these move to 8859 and 5523, and on T1 from 7587 and 5601 to 10 000 and 9143. FNO and DeepONet reach their smallest errors at the end of the run in either case. Derivative accuracy thus delays semiconvergence, as the level Δ \Delta Δ predicts, but does not by itself determine the best reconstruction; more generally, smaller E F E_F E F or E J E_J E J in Table 1 does not imply a smaller coefficient error in Table 2 . We examine this distinction in the next subsection.
From directional derivative accuracy to the Landweber direction
The metric E J E_J E J measures derivative accuracy along the training directions, in the geometry of Γ ^ r \widehat{\Gamma}_r Γ r , whereas the baseline iteration depends on the weighted adjoint action C D F θ ( m k ) ∗ R k C\,DF_\theta(m_k)^*R_k C D F θ ( m k ) ∗ R k ; as discussed in § 6 , accuracy in the first need not imply accuracy in the second.
To examine this distinction without confounding it with architecture-dependent inverse trajectories, we evaluate the derivative diagnostics at two common coefficient fields, the initial point m 0 ≡ 100 m_0 \equiv100 m 0 ≡ 100 and the T2 truth. At such a field m m m , let R = F θ ( m ) − y h R = F_\theta(m) - y_h R = F θ ( m ) − y h be the residual of the surrogate with respect to the noiseless T2 data y h y_h y h , and let q θ = D F θ ( m ) ⊤ W y R q_\theta= DF_\theta(m)^\top W_yR q θ = D F θ ( m ) ⊤ W y R and q = D F ( m ) ⊤ W y R q = DF(m)^\top W_yR q = D F ( m ) ⊤ W y R be the learned and exact adjoint actions, that is, the vector–Jacobian products (VJPs) for this residual. Using the same R R R in both isolates the error of the derivative from that of the forward map. With B θ B_\theta B θ as in (30) , we compare three relative errors,
[ ∑ j ∥ B θ ( m ) h j ∥ W y 2 ∑ j ∥ D F ( m ) h j ∥ W y 2 ] 1 / 2 , ∥ q θ − q ∥ 2 ∥ q ∥ 2 , [ ( q θ − q ) ⊤ C h ( q θ − q ) ] 1 / 2 [ q ⊤ C h q ] 1 / 2 . \left[\frac{\sum_j\|B_\theta(m)h_j\|_{W_y}^2}{\sum_j\|DF(m)h_j\|_{W_y}^2}\right]^{1/2}, \qquad\frac{\|q_\theta-q\|_2}{\|q\|_2}, \qquad\frac{\left[(q_\theta-q)^\top C_h(q_\theta-q)\right]^{1/2}}{\left[q^\top C_hq\right]^{1/2}}. [ ∑ j ∥ D F ( m ) h j ∥ W y 2 ∑ j ∥ B θ ( m ) h j ∥ W y 2 ] 1/2 , ∥ q ∥ 2 ∥ q θ − q ∥ 2 , [ q ⊤ C h q ] 1/2 [ ( q θ − q ) ⊤ C h ( q θ − q ) ] 1/2 .
The first is the pooled error of the Jacobian–vector products (JVPs) along the r r r training directions h j h_j h j of (40) . The second is the error of the VJP in the Euclidean norm, which governs an unpreconditioned Landweber step. The third is the error of the preconditioned directions C h q θ C_hq_\theta C h q θ and C h q C_hq C h q in the norm of M C \mathcal{M}_C M C , in which Lemma 5.10 measures the iteration. With the discrete inner product ⟨ u , v ⟩ M C = u ⊤ C h − 1 v \langle u,v\rangle_{M_C}=u^\top C_h^{-1}v ⟨ u , v ⟩ M C = u ⊤ C h − 1 v , one has ∥ C h v ∥ M C 2 = v ⊤ C h v \|C_hv\|_{M_C}^2=v^\top C_hv ∥ C h v ∥ M C 2 = v ⊤ C h v . The residual R R R depends on the surrogate, so the VJP errors of different architectures use slightly different residuals.
Figure 3 shows that these notions of derivative accuracy can differ substantially. At the initial field, for example, DeepONet has a pooled fixed-direction JVP error of 1.52%, but its ambient same-residual VJP error is 30.08%. After application of the Matérn operator, the corresponding VJP error, measured in M C \mathcal{M}_C M C , is 1.30%. The relative separation is even larger for GenDeepONet, whose fixed-direction error at m 0 m_0 m 0 is only 0.067% against an ambient VJP error of 6.02%.
Figure 3. The three relative derivative errors defined in the text, at the initial field m 0 ≡ 100 m_0 \equiv100 m 0 ≡ 100 and at the T2 truth: pooled JVP error along the training directions, Euclidean (ambient) VJP error, and preconditioned VJP error in the norm of M C \mathcal{M}_C M C .
The same qualitative distinction remains at the T2 truth, although its magnitude is model dependent. In every case the ambient VJP error is larger than the corresponding fixed-direction error, and applying the Matérn geometry reduces the ambient adjoint error. The preconditioned error at the T2 truth nevertheless remains appreciable for GenFNO and GenDeepONet.
This helps explain why the reductions in E J E_J E J reported in § 7.6 do not translate into better reconstructions: the training loss probes a few fixed input directions, whereas the iteration uses the adjoint action for its current residual, weighted by C C C .
Preconditioning and semiconvergence
Section 5 motivates the preconditioner by the fact that the learned derivative is unreliable in rough directions, which are essentially absent from the training data. To test this, we run the same noiseless T2 inversions with the identity in place of the Matérn geometry, keeping the network weights, data and initial field. This setting lies outside the theory of Section 5. The Matérn results are those of Table 2 , and the smallest errors with the identity are also listed in Table 5.
Without preconditioning, three of the four surrogates semiconverge although the data are exact, whereas the exact iteration and the FNO reach their smallest errors at the end of the run. DeepONet reaches its minimum of 9.57% after 275 iterations and ends at 10.49%. GenFNO reaches 8.80% at iteration 129 and ends at 23.57%, and GenDeepONet reaches 9.62% at iteration 448 and ends at 15.20%. With the Matérn geometry the same models stay within 1% of their minima over the budget. The loss of accuracy comes with spurious structure (Figure 4): fine oscillatory texture already at the oracle iterates, and by the end of the run high-frequency speckle, ring-like errors around the inclusions and values pushed to the bounds 0 and 200. The exact-FEM control shows that the problem itself does not require the smoothing. With the identity, exact Landweber reaches 1.62% on T2, against 6.94% with the Matérn geometry, and the FNO likewise improves from 8.86% to 5.63%. We also decreased ℓ C \ell_C ℓ C from 0.1 to 0.005, which moves the geometry gradually from the baseline towards the identity (results not shown). Along this path the best FNO error decreases steadily, reaching 0.64 times its baseline value under the identity, while those of the other surrogates change little. This is consistent with Figure 3 . The FNO has much the smallest unpreconditioned VJP error, so its unfiltered direction remains accurate, whereas for the other surrogates the components removed by the smoothing are dominated by derivative error.
Figure 4. Noiseless T2 reconstructions without preconditioning. Top: each method at its oracle iterate. Bottom: the final iterate, k = 10 000 k = 10\,000 k = 10 000 . Continued iteration produces oscillatory artefacts and projection onto the bounds for DeepONet, GenFNO and GenDeepONet.
The instability without preconditioning therefore comes from the surrogates, not from the inverse problem. As anticipated in § 6.3 , the Matérn geometry does not make the learned derivative more accurate but damps its poorly learned fine-scale components, together with genuine fine-scale information, which is why it improves stability while slowing the recovery of the sharp features of T2.
Surrogate error as a source of semiconvergence. In Lemma 5.10 the surrogate errors enter the level Δ = δ + ε o p + ρ ε D , C / ( 1 + η ) \Delta= \delta+ \varepsilon_{\mathrm{op}} + \rho\varepsilon_{D,C}/(1 + \eta) Δ = δ + ε op + ρ ε D , C / ( 1 + η ) in the same way as the noise. Even with exact data, descent is guaranteed only while the residual exceeds τ Δ \tau\Delta τ Δ , and beyond that point an inaccurate surrogate can drive the iteration away from the truth, as noise would. To test this we use a capacity study of the FNO, in which the width and number of Fourier modes are varied while the training data, the derivative weight λ J = 10 \lambda_J = 10 λ J = 10 and the selection rule are fixed (Table 3 ), and a smaller study in which the number of training fields is also reduced (Table 4 ).
Table 3 : FNO capacity study at λ J = 10 \lambda_J = 10 λ J = 10 , noiseless T2 data. For each capacity the same checkpoint is used with both geometries. A final-to-minimum ratio above one indicates semiconvergence within the 10 000-iteration budget.
Table 3.
Table 4: FNO models trained on fewer fields, at λ J = 10 \lambda_J = 10 λ J = 10 , with the matching 10 000-field models for reference. Noiseless inversion with the baseline Matérn geometry. Errors E F E_F E F and E J E_J E J are validation errors. A dash means that the inversion was not run.
Table 4. Best relative coefficient error e m ( k o r ) e_m(k_{\rm or}) e m ( k or ) , in percent, for noiseless inversion with the baseline Matérn geometry (ν C = 3 / 2 \nu_C = 3/2 ν C = 3/2 ), the two geometries with ν C < 1 / 2 \nu_C < 1/2 ν C < 1/2 that satisfy the regularity condition for T1, and the identity. All Matérn geometries use ℓ C = 0.1 \ell_C = 0.1 ℓ C = 0.1 . The exact-FEM identity run was performed for T2 only.
The two smallest models semiconverge even with the Matérn geometry. At width and modes 8/4, with E F = 1.36 % E_F = 1.36\% E F = 1.36% , the error reaches its minimum of 11.79% at iteration 248 and ends at 14.29%. At 12/6 the minimum is reached at iteration 551 and the final error is 23% larger. Without preconditioning, the same models reach their minima after fewer than 120 iterations and end at 1.8 to 2.1 times the minimum. The behaviour disappears as the surrogate becomes more accurate. At 16/8 only a slight deterioration remains without preconditioning, and from 24/10 onwards, with E F ≤ 0.12 % E_F \le0.12\% E F ≤ 0.12% , every minimum lies at the end of the budget. Reducing the training data has the same effect (Table 4 ). At 8/4, training on 1000 instead of 10 000 fields doubles E F E_F E F to 2.65%, moves the minimum from iteration 248 to 52 and raises the final-to-minimum ratio from 1.21 to 1.73. At 12/6 the ratio rises from 1.23 to 1.38. For the larger models the iteration remains stable even with 1000 fields, and at 24/10 the error is insensitive to the training size. The deterioration is not specific to the structured truth, since the small models trained on 10 000 fields also semiconverge on T1, with ratios of 1.44 and 1.32. Semiconvergence without noise is therefore produced by surrogate error alone, as the level Δ \Delta Δ predicts. The preconditioner delays and reduces it
but does not remove it, and it lowers the final error of the small models but not their best error. Smaller models and less data degrade the forward and derivative accuracy together, so the study cannot tell whether ε o p \varepsilon_{\rm op} ε op or ε D , C \varepsilon_{D,C} ε D , C drives the semiconvergence; each model is also trained only once.
Preconditioners satisfying the regularity condition
The baseline geometry lies outside Theorem 5.15 , because neither truth satisfies m † − m 0 ∈ M C m^\dagger- m_0 \in\mathcal{M}_C m † − m 0 ∈ M C . For T1 the condition holds when ν C < 1 / 2 \nu_C < 1/2 ν C < 1/2 , since M C = H ν C + 1 ( Ω ) \mathcal{M}_C = H^{\nu_C+1}(\Omega) M C = H ν C + 1 ( Ω ) and a Matérn sample with smoothness 3 / 2 3/2 3/2 lies in H s ( Ω ) H^s(\Omega) H s ( Ω ) only for s < 3 / 2 s < 3/2 s < 3/2 . T2 remains outside M C \mathcal{M}_C M C for every ν C \nu_C ν C , since a coefficient with jumps across curves does not belong to H s ( Ω ) H^s(\Omega) H s ( Ω ) for s ≥ 1 / 2 s \ge1/2 s ≥ 1/2 . We repeat the noiseless experiments of § 7.5 with ν C ∈ { 0.25 , 0.4 } \nu_C \in\{0.25, 0.4\} ν C ∈ { 0.25 , 0.4 } and ℓ C ∈ { 0.1 , 0.5 , 1.0 } \ell_C \in\{0.1, 0.5, 1.0\} ℓ C ∈ { 0.1 , 0.5 , 1.0 } in the kernel (43) . The trained networks, data, initial field, step rule (42) and 10 000-iteration budget are unchanged.
Reconstruction accuracy. Table 5 compares the smallest coefficient error reached with ℓ C = 0.1 \ell_C = 0.1 ℓ C = 0.1 with the baseline and with the identity. With ℓ C = 0.1 \ell_C = 0.1 ℓ C = 0.1 , both values of ν C \nu_C ν C improve on the baseline for every method. For ν C = 0.25 \nu_C = 0.25 ν C = 0.25 , the exact-FEM error decreases from 4.12% to 3.34% on T1 and from 6.94% to 6.01% on T2, and the surrogate errors decrease by between 1.5% and 10.5% of their baseline values. Longer length scales, ℓ C = 0.5 \ell_C = 0.5 ℓ C = 0.5 and 1, generally increase the error (results not shown). The ordering of the surrogates on T1 is unchanged, and on T2 only FNO and GenFNO exchange places. The stability observed in § 7.8 is also retained. In every one of the 60 runs the final error is within 5% of the minimum, whereas without preconditioning GenFNO and GenDeepONet end at up to three times their minima. The conclusions drawn from the baseline geometry therefore also hold in geometries covered by the analysis.
Table 6. Diagnostics along the noiseless trajectories with ν C = 0.25 \nu_C = 0.25 ν C = 0.25 and ℓ C = 0.1 \ell_C = 0.1 ℓ C = 0.1 , evaluated every ten iterations. Here ζ k o r \zeta_{k_{\mathrm{or}}} ζ k or is the value of ζ k \zeta_k ζ k at the oracle iterate and k ζ k_{\zeta} k ζ the first evaluated iterate with ζ k ≥ 1 / 2 \zeta_k \ge1/2 ζ k ≥ 1/2 (a dash if there is none). The column e m ( k ζ ) / e m ( k o r ) e_m(k_{\zeta})/e_m(k_{\mathrm{or}}) e m ( k ζ ) / e m ( k or ) is the error at k ζ k_{\zeta} k ζ divided by the oracle error. The M C \mathcal{M}_C M C column states whether ∥ e k ∥ M C \|e_k\|_{\mathcal{M}_C} ∥ e k ∥ M C is non-increasing up to the oracle iterate.
Checking the hypotheses along the iteration. Since the truth is known, we can evaluate along the trajectories the two quantities on which the descent argument of Lemma 5.10 rests. With e k = m k − m h † e_k = m_k - m_h^\dagger e k = m k − m h † , the first is the cone quantity of the exact model, which the tangential cone condition requires to stay below 1 / 2 1/2 1/2 ,
η k = ∥ F ( m k ) − F ( m h † ) − D F ( m k ) e k ∥ W y ∥ F ( m k ) − F ( m h † ) ∥ W y . \eta_k = \frac{\|\mathcal{F}(m_k) - \mathcal{F}(m_h^\dagger) - D\mathcal{F}(m_k)e_k\|_{W_y}}{\|\mathcal{F}(m_k) - \mathcal{F}(m_h^\dagger)\|_{W_y}}. η k = ∥ F ( m k ) − F ( m h † ) ∥ W y ∥ F ( m k ) − F ( m h † ) − D F ( m k ) e k ∥ W y .
The second measures how well the linearisation of the model F ⋆ ∈ { F , F θ } \mathcal{F}_\star\in\{\mathcal{F},\mathcal{F}_{\theta}\} F ⋆ ∈ { F , F θ } that drives the iteration predicts its own residual R k = F ⋆ ( m k ) − y h R_k = \mathcal{F}_\star(m_k) - y_h R k = F ⋆ ( m k ) − y h in the direction of the error. Together with the relative step size, it is
ζ k = ∥ D F ⋆ ( m k ) e k − R k ∥ W y ∥ R k ∥ W y , s k = α k q k T d k ∥ R k ∥ W y 2 . \zeta_k = \frac{\|D\mathcal{F}_\star(m_k)e_k - R_k\|_{W_y}}{\|R_k\|_{W_y}}, \qquad s_k = \frac{\alpha_k q_k^{\mathsf T}d_k}{\|R_k\|_{W_y}^2}. ζ k = ∥ R k ∥ W y ∥ D F ⋆ ( m k ) e k − R k ∥ W y , s k = ∥ R k ∥ W y 2 α k q k T d k .
Repeating the expansion in the proof of Lemma 5.10 in M C \mathcal{M}_C M C , where ∥ d k ∥ M C 2 = q k T d k \|d_k\|_{\mathcal{M}_C}^2 = q_k^{\mathsf T}d_k ∥ d k ∥ M C 2 = q k T d k , gives
∥ e k + 1 ∥ M C 2 − ∥ e k ∥ M C 2 ≤ − α k ∥ R k ∥ W y 2 ( 1 − 2 ζ k ) whenever s k ≤ 1 , \|e_{k+1}\|_{\mathcal{M}_C}^2 - \|e_k\|_{\mathcal{M}_C}^2 \leq-\alpha_k\|R_k\|_{W_y}^2(1 - 2\zeta_k) \qquad\text{whenever } s_k \leq1, ∥ e k + 1 ∥ M C 2 − ∥ e k ∥ M C 2 ≤ − α k ∥ R k ∥ W y 2 ( 1 − 2 ζ k ) whenever s k ≤ 1 ,
provided neither the ℓ ∞ \ell^\infty ℓ ∞ cap nor the projection is active. In words, a step with ζ k < 1 / 2 \zeta_k < 1/2 ζ k < 1/2 and s k ≤ 1 s_k \leq1 s k ≤ 1 reduces the error in M C \mathcal{M}_C M C , so ζ k < 1 / 2 \zeta_k < 1/2 ζ k < 1/2 certifies descent at that step. By Lemma 5.9 , ζ k \zeta_k ζ k is at most η \eta η plus terms proportional to 1 / ∥ R k ∥ 1/\|R_k\| 1/∥ R k ∥ , so the certificate is expected to fail once the residual approaches the level set by discretisation mismatch and surrogate error. Both quantities require the truth, so they test the theory and do not give a stopping rule.
Table 6 gives the diagnostics for ν C = 0.25 \nu_C = 0.25 ν C = 0.25 and ℓ C = 0.1 \ell_C = 0.1 ℓ C = 0.1 ; the other five geometries behave in the same way. Three observations follow. First, the hypotheses on the exact model and the step hold: η k ≤ 0.171 \eta_k \leq0.171 η k ≤ 0.171 on T1 and η k ≤ 0.351 \eta_k \leq0.351 η k ≤ 0.351 on T2 in all 60 runs, the largest values occurring at the common initial field, and s k ≤ 0.41 s_k \le0.41 s k ≤ 0.41 throughout. Second, the certificate is lost much earlier with a surrogate. For the exact model ζ k \zeta_k ζ k reaches 1 / 2 1/2 1/2 late or not at all, whereas for the surrogates it does so in 43 of the 48 runs, after fewer than 700 iterations for the generalised architectures and latest for FNO.
Third, the certificate is conservative. The coefficient error continues to decrease well beyond the first crossing, and in the table stopping there would increase the error by 13%–22% for FNO and DeepONet and by 28%–76% for the generalised architectures. The error measured in M C \mathcal{M}_{C} M C , which Lemma 5.10 predicts to decrease, is non-increasing up to the oracle iterate in 54 of the 60 runs, including all exact-FEM runs, even on T2, which lies outside the theory. The six exceptions are the GenFNO runs with ℓ C = 0.1 \ell_C = 0.1 ℓ C = 0.1 , shown in the table, and two GenDeepONet runs on T2 with larger ℓ C \ell_C ℓ C , the surrogates whose certificate fails earliest. The ℓ ∞ \ell^\infty ℓ ∞ cap is active in at most 30 early iterations of each run, and the projection onto [ 0 , 200 ] [0, 200] [ 0 , 200 ] is active in up to about a third of the iterations in some T2 runs with DeepONet and GenDeepONet. Neither is covered by the analysis.
Computational cost
To keep the comparison fair, the exact-FEM baseline assembles the finite-element operator and its algebraic multigrid preconditioner once per iterate and reuses them for the forward, tangent and adjoint solves. A 10 000-iteration exact inversion took 1.7–2.2 hours on eight CPU cores, and a surrogate inversion 1.8–4.0 minutes on one NVIDIA L40 GPU, a reduction by factors of 26–34 for DeepONet, 34–44 for FNO, 49–63 for GenFNO and 56–72 for GenDeepONet. These are loop times at equal iteration counts, not at equal accuracy, comparing a CPU and a GPU implementation without hardware normalisation. Offline, the 15 000 training, validation and test fields with their derivative labels required 15 000 forward and 240 000 tangent solves, about 1.06 hours of serial-equivalent computation, and training the selected models took between 2.5 and 27 hours on one A100 GPU, excluding hyperparameter search (for FNO and GenDeepONet the time of the run with λ J = 100 \lambda_J = 100 λ J = 100 is used, since the timing records of the selected runs were not retained).
Observational noise and empirical stopping
We finally examine how observational noise, surrogate error and stopping interact. For each truth we add one fixed noise realisation at relative levels 1%, 2% and 5%, the same for the exact and surrogate inversions, so these are paired case studies rather than averages over the noise. Figure 5 shows the trajectories for T1; those for T2 behave in the same way.
Figure 5. Noisy T1 inversions at 1%, 2% and 5% observational noise. Left: residual divided by the raw noise norm δ \delta δ ; circles mark the first raw-δ \delta δ crossing. Right: relative coefficient error; stars denote oracle minima. Surrogate curves use the surrogate residual and should not be interpreted as FEM re-evaluated physical residuals.
The analysis stops at the first iterate with ∥ R k ∥ ≤ τ Δ \lVert R_k\rVert\le\tau\Delta ∥ R k ∥ ≤ τ Δ , with τ > 2 ( 1 + η ) / ( 1 − 2 η ) \tau> 2(1+\eta)/(1-2\eta) τ > 2 ( 1 + η ) / ( 1 − 2 η ) . Since Δ \Delta Δ cannot be computed, we apply the rule retrospectively with the raw level, κ τ = min { k : ∥ R k ∥ W y < τ δ } \kappa_\tau= \min\{k : \lVert R_k\rVert_{W_y} < \tau\delta\} κ τ = min { k : ∥ R k ∥ W y < τ δ } , for τ ∈ { 1 , 1.5 , 2 } \tau\in\{1,1.5,2\} τ ∈ { 1 , 1.5 , 2 } (Table 7 ). With τ = 1 \tau= 1 τ = 1 the surrogate iterations often never stop. At low noise their residuals level off above δ \delta δ , since a learned model cannot fit the data more closely than its own error allows, which is what Δ \Delta Δ accounts for. None of the eight surrogate runs reaches δ \delta δ at 1% noise, although the exact runs do, two of eight do at 2% and all do at 5%. Whenever the rule stops, it stops within 10% of the oracle error. With τ = 1.5 \tau= 1.5 τ = 1.5 or 2 the rule always stops, but within the first 100 iterations, where all models still follow the exact iteration closely, so every model stops at about the same error, 1.5 to 2.3 times the exact-FEM oracle error. This early stopping reflects the discrepancy rule on this problem rather than surrogate error. The values of τ \tau τ admitted by the analysis are larger still: for the largest η k \eta_k η k measured in § 7.9 , 0.17 on T1 and 0.35 on T2, they exceed 3.5 and 9.
Table 7. Retrospective discrepancy stopping with the raw noise level. Entries are relative coefficient errors, at the oracle iterate and at the stopping index k τ k_\tau k τ . Surrogate rows give the range over the four surrogates. When only some of them reach the threshold, the range is over those that do, and their number is given in brackets. For example, (2/4) means that two of the four surrogates reach it. A dash means that the threshold is not reached within 10 000 iterations. At 5% noise on T1 with τ = 2 \tau=2 τ = 2 the threshold is met by the initial field.
Beyond the oracle iterate the exact-FEM error grows fastest, more than doubling by the end of the run at 5% noise on T1, possibly because a surrogate that resolves fine scales less accurately also fits less of the noise; with a single noise realisation we note this only as an observation. At 5% noise the oracle errors of the exact and surrogate iterations differ by less than 0.4 percentage points on both truths, so surrogate error need not dominate once the noise is appreciable.
Can held-out surrogate errors be used for stopping? The level Δ \Delta Δ involves uniform error bounds, so we tested whether thresholds built from held-out test errors could replace it on the 18 noiseless T2 surrogate trajectories that semiconverge. Forward-error thresholds were missed by some trajectories, even at the maximum test error, and thresholds combining forward and derivative errors as in the theory were always crossed, but about 3.3 × 10 3 3.3 \times10^{3} 3.3 × 1 0 3 iterations too early, with errors 1.4–1.8 times the best attainable; finite-test errors are therefore not informative about when to stop, which remains an open question in the noiseless, surrogate-limited regime.
Transient Darcy flow
The reaction benchmark compares four architectures on a problem where exact derivatives are cheap. This example shows that the approach carries over to a time-dependent problem with sparse space–time data, using a single standard architecture: a DeepONet, whose trunk can be evaluated at any space–time point, so that only the observed values need to be computed during the inversion. The example lies outside the assumptions of § 5 , since the surrogate is only piecewise smooth and the truth is rougher than M C \mathcal{M}_{C} M C .
The unknown is a log-permeability m m m on Ω = ( 0 , 1 ) 2 \Omega=(0,1)^2 Ω = ( 0 , 1 ) 2 . For 0 < t ≤ T = 0.2 0<t\le T=0.2 0 < t ≤ T = 0.2 , the pressure p p p satisfies
∂ t p − ∇ ⋅ ( e m ∇ p ) = f in Ω × ( 0 , T ] , e m ∇ p ⋅ n = 0 on ∂ Ω × ( 0 , T ] , p ( ⋅ , 0 ) = 1. (45) \partial_t p-\nabla\cdot\left(e^m\nabla p\right)=f \quad\text{in }\Omega\times(0,T], \qquad e^m\nabla p\cdot n=0 \quad\text{on }\partial\Omega\times(0,T], \qquad p(\cdot,0)=1. \tag*{(45)} ∂ t p − ∇ ⋅ ( e m ∇ p ) = f in Ω × ( 0 , T ] , e m ∇ p ⋅ n = 0 on ∂ Ω × ( 0 , T ] , p ( ⋅ , 0 ) = 1. (45)
The source f f f consists of six injectors and six producers of unit rate, modelled by normalised Gaussians of width 0.04, so that the net source vanishes. The pressure is observed at 24 sensors (the well centres and twelve further observation wells) at the 51 times t ℓ = 0.004 ℓ t_\ell= 0.004\,\ell t ℓ = 0.004 ℓ , giving 1224 data, and F ( m ) \mathcal{F}(m) F ( m ) denotes this vector of observations. The tangent and adjoint equations are given in Appendix § 13 .
Discretisation and training. The parameter lives on a 120 × 120 120 \times120 120 × 120 nodal grid, and the pressure uses continuous piecewise-linear elements on a 256 × 256 256 \times256 256 × 256 -cell mesh with 400 backward-Euler steps. Training coefficients are zero-mean Gaussian Matérn fields with variance 0.25, ν ∈ { 1 , 1.5 , 2 } \nu\in\{1,1.5,2\} ν ∈ { 1 , 1.5 , 2 } and ℓ ∈ { 0.08 , … , 0.30 } \ell\in\{0.08,\ldots,0.30\} ℓ ∈ { 0.08 , … , 0.30 } . We use 20 000 training, 2 000 validation and 2 000 test fields, each with pressures at the sensors and on a 65 × 65 65 \times65 65 × 65 grid, and directional derivatives along 8 of 32 empirical KL directions. The surrogate is a standard DeepONet of about 1.55 × 10 7 1.55 \times10^{7} 1.55 × 1 0 7 parameters, with a convolutional branch acting on m m m and a trunk acting on space-time coordinates. The branch uses ReLU activations and max pooling, so the surrogate is only piecewise smooth in m m m . It is trained with the pressure loss and a directional-derivative loss of weight λ J = 1 \lambda_J = 1 λ J = 1 (not tuned); for comparison, the same architecture is trained with the pressure loss alone. On the test fields, the median relative forward error over all sensors and times is 0.61% with derivative training and 0.66% without it, whereas the median relative error of the directional derivatives falls from 8.8% to 4.0%, and derivative training reduces it on all but 3 of the 16 000 field-direction pairs.
Inversion setup. The truth is a validation field, not used to fit the network weights. Data are observed on 0 ≤ t ≤ 0.2 0 \le t \le0.2 0 ≤ t ≤ 0.2 and generated on a refined mesh with 512 × 512 512 \times512 512 × 512 cells and 800 steps, so that no method inverts its own discretisation. Noisy data add a single fixed noise realisation, scaled so that δ = ϱ ∥ y 0 − 1 ∥ \delta= \varrho\lVert y^0 - 1 \rVert δ = ϱ ∥ y 0 − 1 ∥ for ϱ ∈ { 1 , 2 , 5 } % \varrho\in\{1,2,5\}\% ϱ ∈ { 1 , 2 , 5 } % , where y 0 y^0 y 0 are the clean observations. Both iterations use the nodal Matérn preconditioner with ν C = 0.5 \nu_C = 0.5 ν C = 0.5 and ℓ C = 0.1 \ell_C = 0.1 ℓ C = 0.1 , so that M C = H 3 / 2 \mathcal{M}_C = H^{3/2} M C = H 3/2 , whereas the truth was drawn with ν = 1 \nu= 1 ν = 1 and ℓ = 0.135 \ell= 0.135 ℓ = 0.135 and lies outside it, the step rule (41) with each nodal update capped at 0.75, projection onto [ − 2.8 , 2.8 ] [-2.8,2.8] [ − 2.8 , 2.8 ] and m 0 ≡ 0 m_0 \equiv0 m 0 ≡ 0 . The exact iteration was run for 1000 steps and the surrogate iteration for 20 000; with noise we report the first iterate whose own residual falls below δ \delta δ (τ = 1 \tau= 1 τ = 1 ), and without noise the last iterate.
The aim is prediction as well as recovery. Each reconstruction is simulated with the finite-element model up to t = 0.4 t = 0.4 t = 0.4 , with unchanged rates (S1) and with a changed schedule (S2) in which, from t = 0.2 t = 0.2 t = 0.2 , three injectors and three producers double their rates and the others are shut in. We report the pooled relative error ∥ p r e c − p t ∥ / ∥ p t − 1 ∥ \lVert p_{\mathrm{rec}} - p^t \rVert/\lVert p^t - 1 \rVert ∥ p rec − p t ∥ / ∥ p t − 1 ∥ of the pressures at the twelve injection and production wells. The forecasts are finite-element simulations of the recovered field, since the surrogate was trained only up to t = 0.2 t = 0.2 t = 0.2 .
Results. With noisy data the surrogate iteration stops by the discrepancy principle at every noise level, and its reconstruction error is within 1.3 percentage points of the exact iteration (Table 8 ); the reconstructions are visually almost indistinguishable (Figure 6). The well pressures agree as closely: both methods reproduce the history to about 1%, forecast S1 to below 1% and the changed schedule S2 to 1.3–3.2%, with no systematic advantage for either (Figure 7 ). The error in the recovered field is large, between 44% and 64%, because only 24 sensors inform a rough field, but the predictions of interest are accurate, including under a schedule that the data do not contain.
Table 8: Transient Darcy flow: exact FEM and DeepONet-surrogate Landweber iteration. With noise, the first iterate whose own residual is below δ \delta δ ; without noise, the last iterate. Time is the measured loop time (exact on CPU, surrogate on one L40 GPU). r k / δ r_k/\delta r k / δ is the FEM residual of the stopped iterate relative to the noise level. Well-pressure errors are relative errors at the twelve injection and production wells, from FEM simulation of the recovered field, over the history window and for the forecasts S1 (unchanged rates) and S2 (changed rates) on 0.2 < t ≤ 0.4 0.2 < t \le0.4 0.2 < t ≤ 0.4 .
Table 8. Figure 6. Transient Darcy flow: truth (left) and reconstructions by the exact (top) and DeepONet (bottom) iterations at each noise level, on a common colour scale, at the iterates of Table 8.
Table 5.
Figure 7. Transient Darcy flow: pressures at two injectors and two producers, chosen before the forecasts were computed, from FEM simulation of the fields recovered from data with 2% noise. Top: S1, unchanged rates. Bottom: S2, where from t = 0.2 t = 0.2 t = 0.2 the first injector and producer double their rates and the second are shut in. The dashed line marks the end of the data window.
It is worth noting three further points. First, the surrogate stops on its own residual, and the FEM residual of the stopped iterate exceeds δ \delta δ by 7%, 2% and 0.4% at the three noise levels, so the discrepancy principle holds only approximately in the physical model. Second, the surrogate iteration needs 25, 8 and 3 times as many steps as the exact one, consistent with the level Δ \Delta Δ , which makes the iteration slow down as δ \delta δ approaches the surrogate error. Third, without noise the surrogates trained with and without the derivative loss reach 44.7% and 44.9%, against 43.7% for the exact iteration, so derivative training, which halves the derivative error, does not change the reconstruction, as in the reaction benchmark.
Cost. A surrogate step takes about 15 ms on the GPU, against about 23 s for an exact step (two forward, one tangent and one adjoint solve over 400 time steps). Despite the larger number of steps, the time to stopping falls from 2.7 h, 1.9 h and 16 min to 2.7 min, 35 s and 3 s at 1%, 2% and 5% noise; the exact solver ran serially, so these ratios compare two implementations, not two optimised codes. Offline, generating the 24 000 fields with derivative labels took 1453 allocated CPU-core hours, as much as about 66, 96 and 660 exact inversions, and training took 19 GPU hours. The surrogate therefore pays off when the inverse problem is solved repeatedly with different data from the same wells, and, since this cost is paid before the data arrive, when the field and its predictions must be updated as new measurements come in. The surrogate can also be queried away from the sensors without retraining: on 200 validation fields its median pressure error on the 65 × 65 65 \times65 65 × 65 training grid is 0.91%, against 0.62% at the sensors, and a full-grid evaluation takes 0.16 s against 3.4 s for a finite-element solve, so pressures at, for instance, planned observation wells are available at a small fraction of the cost (we have checked the accuracy only on the training grid).
Conclusions
We have studied nonlinear Landweber iteration in which the forward map and its derivative are replaced by a differentiable neural-operator surrogate. Learning the inverse map directly is not a viable alternative: near a discontinuity, no continuous network approximates it uniformly (Proposition 2.2 ). The forward map, by contrast, can be learned together with its derivative by encoder–network–decoder operators, a class that generalised DeepONets and fully coupled Fourier operators realise exactly. For such a surrogate, a discrepancy principle with the level Δ \Delta Δ , which combines the noise level with the forward and derivative errors, gives descent before stopping and finite termination. With a compact preconditioner, the stopped reconstructions converge as the data and surrogate errors vanish, provided m † − m 0 ∈ M C m^{\dagger} - m_{0} \in\mathcal{M}_{C} m † − m 0 ∈ M C (Theorem 5.15 ).
The experiments support the central mechanism of the analysis: surrogate error acts like data noise. Without preconditioning, three of the four surrogates in the reaction benchmark semiconverge early even with exact data, and develop spurious fine-scale structure, whereas the exact iteration does not. The compact preconditioner removes this instability by damping the poorly learned components of the adjoint direction, but it also slows the recovery of sharp features. Derivative-informed training along a fixed set of directions reduced the derivative test error by 45–94% but hardly changed the best reconstructions, in both the reaction and the Darcy examples; its clearer effect was to delay semiconvergence. What matters for inversion is therefore not the derivative accuracy along the training directions but the accuracy of the learned adjoint direction in the geometry of the iteration. Once the offline cost is paid, the surrogate iterations were between about 25 and 300 times faster than the finite-element ones, which pays off when the inverse problem is solved repeatedly or must be updated as new data arrive.
Several questions remain open. The level Δ \Delta Δ involves uniform error bounds that are not computable in practice, and thresholds built from held-out test errors did not identify when to stop in the noiseless, surrogate-limited regime. A posteriori estimates of the surrogate error along the inverse trajectory, possibly combined with occasional finite-element evaluations, are a natural next step. A second direction is training that targets the weighted derivative error ( D F θ − D F ) C 1 / 2 (D\mathcal{F}_{\theta} - D\mathcal{F})\mathcal{C}^{1/2} ( D F θ − D F ) C 1/2 directly, so that the training and inversion geometries are aligned. The analysis is carried out in Hilbert spaces; its extension to Banach spaces along the lines of Schuster et al. [37 ] would allow penalties better suited to sharp interfaces. Finally, although a discontinuous inverse map cannot be learned directly, a regularised neural operator that balances the approximation and noise-propagation errors may be feasible.
Declarations
Throughout this work, generative AI was used to polish the wording and correct grammar, to proofread parts of the manuscript and suggest clarifications and modifications, to draft figure captions and parts of the descriptions of the numerical experiments, and to write all software used in this paper, following our specifications, including the finite-element solvers, the neural architectures, the training code and the Landweber implementations. The tools were ChatGPT (OpenAI; Astra) and OpenAI Codex (based on GPT-6; desktop application version 26.1016.62342) for the software, and Claude (Anthropic; Opus 5.5) for the text. No AI system ran jobs on the computing cluster, and every reported number was computed by archived scripts from saved results. The solvers were checked by mesh refinement, finite differences and the adjoint identity. The research questions, the theoretical framework and results, the choice of problems and architectures, and all decisions on what to report are our own. We carefully checked all AI-generated content and suggestions and take full responsibility for all mathematical content, software, numerical results, interpretations, and conclusions.
Code and data availability. The code for the reaction benchmark, including scripts that regenerate all its tables and figures, is at https://github.com/Marco-Iglesias-Nottingham/neural-operator-landweber-reaction, with data, trained models and saved results at https://doi.org/10.5281/zenodo.23767592. The code and data for the Darcy example are at https://github.com/Marco-Iglesias-Nottingham/neural-operator-landweber-darcy and https://doi.org/10.5281/zenodo.23879298.
Acknowledgments and Disclosure of Funding
K. Gai and S. Lu were supported by National Key Research and Development Programs of China (No. 2023YFA1009103), NSFC (No. 12631015 and No. 92570106). Simulations were performed using the Sulis Tier 2 HPC platform hosted by the Scientific Computing Research Technology Platform at the University of Warwick. Sulis is funded by EPSRC, United Kingdom, Grant EP/T022108/1 and the HPC Midlands+ consortium.
Approximation proofs
This appendix gives full proofs of Theorems 3.1 and 3.2 . Both proofs follow the same four steps: finite-rank truncation of input and output, control of the derivative of the truncated operator, reduction to a map between Euclidean spaces, and approximation of that map by a network.
Background results
Definition A.1 (Approximation property). A Banach space X X X has the approximation property if, for every compact set K ⊂ X K \subset X K ⊂ X and every ϵ > 0 \epsilon> 0 ϵ > 0 , there is a bounded finite-rank operator T : X → X T : X \to X T : X → X with sup x ∈ K ∥ x − T x ∥ X ≤ ϵ \sup_{x \in K} \lVert x - Tx \rVert_X \le\epsilon sup x ∈ K ∥ x − T x ∥ X ≤ ϵ .
Lemma A.2 ([23 Lemma 18] ). Every Banach space with a Schauder basis has the approximation property. More precisely, if P n P_n P n are the partial-sum projections of the basis, then L : = sup n ∥ P n ∥ < ∞ L := \sup_n \lVert P_n \rVert< \infty L := sup n ∥ P n ∥ < ∞ and P n → I P_n \to I P n → I uniformly on every compact set.
Lemma A.3 ([23 Lemma 21] ). Let X , Y X,Y X , Y be Banach spaces, K ⊂ X K \subset X K ⊂ X compact, and T , T n : X → Y T,T_n : X \to Y T , T n : X → Y continuous with sup x ∈ K ∥ T ( x ) − T n ( x ) ∥ Y → 0 \sup_{x \in K} \lVert T(x) - T_n(x) \rVert_Y \to0 sup x ∈ K ∥ T ( x ) − T n ( x ) ∥ Y → 0 . Then T ( K ) ∪ ⋃ n ≥ 1 T n ( K ) T(K) \cup\bigcup_{n \ge1} T_n(K) T ( K ) ∪ ⋃ n ≥ 1 T n ( K ) is compact in Y Y Y .
Lemma A.4 (Finite-dimensional C 1 C^1 C 1 approximation). Let σ ∈ C 1 ( R ) \sigma\in C^1(\mathbb{R}) σ ∈ C 1 ( R ) be nonpolynomial , let V ⊂ R r V \subset\mathbb{R}^r V ⊂ R r be open, let g ∈ C 1 ( V , R s ) g \in C^1(V,\mathbb{R}^s) g ∈ C 1 ( V , R s ) , and let B ⊂ V B \subset V B ⊂ V be compact. For every ϵ > 0 \epsilon> 0 ϵ > 0 there is a one-hidden-layer network f θ ( z ) = W 2 σ ( W 1 z + b 1 ) + b 2 f_\theta(z) = W_2\sigma(W_1z + b_1) + b_2 f θ ( z ) = W 2 σ ( W 1 z + b 1 ) + b 2 , with σ \sigma σ applied componentwise, such that sup z ∈ B ∥ g ( z ) − f θ ( z ) ∥ < ϵ \sup_{z\in B}\lVert g(z) - f_\theta(z)\rVert< \epsilon sup z ∈ B ∥ g ( z ) − f θ ( z )∥ < ϵ and sup z ∈ B ∥ D g ( z ) − D f θ ( z ) ∥ < ϵ \sup_{z\in B}\lVert Dg(z) - Df_\theta(z)\rVert< \epsilon sup z ∈ B ∥ D g ( z ) − D f θ ( z )∥ < ϵ .
Proof. Let χ ∈ C ∞ ( V ) \chi\in C^\infty(V) χ ∈ C ∞ ( V ) equal one on a neighbourhood of B B B . Extended by zero, χ g \chi g χ g belongs to C 1 ( R r , R s ) C^1(\mathbb{R}^r,\mathbb{R}^s) C 1 ( R r , R s ) and coincides with g g g near B B B . By Pinkus [33 ] Theorem 4.1), for σ ∈ C 1 \sigma\in C^1 σ ∈ C 1 nonpolynomial, linear combinations of ridge functions σ ( w ⋅ z + b ) \sigma(w \cdot z + b) σ ( w ⋅ z + b ) are dense in C 1 ( R r ) C^1(\mathbb{R}^r) C 1 ( R r ) uniformly on compact sets. Apply this to each component of χ g \chi g χ g with a tolerance small enough that the Euclidean and Frobenius norms of the value and Jacobian errors are below ϵ \epsilon ϵ on B B B , and stack the s s s hidden layers into one.
Proof of Theorem 3.1
Throughout, we let ϵ > 0 \epsilon> 0 ϵ > 0 be arbitrary and write D ( m ) : = D F ( m ) D(m) := D\mathcal{F}(m) D ( m ) := D F ( m ) .
Step 1: finite-rank truncations and compact sets. By the approximation property of M \mathcal{M} M , applied with tolerances 1 / n 1/n 1/ n on the compact set K ∪ H \mathcal{K} \cup H K ∪ H , there are bounded finite-rank operators P n : M → M P_n : \mathcal{M} \to\mathcal{M} P n : M → M with ρ n : = sup m ∈ K ∥ m − P n m ∥ M → 0 \rho_n := \sup_{m\in\mathcal{K}}\lVert m - P_nm\rVert_{\mathcal{M}} \to0 ρ n := sup m ∈ K ∥ m − P n m ∥ M → 0 and ρ n H : = sup h ∈ H ∥ h − P n h ∥ M → 0 \rho_n^H := \sup_{h\in H}\lVert h - P_nh\rVert_{\mathcal{M}} \to0 ρ n H := sup h ∈ H ∥ h − P n h ∥ M → 0 . Since K \mathcal{K} K is compact and U U U open, dist ( K , M ∖ U ) > 0 \operatorname{dist}(\mathcal{K},\mathcal{M}\setminus U) > 0 dist ( K , M ∖ U ) > 0 ; discarding finitely many n n n , we may assume P n K ⊂ U P_n\mathcal{K} \subset U P n K ⊂ U for all n n n . By Lemma A.3, the sets Z : = K ∪ ⋃ n P n ( K ) ⊂ U Z := \mathcal{K} \cup\bigcup_n P_n(\mathcal{K}) \subset U Z := K ∪ ⋃ n P n ( K ) ⊂ U and Z H : = H ∪ ⋃ n P n ( H ) Z_H := H \cup\bigcup_n P_n(H) Z H := H ∪ ⋃ n P n ( H ) are compact. Hence F \mathcal{F} F and D D D are uniformly continuous on Z Z Z , with moduli of continuity ω F \omega_{\mathcal{F}} ω F and ω D \omega_D ω D , and L D : = sup a ∈ Z ∥ D ( a ) ∥ < ∞ L_D := \sup_{a\in Z}\lVert D(a)\rVert< \infty L D := sup a ∈ Z ∥ D ( a )∥ < ∞ , R H : = max { 1 , sup h ∈ Z H ∥ h ∥ M } < ∞ R_H := \max\{1,\sup_{h\in Z_H}\lVert h\rVert_{\mathcal{M}}\} < \infty R H := max { 1 , sup h ∈ Z H ∥ h ∥ M } < ∞ . The map ( a , h ) ↦ D ( a ) h (a,h) \mapsto D(a)h ( a , h ) ↦ D ( a ) h is continuous on Z × Z H Z \times Z_H Z × Z H , because ∥ D ( a ) h − D ( a ′ ) h ′ ∥ ≤ ∥ D ( a ) − D ( a ′ ) ∥ ∥ h ∥ + ∥ D ( a ′ ) ∥ ∥ h − h ′ ∥ \lVert D(a)h-D(a')h'\rVert\le\lVert D(a)-D(a')\rVert\lVert h\rVert+\lVert D(a')\rVert\lVert h-h'\rVert ∥ D ( a ) h − D ( a ′ ) h ′ ∥ ≤ ∥ D ( a ) − D ( a ′ )∥ ∥ h ∥ + ∥ D ( a ′ )∥ ∥ h − h ′ ∥ . Therefore W : = F ( Z ) ∪ { D ( a ) h : a ∈ Z , h ∈ Z H } W := \mathcal{F}(Z)\cup\{D(a)h : a \in Z,\ h \in Z_H\} W := F ( Z ) ∪ { D ( a ) h : a ∈ Z , h ∈ Z H } is compact in Y \mathcal{Y} Y . By the approximation property of Y \mathcal{Y} Y , choose a bounded finite-rank operator Q : Y → Y Q : \mathcal{Y} \to\mathcal{Y} Q : Y → Y with
sup y ∈ W ∥ y − Q y ∥ Y ≤ ϵ / 30. (46) \sup_{y\in W}\lVert y-Qy\rVert_{\mathcal{Y}} \le\epsilon/30. \tag*{(46)} y ∈ W sup ∥ y − Q y ∥ Y ≤ ϵ /30. (46)
Since Z Z Z and Z H Z_H Z H do not depend on the index n n n , neither does W W W , and n n n may be chosen after Q Q Q . Choose n n n so large that
ω F ( ρ n ) ≤ ϵ / 6 , ω D ( ρ n ) R H ≤ ϵ / 12 , L D ρ n H ≤ ϵ / 12 , (47) \omega_{\mathcal{F}}(\rho_n) \le\epsilon/6,\qquad\omega_D(\rho_n)R_H \le\epsilon/12,\qquad L_D\rho_n^H \le\epsilon/12, \tag*{(47)} ω F ( ρ n ) ≤ ϵ /6 , ω D ( ρ n ) R H ≤ ϵ /12 , L D ρ n H ≤ ϵ /12 , (47)
write P = P n P = P_n P = P n , and set F P , Q : = Q ∘ F ∘ P \mathcal{F}_{P,Q} := Q \circ\mathcal{F} \circ P F P , Q := Q ∘ F ∘ P on { m : P m ∈ U } \{m : Pm \in U\} { m : P m ∈ U } .
Step 2: accuracy of the truncated operator. For m ∈ K m \in\mathcal{K} m ∈ K , since P m ∈ Z Pm \in Z P m ∈ Z ,
∥ F ( m ) − F P , Q ( m ) ∥ Y ≤ ∥ F ( m ) − F ( P m ) ∥ Y + ∥ ( I − Q ) F ( P m ) ∥ Y ≤ ω F ( ρ n ) + ϵ / 30 < ϵ / 3. \lVert\mathcal{F}(m) - \mathcal{F}_{P,Q}(m)\rVert_{\mathcal{Y}} \le\lVert\mathcal{F}(m) - \mathcal{F}(Pm)\rVert_{\mathcal{Y}} + \lVert(I-Q)\mathcal{F}(Pm)\rVert_{\mathcal{Y}} \le\omega_{\mathcal{F}}(\rho_n) + \epsilon/30 < \epsilon/3. ∥ F ( m ) − F P , Q ( m ) ∥ Y ≤ ∥ F ( m ) − F ( P m ) ∥ Y + ∥( I − Q ) F ( P m ) ∥ Y ≤ ω F ( ρ n ) + ϵ /30 < ϵ /3.
By the chain rule, D F P , Q ( m ) = Q D ( P m ) P D\mathcal{F}_{P,Q}(m) = QD(Pm)P D F P , Q ( m ) = Q D ( P m ) P , and
D ( m ) − Q D ( P m ) P = ( I − Q ) D ( m ) + Q [ D ( m ) − D ( P m ) ] P + Q D ( m ) ( I − P ) . (48) D(m) - QD(Pm)P = (I-Q)D(m) + Q[D(m) - D(Pm)]P + QD(m)(I-P). \tag*{(48)} D ( m ) − Q D ( P m ) P = ( I − Q ) D ( m ) + Q [ D ( m ) − D ( P m )] P + Q D ( m ) ( I − P ) . (48)
Fix m ∈ K m \in\mathcal{K} m ∈ K and h ∈ H h \in H h ∈ H . We bound the three terms applied to h h h without using any bound on ∥ Q ∥ \lVert Q\rVert ∥ Q ∥ : in each case we add and subtract terms so that I − Q I-Q I − Q acts only on elements of W W W . First, D ( m ) h ∈ W D(m)h \in W D ( m ) h ∈ W , so ∥ ( I − Q ) D ( m ) h ∥ Y ≤ ϵ / 30 \lVert(I-Q)D(m)h\rVert_{\mathcal{Y}} \le\epsilon/30 ∥( I − Q ) D ( m ) h ∥ Y ≤ ϵ /30 . Second, since m , P m ∈ Z m, Pm \in Z m , P m ∈ Z and P h ∈ Z H Ph \in Z_H P h ∈ Z H , both D ( m ) P h D(m)Ph D ( m ) P h and D ( P m ) P h D(Pm)Ph D ( P m ) P h lie in W W W , and
∥ Q [ D ( m ) − D ( P m ) ] P h ∥ Y ≤ ∥ ( Q − I ) D ( m ) P h ∥ Y + ∥ [ D ( m ) − D ( P m ) ] P h ∥ Y + ∥ ( I − Q ) D ( P m ) P h ∥ Y ≤ 2 ϵ 30 + ω D ( ρ n ) R H . \lVert Q[D(m)-D(Pm)]Ph\rVert_Y \le\lVert(Q-I)D(m)Ph\rVert_Y+\lVert[D(m)-D(Pm)]Ph\rVert_Y
+\lVert(I-Q)D(Pm)Ph\rVert_Y \le\frac{2\epsilon}{30}+\omega_D(\rho_n)R_H. ∥ Q [ D ( m ) − D ( P m )] P h ∥ Y ≤ ∥( Q − I ) D ( m ) P h ∥ Y + ∥[ D ( m ) − D ( P m )] P h ∥ Y + ∥( I − Q ) D ( P m ) P h ∥ Y ≤ 30 2 ϵ + ω D ( ρ n ) R H .
Third, D ( m ) h D(m)h D ( m ) h and D ( m ) P h D(m)Ph D ( m ) P h lie in W W W , and
∥ Q D ( m ) ( h − P h ) ∥ Y ≤ ∥ ( Q − I ) D ( m ) h ∥ Y + ∥ D ( m ) ( h − P h ) ∥ Y + ∥ ( I − Q ) D ( m ) P h ∥ Y ≤ 2 ϵ 30 + L D ρ n H . \lVert QD(m)(h-Ph)\rVert_Y \le\lVert(Q-I)D(m)h\rVert_Y+\lVert D(m)(h-Ph)\rVert_Y+\lVert(I-Q)D(m)Ph\rVert_Y
\le\frac{2\epsilon}{30}+L_D\rho_n^H. ∥ Q D ( m ) ( h − P h ) ∥ Y ≤ ∥( Q − I ) D ( m ) h ∥ Y + ∥ D ( m ) ( h − P h ) ∥ Y + ∥( I − Q ) D ( m ) P h ∥ Y ≤ 30 2 ϵ + L D ρ n H .
Summing and using (47) gives sup m ∈ K , h ∈ H ∥ ( D ( m ) − D F P , Q ( m ) ) h ∥ Y ≤ 5 ϵ / 30 + ϵ / 12 + ϵ / 12 = ϵ / 3 \sup_{m\in K,\ h\in H}\lVert(D(m)-D_{F_{P,Q}}(m))h\rVert_Y \le5\epsilon/30+\epsilon/12+\epsilon/12=\epsilon/3 sup m ∈ K , h ∈ H ∥( D ( m ) − D F P , Q ( m )) h ∥ Y ≤ 5 ϵ /30 + ϵ /12 + ϵ /12 = ϵ /3 .
Proof of Theorem 3.2
Again let ϵ > 0 \epsilon>0 ϵ > 0 and write D ( m ) : = D F ( m ) D(m):=DF(m) D ( m ) := D F ( m ) .
Step 1: basis projections. Let ( e j ) (e_j) ( e j ) be a Schauder basis of M \mathcal{M} M with coordinate functionals ( e j ∗ ) ⊂ M ∗ (e_j^*)\subset\mathcal{M}^* ( e j ∗ ) ⊂ M ∗ , and ( u j ) (u_j) ( u j ) a Schauder basis of Y \mathcal{Y} Y with coordinate functionals ( u j ∗ ) (u_j^*) ( u j ∗ ) . Let P n x = ∑ j = 1 n e j ∗ ( x ) e j P_nx=\sum_{j=1}^n e_j^*(x)e_j P n x = ∑ j = 1 n e j ∗ ( x ) e j and Q N y = ∑ j = 1 N u j ∗ ( y ) u j Q_Ny=\sum_{j=1}^N u_j^*(y)u_j Q N y = ∑ j = 1 N u j ∗ ( y ) u j be the partial-sum projections, and let L ≥ 1 L\ge1 L ≥ 1 bound ∥ P n ∥ \lVert P_n\rVert ∥ P n ∥ and ∥ Q N ∥ \lVert Q_N\rVert ∥ Q N ∥ for all n , N n,N n , N (Lemma A.2). Then P n → I P_n\to I P n → I uniformly on compact subsets of M \mathcal{M} M , and Q N → I Q_N\to I Q N → I uniformly on compact subsets of Y \mathcal{Y} Y . Moreover P n ∗ f = ∑ j ≤ n f ( e j ) e j ∗ P_n^*f=\sum_{j\le n}f(e_j)e_j^* P n ∗ f = ∑ j ≤ n f ( e j ) e j ∗ for f ∈ M ∗ f\in\mathcal{M}^* f ∈ M ∗ . Since M \mathcal{M} M is reflexive, every Schauder basis of M \mathcal{M} M is shrinking ([1 ] ), so ( e j ∗ ) (e_j^*) ( e j ∗ ) is a basis of M ∗ \mathcal{M}^* M ∗ and P n ∗ → I P_n^*\to I P n ∗ → I strongly on M ∗ \mathcal{M}^* M ∗ , with ∥ P n ∗ ∥ = ∥ P n ∥ ≤ L \lVert P_n^*\rVert=\lVert P_n\rVert\le L ∥ P n ∗ ∥ = ∥ P n ∥ ≤ L .
As in the previous proof, write ρ n = sup m ∈ K ∥ m − P n m ∥ \rho_n=\sup_{m\in K}\lVert m-P_nm\rVert ρ n = sup m ∈ K ∥ m − P n m ∥ , discard finitely many n n n so that P n K ⊂ U P_nK\subset U P n K ⊂ U , and let Z = K ∪ ⋃ n P n ( K ) ⊂ U Z=K\cup\bigcup_nP_n(K)\subset U Z = K ∪ ⋃ n P n ( K ) ⊂ U , which is compact by Lemma A.3. Let ω F , ω D \omega_F,\omega_D ω F , ω D be moduli of continuity of F F F and D D D on Z Z Z . For m ∈ K m\in K m ∈ K ,
∥ F ( m ) − Q N F ( P n m ) ∥ Y ≤ ω F ( ρ n ) + sup y ∈ F ( Z ) ∥ y − Q N y ∥ Y . (49) \lVert F(m)-Q_NF(P_nm)\rVert_Y\le\omega_F(\rho_n)+\sup_{y\in F(Z)}\lVert y-Q_Ny\rVert_Y. \tag*{(49)} ∥ F ( m ) − Q N F ( P n m ) ∥ Y ≤ ω F ( ρ n ) + y ∈ F ( Z ) sup ∥ y − Q N y ∥ Y . (49)
Step 2: the derivative in operator norm. The chain rule and the decomposition (48) , with P = P n P=P_n P = P n and Q = Q N Q=Q_N Q = Q N , give ∥ D ( m ) − Q N D ( P n m ) P n ∥ ≤ T 1 + T 2 + T 3 \lVert D(m)-Q_ND(P_nm)P_n\rVert\le T_1+T_2+T_3 ∥ D ( m ) − Q N D ( P n m ) P n ∥ ≤ T 1 + T 2 + T 3 , where T 1 = ∥ Q N [ D ( m ) − D ( P n m ) ] P n ∥ T_1=\lVert Q_N[D(m)-D(P_nm)]P_n\rVert T 1 = ∥ Q N [ D ( m ) − D ( P n m )] P n ∥ , T 2 = ∥ ( I − Q N ) D ( m ) ∥ T_2=\lVert(I-Q_N)D(m)\rVert T 2 = ∥( I − Q N ) D ( m )∥ and T 3 = ∥ Q N D ( m ) ( I − P n ) ∥ T_3=\lVert Q_ND(m)(I-P_n)\rVert T 3 = ∥ Q N D ( m ) ( I − P n )∥ .
First term. T 1 ≤ L 2 ω D ( ρ n ) T_1\le L^2\omega_D(\rho_n) T 1 ≤ L 2 ω D ( ρ n ) for all $m\in K. Second term. Let B M B_{\mathcal{M}} B M be the closed unit ball of M \mathcal{M} M . The set { D ( m ) : m ∈ K } \{D(m):m\in\mathcal{K}\} { D ( m ) : m ∈ K } is compact in L ( M , Y ) \mathcal{L}(\mathcal{M},\mathcal{Y}) L ( M , Y ) ; for η > 0 \eta>0 η > 0 choose a finite η \eta η -net D ( m 1 ) , … , D ( m t ) D(m_1),\ldots,D(m_t) D ( m 1 ) , … , D ( m t ) with m i ∈ K m_i\in\mathcal{K} m i ∈ K . Each D ( m i ) B M D(m_i)B_{\mathcal{M}} D ( m i ) B M is relatively compact because D ( m i ) D(m_i) D ( m i ) is compact, and every D ( m ) x D(m)x D ( m ) x with m ∈ K m\in\mathcal{K} m ∈ K , x ∈ B M x\in B_{\mathcal{M}} x ∈ B M lies within η \eta η of ⋃ i D ( m i ) B M \bigcup_i D(m_i)B_{\mathcal{M}} ⋃ i D ( m i ) B M . Hence S : = ⋃ m ∈ K D ( m ) B M S:=\bigcup_{m\in\mathcal{K}}D(m)B_{\mathcal{M}} S := ⋃ m ∈ K D ( m ) B M is totally bounded, so S ‾ \overline{S} S is compact, and sup m ∈ K T 2 = sup y ∈ S ∥ ( I − Q N ) y ∥ Y → 0 \sup_{m\in\mathcal{K}}T_2=\sup_{y\in S}\|(I-Q_N)y\|_{\mathcal{Y}}\to0 sup m ∈ K T 2 = sup y ∈ S ∥ ( I − Q N ) y ∥ Y → 0 as N → ∞ N\to\infty N → ∞ .
Third term. Since ∥ A ∥ = ∥ A ∗ ∥ \|A\|=\|A^*\| ∥ A ∥ = ∥ A ∗ ∥ , T 3 ≤ L ∥ D ( m ) ( I − P n ) ∥ = L ∥ ( I − P n ∗ ) D ( m ) ∗ ∥ T_3\le L\|D(m)(I-P_n)\|=L\|(I-P_n^*)D(m)^*\| T 3 ≤ L ∥ D ( m ) ( I − P n ) ∥ = L ∥ ( I − P n ∗ ) D ( m ) ∗ ∥ . By Schauder’s theorem, D ( m ) ∗ D(m)^* D ( m ) ∗ is compact, so D ( m ) ∗ B Y ∗ D(m)^*B_{\mathcal{Y}^*} D ( m ) ∗ B Y ∗ is relatively compact, and the uniformly bounded operators P n ∗ P_n^* P n ∗ converge to I I I uniformly on it. Hence ∥ ( I − P n ∗ ) D ( m ) ∗ ∥ → 0 \|(I-P_n^*)D(m)^*\|\to0 ∥ ( I − P n ∗ ) D ( m ) ∗ ∥ → 0 for each m ∈ K m\in\mathcal{K} m ∈ K . The set { D ( m ) ∗ : m ∈ K } \{D(m)^*:m\in\mathcal{K}\} { D ( m ) ∗ : m ∈ K } is compact in L ( Y ∗ , M ∗ ) \mathcal{L}(\mathcal{Y}^*,\mathcal{M}^*) L ( Y ∗ , M ∗ ) ; if D ( m 1 ) ∗ , … , D ( m t ) ∗ D(m_1)^*,\ldots,D(m_t)^* D ( m 1 ) ∗ , … , D ( m t ) ∗ is an η \eta η -net, then ∥ ( I − P n ∗ ) D ( m ) ∗ ∥ ≤ max i ∥ ( I − P n ∗ ) D ( m i ) ∗ ∥ + ( 1 + L ) η \|(I-P_n^*)D(m)^*\|\le\max_i\|(I-P_n^*)D(m_i)^*\|+(1+L)\eta ∥ ( I − P n ∗ ) D ( m ) ∗ ∥ ≤ max i ∥ ( I − P n ∗ ) D ( m i ) ∗ ∥ + ( 1 + L ) η , so sup m ∈ K T 3 → 0 \sup_{m\in\mathcal{K}}T_3\to0 sup m ∈ K T 3 → 0 as n → ∞ n\to\infty n → ∞ .
Choice of N N N and $n. The sets F ( Z ) \mathcal{F}(Z) F ( Z ) and S ‾ \overline{S} S do not depend on n n n . First choose N N N such that sup y ∈ F ( Z ) ∥ y − Q N y ∥ ≤ ϵ / 4 \sup_{y\in\mathcal{F}(Z)}\|y-Q_Ny\|\le\epsilon/4 sup y ∈ F ( Z ) ∥ y − Q N y ∥ ≤ ϵ /4 and sup K T 2 ≤ ϵ / 6 \sup_{\mathcal{K}}T_2\le\epsilon/6 sup K T 2 ≤ ϵ /6 . Then choose n n n such that ω F ( ρ n ) ≤ ϵ / 4 \omega_{\mathcal{F}}(\rho_n)\le\epsilon/4 ω F ( ρ n ) ≤ ϵ /4 , $2\omega_D(\rho_n)\le\epsilon/6$ and sup K T 3 ≤ ϵ / 6 \sup_{\mathcal{K}}T_3\le\epsilon/6 sup K T 3 ≤ ϵ /6 . With F n , N : = Q N ∘ F ∘ P n \mathcal{F}_{n,N}:=Q_N\circ\mathcal{F}\circ P_n F n , N := Q N ∘ F ∘ P n , (48) and the three bounds give sup m ∈ K ∥ F ( m ) − F n , N ( m ) ∥ Y ≤ ϵ / 2 \sup_{m\in\mathcal{K}}\|\mathcal{F}(m)-\mathcal{F}_{n,N}(m)\|_{\mathcal{Y}}\le\epsilon/2 sup m ∈ K ∥ F ( m ) − F n , N ( m ) ∥ Y ≤ ϵ /2 and sup m ∈ K ∥ D ( m ) − D F n , N ( m ) ∥ L ( M , Y ) ≤ ϵ / 2 \sup_{m\in\mathcal{K}}\|D(m)-D\mathcal{F}_{n,N}(m)\|_{\mathcal{L}(\mathcal{M},\mathcal{Y})}\le\epsilon/2 sup m ∈ K ∥ D ( m ) − D F n , N ( m ) ∥ L ( M , Y ) ≤ ϵ /2 .
Step 3: reduction to Euclidean spaces. Let Ψ : = ( e 1 ∗ , … , e N ∗ ) : M → R N \Psi:=(e_1^*,\ldots,e_N^*):\mathcal{M}\to\mathbb{R}^N Ψ := ( e 1 ∗ , … , e N ∗ ) : M → R N , S c : = ∑ j ≤ N c j e j S c:=\sum_{j\le N}c_je_j S c := ∑ j ≤ N c j e j , Λ : = ( u 1 ∗ , … , u N ∗ ) : Y → R N \Lambda:=(u_1^*,\ldots,u_N^*):\mathcal{Y}\to\mathbb{R}^N Λ := ( u 1 ∗ , … , u N ∗ ) : Y → R N and Φ c : = ∑ j ≤ N c j u j \Phi c:=\sum_{j\le N}c_ju_j Φ c := ∑ j ≤ N c j u j , so that P n = S Ψ P_n=S\Psi P n = S Ψ and Q N = Φ Λ Q_N=\Phi\Lambda Q N = ΦΛ . On the open set V : = { z ∈ R N : S z ∈ U } V:=\{z\in\mathbb{R}^N:S z\in U\} V := { z ∈ R N : S z ∈ U } , the map g ( z ) : = Λ F ( S z ) g(z):=\Lambda\mathcal{F}(S z) g ( z ) := Λ F ( S z ) is C 1 C^1 C 1 by the chain rule, with F n , N = Φ ∘ g ∘ Ψ \mathcal{F}_{n,N}=\Phi\circ g\circ\Psi F n , N = Φ ∘ g ∘ Ψ and D F n , N ( m ) = Φ D g ( Ψ m ) Ψ D\mathcal{F}_{n,N}(m)=\Phi Dg(\Psi m)\Psi D F n , N ( m ) = Φ D g ( Ψ m ) Ψ . The set B : = Ψ ( K ) ⊂ V B:=\Psi(\mathcal{K})\subset V B := Ψ ( K ) ⊂ V is compact.
Step 4: network approximation. Lemma A.4 gives a network f θ f_\theta f θ with value and Jacobian errors below δ ′ : = ϵ / ( 2 max { 1 , ∥ Φ ∥ } max { 1 , ∥ Ψ ∥ } ) \delta':=\epsilon/(2\max\{1,\|\Phi\|\}\max\{1,\|\Psi\|\}) δ ′ := ϵ / ( 2 max { 1 , ∥Φ∥ } max { 1 , ∥Ψ∥ }) on B B B . For F θ : = Φ ∘ f θ ∘ Ψ \mathcal{F}_\theta:=\Phi\circ f_\theta\circ\Psi F θ := Φ ∘ f θ ∘ Ψ and m ∈ K m\in\mathcal{K} m ∈ K , ∥ F n , N ( m ) − F θ ( m ) ∥ Y ≤ ∥ Φ ∥ δ ′ < ϵ / 2 \|\mathcal{F}_{n,N}(m)-\mathcal{F}_\theta(m)\|_{\mathcal{Y}}\le\|\Phi\|\delta'<\epsilon/2 ∥ F n , N ( m ) − F θ ( m ) ∥ Y ≤ ∥Φ∥ δ ′ < ϵ /2 and ∥ D F n , N ( m ) − D F θ ( m ) ∥ ≤ ∥ Φ ∥ δ ′ ∥ Ψ ∥ < ϵ / 2 \|D\mathcal{F}_{n,N}(m)-D\mathcal{F}_\theta(m)\|\le\|\Phi\|\delta'\|\Psi\|<\epsilon/2 ∥ D F n , N ( m ) − D F θ ( m ) ∥ ≤ ∥Φ∥ δ ′ ∥Ψ∥ < ϵ /2 . Together with Step 2, both errors are below ϵ \epsilon ϵ . □
Standard FNO calculus and Lipschitz estimates
Let F θ F N O : m ↦ u \mathcal{F}^{\mathrm{FNO}}_\theta:m\mapsto u F θ FNO : m ↦ u denote the full standard FNO map, with θ \theta θ collecting its trainable parameters. Consider bounded linear lifting and output maps P , Q P,Q P , Q , and layers indexed by t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 : v 0 = P m + p v_0=Pm+p v 0 = P m + p , z t = ( W t + K t ) v t + b t z_t=(W_t+K_t)v_t+b_t z t = ( W t + K t ) v t + b t and v t + 1 = σ ( z t ) v_{t+1}=\sigma(z_t) v t + 1 = σ ( z t ) , with output u = F θ F N O ( m ) = Q v T ( m ) + q u=\mathcal{F}^{\mathrm{FNO}}_\theta(m)=Qv_T(m)+q u = F θ FNO ( m ) = Q v T ( m ) + q . The derivative D F θ F N O ( m ) D\mathcal{F}^{\mathrm{FNO}}_\theta(m) D F θ FNO ( m ) is taken with respect to the input m m m , with θ \theta θ fixed. On a finite grid with a smooth activation, the chain rule gives δ v t + 1 = D t ( W t + K t ) δ v t \delta v_{t+1}=D_t(W_t+K_t)\delta v_t δ v t + 1 = D t ( W t + K t ) δ v t with D t w = σ ′ ( z t ) w D_tw=\sigma'(z_t)w D t w = σ ′ ( z t ) w , and therefore
D F θ F N O ( m ) = Q D T − 1 ( W T − 1 + K T − 1 ) ⋯ D 0 ( W 0 + K 0 ) P . (50) D\mathcal{F}^{\mathrm{FNO}}_\theta(m)=QD_{T-1}(W_{T-1}+K_{T-1})\cdots D_0(W_0+K_0)P. \tag*{(50)} D F θ FNO ( m ) = Q D T − 1 ( W T − 1 + K T − 1 ) ⋯ D 0 ( W 0 + K 0 ) P . (50)
For nonlinear lifting or decoding, their derivatives replace P , Q P,Q P , Q at the corresponding intermediate values. The same formula holds on function spaces whenever these compositions are Fréchet differentiable there, for example with appropriate smooth composition maps on a Sobolev algebra. Scalar smoothness and an L 2 L^2 L 2 mapping property alone do not establish Fréchet differentiability of the Nemytskii map L 2 → L 2 L^2\to L^2 L 2 → L 2 .
On L 2 L^2 L 2 of the torus, or a grid with a unitary discrete Fourier transform, a truncated multiplier satisfies ∥ K t v ∥ 2 = ∑ k ∈ Λ t ∥ R t ( k ) v ^ ( k ) ∥ 2 2 ≤ max k ∈ Λ t ∥ R t ( k ) ∥ 2 2 ∥ v ^ ∥ 2 \|K_tv\|^2=\sum_{k\in\Lambda_t}\|R_t(k)\hat{v}(k)\|_2^2\le\max_{k\in\Lambda_t}\|R_t(k)\|_2^2\|\hat{v}\|^2 ∥ K t v ∥ 2 = ∑ k ∈ Λ t ∥ R t ( k ) v ^ ( k ) ∥ 2 2 ≤ max k ∈ Λ t ∥ R t ( k ) ∥ 2 2 ∥ v ^ ∥ 2 . Also ∥ v ↦ W t v ∥ ≤ ∥ W t ∥ 2 \|v\mapsto W_tv\|\le\|W_t\|_2 ∥ v ↦ W t v ∥ ≤ ∥ W t ∥ 2 . If the activation is globally L σ L_\sigma L σ -Lipschitz, repeated composition consequently gives
Lip ( F θ F N O ) ≤ L Q L P L σ T ∏ t = 0 T − 1 ( ∥ W t ∥ 2 + max k ∈ Λ t ∥ R t ( k ) ∥ 2 ) . (51) \operatorname{Lip}(\mathcal{F}_{\theta}^{\mathrm{FNO}}) \le L_Q L_P L_\sigma^T \prod_{t=0}^{T-1} \left(\lVert W_t\rVert_2 + \max_{k\in\Lambda_t}\lVert R_t(k)\rVert_2\right). \tag*{(51)} Lip ( F θ FNO ) ≤ L Q L P L σ T t = 0 ∏ T − 1 ( ∥ W t ∥ 2 + k ∈ Λ t max ∥ R t ( k ) ∥ 2 ) . (51)
Here L P , L Q L_P,L_Q L P , L Q are the lifting and decoding Lipschitz constants; for an M M M -layer perceptron, L P ≤ L σ M − 1 ∏ j = 1 M ∥ P j ∥ 2 L_P \le L_\sigma^{M-1}\prod_{j=1}^{M}\lVert P_j\rVert_2 L P ≤ L σ M − 1 ∏ j = 1 M ∥ P j ∥ 2 , and similarly for Q Q Q . Biases cancel in layerwise differences. These are upper bounds, not estimates of the realised Lipschitz constant. They apply in the indicated L 2 L^2 L 2 or grid norms; a different norm requires the corresponding boundedness estimates.
Tangent and adjoint for the Darcy example
For a perturbation h h h , the tangent w = D p ( m ) h w = Dp(m)h w = D p ( m ) h of (45) satisfies
∂ t w − ∇ ⋅ ( e m ∇ w ) = ∇ ⋅ ( e m h ∇ p ) , ( e m ∇ w + e m h ∇ p ) ⋅ n = 0 , w ( ⋅ , 0 ) = 0 , \partial_t w-\nabla\cdot\left(e^m\nabla w\right)=\nabla\cdot\left(e^m h\nabla p\right),\qquad\left(e^m\nabla w+e^m h\nabla p\right)\cdot n=0,\qquad w(\cdot,0)=0, ∂ t w − ∇ ⋅ ( e m ∇ w ) = ∇ ⋅ ( e m h ∇ p ) , ( e m ∇ w + e m h ∇ p ) ⋅ n = 0 , w ( ⋅ , 0 ) = 0 ,
and ( D F ( m ) h ) ℓ j = w ( s j , t ℓ ) (DF(m)h)_{\ell j}=w(s_j,t_\ell) ( D F ( m ) h ) ℓ j = w ( s j , t ℓ ) . For a data-space vector r = ( r ℓ j ) r=(r_{\ell j}) r = ( r ℓ j ) , the adjoint z z z runs backwards in time,
− ∂ t z − ∇ ⋅ ( e m ∇ z ) = ∑ ℓ : t ℓ > 0 ∑ j r ℓ j δ s j δ t ℓ , e m ∇ z ⋅ n = 0 , z ( ⋅ , T + ) = 0 , -\partial_t z-\nabla\cdot\left(e^m\nabla z\right)=\sum_{\ell:t_\ell>0}\sum_j r_{\ell j}\delta_{s_j}\delta_{t_\ell},\qquad e^m\nabla z\cdot n=0,\qquad z(\cdot,T^+)=0, − ∂ t z − ∇ ⋅ ( e m ∇ z ) = ℓ : t ℓ > 0 ∑ j ∑ r ℓ j δ s j δ t ℓ , e m ∇ z ⋅ n = 0 , z ( ⋅ , T + ) = 0 ,
so that each observation time contributes a jump of z z z . With the L 2 ( Ω ) L^2(\Omega) L 2 ( Ω ) inner product on the parameter,
D F ( m ) ∗ r = − ∫ 0 T e m ∇ p ⋅ ∇ z d t . DF(m)^*r=-\int_0^T e^m\nabla p\cdot\nabla z\,\mathrm{d}t. D F ( m ) ∗ r = − ∫ 0 T e m ∇ p ⋅ ∇ z d t .
The observation at t = 0 t=0 t = 0 does not depend on m m m . Point evaluation is not bounded on H 1 ( Ω ) H^1(\Omega) H 1 ( Ω ) in two dimensions, so these formulas are formal. The computations use the discrete tangent and adjoint of the backward-Euler finite-element scheme, for which the corresponding identities hold exactly. The discrete adjoint reuses the forward LU factorisation, and the adjoint action is accumulated on the parameter grid through the transpose of the interpolation.