betaFeedback

WorksMathematics

NeRI: Neural Residual Iteration for Elliptic Operator Equations and Variational Inequalities

Abstract

In this work, we propose a so-called Neural Residual Iteration (NeRI), a weak-form neural-network framework for solving a class of elliptic operator equations and variational inequalities, including nonlinear and nonsymmetric problems that may not admit a global energy functional. The key idea is to construct, at each iteration, a state-dependent linear functional whose derivative with respect to the trial function recovers the current weak residual. A single gradient step is then performed on the residual, yielding a residual-driven neural update without requiring a global energy functional or an additional test network. From a function-space perspective, NeRI can be interpreted as a neural-tangent-kernel-preconditioned realization of classical residual iterations, thereby bridging classical iterative solvers and neural-network optimization. We further extend NeRI to elliptic variational inequalities (VIs) through penalized and constraint-preserving formulations. For wide two-layer networks under gradient flow, we prove finite-time lazy-training estimates and derive an H1H^1-error decomposition relative to the exact weak solution. Numerical experiments on several elliptic problems demonstrate the effectiveness of NeRI, particularly for nonsymmetric and low-regularity problems.

Introduction

Partial differential equations (PDEs), and elliptic equations in particular, play a fundamental role in modeling a wide range of stationary phenomena in science and engineering. Their accurate and efficient numerical solution has therefore been a central problem in scientific computing. Classical numerical methods, including finite difference, finite element, finite volume, and spectral methods, have been extensively studied for elliptic equations and have achieved great success in both theory and applications; see, e.g., [4, 9] and the references therein. In particular, variational formulations and finite element methods provide a powerful framework for general elliptic boundary value problems and extend naturally to elliptic variational inequalities arising from unilateral or convex constraints, such as obstacle and contact problems [24, 15]. However, traditional mesh-based methods may become computationally expensive for high-dimensional problems, complex geometries, and multiscale structures.

Neural-network-based methods offer an alternative by representing the unknown solution as a trainable function rather than constructing it on a prescribed mesh. Physics-informed neural networks (PINNs) [36] incorporate the differential equation, boundary conditions, and available data into a residual-based training objective, using automatic differentiation [5] to evaluate the governing equations. These strong-form methods are flexible, but may require expensive evaluation of higher-order derivatives [29] and can be less effective for solutions with limited regularity [19, 18].

For elliptic problems, weak and variational formulations provide an important alternative. Compared with strong-form residual methods, weak-form approaches usually require only lower-order derivatives of the neural trial function and are therefore better suited to solutions with limited regularity. The deep Ritz method [11] uses an energy functional associated with the PDE as the loss function and applies to elliptic problems admitting a variational potential, including linear symmetric problems and certain nonlinear potential equations. Its applicability is limited, however, when the underlying operator is non-potential and no suitable global energy functional is available. To go beyond this setting, the iterative deep Ritz method (IDRM) [17] treats monotone elliptic problems, including non-potential operators, through a sequence of surrogate variational problems. Another weak-form approach is the weak adversarial network (WAN) [43], which parameterizes both trial and test functions and casts the PDE into an operator-norm or minimax formulation. These approaches broaden the applicability of neural solvers, but also introduce additional computational or optimization challenges: IDRM requires solving a sequence of surrogate neural minimization problems, while WAN typically involves two networks and a minimax optimization. Related developments and numerical analyses of neural PDE methods can be found, for example, in [39, 14, 25, 42, 22, 21, 10, 41]. Data-driven operator-learning approaches, such as DeepONet [28, 37] and Fourier neural operators [27, 26], provide another related direction; see [33, 38, 32] for recent reviews.

Recently, neural methods have also been developed for elliptic variational inequalities, where constraints and potentially nonsmooth solutions introduce further challenges. Existing approaches include energy-based and penalty formulations for obstacle problems [45, 6], weak adversarial methods based on regularized gap functions [2], and Prox-PINNs based on proximal reformulations [13]. In particular, the weak adversarial approach in [2] accommodates nonsymmetric operators, while Prox-PINNs [13] use proximal operators to reformulate VIs as nonlinear equations.

Alongside these methodological developments, the neural tangent kernel (NTK) provides a framework for understanding neural-network training through function-space dynamics [20]. Lazy-training theory relates these dynamics to those of the model linearized at initialization [7]. For PINNs, Wang et al. [41] use the NTK to analyze discrepancies in the convergence rates of different loss components. Jin and Wu [22] establish empirical-loss convergence for wide two-layer PINNs trained by stochastic gradient methods for the Poisson equation, relying on the positive definiteness of suitable Gram matrices. Related kernel-based work studies Sobolev-norm accuracy and statistical optimality for learning elliptic equations [31].

In this work, we propose Neural Residual Iteration (NeRI), a framework based directly on the weak operator residual. For an elliptic operator equation A(u)=f\mathcal{A}(u)=f, we construct, at each iterate uku_k, a state-dependent linear functional whose derivative with respect to the trial function recovers A(uk)−f\mathcal{A}(u_k)-f. In the neural parameterization, a single gradient step is then performed, and the residual is re-evaluated at the updated approximation. Unlike IDRM, the state-dependent functional in NeRI is not treated as a surrogate variational problem to be minimized; it is used only to generate the current residual-driven update. Thus, NeRI has no inner minimization loop and is formulated directly as a neural realization of residual iteration. In the potential case, its update direction coincides with the deep Ritz gradient direction, while the construction remains applicable to nonlinear and nonsymmetric non-potential operators. We also provide a weak adversarial interpretation through an implicitly defined test function, without introducing an additional network. From a function-space perspective, NeRI induces, to first order, a neural-tangent-kernel-preconditioned residual iteration, connecting classical operator iterations with neural-network optimization [34, 44]. A closely related non-gradient method is proposed by Peng et al. [35], which updates network parameters using strong-form PDE residuals evaluated by finite differences. NeRI shares this residual-driven principle but uses weak residual pairings involving only first-order spatial derivatives for second-order divergence-form problems. This formulation naturally accommodates low-regularity weak solutions without requiring pointwise differential residuals. Our work further extends the framework to elliptic variational inequalities and develops a Sobolev-space analysis.

An important feature of our analysis is that the neural tangent kernel is treated as an operator from the dual of the Sobolev trial space into the trial space itself. Under a gradient-flow formulation, we prove finite-time lazy-training estimates and a frozen-kernel approximation. For wide two-layer networks, the kernel drift is OP((log⁡m)/m)O_{\mathbb{P}}((\log m)/\sqrt{m}) under suitable activation and initialization assumptions. In contrast to the empirical-loss convergence analysis in [22], we combine these estimates to bound, on fixed finite time intervals, the H1(Ω)H^1(\Omega)-error relative to the exact weak solution, separating finite-width, frozen-flow optimization, and reduced-space approximation errors.

The remainder of the paper is organized as follows. § 2 introduces the operator setting and classical residual iteration. § 3 presents NeRI, its function-space interpretation, and the treatment of Dirichlet boundary conditions. § 4 extends the framework to elliptic VIs. § 5 develops the lazy-training and error analysis. Numerical experiments and concluding remarks are presented in § 6 and § 7, respectively.

Problem setting and classical residual iteration

Let Ω⊂Rd\Omega\subset\mathbb{R}^d be a bounded domain with boundary ∂Ω\partial\Omega, and let VV be the Hilbert space used in the weak formulation, with dual V∗V^* and duality pairing ⟨⋅,⋅⟩V∗,V\langle\cdot,\cdot\rangle_{V^*,V}. For second-order problems, typical choices are V=H1(Ω)V = H^1(\Omega) for natural boundary conditions and V=H01(Ω)V = H^1_0(\Omega) for homogeneous Dirichlet conditions [1]. Nonhomogeneous Dirichlet data can be handled by a lifting u=g^D+wu = \widehat{g}_D + w, where g^D∈H1(Ω)\widehat{g}_D \in H^1(\Omega) extends the boundary data and w∈H01(Ω)w \in H^1_0(\Omega).

We consider the abstract elliptic operator equation

find u∈V:A(u)=fin V∗,(1)\text{find } u \in V:\qquad\mathcal{A}(u)=f\quad\text{in }V^*, \tag*{(1)}

where A:V→V∗\mathcal{A}:V\to V^* is a possibly nonlinear and nonsymmetric operator, and f∈V∗f\in V^* is given. A typical second-order weak form is

⟨A(u),v⟩V∗,V=∫Ω(a0(x,u,∇u)v+a1(x,u,∇u)⋅∇v) dx+∫∂Ωb(x,u)v ds,(2)\langle\mathcal{A}(u),v\rangle_{V^*,V} = \int_{\Omega}\left(a_0(x,u,\nabla u)v+a_1(x,u,\nabla u)\cdot\nabla v\right)\,\mathrm{d}x +\int_{\partial\Omega}b(x,u)v\,\mathrm{d}s, \tag*{(2)}
⟨f,v⟩V∗,V=∫Ω(f0v+f1⋅∇v) dx+∫∂ΩgNv ds.(3)\langle f,v\rangle_{V^*,V} = \int_{\Omega}\left(f_0v+f_1\cdot\nabla v\right)\,\mathrm{d}x +\int_{\partial\Omega}g_Nv\,\mathrm{d}s. \tag*{(3)}

The boundary terms account for Neumann or Robin conditions.

Throughout this paper, we assume that A\mathcal{A} is Lipschitz continuous and strongly monotone.

Assumption 2.1. There exist constants L,γ>0L,\gamma>0 such that, for all u,v∈Vu,v\in V,

∥A(u)−A(v)∥V∗≤L∥u−v∥V,(4)\|\mathcal{A}(u)-\mathcal{A}(v)\|_{V^*}\le L\|u-v\|_V, \tag*{(4)}
⟨A(u)−A(v),u−v⟩V∗,V≥γ∥u−v∥V2.(5)\langle\mathcal{A}(u)-\mathcal{A}(v),u-v\rangle_{V^*,V}\ge\gamma\|u-v\|_V^2. \tag*{(5)}

Under Assumption 2.1, problem (1) admits a unique solution u∗u^* [44]. Let R:V∗→V\mathcal{R}:V^*\to V be the Riesz isomorphism defined by ⟨Rg,h⟩V=⟨g,h⟩V∗,V\langle\mathcal{R}g,h\rangle_V=\langle g,h\rangle_{V^*,V} for g∈V∗g\in V^* and h∈Vh\in V. Given u0∈Vu_0\in V and η>0\eta>0, the Richardson-type residual iteration reads

uk+1=uk−ηR(A(uk)−f),k=0,1,2,…(6)u_{k+1}=u_k-\eta\mathcal{R}\bigl(\mathcal{A}(u_k)-f\bigr),\qquad k=0,1,2,\ldots \tag*{(6)}

The Lipschitz and monotonicity bounds yield

∥uk+1−u∗∥V2≤(1−2ηγ+η2L2)∥uk−u∗∥V2.\lVert u_{k+1}-u^*\rVert_V^2 \le\left(1-2\eta\gamma+\eta^2L^2\right)\lVert u_k-u^*\rVert_V^2.

Hence, the iteration is contractive and converges to u∗u^* whenever 0<η<2γ/L20<\eta<2\gamma/L^2. More generally, one may consider classical function space iterations of the form uk+1=uk−ηT(uk)u_{k+1}=u_k-\eta T(u_k), where T(u∗)=0T(u^*)=0 and u↦u−ηT(u)u\mapsto u-\eta T(u) is a contraction for an appropriate choice of the step size η\eta. This residual-iteration principle is the basis of the neural method developed below.

Neural residual iteration (NeRI)

Motivation and formulation of NeRI

We first recall the variational case. Suppose that there exists a Fréchet-differentiable functional E:V→RE:V\to\mathbb{R} such that

DuE(u)=A(u)−fin V∗.(7)D_uE(u)=\mathcal{A}(u)-f \quad\text{in } V^*. \tag*{(7)}

Then every solution of A(u)=f\mathcal{A}(u)=f is a stationary point of EE. If EE is strongly convex, the solution is its unique minimizer. This variational structure forms the basis of the deep Ritz method, which uses E(uθ)E(u_\theta) as the training objective for a neural-network approximation uθu_\theta. For a linear symmetric operator, the energy takes the familiar quadratic form

E(u)=12⟨A(u),u⟩V∗,V−⟨f,u⟩V∗,V.(8)E(u)=\frac{1}{2}\langle\mathcal{A}(u),u\rangle_{V^*,V}-\langle f,u\rangle_{V^*,V}. \tag*{(8)}

For general nonlinear or nonsymmetric operators, however, such a global energy need not exist, and a fixed Ritz-type objective may therefore be unavailable. In particular, for a nonsymmetric linear operator, the first variation of (8) involves only the symmetric part of A\mathcal{A} and does not recover the original residual.

The classical residual iteration (6), in contrast, does not require A\mathcal{A} to be linear, symmetric, or potential; its contraction relies instead on the Lipschitz continuity and strong monotonicity conditions in Assumption 2.1. Motivated by this observation, NeRI generates a residual-driven neural update without constructing a global energy. In fact, given the current approximation uk∈Vu_k\in V of u∗u^*, we define the state-dependent linear functional

Jk(u):=⟨A(uk)−f,u⟩V∗,V.(9)J_k(u):=\langle\mathcal{A}(u_k)-f,u\rangle_{V^*,V}. \tag*{(9)}

Since A(uk)−f\mathcal{A}(u_k)-f is held fixed when differentiating with respect to uu, its derivative satisfies DuJk(u)=A(uk)−fD_uJ_k(u)=\mathcal{A}(u_k)-f. Thus, the current residual can be recovered as the derivative of a state-dependent functional even when the operator has no global potential.

The preceding construction is formulated in function space. To obtain a practical neural method, we now restrict the trial functions to a neural-network class and translate the residual direction into a parameter update. For the moment, we consider natural boundary conditions and take V=H1(Ω)V=H^1(\Omega). With a sufficiently regular activation function, the neural realization belongs to VV, while the natural boundary conditions are incorporated through the weak formulation. Essential Dirichlet conditions are treated separately below in Subsection 3.3.

Let uθ∈Vu_\theta\in V be a neural-network approximation with parameter vector θ∈Rp\theta\in\mathbb{R}^p, and assume that θ↦uθ\theta\mapsto u_\theta is Fréchet differentiable. At iteration kk, we define the state-dependent objective

Jk(θ):=⟨sg⁡(A(uθk)−f),uθ⟩V∗,V.(10)J_k(\theta):=\left\langle\operatorname{sg}\bigl(\mathcal{A}(u_{\theta_k})-f\bigr),u_\theta\right\rangle_{V^*,V}. \tag*{(10)}

where ‘sg’ denotes the stop-gradient operation: the current residual is treated as a constant when differentiating with respect to θ\theta. In particular, no derivative is propagated through the

Algorithm 1: Neural Residual Iteration (NeRI)

Input: A neural network uθu_\theta with initial parameters θ0\theta_0, step sizes {ηk}k=0K−1\{\eta_k\}_{k=0}^{K-1}, and number of iterations KK

Output: Neural approximation uθKu_{\theta_K}

for k=0,…,K−1k = 0, \ldots, K - 1 do

θk+1=θk−ηk∇θ⟨rk,uθ⟩V∗,V∣θ=θk\theta_{k+1} = \theta_k - \eta_k \nabla_\theta\langle r_k, u_\theta\rangle_{V^*,V}\big|_{\theta=\theta_k} where rk=sg⁡(A(uθk)−f)r_k = \operatorname{sg}(\mathcal{A}(u_{\theta_k}) - f);

end

operator evaluation at uθku_{\theta_k}. We then perform one parameter-gradient step with step size ηk>0\eta_k > 0. By the chain rule, this gives

θk+1=θk−ηk∇θJk(θ)∣θ=θk=θk−ηk(Dθuθk)∗(A(uθk)−f),(11)\theta_{k+1} = \theta_k - \eta_k \nabla_\theta J_k(\theta)\big|_{\theta=\theta_k} = \theta_k - \eta_k (D_\theta u_{\theta_k})^*(\mathcal{A}(u_{\theta_k}) - f), \tag*{(11)}

where Dθuθk:Rp→VD_\theta u_{\theta_k}: \mathbb{R}^p \to V is the parameter-to-function derivative and (Dθuθk)∗:V∗→Rp(D_\theta u_{\theta_k})^*: V^* \to\mathbb{R}^p is its adjoint.

The objective JkJ_k is not minimized to convergence; it is used only to generate a single update direction. After the update, the operator is evaluated again at uθk+1u_{\theta_{k+1}}, and a new state-dependent objective is constructed. Unlike a fixed training loss, the objective therefore changes along the iteration and incorporates the current residual information at each step.

For the analysis below, we use plain gradient descent and its continuous-time gradient-flow limit, as is standard in much of the NTK and lazy-training literature [7, 20, 41, 23]. In practice, adaptive optimizers such as Adam are widely used for neural PDE solvers because of their favorable empirical optimization behavior; accordingly, we mainly use Adam in Section 6. The same stop-gradient objective is compatible with Adam, but the resulting history-dependent dynamics are beyond the scope of the present analysis.

This construction also clarifies the connection with the deep Ritz method. Whenever a functional EE satisfies (7), the chain rule yields ∇θE(uθ)=(Dθuθ)∗(A(uθ)−f)\nabla_\theta E(u_\theta) = (D_\theta u_\theta)^*(\mathcal{A}(u_\theta) - f), which is exactly the NeRI update direction. Thus, the two methods have the same parameter-gradient direction in the potential case, including the linear symmetric setting (8). NeRI, however, does not require such an energy and remains applicable to non-potential operators.

Algorithm 1 summarizes the basic iteration with natural boundary conditions and exact evaluation of the weak residual. Its implementation requires only the weak residual pairing and a standard gradient step, with the residual detached from the computational graph. In practice, the exact duality pairing can be replaced by a quadrature or Monte Carlo approximation.

Induced function-space iteration

We next derive the function-space dynamics induced by NeRI. Although the update is performed in parameter space, it generates a first-order iteration in function space. Assuming that DθuθD_\theta u_\theta is locally Lipschitz, a Taylor expansion gives

uθk+1=uθk+Dθuθk(θk+1−θk)+O(∥θk+1−θk∥2).(12)u_{\theta_{k+1}} = u_{\theta_k} + D_\theta u_{\theta_k}(\theta_{k+1} - \theta_k) + \mathcal{O}(\lVert\theta_{k+1} - \theta_k \rVert^2). \tag*{(12)}

Combining this expansion with (11), we obtain

uθk+1=uθk−ηkDθuθk(Dθuθk)∗(A(uθk)−f)+O(ηk2).(13)u_{\theta_{k+1}} = u_{\theta_k} - \eta_k D_\theta u_{\theta_k}(D_\theta u_{\theta_k})^*(\mathcal{A}(u_{\theta_k}) - f) + \mathcal{O}(\eta_k^2). \tag*{(13)}

The remainders are understood in the VV-norm. The latter bound is uniform in kk provided the Lipschitz constants along the update segments are uniformly bounded and (Dθuθk)∗(A(uθk)−f)(D_\theta u_{\theta_k})^*(\mathcal{A}(u_{\theta_k}) - f) remains bounded.

We define the neural tangent operator at θk\theta_k by

Kθk:=Dθuθk(Dθuθk)∗:V∗→V.(14)K_{\theta_k} := D_\theta u_{\theta_k}(D_\theta u_{\theta_k})^*: V^* \to V. \tag*{(14)}

Equivalently, writing θ=(θ(1),…,θ(p))\theta= (\theta^{(1)}, \ldots, \theta^{(p)}), we have

Kθkg=∑i=1p⟨g,∂θ(i)uθk⟩V∗,V∂θ(i)uθk,g∈V∗.K_{\theta_k} g = \sum_{i=1}^{p} \langle g, \partial_{\theta^{(i)}}u_{\theta_k}\rangle_{V^*,V}\partial_{\theta^{(i)}}u_{\theta_k}, \qquad g \in V^*.

Thus, the induced function-space iteration becomes

uθk+1=uθk−ηkKθk(A(uθk)−f)+O(ηk2).(15)u_{\theta_{k+1}} = u_{\theta_k} - \eta_k K_{\theta_k}\bigl(A(u_{\theta_k}) - f\bigr) + O(\eta_k^2). \tag*{(15)}

The operator KθkK_{\theta_k} is symmetric in the duality sense and positive semidefinite: for all g,h∈V∗g,h \in V^*,

⟨g,Kθkh⟩V∗,V=⟨h,Kθkg⟩V∗,V,⟨g,Kθkg⟩V∗,V=∣(Dθuθk)∗g∣Rp2≥0.\langle g, K_{\theta_k}h\rangle_{V^*,V} = \langle h, K_{\theta_k}g\rangle_{V^*,V}, \qquad\langle g, K_{\theta_k}g\rangle_{V^*,V} = \lvert(D_\theta u_{\theta_k})^*g\rvert_{\mathbb{R}^p}^2 \ge0.

Accordingly, NeRI can be interpreted as a neural-kernel-preconditioned realization of classical residual iterations.

This interpretation is particularly transparent for a linear parameterization uθ=∑i=1pθ(i)ϕiu_\theta= \sum_{i=1}^{p}\theta^{(i)}\phi_i, with fixed functions ϕi∈V\phi_i \in V. In this case, DθuθD_\theta u_\theta is independent of θ\theta, and the tangent operator is

Kϕg=∑i=1p⟨g,ϕi⟩V∗,Vϕi=(∑i=1pϕi⊗ϕi)g,g∈V∗.(16)K_\phi g = \sum_{i=1}^{p}\langle g,\phi_i\rangle_{V^*,V}\phi_i = \left(\sum_{i=1}^{p}\phi_i \otimes\phi_i\right)g, \qquad g \in V^*. \tag*{(16)}

The range of the fixed finite rank kernel KϕK_\phi is Vp:=span⁡{ϕ1,…,ϕp}V_p := \operatorname{span}\{\phi_1,\ldots,\phi_p\}. Since this space is finite-dimensional, KϕK_\phi cannot be uniformly coercive on all of V∗V^* when VV is infinite-dimensional. Nevertheless, its positivity on the corresponding reduced space permits a convergence analysis.

To isolate the role of the kernel in residual correction, we first analyze the fixed-kernel iteration. § 5 then justifies this approximation through finite-time lazy-training estimates for wide two-layer networks. For u0∈Vu_0 \in V and η>0\eta> 0, consider

uk+1=uk−ηKϕ(A(uk)−f).(17)u_{k+1} = u_k - \eta K_\phi\bigl(A(u_k)-f\bigr). \tag*{(17)}

The self-adjoint operator B:=KϕR−1:V→VB := K_\phi\mathcal{R}^{-1}: V \to V has range VpV_p and is positive definite on this space. Hence, there exist constants 0<γK≤MK0 < \gamma_K \le M_K such that

γK∥v∥V2≤⟨v,Bv⟩V≤MK∥v∥V2,v∈Vp.(18)\gamma_K\lVert v\rVert_V^2 \le\langle v,Bv\rangle_V \le M_K\lVert v\rVert_V^2, \qquad v \in V_p. \tag*{(18)}

Its Moore–Penrose pseudoinverse B†B^\dagger therefore defines a norm on VpV_p:

∥v∥B†2:=⟨B†v,v⟩V,v∈Vp.\lVert v\rVert_{B^\dagger}^2 := \langle B^\dagger v,v\rangle_V, \qquad v \in V_p.

The following result combines a finite-rank preconditioned convergence estimate with a classical Céa-type quasi-optimality bound for nonlinear Galerkin approximations; see, e.g., [8, 12].

Theorem 3.1. Suppose Assumption 2.1 holds and let S:=u0+VpS := u_0 + V_p. There exists a unique reduced solution uS∗∈Su_S^* \in S satisfying

⟨A(uS∗)−f,v⟩V∗,V=0,v∈Vp.(19)\langle A(u_S^*)-f,v\rangle_{V^*,V}=0, \qquad v \in V_p. \tag*{(19)}

The iterates of (17) satisfy

∥uk+1−uS∗∥B†2≤∥uk−uS∗∥B†2−η(2γ−ηMKL2)∥uk−uS∗∥V2.(20)\lVert u_{k+1}-u_S^*\rVert_{B^\dagger}^2 \le\lVert u_k-u_S^*\rVert_{B^\dagger}^2-\eta(2\gamma-\eta M_KL^2)\lVert u_k-u_S^*\rVert_V^2. \tag*{(20)}

Consequently, if 0<η<2γ/(MKL2)0 < \eta< 2\gamma/(MKL^2), then

∥uk+1−uS∗∥B†2≤ρ∥uk−uS∗∥B†2,ρ:=1−ηγK(2γ−ηMKL2)∈[0,1).(21)\lVert u_{k+1} - u_S^* \rVert_{B^\dagger}^2 \le\rho\lVert u_k - u_S^* \rVert_{B^\dagger}^2,\qquad\rho:= 1 - \eta\gamma K(2\gamma- \eta MKL^2) \in[0,1). \tag*{(21)}

Moreover, if u∗u^* denotes the solution of A(u∗)=f\mathcal{A}(u^*) = f, then

∥uk−u∗∥V≤MKρk/2∥u0−uS∗∥B†+Lγinf⁡v∈S∥v−u∗∥V.(22)\lVert u_k - u^* \rVert_V \le\sqrt{M_K}\rho^{k/2}\lVert u_0 - u_S^* \rVert_{B^\dagger} + \frac{L}{\gamma}\inf_{v\in S}\lVert v-u^*\rVert_V. \tag*{(22)}

Proof. Let Pp:V→VpP_p: V \to V_p be the orthogonal projection. The reduced operator w↦PpR(A(u0+w)−f)w \mapsto P_p\mathcal{R}(\mathcal{A}(u_0+w)-f) is strongly monotone and Lipschitz on VpV_p, so the reduced problem has a unique solution. Set ekS:=uk−uS∗e_k^S := u_k-u_S^* and rkS:=A(uk)−A(uS∗)r_k^S := \mathcal{A}(u_k)-\mathcal{A}(u_S^*). Since u0,uS∗∈Su_0,u_S^* \in S and all increments in (17) belong to VpV_p, we have ekS∈Vpe_k^S \in V_p for all k≥0k\ge0. By (19), Kϕ(A(uS∗)−f)=0K_\phi(\mathcal{A}(u_S^*)-f)=0, and hence ek+1S=ekS−ηKϕrkSe_{k+1}^S=e_k^S-\eta K_\phi r_k^S. Using Kϕ=BRK_\phi=BR and B†B=PpB^\dagger B=P_p, we obtain

∥ek+1S∥B†2=∥ekS∥B†2−2η⟨rkS,ekS⟩V∗,V+η2⟨rkS,KϕrkS⟩V∗,V.\lVert e_{k+1}^S\rVert_{B^\dagger}^2=\lVert e_k^S\rVert_{B^\dagger}^2-2\eta\langle r_k^S,e_k^S\rangle_{V^*,V}+\eta^2\langle r_k^S,K_\phi r_k^S\rangle_{V^*,V}.

Strong monotonicity gives ⟨rkS,ekS⟩V∗,V≥γ∥ekS∥V2\langle r_k^S,e_k^S\rangle_{V^*,V}\ge\gamma\lVert e_k^S\rVert_V^2, while the upper spectral bound and Lipschitz continuity imply

⟨rkS,KϕrkS⟩V∗,V≤MK∥rkS∥V∗2≤MKL2∥ekS∥V2.\langle r_k^S,K_\phi r_k^S\rangle_{V^*,V}\le M_K\lVert r_k^S\rVert_{V^*}^2\le M_KL^2\lVert e_k^S\rVert_V^2.

This proves (20). Under the stated step-size restriction, substituting ∥ekS∥V2≥γK∥ekS∥B†2\lVert e_k^S\rVert_V^2\ge\gamma K\lVert e_k^S\rVert_{B^\dagger}^2 into (20) yields (21).

We next estimate the reduced-space approximation error. For any w∈Sw\in S, we have uS∗−w∈Vpu_S^*-w\in V_p, so the reduced equation gives

⟨A(uS∗)−A(u∗),uS∗−w⟩V∗,V=0.\langle\mathcal{A}(u_S^*)-\mathcal{A}(u^*),u_S^*-w\rangle_{V^*,V}=0.

Therefore,

γ∥uS∗−u∗∥V2≤⟨A(uS∗)−A(u∗),uS∗−u∗⟩V∗,V=⟨A(uS∗)−A(u∗),w−u∗⟩V∗,V≤L∥uS∗−u∗∥V∥w−u∗∥V.\begin{aligned} \gamma\lVert u_S^*-u^*\rVert_V^2 &\le\langle\mathcal{A}(u_S^*)-\mathcal{A}(u^*),u_S^*-u^*\rangle_{V^*,V}\\ &=\langle\mathcal{A}(u_S^*)-\mathcal{A}(u^*),w-u^*\rangle_{V^*,V}\\ &\le L\lVert u_S^*-u^*\rVert_V\lVert w-u^*\rVert_V. \end{aligned}

Taking the infimum over w∈Sw\in S yields

∥uS∗−u∗∥V≤Lγinf⁡v∈S∥v−u∗∥V.(23)\lVert u_S^*-u^*\rVert_V\le\frac{L}{\gamma}\inf_{v\in S}\lVert v-u^*\rVert_V. \tag*{(23)}

Finally, the upper spectral bound gives ∥v∥V2≤MK∥v∥B†2\lVert v\rVert_V^2\le M_K\lVert v\rVert_{B^\dagger}^2 for v∈Vpv\in V_p. Together with (21), this implies

∥uk−uS∗∥V≤MKρk/2∥u0−uS∗∥B†.\lVert u_k-u_S^*\rVert_V\le\sqrt{M_K}\rho^{k/2}\lVert u_0-u_S^*\rVert_{B^\dagger}.

Combining this estimate with (23) and the triangle inequality proves (22).

Treatment of Dirichlet boundary conditions

Unlike natural boundary conditions, essential Dirichlet conditions are not enforced automatically by the weak formulation. We therefore consider two standard treatments within the NeRI framework: the penalty method and the hard-constrained parameterization. We assume that the Dirichlet data gg admit a lifting g^∈H1(Ω)\hat{g}\in H^1(\Omega) satisfying Tr⁡g^=g\operatorname{Tr}\hat{g}=g. On bounded Lipschitz domains, this is equivalent to requiring g∈H1/2(∂Ω)g\in H^{1/2}(\partial\Omega).

Penalty method

The penalty method augments the NeRI objective with a term that penalizes violations of the prescribed boundary condition

u=gon ∂Ω.(24)u = g \quad\text{on } \partial\Omega. \tag*{(24)}

Let Tr⁡:H1(Ω)→L2(∂Ω)\operatorname{Tr}: H^{1}(\Omega) \to L^{2}(\partial\Omega) be the bounded trace operator. The boundary condition is imposed approximately through

Pβ(u):=β2∥Tr⁡u−g∥L2(∂Ω)2,\mathcal{P}_{\beta}(u) := \frac{\beta}{2}\lVert\operatorname{Tr} u - g\rVert_{L^{2}(\partial\Omega)}^{2},

where β>0\beta> 0 is the penalty parameter. Its Fréchet derivative is DuPβ(u)=βTr⁡∗(Tr⁡u−g)∈H1(Ω)∗D_{u}\mathcal{P}_{\beta}(u) = \beta\operatorname{Tr}^{*}(\operatorname{Tr} u - g) \in H^{1}(\Omega)^{*}, where Tr⁡∗:L2(∂Ω)→H1(Ω)∗\operatorname{Tr}^{*}: L^{2}(\partial\Omega) \to H^{1}(\Omega)^{*} is the adjoint of the trace operator.

The penalty problem is posed on V=H1(Ω)V = H^{1}(\Omega), with A:V→V∗\mathcal{A}: V \to V^{*} and f∈V∗f \in V^{*} interpreted through the weak formulation on this space. For a neural ansatz uθ=Nθu_{\theta} = \mathcal{N}_{\theta}, define the penalized NeRI objective at iteration kk by

Jkβ(θ):=⟨sg⁡(A(uθk)−f),uθ⟩V∗,V+Pβ(uθ).(25)\mathcal{J}_{k}^{\beta}(\theta) := \langle\operatorname{sg}(\mathcal{A}(u_{\theta_k}) - f), u_{\theta}\rangle_{V^{*},V} + \mathcal{P}_{\beta}(u_{\theta}). \tag*{(25)}

We then perform one parameter-gradient step,

θk+1=θk−ηk∇θJkβ(θk)=θk−ηk(Dθuθk)∗(A(uθk)−f+βTr⁡∗(Tr⁡uθk−g)).\theta_{k+1} = \theta_k - \eta_k \nabla_{\theta}\mathcal{J}_{k}^{\beta}(\theta_k) = \theta_k - \eta_k(D_{\theta}u_{\theta_k})^{*}\left(\mathcal{A}(u_{\theta_k}) - f + \beta\operatorname{Tr}^{*}(\operatorname{Tr}u_{\theta_k} - g)\right).

Introducing the penalized operator

Aβ(u):=A(u)+βTr⁡∗(Tr⁡u−g),(26)\mathcal{A}_{\beta}(u) := \mathcal{A}(u) + \beta\operatorname{Tr}^{*}(\operatorname{Tr}u - g), \tag*{(26)}

the induced first-order function-space iteration can be written as

uθk+1=uθk−ηkKθk(Aβ(uθk)−f)+O(ηk2),u_{\theta_{k+1}} = u_{\theta_k} - \eta_k K_{\theta_k}\left(\mathcal{A}_{\beta}(u_{\theta_k}) - f\right) + O(\eta_k^{2}),

where Kθk=Dθuθk(Dθuθk)∗K_{\theta_k} = D_{\theta}u_{\theta_k}(D_{\theta}u_{\theta_k})^{*}. Thus, the penalty method amounts to applying NeRI to Aβ(u)=f\mathcal{A}_{\beta}(u) = f while retaining the original neural parameterization. Increasing β\beta strengthens the boundary penalty but also increases the Lipschitz bound for Aβ\mathcal{A}_{\beta}, which may reduce the admissible step size and worsen the conditioning of the iteration.

Hard-constrained parameterization

A hard-constrained parameterization instead enforces the Dirichlet boundary condition by construction. For (24), one can use an ansatz of the form

uθ=T(θ):=g^+wNθ,(27)u_{\theta} = \mathcal{T}(\theta) := \hat{g} + w\mathcal{N}_{\theta}, \tag*{(27)}

where g^∈H1(Ω)\hat{g} \in H^{1}(\Omega) extends the boundary data, with Tr⁡g^=g\operatorname{Tr}\hat{g} = g, and ww is a boundary factor satisfying

w=0on ∂Ω,w>0in Ω.(28)w = 0 \quad\text{on } \partial\Omega,\qquad w > 0 \quad\text{in } \Omega. \tag*{(28)}

Under suitable regularity assumptions, wNθ∈H01(Ω)w\mathcal{N}_{\theta} \in H_{0}^{1}(\Omega), so the prescribed boundary condition holds for every θ\theta. For more general hard-constrained parameterizations, we refer to [16, 30].

Using the same ambient space V=H1(Ω)V = H^{1}(\Omega), the hard-constrained NeRI objective is

Jkhc(θ):=⟨sg⁡(A(T(θk))−f),T(θ)⟩V∗,V.\mathcal{J}_{k}^{\mathrm{hc}}(\theta) := \langle\operatorname{sg}(\mathcal{A}(\mathcal{T}(\theta_k)) - f), \mathcal{T}(\theta)\rangle_{V^{*},V}.

We then perform the gradient update θk+1=θk−ηk∇θJkhc(θ)∣θ=θk\theta_{k+1} = \theta_k - \eta_k \nabla_{\theta}\mathcal{J}_{k}^{\mathrm{hc}}(\theta)\big|_{\theta=\theta_k}. The induced first-order function-space iteration becomes

uθk+1=uθk−ηkK~θk(A(uθk)−f)+O(ηk2),u_{\theta_{k+1}} = u_{\theta_k} - \eta_k \widetilde{K}_{\theta_k}\left(\mathcal{A}(u_{\theta_k}) - f\right) + O(\eta_k^{2}),

where K~θk:=DθT(θk)(DθT(θk))∗\widetilde{K}_{\theta_k}:=D_\theta T(\theta_k)(D_\theta T(\theta_k))^* is the constrained neural tangent operator. Since DθT(θ)=wDθNθD_\theta T(\theta)=wD_\theta N_\theta takes values in H01(Ω)H_0^1(\Omega), the tangent corrections preserve the boundary condition.

In contrast to the penalty method, the hard-constrained method restricts the trial class to boundary-conforming functions, enforcing the boundary data exactly while changing the effective tangent geometry.

Weak adversarial interpretation via an implicit test function

In this subsection, we interpret NeRI from a weak adversarial perspective. We show that its update is gradient-equivalent to a weak residual pairing with an implicitly defined test function associated with the current residual.

Recall that the operator equation can be viewed as minimizing the squared dual residual norm. By the definition of the dual norm,

min⁡u∈V∥A(u)−f∥V∗2=min⁡u∈Vsup⁡φ∈V, φ≠0∣⟨A(u)−f,φ⟩V∗,V∣2∥φ∥V2.\min_{u\in V}\|\mathcal{A}(u)-f\|_{V^*}^2=\min_{u\in V}\sup_{\varphi\in V,\ \varphi\ne0}\frac{|\langle\mathcal{A}(u)-f,\varphi\rangle_{V^*,V}|^2}{\|\varphi\|_V^2}.

This formulation motivates the weak adversarial network (WAN) [43], which parameterizes both the trial function uθu_\theta and the test function φη\varphi_\eta. A corresponding objective is

L(uθ,φη)=∣⟨A(uθ)−f,φη⟩V∗,V∣2∥φη∥V2.\mathcal{L}(u_\theta,\varphi_\eta)=\frac{|\langle\mathcal{A}(u_\theta)-f,\varphi_\eta\rangle_{V^*,V}|^2}{\|\varphi_\eta\|_V^2}.

A logarithmic variant, defined when the residual pairing is nonzero, is

Llog⁡(uθ,φη)=log⁡∣⟨A(uθ)−f,φη⟩V∗,V∣2−log⁡∥φη∥V2.\mathcal{L}_{\log}(u_\theta,\varphi_\eta)=\log|\langle\mathcal{A}(u_\theta)-f,\varphi_\eta\rangle_{V^*,V}|^2-\log\|\varphi_\eta\|_V^2.

The trial and test networks are then updated alternately through the min–max formulation.

For a nonzero residual, the choice φ=R(A(uθ)−f)\varphi=\mathcal{R}(\mathcal{A}(u_\theta)-f) attains the supremum and satisfies

∣⟨A(uθ)−f,φ⟩V∗,V∣2∥φ∥V2=∥A(uθ)−f∥V∗2.\frac{|\langle\mathcal{A}(u_\theta)-f,\varphi\rangle_{V^*,V}|^2}{\|\varphi\|_V^2}=\|\mathcal{A}(u_\theta)-f\|_{V^*}^2.

Computing this optimal test function generally requires solving a Riesz representation problem. In WAN, it is instead approximated by a separate neural network.

We now relate NeRI to a different implicit test function. Define the unnormalized weak residual pairing

L^(u,φ):=⟨A(u)−f,φ⟩V∗,V.\widehat{\mathcal{L}}(u,\varphi):=\langle\mathcal{A}(u)-f,\varphi\rangle_{V^*,V}.

Assume that A\mathcal{A} is Fréchet differentiable. For fixed φ\varphi, its derivative with respect to uu is DuL^(u,φ)=(DuA(u))∗φD_u\widehat{\mathcal{L}}(u,\varphi)=(D_u\mathcal{A}(u))^*\varphi, where the adjoint is understood in the duality sense. At iteration kk, the NeRI objective has derivative A(uk)−f\mathcal{A}(u_k)-f with respect to the trial function. We therefore seek a test function φ^k\widehat{\varphi}_k satisfying

DuL^(uk,φ^k)=(DuA(uk))∗φ^k=A(uk)−f.D_u\widehat{\mathcal{L}}(u_k,\widehat{\varphi}_k)=(D_u\mathcal{A}(u_k))^*\widehat{\varphi}_k=\mathcal{A}(u_k)-f.

Under Assumption 2.1, both DuA(uk)D_u\mathcal{A}(u_k) and its adjoint are bounded and coercive, hence isomorphisms from VV onto V∗V^*. Thus, the implicit test function is well defined by

φ^k:=((DuA(uk))∗)−1(A(uk)−f).(29)\widehat{\varphi}_k:=((D_u\mathcal{A}(u_k))^*)^{-1}(\mathcal{A}(u_k)-f). \tag*{(29)}

With φ^k\widehat{\varphi}_k held fixed, the derivatives of this pairing and the NeRI objective agree at uku_k. The corresponding parameter gradients therefore also agree when uk=uθku_k=u_{\theta_k}.

Evaluating the pairing at this implicit test function gives

L^(uk,φ^k)=⟨A(uk)−f,((DuA(uk))∗)−1(A(uk)−f)⟩V∗,V.(30)\widehat{\mathcal{L}}(u_k,\widehat{\varphi}_k)=\left\langle\mathcal{A}(u_k)-f,((D_u\mathcal{A}(u_k))^*)^{-1}(\mathcal{A}(u_k)-f)\right\rangle_{V^*,V}. \tag*{(30)}

The following lemma shows that this quantity is comparable to the squared dual residual norm.

Lemma 3.2. Suppose Assumption 2.1 holds and A\mathcal{A} is Fréchet differentiable. Then the quantity defined in (30) satisfies

γL2∥A(uk)−f∥V∗2≤L^(uk,φ^k)≤1γ∥A(uk)−f∥V∗2.(31)\frac{\gamma}{L^2}\lVert\mathcal{A}(u_k)-f\rVert_{V^*}^2 \leq\widehat{\mathcal{L}}(u_k,\widehat{\varphi}_k) \leq\frac{1}{\gamma}\lVert\mathcal{A}(u_k)-f\rVert_{V^*}^2. \tag*{(31)}

Proof. Set Bk:=DuA(uk)B_k := D_u\mathcal{A}(u_k). Differentiating the Lipschitz and strong monotonicity bounds for A\mathcal{A} gives

∥Bkh∥V∗≤L∥h∥V,⟨Bkh,h⟩V∗,V≥γ∥h∥V2,h∈V.\lVert B_k h\rVert_{V^*} \leq L\lVert h\rVert_V,\qquad\langle B_kh,h\rangle_{V^*,V} \geq\gamma\lVert h\rVert_V^2,\qquad h\in V.

The same bounds hold for the duality adjoint Bk∗:V→V∗B_k^*:V\to V^*. Given q∈V∗q\in V^*, set h=(Bk∗)−1qh=(B_k^*)^{-1}q. Since ∥q∥V∗=∥Bk∗h∥V∗≤L∥h∥V\lVert q\rVert_{V^*}=\lVert B_k^*h\rVert_{V^*}\leq L\lVert h\rVert_V, we obtain the lower bound

⟨q,(Bk∗)−1q⟩V∗,V=⟨Bk∗h,h⟩V∗,V≥γ∥h∥V2≥γL2∥q∥V∗2.\langle q,(B_k^*)^{-1}q\rangle_{V^*,V}=\langle B_k^*h,h\rangle_{V^*,V}\geq\gamma\lVert h\rVert_V^2\geq\frac{\gamma}{L^2}\lVert q\rVert_{V^*}^2.

For the upper bound, coercivity gives

γ∥h∥V2≤⟨Bk∗h,h⟩V∗,V=⟨q,h⟩V∗,V≤∥q∥V∗∥h∥V,\gamma\lVert h\rVert_V^2\leq\langle B_k^*h,h\rangle_{V^*,V}=\langle q,h\rangle_{V^*,V}\leq\lVert q\rVert_{V^*}\lVert h\rVert_V,

so ∥h∥V≤γ−1∥q∥V∗\lVert h\rVert_V\leq\gamma^{-1}\lVert q\rVert_{V^*}. Consequently,

⟨q,(Bk∗)−1q⟩V∗,V≤∥q∥V∗∥h∥V≤1γ∥q∥V∗2.\langle q,(B_k^*)^{-1}q\rangle_{V^*,V}\leq\lVert q\rVert_{V^*}\lVert h\rVert_V\leq\frac{1}{\gamma}\lVert q\rVert_{V^*}^2.

Taking q=A(uk)−fq=\mathcal{A}(u_k)-f proves (31).

Unlike WAN, NeRI therefore updates only the trial network, without explicitly computing or parameterizing the test function or solving an adversarial maximization problem.

NeRI for elliptic variational inequalities

Let A:V→V∗\mathcal{A}:V\to V^* and f∈V∗f\in V^* be defined as in § 2, and suppose Assumption 2.1 holds. For a nonempty, closed, and convex set K⊂VK\subset V, we consider the following elliptic variational inequality.

Definition 4.1. Find u∗∈Ku^*\in K such that

⟨A(u∗)−f,v−u∗⟩V∗,V≥0,∀v∈K.(32)\langle\mathcal{A}(u^*)-f,v-u^*\rangle_{V^*,V}\geq0,\qquad\forall v\in K. \tag*{(32)}

We now describe two approaches for incorporating the constraint into NeRI. The first replaces the variational inequality by a penalized operator equation, while the second uses a constraint-preserving neural parameterization.

Penalty method

For concreteness, consider the obstacle set

K:={v∈V:v≥ψ a.e. in Ω},(33)K:=\{v\in V:v\geq\psi\ \text{a.e. in }\Omega\}, \tag*{(33)}

where ψ∈L2(Ω)\psi\in L^2(\Omega) is a prescribed obstacle. To penalize violations of the constraint, define

P(u):=12∫Ω((ψ−u)+)2 dx,(34)\mathcal{P}(u):=\frac{1}{2}\int_{\Omega}\left((\psi-u)^+\right)^2\,\mathrm{d}x, \tag*{(34)}

where s+:=max⁡{s,0}s^{+}:=\max\{s,0\}. Its Fréchet derivative satisfies

⟨DP(u),h⟩V∗,V=−∫Ω(ψ−u)+h dx,h∈V.\langle DP(u),h\rangle_{V^{*},V}=-\int_{\Omega}(\psi-u)^{+}h\,\mathrm{d}x,\qquad h\in V.

Here and below, L2(Ω)L^{2}(\Omega) terms are interpreted as elements of V∗V^{*} through the L2L^{2} pairing.

For ε>0\varepsilon>0, define the penalized operator Aε(u):=A(u)−1ε(ψ−u)+\mathcal{A}_{\varepsilon}(u):=\mathcal{A}(u)-\frac{1}{\varepsilon}(\psi-u)^{+}, and consider the penalized operator equation

Aε(uε)=f.(35)\mathcal{A}_{\varepsilon}(u_{\varepsilon})=f. \tag*{(35)}

The map u↦−(ψ−u)+u\mapsto-(\psi-u)^{+} is monotone and Lipschitz continuous in L2(Ω)L^{2}(\Omega). By the continuous embedding, Aε:V→V∗\mathcal{A}_{\varepsilon}:V\to V^{*} therefore remains strongly monotone and Lipschitz continuous. Under suitable assumptions, standard penalty theory yields uε→u∗u_{\varepsilon}\to u^{*} as ε→0\varepsilon\to0 in an appropriate topology [15, 24].

NeRI can now be applied directly to (35). At iteration kk, define

Jk,εVI(θ):=⟨sg⁡(A(uθk)−1ε(ψ−uθk)+−f),uθ⟩V∗,V.(36)\mathcal{J}_{k,\varepsilon}^{\mathrm{VI}}(\theta):=\left\langle\operatorname{sg}\left(\mathcal{A}(u_{\theta_{k}})-\frac{1}{\varepsilon}(\psi-u_{\theta_{k}})^{+}-f\right),u_{\theta}\right\rangle_{V^{*},V}. \tag*{(36)}

The corresponding parameter update is

θk+1=θk−ηk∇θJk,εVI(θ)∣θ=θk.\theta_{k+1}=\theta_{k}-\eta_{k}\nabla_{\theta}\mathcal{J}_{k,\varepsilon}^{\mathrm{VI}}(\theta)\big|_{\theta=\theta_{k}}.

Alternatively, the penalty can be included explicitly in the objective:

J~k,εVI(θ):=⟨sg⁡(A(uθk)−f),uθ⟩V∗,V+12ε∫Ω(ψ−uθ)+2 dx.(37)\widetilde{\mathcal{J}}_{k,\varepsilon}^{\mathrm{VI}}(\theta):=\left\langle\operatorname{sg}\bigl(\mathcal{A}(u_{\theta_{k}})-f\bigr),u_{\theta}\right\rangle_{V^{*},V}+\frac{1}{2\varepsilon}\int_{\Omega}(\psi-u_{\theta})_{+}^{2}\,\mathrm{d}x. \tag*{(37)}

Although (36) and (37) differ as functions of θ\theta, their parameter gradients coincide at θk\theta_{k}. Thus, the penalty contribution can either be included in the detached residual or differentiated explicitly. Decreasing ε\varepsilon strengthens the penalty but increases the Lipschitz bound for Aε\mathcal{A}_{\varepsilon}, potentially reducing the admissible step size and worsening the conditioning of the iteration.

Hard-constrained parameterization

We next enforce the obstacle constraint in (33) by construction. If essential boundary conditions are also included in KK, they must be imposed simultaneously through a suitable boundary-conforming parameterization.

A simple choice is the squared-network parameterization [16]

u^θ:=ψ+Nθ2.(38)\widehat{u}_{\theta}:=\psi+N_{\theta}^{2}. \tag*{(38)}

Provided that ψ,Nθ2∈V\psi,N_{\theta}^{2}\in V, this ensures u^θ∈K\widehat{u}_{\theta}\in K for every θ\theta.

For this ansatz, the penalty term in (37) vanishes identically, leaving the hard-constrained NeRI objective

J~kVI(θ):=⟨sg⁡(A(u^θk)−f),u^θ⟩V∗,V.(39)\widetilde{\mathcal{J}}_{k}^{\mathrm{VI}}(\theta):=\left\langle\operatorname{sg}\bigl(\mathcal{A}(\widehat{u}_{\theta_{k}})-f\bigr),\widehat{u}_{\theta}\right\rangle_{V^{*},V}. \tag*{(39)}

We then perform the update

θk+1=θk−ηk∇θJ~kVI(θ)∣θ=θk.\theta_{k+1}=\theta_{k}-\eta_{k}\nabla_{\theta}\widetilde{\mathcal{J}}_{k}^{\mathrm{VI}}(\theta)\big|_{\theta=\theta_{k}}.

In our numerical experience, the hard-constrained formulation often outperforms the penalty approach, while enforcing feasibility exactly and avoiding the need to tune an obstacle penalty parameter. Its convergence analysis, however, is more involved, since the nonlinear parameterization changes the tangent geometry and may introduce degeneracies on the contact set. In particular, stationarity of the parameter iteration does not, in general, imply satisfaction of the original VI. A rigorous analysis of these stationary points and the convergence of the hard-constrained NeRI iteration is left for future work.

Lazy training and error analysis

In this section, we analyze lazy training for NeRI under gradient flow (depending on an artificial time tt). We then combine the frozen-kernel approximation with reduced-space estimates to obtain an error decomposition relative to the exact solution.

Let θ0\theta_0 denote the initial network parameters. Writing Gθ:=DθuθG_\theta:= D_\theta u_\theta and Kθ=GθGθ∗K_\theta= G_\theta G_\theta^*, the continuous-time NeRI dynamics are

θ˙t=−Gθt∗rt,rt:=A(uθt)−f,θ(0)=θ0,(40)\dot{\theta}_t = -G_{\theta_t}^*r_t,\qquad r_t := \mathcal{A}(u_{\theta_t}) - f,\qquad\theta(0) = \theta_0, \tag*{(40)}

with θ˙t\dot{\theta}_t denoting the derivative with respect to tt. By the chain rule, ut:=uθtu_t := u_{\theta_t} satisfies the exact function-space equation

u˙t=−Kθt(A(ut)−f),u(0)=uθ0.(41)\dot{u}_t = -K_{\theta_t}\bigl(\mathcal{A}(u_t) - f\bigr),\qquad u(0) = u_{\theta_0}. \tag*{(41)}

An architecture-independent lazy-training theorem

We first establish a general estimate that isolates the mechanism behind kernel freezing. For a twice Fréchet differentiable parameterization uθ:Rpm→Vu_\theta: \mathbb{R}^{p_m} \to V, define

Hθ:=Dθ2uθ:Rpm×Rpm→V,∥Hθ∥bil,V:=sup⁡∣a∣=∣b∣=1∥Hθ[a,b]∥V.H_\theta:= D_\theta^2u_\theta: \mathbb{R}^{p_m} \times\mathbb{R}^{p_m} \to V,\qquad\|H_\theta\|_{\mathrm{bil},V} := \sup_{|a|=|b|=1}\|H_\theta[a,b]\|_V.

For R>0R > 0, set

BR(θ0):={θ:∣θ−θ0∣≤R},hm,R:=sup⁡θ∈BR(θ0)∥Hθ∥bil,V.B_R(\theta_0) := \{\theta: |\theta-\theta_0| \le R\},\qquad h_{m,R} := \sup_{\theta\in B_R(\theta_0)}\|H_\theta\|_{\mathrm{bil},V}.

The following lemma relates Hessian bounds to Jacobian and kernel stability.

Lemma 5.1. Let θ,ϑ∈Rpm\theta,\vartheta\in\mathbb{R}^{p_m}. Suppose that, for some h≥0h \ge0, sup⁡s∈[0,1]∥Hϑ+s(θ−ϑ)∥bil,V≤h\sup_{s\in[0,1]}\|H_{\vartheta+s(\theta-\vartheta)}\|_{\mathrm{bil},V} \le h. Then

∥Gθ−Gϑ∥L(Rpm,V)≤h∣θ−ϑ∣,∥Kθ−Kϑ∥L(V∗,V)≤(∥Gθ∥+∥Gϑ∥)h∣θ−ϑ∣.\|G_\theta-G_\vartheta\|_{\mathcal{L}(\mathbb{R}^{p_m},V)} \le h|\theta-\vartheta|,\qquad\|K_\theta-K_\vartheta\|_{\mathcal{L}(V^*,V)} \le(\|G_\theta\|+\|G_\vartheta\|)h|\theta-\vartheta|.

Proof. Fix q∈Rpmq \in\mathbb{R}^{p_m} and put d:=θ−ϑd := \theta-\vartheta. The map Fq(s):=Gϑ+sdqF_q(s) := G_{\vartheta+sd}q satisfies

Fq′(s)=Hϑ+sd[d,q],∥Fq′(s)∥V≤h∣d∣∣q∣,0≤s≤1.F_q'(s) = H_{\vartheta+sd}[d,q],\qquad\|F_q'(s)\|_V \le h|d||q|,\qquad0 \le s \le1.

The mean-value inequality therefore gives ∥(Gθ−Gϑ)q∥V≤h∣θ−ϑ∣∣q∣\|(G_\theta-G_\vartheta)q\|_V \le h|\theta-\vartheta||q|. Taking the supremum over ∣q∣=1|q|=1 proves the first estimate.

For the tangent operator, write

Kθ−Kϑ=(Gθ−Gϑ)Gθ∗+Gϑ(Gθ∗−Gϑ∗).K_\theta-K_\vartheta=(G_\theta-G_\vartheta)G_\theta^*+G_\vartheta(G_\theta^*-G_\vartheta^*).

The second estimate follows from the first and the identity ∥T∗∥=∥T∥\|T^*\|=\|T\|.

We next bound the residual and parameter displacement on a finite time interval.

Lemma 5.2. Suppose Assumption 2.1 holds and ∥Gθt∥≤MG\|G_{\theta_t}\| \le M_G for 0≤t≤T0 \le t \le T. Then

∥rt∥V∗≤∥r0∥V∗exp⁡(LMG2t),(42)\|r_t\|_{V^*} \le\|r_0\|_{V^*}\exp(LM_G^2t), \tag*{(42)}
∣θt−θ0∣≤MG∥r0∥V∗∫0texp⁡(LMG2s) ds.(43)|\theta_t-\theta_0| \le M_G\|r_0\|_{V^*}\int_0^t\exp(LM_G^2s)\,ds. \tag*{(43)}

Proof. Equation (41) gives

∥u˙t∥V≤∥Kθt∥∥rt∥V∗≤MG2∥rt∥V∗.\|\dot{u}_t\|_V \le\|K_{\theta_t}\|\|r_t\|_{V^*} \le M_G^2\|r_t\|_{V^*}.

Integrating and using Lipschitz continuity of A\mathcal{A}, we obtain

∥rt∥V∗≤∥r0∥V∗+L∥ut−u0∥V≤∥r0∥V∗+LMG2∫0t∥rs∥V∗ ds.\|r_t\|_{V^*} \le\|r_0\|_{V^*} + L\|u_t-u_0\|_V \le\|r_0\|_{V^*} + LM_G^2\int_0^t\|r_s\|_{V^*}\,ds.

Since t↦∥rt∥V∗t\mapsto\|r_t\|_{V^*} is continuous, the integral Gronwall inequality yields (42). The parameter flow then satisfies

∣θ˙t∣≤∥Gθt∥∥rt∥V∗≤MG∥r0∥V∗eLMG2t.|\dot{\theta}_t| \le\|G_{\theta_t}\|\|r_t\|_{V^*} \le M_G\|r_0\|_{V^*}e^{LM_G^2t}.

Integrating from 00 to tt proves the displacement bound.

These estimates allow us to establish lazy training for NeRI. The argument combines Hessian-based stability inside a fixed parameter ball with a bound on the displacement of the trajectory.

Theorem 5.3. Fix T>0T>0 and suppose Assumption 2.1 holds. Assume that for every δ∈(0,1)\delta\in(0,1) there exist a deterministic sequence εm→0\varepsilon_m\to0, events EmE_m with P(Em)≥1−δ\mathbb{P}(E_m)\ge1-\delta for all sufficiently large m∈Nm\in\mathbb{N}, and constants M0,C0,R0>0M_0,C_0,R_0>0 independent of mm, such that on EmE_m

∥Gθ0∥≤M0,∥r0∥V∗≤C0,hm,R0≤εm.\|G_{\theta_0}\|\le M_0,\qquad\|r_0\|_{V^*}\le C_0,\qquad h_{m,R_0}\le\varepsilon_m.

If

R0>RT:=2M0C0∫0Texp⁡(4LM02s) ds,R_0>R_T:=2M_0C_0\int_0^T\exp(4LM_0^2s)\,ds,

then, on EmE_m for all sufficiently large mm, the trajectory exists on [0,T][0,T], remains in BR0(θ0)B_{R_0}(\theta_0), and satisfies

sup⁡0≤t≤T∥Kθt−Kθ0∥L(V∗,V)≤3M0RTεm.(44)\sup_{0\le t\le T}\|K_{\theta_t}-K_{\theta_0}\|_{\mathcal{L}(V^*,V)}\le3M_0R_T\varepsilon_m. \tag*{(44)}

In particular, the tangent-operator drift converges to zero in probability.

Proof. Fix an event EmE_m on which the assumed bounds hold and choose mm large enough that εmR0≤M0\varepsilon_mR_0\le M_0. By Lemma 5.1, the vector field in (40) is locally Lipschitz inside BR0(θ0)B_{R_0}(\theta_0), so a unique local solution exists.

Let τ\tau be the first time the trajectory reaches the boundary of this ball, with τ=+∞\tau=+\infty if no such time exists. Before this exit, Lemma 5.1 gives

∥Gθt−Gθ0∥≤εm∣θt−θ0∣≤εmR0≤M0,\|G_{\theta_t}-G_{\theta_0}\|\le\varepsilon_m|\theta_t-\theta_0|\le\varepsilon_mR_0\le M_0,

and hence ∥Gθt∥≤2M0\|G_{\theta_t}\|\le2M_0. Applying Lemma 5.2 with MG=2M0M_G=2M_0 yields, for t≤Tt\le T before the exit,

∣θt−θ0∣≤2M0C0∫0te4LM02s ds≤RT<R0.|\theta_t-\theta_0|\le2M_0C_0\int_0^t e^{4LM_0^2s}\,ds\le R_T<R_0.

At an exit time τ≤T\tau\le T, this would contradict ∣θτ−θ0∣=R0|\theta_\tau-\theta_0|=R_0. The trajectory thus stays in a compact subset of the ball and extends to TT. In particular,

sup⁡0≤t≤T∣θt−θ0∣≤RT,sup⁡0≤t≤T∥Gθt∥≤2M0.\sup_{0\le t\le T}|\theta_t-\theta_0|\le R_T,\qquad\sup_{0\le t\le T}\|G_{\theta_t}\|\le2M_0.

A second application of Lemma 5.1 gives

∥Kθt−Kθ0∥≤(∥Gθt∥+∥Gθ0∥)εm∣θt−θ0∣≤3M0RTεm.\|K_{\theta_t}-K_{\theta_0}\|\le(\|G_{\theta_t}\|+\|G_{\theta_0}\|)\varepsilon_m|\theta_t-\theta_0|\le3M_0R_T\varepsilon_m.

This proves (44). Since δ>0\delta>0 is arbitrary and εm→0\varepsilon_m\to0, the drift converges to zero in probability.

We next compare NeRI with the frozen-kernel flow. Set Km0:=Kθ0K_m^0 := K_{\theta_0} and let vtv_t solve

v˙t=−Km0(A(vt)−f),v0=uθ0.(45)\dot{v}_t=-K_m^0\left(\mathcal{A}(v_t)-f\right),\qquad v_0=u_{\theta_0}. \tag*{(45)}

To relate this flow to the exact solution, define

Xm:=Ran⁡(Km0),Sm:=uθ0+Xm,Bm:=Km0R−1:V→V.X_m:=\operatorname{Ran}(K_m^0),\qquad S_m:=u_{\theta_0}+X_m,\qquad B_m:=K_m^0\mathcal{R}^{-1}:V\to V.

The operator BmB_m is self-adjoint and positive definite on its finite-dimensional range XmX_m. Hence there exist 0<γK,m≤MK,m0<\gamma_{K,m}\le M_{K,m} such that

γK,m∥w∥V2≤⟨Bmw,w⟩V≤MK,m∥w∥V2,w∈Xm.(46)\gamma_{K,m}\lVert w\rVert_V^2\le\langle B_mw,w\rangle_V\le M_{K,m}\lVert w\rVert_V^2,\qquad w\in X_m. \tag*{(46)}

Let uSm∗∈Smu_{S_m}^*\in S_m denote the unique reduced solution satisfying

⟨A(uSm∗)−f,w⟩V∗,V=0,w∈Xm.(47)\langle\mathcal{A}(u_{S_m}^*)-f,w\rangle_{V^*,V}=0,\qquad w\in X_m. \tag*{(47)}

The Moore–Penrose pseudoinverse Bm†B_m^\dagger defines the norm

∥w∥Bm†2:=⟨Bm†w,w⟩V,w∈Xm.\lVert w\rVert_{B_m^\dagger}^2:=\langle B_m^\dagger w,w\rangle_V,\qquad w\in X_m.

Theorem 5.4. Under the assumptions of Theorem 5.3, there is a constant CTC_T, independent of mm, such that on the same high-probability event, for all sufficiently large mm,

sup⁡0≤t≤T∥uθt−vt∥V≤CTεm.(48)\sup_{0\le t\le T}\lVert u_{\theta_t}-v_t\rVert_V\le C_T\varepsilon_m. \tag*{(48)}

Moreover, the frozen flow satisfies, for t≥0t\ge0,

∥vt−uSm∗∥Bm†≤e−γγK,mt∥v0−uSm∗∥Bm†.(49)\lVert v_t-u_{S_m}^*\rVert_{B_m^\dagger}\le e^{-\gamma\gamma_{K,m}t}\lVert v_0-u_{S_m}^*\rVert_{B_m^\dagger}. \tag*{(49)}

If u∗u^* solves A(u∗)=f\mathcal{A}(u^*)=f, then

∥vt−u∗∥V≤MK,me−γγK,mt∥v0−uSm∗∥Bm†+Lγinf⁡w∈Sm∥w−u∗∥V.(50)\lVert v_t-u^*\rVert_V\le\sqrt{M_{K,m}}e^{-\gamma\gamma_{K,m}t}\lVert v_0-u_{S_m}^*\rVert_{B_m^\dagger}+\frac{L}{\gamma}\inf_{w\in S_m}\lVert w-u^*\rVert_V. \tag*{(50)}

Consequently, for 0≤t≤T0\le t\le T,

∥uθt−u∗∥V≤CTεm+MK,me−γγK,mt∥uθ0−uSm∗∥Bm†+Lγinf⁡w∈Sm∥w−u∗∥V.(51)\lVert u_{\theta_t}-u^*\rVert_V\le C_T\varepsilon_m+\sqrt{M_{K,m}}e^{-\gamma\gamma_{K,m}t}\lVert u_{\theta_0}-u_{S_m}^*\rVert_{B_m^\dagger}+\frac{L}{\gamma}\inf_{w\in S_m}\lVert w-u^*\rVert_V. \tag*{(51)}

Proof. The frozen equation is globally well posed because A\mathcal{A} is globally Lipschitz. Set ξt:=uθt−vt\xi_t:=u_{\theta_t}-v_t. Subtracting the evolution equations gives

ξ˙t=−(Kθt−Km0)rt−Km0(A(uθt)−A(vt)),ξ0=0.\dot{\xi}_t=-(K_{\theta_t}-K_m^0)r_t-K_m^0\left(\mathcal{A}(u_{\theta_t})-\mathcal{A}(v_t)\right),\qquad\xi_0=0.

Hence,

∥ξ˙t∥V≤∥Kθt−Km0∥∥rt∥V∗+L∥Km0∥∥ξt∥V.\lVert\dot{\xi}_t\rVert_V\le\lVert K_{\theta_t}-K_m^0\rVert\lVert r_t\rVert_{V^*}+L\lVert K_m^0\rVert\lVert\xi_t\rVert_V.

On the event of Theorem 5.3,

sup⁡t≤T∥Kθt−Km0∥≤3M0RTεm,∥Km0∥≤M02,\sup_{t\le T}\lVert K_{\theta_t}-K_m^0\rVert\le3M_0R_T\varepsilon_m,\qquad\lVert K_m^0\rVert\le M_0^2,

and Lemma 5.2 gives sup⁡t≤T∥rt∥V∗≤C0e4LM02T\sup_{t\le T}\lVert r_t\rVert_{V^*}\le C_0e^{4LM_0^2T}. Thus, for a constant ATA_T independent of mm,

∥ξt∥V≤ATεmt+LM02∫0t∥ξs∥V ds.\lVert\xi_t\rVert_V\le A_T\varepsilon_m t+LM_0^2\int_0^t\lVert\xi_s\rVert_V\,ds.

The integral Gronwall inequality [3] yields ∥ξt∥V≤ATTeLM22Tϵm\|\xi_t\|_V \le A_T T e^{L M_2^2 T}\epsilon_m for 0≤t≤T0 \le t \le T, proving (48).

We next estimate the frozen flow itself. Since v˙t∈Xm\dot{v}_t \in X_m, we have vt∈Smv_t \in S_m for all t≥0t \ge0. Set et:=vt−uSm∗∈Xme_t := v_t - u^*_{S_m} \in X_m. By (47), Km0(A(uSm∗)−f)=0K_m^0(\mathcal{A}(u^*_{S_m}) - f) = 0, and hence

e˙t=−Km0(A(vt)−A(uSm∗)).\dot{e}_t = -K_m^0(\mathcal{A}(v_t) - \mathcal{A}(u^*_{S_m})).

Using Km0=BmRK_m^0 = B_m R and et∈Xme_t \in X_m, we obtain

12ddt∥et∥Bm∗2=−⟨A(vt)−A(uSm∗),et⟩V∗,V≤−γ∥et∥V2≤−γγK,m∥et∥Bm∗2.\frac{1}{2}\frac{d}{dt}\|e_t\|_{B_m^*}^2 = -\langle\mathcal{A}(v_t) - \mathcal{A}(u^*_{S_m}),e_t\rangle_{V^*,V} \le-\gamma\|e_t\|_V^2 \le-\gamma\gamma_{K,m}\|e_t\|_{B_m^*}^2.

Gronwall’s inequality proves (49), while (46) gives ∥et∥V≤MK,m∥et∥Bm∗\|e_t\|_V \le\sqrt{M_{K,m}}\|e_t\|_{B_m^*}.

Finally, applying the reduced-space estimate (23) with S=SmS=S_m yields

∥uSm∗−u∗∥V≤Lγinf⁡w∈Sm∥w−u∗∥V.(52)\|u^*_{S_m}-u^*\|_V \le\frac{L}{\gamma}\inf_{w\in S_m}\|w-u^*\|_V. \tag*{(52)}

Combining this bound with (49) proves (50). The total error estimate (51) follows from (48) and the triangle inequality.

Estimate (51) separates the total error into finite-width lazy-training, frozen-flow optimization, and reduced-space approximation errors.

Two-layer networks in H1(Ω)^1(\Omega)

We now verify the abstract hypotheses for a two-layer network in V=H1(Ω)V=H^1(\Omega), where Ω⊂Rd\Omega\subset\mathbb{R}^d is bounded and dd is fixed. Consider the two-layer network

uθ(m)(x)=1m∑i=1maiσ(wi⋅x+bi).(53)u^{(m)}_\theta(x)=\frac{1}{\sqrt{m}}\sum_{i=1}^{m}a_i\sigma(w_i\cdot x+b_i). \tag*{(53)}

We use the stochastic order notation of van der Vaart [40]. For random elements ZmZ_m of a normed space (X,∥⋅∥X)(\mathcal{X},\|\cdot\|_{\mathcal{X}}), we write Zm=OP(1)Z_m=O_{\mathrm{P}}(1) if they are bounded in probability: for every ϵ>0\epsilon>0, there exist Mϵ<∞M_\epsilon<\infty and mϵm_\epsilon such that P(∥Zm∥X>Mϵ)≤ϵ\mathbb{P}(\|Z_m\|_{\mathcal{X}}>M_\epsilon)\le\epsilon for all m≥mϵm\ge m_\epsilon. For a deterministic sequence am>0a_m>0, the notation Zm=OP(am)Z_m=O_{\mathrm{P}}(a_m) means Zm/am=OP(1)Z_m/a_m=O_{\mathrm{P}}(1), while Zm=oP(1)Z_m=o_{\mathrm{P}}(1) means ∥Zm∥X→0\|Z_m\|_{\mathcal{X}}\to0 in probability.

Assumption 5.5. The activation satisfies σ∈Cb3(R)\sigma\in C_b^3(\mathbb{R}). At initialization, the blocks (ai,wi,bi)(a_i,w_i,b_i) are independent and identically distributed, all coordinates are sub-Gaussian with width-independent sub-Gaussian norms, and aia_i is centered and independent of (wi,bi)(w_i,b_i).

The centering and independence of the output weights ensure that the initial network is OP(1)O_{\mathrm{P}}(1) in H1(Ω)H^1(\Omega), as shown below.

Lemma 5.6. Under Assumptions 5.5 and 2.1,

∥uθ0(m)∥H1=OP(1),∥Gθ0(m)∥L(Rpm,H1)=OP(1),∥A(uθ0(m))−f∥H1(Ω)∗=OP(1).\|u^{(m)}_{\theta_0}\|_{H^1}=O_{\mathrm{P}}(1),\qquad\|G^{(m)}_{\theta_0}\|_{\mathcal{L}(\mathbb{R}^{pm},H^1)}=O_{\mathrm{P}}(1),\qquad\|\mathcal{A}(u^{(m)}_{\theta_0})-f\|_{H^1(\Omega)^*}=O_{\mathrm{P}}(1).

Proof. Write zi(x)=wi⋅x+biz_i(x)=w_i\cdot x+b_i. By centering and independence, all cross terms vanish in the second moment of the width-normalized sum. Hence,

E∥uθ0(m)∥L2(Ω)2=1m∑i=1mE∫Ωai2σ(zi(x))2 dx≤C∣Ω∣,\mathbb{E}\|u^{(m)}_{\theta_0}\|_{L^2(\Omega)}^2=\frac{1}{m}\sum_{i=1}^{m}\mathbb{E}\int_\Omega a_i^2\sigma(z_i(x))^2\,dx\le C|\Omega|,
E∥∇uθ0(m)∥L2(Ω)2=1m∑i=1mE∫Ωai2σ′(zi(x))2∣wi∣2 dx≤C.\mathbb{E}\|\nabla u^{(m)}_{\theta_0}\|_{L^2(\Omega)}^2=\frac{1}{m}\sum_{i=1}^{m}\mathbb{E}\int_\Omega a_i^2\sigma'(z_i(x))^2|w_i|^2\,dx\le C.

The relevant initialization moments are uniformly bounded, so CC is independent of mm. Markov’s inequality gives the first estimate.

For the Jacobian, the parameter derivatives are

∂aiu=m−1/2σ(zi),∂wi,ju=m−1/2aiσ′(zi)xj,∂biu=m−1/2aiσ′(zi).\partial_{a_i}u=m^{-1/2}\sigma(z_i),\qquad\partial_{w_{i,j}}u=m^{-1/2}a_i\sigma'(z_i)x_j,\qquad\partial_{b_i}u=m^{-1/2}a_i\sigma'(z_i).

Their spatial gradients are

∇∂aiu=m−1/2σ′(zi)wi,∇∂biu=m−1/2aiσ′′(zi)wi,\nabla\partial_{a_i}u=m^{-1/2}\sigma'(z_i)w_i,\qquad\nabla\partial_{b_i}u=m^{-1/2}a_i\sigma''(z_i)w_i,
∇∂wi,ju=m−1/2ai[σ′′(zi)xjwi+σ′(zi)ej].\nabla\partial_{w_{i,j}}u=m^{-1/2}a_i\left[\sigma''(z_i)x_jw_i+\sigma'(z_i)e_j\right].

Since Ω\Omega and σ,σ′,σ′′\sigma,\sigma',\sigma'' are bounded, summing the squared H1H^1 norms gives

∥Gθ0(m)∥HS(Rpm,H1)2≤Cm∑i=1m(1+ai2)(1+∣wi∣2).\lVert G_{\theta_0}^{(m)}\rVert_{\mathrm{HS}(\mathbb{R}^{pm},H^1)}^2\le\frac{C}{m}\sum_{i=1}^{m}(1+a_i^2)(1+\lvert w_i\rvert^2).

Uniform moment bounds and Markov’s inequality make the right-hand side OP(1)O_{\mathbb{P}}(1). Since the operator norm is bounded by the Hilbert–Schmidt norm, the Jacobian estimate follows. Finally, the residual bound follows from the following inequality:

∥A(uθ0(m))−f∥H1(Ω)∗≤L∥uθ0(m)∥H1+∥A(0)−f∥H1(Ω)∗.\lVert\mathcal{A}(u_{\theta_0}^{(m)})-f\rVert_{H^1(\Omega)^*}\le L\lVert u_{\theta_0}^{(m)}\rVert_{H^1}+\lVert\mathcal{A}(0)-f\rVert_{H^1(\Omega)^*}.

□\square

We now derive the following Hessian estimate.

Lemma 5.7. For fixed R>0R>0, Assumption 5.5 implies

sup⁡θ∈BR(θ0)∥Dθ2u(m)∥bil,H1=OP(log⁡mm).(54)\sup_{\theta\in B_R(\theta_0)}\lVert D_\theta^2u^{(m)}\rVert_{\mathrm{bil},H^1}=O_{\mathbb{P}}\left(\frac{\log m}{\sqrt{m}}\right). \tag*{(54)}

Proof. For one neuron, write

ϕ(a,w,b;x)=m−1/2aσ(z),z=w⋅x+b.\phi(a,w,b;x)=m^{-1/2}a\sigma(z),\qquad z=w\cdot x+b.

For block directions p=(α,r,β)p=(\alpha,r,\beta) and q=(α~,r~,β~)q=(\widetilde{\alpha},\widetilde{r},\widetilde{\beta}), set ρp=r⋅x+β\rho_p=r\cdot x+\beta and ρq=r~⋅x+β~\rho_q=\widetilde{r}\cdot x+\widetilde{\beta}. Direct differentiation gives

D2ϕ[p,q]=1m[ασ′(z)ρq+α~σ′(z)ρp+aσ′′(z)ρpρq].(55)D^2\phi[p,q]=\frac{1}{\sqrt{m}}\left[\alpha\sigma'(z)\rho_q+\widetilde{\alpha}\sigma'(z)\rho_p+a\sigma''(z)\rho_p\rho_q\right]. \tag*{(55)}

Boundedness of Ω\Omega gives ∣ρp(x)∣≤CΩ∣p∣\lvert\rho_p(x)\rvert\le C_\Omega\lvert p\rvert and ∣ρq(x)∣≤CΩ∣q∣\lvert\rho_q(x)\rvert\le C_\Omega\lvert q\rvert, so

∥D2ϕ[p,q]∥L2(Ω)≤Cm(1+∣a∣)∣p∣∣q∣.\lVert D^2\phi[p,q]\rVert_{L^2(\Omega)}\le\frac{C}{\sqrt{m}}(1+\lvert a\rvert)\lvert p\rvert\lvert q\rvert.

Taking one spatial derivative yields

∇xD2ϕ[p,q]=1m(α[σ′′(z)wρq+σ′(z)r~]+α~[σ′′(z)wρp+σ′(z)r]+a[σ′′′(z)wρpρq+σ′′(z)rρq+σ′′(z)r~ρp]).\nabla_xD^2\phi[p,q]=\frac{1}{\sqrt{m}}\left(\alpha[\sigma''(z)w\rho_q+\sigma'(z)\widetilde{r}]+\widetilde{\alpha}[\sigma''(z)w\rho_p+\sigma'(z)r]+a[\sigma'''(z)w\rho_p\rho_q+\sigma''(z)r\rho_q+\sigma''(z)\widetilde{r}\rho_p]\right).

Since σ′,σ′′,σ′′′\sigma',\sigma'',\sigma''' are bounded,

∥∇xD2ϕ[p,q]∥L2(Ω)≤Cm(1+∣a∣)(1+∣w∣)∣p∣∣q∣.\lVert\nabla_xD^2\phi[p,q]\rVert_{L^2(\Omega)}\le\frac{C}{\sqrt{m}}(1+\lvert a\rvert)(1+\lvert w\rvert)\lvert p\rvert\lvert q\rvert.

Partition global directions as p=(p1,…,pm)p=(p_1,\ldots,p_m) and q=(q1,…,qm)q=(q_1,\ldots,q_m). Since cross-neuron derivatives vanish, Dθ2uθ[p,q]=∑i=1mD2ϕi[pi,qi]D_\theta^2u_\theta[p,q]=\sum_{i=1}^m D^2\phi_i[p_i,q_i]. The preceding one-neuron bounds and the triangle inequality give

∥Dθ2uθ[p,q]∥H1≤Cmmax⁡i≤m(1+∣ai∣)(1+∣wi∣)∑i=1m∣pi∣∣qi∣≤Cmmax⁡i≤m(1+∣ai∣)(1+∣wi∣)∣p∣∣q∣,\lVert D_\theta^2u_\theta[p,q]\rVert_{H^1}\leq\frac{C}{\sqrt{m}}\max_{i\leq m}(1+|a_i|)(1+|w_i|)\sum_{i=1}^m |p_i||q_i| \leq\frac{C}{\sqrt{m}}\max_{i\leq m}(1+|a_i|)(1+|w_i|)|p||q|,

where the last step is Cauchy–Schwarz over the neuron blocks. For θ∈BR(θ0)\theta\in B_R(\theta_0),

∣ai∣≤∣ai(0)∣+R,∣wi∣≤∣wi(0)∣+R.|a_i|\leq|a_i(0)|+R,\qquad|w_i|\leq|w_i(0)|+R.

The sub-Gaussian maximal bounds, with dd fixed, yield

max⁡i≤m∣ai(0)∣=OP(log⁡m),max⁡i≤m∣wi(0)∣=OP(log⁡m).\max_{i\leq m}|a_i(0)|=O_{\mathbb{P}}(\sqrt{\log m}),\qquad\max_{i\leq m}|w_i(0)|=O_{\mathbb{P}}(\sqrt{\log m}).

Thus the product factor is OP(log⁡m)O_{\mathbb{P}}(\log m) uniformly on BR(θ0)B_R(\theta_0). Taking the supremum over unit p,qp,q proves (54).

Theorem 5.8. Let Ω⊂Rd\Omega\subset\mathbb{R}^d be bounded and suppose Assumptions 2.1 and 5.5 hold. For every fixed T>0T>0,

sup⁡0≤t≤T∥Kθt(m)−Kθ0(m)∥L(H1(Ω)∗,H1(Ω))=OP(log⁡mm).\sup_{0\leq t\leq T}\left\|K_{\theta_t}^{(m)}-K_{\theta_0}^{(m)}\right\|_{\mathcal{L}(H^1(\Omega)^*,H^1(\Omega))}=O_{\mathbb{P}}\left(\frac{\log m}{\sqrt{m}}\right).

If vt(m)v_t^{(m)} solves (45), then

sup⁡0≤t≤T∥uθt(m)−vt(m)∥H1=OP(log⁡mm).\sup_{0\leq t\leq T}\left\|u_{\theta_t}^{(m)}-v_t^{(m)}\right\|_{H^1}=O_{\mathbb{P}}\left(\frac{\log m}{\sqrt{m}}\right).

Consequently, for 0≤t≤T0\leq t\leq T,

∥uθt(m)−u∗∥H1≤MK,me−γγK,mt∥uθ0(m)−uSm∗∥Bm∗+Lγinf⁡w∈Sm∥w−u∗∥H1+OP(log⁡mm).(56)\left\|u_{\theta_t}^{(m)}-u^*\right\|_{H^1}\leq\sqrt{M_{K,m}}e^{-\gamma\gamma_{K,m}t}\left\|u_{\theta_0}^{(m)}-u_{S_m}^*\right\|_{B_m^*} +\frac{L}{\gamma}\inf_{w\in S_m}\left\|w-u^*\right\|_{H^1}+O_{\mathbb{P}}\left(\frac{\log m}{\sqrt{m}}\right). \tag*{(56)}

where the stochastic remainder is uniform on [0,T][0,T].

Proof. Fix T>0T>0 and δ>0\delta>0. Lemma 5.6 gives constants M0,C0>0M_0,C_0>0, independent of mm, such that

P(∥Gθ0(m)∥≤M0, ∥A(uθ0(m))−f∥H1(Ω)∗≤C0)≥1−δ/2\mathbb{P}\left(\left\|G_{\theta_0}^{(m)}\right\|\leq M_0,\ \left\|\mathcal{A}(u_{\theta_0}^{(m)})-f\right\|_{H^1(\Omega)^*}\leq C_0\right)\geq1-\delta/2

for all sufficiently large mm. Choose R0>RTR_0>R_T, with RTR_T defined from these constants in Theorem 5.3. Applying Lemma 5.7 on BR0(θ0)B_{R_0}(\theta_0) gives a constant CδC_\delta such that

P(sup⁡θ∈BR0(θ0)∥Dθ2uθ(m)∥bil,H1≤Cδlog⁡mm)≥1−δ/2.\mathbb{P}\left(\sup_{\theta\in B_{R_0}(\theta_0)}\left\|D_\theta^2u_\theta^{(m)}\right\|_{\mathrm{bil},H^1}\leq C_\delta\frac{\log m}{\sqrt{m}}\right)\geq1-\delta/2.

The intersection has probability at least 1−δ1-\delta, and Theorem 5.3 applies with εm=Cδ(log⁡m)/m\varepsilon_m=C_\delta(\log m)/\sqrt{m}. The kernel-drift bound follows from (44), and Theorem 5.4 gives the frozen-flow approximation. Since δ>0\delta>0 is arbitrary, both estimates have the stated stochastic order. Finally, (56) follows from (51).

Estimate (56) separates the total error into finite-width lazy-training, frozen-flow optimization, and reduced-space approximation errors. The frozen-flow approximation holds for fixed TT, with constants that may depend on TT. These estimates do not by themselves establish convergence to u∗u^* in a joint large-width and long-time limit. Such convergence requires further control of the reduced-space approximation error and compatibility between the width-dependent optimization time and the validity interval of the frozen-flow approximation.

Numerical results

In this section, we evaluate the performance of NeRI on several representative problems and compare it with PINN and WAN. Unless otherwise specified, all experiments use a residual neural network architecture consisting of NblkN_{\mathrm{blk}} residual blocks, each containing two fully connected layers and a residual connection. We set Nblk=2N_{\mathrm{blk}}=2 for two-dimensional problems and Nblk=3N_{\mathrm{blk}}=3 for high-dimensional problems. Each hidden layer has width 100, and the hyperbolic tangent function is used as the activation throughout.

All networks are trained using Adam, and we use cosine annealing to decrease the learning rate from 1.0×10−31.0\times10^{-3} to 1.0×10−81.0\times10^{-8} over the full training run. At each training epoch, the interior and boundary points are sampled independently and uniformly in Ω\Omega and on ∂Ω\partial\Omega, respectively. We use NΩ=10,000N_{\Omega}=10{,}000 interior points in all experiments, with 256 points per boundary segment (including the slit, when present) in two dimensions and 2,000 points per boundary face in higher dimensions. The experiments were implemented in PyTorch and executed on a MacBook Pro equipped with an Apple M5 chip. The implementation code of the numerical experiments is publicly available at https://github.com/JianfengNing/NeRI-Code.

A nonsymmetric elliptic problem with a corner singularity

Let Ω=(−1,1)2∖([0,1)×{0})\Omega=(-1,1)^2\setminus([0,1)\times\{0\}) be the slit domain, consider the nonsymmetric elliptic problem

{−Δu+ux1=fin Ω,u=gon ∂Ω.\begin{cases} -\Delta u+u_{x_1}=f & \text{in }\Omega,\\ u=g & \text{on }\partial\Omega. \end{cases}

The tip of the slit at the origin gives rise to a corner singularity. We prescribe the exact solution u∗(r,θ)=r1/2sin⁡(θ2)u^*(r,\theta)=r^{1/2}\sin\left(\frac{\theta}{2}\right), where (r,θ)(r,\theta) are polar coordinates centered at the origin, and choose ff and gg accordingly. The solution belongs to H1(Ω)H^1(\Omega) but not to H2(Ω)H^2(\Omega). The crack-tip singularity can pose approximation and optimization challenges for standard strong-form PINNs, whereas NeRI and WAN use weak formulations involving only first-order spatial derivatives.

We compare NeRI with PINN and WAN. For all three methods, the Dirichlet boundary condition is imposed through a quadratic penalty with parameter β/2=50\beta/2=50, and the networks are trained for 15,000 epochs. At each epoch, WAN performs one minimization update and two maximization updates.

The numerical approximations and corresponding pointwise absolute errors are shown in Figure 1a. NeRI yields the smallest error among the three methods, while PINN exhibits noticeably larger errors near the singular point and WAN gives an intermediate accuracy. The relative L2L^2 errors obtained by NeRI, PINN, and WAN are 6.24×10−36.24\times10^{-3}, 1.01×10−11.01\times10^{-1}, and 5.95×10−25.95\times10^{-2}, respectively. Figure 1b shows the corresponding error histories during training. The error of NeRI decreases steadily and remains stable throughout the optimization. PINN decreases rapidly at the initial stage but subsequently stagnates at a comparatively high error level, consistent with the difficulty of strong-form training for solutions with limited regularity. WAN also shows a stable initial decrease, but reaches a higher error level than NeRI, illustrating the additional optimization difficulty associated with its min–max formulation.

Numerical results for Example 6.1

Figure 1. (a) The first row presents the exact solution, numerical solutions obtained by NeRI, PINN, and WAN. The second row presents the corresponding absolute errors. (b) Relative L2L^2 error histories during training. Numerical results for Example 6.1.

To more directly connect the numerical experiments with the theoretical setting of § 5, we additionally test NeRI using plain gradient descent and two-layer networks with different widths. The learning rate is fixed at 5×10−35 \times10^{-3}, and all networks are trained for 100,000 epochs. As is commonly observed in neural PDE solvers, adaptive optimizers such as Adam often exhibit more favorable empirical optimization behavior than vanilla gradient descent. Consistently, Table 1 shows that plain gradient descent leads to a larger error than Adam for NeRI in this example, while still achieving reasonable accuracy. The resulting errors also remain below the PINN and WAN errors reported above, although the corresponding architectures and optimizers are different.

Width2550100200
Relative L2L^2 Error4.84×10−24.84 \times10^{-2}4.43×10−24.43 \times10^{-2}4.09×10−24.09 \times10^{-2}4.23×10−24.23 \times10^{-2}

Table 1. Relative L2L^2 errors of NeRI for Example 6.1 using plain gradient descent and two-layer networks with different widths.

A low-regularity problem with a discontinuous diffusion coefficient

Let Ω=(0,1)2\Omega=(0,1)^2 and consider the nonsymmetric elliptic problem

{−∇⋅(a(x)∇u)+ux1=10in Ω,u=0on ∂Ω.\begin{cases} -\nabla\cdot\left(a(x)\nabla u\right)+u_{x_1}=10 & \text{in }\Omega,\\ u=0 & \text{on }\partial\Omega. \end{cases}

The diffusion coefficient is piecewise constant with a(x)=3a(x)=3 for 13≤x1≤23\frac{1}{3}\le x_1\le\frac{2}{3}, and a(x)=1a(x)=1 otherwise. The jumps in aa across the interfaces reduce the global regularity of the solution and make the problem challenging for standard strong-form neural solvers, whereas the weak formulation naturally accommodates the discontinuous coefficient.

We compare NeRI with PINN and WAN. For all three methods, the homogeneous Dirichlet condition is imposed through the hard-constrained parameterization uθ=Nθwu_\theta=\mathcal{N}_\theta w with w(x)=x1(1−x1)x2(1−x2)w(x)=x_1(1-x_1)x_2(1-x_2). The networks are trained for 20,000 epochs. The numerical approximations and corresponding pointwise absolute errors are shown in Figure 2. The reference solution is computed using P1P_1 finite elements on a triangulated uniform 300×300300\times300 grid with centroid quadrature. NeRI gives the smallest error among the three methods and captures the solution behavior across the coefficient interfaces more accurately. PINN exhibits noticeably larger errors, especially near the interfaces. The relative L2L^2 errors for NeRI, PINN, and WAN are 9.13×10−49.13\times10^{-4}, 2.70×10−22.70\times10^{-2}, and 2.89×10−22.89\times10^{-2}, respectively.

Numerical results for Example 6.2

Figure 2. Numerical results for Example 6.2. The first row presents the exact solution, numerical approximations obtained by NeRI, PINN, and WAN. The second row presents the corresponding absolute errors.

We emphasize that the PINN used here is a standard strong-form, single-network baseline. In an additional experiment, an interface-aware PINN using three subnetworks and penalizing jumps in the solution and the normal diffusive flux achieved a relative L2L^2 error of 1.63×10−31.63\times10^{-3}. Such approaches, however, require explicit treatment and sampling of the interfaces and may become more involved for complex geometries or high-dimensional problems. In contrast, NeRI and WAN incorporate the discontinuous coefficient directly through the weak formulation without requiring explicit interface conditions.

High-dimensional pp-Laplace problem

Let Ω=(0,1)10\Omega=(0,1)^{10}. We consider the following pp-Laplace problem studied in [17]:

{−∇⋅(∣∇u∣p−2∇u)+b⃗⋅∇u=fin Ω,u=gon ∂Ω.\begin{cases} -\nabla\cdot\left(|\nabla u|^{p-2}\nabla u\right)+\vec{b}\cdot\nabla u=f & \text{in }\Omega,\\ u=g & \text{on }\partial\Omega. \end{cases}

We set p=1.5p=1.5, f(x)=9−x1∥x∥2f(x)=\frac{9-x_{1}}{\lVert x\rVert_{2}} and b⃗=(1,0,⋯ ,0)\vec{b}=(1,0,\cdots,0). The exact solution is given as u∗(x)=1−∥x∥2u^{*}(x)=1-\lVert x\rVert_{2}. The network is trained for 6,000 epochs, and the Dirichlet boundary condition is imposed through a quadratic penalty with parameter β=200\beta=200. The numerical solution on the two-dimensional slice (x1,x2,0.5,…,0.5)(x_{1},x_{2},0.5,\ldots,0.5) is shown in Figure 3.

Numerical results for Example 6.3

Figure 3. Numerical results for Example 6.3.

The relative L2L^{2} error of NeRI over Ω\Omega is 8.04×10−38.04\times10^{-3}, and the relative L2L^{2} error on the same slice is 1.95×10−21.95\times10^{-2}. For this setting, [17] reports errors of 2.86×10−22.86\times10^{-2}, 6.71×10−26.71\times10^{-2}, and 1.03×10−11.03\times10^{-1} for IDRM, PINN, and WAN, respectively. Moreover, compared with IDRM, NeRI avoids inner surrogate minimization and the associated regularization and stopping parameters. It updates the network directly from the current weak residual, without differentiating through the nonlinear operator, yielding a simpler training procedure.

We note that the pp-Laplace operator with p=1.5p=1.5 does not satisfy Assumption 2.1, so this example lies outside the error analysis of § 5. Nevertheless, the weak adversarial interpretation remains valid where the implicit adjoint problem is well posed and the linearized operator has a strictly positive quadratic form: the pairing is then strictly positive for nonzero residuals, even without uniform equivalence to the squared dual residual norm. Numerical results indicate that NeRI remains effective for the tested problems, while extending the analysis to this broader setting is left for future work.

High-dimensional biharmonic equation

Let Ω=(0,1)10\Omega=(0,1)^{10} and consider the following high-dimensional biharmonic problem:

{Δ2u+b⃗⋅∇u=fin Ω,Δu=0,u=0on ∂Ω.\begin{cases} \Delta^{2}u+\vec{b}\cdot\nabla u=f & \text{in }\Omega,\\ \Delta u=0,\quad u=0 & \text{on }\partial\Omega. \end{cases}

We take b⃗=(1,1,0,…,0)\vec{b}=(1,1,0,\ldots,0) and set V=H2(Ω)∩H01(Ω)V=H^{2}(\Omega)\cap H_{0}^{1}(\Omega). The corresponding weak formulation is

⟨Au−f,v⟩V∗,V=∫ΩΔu Δv dx+∫Ω(ux1+ux2)v dx−∫Ωfv dx.(57)\langle Au-f,v\rangle_{V^{*},V}=\int_{\Omega}\Delta u\,\Delta v\,\mathrm{d}x+\int_{\Omega}(u_{x_{1}}+u_{x_{2}})v\,\mathrm{d}x-\int_{\Omega}fv\,\mathrm{d}x. \tag*{(57)}

The natural boundary condition Δu=0\Delta u=0 on ∂Ω\partial\Omega is incorporated through the weak formulation. To enforce the essential condition u=0u=0 on ∂Ω\partial\Omega, we use the hard-constrained parameterization uθ=Nθwu_{\theta}=N_{\theta}w with w(x)=410∏i=110xi(1−xi)w(x)=4^{10}\prod_{i=1}^{10}x_{i}(1-x_{i}). The exact solution is chosen as u∗(x)=∏i=110sin⁡(πxi)u^{*}(x)=\prod_{i=1}^{10}\sin(\pi x_{i}).

The network is trained for 5,000 epochs. The numerical solution on the two-dimensional slice (x1,x2,0.5,…,0.5)(x_{1},x_{2},0.5,\ldots,0.5) is shown in Figure 4. On this slice, NeRI achieves a relative L2L^{2} error of 5.25×10−35.25\times10^{-3} and a maximum pointwise error of 6.63×10−36.63\times10^{-3}.

Numerical results for Example 6.4

Figure 4. Numerical results for Example 6.4.

A nonsymmetric variational inequality problem

Let Ω=(−1,1)2\Omega= (-1,1)^2. We consider the nonsymmetric variational inequality (32), where

⟨Au−f,v⟩V∗,V:=∫Ω(∇u⋅∇v+ux1v+uv−fv) dx−∫∂Ωgv ds,\langle Au-f,v\rangle_{V^*,V}:=\int_{\Omega}(\nabla u\cdot\nabla v+u_{x_1}v+uv-fv)\,\mathrm{d}x-\int_{\partial\Omega}gv\,\mathrm{d}s,

and the constraint set is K:={u∈H1(Ω):u≥1 a.e. in Ω}K:=\{u\in H^1(\Omega):u\ge1\ \text{a.e. in }\Omega\}. The exact solution is chosen as

u∗(x)={(1−8x12−8x22)2+1,x12+x22≤18,1,otherwise.u^*(x)=\begin{cases} (1-8x_1^2-8x_2^2)^2+1, & x_1^2+x_2^2\le\frac{1}{8},\\ 1, & \text{otherwise}. \end{cases}

The Neumann data gg are determined from u∗u^*, and the source term is given by

f(x)=−Δu∗+ux1∗+u∗−{1,x12+x22>34,0,otherwise.f(x)=-\Delta u^*+u^*_{x_1}+u^*-\begin{cases} 1, & x_1^2+x_2^2>\frac{3}{4},\\ 0, & \text{otherwise}. \end{cases}

The coincidence set is {u∗=1}={x12+x22≥1/8}\{u^*=1\}=\{x_1^2+x_2^2\ge1/8\}. In particular, the problem contains a biactive region where the obstacle is active while the multiplier Au∗−fAu^*-f vanishes. Moreover, the second derivatives of u∗u^* have jumps across the circle {x12+x22=1/8}\{x_1^2+x_2^2=1/8\}, resulting in limited global regularity.

We compare NeRI with Prox-PINN [13] and the weak adversarial approach [2]. For all methods, we use the hard-constrained parameterization uθ=Nθ2+1u_\theta=\mathcal{N}_\theta^2+1 to enforce the obstacle constraint. For this problem, the Prox-PINN loss is

L(θ)=1τ∥max⁡{uθ−τ(Auθ−f),1}−uθ∥L2(Ω)2+β∥∂uθ∂n−g∥L2(∂Ω)2.\mathcal{L}(\theta)=\frac{1}{\tau}\left\|\max\{u_\theta-\tau(Au_\theta-f),1\}-u_\theta\right\|_{L^2(\Omega)}^2+\beta\left\|\frac{\partial u_\theta}{\partial n}-g\right\|_{L^2(\partial\Omega)}^2.

Compared with the standard Prox-PINN formulation, we include the factor 1/τ1/\tau in the first term, which improved the numerical accuracy in our parameter tuning. We use β=102\beta=10^2 and τ=0.01\tau=0.01. For the weak adversarial method, we introduce two constraint-preserving networks uθu=Nθu2+1u_{\theta_u}=\mathcal{N}_{\theta_u}^2+1 and vθv=Nθv2+1v_{\theta_v}=\mathcal{N}_{\theta_v}^2+1, and use the objective

Lreg(θu,θv):=⟨Auθu−f,uθu−vθv⟩V∗,V−γreg∥uθu−vθv∥H1(Ω)2,γreg=5×10−4.\mathcal{L}_{\mathrm{reg}}(\theta_u,\theta_v):=\langle Au_{\theta_u}-f,u_{\theta_u}-v_{\theta_v}\rangle_{V^*,V}-\gamma_{\mathrm{reg}}\left\|u_{\theta_u}-v_{\theta_v}\right\|_{H^1(\Omega)}^2,\qquad\gamma_{\mathrm{reg}}=5\times10^{-4}.

The networks are trained for 20,000 epochs. Figure 5 shows the numerical approximations obtained by NeRI, Prox-PINN, and WAN together with the corresponding pointwise absolute errors. NeRI captures the localized nonsmooth behavior more accurately and yields the smallest error among the three methods. The relative L2L^2 errors for NeRI, Prox-PINN, and WAN are 4.18×10−44.18\times10^{-4}, 3.80×10−33.80\times10^{-3}, and 1.09×10−31.09\times10^{-3}, respectively.

Numerical results for Example 6.5

Figure 5. Numerical results for Example 6.5.

We note that, although not presented here, for smooth problems with high-regularity u∗u^* (e.g., examples in [2]) the Prox-PINN can work quite well and tends to achieve higher accuracy and faster convergence than NeRI and WAN with a suitable choice of τ\tau and β\beta.

Conclusion

In this work, we introduced Neural Residual Iteration (NeRI), a weak-form neural-network framework for a class of elliptic operator equations and variational inequalities, including nonlinear and nonsymmetric problems. At each iteration, NeRI constructs a state-dependent linear functional whose derivative recovers the current weak residual. This construction yields a residual-driven parameter update without requiring a global energy functional or an additional test network. We also extended NeRI to elliptic variational inequalities through penalized and constraint-preserving formulations and provided weak adversarial interpretations without explicit min–max training.

From a function-space perspective, NeRI induces, to first order, a neural-tangent-operator-preconditioned residual iteration. For strongly monotone Lipschitz operator equations, we established convergence of the fixed-kernel iteration to a reduced-space solution and developed a gradient-flow analysis in the Sobolev setting. For wide two-layer networks, we quantified the finite-time kernel drift in L(H1(Ω)∗,H1(Ω))\mathcal{L}(H^1(\Omega)^*, H^1(\Omega)) and derived a direct H1(Ω)H^1(\Omega)-error bound relative to the exact weak solution. This bound separates finite-width lazy-training, frozen-flow optimization, and reduced-space approximation errors, clarifying the role of the neural parameterization in both approximation and residual preconditioning.

Numerical experiments illustrate the effectiveness of NeRI for nonsymmetric and low-regularity problems, including high-dimensional equations and an obstacle-type variational inequality. The reported comparisons show advantages over the tested PINN-type and weak adversarial baselines, while retaining a single-network, single-step update structure.

Several directions remain for future work. One is to develop convergence theory for finite-width networks beyond the lazy-training regime and to extend the explicit Sobolev-space estimates to deep architectures. Another is to analyze the discrete-time neural iteration, accounting for the evolving tangent operator, finite step sizes, and quadrature or Monte Carlo approximation of the weak residual. The use of Adam in the numerical experiments also motivates extending the theory to different optimizers, including momentum and adaptive methods with history-dependent dynamics. Further questions concern convergence guarantees for constraint-preserving VI updates and extensions to more general monotone or nonsmooth operators, time-dependent problems, and coupled PDE systems.

Acknowledgement: This work is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – The Berlin Mathematics Research Center MATH+ (EXC-2046/1, project ID: 390685689). In particular, MH gratefully acknowledges support from the DFG Excellence Cluster MATH+ (EXC 2046) through the project AA5-2 “Robust Multilevel Training of Artificial Neural Networks.” This work is further supported by the DFG through the Research Grants Programme project “Nonsmooth machine-learning-based computational models and methods for ultrasound computed tomography (MLUSCT)” (Project number: 581139020).

References

  1. [1]R. A. Adams and J. J. F. Fournier. Sobolev Spaces, volume 140 of Pure and Applied Mathematics (Amsterdam). Elsevier/Academic Press, Amsterdam, second edition, 2003.
  2. [2]A. Alphonse, M. Hintermüller, A. Kister, C. H. Lun, and C. Sirotĕnko. A neural network approach to learning solutions of a class of elliptic variational inequalities. J. Mach. Learn. Res., 27(157):1–48, 2026.
  3. [3]H. Amann. Ordinary Differential Equations: An Introduction to Nonlinear Analysis, volume 13 of De Gruyter Studies in Mathematics. Walter de Gruyter, Berlin, 1990.
  4. [4]W. F. Ames. Numerical Methods for Partial Differential Equations. Academic Press, 2014.
  5. [5]A. G. Baydin, B. A. Pearl­mutter, A. A. Radul, and J. M. Siskind. Automatic differentiation in machine learning: A survey. J. Mach. Learn. Res., 18(153):1–43, 2018.
  6. [6]X. Cheng, X. Shen, X. Wang, and K. Liang. A deep neural network-based method for solving obstacle problems. Nonlinear Anal. Real World Appl., 72:103864, 2023.DOI
  7. [7]L. Chizat, E. Oyallon, and F. Bach. On lazy training in differentiable programming. In Adv. Neural Inf. Process. Syst., volume 32, 2019.arxiv.org/abs/1812.07956
  8. [8]S. Congreve and T. P. Wihler. Iterative Galerkin discretizations for strongly monotone problems. J. Comput. Appl. Math., 311:457–472, 2017.
  9. [9]M. Dauge. Elliptic Boundary Value Problems on Corner Domains: Smoothness and Asymptotics of Solutions. Springer, 2006.
  10. [10]T. De Ryck and S. Mishra. Numerical analysis of physics-informed neural networks and related models in physics-informed machine learning. Acta Numer., 33:633–713, 2024.DOI
  11. [11]W. E and B. Yu. The deep Ritz method: A deep learning-based numerical algorithm for solving variational problems. Commun. Math. Stat., 6(1):1–12, 2018.arxiv.org/abs/1710.00211
  12. [12]G. Gantner, A. Haberl, D. Praetorius, and B. Stiftner. Rate optimal adaptive FEM with inexact solver for nonlinear operators. IMA J. Numer. Anal., 38(4):1797–1831, 2018.DOI
  13. [13]Y. Gao, Y. Song, Z. Tan, H. Yue, and S. Zeng. Prox-PINNs: A deep learning algorithmic framework for elliptic variational inequalities. arXiv preprint arXiv:2505.14430, 2025.arxiv.org/abs/2505.14430
  14. [14]Z. Gao, L. Yan, and T. Zhou. Failure-informed adaptive sampling for PINNs. SIAM J. Sci. Comput., 45(4):A1971–A1994, 2023.arxiv.org/abs/2210.00279
  15. [15]R. Glowinski, J.-L. Lions, and R. Trémolières. Numerical Analysis of Variational Inequalities, volume 8. Elsevier, 2011.
  16. [16]M. Hintermüller and J. Ning. Constrained neural parameterization for optimization in function spaces. arXiv preprint arXiv:2606.00855, 2026.arxiv.org/abs/2606.00855
  17. [17]T. Hu, B. Jin, and F. Wang. An iterative deep Ritz method for monotone elliptic problems. J. Comput. Phys., 527:113791, 2025.arxiv.org/abs/2501.15186
  18. [18]T. Hu, B. Jin, and Z. Zhou. Solving elliptic problems with singular sources using singularity splitting deep Ritz method. SIAM J. Sci. Comput., 45(4):A2043–A2074, 2023.arxiv.org/abs/2209.02931
  19. [19]T. Hu, B. Jin, and Z. Zhou. Solving Poisson problems in polygonal domains with singularity enriched physics informed neural networks. SIAM J. Sci. Comput., 46(4):C369–C398, 2024.arxiv.org/abs/2308.16429
  20. [20]A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Adv. Neural Inf. Process. Syst., volume 31, 2018.arxiv.org/abs/1806.07572
  21. [21]Y. Jiao, R. Li, P. Wu, J. Z. Yang, and P. Zhang. DRM revisited: A complete error analysis. J. Mach. Learn. Res., 26(115):1–76, 2025.arxiv.org/abs/2407.09032
  22. [22]B. Jin and L. Wu. Convergence of stochastic gradient methods for wide two-layer physics-informed neural networks for the Poisson equation. Neural Networks, page 109335, 2026.DOI
  23. [23]B. Jin and L. Wu. The differential neural tangent kernel and its positivity. arXiv preprint arXiv:2607.10200, 2026.arxiv.org/abs/2607.10200
  24. [24]D. Kinderlehrer and G. Stampacchia. An Introduction to Variational Inequalities and Their Applications. SIAM, 2000.
  25. [25]Y. Lee, A. Kopaničáková, and G. E. Karniadakis. Two-level overlapping additive Schwarz preconditioner for training scientific machine learning applications. Comput. Methods Appl. Mech. Eng., 448:118400, 2026.DOI
  26. [26]K. Li and W. Ye. D-FNO: A decomposed Fourier neural operator for large-scale parametric partial differential equations. Comput. Methods Appl. Mech. Eng., 436:117732, 2025.DOI
  27. [27]Z. Li, N. B. Kovachki, K. Azizzadenesheli, K. Bhattacharya, A. Stuart, A. Anandkumar, et al. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021.
  28. [28]L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nat. Mach. Intell., 3(3):218–229, 2021.
  29. [29]L. Lu, X. Meng, Z. Mao, and G. E. Karniadakis. DeepXDE: A deep learning library for solving differential equations. SIAM Rev., 63(1):208–228, 2021.DOI
  30. [30]L. Lu, R. Pestourie, W. Yao, Z. Wang, F. Verdugo, and S. G. Johnson. Physics-informed neural networks with hard constraints for inverse design. SIAM J. Sci. Comput., 43(6):B1105–B1132, 2021.
  31. [31]Y. Lu, J. Blanchet, and L. Ying. Sobolev acceleration and statistical optimality for learning elliptic equations via gradient descent. In Adv. Neural Inf. Process. Syst., volume 35, pages 33233–33247, 2022.arxiv.org/abs/2205.07331
  32. [32]K. Luo, J. Zhao, Y. Wang, J. Li, J. Wen, J. Liang, H. Soekmadji, and S. Liao. Physics-informed neural networks for PDE problems: A comprehensive review. Artif. Intell. Rev., 58(10):323, 2025.DOI
  33. [33]D. Nganyu Tanyu, J. Ning, T. Freudenberg, N. Heilenkötter, A. Rademacher, U. Iben, and P. Maass. Deep learning methods for partial differential equations and related parameter identification problems. Inverse Problems, 39(10):103001, 2023.DOI
  34. [34]J. M. Ortega and W. C. Rheinboldt. Iterative Solution of Nonlinear Equations in Several Variables. SIAM, 2000.
  35. [35]Y. Peng, D. Hu, and Z.-Q. J. Xu. A non-gradient method for solving elliptic partial differential equations with deep neural networks. J. Comput. Phys., 472:111690, 2023.DOI
  36. [36]M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys., 378:686–707, 2019.DOI
  37. [37]A. Sojitra, M. Dhingra, and O. San. FEDONet: Fourier-embedded DeepONet for spectrally accurate operator learning. J. Comput. Phys., page 114931, 2026.DOI
  38. [38]J. D. Toscano, V. Oommen, A. J. Varghese, Z. Zou, N. Ahmadi Daryakenari, C. Wu, and G. E. Karniadakis. From PINNs to PIKANs: Recent advances in physics-informed machine learning. Mach. Learn. Comput. Sci. Eng., 1(1):15, 2025.DOI
  39. [39]C. Uriarte, D. Pardo, I. Muga, and J. Muñoz-Matute. A deep double Ritz method (D2RM) for solving partial differential equations using neural networks. Comput. Methods Appl. Mech. Eng., 405:115892, 2023.
  40. [40]A. W. van der Vaart. Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 1998.
  41. [41]S. Wang, X. Yu, and P. Perdikaris. When and why PINNs fail to train: A neural tangent kernel perspective. J. Comput. Phys., 449:110768, 2022.arxiv.org/abs/2007.14527
  42. [42]M. Weng, Z. Mao, and J. Shen. Deep collocation method: A framework for solving PDEs using neural networks with error control. SIAM J. Sci. Comput., 48(1):C77–C102, 2026.
  43. [43]Y. Zang, G. Bao, X. Ye, and H. Zhou. Weak adversarial networks for high-dimensional partial differential equations. J. Comput. Phys., 411:109409, 2020.DOI
  44. [44]E. Zeidler. Nonlinear Functional Analysis and Its Applications: II/B: Nonlinear Monotone Operators. Springer Science & Business Media, 2013.
  45. [45]X. E. Zhao, W. Hao, and B. Hu. Two neural-network-based methods for solving elliptic obstacle problems. Chaos Solitons Fractals, 161:112313, 2022.DOI

Paper details

vlnc:W_zvkkra9m
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents