betaFeedback

WorksComputer Science

Conditionally adaptive augmented Lagrangian method for physics-informed learning of forward and inverse problems

Abstract

We present several key advances to the Physics and Equality Constrained Artificial Neural Networks (PECANN) framework, substantially improving its capacity to solve challenging partial differential equations (PDEs). Our enhancements broaden the framework's applicability and improve efficiency. First, we generalize the Augmented Lagrangian Method (ALM) to support multiple, independent penalty parameters for enforcing heterogeneous constraints. Second, we introduce a constraint aggregation technique to address inefficiencies associated with point-wise enforcement. Third, we incorporate a single Fourier feature mapping to capture highly oscillatory solutions with multi-scale features, where alternative methods often require multiple mappings or costlier architectures. Fourth, a novel time-windowing strategy enables seamless long-time evolution without relying on discrete time models. Fifth, and critically, we propose a conditionally adaptive penalty update (CAPU) strategy for ALM that accelerates the growth of Lagrange multipliers for constraints with larger violations, while enabling coordinated updates of multiple penalty parameters. CAPU accelerates the growth of Lagrange multipliers for selectively challenging constraints, enhancing constraint enforcement during training. We demonstrate the effectiveness of PECANN-CAPU across diverse problems, including the transonic rarefaction problem, reversible scalar advection by a vortex, high-wavenumber Helmholtz and Poisson's equations, and inverse heat source identification. The framework achieves competitive accuracy across all cases when compared with established methods and recent approaches based on Kolmogorov-Arnold networks. Collectively, these advances improve the robustness, computational efficiency, and applicability of PECANN to demanding problems in scientific computing.

Introduction

The application of artificial neural networks (ANNs) to the solution of partial differential equations (PDEs) dates back to the early 1990s [2, 3], following the theoretical establishment of ANNs as universal function approximators [4]. With the advent of deep learning–driven by key developments such as back propagating errors in neural networks [5], deep belief networks [6], rectified linear units [7], efficient training of neural networks [8], dropout regularization [9], and stochastic optimization [10]–there has been a renewed interest in leveraging neural networks for solving PDEs [11, 12, 13, 14]. This resurgence was further catalyzed by the introduction of the physics-informed neural networks (PINNs) framework [15].

In the baseline PINN approach [2, 3, 15], governing PDEs are incorporated into training through a composite loss function derived from an unconstrained optimization perspective. This composite loss term combines contributions from residual forms of the PDE, boundary and initial conditions, and sample data, each weighted by tunable hyperparameters. Thus, in the baseline approach, solving a PDE is cast as an unconstrained, multi-objective optimization problem. While PINN performance depends on activation functions [16], sampling strategies [17], and network architectures [18, 1], the formulation and balance of loss terms are central to achieving accurate and stable predictions.

Because training a neural network relies on gradient-based optimizers, large gradients can dominate, leading to unstable behavior where some loss terms are prioritized over others. Balancing these the loss terms in a composite objective function requires manual or dynamic tuning of hyperparameters, which becomes impractical in the absence of validation data or exact solutions [19]. This challenge has driven extensive research into dynamic loss reweighting strategies [18, 20, 21, 22, 23], which are critical for improving PINN’s accuracy and stability. A comprehensive review of these methods and emerging PINN variants is provided by Toscano et al. [24].

Principally speaking, in optimization problems, it is crucial to distinguish which terms should serve as objectives and which should be enforced as constraints [25]. Departing from the multi-objective optimization paradigm in the baseline PINN approach, several works have adopted the constrained optimization perspective. In the self-adaptive PINN method [22], the objective function was formulated as a PDE-constrained optimization problem using a penalty-based method with trainable penalty coefficients. Other works have adopted the exact enforcement of boundary conditions as hard constraints through auxiliary functions [26, 27, 28]. Basir and Senocak [1] proposed the Physics and Equality Constrained Artificial Neural Networks (PECANN) framework to learn the solution of forward an inverse PDE problems.

In the PECANN framework, the augmented Lagrangian method (ALM) [29, 30] is employed to recast the constrained optimization problem into an unconstrained one in a principled manner. ALM provides a formal framework to enforce the constraints and should not be confused with optimizers such as stochastic gradient descent used to minimize an objective function in unconstrained optimization. Notably, ALM integrates the strengths of both the penalty method and the method of Lagrange multipliers from the constrained optimization field. ALM aims for a balance between feasibility and optimality by iteratively updating a penalty parameter to regulate the influence of constraint violations [31, 32, 25].

With a few modifications, the PECANN framework has been successfully applied to a wide range of forward and inverse PDE problems from different domains [33, 34, 35, 36, 37]. Hu et al. [35] demonstrated PECANN’s versatile formulation by inferring interface transmission conditions in the parallel solution of two-dimensional Laplace and Helmholtz equations using a non-overlapping Schwarz-type domain decomposition method. In their approach, both the global solution and the interface transmission conditions–characterized by unknown parameters–were learned simultaneously, while enforcing vanishing PDE residuals and associated boundary conditions as constraints. Olson et al. [37] developed one- and two-level domain decomposition methods to accelerate the PECANN method through parallel computing. Their work highlighted the critical role of differentiating boundary conditions from interface continuity requirements when formulating the loss functions. Lan et al. [34] applied the PECANN framework to develop surrogate models for groundwater flow dynamics by learning unsaturated infiltration solutions from limited initial and boundary data. They showed that PECANN outperformed both PINNs and purely data-driven approaches, and achieved accuracy comparable to finite-difference methods while requiring significantly less data. Most recently, Su et al. [36] applied the PECANN framework to learn unstable solutions of the so-called “bad” Jaulent–Miodek nonlinear equation, which models the motion of nonviscous shallow water wave packets in a flat-bottomed domain under shear forces. To accelerate the training process, Su et al. first pre-trained the network using the PINN methodology and then fine-tuned it with PECANN to achieve higher accuracy. They also highlighted the limitations of both the original PINN and PECANN approaches in solving the Jaulent–Miodek equation over larger computational domains, underscoring the need for the advancements to the PECANN framework introduced in the present work.

Fully-connected feed-forward neural networks, commonly referred to as multi-layer perceptrons (MLPs), are widely employed in machine learning applications. However, recent developments have introduced alternative architectures that aim to enhance expressivity and performance. Among these, Kolmogorov-Arnold Networks (KANs) [38] and Fourier feature networks [39] represent two notable examples. KANs are grounded in the Kolmogorov-Arnold representation theorem, wherein the activation functions of neurons are learned as one-dimensional splines. Despite their theoretical advantages, KANs exhibit significantly slower training times compared to conventional MLPs. In contrast, Fourier feature networks augment the representational capacity of MLPs for high-frequency functions by applying a Fourier feature mapping to the input space.

Wang et al. [40] introduced the PIKAN framework, which integrates KANs with PINNs. Their study substantiates the superior expressive capacity of KANs, while also underscoring their computational inefficiency and limitations in solving PDEs characterized by complex geometries or extensive computational grids. Furthermore, Wang et al. [41] demonstrated that employing multiple Fourier feature mappings–each initialized with distinct variances–in conjunction with a fully-connected MLP architecture significantly enhances the model’s ability to approximate solutions to benchmark PDEs exhibiting high-frequency behavior–scenarios in which conventional PINN models tend to underperform.

In the following, we present several key enhancements to the original PECANN framework [1]. These advancements collectively enable the new framework to learn solutions to challenging problems frequently employed in the development and benchmarking of numerical methods for PDEs.

Technical Formulation

Let us consider a generic equality-constrained optimization problem:

min⁡θJ(θ),subject toCi(θ)=0,∀i∈E,(1)\min_{\theta} \mathcal{J}(\theta), \qquad\text{subject to} \qquad\mathcal{C}_{i}(\theta)=0,\quad\forall i \in\mathcal{E}, \tag*{(1)}

where J(θ)\mathcal{J}(\theta) is the objective function, Ci(θ)\mathcal{C}_{i}(\theta) are constraint functions, and all functions are continuous and real-valued over the primal variable θ∈Rn\theta\in\mathbb{R}^{n}. The set E\mathcal{E} indexes a finite number of equality constraints. We can express all constraints collectively as a vector C(θ)\mathcal{C}(\theta) and reformulate the problem using the augmented Lagrangian method [29, 30]. This leads to the following unconstrained minimax formulation:

min⁡θmax⁡λL(θ;λ,μ)=J(θ)+λTC(θ)+12μ∥C(θ)∥22,(2)\min_{\theta}\max_{\lambda}\mathcal{L}(\theta;\lambda,\mu)=\mathcal{J}(\theta)+\lambda^{T}\mathcal{C}(\theta)+\frac{1}{2\mu}\lVert\mathcal{C}(\theta)\rVert_{2}^{2}, \tag*{(2)}

where L\mathcal{L} is the augmented unconstrained loss, λ\lambda denotes the vector of Lagrange multipliers (also known as dual variables) associated with the constraint vector C\mathcal{C}, μ\mu is a scalar penalty parameter. The relationship between the primal and dual formulations is captured by the minimax inequality (a form of weak duality), which ensures that:

max⁡λmin⁡θL(θ;λ)≤min⁡θmax⁡λL(θ;λ).(3)\max_{\lambda}\min_{\theta}\mathcal{L}(\theta;\lambda)\leq\min_{\theta}\max_{\lambda}\mathcal{L}(\theta;\lambda). \tag*{(3)}

The maximization over λ\lambda at optimization step ee can be performed iteratively using the following update rule, as proposed by Hestenes [29]:

λe←λe−1+μe−1C(θe),(4)\lambda^{e}\leftarrow\lambda^{e-1}+\mu^{e-1}\mathcal{C}(\theta^{e}), \tag*{(4)}

where μe−1\mu^{e-1} is the penalty parameter from the previous step and C(θe)\mathcal{C}(\theta^{e}) is the constraint violation at the current iterate θe\theta^{e}. We should note that μ\mu can be viewed as a global learning rate for the Lagrange multipliers. ALM combines the merits of the penalty method and the Lagrange multiplier method by updating the penalty parameter in such a way that it balances the trade-off between feasibility and optimality, and ensures convergence. We provide a derivation of ALM in Appendix § 13, emphasizing the role of Lagrange multipliers and penalty parameters in its formulation.

Algorithm 1 is the procedure that is commonly found in most textbooks presenting ALM. It is also the algorithm adopted in the original PECANN [1] method. In this procedure, the penalty parameter μ\mu is monotonically increased at each training iteration by an exponential factor β\beta, until it reaches a predefined maximum threshold μmax⁡\mu_{\max} for safeguarding. Bounding the penalty parameter with an upper limit is a common strategy utilized to prevent excessively large values that could otherwise induce numerical instability. We will refer to Algorithm 1 as ALM with monotonic exponential penalty update (MPU).

In the context of neural network training, Algorithm 1 takes as input the initial model parameters θ0\theta^{0}, a maximum penalty cap μmax⁡\mu_{\max}, a multiplicative growth factor β\beta for updating the penalty parameter μ\mu, and the total number of training epochs EE. We should note that in Algorithm 1, both the Lagrange multipliers and the penalty parameter are updated at each iteration [25]. To prevent divergence, the maximum penalty parameter was set to 10410^{4} in default. However, updating the penalty parameter every epoch may lead to overly aggressive growth, potentially disrupting the balance between the objective and constraint terms, as will be shown in a later section. Additionally, selecting an appropriate maximum penalty parameter is often challenging and problem-dependent.

Another strategy to update the penalty parameter μ\mu is to update it only when the constraints have not decreased sufficiently at the current iteration [42, 43]. Algorithm 2 presents an ALM with a conditional penalty parameter update strategy (CPU). A similar strategy has been adopted in the work of Dener et al. [44] to train an encoder-decoder neural network for approximating the Fokker-Planck-Landau collision operator.

Algorithm 1: ALM with monotonic exponential penalty update (MPU) [25]
1 Input: θ^0, β, μ_max, E
2 λ^0 = 1                                      /* Initializing Lagrange multipliers */
3 μ^0 = 1                                      /* Initializing the penalty parameter */
4 for e = 1 to E... do
5     θ^e ← Update(θ^{e−1}) via Eq. 2          /* primal update */
6     λ^e ← λ^{e−1} + μ^{e−1} C(θ^e)          /* dual update */
7     μ^e ← min(β μ^{e−1}, μ_max)               /* penalty parameter update */
8 end
9 Output: θ^E
10 Defaults: β = 2, μ_max = 1 × 10^4
Algorithm 2: ALM with conditional penalty update (CPU) [43]
1 Input: θ^0, β, μ_max, E
2 λ^0 = 1                                      /* Initializing Lagrange multipliers */
3 μ^0 = 1                                      /* Initializing the penalty parameter */
4 P^0 = ∞                                      /* Initializing l^2 norm of constraints */
5 for e = 1 to E... do
6     θ^e ← Update(θ^{e−1}) via Eq. 2          /* primal update */
7     if ‖C(θ^e)‖_2 < P^{e−1} then
8         λ^e ← λ^{e−1} + μ^{e−1} C(θ^e)      /* dual update */
9         μ^e = μ^{e−1}                         /* penalty parameter unchanged */
10    else
11        μ^e ← min(β μ^{e−1}, μ_max)           /* penalty parameter update */
12        λ^e = λ^{e−1}                          /* dual unchanged */
13    end
14    P^e = ‖C(θ^e)‖_2
15 end
16 Output: θ^E
17 Defaults: β = 2, μ_max = 1 × 10^4

Algorithm 2 shares the same inputs as the previous method. P\mathcal{P} is a placeholder for the previous l2l^2 norm of the constraints. As before, μmax⁡\mu_{\max} serves as a safeguard to cap the penalty parameter, and β\beta controls its multiplicative growth. The penalty is again limited to 10410^4 to avoid numerical instability, though determining an appropriate upper bound remains a nontrivial task.

Proposed Method: Conditionally Adaptive Augmented Lagrangian Method

In ALM, when multiple constraints are present, a unique Lagrange multiplier is assigned to each constraint. However, as demonstrated in the MPU and CPU algorithms, a single penalty parameter, μ\mu, which is exponentially increased during training, is commonly employed. Our experience indicates that this approach is insufficient for learning solutions to complex PDEs involving multiple constraints with diverse characteristics—such as those arising from flux conditions, observational data, or governing physical laws. Therefore, we advocate assigning a distinct penalty parameter to each Lagrange multiplier and propose an adaptive update strategy, rather than exponentially increasing μ\mu. This individualized approach allows each penalty parameter to evolve in accordance with the behavior of its associated constraint, thereby improving both training stability and constraint enforcement.

Formally, the augmented Lagrangian loss with constraint-specific penalties is

max⁡λmin⁡θL(θ,λ;μ)=J(θ)+λTC(θ)+12μT[C(θ)⊙C(θ)].(5)\max_{\boldsymbol{\lambda}}\min_{\theta}\mathcal{L}(\theta,\boldsymbol{\lambda};\boldsymbol{\mu}) = \mathcal{J}(\theta) + \boldsymbol{\lambda}^{T}\mathbf{C}(\theta) + \frac{1}{2}\boldsymbol{\mu}^{T}[\mathbf{C}(\theta)\odot\mathbf{C}(\theta)]. \tag*{(5)}

where ⊙\odot denotes the element-wise (Hadamard) product, λ\boldsymbol{\lambda} is a vector of Lagrange multipliers, and μ\boldsymbol{\mu} represents a vector of positive penalty parameters, with μi\mu_i associated to constraint CiC_i. The corresponding dual update at training epoch ee is

λe←λe−1+μe−1⊙C(θe).(6)\boldsymbol{\lambda}^{e}\leftarrow\boldsymbol{\lambda}^{e-1}+\boldsymbol{\mu}^{e-1}\odot\mathbf{C}(\theta^{e}). \tag*{(6)}

The key challenge is to update these penalty parameters independently and adaptively. Notably, since the gradient of the augmented loss (5) with respect to λ\boldsymbol{\lambda} is precisely the set of constraints C\mathbf{C}, the dual update (6) in ALM shares a formal resemblance to a gradient ascent step on a concave objective. In this sense, designing adaptive updates for distinct penalty parameters can be loosely regarded as analogous to developing adaptive learning rates in unconstrained optimization algorithms.

We draw inspiration from one of the adaptive algorithms for convex optimization, the RMSProp method proposed by Hinton [45]. RMSProp adjusts individual learning rates based on the moving averages of squared gradients, building an adaptive path of step sizes responsive to gradient history. If we apply this adaptive strategy for the dual update stage of ALM (6), the moving average of the squared ii-th constraint is expressed as

vˉi←ζvˉi+(1−ζ)Ci2(θ),(7)\bar{v}_{i}\leftarrow\zeta\bar{v}_{i}+(1-\zeta)C_{i}^{2}(\theta), \tag*{(7)}

where ζ\zeta is the smoothing coefficient (default ζ=0.99\zeta=0.99). The corresponding penalty parameter is then computed as

μi(RMSprop)←ηivˉi+ϵ,(8)\mu_{i}^{(\mathrm{RMSprop})}\leftarrow\frac{\eta_{i}}{\sqrt{\bar{v}_{i}+\epsilon}}, \tag*{(8)}

where ηi\eta_i is a specific scalar that serves as the penalty scaling factor, and ϵ\epsilon is a small constant (default ϵ=10−16\epsilon=10^{-16}) for numerical stability. However, this inverse dependence on the moving average of the ii-th constraint vˉi\bar{v}_i creates a serious conflict with the fundamental principle of ALM, which is enforcing stronger penalties when constraint violations are large. Instead, with (7) and (8), when a constraint value gradually increases for a short period during training (so that vˉi\bar{v}_i rise), μi(RMSprop)\mu_i^{(\mathrm{RMSprop})} inadvertently decreases, which runs counter to the aforementioned principle. Equally important, the resulting update of the Lagrange multiplier is

λi←λi+μi(RMSprop)Ci(θ)=λi+ηiCi(θ)vˉi+ϵ,(9)\lambda_i\leftarrow\lambda_i+\mu_i^{(\mathrm{RMSprop})}C_i(\theta)=\lambda_i+\eta_i\frac{C_i(\theta)}{\sqrt{\bar{v}_i+\epsilon}}, \tag*{(9)}

which notably provides no direct control over the penalty parameter, μ\mu. In this rule (Eq. (9)), the growth of each λi\lambda_i is governed solely by the constant ηi\eta_i and the ratio between the current constraint Ci(θ)\mathcal{C}_i(\theta) and its RMS-averaged history vˉi\sqrt{\bar{v}_i}. Since this ratio typically remains close to unity, λi\lambda_i keeps nearly the same rate of growth, essentially decided by ηi\eta_i, even when constraint violations increase. To illustrate these pathologies in practice, we present a specific failure case in Appendix § 11. Figure A.21(a) shows the resulting Lagrange multipliers and penalty parameter, obtained by applying the rule in Eq. (9), plotted as a function of epochs.

To address the inconsistency arising from directly adopting RMSProp’s adaptive strategy, we introduce a safeguard by applying a maximum operation that retains the larger of the previous penalty parameter and its updated value from RMSProp.

μi←max⁡(μi,ηivˉi+ϵ).(10)\mu_i \leftarrow\max\left(\mu_i, \frac{\eta_i}{\sqrt{\bar{v}_i}+\epsilon}\right). \tag*{(10)}

Despite its simplicity, this modification drastically changed the evolution of both Lagrange multipliers and penalty parameters during training, proving highly effective, as demonstrated in Figure A.21(b). By safeguarding the penalty parameter, unintended reductions during training are prevented, while the corresponding Lagrange multiplier can be rapidly amplified via Eq. (6) in response to increased constraint violations. As a result, the optimization process is guided back toward feasibility precisely at the stages when constraint satisfaction is most critical. At the same time, the RMSProp contribution in {(10)} allows μi\mu_i to increase gradually when constraint violations diminish smoothly during optimization, thereby strengthening enforcement progressively as constraint satisfaction is approached.

In the literature on adaptive learning-rate optimizers, AMSGrad [46] employs a related mechanism to address the convergence deficiencies of Adam [10] by replacing the exponentially weighted second-moment estimates with a non-decreasing sequence defined via a maximum operator. However, AMSGrad is not suitable for our ALM context. Preventing a drift toward infeasibility here would require an AMSGrad-like method to employ a minimum operator on the historical squared constraint values and set the first-moment estimate equal to the current constraint, a configuration that maintains consistency with the dual update rule in Eq. (6).

Another important consideration is whether it’s necessary to update the dual variables λ\boldsymbol{\lambda} at every epoch. Consistent with several studies [31, 47, 48], our viewpoint is that such frequent updates are unnecessary until the current unconstrained optimization subproblem for minimizing with respect to θ\theta (see Eq. (5)) has converged sufficiently, given the previous values λe−1\boldsymbol{\lambda}^{e-1} and μe−1\boldsymbol{\mu}^{e-1}. We define the rate of decrease of the augmented Lagrangian loss between successive epochs as

LeLe−1=ω,(11)\frac{\mathcal{L}^{e}}{\mathcal{L}^{e-1}} = \omega, \tag*{(11)}

such that we simply declare convergence when ωt<ω≤1.0\omega_t < \omega\le1.0, where Le\mathcal{L}^{e} and Le−1\mathcal{L}^{e-1} denote the current and previous augmented Lagrangian loss, respectively, and ωt∈(0,1)\omega_t \in(0,1) is a user-defined threshold. The default value here is ωt=0.999\omega_t = 0.999. Meanwhile, ω>1\omega> 1 indicates that the gradient-based optimizer may perform stochastic updates leading to greater constraint violations. Once either condition is satisfied, i.e., ω≥ωt\omega\ge\omega_t, the Lagrange multipliers and penalty parameters are updated accordingly, and the training proceeds to the next unconstrained optimization subproblem with updated λe\boldsymbol{\lambda}^{e} and μe\boldsymbol{\mu}^{e}.

Algorithm 3: ALM with conditionally adaptive penalty updates (CAPU)

  1. Defaults: ζ=0.99\zeta= 0.99, ωt=0.999\omega_t = 0.999, ϵ=10−16\epsilon= 10^{-16}

  2. Input: θ0\theta^0, EE, η\boldsymbol{\eta}

  3. λ0←1\boldsymbol{\lambda}^0 \gets1 /* Initializing Lagrange multipliers */

  4. μ0←1\boldsymbol{\mu}^0 \gets1 /* Initializing penalty parameters */

  5. vˉ0←0\bar{\mathbf{v}}^0 \gets0 /* Initializing averaged square-gradients */

  6. L0←∞\mathcal{L}^0 \gets\infty /* Initializing augmented Lagrangian loss */

  7. for e=1e = 1 to EE... do

  8. θe←Update⁡(θe−1)\theta^e \gets\operatorname{Update}(\theta^{e-1}) /* primal update */

  9. Le←\mathcal{L}^e \gets (5) with θe\theta^e, λe−1\boldsymbol{\lambda}^{e-1}, μe−1\boldsymbol{\mu}^{e-1}, …\ldots /* augmented loss update */

  10. vˉe←ζvˉe−1+(1−ζ)(C(θe)⊙C(θe))\bar{\mathbf{v}}^e \gets\zeta\bar{\mathbf{v}}^{e-1} + (1-\zeta)(\mathbf{C}(\theta^e) \odot\mathbf{C}(\theta^e)) /* square-gradient update */

  11. if Le/Le−1≥ωt\mathcal{L}^e/\mathcal{L}^{e-1} \geq\omega_t then

  12. λe←λe−1+μe−1⊙C(θe)\boldsymbol{\lambda}^e \gets\boldsymbol{\lambda}^{e-1} + \boldsymbol{\mu}^{e-1} \odot\mathbf{C}(\theta^e) /* dual update */

  13. μe←max⁡(μe−1,ηvˉe+ϵ)\boldsymbol{\mu}^e \gets\max\left(\boldsymbol{\mu}^{e-1}, \frac{\boldsymbol{\eta}}{\sqrt{\bar{\mathbf{v}}^e}+\epsilon}\right) /* penalty update */

  14. end

  15. Output: θE\theta^E

The complete training procedure is presented in Algorithm 3. Here, the smoothing constant ζ\zeta and the stability parameter ϵ\epsilon are set to their default values from RMSProp. The convergence threshold is specified as ωt=0.999\omega_t = 0.999. The vector of penalty scaling factors η={ηi:i∈E}\boldsymbol{\eta} = \{\eta_i : i \in\mathcal{E}\} is provided as an input to the algorithm, along with the initialized neural network parameters θ0\theta^0 and the total number of training epochs EE. The Lagrange multiplier vector λ\boldsymbol{\lambda} and the penalty parameter vector μ\boldsymbol{\mu} are both initialized to 1.0, and the moving averages of squared gradients, vˉ\bar{\mathbf{v}}, are initialized to zero. The initial augmented loss L0\mathcal{L}^0 is set to infinity to serve as a placeholder for the prior epoch’s loss.

In Appendix § 13, we follow existing proofs of optimality and feasibility of the well-established augmented Lagrangian method, and present a proof demonstrating the boundedness of the penalty parameters and the convergence property of the Lagrange multipliers associated with the CAPU algorithm.

The vector η\boldsymbol{\eta} plays a key role in controlling the growth rate of the Lagrange multipliers λ\boldsymbol{\lambda}, as previously discussed. The optimization framework in our setting, as defined in (5), includes both primal (θ\theta) and dual (λ\boldsymbol{\lambda}) updates. If the dual update overwhelms the other, rapid growth of λ\boldsymbol{\lambda} can hinder convergence by dominating the objective, similar to issues encountered in MPU and CPU strategies. Conversely, if λ\boldsymbol{\lambda} grows too slowly, the multipliers may take longer to reach effective magnitudes, resulting in slow progress toward feasibility. In RMSProp for gradient descent, a typical recommended value for the learning rate is 0.01. Since RMSProp is a first-order optimizer, we recommend using ηi=0.01\eta_i = 0.01 as well, when the primal update employs a first-order method like Adam [10]. In contrast, for quasi-Newton optimizers such as L-BFGS [49], which converge faster, a larger value such as ηi=1\eta_i = 1 is advisable to balance constraint enforcement and objective minimization effectively. A sensitivity study of external input ηi\eta_i and internal hyperparameters ζ\zeta, ωt\omega_t is presented and compared against other popular physics-informed approaches in Appendix § 12.

Constrained optimization formulation for solving forward and inverse problems

We begin by formulating the general constrained optimization problem for learning a generic PDE solution subject to boundary and initial conditions. This formulation is designed to accommodate the proposed CAPU algorithm and incorporate a constraint aggregation technique.

Consider a generic PDE problem for a scalar function u(x,t):Rd+1→Ru(\boldsymbol{x},t):\mathbb{R}^{d+1}\to\mathbb{R} on the domain Ω⊂Rd\Omega\subset\mathbb{R}^{d} with its boundary ∂Ω\partial\Omega:

F(x,t;∂u∂t,∂2u∂t2,⋯ ,∂u∂x,∂2u∂x2,⋯ ,ν)=0,∀(x,t)∈U,B(x,t,g;u,∂u∂x,⋯ )=0,∀(x,t)∈∂U,I(x,t,h;u,∂u∂t,⋯ )=0,∀(x,t)∈Γ,(12)\begin{aligned} \mathcal{F}\left(\boldsymbol{x},t;\frac{\partial u}{\partial t},\frac{\partial^{2}u}{\partial t^{2}},\cdots,\frac{\partial u}{\partial x},\frac{\partial^{2}u}{\partial x^{2}},\cdots,\boldsymbol{\nu}\right)&=0,\quad\forall(\boldsymbol{x},t)\in\mathcal{U},\\ \mathcal{B}\left(\boldsymbol{x},t,g;u,\frac{\partial u}{\partial x},\cdots\right)&=0,\quad\forall(\boldsymbol{x},t)\in\partial\mathcal{U},\\ \mathcal{I}\left(\boldsymbol{x},t,h;u,\frac{\partial u}{\partial t},\cdots\right)&=0,\quad\forall(\boldsymbol{x},t)\in\Gamma, \tag*{(12)} \end{aligned}

where F\mathcal{F} is the PDE residual involving differential operators and parameter vector ν\boldsymbol{\nu}; B\mathcal{B} and I\mathcal{I} are the residuals of the boundary condition (BC) and initial condition (IC), incorporating source functions g(x,t)g(\boldsymbol{x},t) and h(x,t)h(\boldsymbol{x},t), respectively. The domain is defined as U={(x,t)∣x∈Ω,t=[0,T]}\mathcal{U}=\{(\boldsymbol{x},t)\mid\boldsymbol{x}\in\Omega,t=[0,T]\}, with boundary ∂U={(x,t)∣x∈∂Ω,t=[0,T]}\partial\mathcal{U}=\{(\boldsymbol{x},t)\mid\boldsymbol{x}\in\partial\Omega,t=[0,T]\}, and initial surface Γ={(x,t)∣x∈∂Ω,t=0}\Gamma=\{(\boldsymbol{x},t)\mid\boldsymbol{x}\in\partial\Omega,t=0\}.

Original PECANN formulation with point-wise constraints

Considering the generic PDE problem (12), Basir and Senocak [1] formulated the PECANN method as a constrained optimization problem where boundary and initial conditions at each collocation point were treated as a constraint:

min⁡θ∑i=1NF∥F(x(i),t(i);ν,θ)∥22,subject toϕ(B(x(i),t(i),g(i);θ))=0,∀(x(i),t(i),g(i))∈∂U, i=1,⋯ ,NB,ϕ(I(x(i),t(i),h(i);θ))=0,∀(x(i),t(i),h(i))∈Γ, i=1,⋯ ,NI,(13)\begin{aligned} \min_{\boldsymbol{\theta}}\quad&\sum_{i=1}^{N_F}\left\|\mathcal{F}(\boldsymbol{x}^{(i)},t^{(i)};\boldsymbol{\nu},\boldsymbol{\theta})\right\|_{2}^{2},\\ \text{subject to}\quad&\phi\left(\mathcal{B}(\boldsymbol{x}^{(i)},t^{(i)},g^{(i)};\boldsymbol{\theta})\right)=0,\quad\forall(\boldsymbol{x}^{(i)},t^{(i)},g^{(i)})\in\partial\mathcal{U},\ i=1,\cdots,N_B,\\ &\phi\left(\mathcal{I}(\boldsymbol{x}^{(i)},t^{(i)},h^{(i)};\boldsymbol{\theta})\right)=0,\quad\forall(\boldsymbol{x}^{(i)},t^{(i)},h^{(i)})\in\Gamma,\ i=1,\cdots,N_I, \tag*{(13)} \end{aligned}

where NFN_F, NBN_B, NIN_I are the number of collocation points in U\mathcal{U}, ∂U\partial\mathcal{U} and Γ\Gamma respectively. ϕ\phi is a convex distance function, which we discuss in more detail in section 3.1.4. We should note that number of Lagrange multipliers scale with the number of constraints NBN_B, NIN_I. For PDE problems, the number of collocation points typically grows with the complexity of the solution, leading to thousands of constraints that must be satisfied (NBN_B and NIN_I are large). Consequently, the efficiency of the optimization process degrades. Moreover, point-wise constraint enforcement hinders the adoption of standard deep learning techniques such as mini-batch training. To address these limitations, we propose a new formulation based on constraint aggregation.

Revised PECANN formulation with constraint aggregation

In large-scale design or topology optimization, failure criteria are often expressed as maximum or minimum functions, which are inherently non-differentiable. To address this, constraint aggregation techniques have been developed to approximate these criteria with continuously differentiable functions, enabling sensitivity analysis via the adjoint method [50, 51, 52].

As illustrated in the example problem in § 4.1, when the number of point-wise constraints becomes large, Lagrange multipliers associated with a specific type of constraint tend to follow a probabilistic distribution with a well-defined expected value. Motivated by this observation, we aggregate point-wise constraints on each boundary using the mean squared residuals (MSR) metric, ensuring smoothness and differentiability. This aggregation reduces the number of Lagrange multipliers from thousands to just a few, making them easier to manage and enabling mini-batch training when full-batch training becomes impractical due to memory limitations.

Considering the generic PDE problem defined in Eq. (12), we now express our new formulation with constraint aggregation as follows:

min⁡θ J(θ)=1NF∑i=1NF∥F(x(i),t(i);ν,θ)∥22,subject to  CB=1NB∑i=1NBϕ(B(x(i),t(i),g(i);θ)):=0,CI=1NI∑i=1NIϕ(I(x(i),t(i),h(i);θ)):=0.(14)\begin{aligned} \min_{\theta}\ \mathcal{J}(\theta) &= \frac{1}{N_F}\sum_{i=1}^{N_F}\left\|\mathcal{F}(x^{(i)},t^{(i)};\nu,\theta)\right\|_2^2,\\ \text{subject to }\ C_B &= \frac{1}{N_B}\sum_{i=1}^{N_B}\phi\left(\mathcal{B}(x^{(i)},t^{(i)},g^{(i)};\theta)\right):=0,\\ C_I &= \frac{1}{N_I}\sum_{i=1}^{N_I}\phi\left(\mathcal{I}(x^{(i)},t^{(i)},h^{(i)};\theta)\right):=0. \tag*{(14)} \end{aligned}

where ϕ\phi is a quadratic function, but other options are possible [51].

In Appendix § 13.§ 1, we present a derivation establishing a connection between the point-wise constraint formulation and the aggregated constraint formulation. Interestingly, an extra penalty term dependent on the variance of the constraints appears with point-wise enforcement, a term that can be large for complex problems with challenging constraints. The benefits of using constraint aggregation on accuracy levels are demonstrated in § 4.1.

Extension to PDE-constrained inverse problems

In the original PECANN approach for inverse problems [1], the loss function was minimized in a composite manner, incorporating both the governing PDE and noisy observational data into a single objective, while high-fidelity data were enforced as constraints. However, discrepancies in physical scales between PDE terms and noisy measurements can hinder the inference process. To address this challenge in inverse problems with multi-fidelity data, we propose a reformulation that prioritizes minimizing the loss associated with noisy observations, while treating the governing PDE and any available high-fidelity data as constraints. Notably, this reformulation is made possible by the design of our CAPU algorithm, which assigns independent penalty parameters to each constraint and updates them adaptively. In the absence of noisy data, the PDE residual naturally serves as the objective, with high-fidelity data enforced as constraints.

To formulate this revision, consider a generic inverse PDE problem with unknown parameters ν\nu, and a set of noisy measurements {(x(i),t(i)),u^(i)}i=1NM\{(x^{(i)},t^{(i)}),\hat{u}^{(i)}\}_{i=1}^{N_M}. The problem can be formulated as follows:

min⁡θ,νJ(θ)=1NM∑i=1NM∥u(x(i),t(i);θ)−u^(x(i),t(i))∥22,\min_{\theta,\nu} J(\theta) = \frac{1}{N_M}\sum_{i=1}^{N_M}\left\lVert u(x^{(i)},t^{(i)};\theta)-\hat{u}(x^{(i)},t^{(i)})\right\rVert_2^2,

subject to CF=1NF∑i=1NFϕ(F(x(i),t(i);ν,θ)):=0,C_F=\frac{1}{N_F}\sum_{i=1}^{N_F}\phi(\mathcal{F}(x^{(i)},t^{(i)};\nu,\theta)):=0,

CB:=0;CI:=0;⋯C_B:=0;\quad C_I:=0;\quad\cdots {#eq:15}

If high-fidelity data is available, we can incorporate it as an additional constraint into the above formulation along with other constraints. In the context of CAPU, the choice of penalty scaling factors η\eta should reflect the relative priority of constraints. Specifically, the scaling factor for the PDE constraint ηF\eta_F is recommended to be smaller than those for high-fidelity data sources (ηB&ηI)(\eta_B \& \eta_I), such as boundary and initial conditions. This prioritization encourages the optimization process to satisfy boundary and initial conditions before focusing on reducing the PDE residual.

Role of convex distance functions in constraint enforcement and aggregation

In the original PECANN formulation with point-wise constraints, a quadratic distance function ϕ\phi is adopted [1]. In the present revised formulation (Eq. (14)), we adopt a constraint aggregation technique based on the mean squared residual (MSR) metric, which inherits the quadratic distance function ϕ\phi implicitly. Consequently, the associated penalty terms in ALM are proportional to the fourth power of the operator residuals in both formulations. Note that the augmented Lagrangian loss has separate contributions from the penalty term and the Lagrange multiplier (see Eq. (2)). For the ith boundary constraint,

ϕ=∣⋅∣2⇒CB(i)∝∣B(i)∣2,12μBCB(i)2∝∣B(i)∣4.(15)\phi=|\cdot|^2\Rightarrow C_B^{(i)}\propto|\mathcal{B}^{(i)}|^2,\qquad\frac{1}{2}\mu_B C_B^{(i)2}\propto|\mathcal{B}^{(i)}|^4. \tag*{(15)}

When ϕ\phi is an absolute-value function, the aggregated constraint reduces to the mean absolute error (MAE), leading to penalty terms proportional to the square of the residuals of the boundary condition operator

ϕ=∣⋅∣⇒CB(i)∝∣B(i)∣,12μBCB(i)2∝∣B(i)∣2.(16)\phi=|\cdot|\Rightarrow C_B^{(i)}\propto|\mathcal{B}^{(i)}|,\qquad\frac{1}{2}\mu_B C_B^{(i)2}\propto|\mathcal{B}^{(i)}|^2. \tag*{(16)}

Basir and Senocak [1] tested both options and found the quadratic distance function to be superior due to its smoothness and differentiability.

The choice of the distance function ϕ\phi provides a framework for interpreting alternative strategies within PINNs from the perspective of constrained optimization. A notable example is the self-adaptive PINN (SA-PINN) work [22]. Upon close examination of the SA-PINN formulation, we can see that the self-adaptive behavior is realized through a minimax formulation in the inverse form (i.e., weak duality):

max⁡ωmin⁡θJ(θ)+m(ω)T[C(θ)⊙C(θ)],(17)\max_{\omega}\min_{\theta}\mathcal{J}(\theta)+m(\omega)^T[\mathcal{C}(\theta)\odot\mathcal{C}(\theta)], \tag*{(17)}

where the objective J(θ)\mathcal{J}(\theta) is calculated of the MSR of the boundary conditions, and the constraint vector includes point-wise residuals of the PDE in the entire domain (∣F(x(i),t(i))∣)(|\mathcal{F}(x^{(i)},t^{(i)})|) and initial conditions (∣I(x(i),0)∣|\mathcal{I}(x^{(i)},0)|). Eq. (17) closely resembles the penalty terms given in Eq. (3.1.4), specifically those that arise when ϕ\phi is defined as an absolute-value function with point-wise enforcement of constraints.

A key element of the SA-PINN method is the mask function

m(ω)=cωq,(18)m(\boldsymbol{\omega})=c\boldsymbol{\omega}^{q}, \tag*{(18)}

where ω\boldsymbol{\omega} represents the self-adaptive weights and a polynomial function is assumed for the mask function. Note that in the SA-PINN paper [22], self-adaptive weights are denoted by the symbol λ\lambda. Here, we use ω\omega to distinguish it from the Lagrange multipliers, λ\lambda, used in ALM.

Comparing Eq. (17) with the augmented Lagrangian in Eq. (5), we observe that Eq. (17) lacks the constraint term with the Lagrange multipliers (λ\lambda), which are key elements of ALM for enforcing feasibility through optimality conditions and ensuring boundedness alongside the penalty parameter, μ\mu [31]. This distinction is subtle, yet it is critical for differentiating the PECANN methodology from the SA-PINN method.

Thus, the SA-PINN approach can be viewed as an adaptive penalty method distinct from the augmented Lagrangian method in PECANN. In conventional penalty methods, achieving exact feasibility theoretically requires the penalty parameter to approach infinity, driving the solution to satisfy all constraints. But, in practice, such a large penalty parameter increase the condition number of the Hessian matrix, leading to ill-conditioning in the optimizer. The examples presented in [22] show that the self-adaptive weights ω\omega typically remain at a modest level below 20. It would therefore be a valuable exercise to systematically study SA-PINN performance when the mask functions are amplified toward much larger penalization values.

Integration of Fourier feature mappings into the PECANN framework

A Fourier feature mapping, γ\gamma, projects input coordinates x∈Rd\mathbf{x}\in\mathbb{R}^{d} into a higher-dimensional space via sinusoidal transformations, thereby enhancing the expressive capacity of MLPs [39, 41]. This technique is theoretically grounded in Bochner’s theorem and can be interpreted as a Fourier approximation of a stationary kernel function [39], enabling the network to more effectively capture high-frequency components in the input space.

A single Fourier feature mapping is defined as:

γ(x)=[cos⁡(2πBx)sin⁡(2πBx)](19)\gamma(\mathbf{x})= \begin{bmatrix} \cos(2\pi\mathbf{B}\mathbf{x})\\ \sin(2\pi\mathbf{B}\mathbf{x}) \end{bmatrix} \tag*{(19)}

where B∈Rm×d\mathbf{B}\in\mathbb{R}^{m\times d} consists of entries sampled from a Gaussian distribution N(0,σ)\mathcal{N}(0,\sigma), which remain fixed during training. The resulting transformed input is then fed into a standard fully connected neural network, with the following layer-wise operations:

H1=γ(x),(20)\mathbf{H}_{1}=\gamma(\mathbf{x}), \tag*{(20)}
Hℓ=ψ(Wℓ⋅Hℓ−1+bℓ),ℓ=2,…,H,(21)\mathbf{H}_{\ell}=\psi(\mathbf{W}_{\ell}\cdot\mathbf{H}_{\ell-1}+\mathbf{b}_{\ell}),\quad\ell=2,\ldots,H, \tag*{(21)}
fθ(x)=WL+1⋅HL+bL+1,(22)f_{\theta}(\mathbf{x})=\mathbf{W}_{L+1}\cdot\mathbf{H}_{L}+\mathbf{b}_{L+1}, \tag*{(22)}

where ψ\psi represents the activation function.

According to Wang et al. [41], both the number of Fourier feature mappings and the choice of the scaling parameter σ\sigma are problem-dependent. Typical values such as $\sigma=1,20,50,100 enable the network to capture different frequency scales in the input function. Our experience has been different. In § 4.3, we test σ∈[0.25,16]\sigma\in[0.25, 16] and demonstrate through numerical experiments that a single layer of Fourier feature mappings with the default setting σ=1\sigma= 1 is sufficient when combined with ALM and the CAPU strategy under the constrained optimization formulation of PECANN. Fourier feature mapping is only adopted for multi-scale problems with highly oscillatory solutions presented in § 4.3 and § 4.4.1.

Application to forward and inverse PDE problems

We introduce several key enhancements to the original PECANN method [1] that improve its accuracy, stability, and generalization across diverse PDE problems. We construct a suite of benchmark problems, each specifically designed to isolate and evaluate a different aspect of the new framework. Before presenting detailed results, we summarize the key outcomes of each example problem.

  • § 4.1 (Unsteady heat transfer in a composite medium): This section analyzes the original PECANN formulation with point-wise constraint enforcement and demonstrates that the new formulation with constraint aggregation effectively satisfies boundary and flux continuity. Given the superior outcome of this example, the constraint aggregation formulation is adopted in all remaining examples.

  • § 4.2 (Transonic rarefaction in the inviscid Burgers’ equation): This example provides a quantitative comparison of penalty update strategies within the ALM, demonstrating that PECANN-CAPU effectively controls network expressiveness in discontinuous and sub-differentiable regions without modifying the governing physics or the underlying method.

  • § 4.3 (1D Poisson’s equation with multi-scale solutions): This example demonstrates that standard MLPs, when paired with either sufficiently dense collocation points or a single Fourier feature mapping (σ=1\sigma= 1), can effectively learn complex solution structures within the PECANN-CAPU framework. The results challenge the prevailing assumption that a neural network architecture incapable of solving a regression task will necessarily fail when applied to PDE learning, thereby underscoring the critical role of the loss function formulation and the optimization strategy employed.

  • § 4.4 (2D Helmholtz equation with high-frequency and multi-scale solutions): This example demonstrates that algorithmic innovation can rival architectural complexity. We compare our proposed PECANN-CAPU framework (using MLPs) against the physics-informed Chebyshev Kolmogorov-Arnold Networks (cPIKAN) [53], highlighting PECANN-CAPU’s ability to achieve competitive accuracy. Furthermore, we show that integrating MLPs with a single Fourier feature mapping yields additional gains in solution fidelity.

  • § 4.5 (Reversible advection of a passive scalar by a single vortex): This example highlights the PECANN-CAPU framework’s ability to perform stable, long-time integration of PDEs. By employing a non-overlapping time-windowing strategy, PECANN-CAPU achieves accuracy that is competitive with or superior to specialized computational methods designed for two-phase flow simulations.

  • § 4.6 (Inference of space-wise heat source in a transient heat conduction problem): This example evaluates the PECANN-CAPU framework with constraint aggregation for inverse PDE problems, demonstrating smooth and accurate reconstructions even when utilizing sparse and noisy data.

  • §Appendix A compares two constrained optimization methods, PECANN-CAPU and PECANN with direct application of RMSProp, under the constraint-aggregation formulation (14). The results demonstrate that PECANN-CAPU consistently drives the optimization toward feasibility with the fastest convergence rate.

  • §Appendix B examines the sensitivity of the PECANN-CAPU results to its internal hyperparameters and one external input. The accuracy of these results is assessed relative to other published works.

  • §Appendix C presents the theoretical underpinnings of the ALM, the mathematical relationship between the point-wise and aggregation formulations for constraints, and an analysis of the current CAPU algorithm regarding the boundedness of its penalty parameters and the convergence behavior of its Lagrange multipliers.

We define two metrics to assess the accuracy of our predictions. Given an nn-dimensional vector of predictions u^∈Rn\hat{u} \in\mathbb{R}^{n} and an nn-dimensional vector of exact values u∈Rnu \in\mathbb{R}^{n}, the relative Euclidean or l2l^{2} norm and infinity norm l∞l^{\infty} norm respectively

Er(u^,u)=∥u^−u∥2∥u∥2,E∞(u^,u)=∥u^−u∥∞,(23)\mathcal{E}_{r}(\hat{u},u)=\frac{\lVert\hat{u}-u\rVert_{2}}{\lVert u\rVert_{2}},\qquad\mathcal{E}_{\infty}(\hat{u},u)=\lVert\hat{u}-u\rVert_{\infty}, \tag*{(23)}

where ∥⋅∥2\lVert\cdot\rVert_{2} denotes the Euclidean norm, and ∥⋅∥∞\lVert\cdot\rVert_{\infty} denotes the maximum norm.

Unless otherwise stated, we use standard MLPs parameterized by θ\theta, with hyperbolic tangent activation functions and Xavier initialization scheme [54]. When selected, L-BFGS optimizer available in the Pytorch package [55] is employed with strong Wolfe line search function. The penalty scaling factor η\boldsymbol{\eta} in CAPU strategy is set to 0.01 when using Adam for the primal update, and 1.0 when using L-BFGS. For evaluations involving statistical performance such as the relative l2l^{2} norm, the mean prediction and its standard deviation are calculated based on five independent trials.

Forward problem: unsteady heat transfer in a composite medium

We consider the time-dependent heat transfer problem in a composite material where temperature and heat fluxes are matched across the interface [56]. This canonical problem has initial condition, boundary condition and flux constraints, which makes it a suitable example to demonstrate the effectiveness of PECANN-CAPU with constraint aggregation and contrast it against the original PECANN with point-wise enforcement of constraints.

We formulate the time-dependent heat conduction in a composite medium problem as follows:

∂u(x,t)∂t=∂∂x[κ(x,t)∂u(x,t)∂x]+s(x,t),(x,t)∈Ω×[0,τ],(24)\frac{\partial u(x,t)}{\partial t}=\frac{\partial}{\partial x}\left[\kappa(x,t)\frac{\partial u(x,t)}{\partial x}\right]+s(x,t),\qquad(x,t)\in\Omega\times[0,\tau], \tag*{(24)}

where uu is the temperature, ss is heat source, κ\kappa is the thermal conductivity. The domain Ω=Ω1∪Ω2\Omega=\Omega_{1}\cup\Omega_{2} consists of two non-overlapping subdomains: Ω1={x∣−1≤x<0}\Omega_{1}=\{x\mid-1\leq x<0\} and Ω2={x∣0<x≤1}\Omega_2 = \{x|0 < x \le1\}. To assess the accuracy of the learned solutions, we prescribe the thermal conductivity and exact solution as:

[κ,u]T={[1,tsin⁡(3πx)]T,(x,t)∈Ω1×[0,2],[3π,tx]T,(x,t)∈Ω2×[0,2].(25)[\kappa, u]^T = \begin{cases} [1, t\sin(3\pi x)]^T, & (x,t) \in\Omega_1 \times[0,2],\\ [3\pi, tx]^T, & (x,t) \in\Omega_2 \times[0,2]. \end{cases} \tag*{(25)}

Let us introduce an auxiliary flux parameter q(x,t)=κ(x,t)∂u∂xq(x,t) = \kappa(x,t)\frac{\partial u}{\partial x} to reduce the original second order PDE into the following system of first-order PDEs

F(x,t):=∂u(x,t)∂t−∂q(x,t)∂x−s(x,t),Q(x,t):=q(x,t)−κ(x,t)∂u(x,t)∂x,B(x,t):=u(x,t)−g(x,t),I(x,t):=u(x,0)−h(x,0),(26)\begin{aligned} \mathcal{F}(x,t) &:= \frac{\partial u(x,t)}{\partial t} - \frac{\partial q(x,t)}{\partial x} - s(x,t),\\ \mathcal{Q}(x,t) &:= q(x,t) - \kappa(x,t)\frac{\partial u(x,t)}{\partial x},\\ \mathcal{B}(x,t) &:= u(x,t) - g(x,t),\\ \mathcal{I}(x,t) &:= u(x,0) - h(x,0), \tag*{(26)} \end{aligned}

where Q\mathcal{Q} represents the residuals of the flux operators. The corresponding source term s(x,t)s(x,t) and boundary/initial functions g(x,t)g(x,t) and h(x,t)h(x,t) can be calculated exactly using Eqs. (25) and (27).

We use a MLP with two inputs (x,t)(x,t) and two outputs (u,q)(u,q), consisting of six hidden layers with 60 neurons each. The network is trained for 500,000 epochs using the Adam optimizer and a ReduceLROnPlateau scheduler from PyTorch [55], which reduces the learning rate by a factor of 0.98 after 100 stagnant epochs, eventually driving it to a near-zero value. For the purpose of this problem, NF=NQ=8192N_F = N_Q = 8192 collocation points are generated once before training, together with NB=2×8192N_B = 2 \times8192 boundary points and NI=8192N_I = 8192 initial-condition points.

We begin by solving the problem using the point-wise constraint formulation described in § 3.1.1. Results at t=1t = 1 are shown in Figure 1, where panels (a) and (b) depict the temperature and heat flux distributions in the composite medium, respectively. Our proposed CAPU algorithm yields results in excellent agreement with the exact solution. In contrast, both the monotonic (MPU) and conditional (CPU) penalty update strategies produce solutions that deviate significantly from the reference, except at the boundary locations.

Profiles of predicted temperature and heat flux in a composite medium

Figure 1. Heat transfer in a composite medium: Profiles of (a) predicted temperature and (b) predicted heat flux at t=1t = 1, obtained using different penalty update strategies with the point-wise constraint formulation from § 3.1.1.

Despite the strong performance of the CAPU strategy, constraints within the PECANN framework are enforced in a point-wise manner. In Figure 2, we present the distribution of point-wise Lagrange multipliers at the final training epoch. Notably, the multipliers of each constraint type exhibit distinct and non-uniform distributions characterized by well-defined most probable values, particularly when the CAPU algorithm is employed, as shown in Figure 2(c). This observation suggests that constraint enforcement strategies based on constraint aggregation—rather than point-wise enforcement at every collocation or observational data point—may be both viable and computationally advantageous.

Distributions of Lagrange multipliers for MPU, CPU, and CAPU

Figure 2. Time-dependent heat conduction in a composite medium: Distribution of Lagrange multipliers obtained using the point-wise constraint formulation from § 3.1.1. (a) MPU (Algorithm 1); (b) CPU (Algorithm 2); (c) CAPU (Algorithm 3, proposed).

Figure 3 first illustrates the superior performance of the CAPU algorithm with the new aggregation formulation (§ 3.1.2), and then compares the evolution of evaluation metrics with that obtained using point-wise constraint enforcement. Panels (a) and (b) show the space-time temperature solution and the absolute-error distribution using CAPU, while panels (d) and (e) display the corresponding heat flux and its error. These results demonstrate close agreement with the exact solutions. Models trained with the MPU and CPU strategies failed to converge; therefore, their temperature and heat flux predictions are omitted. Their poor behavior of relative l2l^{2} error Er(u^,u)\mathcal{E}_{r}(\hat{u},u) is nonetheless shown in panels (c) and (f) for comparison. These results demonstrate that, under the constraint-aggregation formulation (14), the CAPU algorithm consistently outperforms both MPU and CPU.

Time-dependent heat conduction in a composite medium

Figure 3. Time-dependent heat conduction in a composite medium: Results are obtained using the constraint-aggregation formulation described in § 3.1.2. The prediction using CAPU with point-wise constraints is also included. Panels (a)–(c) show the contours of predicted temperature, the absolute error obtained with CAPU, and the evolution of the relative l2l^2 error Er(u^,u)\mathcal{E}_r(\hat{u},u), respectively, with the latter including comparisons among MPU, CPU, and CAPU (point-wise); panels (d)–(f) present the corresponding heat flux results.

Compared with the result using point-wise enforcement in Figure 3(c) and (f), CAPU under constraint-aggregation formulation achieves faster convergence in the early stages of training and stabilizes at lower error norms. The statistical performance of Er(u^,u)\mathcal{E}_{r}(\hat{u},u) further indicates that the new formulation with constraint aggregation achieves higher accuracy, with 5.42±2.41×10−35.42 \pm2.41 \times10^{-3} compared to 1.28±0.33×10−21.28 \pm0.33 \times10^{-2} for the point-wise formulation.

Overall constraint aggregation not only result in computational savings by reducing the number of Lagrange multipliers from thousands to a few, but it also improve accuracy levels. Based on these findings, we adopt PECANN with constraint aggregation for all remaining examples.

Forward problem: transonic rarefaction in inviscid Burgers’ equation

Nonlinear hyperbolic conservation laws frequently encounter rarefaction wave and shock phenomena. Accurately capturing high gradients and discontinuities without introducing spurious oscillations is crucial and constitutes a challenging task for any numerical method. The baseline PINN approach is known to struggle learning the solution of PDEs with discontinuities. Several enhancements to the baseline PINN model have been proposed to improve upon this deficiency of the baseline PINN model [57]. For instance, Liu et al. [58] observed that training performance deteriorates due to transition points and introduced a local gradient-dependent weight into the governing equations. Comparable strategies, including the addition of artificial viscosity [59], help stabilize training by weakening the network’s representation capability near discontinuities.

To illustrate the versatility of our proposed PECANN-CAPU framework in handling problems characterized by discontinuities and sharp gradients, we examine the transonic rarefaction scenario governed by the inviscid Burgers’ equation. This benchmark problem serves as a basis for quantitatively comparing three distinct penalty update strategies within ALM, corresponding to Algorithms 1–3.

Specifically, the inviscid transonic rarefaction problem is considered within the spatiotemporal domain U={(x,t)∣x∈[−0.5,0.5],t∈[0,1]}\mathcal{U} = \{(x,t) \mid x \in[-0.5, 0.5], t \in[0, 1]\}:

PDE:∂u∂t+u∂u∂x=0,IC:u(x,t=0)={−0.2,x≤0,0.4,x>0,BCs:{u(x=−0.5,t)=−0.2,u(x=0.5,t)=0.4.(27)\begin{aligned} \text{PDE:}\quad&\frac{\partial u}{\partial t} + u\frac{\partial u}{\partial x} = 0,\\ \text{IC:}\quad&u(x,t=0) = \begin{cases} -0.2, & x \le0,\\ 0.4, & x > 0, \end{cases}\\ \text{BCs:}\quad& \begin{cases} u(x=-0.5,t) = -0.2,\\ u(x=0.5,t) = 0.4. \end{cases} \tag*{(27)} \end{aligned}

The exact solution is given by:

u(x,t)={−0.2,if x≤−0.2t,x/t,if −0.2t≤x≤0.4t,0.4,if x≥0.4t.(28)u(x,t) = \begin{cases} -0.2, & \text{if } x \le-0.2t,\\ x/t, & \text{if } -0.2t \le x \le0.4t,\\ 0.4, & \text{if } x \ge0.4t. \end{cases} \tag*{(28)}

The initial contact discontinuity evolves into a transonic rarefaction wave with a sign change in the characteristic speed, located at a sonic point (u=0u = 0) [60]. Accurately predicting this initial discontinuity is crucial, as any error will propagate and expand over time, creating an error-amplifying scenario.

In order to exclude the effects of random sampling, especially in high-gradient regions, uniform structured meshes are employed across the domain U\mathcal{U}. During training, the coarse mesh uses 66×3366 \times33 points in space (xx) and time (tt), comprising 64×3264 \times32 interior points, 64 initial condition points at t=0t = 0, and 33 points per boundary. The fine mesh adopts a 130×65130 \times65 resolution, and all results are evaluated on a 258×129258 \times129 mesh. The MLP employed comprises three hidden layers with 20 neurons per layer. We adopt Adam optimizer, with its default learning rate (10−310^{-3}) across 10,000 epochs. Note that we do not modify the governing physics in our approach.

Figure 4 compares the predicted and exact solutions through their mean profiles and standard deviations at initial and final times with coarse and fine mesh, as well as the evolution of relative l2l^2 norms during training. All the predictions generally adhere to the prescribed initial data in panel (a), while minor overfitting is observed near the discontinuity, which mesh refinement can effectively mitigates as shown in panel (d). In contrast, the final time results in panel (b) demonstrate that only the CAPU algorithm maintains excellent agreement with the exact solution, showing both high accuracy and consistency. The performance of MPU cannot be improved even with fine mesh in panel (e), due to the challenges in handling the transition point of rarefaction wave. Furthermore, panel (c) and (f) illustrate the superior convergence rate of CAPU in terms of the relative l2l^2-norm, despite occasional spikes in the progress.

Figure 4. Transonic rarefaction: Panels (a)–(c) show coarse-mesh results; panels (d)–(f) show fine-mesh counterparts. Specifically: (a, d) predictions at t=0t = 0; (b, e) predictions at t=1t = 1; (c, f) evolution of relative l2l^2 error norms. Curves represent mean predictions; shaded areas indicate standard deviation bands.

Table 1 summarizes the mean and standard deviation of the relative l2l^2-norms at the end of the training. The results demonstrate that CAPU outperforms both MPU and CPU across all metrics, exhibiting significantly lower mean errors and superior robustness (smaller standard deviations) on both coarse and fine meshes. Mesh refinement generally enhances statistical performance, with additional residual points improving the capture of the residual loss particularly in high-gradient regions. Nevertheless, even with quadrupled sampling density, both the MPU and CPU consistently fail to achieve relative l2l^2 errors below 10−110^{-1}.

AlgorithmMesh 66 ×\times 33Mesh 130 ×\times 65
PECANN-MPU2.00±1.27×10−12.00 \pm1.27 \times10^{-1}1.66±1.23×10−11.66 \pm1.23 \times10^{-1}
PECANN-CPU2.61±1.25×10−12.61 \pm1.25 \times10^{-1}9.89±4.24×10−29.89 \pm4.24 \times10^{-2}
PECANN-CAPU2.93±0.51×10−22.93 \pm0.51 \times10^{-2}1.93±0.52×10−21.93 \pm0.52 \times10^{-2}

Table 1. Transonic Rarefaction: Relative l2l^2-norms over 5 trials using different penalty updating algorithms and two levels of uniform meshes.

Figure 5 shows the contours of exact solution, the prediction and the absolute point-wise errors from the best trial of the CAPU algorithm with fine mesh. The main errors are concentrated around the discontinuity. This occurs because smooth function approximation can never replace the sharp discontinuity, leading to the maximum point-wise error keeping relatively large despite the overall low global errors.

Three 3D surface plots showing the exact solution, CAPU prediction, and absolute point-wise error

Figure 5. Transonic rarefaction: (a) exact solution, (b) prediction of the CAPU algorithm from the best fine-mesh trial, (c) the corresponding absolute point-wise error.

To better understand the inner-workings of handling discontinuities, Figure 6 compares the evolution of penalty parameters, Lagrange multipliers, and corresponding constraints, along with objective and augmented losses, using the worst-performing trials with fine mesh for each algorithm. In panel (a), the penalty parameter of MPU grow exponentially and rapidly reach the maximum value μmax⁡=104\mu_{\max} = 10^4, while CPU reaches this limit more gradually by around epoch 50. In contrast, CAPU keeps the penalty parameters fixed at their initial values until

Figure 6. Transonic rarefaction problem: evolution of (a) penalty parameters, (b) Lagrange multipliers for boundary condition (BC), (c) Lagrange multipliers for initial condition (IC), (d) augmented losses and objectives, (e) BC constraints, (f) IC constraints from the worst trials of three strategies with fine mesh.

the augmented loss satisfies a convergence criterion, after which each parameter increases independently. Notably, μB\mu_B of CAPU consistently exceeds μI\mu_I while remaining below MPU’s prescribed upper bound.

Figure 6(b) and (c) show the evolution of Lagrange multipliers λB\lambda_B and λI\lambda_I. MPU demonstrates rapid growth (λB>103\lambda_B > 10^3, λI>104\lambda_I > 10^4 in 100 epochs) driven by its aggressive μ\mu update, while CPU shows similar but dampened behavior. In contrast, CAPU maintains initial λ\lambda values before gradually converging toward the much lower values. Based on the evaluation metrics, CAPU achieves the most accurate approximation of the near-optimal solution. This indicates that the Lagrange multipliers learned by CAPU are close to optimal, whereas the excessively rapid growth observed in MPU and CPU leads to deviations from the optimal values. Panel (d) shows CAPU achieves lower objective (residual-averaged) and augmented losses than MPU/CPU, with a more stable ratio indicating that MPU/CPU’s premature constraint over-prioritization leads to excessive Lagrange multipliers and optimization imbalance. Augmented loss spikes correlate with sudden CB/CIC_B/C_I violations in panels (e) and (f). The gradual reduction of CIC_I in the CAPU algorithm indicates that it progressively improves network expressiveness near discontinuities, unlike MPU and CPU, which prematurely overemphasize it.

Forward problem: 1D Poisson’s equation

Wang et al. [40] employed a one-dimensional Poisson’s equation with a multi-scale solution—originally introduced by Wang et al. [41]—as a regression task to illustrate the superior expressive capacity of KANs compared to standard MLPs. In support of their argument, they made the bold assertion that “if function fitting fails, solving PDEs will certainly fail,” and substantiated this claim by analyzing the convergence behavior of KANs and MLPs through the eigenvalue spectra of the Neural Tangent Kernel (NTK) matrix. For the same problem, Wang et al. [41] demonstrated that PINNs, conventional or equipped with a single Fourier feature mapping, failed to accurately learn the solution. However, by employing two distinct Fourier feature mappings, they achieved a relative l2l^2 error norm of 1.36×10−31.36 \times10^{-3}.

To demonstrate the versatility of our PECANN-CAPU algorithm in solving multi-scale problems we apply our approach to the same 1D Poisson’s equation considered on the domain Ω={x∣0≤x≤1}\Omega= \{x \mid0 \le x \le1\}.

∇2u=s,in Ω,u=0,on ∂Ω.(29)\nabla^2 u = s,\quad\text{in }\Omega, \qquad u = 0,\quad\text{on }\partial\Omega. \tag*{(29)}

An exact solution to Eq. (29) is assumed in the following form:

u(x)=sin⁡(2πx)+0.1sin⁡(bπx),∀x∈Ω,(30)u(x) = \sin(2\pi x) + 0.1\sin(b\pi x),\quad\forall x \in\Omega, \tag*{(30)}

where bb is a user-defined wavenumber. The corresponding source function s(x)s(x) is computed by substituting this solution into Eq. (29). Wang et al. [41] set b=50b = 50 to highlight the severe limitations of conventional PINNs, as well as PINNs with a single Fourier feature mappings, in learning the correct solution.

Following Wang et al. [40], we adopt the same feed-forward neural network with four hidden layers of 100 neurons each. The model is trained for 100,000 epochs using the Adam optimizer and the same ReduceLROnPlateau scheduler described in § 4.1. To avoid bias from point placement, a fixed set of uniformly distributed residual points and two boundary points are used as the batch during training.

Before introducing Fourier feature mappings, we first assess our PECANN-CAPU method with standard MLPs across a range of wavenumbers bb (8–50) and batch sizes (128–512). Table 2 presents the relative l2l^2 norm Er\mathcal{E}_r of the predictions. As the solution complexity grows with bb, the mean and standard deviation of Er\mathcal{E}_r increase—rising from 10−610^{-6} to 10010^{0} at a fixed batch size (i.e. number of collocation points) of 128. Despite this, the most challenging case (b=50b = 50) still yields a low error of Er=9.153×10−3\mathcal{E}_r = 9.153 \times10^{-3} in the best trial. To better capture high-frequency features at b=50b = 50, we test larger batch sizes (256 and 512), observing improved performance: the mean error drops to 3.690×10−23.690 \times10^{-2}, with the best trial converging to the 10−310^{-3} level.

bbBatchRelative l2l^2 mean ±\pm stdbest
81288.49±6.27×10−68.49 \pm6.27 \times10^{-6}8.37×10−78.37 \times10^{-7}
161281.51±1.06×10−51.51 \pm1.06 \times10^{-5}5.13×10−65.13 \times10^{-6}
321288.36±2.72×10−48.36 \pm2.72 \times10^{-4}5.17×10−45.17 \times10^{-4}
501281.37±2.15×1001.37 \pm2.15 \times10^{0}9.15×10−39.15 \times10^{-3}
502562.04±3.96×10−12.04 \pm3.96 \times10^{-1}1.07×10−31.07 \times10^{-3}
505123.69±4.78×10−23.69 \pm4.78 \times10^{-2}1.08×10−31.08 \times10^{-3}

Table 2. Poisson’s equation: Relative l2l^2 norms for different wavenumbers bb in the exact solution (Eq. (30)) and varying batch (i.e. total number of collocation points) sizes, using PECANN-CAPU framework with standard MLPs.

Figure 7 presents the mean and standard deviation of predictions, along with the best-performing trial, for different batch sizes using the CAPU algorithm. As the batch size increases from Figure 7(a) to (c), the mean predictions better align with the exact solution, and the uncertainty bands shrink. The inset in Figure 7(c) confirms this trend, though minor discrepancies remain. In all cases, the best-performing trials closely match the exact solution.

Poisson’s equation predictions for batch sizes 128, 256, and 512

Figure 7. Poisson’s equation (b=50b = 50 in Eq. (30)): Mean and standard deviation of the predictions, along with the best-performing trial, for batch sizes of (a) 128, (b) 256, and (c) 512 using the PECANN-CAPU framework with standard MLPs.

Figure 8 presents the evolution of the objective and boundary constraint for the best-performing trials. In all cases, the boundary constraint remains tightly satisfied around 10−1010^{-10}. However, with a batch size of 128 (panel a), the objective exhibits strong oscillations, unlike the more stable trends in panels (b) and (c). Despite reaching a lower final value (around 10−210^{-2}), the case in (a) yields the highest relative l2l^2 error, suggesting overfitting due to insufficient residual sampling. The observed oscillations likely stem from unresolved high-frequency components, and the decaying learning rate fails to stabilize training. These results underscore the importance of adequate residual sampling for capturing complex solution features.

Evolution of the objective and boundary constraint for batch sizes 128, 256, and 512

Figure 8. Poisson’s equation (b=50b = 50 in Eq. (30)): Evolution of the objective and boundary constraint of the best-performing trial for batch sizes of (a) 128, (b) 256, and (c) 512 using the CAPU algorithm.

Beyond the observed overfitting, our results demonstrate that while MLPs are commonly perceived as limited in their capacity to model multi-scale solutions, they can nonetheless effectively solve this forward problem when employed within the PECANN framework and optimized using the CAPU algorithm. We therefore argue that it is premature to dismiss MLPs as inherently incapable of representing multi-scale phenomena. Crucially, when evaluating the expressive capacity of a neural network in physics-informed learning of PDEs, the formulation of the optimization problem is as important as the choice of network architecture.

Enhancements with a single Fourier feature mappings

We adopt the same wide and shallow neural network as in Wang et al. [41], with two hidden layers of 100 units each. The network is trained with the Adam optimizer for 40,000 epochs, using an exponential learning rate decay (factor 0.9 every 1000 iterations). Residual points are resampled randomly at each epoch. Instead of multiple, tuned Fourier feature mappings as adopted in [41], we apply the original method from Tancik et al. [39], replacing the first hidden layer with a single Fourier feature mapping matrix B∈R1×50\mathbf{B} \in\mathbb{R}^{1 \times50} only. We note that Wang et al. [41] did not obtain good predictions with a single layer of Fourier feature mappings.

Figure 9 summarizes the mean and standard deviation of the relative l2l^2 error (Er\mathcal{E}_r) over 10 trials for varying Gaussian deviations σ\sigma and two batch sizes (128 and 512). Best-performing trial results are also shown. The mean Er\mathcal{E}_r remains stable below 10−210^{-2} for σ\sigma values between 0.25 and 1. As σ\sigma increases beyond 1, the mean Er\mathcal{E}_r rises, indicating reduced prediction accuracy. In contrast, larger batch sizes generally yield lower errors, particularly for smaller σ\sigma. Notably, σ=1\sigma= 1 provides the best overall performance and consistency; increasing the batch size to 512 further reduces the mean error to the order of 10−510^{-5}. These findings indicate that a single Fourier feature mapping with σ=1\sigma= 1 is sufficient to capture both low- and high-frequency components in multi-scale solutions when using PECANN-CAPU algorithm. This contrasts with prior findings in Wang et al. [41], which suggests the need for multiple Fourier features with tuning different σ\sigma values to separately address different frequency components.

Variation in the statistical performance of relative $l^2$ norms for Poisson’s equation

Figure 9. Variation in the statistical performance of relative l2l^2 norms for Poisson’s equation (b=50b = 50 in Eq. (30)), shown as a function of Gaussian deviation σ∈[0.25,16]\sigma\in[0.25, 16]. A single Fourier feature mapping is used within the PECANN-CAPU framework.

An important observation is that the performance of the CAPU algorithm is sensitive to the choice of Gaussian deviation σ\sigma, with larger values leading to significant degradation in accuracy. To investigate this further, we examine σ=1\sigma= 1, 4, and 16, and present the best-performing predictions along with the evolution of loss terms in Figure 10. As shown in panel (a), only σ=1\sigma= 1 produces smooth and accurate predictions, while larger values—especially σ=16\sigma= 16—lead to discontinuities and wedge-like artifacts. Panel (b) shows that the objective (averaged residual loss) initially drops sharply in all cases and oscillates around 10−210^{-2}—a lower level than in standard networks trained with CAPU—indicating that Fourier features enhance expressiveness. However, this increased capacity also raises the risk of overfitting, particularly for large σ\sigma, as reflected in poor predictive performance. This also explains the improved accuracy at batch size 512, where more residual points help counteract overfitting. Notably, for σ=16\sigma= 16, the boundary constraint CBC_B fails to decrease, despite continued growth of the Lagrange multiplier λB\lambda_B under CAPU. While increasing the penalty scaling factor ηB\eta_B might accelerate λB\lambda_B and better enforce the constraint, we recommend simply using the standard Gaussian deviation σ=1\sigma= 1 when applying CAPU with a single Fourier feature mapping, as it yields accurate, consistent predictions and faster convergence.

Predictions and loss evolution for Poisson’s equation

Figure 10. Poisson’s equation (b=50b = 50 in Eq. (30)): (a) Prediction from the best-performing trials for σ=1\sigma= 1, 4, and 16 with a batch size of 128; (b) corresponding evolution of the objective (solid lines) and boundary constraint (dotted lines). A single Fourier feature mapping is used within the PECANN-CAPU framework.

To examine the limitations of the CAPU algorithm with a single Fourier feature mapping (σ=1\sigma= 1), we increase the wavenumber of the exact solution up to b=128b = 128. To mitigate overfitting, 512 residual points are randomly sampled at each epoch. Table 3 reports the relative ℓ2\ell^2 error statistics. Although performance slightly declines with increasing bb, the method still achieves accurate and consistent results even at b=128b = 128, demonstrating its effectiveness on highly multiscale problems.

bbRelative ℓ2\ell^2
mean ±\pm stdbest
503.29±0.88×10−53.29 \pm0.88 \times10^{-5}2.02×10−52.02 \times10^{-5}
645.85±3.23×10−55.85 \pm3.23 \times10^{-5}2.86×10−52.86 \times10^{-5}
1284.36±3.14×10−3\mathbf{4.36 \pm3.14 \times10^{-3}}6.50×10−4\mathbf{6.50 \times10^{-4}}

Table 3. Poisson’s equation: relative ℓ2\ell^2 errors for different wavenumbers bb of the exact solution (Eq. (30)) using a single Fourier feature mapping with standard normal distribution σ=1\sigma= 1. Results are obtained using the CAPU algorithm with the Adam optimizer over 4×1044 \times10^4 epochs and a batch size of 512.

The 1D Poisson’s equation with a multi-scale solution example show that, given a sufficiently large number of collocation points, standard MLPs are capable of learning the solutions despite the claims made in Wang et al. [40]. Moreover, when the number of collocation points is reduced, the incorporation of a single Fourier feature mapping proves sufficient to achieve excellent accuracy. Overall, the performance of the PECANN-CAPU framework substantially improves with a σ=1\sigma= 1 Fourier feature mapping, enabling accurate resolution of both low- and high-frequency components in multiscale problems when sufficient residual points are used.

Our empirical observations underscore the critical importance of the underlying constrained optimization formulation and its associated algorithms in physics-informed learning, particularly for accurately capturing multi-scale and highly oscillatory solutions. We note that our conclusion does not diminish the relevance of advances in neural network architectures; rather, it highlights the complementary importance of optimization strategies in achieving robust and efficient learning.

Forward problem: 2D Helmholtz equation with high-frequency and multi-scale solutions

The solution of the Helmholtz equation, across varying levels of complexity, has been extensively utilized within the physics-informed machine learning community as a benchmark to evaluate the effectiveness of different methodologies, including APINNs [61], FBPINNs [62], and PIKANs [53].

The 2D Helmholtz equation in the domain Ω={(x,y)∣−1≤x,y≤1}\Omega= \{(x, y) \mid-1 \leq x, y \leq1\} is defined as:

∇2u+k2u=s,in Ω,u=g,on ∂Ω,(31)\begin{aligned} \nabla^2 u + k^2 u &= s, \quad\text{in } \Omega,\\ u &= g, \quad\text{on } \partial\Omega, \tag*{(31)} \end{aligned}

where kk is the Helmholtz wavenumber.

Here, we will consider two categories of problems. The first adopts the same setup used to assess PIKANs [53]. This problem is first employed to better understand the sensitivity of the predicted solution to the hyperparameters of the CAPU algorithm (see Appendix § 12), and to highlight that algorithmic innovation plays a role as crucial as architectural design. The second problem is much more challenging than the first one. It involves a localized Gaussian-like source term with a highly oscillatory solution. This case is used to further assess the PECANN-CAPU approach augmented with a single Fourier feature mapping.

The exact solution to this first problem with k=1k = 1 is given by:

u(x,y)=sin⁡(a1πx)sin⁡(a2πy),∀(x,y)∈Ω.(32)u(x,y) = \sin(a_1\pi x)\sin(a_2\pi y), \quad\forall(x,y) \in\Omega. \tag*{(32)}

Given the wavenumbers (a1,a2)(a_1,a_2), the corresponding source terms s(x,y)s(x,y) and g(x,y)g(x,y) are computed by substitution into Eq. (31). Here, a1=a2=6a_1 = a_2 = 6 is set to produce a high-frequency solution.

A MLP with H=6H = 6 hidden layers with W=128W = 128 neurons per layer is employed. The model is trained for 500,000 epochs using the Adam optimizer on a 51×5151 \times51 uniform collocation grid, and evaluated on a finer 201×201201 \times201 mesh. Given Adam’s first-order nature, a decaying learning rate is essential for convergence over long durations, and we adopt the same ReduceLROnPlateau scheduler as in the previous subsection. The CAPU algorithm is applied with a penalty scaling factor of η=1\eta= 1, following the insights gained from the sensitivity study presented in Appendix § 12.

Table 4 presents a statistical comparison of relative l2l^2 norms, Er\mathcal{E}_r, across different methods. Under the same optimizer, both CPU and CAPU achieve the best-trial performance, with Er≈3×10−3\mathcal{E}_r \approx3 \times10^{-3}, comparable to cPIKAN with RBA. However, CAPU consistently attains high accuracy, exhibiting a mean error approximately two orders of magnitude lower and the smallest standard deviation among the PECANN framework. An additional configuration further enhances accuracy: when trained with the L-BFGS optimizer, even a shallow network comprising three hidden layers with 40 neurons each attains both accuracy and robustness, achieving a mean Er\mathcal{E}_r of 1.33×10−31.33 \times10^{-3} and a minimum of 5.61×10−45.61 \times10^{-4}.

MethodN. ParamsHHWWOptim.Relative l2l^2 mean ±\pm stdbest
PINN [53]823046128Adam-4.30×10−24.30 \times10^{-2}
cPIKAN+RBA [53]20960532Adam-3.81×10−33.81 \times10^{-3}
PECANN-MPU823046128Adam12.53±7.93×10−112.53 \pm7.93 \times10^{-1}1.20×10−11.20 \times10^{-1}
PECANN-CPU823046128Adam1.30±2.54×10−11.30 \pm2.54 \times10^{-1}3.36×10−3\mathbf{3.36 \times10^{-3}}
PECANN-CAPU823046128Adam6.69±3.38×10−3\mathbf{6.69 \pm3.38 \times10^{-3}}3.68×10−33.68 \times10^{-3}
PECANN-CAPU3441340L-BFGS1.33±0.653×10−31.33 \pm0.653 \times10^{-3}5.61×10−45.61 \times10^{-4}

Table 4. Helmholtz equation (Eq. 33 with a1=a2=6a_1 = a_2 = 6): Relative l2l^2 comparison between different methods and the PECANN-CAPU method. All methods use the Adam optimizer for 5×1055 \times10^5 epochs, except the last case that adopt L-BFGS for 5×1045 \times10^4 epochs.

Figure 11(a) displays PECANN predictions obtained using the CAPU algorithm. As seen in panel (b), we observe that absolute error level is generally very low, around 10−310^{-3}.

Helmholtz equation prediction and absolute point-wise error

Figure 11. Helmholtz equation (Eq. (32) with a1=a2=6a_1 = a_2 = 6): (a) PECANN-CAPU best trial prediction, (b) the corresponding absolute point-wise error.

Figure 12 shows the evolution of all the loss terms and evaluation metrics for the CAPU algorithm with Adam. In panel (a), a smooth and gradual increase of the penalty parameter μB\mu_B is observed, interacting with the progressive decrease of the boundary constraint CB\mathcal{C}_B, as governed by the update rule in Eq. (10). This interaction rapidly amplifies the Lagrange multiplier, leading to λB>μB\lambda_B > \mu_B during the early training stage; subsequently, the growth rate slows, and the constraint stabilizes. In panel (b), the network converges after approximately 10510^5 epochs, with the relative l2l^2 norm stabilizing below 10−210^{-2}. Note that the shallow network trained with L-BFGS converges also fast-by 10410^4 epochs-with a lower mean error near 10−310^{-3}.

Evolution of objective and constraint losses and norms

Figure 12. Helmholtz equation (Eq. (32) with a1=a2=6a_1 = a_2 = 6): Evolution of mean and standard deviation of (a) the objective and constraint losses, (b) l∞l^\infty and relative l2l^2 norms using CAPU.

A much more challenging Helmholtz equation problem

In their FBPINN work, Dolean et al. [62] considered the 2D Helmholtz equation (31) on the domain Ω={(x,y)∣0≤x,y≤1}\Omega= \{(x,y) \mid0 \le x,y \le1\}, with zero Dirichlet boundary conditions g=0g = 0 and a localized Gaussian-like source term:

s(x)=12πσ2exp⁡(−∥x−0.5∥222σ2),(33)s(\mathbf{x}) = \frac{1}{2\pi\sigma^2}\exp\left(-\frac{\lVert\mathbf{x} - 0.5 \rVert_2^2}{2\sigma^2}\right), \tag*{(33)}

The wavenumber and the width of the Gaussian source term are defined as:

k=2Lπ1.6,σ=0.82L,(34)k = \frac{2^L\pi}{1.6}, \qquad\sigma= \frac{0.8}{2^L}, \tag*{(34)}

The solution complexity can be increased systematically by increasing the parameter LL.

This specific problem represents a more realistic application of the Helmholtz equation and has no analytical solution. Therefore, we generate reference data using a second-order central difference scheme on a 361×361361 \times361 uniform grid, using the same finite difference code adopted in [62]. It is worth noting that for L≥6L \ge6, the finite-difference solution shows inconsistencies, even after increasing the mesh resolution. The four different methods deployed in Dolean et al. [62] failed to produce a common solution. Therefore, we restrict our analysis to cases with L≤5L \le5. We used the less complex L=4L = 4 case to better understand the improvement of our CAPU algorithm relative to the original RMSProp scheme. Those results are presented in Appendix § 11.

Setting L=5L = 5 results in a highly concentrated source term and a large Helmholtz wavenumber, thereby significantly increasing the complexity of the problem. To enhance the approximation capability of the PECANN framework with the CAPU algorithm, we augment the network architecture by incorporating a low-frequency Fourier feature mapping, sampled from a standard Gaussian distribution (σ=1\sigma= 1). The training data consist of 1602160^2 randomly sampled residual points and 160 boundary points per edge.

Table 5 presents a comparison of the relative l2l^2 errors between PECANN predictions using standard MLPs and those using Fourier feature mappings, with comparable parameter counts. Additional comparisons investigate various Fourier-based architectures by varying the number of hidden layers (HH) and the number of neurons per layer (WW).

ArchitectureN. ParamsHHWWRelative l2l^2 mean ±\pm stdbest
MLP9213209.20±0.24×10−19.20 \pm0.24 \times10^{-1}8.83×10−18.83 \times10^{-1}
Fourier (σ=1\sigma= 1)8613207.42±0.91×10−17.42 \pm0.91 \times10^{-1}6.14×10−16.14 \times10^{-1}
MLP34413408.61±0.18×10−18.61 \pm0.18 \times10^{-1}8.43×10−18.43 \times10^{-1}
Fourier (σ=1\sigma= 1)33819201.78±0.43×10−11.78 \pm0.43 \times10^{-1}1.33×10−11.33 \times10^{-1}
Fourier (σ=1\sigma= 1)33213401.45±0.38×10−11.45 \pm0.38 \times10^{-1}9.64×10−29.64 \times10^{-2}
Fourier (σ=1\sigma= 1)716118205.25±2.52×10−15.25 \pm2.52 \times10^{-1}2.19×10−12.19 \times10^{-1}
Fourier (σ=1\sigma= 1)73813608.89±1.16×10−28.89 \pm1.16 \times10^{-2}6.73×10−26.73 \times10^{-2}
Fourier (σ=1\sigma= 1)130413809.22±0.54×10−29.22 \pm0.54 \times10^{-2}8.28×10−28.28 \times10^{-2}

Table 5. 2D Helmholtz equation with a localized source term (Eq. (33) with L=5L = 5): Relative l2l^2 norms of PECANN predictions with standard MLPs and networks with a σ=1\sigma= 1 Fourier feature mapping, using the CAPU algorithm. Models are trained with the L-BFGS optimizer for 8×1048 \times10^4 epochs and the CAPU algorithm.

High mean Er\mathcal{E}_r values on the order of 10−110^{-1} indicate that a width of W=20W = 20 is insufficient for solving the L=5L = 5 case, regardless of using MLP or Fourier-feature architectures. Doubling the width to W=40W = 40 markedly improves both accuracy and consistency when Fourier features are included, with the best-performing trial achieving Er=9.642×10−2\mathcal{E}_r = 9.642 \times10^{-2}—outperforming the corresponding MLP by an order of magnitude. Among models with similar parameter counts, shallower, wider Fourier networks (e.g., H=3H = 3, W=40W = 40) outperform deeper ones (e.g., H=9H = 9, W=20W = 20), with the gap widening at matched sizes around 7200 parameters. Accuracy continues to improve with model size in wide configurations, converging at H=3H = 3, W=60W = 60 with a minimum Er=6.726×10−2\mathcal{E}_r = 6.726 \times10^{-2}. This degradation in deeper networks may stem from vanishing gradients and the limitations of the tanh actuation function in representing periodic nonlinearities [63], which motivates further investigation into the role of Fourier feature mapping at the input layer.

To better understand the impact of Fourier feature mapping under similar parameter counts, we analyze the probability density distributions of elements in the randomized matrix B. Figure 13 shows the mean and standard deviation bands across five trials for networks of varying width, compared to the standard normal distribution N(0,1)\mathcal{N}(0, 1). For W=20W = 20, the distributions deviate significantly: although the mean remains symmetric, the standard deviation is clearly below 1, with wide variation. Increasing to W=40W = 40 reduces this mismatch, consistent with earlier results—models with W=20W = 20 consistently underperform, while doubling the width improves accuracy with some variability. As WW increases to 60 and 80, the distributions increasingly align with N(0,1)\mathcal{N}(0, 1), yielding further but saturating gains. This plateau in accuracy, as shown in Table 5, suggests diminishing returns from further distributional alignment. Given the lack of an exact analytical solution, we do not seek higher accuracy relative to the reference, which may itself be affected by mesh resolution or discretization scheme.

Probability density distributions of elements in the randomized matrix B for Fourier networks with varying neuron counts, compared to the standard normal distribution

Figure 13. 2D Helmholtz equation with a localized source term (Eq. (33) with L=5L = 5): Mean and deviation bands of probability density distributions with elements in the random matrix B over five trials for Fourier networks with varying neuron counts, compared to the standard normal distribution.

Figure 14 compares MLP and Fourier-featured networks by tracking the evolution of the objective J\mathcal{J} and boundary constraint CB\mathcal{C}_{B}. In panel (a), MLPs with W=20W = 20 and W=40W = 40 converge to J≈100\mathcal{J} \approx10^{0}, while in panel (b), their CB\mathcal{C}_{B} values rise and plateau just below 10−410^{-4}, indicating boundary violations and waveform leakage beyond the domain for L=5L = 5. This reflects the MLPs’ limited capacity at higher Helmholtz wavenumber, in contrast to their adequate performance at L=4L = 4 (Appendix § 11). While increasing the penalty scaling factor (e.g., to 10) may enhance boundary enforcement, the slow decay of J\mathcal{J} highlights the inherent challenge of this high-frequency case.

With a low-frequency Fourier mapping (σ=1\sigma= 1) at the input, the W=40W = 40 network shows a rapid initial drop in J\mathcal{J}—well below that of the corresponding MLP—followed by a slower decline around epoch 10410^{4} in Figure 14(a). This slowdown corresponds to a gradual reduction in the boundary constraint CB\mathcal{C}_{B}, as shown in Figure 14(b), where CAPU steadily enforces the boundary condition. Both losses converge around epoch 3×1043 \times10^{4}, with J≈10−3\mathcal{J} \approx10^{-3} and CB\mathcal{C}_{B} below 10−810^{-8}. Increasing the width to W=60W = 60 further improves performance: J\mathcal{J} rapidly drops below 10−210^{-2} and briefly plateaus, while CAPU effectively suppresses boundary violations, leading both terms to converge before epoch 2×1042 \times10^{4}.

Evolution of the mean and standard deviation of the objective and boundary constraint for different MLP and Fourier-featured network architectures

Figure 14. 2D Helmholtz equation with a localized source term (Eq. (33) with L=5L = 5): Evolution of the mean and standard deviation of the objective (a) and boundary constraint (b) for different MLP and Fourier-featured network architectures.

Reference solution, prediction, and absolute point-wise error for the 2D Helmholtz equation

Figure 15. 2D Helmholtz equation with a localized source term (Eq. (33) with L=5L = 5): (a) reference solution, (b) prediction of the best-performing trial using the CAPU algorithm with Fourier-featured network of 3 hidden layers and 60 neurons per layer, (c) the corresponding absolute point-wise error.

Figure 15 illustrates the reference solution of L=5L = 5, and the prediction from the lowest-Er\mathcal{E}_{r} trial, as well as the absolute point-wise error. Overall, the agreement between the exact and numerical solution is very good for this challenging case. The primary absolute error is concentrated at the center of the square domain, with a maximum value of 7.843×10−37.843 \times10^{-3}.

Forward problem: reversible advection of a passive scalar by a single vortex

Baseline PINN models struggle with long-time dynamics when trained over the entire spatio-temporal domain. Wang et al. [64] showed that this failure happens because baseline PINN models exhibit an implicit bias toward minimizing PDE residuals at later times before fitting initial conditions. A practical alternative to address this issue is to use discrete time models with time-marching schemes such as the Runge-Kutta methods [15]. Wang et al. [64] proposed a causal training algorithm that restores physical causality in PINNs by adaptively re-weighting the PDE residual loss toward initial times during training of continuous time models.

In contrast to previously discussed approaches, we adopt a time-windowing strategy that is enabled by the constrained optimization formulation employed in the PECANN framework. In our approach the final solution of a time window is enforced as an initial condition constraint for the next time window in a principled fashion.

To evaluate this time-windowing strategy, we consider a challenging benchmark problem: reversible scalar advection by a single vortex flow. Originally introduced by LeVeque [65] and Rider and Kothe [66], this benchmark is well-established for assessing various interface tracking and front capturing techniques used in two-phase flow simulations, including the volume-of-fluid (VOF) method [66, 67], the level-set method [68, 69], and the moment-of-fluid (MOF) method [70].

In this benchmark, a circular dye distribution is advected and then the flow is reversed to observe to what accuracy level the original circular shape is recovered. This process serves as a stringent test of a numerical method’s conservation and diffusion properties. For PECANN, successful recovery also highlights its capability for accurate long-time PDE integration.

The test problem is governed by the advection of a passive scalar ϕ\phi by a predefined time-dependent flow field without any physical diffusion in the scalar transport equation

∂ϕ∂t+u⋅∇ϕ=0,(35)\frac{\partial\phi}{\partial t}+\mathbf{u}\cdot\nabla\phi=0, \tag*{(35)}

where ϕ∈[0,1]\phi\in[0,1] is the volume fraction function for a scalar field, which is advected by the velocity field u\mathbf{u}. The initial condition is defined by setting ϕ=1\phi=1 within a circle of radius 0.15 centered at (0.5,0.75)(0.5,0.75) inside a unit square domain, and ϕ=0\phi=0 elsewhere. This results in a scalar field with a discontinuous interface. For the domain boundaries, ϕ\phi is always set to zero.

A time-dependent, divergence-free velocity field is defined by the following stream function:

Ψ(x,y,t)=1πsin⁡2(πx)sin⁡2(πy)cos⁡(πtT),x,y∈[0,1],(36)\Psi(x,y,t)=\frac{1}{\pi}\sin^{2}(\pi x)\sin^{2}(\pi y)\cos\left(\frac{\pi t}{T}\right),\qquad x,y\in[0,1], \tag*{(36)}

where TT is the specified period, chosen as 8.0 for this test.

To predict the long-time evolution of this scalar advection problem, we propose a non-overlapping time-marching strategy. Unlike conventional finite-difference-based time-marching schemes, our approach fundamentally differs in its formulation and implementation. Specifically, the entire temporal domain is partitioned into smaller, non-overlapping subdomains. The predicted solution within each temporal-spatial subdomain is subsequently employed as an initial condition constraint for the succeeding subdomain. This sequential conditioning is facilitated by the underlying constrained optimization framework of PECANN, which enables principled enforcement of temporal-spatial consistency within each subdomain.

In the present problem, each time window spans Δt=0.2\Delta t=0.2, dividing the total simulation period into 40 uniform intervals. A separate neural network models each subdomain, taking (x,y,t)(x,y,t) as input and producing a scalar output ϕ\phi. The network architecture consists of 6 hidden layers (H=6H=6), each with 40 neurons. Since the passive scalar ϕ\phi is bounded between 0 and 1 as prior knowledge, a sigmoid activation function is employed in the output layer. Accordingly, the output in the iith subdomain is given by

ϕi=fθi(x,y,t)=sigmoid⁡(WH+1⋅HH+bH+1),for t∈[(i−1)Δt,iΔt],(37)\phi_i=f_{\theta_i}(x,y,t)=\operatorname{sigmoid}\left(\mathbf{W}_{H+1}\cdot\mathbf{H}_H+\mathbf{b}_{H+1}\right),\qquad\text{for }t\in[(i-1)\Delta t,i\Delta t], \tag*{(37)}

where θi\theta^{i} denotes the trainable parameters of the iith model. The time-marching update is performed via:

ϕ(x,y,(i−1)Δt;θi)←ϕ(x,y,(i−1)Δt;θi−1).(38)\phi(x,y,(i-1)\Delta t;\theta^{i}) \leftarrow\phi(x,y,(i-1)\Delta t;\theta^{i-1}). \tag*{(38)}

Each subdomain is sampled using two levels of uniform meshes: a coarse mesh 129×129×11129 \times129 \times11 and a fine mesh 257×257×11257 \times257 \times11. The first two dimensions correspond to equally spaced points in the spatial domain, aligned with the setup in Xie et al. [71], while the third dimension denotes the number of points along the temporal axis spanning the time window Δt=0.02\Delta t = 0.02. The sampled points include interior residual points as well as boundary and initial condition points. Training begins with the Adam optimizer for 3000 epochs, followed by L-BFGS for 2000 epochs. The CAPU algorithm is applied with a penalty scaling factor η=0.01\eta= 0.01 for both initial and boundary constraints.

Figure 16 displays the time evolution of predicted scalar on the fine mesh at five snapshots (t=0,2,4,6,8t = 0, 2, 4, 6, 8). Initially, a circular passive scalar is accurately captured at t=0t = 0. As the vortex flow progresses, the scalar is stretched into a spiral structure with a thin, elongated tail, reaching maximum deformation at t=4t = 4. This process introduces strong interface distortion and broadens the discontinuity, creating a highly challenging scenario for interface tracking methods to capture. In the second half of the simulation, the vortex rotation reverses direction to recover the original shape. At t=6t = 6, the interface resembles that at t=2t = 2, indicating a near-symmetric reversal of deformation. By the end of the period (t=8t = 8), the original circular shape is nearly restored, demonstrating superior conservation and minimal numerical smearing in the method.

Reversible transport of a passive scalar field by a vortex

Figure 16. Reversible transport of a passive scalar field by a vortex. Scalar field at t=0t = 0, 22, 44, 66, and 88 on the fine mesh using the CAPU algorithm.

Figure 17 shows the evolution of loss terms and optimization parameters across the 40 subdomain models trained with the CAPU algorithm. In panel (a), switching from Adam to L-BFGS yields significant convergence improvements, with the objective J\mathcal{J} dropping by nearly three orders of magnitude and stabilizing below 10−410^{-4}. The initial constraint CIC_{I} consistently decreases to around 10−610^{-6}, while the boundary constraint CBC_{B} falls sharply below 10−1010^{-10}—demonstrating CAPU’s effectiveness in enforcing constraints within each time window. This robust performance ensures more reliable predictions for subsequent windows and contributes to the method’s superior accuracy.

Evolution of losses, Lagrange multipliers, and penalty parameters during training

Figure 17. Reversible transport of a passive scalar field by a vortex: Evolution of the mean and standard deviation during training across all 40 subdomain models on the fine mesh using the CAPU algorithm. (a) Loss terms (objective, boundary, and initial constraints); (b) Corresponding Lagrange multipliers and penalty parameters.

In Figure 17(b), the boundary penalty parameter μB\mu_{B} rapidly approaches its upper bound of 10610^{6}, triggered by the sharp decline in CBC_{B}. This limit is implicitly imposed by the adaptive update rule in Eq. (10), given η=0.01\eta= 0.01 and ϵ=10−16\epsilon= 10^{-16}. In contrast, Lagrange multipliers stabilize early in training.

By plotting scalar contour lines at ϕ=0\phi= 0 to 11 in increments of 0.20.2, Figure 18 presents the predicted interface locations for both coarse (panel a) and fine (panel b) meshes at midpoint (t=4t = 4) and at final time (t=8t = 8). In panel (a), the CAPU algorithm consistently captures the interface over the entire simulation, even on the coarse mesh. At t=4t = 4, the prediction exhibits a tightly wound spiral, while at t=8t = 8, it evolves into an irregular pattern around the original circular region, indicating degradation of scalar information.

Predicted interfaces at intermediate and final times for coarse and fine meshes

Figure 18. Reversible transport of a passive scalar field by a vortex: Predicted interfaces at intermediate and final times, represented by the iso-contours ϕ=0\phi= 0 to 11 in increments of 0.20.2, using the CAPU algorithm with (a) the coarse mesh and (b) the fine mesh.

Although the predicted scalar field on the coarse mesh yields acceptable results, it exhibits noticeable imperfections in reconstructing the original circular geometry. In contrast, simulations performed on the fine mesh demonstrate significant improvements: the solution evolves into a smooth and coherent spiral structure at the midpoint and accurately recovers an almost perfectly circular profile by the final time.

Next, we compare the quality of our predictions for this challenging benchmark problems against published studies that implement a variety of interface tracking methods using finite difference/volume type of methods. Xie et al. [71] compiled results from 17 different studies including structured and unstructured grids. The following metric was used to assess the accuracy of each method

Error⁡=∑i=1Ne∣ϕpi−ϕei∣Ωi,(39)\operatorname{Error} = \sum_{i=1}^{N_e} \left\lvert\phi_{pi} - \phi_{ei} \right\rvert\Omega_i, \tag*{(39)}

where ϕpi\phi_{pi} and ϕei\phi_{ei} denote the predicted and exact values on grid cell ii, and Ωi\Omega_i represents the area of that cell. Error values range from a worst case of 5.02×10−35.02 \times10^{-3} to a best case of 5.09×10−45.09 \times10^{-4} for the present problem.

Table 6 presents a comparison of our method’s mean error estimate and its standard deviation–computed over five trials–at the final time (t=8.0t = 8.0) against selected benchmark studies, including the average of the error from 17 different methods as compiled in [71]. Our approach achieves a mean error of 1.113×10−31.113 \times10^{-3}, underscoring its competitive accuracy relative to established interface tracking methods.

AlgorithmError
Rider and Kothe [66]1.44×10−31.44 \times10^{-3}
EMFPA/Youngs [72]2.13×10−32.13 \times10^{-3}
Stream/Youngs [67]2.16×10−32.16 \times10^{-3}
CLSVOF [73]1.93×10−31.93 \times10^{-3}
THINC/QQ [74]3.12×10−33.12 \times10^{-3}
CBLSVOF [71]2.73×10−32.73 \times10^{-3}
Mean of various works from [71]2.255±1.167×10−32.255 \pm1.167 \times10^{-3}
Current Method1.113±0.210×10−31.113 \pm0.210 \times10^{-3}

Table 6. Reversible transport of a passive scalar field by a vortex: numerical errors at T=8T = 8 with different numerical methods and our proposed method

Inverse problem: estimation of space-wise heat source in transient heat conduction

In Section 3.1.3, we extended the PECANN-CAPU framework with constraint aggregation to inverse PDE problems. Here, we apply this formulation to jointly learn the temperature field and a spatially dependent heat source from noisy temperature measurements. Specifically, we consider the inverse identification of a space-wise heat source term problem from Hasanov [75], in which the temporal component of the source is known and aim to infer its spatial part.

The governing equation (24) defined in the domain U={(x,t)∣0≤x,t≤1}\mathcal{U} = \{(x,t) \mid0 \le x,t \le1\} is adopted with the following separable source term,

s(x,t)=F(x)H(t),(40)s(x,t) = F(x)H(t), \tag*{(40)}

where H(t)=5exp⁡(−3t)H(t) = 5\exp(-3t) is prescribed and the thermal conductivity is given by κ(x)=4+x2\kappa(x) = 4 + x^2. This type of inverse problem is designated as ISPF in [75]. In addition to known boundary and initial conditions, we incorporate final-time temperature measurements corrupted by Gaussian noise:

uT(x)=(1+n)u(x,Tf),n∼N(0,1).(41)u_T(x) = (1+n)u(x,T_f), \qquad n \sim\mathcal{N}(0,1). \tag*{(41)}

The exact solution and the corresponding source are as follows:

u(x,t)=5exp⁡(−3t)sin⁡(πx),u(x,t) = 5\exp(-3t)\sin(\pi x),
F(x)=(−3+π2(4+x2))sin⁡(πx)−2πx.(42)F(x) = \left(-3+\pi^2(4+x^2)\right)\sin(\pi x)-2\pi x. \tag*{(42)}

Figure Figure 19a indicates the problem setup via the sampling configuration, which includes the known boundary and initial points, noisy measurements, and residual points representing the PDE. The corresponding exact, time-dependent temperature field is shown in Figure 19b.

Inverse problem: (a) distribution of residual, boundary, and initial points along with noisy data from the final-time measurement; (b) exact, time-dependent temperature field, $u(x,t)$

Figure 19. Inverse problem: (a) distribution of residual, boundary, and initial points along with noisy data from the final-time measurement; (b) exact, time-dependent temperature field, u(x,t)u(x,t).

We cast this inverse problem as a PDE-constrained optimization problem using a neural network with inputs (x,t)(x,t), one hidden layer of 40 units, and outputs (u,F)(u,F). To ensure that FF depends only on xx, we evaluate FF with a constant temporal input, effectively removing any influence from tt. This is implemented as:

u(x,t),_=fθ(x,t),u(x,t),\_ = f_{\theta}(x,t),
_,F(x)=fθ(x,tconst),(43)\_,F(x) = f_{\theta}(x,t_{\mathrm{const}}), \tag*{(43)}

where _\_ denotes a placeholder output that is computed but discarded, tconstt_{\mathrm{const}} is a fixed value as 0. Thus, the network performs two forward passes: first, using the true (x,t)(x,t) to obtain uu; second, using (x,tconst)(x,t_{\mathrm{const}}) to obtain FF. The loss formulation follows Eq. (15), where the averaging measurement loss is treated as the objective J\mathcal{J}, and the PDE residual CFC_F, boundary CBC_B, and initial condition CIC_I losses are enforced as constraints. Training samples include 10410^4 collocation points for the PDE residual, 128 points for the initial condition, each boundary, and the final-time measurement. The model is trained for 10410^4 epochs using the L-BFGS optimizer, with the CAPU algorithm applied. Penalty scaling factors are set to ηF=0.01\eta_F=0.01 for CFC_F and 1 for the remaining constraints, as detailed in subsection 3.1.3.

Figure Figure 20 illustrates the effect of final-time measurements uT(x)u_T(x) at three noise levels (n=0,1%,5%)(n=0,1\%,5\%) on the predicted spatial source F(x)F(x), compared against the exact solution and numerical results from Hasanov [75], along with the evolution of the relative l2l^2 error Er(u^,u)E_r(\hat{u},u) during training. As noise increases, the measurements in panel (a) scatter around the noiseless data. In panel (b), the network-based prediction of F(x)F(x) remains smooth and accurate in the central region but shows increasing deviation near the boundaries. In contrast, the numerical result of Hasanov [75] exhibits noticeable oscillations near the center, despite accurate boundary predictions. Panel (c) illustrates that the relative error, Er(u^,u)\mathcal{E}_r(\hat{u},u), consistently falls below 10−210^{-2} across all noise levels, albeit with a gradual degradation in performance as the noise increases. This example highlights a key advantage of the PECANN-CAPU framework over traditional approaches to inverse problems–namely, its robustness in producing smooth and accurate predictions even under high observational noise.

Figure 20. Inverse problem: (a) temperature measurements at the final time with different noise levels, (b) predicted spatial source F(x)F(x) compared with the exact solution and numerical results from Hasanov [75], and (c) evolution of the relative l2l^2 error in the predicted temperature over training.

Conclusion

The PECANN framework [1] relies on constrained optimization via the augmented Lagrangian method (ALM) [29, 30] to enforce boundary and initial conditions on PDE solutions as equality constraints. We showed that conventional ALM with a single penalty parameter update strategy often fail for complex problems with multiple, diverse constraints. We addressed this limitation by introducing the “conditionally adaptive penalty update (CAPU)” strategy, which assigns unique penalty parameters to each constraint and updates them based on the magnitude of constraint violations. Theoretical analysis and empirical examples confirm CAPU’s ability to ensure boundedness of penalty parameters and convergence of Lagrange multipliers.

To further expand the PECANN–CAPU framework’s capacity for complex PDE problems in forward and inverse modeling scenarios, we incorporated constraint aggregation—a common technique in design and topology optimization—into its formulation. The original PECANN approach enforced constraints point-wise at every collocation point, a method that becomes computationally cumbersome and degrades performance as problem complexity increases, especially when using the proposed CAPU algorithm. Our experiments revealed that the Lagrange multipliers associated with a given constraint type follow a well-defined distribution and mean. Motivated by this observation, we aggregated constraints using a mean squared residual metric, which drastically reduced the number of Lagrange multipliers and penalty parameters from thousands to just a few. Equally important, we demonstrated through numerical examples and theoretical analysis that constraint aggregation enhances accuracy by eliminating the additional variance term in the objective function that arises from the point-wise enforcement of a large number of constraints.

Our revised formulation significantly improves the expressive power of the original PECANN framework. Standard MLPs often struggle to represent highly oscillatory PDE solutions with multi-scale features. Through case studies involving 1D Poisson’s and 2D Helmholtz equations of varying complexity, we demonstrated that PECANN–CAPU with constraint aggregation substantially enhances MLP capacity. Our approach achieves accuracy on par with physics-informed methods based on Kolmogorov–Arnold networks. An interesting implication of this demonstration is that pure regression is insufficient to evaluate network architecture in physics informed learning. Recognizing the inherent limitations of MLPs, we further enhanced expressive power by incorporating a single Fourier feature mapping with σ=1\sigma= 1. Another key strength of PECANN–CAPU lies in training efficiency. In our experiments, shallow and compact networks trained with L-BFGS consistently outperformed deeper networks trained with Adam.

To assess the PECANN–CAPU framework’s capability in predicting long-time PDE evolution, we applied it to the challenging benchmark of reversible transport of a passive scalar by a single vortex, a problem often used to evaluate interface-tracking methods in two-phase flow simulations. To enable this long-time integration, we introduced a time-windowing strategy within the constrained optimization formulation. In this approach, the terminal state of each time window is enforced as an initial constraint for the subsequent one. This strategy is uniquely enabled by the CAPU design, which inherently supports multiple independent constraints, each evolving with its own adaptively updated penalty parameter. Our results show that PECANN–CAPU achieves accuracy comparable to advanced schemes developed specifically for two-phase flows.

Overall, the success of PECANN–CAPU with constraint aggregation across a diverse set of PDE problems—in both forward and inverse scenarios—demonstrates that the introduced enhancements substantially expand the scope of the original PECANN formulation [1] for tackling challenging problems in computational science. More broadly, the augmented Lagrangian method, when integrated with CAPU and constraint aggregation, provides a principled and versatile mechanism for handling multiple, heterogeneous constraints, highlighting its promise beyond training physics-informed neural networks.

CRediT authorship contribution statement

Qifeng Hu: Conceptualization (Supporting), Data curation, Formal analysis (Lead), Investigation (Lead), Methodology (Equal), Software (Lead), Validation, Visualization, Writing - original draft (Lead), Writing – review & editing (Equal) Shamsulhaq Basir: Conceptualization (Supporting), Investigation (Supporting), Methodology (Equal), Software - original version, Writing – original draft (Supporting), Writing – review & editing (Supporting) Inanc Senocak: Conceptualization (Lead), Formal analysis (Supporting), Funding acquisition, Investigation (Supporting), Methodology (Equal), Project administration, Resources, Supervision, Writing – original draft (Supporting), Writing – review & editing (Equal).

Data availability

All the codes used to produce the results in this work will be publicly available at https://github.com/HiPerSimLab/PECANN/CAPU upon acceptance of the manuscript.

Declaration of computing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Declaration of generative AI and AI-assisted technologies in the writing

During the preparation of this work the author(s) used Microsoft Copilot and ChatGPT in order to assist with improving the clarity and quality of the English language. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the content of the publication.

Acknowledgments

This material is based upon work supported by the National Science Foundation under Grant No. 1953204 and in part by the University of Pittsburgh Center for Research Computing through the resources provided.

Evaluation of PECANN-CAPU and Direct PECANN-RMSProp

This appendix provides a comparison between the adaptive strategy from RMSProp and our proposed CAPU to better comprehend the merits of CAPU. Both strategies adopt the PECANN formulation with constraint aggregation.

We consider the 2D Helmholtz problem, Eq. (31), defined by the source terms in Eqs. (33) and (34) with L=4L = 4. Note that the more complex L=5L = 5 case is studied in § 4.4.1. A compact MLP with three hidden layers of 20 neurons each is trained for 80,000 epochs using the L-BFGS optimizer. The training data consist of 6400 randomly sampled residual points and 80 boundary points per edge.

Figure A.21 show the evolution of the objective J\mathcal{J}, boundary constraint CBC_{\mathrm{B}}, penalty parameter μB\mu_{\mathrm{B}}, and Lagrange multiplier λB\lambda_{\mathrm{B}} over the epochs, for PECANN-RMSProp and PECANN-CAPU, respectively, with the corresponding final predictions.

2D Helmholtz equation with a localized source term: evolution of the objective, boundary constraint, Lagrange multiplier, and penalty parameter, and predicted fields

Figure A.21. 2D Helmholtz equation with a localized source term (Eq. (33)) with L=4L=4: evolution of the objective, boundary constraint, Lagrange multiplier, and penalty parameter using (a) the PECANN-RMSProp algorithm, (b) the proposed PECANN-CAPU algorithm. Panels (c–d) show the corresponding predicted fields of these algorithms at their final epochs.

During optimization, as shown in Figs. A.21(a–b), the constraint violation CBC_B starts to increase after approximately 10210^2 epochs, as the objective J\mathcal{J} continues to decrease, indicating that the training is progressively driven towards an infeasible region. In this scenario, RMSProp in Fig. A.21(a) promotes a reduction of μB\mu_B and produces an almost linear, weakly responsive update of λB\lambda_B, as discussed in § 3. Lagrange multipliers show no sign of convergence even after 10510^5 epochs. We also observe premature escalation of μB\mu_B and unnecessary updates of λB\lambda_B during the early training stage, when optimization automatically continues to improve feasibility.

In contrast, PECANN-CAPU in Fig. A.21(b) exhibits stronger enforcement response under larger constraint violations by maintaining μB\mu_B above 10310^3, determined by the maximum operation in Eq. (10). It is accompanied by a pronounced increase in λB\lambda_B compared to other training stages. Consequently, CBC_B begins decreasing shortly after 10310^3 epochs, indicating that the optimization is driven back toward feasibility. As CBC_B continues to diminish,Lagrange multiplier, λB\lambda_B, converges to ≈2,000\approx2{,}000 after ≈5×104\approx5 \times10^4 epochs.

The final predictions shown in Figs. A.21(c–d) further highlight the difference between feasible and infeasible solutions produced by the two algorithms. The Gaussian-like source located at the geometric center of the domain induces outward wave propagation, leading to pronounced boundary violations around the 10310^3-epoch mark. As illustrated in Fig. A.21(c), PECANN-RMSProp just begins to recover toward feasibility, suggesting that it would need order of magnitude more epochs than PECANN-CAPU to potentially achieve a feasible solution in panel (d). This is evident from the evolution plots displayed on a log⁡10\log_{10} scale along the x-axis. Eventually, the CAPU algorithm achieves a relative l2l^2 norm of Er=(8.906±5.189)×10−2\mathcal{E}_r = (8.906 \pm5.189) \times10^{-2}, with the best-performing trial reaching 2.909×10−22.909 \times10^{-2}, confirming the efficacy of the CAPU strategy.

Sensitivity to Hyperparameter Selection

We use the exact solution of the Helmholtz equation in Eq. (32) as a reference case. To examine the sensitivity of PECANN-CAPU predictions to its hyperparameters, we adopt the configuration a1=1a_1 = 1 and a2=4a_2 = 4, following the setup in [53].

The sole external input to Algorithm 3, the penalty scaling factor ηB\eta_B for BC, is varied from 0.01 (as used in RMSProp) to 1. We also vary the internal hyperparameters: the smoothing coefficient ζ\zeta (from the default value of 0.99 to 0.9) and the conditional factor ωt\omega_t (from 0.999 to 0.99). We adopt a shallow MLP with two hidden layers (H=2H = 2) and 16 neurons per layer (W=16W = 16) and train it using the L-BFGS optimizer with a limit of 1800 epochs. The same number of collocation and evaluation points are used as described in § 4.4.

Table B.7 reports the relative l2l^2 norm of the mean error and its standard deviation computed over five trials for the PECANN-CAPU method. The accuracy of PECANN-CAPU predictions shows a modest sensitivity to ηB\eta_B, which controls the growth of the Lagrange multiplier. Setting ηB=1\eta_B = 1 reduces both the mean and standard deviation of the relative error compared to ηB=0.01\eta_B = 0.01 by approximately an order of magnitude. In contrast, variations in internal hyperparameters ωt\omega_t and ζ\zeta have only minor effects.

While the best CAPU performance is achieved with ηB=1\eta_B = 1 and a tuned value of ζ=0.9\zeta= 0.9, we caution against excessive tuning of internal hyperparameters. Reducing the smoothing coefficient to ζ=0.9\zeta= 0.9 enhances responsiveness by assigning greater weight to recent changes in the squared averages. While this adjustment yields favorable results in the present simple case, where constraint violations decrease steadily during training, such aggressive responsiveness may neglect gradual trends, leading to undesirable behavior in more complex scenarios, such as the cases discussed in Appendix § 11 and § 4.2. Therefore, we recommend retaining the default values for the internal hyperparameters, and selecting the external input—the penalty scaling factors for BC and IC—as 1.0 when using the L-BFGS optimizer, as analyzed in § 3.

To assess the influence of CAPU hyperparameters on convergence, three configurations, (ηB=0.01\eta_B = 0.01, ζ=d.v.\zeta= d.v.), (ηB=1.0\eta_B = 1.0, ζ=d.v.\zeta= d.v.), and (ηB=1.0\eta_B = 1.0, ζ=0.9\zeta= 0.9), are evaluated along with the MLP algorithm. Figure B.22 illustrates the evolution of the penalty parameter μB\mu_B, the corresponding Lagrange multiplier λB\lambda_B, and the relative l2l^2 error Er\mathcal{E}_r for their best-performing trials. As shown in Fig. B.22(a), all CAPU configurations exhibit a delayed increase in μB\mu_{\mathrm{B}} and λB\lambda_{\mathrm{B}} during the first 500 epochs compared to MPU, due to the convergence condition embedded in the CAPU algorithm. Beyond this stage, configurations with larger penalty scaling factor ηB\eta_{\mathrm{B}} begin to grow earlier, while the one with a smaller smoothing coefficient ζ\zeta rises more rapidly. As discussed previously, this behavior associated with smaller ζ\zeta can only occur when the optimization continues toward the enhancement of feasibility.

MethodηB\eta_{\mathrm{B}}ωt\omega_{\mathrm{t}}ζ\zetaEpochsRelative l2l^2 mean ±\pm stdbest trial
PECANN-CAPU0.01d.v.d.v.18003.40±1.28×10−23.40 \pm1.28 \times10^{-2}1.74×10−21.74 \times10^{-2}
0.99d.v.1.75±0.63×10−21.75 \pm0.63 \times10^{-2}7.56×10−37.56 \times10^{-3}
d.v.0.92.84±0.90×10−22.84 \pm0.90 \times10^{-2}1.29×10−21.29 \times10^{-2}
1d.v.d.v.8.20±4.20×10−38.20 \pm4.20 \times10^{-3}4.22×10−34.22 \times10^{-3}
0.99d.v.8.60±5.97×10−38.60 \pm5.97 \times10^{-3}4.18×10−34.18 \times10^{-3}
d.v.0.95.14±2.88×10−35.14 \pm2.88 \times10^{-3}2.68×10−32.68 \times10^{-3}

Table B.7. Helmholtz equation (Eq. (32) with a1=1a_1 = 1 and a2=4a_2 = 4): Relative l2l^2 error comparison of PECANN-CAPU using one input and two internal hyperparameters, where the default values (ωt=0.999\omega_{\mathrm{t}} = 0.999, ζ=0.99\zeta= 0.99) are denoted with d.v.. The network has two hidden layers with 16 neurons per layer (H=2H = 2, W=16W = 16).

Evolution of penalty parameters and relative $l^2$ norm

Figure B.22. Helmholtz equation (Eq. (32) with a1=1a_1 = 1 and a2=4a_2 = 4): (a) Evolution of penalty parameters (dotted lines) for BC and the corresponding Lagrange multipliers (solid lines) of the best-performing trials using three CAPU configuration and MPU, extended to 5000 epochs. and (b) the corresponding evolution of the relative l2l^2 norm.

Before the 1800th epoch, MPU shows a superior convergence performance in terms of Er\mathcal{E}_{r} in Fig. B.22(b). In fact, the ranking of configurations at this point closely follows the order of their Lagrange multipliers in Fig. B.22(a). MPU appears to converge fastest early on because the constraint decreases to a negligible level, leading λB\lambda_{\mathrm{B}} to rapidly converge to about 10310^3 as μB\mu_{\mathrm{B}} reaches its upper bound at a fixed rate. In contrast, CAPU resists such abrupt growth through its adaptive mechanism–beneficial for complex problems but limiting here–resulting in slower convergence. With further training, CAPU with ηB=1\eta_{\mathrm{B}} = 1 gradually increases both μB\mu_{\mathrm{B}} and λB\lambda_{\mathrm{B}} until convergence near λB≈103\lambda_{\mathrm{B}} \approx10^3, whereas the ηB=0.01\eta_{\mathrm{B}} = 0.01 configuration requires additional computation to reach convergence. Overall, CAPU achieves the lowest observed relative error.

MethodN. ParamsHHWWOptim.EpochsRelative l2l^2 mean ± stdRelative l2l^2 best trial
baseline PINN [53]337216L-BFGS1800-1.03×10−21.03 \times10^{-2}
cPIKAN+RBA [53]35028L-BFGS1800-3.81×10−33.81 \times10^{-3}
SA-PINN [22]7851450Adam & L-BFGS10410^{4} & 10410^{4}3.20±0.22×10−33.20 \pm0.22 \times10^{-3}-
PECANN-MPU337216L-BFGS50005.33±2.52×10−35.33 \pm2.52 \times10^{-3}1.06×10−31.06 \times10^{-3}
PECANN-CPU337216L-BFGS50006.18±4.08×10−36.18 \pm4.08 \times10^{-3}4.32×10−34.32 \times10^{-3}
PECANN-CAPU337216L-BFGS50004.88±6.65×10−34.88 \pm6.65 \times10^{-3}6.96×10−46.96 \times10^{-4}
baseline PINN337216L-BFGS50005.64±5.47×10−25.64 \pm5.47 \times10^{-2}1.39×10−21.39 \times10^{-2}
PINN with λB=1000\lambda_B = 1000337216L-BFGS50002.56±1.59×10−32.56 \pm1.59 \times10^{-3}5.87×10−45.87 \times10^{-4}

Table B.8. Helmholtz equation (Eq. (32) with $a_1 = 1$ and $a_2 = 4$): Relative l2l^2 comparison among several methods and the PECANN-CAPU algorithm. The SA-PINN [22] was trained with 10510^5 residual points and 400 boundary points. PINNs are re-evaluated over 5000 training epochs, both with and without the near-optimal Lagrange multiplier λB=1000\lambda_B = 1000 retrieved from the PECANN CAPU run.

Table B.8 presents a comparison of the three PECANN algorithms in this study with results reported in the literature [22, 53], as well as our additional tests of PINNs with and without the near-optimal Lagrange multiplier λB=1000\lambda_B = 1000 identified via the PECANN framework. SA-PINN exhibits highly consistent performance in the cited work; however, it relies on a considerably larger network configuration (H=4H = 4, W=50W = 50) and requires substantially more training epochs and collocation points. When using the PECANN framework, both the MPU and CPU algorithms show effectiveness with comparable accuracy, due to the relative simplicity of the PDE problem, The most effective PECANN-CAPU configuration (ηB=1\eta_B = 1, default internal hyperparameters) was subjected to extended training up to 5000 epochs. The resulting predictions achieved an error level comparable to the mean error reported for SA-PINN [22]. Across the five trials conducted, PECANN-CAPU yielded a minimum relative l2l^2 error of 6.96×10−46.96 \times10^{-4}.

In the re-evaluation, PINN achieves its best and most consistent performance when provided with a near-optimal Lagrange multiplier λB=1000\lambda_B = 1000, retrieved from the PECANN-CAPU framework. Use of this optimal value yields a significant improvement over the baseline PINN, highlighting the advantage of a constrained-optimization perspective in determining the optimal weight.

While MPU exhibits superior computational efficiency in this specific, simple case, it fails to yield acceptable results for more complex problems. We assert that to develop a robust, generalizable method applicable across a range of challenges, it is pragmatic to accept a minor efficiency trade-off in simpler cases to guarantee superior performance in complex scenarios. The computational overhead associated with training for 5000 epochs remains modest, as this is offset by the use of a compact network and a reduced number of collocation points relative to the cited works in Table B.8.

Theoretical underpinnings of the PECANN-CAPU algorithm

This section begins with a derivation of the augmented Lagrangian method (ALM) [29, 30], providing a theoretical foundation for the proposed PECANN-CAPU algorithm. First, we examine the benefits of introducing quadratic penalty terms in classical Lagrange multipliers method. We then leverage this analysis to discuss the CAPU algorithm’s resulting boundedness and convergence properties.

Starting from the method of Lagrange multipliers, the constrained optimization problem in Eq. (1) can be expressed as an unconstrained saddle-point problem

min⁡θmax⁡λˉJ(θ)+λˉTC(θ),(44)\min_{\theta}\max_{\bar{\lambda}} \mathcal{J}(\theta) + \bar{\lambda}^{T}\mathcal{C}(\theta), \tag*{(44)}

where λˉ\bar{\lambda} denotes the classical Lagrange multipliers, distinct from those utilized in the Augmented Lagrangian Method (ALM).

The maximization step in (C.1) with respect to each λˉi∈λˉ\bar{\lambda}_i \in\bar{\lambda} can be highly non-smooth, since the multiplier becomes unbounded (i.e., approaches ∞\infty) unless the constraint Ci=0C_i = 0 is exactly satisfied.

To mitigate the issue, a smoothing mechanism is introduced by adding a proximal term that penalizes deviations of λˉ\bar{\lambda} from their previous estimates λ\lambda. This modification yields the following regularized formulation:

min⁡θmax⁡λˉL(θ,λˉ;μ)=J(θ)+λˉTC(θ)−(12μ)T[(λˉ−λ)⊙(λˉ−λ)],(45)\min_{\theta}\max_{\bar{\lambda}} \mathcal{L}(\theta,\bar{\lambda};\mu) = \mathcal{J}(\theta) + \bar{\lambda}^{T}\mathcal{C}(\theta) - \left(\frac{1}{2\mu}\right)^{T}[(\bar{\lambda}-\lambda)\odot(\bar{\lambda}-\lambda)], \tag*{(45)}

where μ\mu are positive penalty parameters, controlling the strength of the regularization. Maximizing the regularized loss in Eq. (C.2) with respect to λˉ\bar{\lambda} is straightforward. By setting the gradient to zero, we obtain the first-order optimality condition:

C(θ)−1μ(λˉ−λ)=0,(46)\mathcal{C}(\theta)-\frac{1}{\mu}(\bar{\lambda}-\lambda)=0, \tag*{(46)}

which is equivalent to:

λˉ=λ+μ⊙C(θ).(47)\bar{\lambda}=\lambda+\mu\odot\mathcal{C}(\theta). \tag*{(47)}

Substituting Eq. (C.4) back into Eq. (C.2) recovers the standard augmented Lagrangian form:

min⁡θmax⁡λL(θ,λ;μ)=J(θ)+λTC(θ)+12μT[C(θ)⊙C(θ)].(48)\min_{\theta}\max_{\lambda}\mathcal{L}(\theta,\lambda;\mu)=\mathcal{J}(\theta)+\lambda^{T}\mathcal{C}(\theta)+\frac{1}{2}\mu^{T}[\mathcal{C}(\theta)\odot\mathcal{C}(\theta)]. \tag*{(48)}

This confirms that the prior estimates λ\lambda act as the Lagrange multipliers in the ALM, while the smoothing parameters μ\mu govern the regularization strength of the quadratic penalty terms applied to the constraints.

Analyzing the first-order optimality condition given in Eq. (C.3), we observe that constraint satisfaction is achieved when the updated multipliers λˉ\bar{\lambda} remain close to their previous values λ\lambda. Moreover, unlike standard penalty methods, the penalty parameters μ\mu do not need to grow indefinitely for the constraints C(θ)\mathcal{C}(\theta) to approach zero.

From Eq. (C.4), we can approximate the optimal value λˉ\bar{\lambda} from the current estimate, which justifies the principled update rule for λ\lambda, given by Eq. (4).

Overall, this analysis demonstrates the strength of ALM in effectively balancing feasibility and optimality, distinguishing it from pure Lagrange multiplier or pure penalty methods. ALM achieves this balance by combining penalty-based regularization with dual variable updates.

Mathematical relationship between point-wise and aggregation formulations

For clarity of exposition, consider a forward problem in which the PDE residual losses are denoted by JF(i)\mathcal{J}_{F}^{(i)}, i∈{1,2,…,NF}i \in\{1,2,\ldots,N_{F}\}, constrained by boundary losses by CB(j)\mathcal{C}_{B}^{(j)}, j∈{1,2,…,NB}j \in\{1,2,\ldots,N_{B}\}. The point-wise augmented Lagrangian loss is written as

Lpw=∑iJF(i)+∑jλjCB(j)+∑j12μjCB(j)2.(49)\mathcal{L}_{pw} = \sum_{i} \mathcal{J}_{F}^{(i)} + \sum_{j} \lambda_{j}\mathcal{C}_{B}^{(j)} + \sum_{j} \frac{1}{2}\mu_{j}\mathcal{C}_{B}^{(j)2}. \tag*{(49)}

To explore the connection with the constraint-aggregation formulation, assume that all Lagrange multipliers take a common value λj=cλ\lambda_{j}=c_{\lambda} and all penalty parameters take a common value μj=cμ\mu_{j}=c_{\mu}. Introducing the sample means J‾F=1NF∑iJF(i)\overline{\mathcal{J}}_{F}=\frac{1}{N_{F}}\sum_{i}\mathcal{J}_{F}^{(i)}, C‾B=1NB∑jCB(j)\overline{\mathcal{C}}_{B}=\frac{1}{N_{B}}\sum_{j}\mathcal{C}_{B}^{(j)}, and using the standard identity relating the mean of squares and the variance,

1NB∑jCB(j)2=C‾B2+Var⁡(CB).(50)\frac{1}{N_{B}}\sum_{j}\mathcal{C}_{B}^{(j)2}=\overline{\mathcal{C}}_{B}^{2}+\operatorname{Var}(\mathcal{C}_{B}). \tag*{(50)}

The expression for Lpw\mathcal{L}_{pw} can be rewritten as

Lpw=NF[J‾F+cλNBNFC‾B+12cμNBNFC‾B2+12cμNBNFVar⁡(CB)].(51)\mathcal{L}_{pw}=N_{F}\left[\overline{\mathcal{J}}_{F}+\frac{c_{\lambda}N_{B}}{N_{F}}\overline{\mathcal{C}}_{B}+\frac{1}{2}\frac{c_{\mu}N_{B}}{N_{F}}\overline{\mathcal{C}}_{B}^{2}+\frac{1}{2}\frac{c_{\mu}N_{B}}{N_{F}}\operatorname{Var}(\mathcal{C}_{B})\right]. \tag*{(51)}

Considering the constraint-aggregation augmented loss

Lavg=J‾F+λBC‾B+12μBC‾B2,(52)\mathcal{L}_{avg}=\overline{\mathcal{J}}_{F}+\lambda_{B}\overline{\mathcal{C}}_{B}+\frac{1}{2}\mu_{B}\overline{\mathcal{C}}_{B}^{2}, \tag*{(52)}

we define the Lagrange multiplier and penalty parameter as

λB=cλNBNF,μB=cμNBNF.(53)\lambda_{B}=\frac{c_{\lambda}N_{B}}{N_{F}},\qquad\mu_{B}=\frac{c_{\mu}N_{B}}{N_{F}}. \tag*{(53)}

Then the point-wise augmented loss satisfies

Lpw=NF[Lavg+12μBVar⁡(CB)].(54)\mathcal{L}_{pw}=N_{F}\left[\mathcal{L}_{avg}+\frac{1}{2}\mu_{B}\operatorname{Var}(\mathcal{C}_{B})\right]. \tag*{(54)}

This final expression shows that, under the assumption of constant values for λj\lambda_{j} and μj\mu_{j}, the point-wise formulation is precisely NFN_{F} times its constraint-aggregation counterpart, added by an extra penalty term acting on the variance of the boundary constraint residuals.

Apart from the scaling effects induced by the number of sampling points and the relationship in Eq. (C.10), the variance penalty term may introduce undesirable outcomes. As shown in Figure 3(c,Figure 3f), the formulation with constraint aggregation achieves accuracy levels that are an order of magnitude superior to (or lower error than) point-wise constraints.

Although this term can help suppress fluctuations in constraint evaluations in certain cases, it may equally degrade convergence toward feasibility. We have encountered this issue specifically in inverse problems where point-wise PDE residuals are incorporated as constraints, as these residuals typically exhibit larger intrinsic variability throughout the optimization process.

Boundedness and Convergence of the CAPU Algorithm

We demonstrate that the CAPU algorithm ensures parameter boundedness and multiplier convergence as constraint violations diminish. This guarantees numerical stability and prevents unbounded parameter growth during the final optimization stages.

Lemma 1 (Boundedness of CAPU penalty parameters). Let μi\mu_i and λi\lambda_i be the penalty parameter and Lagrange multiplier corresponding to the iith constraint CiC_i in the PECANN-CAPU algorithm. If the moving average of the squared constraint, denoted by νˉi\bar{\nu}_i, satisfies νˉi≪ϵ\bar{\nu}_i \ll\epsilon, where ϵ\epsilon is a small positive constant introduced for numerical stability in the denominator of the update rule (10), then μi\mu_i asymptotically approaches a finite upper bound given by

μi→ηiϵ.(55)\mu_i \to\frac{\eta_i}{\sqrt{\epsilon}}. \tag*{(55)}

Consequently, when the constraint is sufficiently stabilized below ϵ\sqrt{\epsilon}, the moving average νˉi\bar{\nu}_i remains on the order of ϵ\epsilon, ensuring that μi\mu_i remains bounded.

Proof. From the CAPU update rule (10), the denominator involves νˉi+ϵ\sqrt{\bar{\nu}_i}+\epsilon. When νˉi≪ϵ\bar{\nu}_i \ll\epsilon, this term becomes approximately ϵ\sqrt{\epsilon}, leading to

μi≈ηiϵ.\mu_i \approx\frac{\eta_i}{\sqrt{\epsilon}}.

Thus, the penalty parameter μi\mu_i converges to a finite value rather than diverging as required for pure quadratic penalty methods. In this regime, the constraint remains stabilized, and νˉi\bar{\nu}_i stays near ϵ\epsilon, preventing unbounded growth of μi\mu_i. Hence, μi\mu_i is bounded as stated, which is also consistently observed in many experiments, for example, shown in Figure 17.

Theorem 1 (Convergence of Lagrange multipliers in CAPU). Based on Lemma 1, suppose that the iith constraint CiC_i is sufficiently stabilized such that Ci≪ϵC_i \ll\sqrt{\epsilon}. Then, under the CAPU update rule, the corresponding Lagrange multiplier λi\lambda_i converges to a finite value as the optimization proceeds.

Proof. When the optimization process reaches a stabilized regime, the conditional check in Algorithm 3 is satisfied at each epoch, and the updates of λi\lambda_i proceed with small increments. According to the update rule for the iith Lagrange multiplier,

λi(e)−λi(e−1)=μiCi,\lambda_i^{(e)}-\lambda_i^{(e-1)}=\mu_i C_i,

and by Lemma 1, μi\mu_i remains bounded by μi<ηiϵ\mu_i < \frac{\eta_i}{\sqrt{\epsilon}}. Given that the constraint magnitude satisfies Ci≪ϵC_i \ll\sqrt{\epsilon}, we have

∣λi(e)−λi(e−1)∣<ηiϵCi<ηi.(56)\left|\lambda_i^{(e)}-\lambda_i^{(e-1)}\right| < \frac{\eta_i}{\sqrt{\epsilon}}C_i < \eta_i. \tag*{(56)}

Therefore, the Lagrange multiplier update becomes progressively smaller as the constraint stabilizes, implying that λi\lambda_i converges asymptotically to a finite limit.

References

  1. [1]S. Basir, I. Senocak, Physics and equality constrained artificial neural networks: Application to forward and inverse problems with multi-fidelity data fusion, J. Comput. Phys. 463 (2022) 111301.DOI
  2. [2]M. W. M. G. Dissanayake, N. Phan-Thien, Neural-network-based approximations for solving partial differential equations, Commun. Numer. Meth. Eng. 10 (1994) 195–201.DOI
  3. [3]B. P. van Milligen, V. Tribaldos, J. A. Jiménez, Neural network differential equation and plasma equilibrium solver, Phys. Rev. Lett. 75 (1995) 3594–3597.pubmed.ncbi.nlm.nih.gov/10059679
  4. [4]K. Hornik, M. Stinchcombe, H. White, Multilayer feedforward networks are universal approximators, Neural Netw. 2 (1989) 359–366.DOI
  5. [5]D. E. Rumelhart, G. E. Hinton, R. J. Williams, Learning representations by back-propagating errors, Nature 323 (1986) 533–536.
  6. [6]G. E. Hinton, S. Osindero, Y.-W. Teh, A fast learning algorithm for deep belief nets, Neural Comput. 18 (2006) 1527–1554.pubmed.ncbi.nlm.nih.gov/16764513
  7. [7]V. Nair, G. E. Hinton, Rectified linear units improve restricted boltzmann machines, in: Proc. Int. Conf. Mach. Learn., 27, 2010, p. 807–814.
  8. [8]S. Ioffe, C. Szegedy, Batch normalization: accelerating deep network training by reducing internal covariate shift, in: Proc. Int. Conf. Mach. Learn. volume 37, 2015, p. 448–456.
  9. [9]N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: a simple way to prevent neural networks from overfitting, J. Mach. Learn. Res. 15 (2014) 1929–1958.
  10. [10]D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, in: Y. Bengio, Y. LeCun (Eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL: http://arxiv.org/abs/1412.6980.
  11. [11]W. E, B. Yu, The deep Ritz method: A deep learning-based numerical algorithm for solving variational problems, Commun. Math. Stat. 6 (2018) 1–12.arxiv.org/abs/1710.00211
  12. [12]J. Han, A. Jentzen, W. E, Solving high-dimensional partial differential equations using deep learning, Proc. Natl. Acad. Sci. U.S.A 115 (2018) 8505–8510.DOI
  13. [13]J. Sirignano, K. Spiliopoulos, DGM: A deep learning algorithm for solving partial differential equations, J. Comput. Phys. 375 (2018) 1339–1364.DOI
  14. [14]Y. Zhu, N. Zabaras, P.-S. Koutsourelakis, P. Perdikaris, Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data, J. Comput. Phys. 394 (2019) 56–81.DOI
  15. [15]M. Raissi, P. Perdikaris, G. E. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys. 378 (2019) 686–707.DOI
  16. [16]A. D. Jagtap, K. Kawaguchi, G. E. Karniadakis, Adaptive activation functions accelerate convergence in deep and physics-informed neural networks, J. Comput. Phys. 404 (2020) 109136.DOI
  17. [17]C. Wu, M. Zhu, Q. Tan, Y. Kartha, L. Lu, A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks, Comput. Methods Appl. Mech. Eng. 403 (2023) 115671.DOI
  18. [18]S. Wang, Y. Teng, P. Perdikaris, Understanding and mitigating gradient flow pathologies in physics-informed neural networks, SIAM J. Sci. Comput. 43 (2021) A3055–A3081.
  19. [19]S. Basir, I. Senocak, Critical investigation of failure modes in physics-informed neural networks, in: AIAA SCITECH 2022 Forum, American Institute of Aeronautics and Astronautics, 2022. doi:10.2514/6.2022-2353.DOI
  20. [20]R. van der Meer, C. W. Oosterlee, A. Borovykh, Optimally weighted loss functions for solving pdes with neural networks, J. Comput. Appl. Math. 405 (2022) 113887.
  21. [21]D. Liu, Y. Wang, A dual-dimer method for training physics-constrained neural networks with minimax architecture, Neural Networks 136 (2021) 112–125.DOI
  22. [22]L. D. McClenny, U. M. Braga-Neto, Self-adaptive physics-informed neural networks, J. Comput. Phys. 474 (2023) 111722.DOI
  23. [23]S. J. Anagnostopoulos, J. D. Toscano, N. Stergiopulos, G. E. Karniadakis, Residual-based attention in physics-informed neural networks, Comput. Methods Appl. Mech. Eng. 421 (2024) 116805.DOI
  24. [24]J. D. Toscano, V. Oommen, A. J. Varghese, Z. Zou, N. Ahmadi Daryakenari, C. Wu, G. E. Karniadakis, From pinns to pikans: recent advances in physics-informed machine learning, Mach. Learn. Comput. Sci. Eng. 1 (2025).DOI
  25. [25]J. R. Martins, A. Ning, Engineering design optimization, Cambridge University Press, 2021.DOI
  26. [26]I. E. Lagaris, A. Likas, D. I. Fotiadis, Artificial neural networks for solving ordinary and partial differential equations, IEEE Trans. Neural Netw. 9 (1998) 987–1000.DOI
  27. [27]J. Berg, K. Nyström, A unified deep artificial neural network approach to partial differential equations in complex geometries, Neurocomputing 317 (2018) 28–41.DOI
  28. [28]L. Lu, R. Pestourie, W. Yao, Z. Wang, F. Verdugo, S. G. Johnson, Physics-informed neural networks with hard constraints for inverse design, SIAM J. Sci. Comput. 43 (2021) B1105–B1132.
  29. [29]M. R. Hestenes, Multiplier and gradient methods, J. Optim. Theory Appl. 4 (1969) 303–320.DOI
  30. [30]M. J. Powell, A method for nonlinear constraints in minimization problems, in: R. Fletcher (Ed.), Optimization: Symposium of the Institute of Mathematics and Its Applications, University of Keele, England, 1968, Academic Press, London,New York, 1969, pp. 283–298.
  31. [31]E. G. Birgin, J. M. Martínez, Practical Augmented Lagrangian Methods for Constrained Optimization, Society for Industrial and Applied Mathematics, 2014. doi:10.1137/1.9781611973365.DOI
  32. [32]J. Nocedal, S. Wright, Numerical optimization, Springer Science & Business Media, 2006.
  33. [33]S. Basir, I. Senocak, An adaptive augmented lagrangian method for training physics and equality constrained artificial neural networks, 2023. arXiv:2306.04904.arxiv.org/abs/2306.04904
  34. [34]P. Lan, J. Su, S. Zhang, Surrogate modeling for unsaturated infiltration via the physics and equality-constrained artificial neural networks, J. Rock Mech. Geotech. Eng. 16 (2024) 2282–2295.DOI
  35. [35]Q. Hu, S. Basir, I. Senocak, Non-overlapping, Schwarz-type domain decomposition method for physics and equality constrained artificial neural networks, Comput. Methods Appl. Mech. Eng. 436 (2025) 117706.
  36. [36]J.-J. Su, G.-L. Tao, R. Li, S. Zhang, Physics-driven deep learning methods and numerically intractable “bad” Jaulent–Miodек equation, Chaos 35 (2025).DOI
  37. [37]E. Olson, S. Won, X. Tu, Scalable two-level domain decomposition methods for the physics and equality constrained artificial neural networks, Comput. Method. Appl. Mech. Eng. 448 (2026) 118437.DOI
  38. [38]Z. Liu, Y. Wang, S. Vaidya, F. Ruehle, J. Halverson, M. Soljacić, T. Y. Hou, M. Tegmark, KAN: Kolmogorov–Arnold Networks, in: The Thirteenth International Conference on Learning Representations, 2025. URL: https://openreview.net/forum?id=Ozo7qJ5vZi.DOI
  39. [39]M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. T. Barron, R. Ng, Fourier features let networks learn high frequency functions in low dimensional domains, in: Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Curran Associates Inc., Red Hook, NY, USA, 2020.arxiv.org/abs/2006.10739
  40. [40]Y. Wang, J. Sun, J. Bai, C. Anitescu, M. S. Eshaghi, X. Zhuang, T. Rabczuk, Y. Liu, Kolmogorov–arnold-informed neural network: A physics-informed deep learning framework for solving forward and inverse problems based on Kolmogorov–Arnold networks, Comput. Methods Appl. Mech. Eng. 433 (2025) 117518.DOI
  41. [41]S. Wang, H. Wang, P. Perdikaris, On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks, Comput. Methods Appl. Mech. Eng. 384 (2021) 113938.DOI
  42. [42]M. Bierlaire, Optimization : Principles and Algorithms, EPFL Press, Lausanne, 2018.
  43. [43]S. Wright, J. Nocedal, et al., Numerical optimization, Springer Science 35 (1999) 7.
  44. [44]A. Dener, M. A. Miller, R. M. Churchill, T. Munson, C.-S. Chang, Training neural networks under physical constraints using a stochastic augmented lagrangian approach, arXiv preprint arXiv:2009.07330 (2020).arxiv.org/abs/2009.07330
  45. [45]H. Geoffrey, S. Nitish, S. Kevi, Lecture slides for csc321: Introduction to neural networks and machine learning, 2014. URL: https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf, accessed on March 23, 2023.
  46. [46]S. J. Reddi, S. Kale, S. Kumar, On the convergence of adam and beyond, in: Proceedings of the International Conference on Learning Representations, 2018. Available online: https://openreview.net/pdf?id=ryQu7f-RZ.arxiv.org/abs/1904.09237
  47. [47]N. Echebest, M. D. Sánchez, M. L. Schuvert, Convergence results of an augmented lagrangian method using the exponential penalty function, Journal of Optimization Theory and Applications 168 (2016) 92–108.DOI
  48. [48]R. Andreani, K. R. Couto, O. P. Ferreira, G. Haeser, Constraint qualifications and strong global convergence properties of an augmented lagrangian method on riemannian manifolds, SIAM Journal on Optimization 34 (2024) 1799–1825.arxiv.org/abs/2306.14345
  49. [49]J. Nocedal, Updating quasi-Newton matrices with limited storage, Math. Comput. 35 (1980) 773–782.DOI
  50. [50]N. M. K. Poon, J. R. R. A. Martins, An adaptive approach to constraint aggregation using adjoint sensitivity analysis, Struct. Multidiscip. Optim. 34 (2006) 61–73.DOI
  51. [51]G. J. Kennedy, J. E. Hicken, Improved constraint-aggregation methods, Comput. Meth. Appl. Mech. Engr. 289 (2015) 332–354.DOI
  52. [52]K.-S. Zhang, Z.-H. Han, Z.-J. Gao, Y. Wang, Constraint aggregation for large number of constraints in wing surrogate-based optimization, Struct. Multidiscip. Optim. 59 (2018) 421–438.DOI
  53. [53]K. Shukla, J. D. Toscano, Z. Wang, Z. Zou, G. E. Karniadakis, A comprehensive and fair comparison between mlp and kan representations for differential equations and operator networks, Comput. Methods Appl. Mech. Eng. 431 (2024) 117290.
  54. [54]X. Glorot, Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, in: Y. W. Teh, M. Titterington (Eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, PMLR, Chia Laguna Resort, Sardinia, Italy, 2010, pp. 249–256.
  55. [55]A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köp̈f, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, PyTorch: An Imperative Style, High-Performance Deep Learning Library, Curran Associates Inc., Red Hook, NY, USA, 2019.arxiv.org/abs/1912.01703
  56. [56]J. Baker-Jarvis, R. Inguva, Heat conduction in layered, composite materials, J. Appl. Phys. 57 (1985) 1569–1573.
  57. [57]J. Abbasi, A. D. Jagtap, B. Moseley, A. Hiorth, P. Østebø Andersen, Challenges and advancements in modeling shock fronts with physics-informed neural networks: A review and benchmarking study, Neurocomputing 657 (2025) 131440.
  58. [58]L. Liu, S. Liu, H. Xie, F. Xiong, T. Yu, M. Xiao, L. Liu, H. Yong, Discontinuity computing using physics-informed neural networks, J. Sci. Comput 98 (2023).DOI
  59. [59]E. J. R. Coutinho, M. Dall’Aqua, L. McClenny, M. Zhong, U. Braga-Neto, E. Gildin, Physics-informed neural networks with adaptive localized artificial viscosity, Journal of Computational Physics 489 (2023) 112265.DOI
  60. [60]E. F. Toro, Riemann Solvers and Numerical Methods for Fluid Dynamics: A Practical Introduction, third ed., Springer-Verlag Berlin Heidelberg, 2009.
  61. [61]Z. Hu, A. D. Jagtap, G. E. Karniadakis, K. Kawaguchi, Augmented physics-informed neural networks (APINNs): A gating network-based soft domain decomposition methodology, Eng. Appl. Artif. Intell. 126 (2023) 107183.DOI
  62. [62]V. Dolean, A. Heinlein, S. Mishra, B. Moseley, Multilevel domain decomposition-based architectures for physics-informed neural networks, Comput. Methods Appl. Mech. Eng. 429 (2024) 117116.DOI
  63. [63]V. Sitzmann, J. N. P. Martel, A. W. Bergman, D. B. Lindell, G. Wetzstein, Implicit neural representations with periodic activation functions, 2020. URL: https://arxiv.org/abs/2006.09661. arXiv:2006.09661.
  64. [64]S. Wang, S. Sankaran, P. Perdikaris, Respecting causality for training physics-informed neural networks, Comput. Methods Appl. Mech. Eng. 421 (2024) 116813.DOI
  65. [65]R. J. LeVeque, High-resolution conservative algorithms for advection in incompressible flow, SIAM J. Numer. Anal. 33 (1996) 627–665.DOI
  66. [66]W. Rider, D. Kothe, Reconstructing volume tracking, J. Comput. Phys. 141 (1998) 112–152.DOI
  67. [67]D. Harvie, D. Fletcher, A new volume of fluid advection algorithm: the stream scheme, J. Comput. Phys. 162 (2000) 1–32.DOI
  68. [68]D. Enright, R. Fedkiw, J. Ferziger, I. Mitchell, A hybrid particle level set method for improved interface capturing, J. Comput. Phys. 183 (2002) 83–116.DOI
  69. [69]S. E. Hieber, P. Koumoutsakos, A lagrangian particle level set method, J. Comput. Phys. 210 (2005) 342–367.DOI
  70. [70]H. T. Ahn, M. Shashkov, Adaptive moment-of-fluid method, J. Comput. Phys. 228 (2009) 2792–2821.
  71. [71]B. Xie, P. Jin, Y. Du, S. Liao, A consistent and balanced-force model for incompressible multiphase flows on polyhedral unstructured grids, Int. J. Multiphase Flow 122 (2020) 103125.DOI
  72. [72]J. Lopez, J. Hernandez, P. Gomez, F. Faura, A volume of fluid method based on multidimensional advection and spline interface reconstruction, J. Comput. Phys. 195 (2004) 718–742.DOI
  73. [73]N. Singh, B. Premachandran, A coupled level set and volume of fluid method on unstructured grids for the direct numerical simulations of two-phase flows including phase change, Int. J. Heat Mass Trans. 122 (2018) 182–203.DOI
  74. [74]B. Xie, F. Xiao, Toward efficient and accurate interface capturing on arbitrary hybrid unstructured grids: the thinc method with quadratic surface representation and gaussian quadrature, J. Comput. Phys. 349 (2017) 415–440.
  75. [75]A. Hasanov, Identification of spacewise and time dependent source terms in 1d heat conduction equation from temperature measurement at a final time, Int. J. Heat Mass Trans. 55 (2012) 2069–2080.DOI

Paper details

vlnc:W_spxvgx2u
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents