betaFeedback

WorksComputer Science

Robust Matrix-Free Newton-Krylov Solvers via Automatic Differentiation

Abstract

Jacobian-Free Newton-Krylov (JFNK) methods avoid forming the full Jacobian, but still require Jacobian-vector products, i.e., Gateaux derivatives of the nonlinear residual along Krylov directions. In standard Finite Differences (FD) formulations, these products are obtained by perturbing the Newton state and differencing residuals, making the linearization sensitive to round-off error and floating-point precision. This work evaluates the global impact of forward-mode Automatic Differentiation (AD) as a replacement for FD Jacobian-vector product in finite-precision JFNK solvers. The comparison keeps the discretization, Newton iteration, line search, Krylov methods, tolerances, and CPU/GPU backend fixed, only varying linearization strategy. Benchmarks include Burgers dynamics, Su-Olson radiation diffusion, reaction-diffusion, and nonlinear time-harmonic Maxwell equations, each evaluated in different nonlinear regimes. By preventing degradation of the Krylov operator, AD accelerates computation by 2-3 orders of magnitude across both CPU and GPU architectures. More importantly, it drastically improves global solver robustness, achieving a minimum completion rate of 95%, compared to just 42% for FD. Ultimately, accurate Gateaux derivatives unify performance and accuracy in JFNK methods, making AD the optimal choice for stiff nonlinear and reduced-precision environments.

Introduction

The solution of nonlinear problems is central to numerous scientific and engineering applications, making robust nonlinear solvers indispensable in computational physics. In particular, Jacobian-Free Newton-Krylov (JFNK) algorithms are extensively employed for large-scale systems governed by Partial Differential Equations (PDEs). These methods are advantageous because they couple the rapid convergence characteristics of Newton’s method with the memory efficiency of matrix-free Krylov subspace linear solvers [1, 2]. Additionally, JFNK provides a highly efficient computational framework capable of addressing both implicit time-dependent initial boundary value problems (IBVPs) and boundary value problems (BVPs). Instead of forming and storing the full Jacobian matrix, the Newton correction is computed using Krylov subspace methods such as GMRES [3], Conjugate Gradient (CG) [4], or BiCGSTAB [5]. These methods require only the action of the Jacobian on vectors. This matrix-free formulation avoids explicit Jacobian assembly, but it does not avoid linearization. At each Newton iteration, the Krylov solver requires repeated Jacobian-vector products, which correspond to Gateaux derivatives of the nonlinear residual along Krylov directions. The quality of these products directly affects the Krylov subspace, the computed Newton correction, and therefore the convergence of the full nonlinear solve. For this reason, the Jacobian-vector product is a critical numerical component of the JFNK method which links accuracy to performance.

In standard JFNK implementations, Jacobian-Vector Products (JVPs) are commonly approximated by Finite Differences (FD) of the nonlinear residual [1, 6]. This requires perturbing the current Newton state along a Krylov direction and differencing two residual evaluations. The perturbation size must be selected by the user or by a heuristic rule. However, its choice might have large implications. If the perturbation is too large, the derivative is biased by truncation error. If it is too small, cancellation and round-off might dominate. This sensitivity becomes especially important when using reduced precision, e.g. single-precision (FP32) instead of double-precision (FP64). In these cases, the admissible range of perturbation sizes narrows and inaccurate JVP can lead to Krylov stagnation, poor Newton corrections, and solver failure. The broader question of when derivatives should be trusted over finite differences has also been studied in the context of noisy computational functions [7].

The increased use of reduced-precision arithmetic, spurred by Machine Learning (ML) workloads and accelerators with FP32 and half-precision (FP16 units), makes this issue increasingly important. In several areas of computational science, including computational fluid dynamics, weather and climate modelling, and accelerator-based simulation, single- and mixed-precision are attractive because they reduce memory traffic and can improve throughput on modern hardware [8, 9, 10, 11, 12]. However, the benefit of reduced precision is not automatic for nonlinear solvers. The inner Krylov iteration can be affected by round-off error, loss of orthogonality in the Arnoldi or Lanczos process, and inaccurate FD Jacobian-vector products [13]. Consequently, the linearization strategy can determine whether a JFNK solver actually benefits from reduced precision.

Automatic Differentiation (AD) [14] provides an alternative to FD for computing derivatives of numerical algorithms [15, 16, 17]. AD has been used to compute sparse Jacobians and derivative matrices in scientific computing [18, 19], and modern tools now support increasingly complex software environments and parallel programming models [20]. AD has also been incorporated into scientific computing frameworks and multiphysics environments, including PETSc-based Newton–Krylov workflows, MOOSE, and Trilinos toolchains [21, 22, 23]. More recently, differentiable programming has renewed interest in differentiating numerical solvers for differential equations, especially in inverse problems, data assimilation, and physics-informed learning [24, 25, 26]. Although AD offers advantages, JFNK solvers have been traditionally implemented with FD to avoid coding complexity and performance overheads associated with it. In this work, we use AD as a numerical device within a matrix-free Newton–Krylov method. Forward-mode AD computes the Jacobian-vector product required by the Krylov solver by directly differentiating the implemented discrete residual.

Replacing FD JVPs with AD alters the linearized operator presented to the Krylov solver. JFNK tightly couples multiple iterative layers, including the outer continuation loop, Newton iterations, Krylov subspace construction, and line search. Because of this nested integration, errors in the JVP can easily cascade. Therefore, the impact of AD must be evaluated across the complete nonlinear solver, not just as an isolated derivative.

The objective of this paper is to provide a complete evaluation of forward-mode AD as a replacement for FD JVP in JFNK solvers for nonlinear PDEs. We measure how this choice affects Krylov convergence, Newton convergence, execution time, and solver failure rates, at the same time comparing any possible additional overhead introduced by AD. In this work, robustness refers to the ability of the full nonlinear solver to converge reliably across problem classes, parameter regimes, floating-point precisions, and Krylov methods without stagnation or problem-specific tuning. Our benchmark suite covers both time-dependent IBVPs and frequency-domain BVPs. It includes viscous Burgers dynamics, Su-Olson radiation diffusion, reaction-diffusion, and nonlinear time-harmonic Maxwell equations in Kerr media. These problems span nonsymmetric and symmetric systems, in parameter regimes with different degrees of stiffness. We evaluate FP64 and FP32 arithmetic, CPU and GPU execution, and multiple Krylov solvers. The goal is to identify how Jacobian-vector product construction affects the reliability and performance of practical matrix-free nonlinear PDE solvers. We demonstrate that AD improves robustness primarily by preventing finite-precision degradation of the Krylov operator. The largest differences appear in FP32, where FD JVP can cause severe Krylov stagnation. In a representative FP32 Burgers case, FD approximations inflated the iteration count to near the GMRES cap, requiring hundreds of thousands of total Krylov iterations to reach convergence. By comparison, AD maintained a tightly bounded Krylov subspace, reducing the iterations by 491×491 \times and accelerating the full nonlinear solve by 169×169 \times, recovering the same converged solution.

Preliminaries

We consider a continuous nonlinear PDE problem generally written as

G(x1,…,xn,u,∂u∂x1,…,∂u∂xn,∂2u∂xi∂xj,…)=0,(1)\mathbf{G}\left(x_1,\ldots,x_n,u,\frac{\partial u}{\partial x_1},\ldots,\frac{\partial u}{\partial x_n},\frac{\partial^2u}{\partial x_i\partial x_j},\ldots\right)=0, \tag*{(1)}

where u=u(x1,…,xn)u=u(x_1,\ldots,x_n) is the state variable and G\mathbf{G} the nonlinear differential operator. After spatial discretization, and time discretization when applicable, the problem is written as a nonlinear residual equation

F(xk,x∗)=0,(2)\mathbf{F}(\mathbf{x}^k,\mathbf{x}^*)=0, \tag*{(2)}

where xk\mathbf{x}^k is the current Newton iterate and x∗\mathbf{x}^* is the previously converged state or initial guess used to define the current nonlinear solve. At each Newton iteration, the residual is linearized around xk\mathbf{x}^k. Omitting the explicit dependence on x∗\mathbf{x}^*, the first-order expansion gives

F(xk+1)≈F(xk)+J(xk)(xk+1−xk),(3)\mathbf{F}(\mathbf{x}^{k+1})\approx\mathbf{F}(\mathbf{x}^k)+\mathcal{J}(\mathbf{x}^k)(\mathbf{x}^{k+1}-\mathbf{x}^k), \tag*{(3)}

where J=F′\mathcal{J}=\mathbf{F}' is the Jacobian of the discrete residual. Defining the Newton correction as δxk=xk+1−xk\delta\mathbf{x}^{k}=\mathbf{x}^{k+1}-\mathbf{x}^{k}, the Newton step is obtained from

J(xk)δxk=−F(xk).(4)\mathcal{J}(\mathbf{x}^{k})\delta\mathbf{x}^{k}=-\mathbf{F}(\mathbf{x}^{k}). \tag*{(4)}

JFNK methods solve Eq. (4) with a Krylov method without explicitly assembling J\mathcal{J} [1, 2]. The nonlinear iteration is stopped when

∥F(xk,x∗)∥∥F(x∗,x∗)∥<tol⁡Newton.(5)\frac{\left\lVert\mathbf{F}(\mathbf{x}^{k},\mathbf{x}^{*})\right\rVert}{\left\lVert\mathbf{F}(\mathbf{x}^{*},\mathbf{x}^{*})\right\rVert}<\operatorname{tol}_{\mathrm{Newton}}. \tag*{(5)}

A backtracking line search can be applied after the Krylov solve by replacing xk+1=xk+δxk\mathbf{x}^{k+1}=\mathbf{x}^{k}+\delta\mathbf{x}^{k} with xk+1=xk+αδxk\mathbf{x}^{k+1}=\mathbf{x}^{k}+\alpha\delta\mathbf{x}^{k}, where α\alpha is reduced until sufficient nonlinear residual decrease is obtained [6].

The main operation in a matrix-free Newton-Krylov method is the Jacobian-vector product. At Krylov iteration mm, the Krylov solver provides a direction vm\mathbf{v}_{m} and requires J(xk)vm\mathcal{J}(\mathbf{x}^{k})\mathbf{v}_{m}, i.e., the Gateaux derivative of the discrete residual in the direction vm\mathbf{v}_{m}:

J(xk)vm=ddϵF(xk+ϵvm)∣ϵ=0.(6)\mathcal{J}(\mathbf{x}^{k})\mathbf{v}_{m}=\left.\frac{\mathrm{d}}{\mathrm{d}\epsilon}\mathbf{F}(\mathbf{x}^{k}+\epsilon\mathbf{v}_{m})\right|_{\epsilon=0}. \tag*{(6)}

This product determines the linear operator seen by the Krylov solver. Consequently, errors in the Jacobian-vector product affect the Krylov subspace, the computed Newton correction, and the convergence of the full nonlinear solve. In standard JFNK implementations, Eq. (6) is approximated by finite differences,

J(xk)vm≈F(xk+ϵvm)−F(xk)ϵ.(7)\mathcal{J}(\mathbf{x}^{k})\mathbf{v}_{m}\approx\frac{\mathbf{F}(\mathbf{x}^{k}+\epsilon\mathbf{v}_{m})-\mathbf{F}(\mathbf{x}^{k})}{\epsilon}. \tag*{(7)}

The perturbation size ϵ\epsilon is usually chosen using a machine-precision-based rule, for example

ϵ=ϵmachmax⁡(1,∥xk∥),(8)\epsilon=\sqrt{\epsilon_{\mathrm{mach}}}\max(1,\left\lVert\mathbf{x}^{k}\right\rVert), \tag*{(8)}

possibly with additional clipping. This choice balances truncation error against cancellation and round-off error [1]. In FP32, this balance becomes more fragile because the range of useful perturbation sizes is narrower.

Forward-mode AD provides an alternative to the FD approach. Instead of differencing two residual evaluations, AD differentiates the implemented discrete residual directly and returns the primal residual together with its directional derivative,

jvp⁡(F,xk,vm)=(F(xk),J(xk)vm).(9)\operatorname{jvp}(\mathbf{F},\mathbf{x}^{k},\mathbf{v}_{m})=\left(\mathbf{F}(\mathbf{x}^{k}),\mathcal{J}(\mathbf{x}^{k})\mathbf{v}_{m}\right). \tag*{(9)}

AD differentiates the operations present in the residual evaluation. For smooth and deterministic residuals, this provides a consistent matrix-free linearization up to floating-point arithmetic. If the residual contains nonsmooth switches, adaptive discontinuities, or noisy embedded solvers, AD may differentiate these algorithmic artifacts. In such cases, carefully chosen finite differences can sometimes act as a macroscopic smoothing mechanism [7]. The present work focuses on the regime where the dominant difficulty is the finite-precision degradation of FD Jacobian-vector products.

For a fixed Newton iterate, Krylov methods approximate the solution of Eq. (4) in the subspace

Km=span⁡{r0,Jr0,J2r0,…,Jm−1r0},(10)\mathcal{K}_m = \operatorname{span}\left\{\mathbf{r}_0, \mathcal{J}\mathbf{r}_0, \mathcal{J}^{2}\mathbf{r}_0, \ldots, \mathcal{J}^{m-1}\mathbf{r}_0\right\}, \tag*{(10)}

where r0=−F−Jv0\mathbf{r}_0=-\mathbf{F}-\mathcal{J}\mathbf{v}_0 is the initial linear residual. The Krylov solver therefore repeatedly queries the Jacobian-vector product. Any noise or bias in this oracle changes the effective Krylov operator and can lead to stagnation or inaccurate Newton corrections.

The Krylov method must be chosen according to the properties of the linearized problem. CG is efficient and uses short recurrences, but it is appropriate only for symmetric positive-definite systems [4]. GMRES is more generally applicable to nonsymmetric systems and minimizes the residual norm over Km\mathcal{K}_m, but it requires storing an expanding orthogonal basis and is often restarted in practice [3]. BiCGSTAB also applies to nonsymmetric systems and has a lower memory footprint, but its convergence can be less monotone and more sensitive to ill conditioning [5].

Floating-point precision affects both the JVP and the Krylov basis construction. In FD JVPs, round-off and cancellation enter through Eq. (7). In Krylov methods, reduced precision can also degrade [13]. Inaccurate JVP can increase Krylov iteration counts, produce poorer Newton corrections, trigger additional line-search steps, and ultimately cause nonlinear solver failure.

Methodology

Solver

The numerical experiments use a matrix-free JFNK solver designed to isolate the effect of the Jacobian–vector product (JVP) evaluation. Figure 1 summarizes the nested solver workflow. The outer loop advances the continuation variable: time for IBVPs and angular frequency ω\omega for the time-harmonic Maxwell BVP. At each outer step, the nonlinear residual is solved by Newton iterations. Each Newton step calls an inner Krylov solver, which accesses the linearized problem only through matrix-free products J(xk)vm\mathcal{J}(x^k)v_m. After the Krylov solve, a backtracking line search is applied using the same residual evaluation for both FD and AD cases.

Figure 1. Nested matrix-free Newton-Krylov workflow used in the experiments. The same outer continuation loop, time for (a) and frequency for (b), Newton iteration, Krylov solver, and line search are used for both linearization strategies. The only algorithmic difference is the JVP evaluation inside the Krylov loop, where either an FD residual difference or a forward-mode AD call is used.

The FD implementation uses the FD JVP in Eq. (7), with the perturbation size selected according to Eq. (8). The AD implementation uses the forward-mode JVP in Eq. (9), applied to the same discrete residual function FF. In both cases, the Krylov solver receives a matrix-free linear operator. No Jacobian matrix is assembled or stored. This makes the FD and AD variants identical except for the routine that evaluates J(xk)vm\mathcal{J}(x^k)v_m. The Krylov solvers are implemented through matrix-free linear operator interfaces. Each time the Krylov method requests a matrix-vector product, the operator calls either the FD or AD JVP routine. We use GMRES and BiCGSTAB for nonsymmetric systems. tems, and CG when the linearized system is symmetric positive definite. This allows the effect of the JVP construction to be evaluated together with the choice of Krylov method, without changing the nonlinear solver. The implementation combines JAX for residual evaluation and forward-mode AD with SciPy and CuPy for Krylov linear algebra. Figure 2 shows an example computational graph produced by jax.jvp for the Maxwell residual. The graph illustrates that the AD implementation evaluates the primal residual and the corresponding directional derivative from the same discrete residual function. This provides the matrix-free product required by the Krylov solver without assembling the Jacobian. On CPUs, the Krylov methods are called through scipy.sparse.linalg [27]; on GPUs, the corresponding CuPy routines are used [28]. JAX provides the AD JVP through jax.jvp [29]. We do not use JAX-native Krylov solvers because their dynamic iteration counts can lead to substantial compilation overhead. Instead, JAX evaluates the residual and AD operations, while SciPy or CuPy performs the iterative linear algebra. On GPUs, JAX and CuPy arrays are exchanged through DLPack zero-copy views, avoiding host-device transfers during the Krylov iterations.

Figure 2. Example computational graph generated by jax.jvp for the time-harmonic Maxwell residual. The primal input is the current field state and the tangent input is the Krylov direction. The AD trace evaluates both the nonlinear residual and the directional derivative J(xk)vm\mathcal{J}(\mathbf{x}^{k})\mathbf{v}_{m}, which is passed to the matrix-free Krylov solver.

Benchmark Problems

The benchmark suite is designed to test how the JVP evaluation affects the robustness of the same JFNK workflow across nonlinear PDEs with difdifferent algebraic and physical structure. We consider three time-dependent initial-boundary value problems (IBVPs) and one frequency-domain nonlinear boundary value problem (BVP), spanning nonsymmetric and symmetric systems, scalar and vector unknowns, real and complex fields, and regimes with different stiffness levels. For the time-dependent problems, the residual is built from an implicit Crank-Nicolson time discretization and second-order finite differences in space; the previous time-step solution is used as the Newton initial guess. For the Maxwell problem, the continuation parameter is the angular frequency ω\omega, and the initial guess is obtained from the corresponding linearized problem. Further discretization details are given in Appendix § 7.

The first problem is the two-dimensional viscous Burgers equation,

∂U∂t+(U⋅∇)U=ν∇2U,(11)\frac{\partial\mathbf{U}}{\partial t} + (\mathbf{U}\cdot\nabla)\mathbf{U} = \nu\nabla^{2}\mathbf{U}, \tag*{(11)}

where U\mathbf{U} is the velocity field and ν\nu is the kinematic viscosity. The advective nonlinearity produces nonsymmetric linearized systems, making this problem a representative test case for nonsymmetric Krylov solvers.

The second problem is the Su-Olson radiation-diffusion system [30],

{∂U∂t=13∇2U−(U−V)+Q(x,t),∂V∂t=ξ(U−V),(12)\begin{cases} \frac{\partial U}{\partial t} = \frac{1}{3}\nabla^{2}U - (U - V) + Q(\mathbf{x},t),\\ \frac{\partial V}{\partial t} = \xi(U - V), \tag*{(12)} \end{cases}

where UU is the radiation energy density, VV is the material energy density, Q(x,t)Q(\mathbf{x},t) is an imposed source, and ξ\xi controls the material response. This problem tests coupled residuals with different source configurations and material parameters.

The third problem is the scalar reaction-diffusion equation,

∂u∂t=D∇2u−u3,(13)\frac{\partial u}{\partial t} = D\nabla^{2}u - u^{3}, \tag*{(13)}

where DD is the diffusion coefficient and u3u^{3} is a nonlinear sink term. This case provides a structured nonlinear system in which conjugate-gradient-based solvers can also be tested when the linearized problem is symmetric positive definite.

The fourth problem is the time-harmonic Maxwell equation in a Kerr medium,

∇×∇×E−ω2μ0ε(E)E=iωμ0J,(14)\nabla\times\nabla\times\mathbf{E} - \omega^{2}\mu_{0}\varepsilon(\mathbf{E})\mathbf{E} = i\omega\mu_{0}\mathbf{J}, \tag*{(14)}

where E\mathbf{E} is the complex electric field, ω\omega is the angular frequency, μ0\mu_0 is the vacuum permeability, and J\mathbf{J} is the imposed current density. The field-dependent permittivity is

ε(E)=ε0(1−0.05i)(1+χ∣E∣),(15)\varepsilon(\mathbf{E}) = \varepsilon_0(1 - 0.05i)(1 + \chi|\mathbf{E}|), \tag*{(15)}

where χ\chi controls the Kerr nonlinearity. This BVP is the stiffest benchmark in the suite because resonant regimes can strongly amplify the field and make the linearized systems ill conditioned.

Experimental design

For each benchmark problem, we vary the initial condition or source configuration together with a single physical control parameter. The Burgers equation is tested using Taylor–Green vortex (TGV), double shear layer (DSL), and four-vortex collision (4VC) initial conditions with ν∈{0.1,0.05,0.01}\nu\in\{0.1, 0.05, 0.01\}. The Su–Olson problem is tested using static and moving source configurations with ξ∈{1,0.1,0.01}\xi\in\{1, 0.1, 0.01\}. The reaction–diffusion problem is tested using Gaussian and sinusoidal initial data with D∈{0.1,0.01,0.001}D \in\{0.1, 0.01, 0.001\}. The Maxwell–Kerr problem is tested using Gaussian and dipole current sources with χ∈{0.5,0.1,0.05}\chi\in\{0.5, 0.1, 0.05\}. Together with the grid choices described below, this parameter sweep produces 480 benchmark simulations organized into 54 nonlinear problems.

The grid sizes are selected according to the target hardware. CPU runs use 64×6464 \times64 and 128×128128 \times128 grids, while GPU runs use 256×256256 \times256 and 512×512512 \times512 grids. The time-dependent problems are advanced with Courant number C=1C = 1, which gives developed nonlinear dynamics while keeping the time-integration setup fixed across all solvers. For the Maxwell–Kerr problem, a preliminary frequency sweep is used to identify strongly resonant regions. The reported runs are then restricted to frequency ranges that remain well posed while still producing stiff nonlinear solves.

Each configuration is evaluated using both JVP strategies, FD and AD, in single- and double-precision. For nonsymmetric systems, we test GMRES and BiCGSTAB. For the reaction–diffusion problem, where the linearized system is symmetric positive definite, we also include CG. This gives eight solver combinations for the nonsymmetric problems and twelve for the reaction–diffusion problem, for a total of 480 solver runs per backend.

The nonlinear and linear tolerances are chosen according to the floating-point precision to avoid both premature convergence and over-solving. In FP32, we set tolNewton=10−4\mathrm{tol}_{\mathrm{Newton}} = 10^{-4}, tolKrylov=10−6\mathrm{tol}_{\mathrm{Krylov}} = 10^{-6}. In FP64, we set tolNewton=\mathrm{tol}_{\mathrm{Newton}} = 10−610^{-6}, tolKrylov=10−8\mathrm{tol}_{\mathrm{Krylov}} = 10^{-8}. The maximum number of Newton iterations is 15 for the time-dependent IBVPs and 50 for the Maxwell BVP. The maximum Krylov iteration count is set to 100. For restarted GMRES, this corresponds to the maximum number of restart cycles. These settings are kept fixed for FD and AD so that differences in runtime, iteration count, and convergence behavior reflect the JVP evaluation rather than different solver tolerances.

Metrics and failure criteria

We record JFNK metrics that distinguish between Krylov stagnation, poor Newton progress, and complete nonlinear failure. For each run, we collect the total execution time, total number of Krylov iterations, number of Newton iterations, residual histories, line-search activity, and convergence status. A run is considered successful if the nonlinear residual reaches the prescribed Newton tolerance for the required time steps or continuation points. A run is classified as failed if any of the following conditions is met:

  1. the simulation terminates prematurely because of numerical instability, overflow, non-finite values, or breakdown of the linear or nonlinear solver;

  2. for time-dependent IBVPs, more than 20% of the time steps fail to converge within two orders of magnitude of the requested Newton tolerance;

  3. for the Maxwell BVP, more than 20% of the frequency steps fail to converge to the requested Newton tolerance.

The second criterion is included to avoid over-penalizing transient convergence difficulties in time-dependent nonlinear problems. In some strongly nonlinear regimes, especially when FD JVPs are used, a small number of time steps may exhibit poor Newton convergence while the overall simulation remains accurate and stable. Such cases are therefore distinguished from systematic solver failure.

To compare solver performance across a heterogeneous benchmark suite, we use Dolan–Moré performance profiles [31]. These profiles summarize both performance and robustness without allowing a small number of difficult or unusually easy cases to dominate the comparison. Let P\mathcal{P} denote the set of benchmark configurations and S\mathcal{S} the set of solver variants being compared. For each problem p∈Pp \in\mathcal{P} and solver s∈Ss \in\mathcal{S}, let tp,st_{p,s} be a performance metric, such as wall-clock time or total Krylov iterations. The performance ratio is defined as

rp,s=tp,smin⁡s∈Stp,s.(16)r_{p,s} = \frac{t_{p,s}}{\min_{s \in S} t_{p,s}}. \tag*{(16)}

Thus, rp,s=1r_{p,s} = 1 means that solver ss is the best performer for problem pp according to the chosen metric. Failed runs are assigned rp,s=∞r_{p,s} = \infty, so they do not contribute to the profile at finite values of τ\tau. The performance profile of solver ss is then

ρs(τ)=1∣P∣∣{p∈P:rp,s≤τ}∣,τ≥1.(17)\rho_s(\tau) = \frac{1}{|\mathcal{P}|} \left|\{p \in\mathcal{P} : r_{p,s} \leq\tau\}\right|,\qquad\tau\geq1. \tag*{(17)}

The value ρs(1)\rho_s(1) gives the fraction of benchmark configurations for which solver ss is the fastest or requires the fewest iterations. As τ\tau increases, ρs(τ)\rho_s(\tau) measures the fraction of problems solved within a factor τ\tau of the best solver. The limiting value reached at large τ\tau corresponds to the solver success rate over the benchmark suite.

Results

Verification and representative solutions

Before comparing linearization strategies, we verify that the benchmark problems produce the expected physical behavior and that the discretization error is not dominated by the choice of JVP evaluation. Figure 3–Figure 6 show representative solutions from the four benchmark classes: Burgers dynamics, Su–Olson radiation diffusion, reaction–diffusion, and the nonlinear time-harmonic Maxwell problem. These cases illustrate the range of regimes used in the performance study, including advective steepening, coupled radiation–material transport, nonlinear dissipation, and resonant electromagnetic fields.

Taylor–Green vortex velocity and magnitude fields

Figure 3. Taylor–Green vortex simulation for the Burgers’ equation on a 512×512512 \times512 grid after 1000 time steps. The computational domain is x,y∈[0,2π)x,y \in[0,2\pi) with fully periodic boundary conditions. The kinematic viscosity is set to ν=0.05\nu= 0.05 to induce steep velocity gradients and macroscopic shock formation. The Courant number is C=1C = 1.

Symmetric Gaussian source simulation for the time-harmonic Maxwell equations

Figure 6. Symmetric Gaussian source simulation for the time-harmonic Maxwell equations on a 512×512512 \times512 grid at normalized frequency ω=18\omega= 18, with χ=0.05\chi= 0.05. The computational domain is x,y∈[0,1]x,y \in[0,1] with Perfect Electric Conductor (PEC) boundary conditions, representing a fully enclosed resonant cavity.

Figure 7 summarizes the verification results. The Burgers Taylor–Green vortex shows comparable temporal error growth in FP32 and FP64 across grid resolutions. The spatial convergence at final time follows the expected second-order behavior of the central-difference discretization, using a 1024 × 1024 AD-FP64 solution as reference. A precision floor is visible for FD in FP32 at the finest grid, but the dominant AD/FD differences observed later arise from Krylov convergence rather than from the baseline discretization error.

Verification and convergence behavior of the JFNK solver

Figure 7. Verification and convergence behavior of the JFNK solver. The reference solution is computed on a 1024×10241024 \times1024 grid with AD in FP64. (a) Temporal error evolution for the Burgers Taylor–Green vortex with C = 1. (b) Spatial convergence at final time, showing the expected second-order behavior from the central-difference discretization. AD and FD produce comparable discretization errors, although a precision floor appears for FD in FP32 at the finest grid. (c) Newton convergence for the nonlinear Maxwell–Kerr problem at different frequencies, showing slower convergence near resonant regimes.

The Newton convergence curves for the Maxwell–Kerr problem show the increasing difficulty of the nonlinear solve near resonant regimes. Frequencies closer to resonance require more Newton iterations, while non-resonant cases converge more rapidly. This confirms that the benchmark suite contains both relatively benign nonlinear time steps and stiff steady nonlinear solves.

Su–Olson radiation diffusion scalar fields

Figure 4. Su-Olson radiation diffusion simulation of a orbiting source in circular trajectory with Gaussian amplitude, characterized by unit amplitude and standard deviation 0.4. Shown are the scalar fields U(x,y)U(x,y), V(x,y)V(x,y), and their combined intensity U+VU + V. The computational domain is x,y∈[−5,5]x,y \in[-5,5]. The parameters are Q0=1Q_0 = 1 and ξ=0.1\xi= 0.1. The simulation is performed on a 512×512512 \times512 grid with C=1C = 1 after 8000 time steps.

JVP cost and Krylov stagnation

The end-to-end performance gains of AD are not derived from cheaper derivative evaluations. As shown in Table 1, AD and FD exhibit comparable per-call costs on both CPU and GPU backends, with AD being slightly more expensive in some cases and slightly faster in others. This indicates that the performance bottleneck is not the cost of evaluating J(xk)vmJ(x^k)v_m in isolation, but rather lies within the Krylov solver itself. Hence, the dominant factor governing runtime is not the per-iteration Jacobian-vector product, but the number and effectiveness of Krylov iterations required to complete the nonlinear solve.

JVPPrecisionApple M4NVIDIA A100
Avg time (μ\mus)% of solverAvg time (μ\mus)% of solver
FDFP32211±12.0%211 \pm12.0\%33.8%256±8.1%256 \pm8.1\%30.7%
FDFP64396±25.4%396 \pm25.4\%14.6%282±10.1%282 \pm10.1\%14.9%
ADFP32286±42.2%286 \pm42.2\%14.3%214±6.4%214 \pm6.4\%12.5%
ADFP64426±20.1%426 \pm20.1\%18.0%228±9.8%228 \pm9.8\%12.8%

Table 1. Average Jacobian–vector product (JVP) time for a 256×256256 \times256 Burgers TGV case over 100 time steps. AD and FD have comparable per-call costs, so the dominant runtime differences observed later are mainly due to changes in Krylov and Newton convergence rather than to the cost of an individual JVP.

Reaction–diffusion solutions at $t_0$, $t_1$, and $t_2$

Figure 5. Reaction–diffusion solution on a sinusoidally perturbed grid, illustrating the formation of spatial patterns in u(x,y)u(x,y). The computational domain is x,y∈[−0.5,0.5]x,y \in[-0.5,0.5]. The diffusion coefficient is set to D=0.01D = 0.01. The simulation is performed on a 512×512512 \times512 grid with C=1C = 1. Results are shown at the initial condition (t0t_0), after 1000 time steps (t1t_1), and after 2000 time steps (t2t_2).

The main bottleneck appears inside the Krylov solver. Table 2 shows a representative single-precision Burgers 4-vortex collision (4VC) case in which FD and AD solve the same nonlinear problem with identical Newton and Krylov settings. With FD-JVPs, GMRES repeatedly reaches iteration

JVP MethodSolver Time (s)Newton It.Total GMRES It.Converged
Finite Difference (FD)99.72200196,300✓
Automatic Diff. (AD)0.59200400✓
169×169\times–491×491\times–

Table 2. Representative FP32 Burgers 4VC case comparing FD and AD JVPs with GMRES. Same residual discretization, Newton iteration, line search, Krylov tolerance, and iteration limits are used. Both produce a converged solution with the same number of Newton iterations, but FD repeatedly reaches iteration counts close to the GMRES iteration cap, while AD only requires a few Krylov iterations per Newton loop.

counts close to the iteration cap, accumulating 196,300 Krylov iterations. With AD-JVPs, the same solve converges with only 400 Krylov iterations, corresponding to an average of 2 Krylov iterations per Newton step. As a result, the wall-clock solver time decreases from 99.72 to 0.59 s.

This behavior explains why AD can produce large end-to-end speedups even when the cost of an individual AD-JVP is comparable to, or slightly higher than, an FD-JVP. In FP32, the FD perturbation is more sensitive to cancellation and round-off error. The resulting JVPs define a noisy or

biased effective operator for GMRES. Since GMRES constructs its Krylov basis from repeated applications of this operator, errors in the JVP can slow residual minimization, trigger restarts, and eventually force the linear solve to stagnate at the iteration cap.

AD avoids this failure mode by differentiating the implemented discrete residual directly. The Krylov solver therefore receives a more consistent matrix-free linearization of the same residual function, without the need to tune a FD perturbation size. The observed speedups are therefore a solver-level effect: AD reduces the number of Krylov iterations and improves the Newton correction, rather than merely accelerating the evaluation of a single derivative.

Robustness and performance profiles

We next compare solver robustness and efficiency over the full benchmark suite using Dolan–Moré performance profiles. The profiles are built from wall-clock time, with failed runs assigned an infinite performance ratio as described in § 3.4. Thus, the height of each curve at small τ\tau measures efficiency, while the plateau at large τ\tau measures the fraction of benchmark configurations successfully solved.

Figure 8(a) shows the CPU performance profiles. AD-based solvers dominate the upper-left region of the plot, indicating that they are both faster on a larger fraction of problems and more likely to complete the benchmark suite. Among the AD variants, BiCGSTAB is often the fastest option, especially in FP32, because of its low memory footprint and short recurrences. GMRES with AD is slightly less efficient in some cases, but it remains more reliable for difficult nonsymmetric problems. In contrast, FD-based solvers show lower plateaus and much larger performance ratios, particularly in FP32, where FD JVPs can lead to Krylov stagnation.

CPU performance profile and reaction–diffusion solver comparison plots

Figure 8. CPU performance results on the Apple M4 backend. (a) Dolan–Moré performance profiles over the benchmark suite. AD-based JVPs provide higher success rates and lower performance ratios than FD-based JVPs. (b) Reaction–diffusion case on a 128 ×\times 128 grid with D=0.01D = 0.01, illustrating the benefit of CG when the linearized system is symmetric positive definite.

The GPU profiles in Figure 9(a) show the same qualitative behavior. AD-based solvers again occupy the most favorable region of the plot. BiCGSTAB with AD gives the best raw performance for many configurations, while GMRES with AD provides the most reliable behavior across the full set of nonlinear problems. FD-based solvers remain substantially less robust, especially in FP32. Although the A100 backend can partially mask large iteration counts through higher throughput, the profile still shows that poor FD linearizations lead to slower and less reliable solves.

GPU performance profiles and reaction–diffusion solver comparison

Figure 9. GPU performance results on the NVIDIA A100 backend. (a) Dolan–Moré performance profiles over the benchmark suite. AD-based solvers achieve higher robustness and better time-to-solution than FD-based solvers. (b) Reaction–diffusion case on a 512×512512 \times512 grid with D=0.01D = 0.01. For this SPD problem, CG with AD is the most efficient Krylov choice.

The reaction-diffusion panels in Figure 8(b) and Figure 9(b) isolate a case where the linearized system is symmetric positive definite. In this setting, CG is the appropriate Krylov method and gives the lowest runtimes when paired with AD. This confirms that the best Krylov method depends on the algebraic structure of the linearized problem: CG is preferable for SPD systems, BiCGSTAB is often fastest for moderately difficult nonsymmetric systems, and GMRES is more reliable for stiff or ill-conditioned nonsymmetric regimes.

Overall, the performance profiles show that the advantage of AD is not limited to a single problem or backend. Across CPU and GPU runs, AD improves the success rate and reduces the time-to-solution by providing a more reliable matrix-free linearization to the Krylov solver. The largest separation from FD occurs in single precision, where the FD perturbation is most sensitive to round-off and cancellation.

Krylov solver selection and failure analysis

The performance profiles indicate that JVP construction and Krylov solver choice must be considered together. Table 3 summarizes the failed configurations for the CPU and GPU solvers across all 480 benchmark simulations for each backend. Most failures are associated with FD-based JVPs:

ProblemPrecision / SolverAD failures CPUAD failures GPUFD failures CPUFD failures GPU
Burgers TGVFP32 BiCGSTAB2–66
Burgers TGVFP32 GMRES2–66
Burgers TGVFP64 BiCGSTAB2–42
Burgers TGVFP64 GMRES2–62
Burgers 4VCFP32 BiCGSTAB––64
Burgers 4VCFP32 GMRES––3–
Burgers 4VCFP64 BiCGSTAB––31
Burgers 4VCFP64 GMRES––61
Burgers DSLFP32 BiCGSTAB––66
Burgers DSLFP32 GMRES––66
Burgers DSLFP64 BiCGSTAB––54
Burgers DSLFP64 GMRES––64
Maxwell–Kerr dipoleFP32 BiCGSTAB–3–3
Maxwell–Kerr dipoleFP64 BiCGSTAB–3–3
Reaction–diffusion GaussianFP32 BiCGSTAB–––6
Reaction–diffusion GaussianFP64 BiCGSTAB–––2
Reaction–diffusion sinusoidalFP32 BiCGSTAB–––6
Reaction–diffusion sinusoidalFP64 BiCGSTAB–––2
Total866364

Table 3. Failed solver configurations. The Su-Olson test cases achieved no failures. Divergence within AD configurations was rare: on the CPU, it was limited to 8 total failures in the Burgers TGV problem, while on the GPU, AD failures were confined to the ill-conditioned Maxwell problem with BiCGSTAB (6 failures). Majority of recorded failures (88.7% on CPU, 91.4% on GPU) were driven by FD JVPs. CPU and GPU implementations demonstrated nearly identical global robustness (with 71 and 70 failures over 480 simulations), though direct comparison is limited given different grid resolutions.

63 out of 71 failures on the CPU backend and 64 out of 70 failures on the GPU backend. AD-based failures are comparatively rare and are concentrated in configurations where BiCGSTAB is applied to the stiff Maxwell–Kerr dipole problem.

The table shows that the main robustness difference is not caused by the hardware backend, but by the combination of linearization strategy, precision, and Krylov solver. FD failures are most common in Burgers configurations, especially in FP32, where the FD perturbation is most sensitive to cancellation and round-off. These failures are consistent with the stagnation behavior discussed in § 4.2: inaccurate JVPs degrade the effective Krylov operator and can prevent the linear residual from decreasing within the iteration limit.

AD substantially reduces this failure mode because the Krylov solver receives the directional derivative of the implemented residual rather than a FD approximation. However, AD does not remove all solver sensitivity. The remaining AD failures occur with BiCGSTAB for the Maxwell dipole case on the GPU. This is consistent with the non-monotone convergence behavior of BiCGSTAB: its short recurrences are efficient, but the method is more vulnerable to breakdown in stiff and ill-conditioned nonsymmetric systems. GMRES with AD is therefore the more reliable choice for the Maxwell–Kerr problem, even when BiCGSTAB is faster on easier nonsymmetric cases.

For the reaction–diffusion problem, the linearized system admits a symmetric positive-definite structure, and CG is the preferred Krylov method. The reaction–diffusion timing panels in Figure 8(b) and Figure 9(b) show that CG with AD gives the lowest runtimes for this class of problems. This confirms that AD improves the quality of the matrix-free linearization, but the Krylov method should still be selected according to the algebraic structure of the linearized system.

Overall, the results suggest the following practical solver choices. For stiff nonsymmetric problems, AD with GMRES provides the most robust configuration. For moderately difficult nonsymmetric problems, AD with BiCGSTAB often gives the best time-to-solution. For symmetric positive-definite problems, AD with CG is preferable. FD-based JVPs are most fragile in FP32 and should be used with caution when the Krylov solve is sensitive to perturbation error.

Conclusions

This work evaluates FD and AD JVPs in matrix-free JFNK solvers for nonlinear PDEs, covering both time-marching IBVPs and frequency-domain BVPs. The main conclusion is that Jacobian-vector product construction is a critical numerical design choice in JFNK solvers. Although JFNK methods avoid forming the Jacobian, the Krylov solver still acts on a matrix-free representation of the local linearized residual. When this representation is obtained by finite differences, the perturbation parameter introduces sensitivity to residual scaling, cancellation, and round-off error. This sensitivity is particularly important in FP32, where the useful range of perturbation sizes is narrower. In the tested problems, inaccurate FD JVPs degraded the Krylov operator, increased iteration counts, and in some cases prevented the nonlinear solve from converging. Forward-mode AD mitigates this failure mode by differentiating the implemented discrete residual directly. The resulting JVP improve the quality of the Krylov subproblem, leading to fewer Krylov iterations, more reliable Newton convergence, and much lower failure rates. The most significant differences are observed in single precision, with a clear separation appearing in the Dolan–Moré performance profiles. All AD configurations surpassed a 95% completion rate, whereas FD performance dropped as low as 64% on CPU and 42% on GPU. Speedups exceed three orders of magnitude for the CPU solver and two orders of magnitude for the GPU solver in completed simulations. For a representative converged FP32 Burgers case, AD achieved a 169×169 \times speedup over FD by reducing GMRES iterations by 491×491 \times for the same number of Newton iterations.

These gains were not caused by a lower cost per Jacobian-vector product. In the present implementation, AD and FD had comparable per-call costs. Instead, the improvement came from reducing the number of Krylov and Newton iterations required by the full nonlinear solve.

The performance profiles further demonstrate that the benefits of AD are not limited to a single backend. On both CPU and GPU, AD-based solvers achieved higher success rates and better time-to-solution than FD-based solvers. The GPU backend can partially hide inflated iteration counts through higher throughput, but it does not remove the underlying algorithmic inefficiency caused by poor FD linearizations. Thus, the advantage of AD should be interpreted primarily as an improvement in solver robustness, with performance gains following from more reliable Krylov convergence.

The choice of Krylov method remains problem dependent. For nonsymmetric problems, BiCGSTAB with AD was often the fastest option when the linearized systems were moderately stiff. However, its non-monotone convergence behavior made it less reliable for the stiff Maxwell–Kerr dipole cases. GMRES with AD was more robust in these regimes, although sometimes at higher cost, as expected. For reaction-diffusion problems with symmetric positive-definite linearized systems, CG with AD was the most efficient choice. These results indicate that AD improves the matrix-free linearization, but the Krylov solver should still be selected according to the algebraic structure and stiffness of the problem.

The conclusions should also be interpreted within the scope of the residuals considered here. AD differentiates the implemented discrete residual, not the continuous PDE. For smooth and deterministic residual evaluations, this provides a consistent matrix-free linearization and avoids the perturbationsize sensitivity of finite differences. If the residual contains nonsmooth limiters, adaptive discontinuities, stochastic components, or noisy embedded solvers, AD may differentiate algorithmic artifacts. In such cases, carefully chosen finite differences or modified linearization strategies may still be useful. The present results identify the regime where AD is most beneficial: smooth finite-precision JFNK residuals for which FD perturbation error is the dominant source of Krylov degradation.

Overall, this study demonstrates that robust matrix-free JFNK performance depends not only on the choice of Krylov solver or hardware backend, but fundamentally on the construction of the Jacobian-vector products. For the nonlinear PDE benchmarks examined, forward-mode AD provides a more reliable finite-precision linearization than finite differences, particularly within FP32 and stiff nonlinear regimes.

Acknowledgments

This work has received funding from the Swedish Research Council’s Research Environment grant (SEE-6GIA 2024-06482). We thank the CuPy developers for their support during our implementation of the CuPy BiCGSTAB Krylov solver.

Numerical Treatment of Test Cases

We evaluate the JFNK solver across two distinct families of PDEs: the time-marching initial boundary value problems (IBVPs) and boundary value problems (BVPs).

Time-marching systems are generally well-conditioned for implicit solvers. The discretization of the time derivative introduces a diagonally dominant mass matrix (scaled by 1/Δt1/\Delta t) into the Jacobian, and the converged solution from the previous time step provides a highly accurate initial guess within the Newton quadratic convergence radius. Conversely, BVPs seek a global field distribution coming from a steady-state or frequency domain problem. Lacking a time derivative, the Jacobian is dictated entirely by high-order spatial operators and highly localized nonlinearities, causing severe stiffness, ill-conditioned linear systems. Furthermore, boundary value problems (BVPs) require initialization from an approximate initial guess, placing significant strain on the global convergence properties of the Newton method and its associated line search. In addition, their solutions are complex-valued fields, which effectively doubles the memory requirements during the computation.

Burgers' Equation

The 2D transient Burgers’ equation, driven by convective nonlinearity, is given by:

∂U∂t+(U⋅∇)U=ν∇2U(18)\frac{\partial\mathbf{U}}{\partial t} + (\mathbf{U} \cdot\nabla)\mathbf{U} = \nu\nabla^{2}\mathbf{U} \tag*{(18)}

where U=(u,v)T\mathbf{U} = (u,v)^{T} represents the velocity field and ν\nu denotes the kinematic viscosity.

Discretization. Defining discrete second-order central difference operators for advection, Adv⁡(U)=uDxU+vDyU\operatorname{Adv}(\mathbf{U}) = uD_x\mathbf{U}+vD_y\mathbf{U}, and diffusion, Lap⁡(U)=DxxU+DyyU\operatorname{Lap}(\mathbf{U}) = D_{xx}\mathbf{U}+D_{yy}\mathbf{U}, the nonlinear residual at Newton iteration kk (targeting time level n+1n+1) is expressed in vector form as:

F(Uk,Un)=Uk−Un+Δt2[Adv⁡(Uk)−νLap⁡(Uk)+Adv⁡(Un)−νLap⁡(Un)](19)\mathbf{F}(\mathbf{U}^{k},\mathbf{U}^{n}) = \mathbf{U}^{k}-\mathbf{U}^{n}+\frac{\Delta t}{2}\left[\operatorname{Adv}(\mathbf{U}^{k})-\nu\operatorname{Lap}(\mathbf{U}^{k})+\operatorname{Adv}(\mathbf{U}^{n})-\nu\operatorname{Lap}(\mathbf{U}^{n})\right] \tag*{(19)}

The time step is constrained adaptively via Courant–Friedrichs–Lewy (CFL) condition:

Δt=C⋅min⁡(Δxmax⁡∣u∣,Δymax⁡∣v∣,(2ν(1Δx2+1Δy2))−1)(20)\Delta t = C \cdot\min\left(\frac{\Delta x}{\max|u|},\frac{\Delta y}{\max|v|},\left(2\nu\left(\frac{1}{\Delta x^{2}}+\frac{1}{\Delta y^{2}}\right)\right)^{-1}\right) \tag*{(20)}

Initial and Boundary Conditions. The solver is evaluated against three fully periodic benchmark configurations:

  • Taylor-Green Vortex (TGV): An analytical baseline tracking exponential kinetic energy decay, initialized as u=sin⁡(x)cos⁡(y)u = \sin(x)\cos(y) and v=−cos⁡(x)sin⁡(y)v = -\cos(x)\sin(y).

  • Double Shear Layer (DSL): A rigorous nonlinear stress test driven by Kelvin-Helmholtz instabilities (ρ=30\rho= 30, δ=0.05\delta= 0.05):

u(x,y)={tanh⁡(ρ(y−π2))y≤πtanh⁡(ρ(3π2−y))y>π,v(x,y)=δsin⁡(x)(21)u(x,y) = \begin{cases} \tanh\left(\rho\left(y-\frac{\pi}{2}\right)\right) & y \le\pi\\ \tanh\left(\rho\left(\frac{3\pi}{2}-y\right)\right) & y > \pi \end{cases}, \qquad v(x,y) = \delta\sin(x) \tag*{(21)}
  • 4-Vortex Collision (4VC): Evaluates line search robustness during violent flow reorganization via four alternating-sign Gaussian vortices (R=0.5R = 0.5):

U(x,y)=∑i=14Γi(−(y−cy,i)x−cx,i)e−ri2/R2(22)\mathbf{U}(x,y) = \sum_{i=1}^{4}\Gamma_i\begin{pmatrix}-(y-c_{y,i}) \\ x-c_{x,i}\end{pmatrix}e^{-r_i^{2}/R^{2}} \tag*{(22)}

where ri2=(x−cx,i)2+(y−cy,i)2r_i^{2} = (x-c_{x,i})^{2}+(y-c_{y,i})^{2}.

Non-Equilibrium Radiative Diffusion (Su-Olson)

The Su-Olson problem models dimensionless non-equilibrium radiation transport, coupling a diffusing radiation energy field UU to a localized material energy field VV:

∂U∂t=13∇2U−(U−V)+Q(x,t),∂V∂t=ξ(U−V)(23)\frac{\partial U}{\partial t} = \frac{1}{3}\nabla^2 U - (U - V) + Q(\mathbf{x},t), \qquad\frac{\partial V}{\partial t} = \xi(U - V) \tag*{(23)}

In the physical regime where ξ≪1\xi\ll1, the disparity in timescales between rapid radiation diffusion and slow localized material absorption renders the system highly stiff.

Discretization. The coupled discrete residual under Crank–Nicolson integration is formulated as:

FU=Uk−Un−Δt2[13Lap⁡(Uk+Un)−(Uk+Un)+(Vk+Vn)+2Q]F_U = U^k - U^n - \frac{\Delta t}{2}\left[\frac{1}{3}\operatorname{Lap}(U^k + U^n) - (U^k + U^n) + (V^k + V^n) + 2Q\right]
FV=Vk−Vn−Δt2ξ[(Uk+Un)−(Vk+Vn)](24)F_V = V^k - V^n - \frac{\Delta t}{2}\xi\left[(U^k + U^n) - (V^k + V^n)\right] \tag*{(24)}

subject to the multi-physics CFL constraint:

Δt=min⁡(Cmin⁡(Δx,Δy)2D,Cξ+δ)(25)\Delta t = \min\left(C\frac{\min(\Delta x,\Delta y)^2}{D},\frac{C}{\xi+\delta}\right) \tag*{(25)}

Configurations. We explore two spatial configurations. The classic 1D Su-Olson profile maps to a 2D grid with periodic yy-boundaries, applying a cold start (U=V=0U = V = 0) driven by a central, stationary box source Q0Q_0. The dynamic 2D configuration stresses fully enclosed spatial coupling within strict Dirichlet boundaries, initializing decaying concentric rings and an orbiting Gaussian source:

Q(x,y,t)=Q0exp⁡(−(x−Rcos⁡(ωt))2+(y−Rsin⁡(ωt))22σ2)(26)Q(x,y,t) = Q_0 \exp\left(-\frac{(x - R\cos(\omega t))^2 + (y - R\sin(\omega t))^2}{2\sigma^2}\right) \tag*{(26)}

Nonlinear Reaction-Diffusion

This equation models a scalar concentration field uu subject to a volumetric nonlinear depletion sink:

∂u∂t=D∇2u−u3(27)\frac{\partial u}{\partial t} = D\nabla^2u - u^3 \tag*{(27)}

Because the analytical derivative of the sink is strictly non-positive (f′(u)=−3u2≤0f'(u) = -3u^2 \le0) and the discrete Laplacian is negative-definite, the Jacobian matrix, J=I−Δt2DL+3Δt2(uk)2I\mathcal{J} = \mathbf{I} - \frac{\Delta t}{2}D\mathbf{L} + \frac{3\Delta t}{2}(u^k)^2\mathbf{I}, is inherently Symmetric Positive Definite (SPD). This property guarantees Conjugate Gradient (CG) convergence, allowing for direct comparisons against GMRES and BiCGSTAB both on CPU and GPUs.

The domain is entirely bounded by homogeneous Dirichlet conditions. We initialize the system with three concentration profiles: a single Gaussian, a Multi-Gaussian with four interacting peaks, and a high-gradient Sinusoidal field u(x,y)=∣sin⁡(kxπx)sin⁡(kyπy)∣u(x,y) = \lvert\sin(k_x\pi x)\sin(k_y\pi y)\rvert to evaluate rapid gradient decay. The system is discretized analogously to previous cases, with CFL condition:

Δt=CΔx2/D(28)\Delta t = C\Delta x^2 / D \tag*{(28)}

Time-Harmonic Maxwell's Equations in Kerr Media

The time-harmonic Maxwell’s wave equation in a nonlinear medium is defined as:

∇×∇×E−ω2μ0ε(E)E=iωμ0J(29)\nabla\times\nabla\times\mathbf{E} - \omega^2\mu_0\varepsilon(\mathbf{E})\mathbf{E} = i\omega\mu_0\mathbf{J} \tag*{(29)}

For the 2D Transverse Electric (TE) formulation (E=(Ex,Ey,0)T\mathbf{E} = (E_x,E_y,0)^T), we model a Kerr-type medium with complex dielectric loss:

ε(E)=ε0(1−0.05i)(1+χ∣Ex∣2+∣Ey∣2)(30)\varepsilon(\mathbf{E}) = \varepsilon_0(1 - 0.05i)\left(1 + \chi\sqrt{\lvert E_x\rvert^2 + \lvert E_y\rvert^2}\right) \tag*{(30)}

where χ=0.05\chi= 0.05, while ε0=1.0\varepsilon_0 = 1.0 and μ0=1.0\mu_0 = 1.0. This localized material dependence fundamentally cross-couples the permittivity to the unknown field solution, rendering the system exceptionally ill-conditioned, especially at resonant frequencies (see Fig. A.1).

Discretization and Initialization. Applying second-order central differences to the curl-curl operator yields fully coupled residuals FxF_x and FyF_y. PEC boundaries are enforced explicitly via F=Etangential=0F = E_{\mathrm{tangential}} = 0. To guarantee Newton-Krylov stability, the initial field guess is computed using the Born approximation, solving the naturally linearized wave equation at E=0\mathbf{E} = 0 where ε=ε0(1−0.05i)\varepsilon= \varepsilon_0(1 - 0.05i).

Figure A.1. Resonance plot for the Maxwell equations in a Kerr medium with χ=0.05\chi= 0.05, normalized by the magnitude of the current.

Preconditioning. To accelerate inner Krylov solvers near structural resonances (ω)(\omega), we deploy a matrix-free, block-diagonal Complex Shifted Laplacian (CSLP) preconditioner. The local permittivity is artificially damped (εprecond=ε(E)(1−0.5i)\varepsilon_{\mathrm{precond}} = \varepsilon(\mathbf{E})(1 - 0.5i)) to shift the operator’s eigenvalues into the right half-plane. The decoupled diagonal components are:

Mxx=2Δy2−ω2μ0εprecond,Myy=2Δx2−ω2μ0εprecond(31)M_{xx} = \frac{2}{\Delta y^{2}} - \omega^{2}\mu_{0}\varepsilon_{\mathrm{precond}}, \qquad M_{yy} = \frac{2}{\Delta x^{2}} - \omega^{2}\mu_{0}\varepsilon_{\mathrm{precond}} \tag*{(31)}

Source Currents. The frequency response is mapped via two applied source currents (J\mathbf{J}):

  • Symmetric Gaussian: Jx=exp⁡(−(x−x0)2+(y−yi)22σ2)J_x = \exp\left(-\frac{(x-x_0)^2+(y-y_i)^2}{2\sigma^2}\right) and Jy=0J_y = 0.

  • Antisymmetric Dipole:

Jx=exp⁡(−(x−x0)2+(y−y1)22σ2)−exp⁡(−(x−x0)2+(y−y2)22σ2)(32)J_x = \exp\left(-\frac{(x-x_0)^2+(y-y_1)^2}{2\sigma^2}\right) - \exp\left(-\frac{(x-x_0)^2+(y-y_2)^2}{2\sigma^2}\right) \tag*{(32)}

and Jy=0J_y = 0.

References

  1. [1]D. Knoll, D. Keyes, Jacobian-free Newton–Krylov methods: a survey of approaches and applications, Journal of Computational Physics 193 (2 (2004) 357–397. doi:10.1016/j.jcp.2003.08.010. URL https://linkinghub.elsevier.com/retrieve/pii/S0021999103004340
  2. [2]C. T. Kelley, Iterative Methods for Linear and Nonlinear Equations, Frontiers in Applied Mathematics, Society for Industrial and Applied Mathematics, 1995. doi:10.1137/1.9781611970944. URL https://epubs.siam.org/doi/10.1137/1.9781611970944DOI
  3. [3]Y. Saad, M. H. Schultz, GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems, SIAM Journal on Scientific and Statistical Computing 7 (3) (1986) 856–869. doi:10.1137/0907058. URL http://epubs.siam.org/doi/10.1137/0907058DOI
  4. [4]M. R. Hestenes, E. Stiefel, Methods of Conjugate Gradients for Solving Linear Systems, Journal of Research of the National Bureau of Standards 49 (6) (1952) 409–436.DOI
  5. [5]H. A. Van Der Vorst, Bi-CGSTAB: A Fast and Smoothly Converging Variant of Bi-CG for the Solution of Nonsymmetric Linear Systems, SIAM Journal on Scientific and Statistical Computing 13 (2) (1992) 631–644. doi:10.1137/0913035. URL http://epubs.siam.org/doi/10.1137/0913035DOI
  6. [6]M. Pernice, H. F. Walker, NITSOL: A Newton Iterative Solver for Nonlinear Systems, SIAM Journal on Scientific Computing 19 (1) (1998) 302–318. doi:10.1137/S1064827596303843. URL https://epubs.siam.org/doi/S1064827596303843DOI
  7. [7]J. J. Moré, S. M. Wild, Do you trust derivatives or differences?, Journal of Computational Physics 273 (2014) 268–277. doi:10.1016/j.jcp.2014.04.056. URL https://linkinghub.elsevier.com/retrieve/pii/S0021999114003325DOI
  8. [8]S. T. K. Lang, A. Dawson, M. Diamantakis, P. Dueben, S. Hatfield, M. Leutbecher, T. Palmer, F. Prates, C. D. Roberts, I. Sandu, N. Wedi, More accuracy with less precision, Quarterly Journal of the Royal Meteorological Society 147 (741) (2021) 4358–4370, eprint: https://rmets.onlinelibrary.wiley.com/doi/pdf/10.1002/qj.4181. doi:10.1002/qj.4181. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/qj.4181DOI
  9. [9]S. Hatfield, A. Subramanian, T. Palmer, P. Düben, Improving Weather Forecast Skill through Reduced-Precision Data Assimilation, Monthly Weather Review 146 (1) (2018) 49–62. doi:10.1175/MWR-D-17-0132.1. URL https://journals.ametsoc.org/view/journals/mwre/146/1/mwr-d-17-0132.1.xmlDOI
  10. [10]G. Freytag, J. V. F. Lima, P. Rech, P. O. A. Navaux, Impact of Reduced and Mixed-Precision on the Efficiency of a Multi-GPU Platform on CFD Applications, in: O. Gervasi, B. Murgante, S. Misra, A. M. A. C. Rocha, C. Garau (Eds.), Computational Science and Its Applications – ICCSA 2022 Workshops, Springer International Publishing, Cham, 2022, pp. 570–587. doi:10.1007/978-3-031-10542-5_39.DOI
  11. [11]M. Karp, R. Stanly, T. Mukha, L. Galimberti, S. Toosi, H. Song, L. Dalcin, S. Rezaeiravesh, N. Jansson, S. Markidis, M. Parsani, S. Bose, S. Lele, P. Schlatter, Effects of lower floating-point precision on scale-resolving numerical simulations of turbulence, Journal of Computational Physics 549 (2026) 114600. doi:10.1016/j.jcp.2025.114600. URL https://linkinghub.elsevier.com/retrieve/pii/S0021999125008824DOI
  12. [12]B. Siklósi, P. K. Sharma, D. J. Lusher, I. Z. Reguly, N. D. Sandham, Reduced and mixed precision turbulent flow simulations using explicit finite difference schemes, Future Generation Computer Systems 175 (2026) 108111. doi:10.1016/j.future.2025.108111. URL https://www.sciencedirect.com/science/article/pii/S0167739X25004054DOI
  13. [13]L. Giraud, J. Langou, M. Rozloznik, The loss of orthogonality in the Gram-Schmidt orthogonalization process, Computers & Mathematics with Applications 50 (7) (2005) 1069–1075. doi:10.1016/j.camwa.2005.08.009. URL https://www.sciencedirect.com/science/article/pii/S0898122105003366DOI
  14. [14]F. Sapienza, J. Bolibar, F. Schäfer, B. Groენke, A. Pal, V. Boussange, P. Heimbach, G. Hooker, F. Pérez, P.-O. Persson, C. Rackauckas, Differentiable Programming for Differential Equations: A Review, arXiv:2406.09699 [math] (Oct. 2025). doi:10.48550/arXiv.2406.09699. URL http://arxiv.org/abs/2406.09699
  15. [15]G. Corliss, C. Faure, A. Griewank, L. Hascoët, U. Naumann (Eds.), Automatic Differentiation of Algorithms: From Simulation to Optimization, Springer New York, New York, NY, 2002. doi:10.1007/978-1-4613-0075-5. URL http://link.springer.com/10.1007/978-1-4613-0075-5DOI
  16. [16]A. Griewank, D. Juedes, J. Utke, Algorithm 755: ADOL-C: a package for the automatic differentiation of algorithms written in C/C++, ACM Trans. Math. Softw. 22 (2) (1996) 131–167. doi:10.1145/229473.229474. URL https://dl.acm.org/doi/10.1145/229473.229474
  17. [17]L. Hascoet, V. Pascual, The Tapenade automatic differentiation tool: Principles, model, and specification, ACM Trans. Math. Softw. 39 (3) (2013) 20:1–20:43. doi:10.1145/2450153.2450158. URL https://dl.acm.org/doi/10.1145/2450153.2450158DOI
  18. [18]T. F. Coleman, A. Verma, The Efficient Computation of Sparse Jacobian Matrices Using Automatic Differentiation, SIAM Journal on Scientific Computing 19 (4) (1998) 1210–1233. doi:10.1137/S1064827595295349. URL http://epubs.siam.org/doi/10.1137/S1064827595295349DOI
  19. [19]W. Xu, T. F. Coleman, Efficient (Partial) Determination of Derivative Matrices via Automatic Differentiation, SIAM Journal on Scientific Computing 35 (3) (2013) A1398–A1416. doi:10.1137/11085061X. URL http://epubs.siam.org/doi/10.1137/11085061XDOI
  20. [20]W. S. Moses, S. H. K. Narayanan, L. Paehler, V. Churavy, M. Schanen, J. Hüсkelheim, J. Doerfert, P. Hovland, Scalable automatic differentiation of multiple parallel paradigms through compiler augmentation, in: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC ’22, IEEE Press, Dallas, Texas, 2022, pp. 1–18. URL https://dl.acm.org/doi/10.5555/3571885.3571964DOI
  21. [21]P. D. Hovland, L. C. McInnes, Parallel simulation of compressible flow using automatic differentiation and PETSc, Parallel Computing (2001) 503–519.DOI
  22. [22]A. Lindsay, R. Stogner, D. Gaston, D. Schwen, C. Matthews, W. Jiang, L. K. Aagesen, R. Carlsen, F. Kong, A. Slaughter, et al., Automatic differentiation in metaphysical and its applications in moose, Nuclear Technology (2021) 1–18.DOI
  23. [23]M. Mayr, A. Heinlein, C. A. Glusa, S. Rajamanickam, M. Arnst, R. A. Bartlett, L. Berger-Vergiat, E. G. Boman, K. D. Devine, G. Harper, M. A. Heroux, M. Hoemmen, J. J. Hu, B. Kelley, D. P. Kouri, P. Kuberry, K. Kim, K. Liegois, C. C. Ober, R. P. Pawlowski, C. Pearson, M. Perego, E. T. Phipps, D. Ridzal, N. V. Roberts, C. M. Siefert, H. K. Thornquist, R. Tomasetti, C. R. Trott, R. S. Tuminaro, J. M. Willenbring, M. Wolf, I. Yamazaki, Trilinos: Enabling Scientific Computing across Diverse Hardware Architectures at Scale, ACM Transactions on Mathematical Software published online ahead of print (2026). doi:10.1145/3802822. URL https://doi.org/10.1145/3802822
  24. [24]M. Raissi, P. Perdikaris, G. E. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational Physics 378 (2019) 686–707. doi:10.1016/j.jcp.2018.10.045. URL https://www.sciencedirect.com/science/article/pii/S0021999118307125DOI
  25. [25]J. Cho, S. B. Yun, S. Nam, Y. Hong, H. Yang, E. Park, Separable Physics-Informed Neural Networks: 37th Conference on Neural Information Processing Systems, NeurIPS 2023, Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023 (2023). URL https://www.scopus.com/pages/publications/85182242601arxiv.org/abs/2306.15969
  26. [26]S. Markidis, The Old and the New: Can Physics-Informed Deep-Learning Replace Traditional Linear Solvers?, Frontiers in Big Data 4 (2021) 669097. doi:10.3389/fdata.2021.669097. URL https://www.frontiersin.org/articles/10.3389/fdata.2021.669097/fullDOI
  27. [27]P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, İ. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, SciPy 1.0 Contributors, SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python, Nature Methods 17 (2020) 261–272. doi:10.1038/s41592-019-0686-2.DOI
  28. [28]R. Okuta, Y. Unno, D. Nishino, S. Hido, C. Loomis, Cupy: A numpy-compatible library for nvidia gpu calculations, in: Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Thirty-first Annual Conference on Neural Information Processing Systems (NIPS), 2017. URL http://learningsys.org/nips17/assets/papers/paper_16.pdf
  29. [29]J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, Y. Katariya, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, Q. Zhang, JAX: composable transformations of Python+NumPy programs (2018). URL http://github.com/jax-ml/jax
  30. [30]J. I. Castor, Radiation hydrodynamics, Cambridge University Press, Cambridge, 2004, meeting Name: Summer School on Radiative Transfer and Radiation Hydrodynamics.
  31. [31]E. D. Dolan, J. J. Moré, Benchmarking optimization software with performance profiles, Mathematical Programming 91 (2) (2002) 201–213. doi:10.1007/s101070100263. URL https://doi.org/10.1007/s101070100263

Paper details

vlnc:W_tgu3849g
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents