WorksEngineering
Implicit DeepReach Without Implicit Differentiation
Abstract
We consider the problem of approximating backward reachable and reach-avoid tubes for high-dimensional nonlinear systems whose Hamiltonians require numerical optimization over constrained controls and disturbances. Existing neural Hamilton-Jacobi (HJ) reachability methods alleviate the dimensionality bottleneck of grid-based solvers, but evaluating Hamiltonians without closed-form expressions introduces an additional computational challenge. We show that, under suitable assumptions, the gradients required for value-function training can be computed without differentiating through the inner optimization problem. Specifically, we establish an envelope-type gradient identity for the saddle point Hamiltonian, under which the optimal control and disturbance enter backpropagation as constants without gradient tracking. This enables end-to-end neural HJ reachability with numerically evaluated Hamiltonians while avoiding both solver unrolling and implicit differentiation. We demonstrate the accuracy and computational efficiency of the proposed approach on nonlinear reachability and reach-avoid problems of different scales.
Introduction
Hamilton–Jacobi (HJ) reachability provides a principled framework for reasoning about safety and robustness in dynamical systems. Given a target or unsafe set, reachability analysis characterizes the states from which an adversarial disturbance can force the system into the target despite the action of a controller [33, 17]. The resulting backward reachable tube (BRT) provides both a safety certificate and an associated feedback controller, making HJ reachability a powerful tool for safety-critical control.
A primary challenge in HJ reachability is the curse of dimensionality. Classical methods solve the associated Hamilton-Jacobi-Isaacs (HJI) equation or variational inequality on a state-time grid [34, 21], these methods are accurate but restricted to relatively low-dimensional systems. Neural approaches such as DeepReach [2] alleviate this bottleneck by representing the value function with a neural network and enforcing the HJI equation at sampled state-time points. Subsequent work has further improved the scalability and accuracy of this paradigm through additional supervision and guidance [40, 42].
In this work, we focus on a separate bottleneck that arises when the Hamiltonian of the system does not admit a closed form solution and is instead defined implicitly by a saddle point problem. While optimal control and disturbance inputs can admit closed-form expressions in some standard reachability problems, more general nonlinear dynamics, coupled control–disturbance interactions, and complex admissible input sets generally require solving a numerical optimization problem to evaluate the Hamiltonian [5, 11 b]. Recent approaches address such Hamiltonians through learned surrogates, data-driven approximations, or conservative bounds [7, 27] (Chilakamarri et al. 2025). These methods make the Hamiltonian more tractable before or outside the value-function computation, but they usually require additional computation or data collection and can introduce approximation error.
An alternative is to retain the numerical Hamiltonian optimization directly within neural HJI training, as explored in related implicit deep learning approaches for optimal control [12]. However, embedding the optimization into the training loop introduces an additional differentiation bottleneck. Each value-function update requires repeatedly solving the pointwise control-disturbance game, and conventional differentiation approaches propagate gradients through its solution. Unrolling the inner solver can incur substantial memory and computational overhead, while exact implicit differentiation typically requires solving an additional linear system for each Hamiltonian evaluation. In this work, we show that neither approach is necessary. By establishing an envelope-type gradient identity for the saddle point Hamiltonian, we obtain the gradients required for neural HJI training without differentiating through the inner optimization. In particular, our main contributions are: (i) End-to-end HJ reachability with implicit Hamiltonians. We develop a neural HJ reachability framework that directly incorporates numerically solved control-disturbance games into value-function training, enabling the treatment of nonlinear systems without tractable closed-form Hamiltonians. (ii) Theoretically justified gradients without optimizer sensitivities. Using a saddle point envelope argument, we establish exact gradient formulas under light assumptions, eliminating the need for solver unrolling or implicit differentiation through the inner optimization. (iii) Numerical validation. We demonstrate the accuracy and scalability of our approach on nonlinear reachability and reach-avoid problems in dimensions ranging from 2 to 50, showing close agreement with reference solutions and reliable closed-loop performance.
Background
Hamilton-Jacobi Reachability
Consider the controlled dynamical system over a finite time horizon
where is the state, is the control, and is an adversarial disturbance with denoting an arbitrary initial time and is a fixed terminal time. Let the target set be represented by
for a sufficiently regular target function . The target set can represent either a set of goal states or unsafe states, and is typically chosen as a signed distance function.
The Backward Reachable Tube (BRT) ([33, 2]) of a given problem consists of states from which the disturbance can force the system into at some time over despite the action of the controller. This is characterized via the value function
and the BRT is given by the zero sublevel set of the value function for any given , that is,
Define the HJI Hamiltonian
where is the adjoint variable to the dynamics. Under standard regularity assumptions and the Isaacs condition, the value function satisfies the HJI variational inequality (VI), which reads
with terminal condition for all . Crucially, the VI formulation accounts for the possibility of reaching the target set at any time within the horizon, rather than only at the terminal time, distinguishing it from the standard HJI PDE.
Similarly, we can define the Backward Reach-Avoid Tube (BRAT) ([9]), which characterizes states from which the controller can reach the target set despite adversarial disturbances, while avoiding an unsafe set , defined as
Here is also typically chosen as a signed distance function describing obstacles or unsafe states of the system, such as joint positions that violate a robot’s safety requirements. We note for goal-reaching problems, the roles of the control and disturbance in the Hamiltonian are reversed. Nonetheless, the BRAT can be similarly defined as the zero sublevel set of the corresponding value function , which satisfies a slightly different variational inequality
with terminal condition . Here the outer maximum enforces the state constraint by excluding trajectories that enter the unsafe set during the horizon. The BRAT , can be recovered similarly following (4).
Neural HJ Reachability
To alleviate the curse of dimensionality of grid-based HJ solvers, neural approaches such as DeepReach ([2]) approximate the value function using a neural network parameterized by . Following the physics-informed neural network (PINN) paradigm ([37, 45]), the network is trained by minimizing the HJI variational inequality residual and enforcing the terminal condition at sampled state-time points. Specifically, the training objective takes the form
where
Here, and denote the sampling distributions over interior state-time points and terminal states, respectively, and balances the two loss terms. The formulation above corresponds to the BRT problem; the BRAT loss follows analogously using the reach-avoid variational inequality.
A key requirement of these approaches is the repeated evaluation of the Hamiltonian during training. Existing neural HJ methods typically consider settings in which the saddle point problem admits a closed-form solution. However, for more complex dynamics, determining the optimal control and disturbance may require numerical optimization. Addressing the resulting differentiability and computational cost bottlenecks remains an open challenge for neural HJ methods.
Neural HJ Reachability with Implicit Hamiltonians
As discussed in § 2, existing neural HJ reachability methods typically rely on Hamiltonians that admit closed-form evaluations. For general nonlinear dynamics with constrained controls and disturbances, the Hamiltonian is defined implicitly through a saddle point optimization problem. Under a standard projected gradient ascent-descent (PGDA) formulation, the inner optimization can be recast as the fixed-point problem over with each iteration defined as
Here, and denote projections onto the admissible sets, and are the step sizes. Under suitable convex–concave assumptions PGDA can provably converge, more general saddle point problems may require alternative algorithms [19, 23, 30]. Regardless of the solver, differentiating through the inner optimization introduces an additional computational bottleneck in neural HJI training. Two conventional approaches exist: (i) Automatic Differentiation (AD): Unrolling the inner solver and backpropagating through its iterations, which can incur substantial computational and memory overhead especially as problem size and number of inner iterations increase. (ii) Implicit Differentiation: Using the implicit function theorem (IFT) [28] it is possible to compute the sensitivities of the saddle point directly under
| Algorithm 1: | Gradient-free neural HJ reachability with implicit Hamiltonians. |
| Input: | Dynamics , target , value network , learning rate |
| Output: | Trained value function |
| while not converged do | |
| Sample | |
| for do | |
| end | |
| end |
Table 1.
Here we assume an interior saddle point for simplicity. While efficient, boundary projections can lead to singular or ill-conditioned Jacobians and subsequent numerical instabilities, especially in large-scale problems, as demonstrated in [25]. Both approaches introduce additional differentiation costs on top of the repeated saddle point solves required during training, rendering them impractical at scale.
In this work, we show that differentiating through the inner optimization is unnecessary. Leveraging a saddle point envelope type argument, we establish exact gradients for neural HJI training without differentiating through the inner solver, which has also been observed in the optimal transport setting Amos [1]; Park et al. [35]. We state the formal theory in § 3.1 and present the resulting end-to-end training algorithm in § 3.2.
Theoretical Analysis
We now establish an envelope type gradient identity that enables neural HJ reachability training without differentiating through the inner saddle point optimization.
Notation. We write , and use and interchangeably. Let and , so that by (5).
Assumptions. For an arbitrary parameter , we assume: (A1) and are nonempty, compact and convex; (A2) is ; (A3) the mapping is differentiable at ; (A4) there exists such that, for every , the Isaacs condition holds; (A5) the function has a unique saddle point on , which we denote by .
Theorem 1 Under (A1)–(A5), the mapping is differentiable at , with
In other words, the extra term that would result from chain rule does not appear. Thus, the saddle point can be treated as a constant during backpropagation, with no gradients propagated through or the solver that computes it.
We defer the proof of Theorem 1 to Appendix Section E, numerical validation of Theorem 1 is also provided in Appendix Section C.
Remark. We highlight several important properties of Theorem 1. (i) Assumptions (A1)–(A5) are compatible with standard HJ reachability settings involving compact convex input constraints, sufficiently regular dynamics, and locally well-defined saddle points. In particular, the result does not require strong convexity-concavity or second-order nonsingularity conditions crucial for IFT. (ii) Theorem 1 applies pointwise at any satisfying these assumptions and therefore extends throughout training wherever they continue to hold. We validate this numerically and measure the effect of inexact inner solves in Appendix Section C, enabling our proposed Algorithm 1. (iii) The result applies to arbitrary compact and convex control and disturbance sets, beyond simple box constraints.
Moreover, the gradient identity is independent of the numerical saddle point solver. (iv) Finally, the result does not require differentiability of the optimal control and disturbance mappings with respect to the network parameters. Consequently, the gradient identity remains valid for saddle points in the interior or on the boundary of the admissible sets.
Gradient-Free Training for Neural HJ Reachability with Implicit Hamiltonians
Following the envelope identity established in § 3.1, we develop an end-to-end neural HJ training framework that incorporates numerically evaluated Hamiltonians without differentiating through the inner saddle-point solver. We describe its main implementation components below.
Exact terminal condition enforcement through network parameterization. Following Singh et al., [41], we enforce the terminal condition by parameterizing the value function as
where denotes the neural network output. Unlike Singh et al., [41], we opt for an exponential decay to reduce the influence of the nonsmooth terminal function away from , which empirically we observe improves training. This parameterization satisfies the terminal condition exactly, eliminating the need for the terminal loss (11).
Saddle point solver. We primarily employ PGDA to solve the inner saddle point problem, using adaptive step sizes based on the Barzilai–Borwein (BB) method Barzilai and Borwein, [4] to improve practical convergence. The step sizes are updated as
Here, and are predefined bounds for numerical stability, and step sizes are updated independently for each sample. This provides a computationally inexpensive adaptive step-size selection, particularly useful when the saddle point objective is relatively flat. We evaluate its effectiveness against a fixed step size solver in Section C. We note that while PGDA convergence depends on the structure of the underlying game, the proposed training framework is independent of the saddle point solver and can accommodate any alternative numerical methods.
Combining the exact terminal-condition parameterization with gradient-free Hamiltonian evaluation yields the end-to-end training procedure summarized in Algorithm 1. After training, the BRT is recovered from the zero sublevel set of , and the corresponding feedback policy can be obtained using the same saddle point solver. For conciseness, Algorithm 1 is written in BRT formulation; the BRAT extension follows analogously.
Experiments
We consider three numerical experiments, covering both BRT and BRAT problems, with state dimensions from 2 to 50. In all cases, the Hamiltonians are implicitly defined and require iterative solvers to compute the saddle points, and optimal saddle points occur both in the interior and on the
caption: Learned and reference value functions for the energy-constrained pursuit-evasion problem. Left: Learned ; Middle: grid-based reference; Right: BRT volume differences.
boundaries of the admissible sets, providing sufficient numerical evidence supporting our approach. All experiments were conducted on a single NVIDIA RTX 5090 GPU. Code will be made available upon publication of the work. We refer to Appendix Section B and Section D for additional details.
Energy Constrained Pursuit–Evasion Example
We first consider a two-player zero-sum differential game with state , where denotes the relative distance between the players and their relative energy resource, with indicating an advantage for the evader. The evader and pursuer select controls , respectively, under the dynamics
The target set is defined by , with terminal function , the target set is reached when either the relative distance falls below the given threshold or the evader exhausts its resource advantage.
Given the finite time horizon , the BRT can be defined as in § 2, the Hamiltonian is separable in and , allowing their optimal values to be computed independently from their respective stationarity conditions
subject to the input constraints, with and . The Hamiltonian is provably convex-concave over the input domain, and the separable structure enables comparison against a grid-based level-set solver while retaining nontrivial numerical optimization of the optimal inputs.
We refer to Appendix Section D for details on model parameterization and training details. As shown in Figure 1, the learned value function closely matches the grid-based reference solution, achieving a training loss on the order of and a BRT volume error of only 0.11%.
Extending to High Dimensions
We extend the 2D energy-constrained pursuit-evasion example to independent evader-pursuer pairs, with joint state and dynamics
The target set is defined by , where so that the target is reached whenever any pair violates its separation or energy threshold.. We refer to Appendix Section B for the complete problem formulation.
We consider 10D and 50D extensions to evaluate scalability of our proposed method. As shown in Figure 5, training converges in both cases, with increasing residual losses at higher dimensions given the added complexity. To assess the learned BRTs without grid-based reference solutions, we perform a closed-loop consistency test on 2048 randomly sampled initial states for each problem, comparing BRT predictions with rollouts under the learned saddle-point policy, similar to that of [2]. The column-normalized confusion matrices in Figure 2 show that agreement remains high, particularly for states predicted to be unreachable (99.1% in 10D and 96.4% in 50D). Agreement for predicted-reachable states decreases to 94.7% and 85.4%, respectively, indicating greater classification disagreement as dimensionality increases.

Figure 2. Closed-loop consistency of learned BRT predictions across state dimensions, quantifying mismatches between BRT predictions and closed-loop rollouts. Confusion matrices are column-normalized by predicted class.

Figure 5. Training loss curves for the energy-constrained pursuit-evasion problems in 2D, 10D, and 50D. The losses decrease and stabilize across all three settings, demonstrating convergence of our proposed neural HJ training procedure. Higher-dimensional problems exhibit larger final residual losses.
Two-Vehicle Collision Avoidance in 6D
We consider a two-vehicle collision avoidance problem based on Bansal and Tomlin [2], with joint state . Unlike the standard constant speed setting, we introduce steering-dependent speed for the evader, resulting in the modified dynamics
where denote the evader’s and pursuer’s turn-rate controls, respectively. This modification introduces nonlinear control dependence into the Hamiltonian, requiring numerical optimization of the evader’s optimal control. We follow the capture target formulation of Bansal and Tomlin [2], with the complete problem definition given in Appendix Section B.
Training yielded a MSE loss of approximately . We note although the model is trained directly in the full 6D state space, a reference solution can be computed in 3D using relative coordinates, exploiting the translation and rotation symmetries of the dynamics. As shown in Figure 3, the learned BRT closely matches a grid-based reference solution across multiple two-dimensional slices, with slightly conservative predictions. Importantly, we obtain the desired results without enforcing said symmetries during neural HJ training.

Figure 3. Learned and reference BRTs across relative-heading slices, showing close agreement with a grid-based reference solution.
We further evaluate the learned value function through closed-loop trajectory rollouts in Figure 6 in the Appendix, using the same numerical Hamiltonian optimization as during training. The trajectories demonstrate both successful evasion and capture, with the inner solver achieving residual errors below across the tested scenarios. The steering-dependent speed also changes the reachable set compared with the constant speed setting: by reducing its speed during rapid turning, the evader can escape from certain configurations that would otherwise lead to capture. The trajectories also illustrate how initially reachable states may leave the BRT over time.

Figure 6. Closed-loop trajectories for the two-vehicle collision avoidance problem, using feedback controls obtained from the learned value function and online Hamiltonian optimization. Left & Right: successful evasion; Middle: capture. The learned value function is overlaid along each trajectory to illustrate its temporal evolution.
Quadrotor Reach-Avoid Control under Adversarial Disturbances
We consider a 6D planar quadrotor reach-avoid problem under adversarial wind disturbances. The quadrotor is controlled through two bounded rotor thrusts and must reach a prescribed target region while avoiding ground collision and excessive pitch angles. The wind disturbance enters through nonlinear velocity-dependent drag, requiring numerical optimization to determine the worst case disturbance. We refer to Appendix Section B for the complete dynamics and problem formulation.
To address training difficulties arising from the nested BRAT variational inequality, we adopt a curriculum learning strategy common in PINNs literature [2, 20, 44], where we gradually extend the sampled time horizon from to during training. We evaluate the learned value function through a closed-loop consistency test on 2048 randomly sampled initial states at . As shown in Table 1, the rollout outcomes agree with the predicted reach-avoid classifications in over 90% of cases for both predicted classes, with most discrepancies occurring near the zero level set. We further visualize representative trajectories in Figure 4, where feedback controls and worst-case disturbances derived from the learned value function produce outcomes consistent with the predicted BRAT.

Figure 4. Rollouts for quadrotor control under disturbances using the learned . Successful and failed trajectories are shown, with outcomes consistent with the BRAT predictions.
| Reached | Crashed | Timeout | Total | ||
| All | 837 (90.8%) | 71 (7.7%) | 14 (1.5%) | 922 | |
| Boundary | 46 | 44 | 7 | 97 | |
| Confident | 791 | 27 | 7 | 825 | |
| All | 62 (5.5%) | 911 (80.9%) | 153 (13.6%) | 1126 | |
| Boundary | 24 | 53 | 4 | 81 | |
| Confident | 38 | 858 | 149 | 1045 |
Table 1. Closed-loop consistency of BRAT predictions using learned value function. “Boundary” denotes . Both crashed and timeout cases have .
Conclusion
We presented an end-to-end neural HJ reachability framework for nonlinear systems with implicitly defined Hamiltonians. Using a perturbation argument based on saddle-point inequalities, we prove that gradients can be computed without differentiating through the inner optimization, avoiding the overhead and numerical instability under solver unrolling or implicit differentiation. Experiments on both reachability and reach-avoid problems demonstrate the effectiveness and scalability of our approach, extending neural HJ methods to a much broader and more practical class of problems.
Related Work
We provide a review of the literature on neural HJ reachability and subsequent works that extend these methods to more complex and challenging problems. Additionally, we provide an overview of recent advances in methods for handling implicit Hamiltonians.
High-dimensional neural HJ reachability. DeepReach [2] introduced a mesh-free neural approach for solving HJ reachability problems in dimensions beyond the range of conventional grid-based solvers. Rather than discretizing the state space, DeepReach directly parameterizes the value function and minimizes the HJI variational inequality residual. Subsequent approaches have improved the scalability and training of neural HJ solvers. Sharpless et al. [40] augment the HJI loss with supervision generated from tractable linearized differential games and demonstrate neural value-function learning for systems with tens of state dimensions. Similarly, Feng et al. [8] and MADR [42] leverage MPC-based supervision to improve neural HJ reachability learning. These methods train the HJ value function end-to-end, but assume that the Hamiltonian required by the HJI residual can be evaluated efficiently. On the theoretical side, Hofgard [16] establishes conditional convergence guarantees for DeepReach, while Lin and Bansal [26] provide formal safety assurances through error bounds and reachable-tube corrections. Related theoretical work also studies neural approximation of HJ-type PDEs [38].
HJ reachability without closed-form Hamiltonians. Several recent works address HJ reachability problems in which the Hamiltonian is difficult to evaluate or unavailable in closed form. (Chilakamarri et al., 2025) learn a neural Hamiltonian surrogate from black-box dynamics samples for subsequent use in HJ solvers. [7] construct a conservative data-driven Hamiltonian directly from trajectory data, while [27] extend HJ-based verification to complex perception-based controllers under state uncertainty. These approaches make a difficult Hamiltonian tractable through approximation or bounding before it is used by the value function solver. In contrast, we retain the numerical optimization defining the Hamiltonian as an explicit component of neural HJI training and differentiate the value-function objective through its solution.
Implicit Hamiltonians in Control. Implicit optimization layers have been studied broadly in machine learning and differentiable optimization, with Jacobian-Free Backpropagation (JFB), which provides an inexpensive alternative to exact implicit differentiation [10, 15, 14, 31 b]. In control, [25] combine operator splitting with JFB to enable scalable end-to-end training through embedded CBF-QP safety layers, which demonstrates that Jacobian-free differentiation can substantially reduce the computational cost of repeatedly differentiating through optimization problems along control trajectories. More closely related to our setting, [11] develop an end-to-end method for high-dimensional optimal control in which the Hamiltonian minimization is solved implicitly and differentiated using JFB. Subsequent work establishes convergence guarantees for the resulting approximate-gradient training procedure [12]. Imitation-based function-encoder policies instead sidestep the Hamiltonian, transferring zero-shot across tasks [24]. As shown in § 3.1, the particular structure of the HJ reachability problem allows for exact gradient evaluation of the training objective without requiring approximate differentiation techniques, providing a more direct and natural alternative to existing approaches for handling implicit Hamiltonians.
Additional Experimental Details
This appendix provides supplementary problem formulations, Hamiltonian derivations, and implementation details for the numerical experiments presented in § 4.
High-Dimensional Energy-Constrained Pursuit–Evasion
We extend the two-player pursuit–evasion problem in § 4 to independent evader-pursuer pairs, with joint state . For each pair , and denote the relative distance and energy resource, respectively, with controls and . The dynamics are
where may vary across pairs to model heterogeneous maneuverability and energy consumption rate. The target set consists of states in which at least one pair violates its separation or energy threshold, and is represented by the terminal function
The corresponding target set is
Since the individual evader–pursuer pairs are dynamically decoupled, the Hamiltonian separates as , where and . Thus, the optimal control and disturbance can be computed independently for each pair using the same numerical optimization procedure as in the 2D case. We also note that under suitable and we can preserve the convex-concave structure of the Hamiltonian, satisfying the necessary assumptions for the theory.
As the problem dimension increases, the reachable set occupies a larger portion of the state space, leading to an imbalance between reachable and unreachable states under uniform sampling. To address this imbalance and stabilize training, we employ importance sampling to ensure that decreasing training loss leads to improved closed-loop performance.
Two-Vehicle Collision Avoidance
We consider a finite-horizon collision avoidance problem between two planar vehicles. The joint state is , where and denote the positions and headings of the pursuer and evader, respectively. We modify the standard constant-speed dynamics [2] by introducing steering dependent speed for the evader:
where denote the evader’s and pursuer’s turn-rate controls, respectively. The evader’s speed decreases with the magnitude of its turn rate, introducing nonlinear control dependence into the Hamiltonian. For the time horizon , the capture target is defined by
with . The corresponding Hamiltonian is
where . While the pursuer’s optimal turn rate admits a closed-form solution, the evader’s optimal control is determined through an inner optimization problem due to the nonlinear dependence of the Hamiltonian on its turn rate.
We further observe that the curvature of the evader’s maximization objective is determined entirely by the sign of the term . Specifically, the objective is strongly concave for and strongly convex for over the admissible control set. Consequently, the global maximizer either lies on the domain boundary or can be provably recovered through methods such as projected gradient ascent. Finally, we use the computational domain
with state-time samples drawn uniformly during training.
Quadrotor Reach-Avoid Control under Adversarial Disturbances
We consider a six-dimensional planar quadrotor subject to adversarial wind disturbances, with dynamics and constraints specified below. Let denote the quadrotor’s position, velocity, pitch angle and pitch rate, respectively. The two thrusters of the rotor can be controlled directly via , the adversary chooses wind velocity . The quadrotor operates under standard dynamics as in [29, 39], which are given by
Here kg is the mass, kg m is pitch inertia, m is the rotor’s arm length, we use m/s to denote gravity. We also set the maximum thrust to N and the maximum wind speed to m/s. While the rotor dynamics are affine in control, the wind disturbance enters via a nonlinear drag force which we define here as
Here the force model combines linear and quadratic drag based on relative air velocity, while not the highest fidelity aerodynamics model, it is still commonly used for simulation and analysis ([13, 22]). For our example we consider coefficients and . The terminal time is set to s. For simplicity, we consider a box region over which we sample and solve the resulting BRAT problem, which we define as .
The reach target of the problem is a boxed region over and at , we define the reach and avoid set functions as
and subsequently the target and unsafe sets as
following the conventions established in § 2. We note that the unsafe set encodes two failure scenarios: the first occurs when the quadrotor reaches the ground, and the second occurs when the pitch angle exceeds the prescribed safety limit, potentially leading to a loss of stability and a drone flip-over. We place no direct target restrictions on , although a successful arrival at the target must nevertheless satisfy . The optimal wind disturbance is determined by the constrained optimization problem
Under our choice of parameters, the maximization admits a unique solution for every state and adjoint, though obtaining always requires an iterative solver, making this a challenging yet suitable testbed for our proposed approach.
Additional Numerical Results
Numerical Verification of The Theory
We provide numerical verification of our main theoretical results § 3.1 here in this section. We mainly consider the the 2D energy constrained pursuit–evasion problem as the example to focus on. Our focus is to validate the accuracy of the training gradient under our proposed approach in saddle point optimization. For simplicity we apply steps of projected gradient ascent/descent with fixed step size of 0.1. Early stopping is disabled to ensure a fair comparison. We compare two different scenarios:
Gradient Free (ours): the inner optimization is untracked and only the obtained saddle point is supplied to the loss function.
Automatic Differentiation (AD): each inner optimization step is fully tracked and unrolled during backpropagation.
For value function construction we consider two different parameterizations: (i) a polynomial parameterization for the value function with sensitivity under control, and (ii) a neural network with the same size used in our numerical testing. For both cases we initialize the value functions so that the adjoints remain strictly positive,thus ensuring the min-max structure and a strictly interior saddle point. for each testing case, we sample with batch size 1024 uniformly over the domain. We report gradient consistency results for both test cases under different solver iterations and display the results in Table 2. The gradient accuracy over the trainable weights is governed by the accuracy of the inner solve, and vanishes with it. As increases, the Hamiltonian residuals and decrease, and decreases with them. For the polynomial the solve converges to machine precision by , and the two gradients coincide exactly at . The neural network converges more slowly because under random initialization its adjoint is small and the Hamiltonian is only weakly curved in the controls. Nevertheless the gradient discrepancies are negligible at for both cases especially regarding the gradient norms. For cases where the saddle points lie on the boundaries, the results follow trivially. Specifically, when evaluated at the exact boundary, the gradient difference is exactly 0, since no fixed-point error needs to be accounted for. Overall the numerical results fully verifies our theoretical claims.
| Polynomial | Neural Network | |||||
| 1 | ||||||
| 2 | ||||||
| 5 | ||||||
| 10 | ||||||
| 20 | ||||||
| 50 | ||||||
| 100 | ||||||
| 200 | ||||||
| 500 | 0 |
Table 2. Gradient consistency test under different value function parameterizations and solver accuracies. Here we define . All terms are in 2-norm unless otherwise specified. Gradient norm for the polynomial case is and for the neural network case .
Inner Solver Comparison
The inner saddle point problem (5) is solved approximately at every training step. While our theory in § 3.1 ensures gradient consistency at the fixed point, the quality of the truncated solve can still affect training. We hereby conduct an ablation study and compare between fixed and adaptive step size selection in the saddle point optimization under the same gradient descent rule. We test on the 10D energy constrained pursuit–evasion example, where both optimizers are given the same budget of 200 iterations. Two copies of the value network are trained under exactly the same sampling and with identical weight initialization; hence, any difference in training is due to the inner solver alone. We also keep a reference solution for the fixed point solver, which uses PGDA with a fixed step size of 0.1 and 5000 iterations. This achieves a median fixed point error of at each sampling mini-batch and serves as a ground truth reference when comparing gradient accuracies. We run a total of 10000 training steps for the test.
We present the results in Table 3. Under the same iteration budget, adaptive step sizing yields a substantially more accurate inner solve than a carefully selected fixed step size. Importantly, the computational overhead is marginal, allowing us to maintain the fast training required for neural HJ reachability at large. However, we note that despite the lower solver accuracy, using plain PGDA
| % of stationary samples* | Hamiltonian gap | Gradient alignment | Final Training loss | Per training step solver time | |
| Fixed | 22% | 4.8% | 0.965 | 0.214s | |
| Adaptive | 98.9% | 0.4% | 0.999 | 0.247s |
Table 3. Inner solver accuracy comparison between fixed and adaptive step size selection. Hamiltonian gap and gradient alignment measure relative value and gradient errors against the reference solution. Numbers averaged across 10K training steps except for the final loss.
with a fixed step size does not lead to significant errors in either the gradient updates or the overall training loss behavior. We attribute this to the stochasticity of Adam and mini-batch training in general. The key takeaway we want to highlight is that convergence of the value function can still be expected even when the inner solve admits a small error.
Training Setup and Details on Numerical Experiments
We provide details on the training setup used in this paper. We consider a total of five different experiments. In all examples the value function is defined per (14), with being a residual MLP with tanh activations and terminal condition enforced exactly by construction. Training only minimizes the VI residual at sampled locations. For all examples except for the quadrotor, we employ a uniform sampling scheme over the state-time space. We train all networks with Adam optimizer [18] with an initial learning rate of . The learning rate is halved whenever training hits a plateau, the patience for which we set differently for each example. The default solver for the inner optimization is PGDA with Barzilai–Borwein adaptive step sizing, as introduced in § 3 with initial step size . We terminate early whenever the fixed-point residual reaches . The maximum number of iterations for the inner optimization is set separately for each individual example. We try to balance between solver accuracy and speed during training. All code is implemented in PyTorch [36], and all runs use random seed 0; problem-specific settings are listed in Table 4.
| Example | Training Steps | Batch Size | Network Size (Width, # of Layers) | LR patience | Inner Iters. | |
| Energy Constrained Pursuit–Evasion (2D) | 1.5 | 30k | 256 | (128, 3) | 2000 | 100 |
| Energy Constrained Pursuit–Evasion (10D) | 1.5 | 50k | 256 | (128, 4) | 3000 | 200 |
| Energy Constrained Pursuit–Evasion (50D) | 1.5 | 100k | 256 | (256, 4) | 5000 | 500 |
| Two-Vehicle Collision Avoidance | 1.0 | 50k | 1024 | (128, 4) | 3000 | 100 |
| Planar Quadrotor (BRAT) | 2.0 | 70k | 1024 | (256, 4) | 3000 |
Table 4. Problem-specific training settings. Inner iterations are the maximum number of iterations of the inner solver.
Proof of Theorem
For completeness, we restate the assumptions and main theorem from § 3.1 and provide its detailed proof, together with the supporting lemmas.
Notations. Write . We denote
where we slightly abuse notations, and used and interchangeably, and thus
Assumptions. For an arbitrary parameter , we assume the following:
(A1) and are nonempty, compact and convex;
(A2) is ;
(A3) the mapping is differentiable at ;
(A4) there exists such that, for every , the Isaacs condition
holds;
(A5) the function has a unique saddle point on , which we denote by .
Theorem 1 Under (A1)–(A5), the mapping is differentiable at , with
In other words, the extra term that would result from chain rule does not appear. Thus, the saddle point can be treated as a constant during backpropagation, with no gradients propagated through or the solver that computes it.
Proof sketch: The key idea of our proof is to avoid differentiating through the saddle point solution map, which may be nondifferentiable. Instead, we directly analyze perturbations of the Hamiltonian using saddle point inequalities. Together with compactness and uniqueness, these bounds establish differentiability of the Hamiltonian and yield the desired gradient identity through the chain rule.
Proof. Let denote the set of saddle points at , that is,
Step 1: Existence of Saddle Point. By (A4), for every , the Isaacs condition
holds. Here, the extrema are attained by compactness and continuity. Let attain the outer maximum on the left and the outer minimum on the right. On one hand,
where in the first step, we used the property of minimum, and in the second step, we used the definition of in (28). On the other hand,
using the same argument. Combining these two inequalities, we have
Next, we argue that is a saddle point. On one hand, we have, for every ,
by the definition of . On the other hand, similarly, for every ,
by the definition of . Together, (30) and (31) establish that is a saddle point, that is . Thus, for every , which is the existence of saddle point. Moreover, by the definition of the Hamiltonian , (29) becomes
Step 2: Linearization Error Bound. We bound the error of linearizing around . Given any fixed , write and let be any saddle point at (which exists by Step 1).
Upper Bound. Using (SV) of Lemma 2, we have
Subtracting these two inequalities, we have
Lower Bound. Similarly, using (SV) of Lemma 2, we have
Subtracting these two inequalities, we have
Error Bound. Let the linearization error be
Plugging (33) and (34) into (35), we have
Next, we establish the linearization error bound as
by the definition (35), and , and
by applying the Cauchy–Schwarz inequality to (36), where
Step 3: Differentiability of . We prove that is differentiable at . By Lemma 3, as , then for any choice of saddle point (which exists by Step 1), . By (A2), is continuous, thus as . Therefore,
as . Since does not depend on the choice of the saddle point , this implies that is differentiable at , and .
Step 4: Gradient-free. By (A3), is differentiable at , thus by Step 3 and the chain rule,
Supporting Lemmas.
Lemma 2 (Saddle Value). [43 Section 5.4.2] If is a saddle point at , then and
Lemma 3 (Continuity of saddle points at ). Let denote the set of saddle points at , that is,
Under (A1), (A2), and (A5), if the sequence with (defined in (A4)) and , then .
Proof. Write and for all . Suppose the contrary, that is, there exists such that there is a subsequence with for all .
By (A1), since is compact, by the Bolzano-Weierstrass theorem [3 Theorem 3.4.8], there exists a convergent subsequence of such that and
by continuity of . Also, remark that by assumption. Since , we have [43 Section 5.4.2]
Consider . By (A2), is continuous on . Together with continuity of inner product, they imply is jointly continuous in . Thus, taking gives
This implies that is a saddle point at [43 Section 5.4.2]. On the other hand, by (A5), is a singleton. Thus, we have , but this contradicts (41) and thus contradicts our original assumption. ▪
References
- [1]Brandon Amos. On amortizing convex conjugates for optimal transport. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=TQ5WUuS_4ai.arxiv.org/abs/2210.12153
- [2]Somil Bansal and Claire J. Tomlin. DeepReach: A deep learning approach to high-dimensional reachability. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 1817–1824. IEEE, 2021. doi: 10.1109/ICRA48506.2021.9561949.DOI
- [3]Robert G Bartle and Donald R Sherbert. Introduction to real analysis, volume 2. Wiley New York, 2000.
- [4]Jonathan Barzilai and Jonathan M Borwein. Two-point step size gradient methods. IMA journal of numerical analysis, 8(1):141–148, 1988.DOI
- [5]John T. Betts. Practical Methods for Optimal Control and Estimation Using Nonlinear Programming. SIAM, 2010.DOI
- [7]Jason J. Choi, Christopher A. Strong, Koushil Sreenath, Namhoon Cho, and Claire J. Tomlin. Data-driven hamiltonian for direct construction of safe set from trajectory data. In 2025 IEEE 64th Conference on Decision and Control (CDC), pages 2951–2958. IEEE, 2025. doi: 10.1109/CDC57313.2025.11312763.DOI
- [8]Zeyuan Feng, Le Qiu, and Somil Bansal. Bridging Model Predictive Control and Deep Learning for Scalable Reachability Analysis. In Proceedings of Robotics: Science and Systems, Los Angeles, CA, USA, June 2025. doi: 10.15607/RSS.2025.XXI.114.DOI
- [9]Jaime F Fisac, Mo Chen, Claire J Tomlin, and S Shankar Sastry. Reach-avoid problems with time-varying dynamics, targets and constraints. In Proceedings of the 18th international conference on hybrid systems: computation and control, pages 11–20, 2015.arxiv.org/abs/1410.6445
- [10]Samy Wu Fung, Howard Heaton, Qiwei Li, Daniel McKenzie, Stanley Osher, and Wotao Yin. JFB: Jacobian-free backpropagation for implicit networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6648–6656, 2022.
- [11]Eric Gelphman, Deepanshu Verma, Nicole Tianjiao Yang, Stanley Osher, and Samy Wu Fung. End-to-end training of high-dimensional optimal control with implicit hamiltonians via jacobian-free backpropagation. In 2026 American Control Conference (ACC). IEEE, 2026a.arxiv.org/abs/2510.00359
- [12]Eric Gelphman, Deepanshu Verma, Nicole Tianjiao Yang, Stanley Osher, and Samy Wu Fung. On the convergence of jacobian-free backpropagation for optimal control problems with implicit hamiltonians. arXiv preprint arXiv:2602.00921, 2026b.arxiv.org/abs/2602.00921
- [13]Gautier Hattenberger, Murat Bronz, and Jean-Philippe Condomines. Evaluation of drag coefficient for a quadrotor model. International Journal of Micro Air Vehicles, 15:17568293221148378, 2023.
- [14]Howard Heaton and Samy Wu Fung. Explainable ai via learning to optimize. Scientific Reports, 13 (1):10103, 2023.DOI
- [15]Howard Heaton, Samy Wu Fung, Aviv Gibali, and Wotao Yin. Feasibility-based fixed point networks. Fixed Point Theory and Algorithms for Sciences and Engineering, 2021(1):21, 2021.arxiv.org/abs/2104.14090
- [16]William Hofgard. Convergence guarantees for neural network-based hamilton–jacobi reachability. In NeurIPS 2024 Workshop on Data-driven and Differentiable Simulations, Surrogates, and Solvers, 2024. URL https://openreview.net/forum?id=SH1isYEjz9.arxiv.org/abs/2410.02904
- [17]Rufus Isaacs. Differential Games: A Mathematical Theory with Applications to Warfare and Pursuit, Control and Optimization. John Wiley & Sons, New York, 1965.
- [18]Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.arxiv.org/abs/1412.6980
- [19]Galina M Korpelevich. The extragradient method for finding saddle points and other problems. Matecon, 12:747–756, 1976.
- [20]Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in neural information processing systems, 34:26548–26560, 2021.arxiv.org/abs/2109.01050
- [21]Sudeep Kundu and Karl Kunisch. Policy iteration for hamilton-jacobi-isaacs equations with control constraints and comparison with hamilton-jacobi-bellman equations. arXiv preprint arXiv:2609.29368, 2026.arxiv.org/abs/2609.29368
- [22]Minhyeok Kwon and Yongsoon Eun. Quadrotor dynamics in a wind field: Equilibria analysis and energy dissipation. International Journal of Control, Automation and Systems, 22(11):3275–3284, 2024.DOI
- [23]Alistair Letcher, David Balduzzi, Sébastien Racan iere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel. Differentiable game mechanics. Journal of Machine Learning Research, 20 (84):1–40, 2019.arxiv.org/abs/1905.04926
- [24]Xingjian Li, Kelvin Kan, Deepanshu Verma, Krishna Kumar, Stanley Osher, and Ján Drgoňa. Zero-shot transferable solution method for parametric optimal control problems. In 2026 American Control Conference (ACC), pages 2003–2010. IEEE, 2026a.arxiv.org/abs/2509.18404
- [25]Xingjian Li, Kelvin Kan, Deepanshu Verma, Krishna Kumar, Stanley Osher, and Samy Wu Fung. End-to-end learning of safe optimal feedback control in high dimensions with control barrier function layers. arXiv preprint arXiv:2607.20674, 2026b.arxiv.org/abs/2607.20674
- [26]Albert Lin and Somil Bansal. Generating formal safety assurances for high-dimensional reachability. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 10525–10531. IEEE, 2023.arxiv.org/abs/2209.12336
- [27]Albert Lin, Alessandro Pinto, and Somil Bansal. Robust verification of controllers under state uncertainty via hamilton–jacobi reachability analysis. In Proceedings of the 8th Annual Learning for Dynamics and Control Conference, volume 331 of Proceedings of Machine Learning Research, pages 1432–1446. PMLR, 2026.arxiv.org/abs/2511.14755
- [28]Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International conference on artificial intelligence and statistics, pages 1540–1552. PMLR, 2020.arxiv.org/abs/1911.02590
- [29]Robert Mahony, Vijay Kumar, and Peter Corke. Multirotor aerial vehicles: Modeling, estimation, and control of quadrotor. IEEE robotics & automation magazine, 19(3):20–32, 2012.DOI
- [30]Eric Mazumdar, S Shankar Sastry, and Michael I Jordan. On finding local nash equilibria (and only local nash equilibria) in zero-sum games. ACM/IMS Journal of Data Science, 2(2):1–26, 2025.arxiv.org/abs/1901.00838
- [31]Daniel McKenzie, Howard Heaton, and Samy Wu Fung. Differentiating through integer linear programs with quadratic regularization and davis-yin splitting. Transactions on Machine Learning Research, 2024a. ISSN 2835-8856. URL https://openreview.net/forum?id=H8IaxrANWl.arxiv.org/abs/2301.13395
- [33]Ian M. Mitchell, Alexandre M. Bayen, and Claire J. Tomlin. A time-dependent hamilton–jacobi formulation of reachable sets for continuous dynamic games. IEEE Transactions on Automatic Control, 50(7):947–957, 2005a. doi: 10.1109/TAC.2005.851439.
- [34]Ian M Mitchell, Alexandre M Bayen, and Claire J Tomlin. A time-dependent hamilton-jacobi formulation of reachable sets for continuous dynamic games. IEEE Transactions on automatic control, 50(7):947–957, 2005b.
- [35]Yesom Park, Eric Gelphman, Stanley Osher, and Samy Wu Fung. Implicit neural optimal transport via fixed-point optimization. arXiv preprint arXiv:2605.10792, 2026.arxiv.org/abs/2605.10792
- [36]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1912.01703
- [37]Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686–707, 2019.
- [38]Hayden Schaeffer. Neural network approximation and structural complexity of semilinear pdes with lower bounds. arXiv preprint arXiv:2610.40384, 2026.
- [39]David SeWell, Xingjian Li, Stepan Tretjakov, Krishna Kumar, and David Fridovich-Keil. Neural operators for multi-task control and adaptation. Transactions on Machine Learning Research, 2026. ISSN 2835-8856. URL https://openreview.net/forum?id=jciObOZ5Wm.arxiv.org/abs/2604.03449
- [40]William Sharpless, Zeyuan Feng, Somil Bansal, and Sylvia Herbert. Linear supervision for nonlinear, high-dimensional neural control and differential games. In Proceedings of the 7th Annual Learning for Dynamics and Control Conference, volume 283 of Proceedings of Machine Learning Research, pages 365–377. PMLR, 2025.arxiv.org/abs/2412.02033
- [41]Aditya Singh, Zeyuan Feng, and Somil Bansal. Exact imposition of safety boundary conditions in neural reachable tubes. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 5489–5495. IEEE, 2025.arxiv.org/abs/2404.00814
- [42]Ryan Teoh, Sander Tonkens, William Sharpless, Aijia Yang, Zeyuan Feng, Somil Bansal, and Sylvia Herbert. MADR: MPC-guided adversarial DeepReach. arXiv preprint arXiv:2510.18845, 2025.arxiv.org/abs/2510.18845
- [43]Lieven Vandenberghe and Stephen Boyd. Convex optimization, volume 1. Cambridge university press Cambridge, 2004.
- [44]Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality is all you need for training physics-informed neural networks. arXiv preprint arXiv:2203.07404, 2022.arxiv.org/abs/2203.07404
- [45]Sifan Wang, Shyam Sankaran, Hanwen Wang, and Paris Perdikaris. An expert’s guide to training physics-informed neural networks. arXiv preprint arXiv:2308.08468, 2023.arxiv.org/abs/2308.08468