betaFeedback

WorksEngineering

PDE Control Gym: A Benchmark for Data-Driven Boundary Control of Partial Differential Equations

Abstract

Over the last decade, data-driven methods have surged in popularity, emerging as valuable tools for control theory. As such, neural network approximations of control feedback laws, system dynamics, and even Lyapunov functions have attracted growing attention. With the ascent of learning based control, the need for accurate, fast, and easy-to-use benchmarks has increased. In this work, we present the first learning-based environment for boundary control of PDEs. In our benchmark, we introduce three foundational PDE problems - a 1D transport PDE, a 1D reaction-diffusion PDE, and a 2D Navier-Stokes PDE - whose solvers are bundled in an user-friendly reinforcement learning gym. With this gym, we then present the first set of model-free, reinforcement learning algorithms for solving this series of benchmark problems, achieving stability, although at a higher cost compared to model-based PDE backstepping. With the set of benchmark environments and detailed examples, this work significantly lowers the barrier to entry for learning-based PDE control - a topic largely unexplored by the data-driven control community. The entire benchmark is available on Github along with detailed documentation and the presented reinforcement learning models are open sourced.

Introduction

As learning-based control has exploded across both academia and industry, the need for fast, and accurate bench-marking is heightened. For example, perhaps the most visible impact of proper bench-marking is in the field of computer-vision resulting in 15-years of breakthrough results from AlexNet Krizhevsky et al. [22] to neural radiance fields (NeRFs) Mildenhall et al. [31]. Despite this, the control community has, justifiably, forgone consistent efforts in bench-marking as the community spawned from an applied mathematics perspective where the focus was behind provable stability guarantees. However, given the recent exploration surrounding data-driven control methods Berberich et al. [2]; Feng et al. [11], designing fast, well-documented, and challenging benchmarks is of utmost importance to ensure new learning-based control approaches are consistently advancing the state of the art.

In this work, we develop a benchmarking suite for learning-based boundary control of PDEs. Boundary control of PDEs is of elevated importance compared to control across the full domain as many real world problems cannot control the PDE across the entire domain, but only at the boundary input. Thus, boundary control is physically more realistic as the actuation and sensing are generally non-intrusive Krstic and Smyshlyaev [24]. For example, in fluid flows, the engineer only gets control access to the surrounding walls containing the fluid Di Meglio et al. [9] or in temperature manufacturing Bian et al. [5], the engineer is typically unable to set the temperature of entire plate, but only a specific edge. Furthermore, boundary control is extremely powerful in modeling macro-level traffic congestion Huan Yu [17] as modern highways typically only enable actuation and sensing at the on/off ramps. Lastly, we briefly mention that boundary control is one of the only approaches to handle chemical and combustion processes Izadi et al. [18] and moving boundary problems such as the Stefan Problem with applications to both 3D printing Koga and Krstic [21] and the excitation of neuron growth for abating neurological diseases such as Parkinson’s and Alzheimer’s Demir et al. [8]. However, for a majority of these applications, each researcher typically develops their own simulations and thus there is no standard library with a universal set of problems to test new algorithms. Furthermore, for even the standard model-based algorithms such as PDE backstepping, it is challenging to implement the control algorithm as this typically requires the solution of a separate Goursat PDE Vazquez et al. [44]. Thus, in this work, we introduce the first library containing a set of general PDE control problems and implementations of their corresponding model-based control algorithms that can be easily modified to fit the wide array of aforementioned target applications.

Contributions This paper has three main contributions. First, we introduce, design, and formalize the first benchmark suite for PDE control including 3 classical problems ranging from boundary stabilization for 1D transport (hyperbolic) and reaction-diffusion (parabolic) PDEs to trajectory following for the 2D Navier-Stokes PDEs. Along with the proposal of the PDE control benchmarking suite, we parameterize the numerical scheme implementations as RL gyms - effectively decoupling the PDE solvers from the controller design, enabling the use of any pre-implemented learning algorithm for PDE control. Second, utilizing our benchmark suite, we train the first set of model-free RL controllers which effectively stabilize hyperbolic and parabolic PDE problems and achieve effective tracking for the 2D Navier-Stokes equations. We then compare the resulting controllers to classical algorithms such as PDE backstepping and adjoint-based optimization highlighting the trade-offs between performance. Lastly, we provide extensive documentation and numerous examples for the training of RL controllers, implementation of classical control algorithms and of course the integration of new PDE control problems into the benchmark suite.

Related Work

Learning-based PDE benchmarks

Currently, to the author’s knowledge, there are no benchmarking suites focused on boundary control of PDEs as most benchmarks such as [42, 13] present datasets for learning PDE solution maps from initial conditions. These benchmarks are typically used for comparing neural network-based PDE solvers like neural operators Lu et al. (2021); Li et al. [28] and PINNs Raissi et al. [37]. Although these benchmarks are effective, they do not allow users to incorporate boundary control or alter the PDEs within their datasets. The ϕ\phiFlow framework Holl et al. [16] provides a flexible PDE solver that supports automatic differentiation for the calculation PDE derivatives to be used in control algorithms. While adaptable for PDE control, it does not natively support RL algorithms and lacks a set of model-based controllers for learning-based control comparisons. Lastly, it is worth noting that while there are ODE control suites like Duan et al. [10]; Tassa et al. [43] tailored for ODE control tasks. Lastly, we mention a concurrent work that also integrates in-domain PDE control into a RL library Zhang et al. [47]; however, the class of problems they focus on - namely in-domain control - is different from the boundary control problems we benchmark in this gym. Thus, the PDE control gym, to our knowledge, represents the first PDE-focused benchmarking suite for learning-based boundary-control algorithms.

Benchmarking for machine learning in PDEsCompilation of a premade datasetSupports controlDifferentiable PDE solverSupports custom PDEsSupports reinforcement learningImplementation of model-based control
PhiFlow Holl et al. [16]✗✓✓✓✗✗
PDEBench Takamoto et al. [42] (created from PhiFlow)✓✗✗✗✗✗
PDEArena Gupta and Brandstetter [13] (created from PhiFlow)✓✗✗✗✗✗
PDE Control Gym (Ours)✗✓✗✓✓✓

Table 1. Comparison of benchmarks for machine learning in PDEs.

Learning as a tool for PDE control

As with most scientific disciplines, machine learning has had a broad impact in PDE control. In 1D PDE problems, a series of work has been developed to use neural operators for approximating control feedback laws Bhan et al. [3]; Krstic et al. [25]; Qi et al. [35], under a supervised learning framework with provable stability guarantees. Furthermore, an optimal control approach using PINNs is explored in Mowlavi and Nabi [33]. Additionally, the closest paper to this work is by Yu et al. [46] who presented the first exploration utilizing RL for PDE boundary control. However, they do not explore the benchmark PDE problems presented in this paper instead focusing on Aw-Rascale-Zhang (ARZ) traffic model.

Reinforcement learning in control

Reinforcement learning (RL) has demonstrated significant success in various control applications, including robotics Brunke et al. [6], power systems Chen et al. [7], and autonomous driving Kiran et al. [20]. From a controls perspective, deep RL (DRL) algorithms learn feedback laws that maps observations (states) into actions (control inputs), typically via neural networks (NN). These RL controllers are trained to optimize specific reward functions, such as the L2L^2 spatial norm of states for stabilization tasks Yu et al. [46]. The most appealing feature of deep RL is its model-free nature, allowing it to control complex systems without requiring explicit model estimations. Consequently, RL has potential to outperform model-based control methods in highly complex tasks with hard-to-model dynamics. To demonstrate the use of PDE Control Gym, we conduct experiments using off-the-shelf RL algorithms implemented with Stable-Baselines3 Raffin et al. [36]. We selected the off-policy soft actor-critic (SAC) (Haarnoja et al., [14]) and on-policy proximal policy optimization (PPO) (Schulman et al., [38]) algorithms for their demonstrated efficiency in solving challenging continuous control tasks Duan et al. [10].

Formalization of PDE Control Problems

General PDE control problem

We consider a partial differential equation (PDE) defined on a domain X\mathcal{X}, which can be either one-dimensional (1D), X=[0,1]⊂R\mathcal{X} = [0,1] \subset\mathbb{R}, or two-dimensional (2D), X=[0,1]×[0,1]⊂R2\mathcal{X} = [0,1] \times[0,1] \subset\mathbb{R}^{2}. The time domain is T=[0,T]⊂R+\mathcal{T} = [0,T] \subset\mathbb{R}^{+}. Let u(x,t)u(x,t), x∈Xx \in\mathcal{X}, t∈Tt \in\mathcal{T} describe the state of the system governed by the PDE according to the dynamics

∂u∂t=P(u,∂u∂x,∂2u∂x2,…,U(t)),(1)\frac{\partial u}{\partial t} = \mathcal{P}\left(u,\frac{\partial u}{\partial x},\frac{\partial^{2}u}{\partial x^{2}},\ldots,U(t)\right), \tag*{(1)}

where P\mathcal{P} is the partial differential equation(s) that model(s) the system dynamics, and U(t)U(t) is the control function. Then, the goal of a PDE control problem is to optimize a cost function (e.g. regulate u(x,t)u(x,t) to be the desired trajectory while reducing the control cost, stabilize the PDEs) from just boundary inputs. Note that in some methods such as PDE backstepping, optimization is forgone in favor of just asymptotic stabilization as the infinite dimensional nature of PDE control is extremely challenging.

Markov decision processes (MDPs) for PDE control

We give a brief overview of the components of the MDP governing both the 1D and 2D support problems. More details about the specific MDPs governing the examples in § 5 can be found in the Appendix § 7.§ 2 and B.2.

Supported Configuration1D Hyperbolic Sensing (o(t)o(t))1D Hyperbolic Actuation (a(t)a(t))1D Parabolic Sensing (o(t)o(t))1D Parabolic Actuation (a(t)a(t))2D Navier-Stokes Sensing (o(t)o(t))2D Navier-Stokes Actuation (a(t)a(t))
full-stateu(x,t)u(x,t)u(1,t)u(1,t)u(x,t)u(x,t)u(1,t)u(1,t)u(x,y,t)u(x,y,t)u(x,1,t)u(x,1,t)
collocatedux(1,t)u_x(1,t)u(1,t)u(1,t)ux(1,t)u_x(1,t)u(1,t)u(1,t)——
anti collocatedu(0,t)u(0,t)u(1,t)u(1,t)————
anti collocatedux(0,t)u_x(0,t)u(1,t)u(1,t)ux(0,t)u_x(0,t)u(1,t)u(1,t)——
full-stateu(x,t)u(x,t)ux(1,t)u_x(1,t)u(x,t)u(x,t)ux(1,t)u_x(1,t)u(x,y,t)u(x,y,t)u(x,0,t)u(x,0,t)
collocatedu(1,t)u(1,t)ux(1,t)u_x(1,t)u(1,t)u(1,t)ux(1,t)u_x(1,t)——
anti collocatedu(0,t)u(0,t)ux(1,t)u_x(1,t)————
anti collocatedux(0,t)u_x(0,t)ux(1,t)u_x(1,t)ux(0,t)u_x(0,t)ux(1,t)u_x(1,t)——
full-state————u(x,y,t)u(x,y,t)u(1,y,t)u(1,y,t)
full-state————u(x,y,t)u(x,y,t)u(0,y,t)u(0,y,t)

Table 2. Configurations for actuation and sensing supported by the PDE Control Gym for the three problems. Full state indicates the measurement is the entire PDE state, collated indicates that the sensing and measurement is done at the same boundary point, and anti-collocated indicates sensing and measurement are done at opposite boundary points. The configurations marked in blue correspond to the experiment examples in § 5.

State and Observation Space. In all PDE control problems addressed, the state s(t)s(t) at time tt is represented by the PDE value u(x,t)u(x,t), x∈Xx \in\mathcal{X}. To enhance flexibility for different PDE tasks such as observer design, we have developed different partial state measurement settings, which we denote as the observation space o(t)o(t). The types of sensing supported for each problem are detailed in Table 3.2. Furthermore, we offer the ability to introduce custom noise functions (with built-in support for Gaussian noise). This allows users to simulate real-world sensor noise in their experiments.

Action Space. The action a(t)=U(t)a(t) = U(t) for both the RL and control agents is determined by the actuation locations and boundary condition type. In 1D hyperbolic and parabolic systems, we consider both Neumann and Dirichlet boundary actuation U(t)∈RU(t) \in\mathbb{R} at either boundary, with four possible cases: 1) u(0,t)=U(t)u(0,t) = U(t), 2) u(1,t)=U(t)u(1,t) = U(t), 3) ux(0,t)=U(t)u_x(0,t) = U(t), and 4) ux(1,t)=U(t)u_x(1,t) = U(t). Considering the symmetry between the boundaries x=0x = 0 and x=1x = 1, there are eight distinct combinations for the 1D Hyperbolic PDE problem and six for the 1D Parabolic PDE problem, including an additional boundary condition at u(0,t)u(0,t). These combinations are outlined in the first two sections of Table 3.2. For the 2D Navier–Stokes problem, we consider Dirichlet-type boundary actuation on any of the four boundaries: top, bottom, left, and right. The gym also allows users to customize actuator positions, enabling research into optimizing both location and actuation type.

State Evolution. To simulate the PDE system evolution with the control input U(t)U(t), we use a first-order Taylor approximation for temporal evolution,

u(t+Δt)=u(t)+Δt⋅P(u(t),∂u∂x,∂2u∂x2,…,U(t)).(2)u(t+\Delta t) = u(t) + \Delta t \cdot\mathcal{P}\left(u(t), \frac{\partial u}{\partial x}, \frac{\partial^{2}u}{\partial x^{2}}, \ldots, U(t)\right). \tag*{(2)}

Spatial derivatives are approximated using appropriate finite difference schemes and are explicitly given for each gym environment in the Appendix § 7.§ 1 and B.1. In practice, selecting the time step Δt\Delta t and spatial discretization Δx\Delta x for each problem requires careful consideration, particularly based on the number of approximated spatial derivatives. Nonetheless, we found that reasonable choices such as Δx=0.01\Delta x = 0.01, Δt=0.0001\Delta t = 0.0001 yield both fast and numerically stable results.

Reward. Reward shaping plays a pivotal role in the training of RL algorithms. Generally speaking, for stabilization tasks, it is appropriate to employ a form of trajectory-based reward

∫0T(∫x∈X∥u(x,t)∥2 dx+∥U(t)∥2)dt,(3)\int_{0}^{T}\left(\int_{x \in\mathcal{X}} \lVert u(x,t)\rVert^{2} \, dx + \lVert U(t)\rVert^{2}\right)dt, \tag*{(3)}

which minimizes the state magnitudes and control efforts. For tracking tasks, a trajectory reward as

∫0T(∫x∈X∥u(x,t)−uref(x,t)∥2 dx+∥U(t)−Uref(t)∥2)dt,(4)\int_{0}^{T}\left(\int_{x \in\mathcal{X}} \lVert u(x,t)-u_{\mathrm{ref}}(x,t)\rVert^{2} \, dx + \lVert U(t)-U_{\mathrm{ref}}(t)\rVert^{2}\right)dt, \tag*{(4)}

is a reasonable choice as it penalizes deviations from the reference trajectory given in both state uref(x,t)u_{\mathrm{ref}}(x,t) and control actions Uref(t)U_{\mathrm{ref}}(t). However, in practice, for the 1D hyperbolic and parabolic PDE problems, we found that the reward as given in (3) was insufficient for training and thus we use a specifically tuned reward that penalizes the difference of the L2L_{2} norms between the current state and next state after action a(t)a(t) (presented in Appendix § 7.§ 2 and B.2).

Benchmark PDE Control Tasks

1D Hyperbolic (transport) PDEs

We consider the benchmark transport PDE in the form

ut(x,t)=ux(x,t)+β(x)u(0,t),(5)u_t(x,t) = u_x(x,t) + \beta(x)u(0,t), \tag*{(5)}

for x∈[0,1)x \in[0,1), t∈[0,T]t \in[0,T]. Physically, (5) is a “transport process (from x=1x = 1 towards x=0x = 0) with recirculation” of the outlet variable u(0,t)u(0,t). Recirculation causes instability and the goal is to stabilize “full-state” recirculation from only boundary inputs. In practice, we consider the same PDE as studied in Bhan et al. [3] where β\beta is governed by the Chebyshev polynomial β(x)=5cos⁡(γcos⁡−1(x))\beta(x) = 5\cos(\gamma\cos^{-1}(x)) and Dirichlet actuation u(1,t)=U(t)u(1,t) = U(t). Classically, this PDE, with recirculation, has been a seminal-benchmark for PDE backstepping, as the 1D transport PDE can model a wide range of applications from chemical processes to shallow water waves and traffic flows Krstic and Smyshlyaev [23].

Model-based backstepping control. The backstepping controller given by the following

U(t)=∫01k(1−y)u(y,t) dy,(6)U(t) = \int_{0}^{1} k(1-y)u(y,t)\,\mathrm{d}y, \tag*{(6)}
k(x)=−β(x)+∫0xβ(x−y)k(y) dy,(7)k(x) = -\beta(x) + \int_{0}^{x}\beta(x-y)k(y)\,\mathrm{d}y, \tag*{(7)}

for x∈[0,1]x \in[0,1]. (6) results in stabilization of (5) Krstic and Smyshlyaev [23]. In practice, the backstepping kernel (7) is implemented using the successive approximations approach although a Laplace transform approach is also viable Krstic and Smyshlyaev [24].

1D Parabolic (reaction-diffusion) PDEs

We consider the benchmark reaction-diffusion PDEs governed by recirculation function λ(x)\lambda(x) as

ut(x,t)=uxx(x,t)+λ(x)u(x,t),(8)u_t(x,t) = u_{xx}(x,t) + \lambda(x)u(x,t), \tag*{(8)}
u(0,t)=0,(9)u(0,t) = 0, \tag*{(9)}

with Dirichlet or Neumann actuation at x=1x = 1. Again, instability is caused by the λ(x)u(x,t)\lambda(x)u(x,t) term otherwise the problem would simplify to the classical heat equation. This PDE appears in different applications ranging from a chemical tubular reactor Shi et al. [39] to electro-chemical battery models Moura et al. [32] and diffusion in social networks Wang et al. [45].

Model-based backstepping control. For the PDEs (8), (9), with Dirichlet boundary actuation u(1,t)=U(t)u(1,t) = U(t), the backstepping controller with full state measurement is given by the following Smyshlyaev and Krstic [40, 41],

U(t)=∫01k(1,y)u(y,t) dy.(10)U(t) = \int_{0}^{1} k(1,y)u(y,t)\,\mathrm{d}y. \tag*{(10)}

where k(x,y)∈C2(T~)k(x,y) \in C^2(\widetilde{\mathcal{T}}), T~={0≤y≤x≤1}\widetilde{\mathcal{T}} = \{0 \le y \le x \le1\}.

kxx(x,y)−kyy(x,y)=λ(y)k(x,y),∀(x,y)∈T~,(11)k_{xx}(x,y) - k_{yy}(x,y) = \lambda(y)k(x,y), \quad\forall(x,y) \in\widetilde{\mathcal{T}}, \tag*{(11)}
k(x,0)=0,(12)k(x,0)=0, \tag*{(12)}
k(x,x)=−12∫0xλ(y) dy,(13)k(x,x)=-\frac{1}{2}\int_{0}^{x}\lambda(y)\,\mathrm{d}y, \tag*{(13)}

where T~={0<y≤x<1}\widetilde{T}=\{0<y\leq x<1\}.

2D Navier-Stokes PDEs

We consider the 2D in-compressible Navier-Stokes equations as the third benchmark control task,

∇⋅u=0,(14)\nabla\cdot\boldsymbol{u}=0, \tag*{(14)}
∂u∂t+u⋅∇u=−1ρ∇p+ν∇2u.(15)\frac{\partial\boldsymbol{u}}{\partial t}+\boldsymbol{u}\cdot\nabla\boldsymbol{u}=-\frac{1}{\rho}\nabla p+\nu\nabla^{2}\boldsymbol{u}. \tag*{(15)}

With slight abuse of notation, we denote the spatial variable (in 2D) as x=(x,y)∈X=[0,1]×[0,1]\boldsymbol{x}=(x,y)\in\mathcal{X}=[0,1]\times[0,1], and u=(u,v):X×T→R2\boldsymbol{u}=(u,v):\mathcal{X}\times\mathcal{T}\to\mathbb{R}^{2} represents 2D velocity field, ν\nu is the kinematic viscosity of the fluid, ρ\rho is the fluid density, and pp is the pressure field. Navier-Stokes equation is fundamental in fluid dynamics with extensive applications including aerodynamic design, pollution modeling, and wind turbine flows Jameson et al. [19]; Li et al. [28]. For the experiments, we consider boundary control along the top boundary as u(x,0,t)=U(x,t)\boldsymbol{u}(x,0,t)=U(x,t), ∀x∈[0,1]\forall x\in[0,1]. All other boundary conditions are Dirichlet boundary conditions where the velocity is set to be 0. The task here is to find boundary control U(x,t)U(x,t) such that the resulting velocity field is close to the reference trajectory given in both desired velocity field uref(x,t)u_{ref}(x,t) and desired actions Uref(x,t)U_{ref}(x,t).

Model-based optimization-based control. We provide an optimization-based controller based on Pyta et al. [34] as the model-based control baseline.

min⁡U(x,t) J(U(⋅,t),u)=12∫T∫X∥u(x,t)−uref(x,t)∥2 dxdt+γ2∫T∥U(⋅,t)−Uref(⋅,t)∥2 dt(16)\begin{aligned} \min_{U(x,t)}\ J(U(\cdot,t),\boldsymbol{u})={}&\frac{1}{2}\int_{\mathcal{T}}\int_{\mathcal{X}}\lVert\boldsymbol{u}(x,t)-u_{ref}(x,t)\rVert^{2}\,\mathrm{d}x\mathrm{d}t\\ &+\frac{\gamma}{2}\int_{\mathcal{T}}\lVert U(\cdot,t)-U_{ref}(\cdot,t)\rVert^{2}\,\mathrm{d}t \tag*{(16)} \end{aligned}
s.t.xref‘(14a)‘,xref‘(14b)‘,u(x,0,t)=U(x,t), ∀x∈[0,1].(17)\mathrm{s.t.}\quad{xref}`(14a)`, {xref}`(14b)`,\qquad\boldsymbol{u}(x,0,t)=U(x,t),\ \forall x\in[0,1]. \tag*{(17)}

To ensure computational tractability, the control actions are set to be the tangential, uniform velocity, i.e., u(x,0,t)=U(t)u(x,0,t)=U(t), v(x,0,t)=0v(x,0,t)=0, followed in Pyta et al. [34]. The optimal control actions are obtained by solving the PDE-constrained optimization problem presented in (15). This solution employs Lagrange multipliers, utilizing the adjoint method McNamara et al. [30]; Gunzburger [12] for gradient computation of the Lagrangian function.

Experiments

For each of the 3 environments in PDE Control Gym, we implemented baseline model-based control algorithms as well as off-the-shelf RL algorithms including soft actor-critic (SAC) (Haarnoja et al. [14]) and proximal policy optimization (PPO) (Schulman et al. [38]) trained using Stable-Baselines3 (Parameters available in Appendix § 7.§ 2 and B.2). We note that all the experiments can be trained in under 1 hour (Nvidia RTX 3090ti) and entire trajectories can be simulated in seconds.

1D Hyperbolic (transport) PDEs

Experimental design Our experimental setup for the Hyperbolic 1D problem, detailed in § 4.1, considers full state measurements and boundary control u(1,t)=U(t)u(1,t)=U(t). We use the Chebyshev polynomial recirculation function β(x)=5cos⁡(γcos⁡−1(x))\beta(x)=5\cos(\gamma\cos^{-1}(x)) from Bhan et al. [3], with γ=7.35\gamma=7.35 (future studies may vary γ\gamma). Each episode is initiated from a random initial condition, u(x,0)∼Uniform⁡(1,10)u(x,0)\sim\operatorname{Uniform}(1,10). This setup presents a challenging control scenario, as the open loop system (U(t)=0)(U(t)=0) is unstable (See Figure 5 in Appendix).

Figure 5. Open-loop (U(t)=0U(t)=0) instability of transport PDE for u(x,0)=1,10u(x,0)=1,10. Instability of the 1D transport PDE with β(x)=5cos⁡(7.35cos⁡−1(x))\beta(x)=5\cos(7.35\cos^{-1}(x)) under a openloop control signal (U(t)=0U(t)=0).

Results We now present detailed results on the policies trained and their comparison with model-based backstepping. In the left of Figure 1, we present the average reward functions for both RL algorithms over 1 million training steps. Then, in Table 3, we present the average reward where we run the trained final RL policies, and the model-based backstepping policy for 50 test episodes with different initial conditions, noting that model-based backstepping performs the best. Additionally, in Figure 2 we provide a comparison across all 3 control approaches where u(x,0)=10u(x,0)=10. We can clearly see that although all 3 policies are stabilizing for the examples, model-based PDE backstepping again performs the best and the RL control signals are high oscillatory leaving room for improvement in applying model-free PDE control algorithms.

AlgorithmHyperbolic PDE Average Episode Reward for Trained Policy ↑\uparrowParabolic PDE Average Episode Reward for Trained Policy ↑\uparrowNavier-Stokes PDE Average Episode Reward for Trained Policy ↑\uparrow
Model-based246.3299.1-7.931
PPO172.3293.3-5.370
SAC184.2229.1-17.829

Table 3. Resulting control algorithm performance on 50 test episodes in each gym (larger value indicates better performance). The model-based algorithm for the hyperbolic and parabolic PDEs are the backstepping schemes given in (6), (7) and (10), (11), (12), (13) respectively while the method for the Navier-Stokes PDE solves the optimization problem (15).

Figure 1. Rewards for training PPO (blue) and SAC (orange) on the 1D transport PDE, 1D reaction-diffusion PDE, and 2D Navier-Stokes PDE from left to right. The solid lines represent the mean and the shaded bounds are 95% confidence intervals across 5 seeds.

Figure 2. Example of the 1D transport PDE system stabilization using backstepping, PPO, and SAC (left to right) under initial conditions u(x,0)=10u(x,0)=10. The recirculation coefficient is defined as β(x)=5cos⁡(γcos⁡−1(x))\beta(x)=5\cos(\gamma\cos^{-1}(x)) with γ=7.35\gamma=7.35.

1D Parabolic (reaction-diffusion) PDEs

Experimental design We adopt the same approach as the 1D Hyperbolic PDE in § 5.1 except that the dynamics are now governed by (8), (9), with full state measurements and u(1,t)=U(t)u(1,t)=U(t). The time-horizon is shortened to 1 second as the algorithms are able to stabilize faster. We choose λ(x)=50cos⁡(γcos⁡−1(x))\lambda(x)=50\cos(\gamma\cos^{-1}(x)) where γ\gamma is fixed to be 8 (future studies may vary γ\gamma). At each episode, the initial conditions are uniformly randomized according to u(x,0)∼Uniform⁡(1,10)u(x,0)\sim\operatorname{Uniform}(1,10), and we note that the system is always open-loop unstable for all possible initial conditions (See Figure 8 in Appendix). For training, we follow the same procedure as § 5.1 except that we require a finer simulation resolution of Δx=0.005\Delta x=0.005, and the PDE is simulated at Δt=0.00001\Delta t=0.00001 due to the approximation of the second spatial derivative in the reaction-diffusion PDE.

Figure 8. Instability of the reaction-diffusion with β(x)=50cos⁡(8cos⁡−1(x))\beta(x)=50\cos(8\cos^{-1}(x)) under a openloop control signal (U(t)=0U(t)=0).

Results Figure 1 presents the average reward over 1 million training steps. Unlike the hyperbolic PDE where both RL algorithms performed relatively equal, the PPO algorithm achieved better performance during training which is corroborated by the testing rewards in the middle column of Table 3. Figure 3 demonstrates a test case with u(x,0)=10u(x,0)=10. Similar to the transport PDE, we observe oscillations in the RL feedback laws, suggesting potential improvement via enforcing continuity constraints as in Asadi et al. [1]. Notably, in Figure 3, perhaps due to reward shaping, PPO differs from the backstepping controller’s approach, but maintains excellent performance.

Figure 3. Example of reaction-diffusion PDE system stabilization using backstepping, PPO, and SAC (left to right) under initial conditions u(x,0)=10u(x,0)=10. The recirculation coefficient using the Chebyshev polynomial defined as λ(x)=50cos⁡(γcos⁡−1(x))\lambda(x)=50\cos(\gamma\cos^{-1}(x)) with γ=8\gamma=8.

2D Navier-Stokes PDEs

Experimental design For the Navier-Stokes 2D problem (§ 4.3), both velocity components are zero initially, i.e., u(x,y,0)=v(x,y,0)=0u(x,y,0)=v(x,y,0)=0. We apply boundary control on the top boundary with tangential, uniform controlled velocity, setting u(x,1,t)=U(t)∈Ru(x,1,t)=U(t)\in\mathbb{R} and v(x,1,t)=0v(x,1,t)=0. For implementation, we discretize the state space with a spatial step of Δx=0.05\Delta x=0.05 and the PDE is simulated at Δt=0.001\Delta t=0.001. The reward for training is derived from the negative of the cost in optimization (15). The reference velocity vector uref\mathbf{u}_{\mathrm{ref}} is the resulted velocity vector under the boundary control U(t)=3−5tU(t)=3-5t, and Uref=2.0U_{\mathrm{ref}}=2.0.

Results Figure 1 (right) shows the average reward per episode for PPO and SAC, with PPO outperforming SAC both in terms of higher final rewards and more stable training curves. Table 3 (right) details the average episodic rewards over 50 test episodes, where PPO surpasses both SAC and the model-based optimization algorithm, which often gets stuck in local optima. Despite the reward difference, on a singular example presented in Figure 4, all methods effectively track the reference velocity vectors.

Figure 4. Example of Navier-Stokes PDE tracking using optimization-based control, PPO, and SAC under initial conditions u(x,y,0)=0u(x,y,0)=0 at t=0t=0 (top) and t=0.2t=0.2 (bottom). Red and black arrows represent the actual and reference velocity field respectively. The background color represents the magnitude of the velocity vector.

Conclusion

Future work Throughout this paper, we have mentioned several avenues for future research based on the PDE Control Gym. As such, we conclude this work by briefly summarizing these ideas. We employed relatively simple policy network architectures in our RL algorithms, not fully fine-tuning them to the specific problems. The PDE Control Gym presents opportunities to optimize policy network structures, improve reward shaping, and develop better RL algorithms for PDE control tasks. Additionally, our experiments were based on time-invariant linear instability coefficients where β(x)\beta(x) and λ(x)\lambda(x) are unknown but static during RL training. Thus, there is much to be explored for model-free controllers when considering time-varying models, adaptive control, and sensing noise. Furthermore, given the superior performance of backstepping controllers, investigating the potential of pre-training RL methods through imitation learning could be a valuable direction.

Conclusion In this study, we introduced the first benchmark suite for learning-based boundary control of PDEs. We developed RL gyms for three fundamental PDE control problems: the 1D transport PDE, 1D reaction-diffusion PDE, and 2D Navier Stokes PDE. This gym allows for the separation of algorithm design from the numerical implementation of PDEs. Moreover, we trained a series of model-free RL models on the three benchmarks and compared their performance with model-based PDE backstepping and optimization methods. Finally, our work discussed multiple avenues for future research, aiming to inspire new research in the challenging field of PDE control.

Hyperbolic Partial Differential Equation

First-Order Finite Difference Scheme

We consider the benchmark transport PDE in the form

ut=ux+β(x)u(0,t),(x,t)∈[0,1)×R+,(18)u_t = u_x + \beta(x)u(0,t), \qquad(x,t) \in[0,1) \times\mathbb{R}_{+}, \tag*{(18)}

with a single boundary condition representing the control input in either Dirichlet Neumann form at x=1x=1 (u(1,t)=U(t)u(1,t)=U(t)). Notice that instability is caused by the recirculation term β(x)u(0,t)\beta(x)u(0,t) - otherwise the resulting PDE with be considered an instantiation of the inviscid Burger’s equation. For the numerical scheme, consider the first-order Taylor approximation for uu and the resulting substitution of derivatives yields

ujn+1=ujn+Δt(uj+1n−ujnΔx+βju0n),(19)u_j^{n+1} = u_j^n + \Delta t \left( \frac{u_{j+1}^n-u_j^n}{\Delta x} + \beta_j u_0^n \right), \tag*{(19)}

where Δt\Delta t denotes the temporal timestep, Δx\Delta x denotes the spatial timestep, n=0,…,Ntn=0,\ldots,N_t, j=0,…,Nxj=0,\ldots,N_x where NtN_t and NxN_x are the total number of temporal and spatial steps respectively. To enforce Neumann boundary conditions, let uζn∣ζ=Nxu_\zeta^n|_{\zeta=N_x} represent the spatial derivative at time tt of spatial point NxN_x which is given by the user as control input. Then, we have

uζn∣ζ=Nx=uNxn−uNx−1nΔx,(20)u_\zeta^n|_{\zeta=N_x} = \frac{u_{N_x}^n-u_{N_{x-1}}^n}{\Delta x}, \tag*{(20)}

which is rearranged for the final boundary point

uNxn=uNx−1n+(Δx)uζn∣ζ=Nx.(21)u_{N_x}^n = u_{N_{x-1}}^n + (\Delta x)u_\zeta^n|_{\zeta=N_x}. \tag*{(21)}

In the case of Dirichlet boundary conditions, the computation is straightforward as uNxnu_{N_x}^n is directly set as the given control input. For stabilization of the finite-difference scheme, it is recommended that timestep be much much smaller than the spatial step LeVeque [27].

Details on Numerical Implementations for Benchmark 1D Hyperoblic PDE

Reinforcement learning baselines: Hyperparameters for proximal policy optimization (PPO) and soft-actor critic (SAC)

We provide the entire set of hyperparameters for training the PPO and SAC algorithms in Table 4 and 5 respectively. For a details on the RL learning algorithms, see Schulman et al. [38] (PPO) and Haarnoja et al. [15] (SAC).

PPO ParameterValue
Learning Rate0.0003
Num Steps per Update2048
Batch size64
Num Epochs per Surrogate Loss Update10
Discount Factor γ\gamma0.99
Bias vs Variance Trade-Off for Advantage Estimator0.95
Clipping Parameter ϵ\epsilon0.2
Entropy Coefficient for Loss0.0
Value Function Coefficient for Loss0.5
Max Value for Gradient Clipping0.5

Table 4. Parameters for PPO Model Trained

Experimental Design for 1D Hyperbolic PDE

We now discuss the construction of each episode for reinforcement learning which is concurrently summarized as a Markov Decision Process (MDP) in Table 6. For this work, we consider a simplified version of the PDE where γ\gamma is fixed for the entire training process to γ=7.35\gamma= 7.35. Then, each episode randomly begins with an initial condition sampled from u(x,0)∼Uniform⁡(1,10)u(x, 0) \sim\operatorname{Uniform}(1, 10). This creates a sufficiently challenging PDE to control as for any initial condition, the PDE is open loop unstable. As an example, we present the PDEs for u(x,0)=1,10u(x, 0) = 1, 10 in the left and right of Figure 5.

Markov Decision Process TupleParameter for Hyperbolic PDE Benchmark Example
S\mathcal{S} - state spaceS⊆C[0,1]\mathcal{S} \subseteq C[0,1]
A\mathcal{A} - action spaceA⊆{x∈R:−20≤x≤20}\mathcal{A} \subseteq\{x \in\mathbb{R} : -20 \le x \le20\}
O\mathcal{O} - observation spaceO=S\mathcal{O} = \mathcal{S} (can be varied to user choice)
p0p_0 - initial sample pdf for u(x,0)∈Su(x,0) \in\mathcal{S}p0⊆{f∈C[0,1]∣f(x)=c, ∀x∈[0,1]}p_0 \subseteq\{f \in C[0,1] \mid f(x) = c,\ \forall x \in[0,1]\} where c∼Uniform⁡(1,10)c \sim\operatorname{Uniform}(1,10)
pf(⋅∣xt,at)p_f(\cdot\mid x_t,a_t) - state transitionDynamics described by PDE in (5) with u(1,t)=atu(1,t) = a_t
TT - time horizon5 seconds
ra(s,s′)r_a(s,s') - reward (s∈S)(s \in\mathcal{S})ra(s,s′)=−1∗∥s′−s∥L2r_a(s,s') = -1 * \lVert s' - s \rVert_{L_2}
q(sT,a0,…,aT)q(s_T,a_0,\ldots,a_T) terminal cost (sT∈S)(s_T \in\mathcal{S})q(sT,a0,…,aT)={0∥sT∥L2>ζσ−1/η∗∑τ=0T∣aτ∣L1−∥st∥L2∥sT∥L2≤ζq(s_T,a_0,\ldots,a_T) = \begin{cases} 0 & \lVert s_T \rVert_{L_2} > \zeta\\ \sigma- 1/\eta* \sum_{\tau=0}^{T} \lvert a_\tau\rvert_{L_1} - \lVert s_t \rVert_{L_2} & \lVert s_T \rVert_{L_2} \le\zeta\end{cases} where σ,η,ζ\sigma,\eta,\zeta are hyperparameters

Table 6. Markov Decision Process describing the PDE example in Section A.2

For the RL policies, we utilized a novel reward function designed to accurately handle the challenging oscillations of hyperbolic PDEs (See Figure 6 with the backstepping controller for an example). With this, we developed the following reward function for 1D PDEs:

Figure 6. Examples of PDE system stabilization using backstepping, PPO, and SAC (left to right) under two different initial conditions u(x,0)=1,10u(x,0)=1,10. The PDE has functional recirculation coefficient using the Chebyshev polynomial defined as β(x)=5cos⁡(γcos⁡−1(x))\beta(x)=5\cos(\gamma\cos^{-1}(x)) with γ=7.35\gamma=7.35. The control signal for each approach is marked in red and in Figure 7.

Figure 7. The corresponding control signal given to the PDEs in Figure 6. The control signals on the left match the PDE with u(x,0)=1u(x,0)=1 (top-row of Figure 6) and the signals on the right correspond to the PDE with u(x,0)=10u(x,0)=10 (bottom-row of Figure 6) respectively.

ra(s,s′)=−1∗∥s′−s∥L2,(22)r_a(s,s') = -1 * \lVert s' - s \rVert_{L_2}, \tag*{(22)}
q(sT,a0,…,aT)={0∥sT∥L2>20,σ−1/η∗∑τ=0T∣aτ∣L1−∥st∥L2∥sT∥L2≤20(23)q(s_T,a_0,\ldots,a_T) = \begin{cases} 0 & \lVert s_T \rVert_{L_2} > 20, \\ \sigma- 1/\eta* \sum_{\tau=0}^{T} \lvert a_\tau\rvert_{L_1} - \lVert s_t \rVert_{L_2} & \lVert s_T \rVert_{L_2} \le20 \end{cases} \tag*{(23)}

where σ,η,ζ\sigma,\eta,\zeta were hyperparameters set to 300, 1000, 20. The intuition of this reward function is in two components. First, ra(s,s′)r_a(s,s') rewards control inputs that force the state to become smaller compared to its previous state leading to an encouragement towards stabilization. Then, at the termination of the episode, if the final state L2L_2 norm is small enough (≤20\le20), we give the policy a large reward which is now penalized by the control costs - ie, we only value control costs for the policy after we know that the policy for the episode is stabilizing. This trade-off is commonly found in a series of optimal control problems - but due to the challenging nature of PDEs, one is usually satisfied with just stabilization guarantees without limits on the control inputs. This highlights one of the advantages of using model-free approaches where one can begin to enforce a stabilization first, minimization of control second approach. Lastly, we note that many rewards will encourage successful policies and we leave it to the community to develop better tuned rewards for these benchmark PDE problems.

For training, we run each model with parameters listed in Table 4 and Table 5 (standard defaults for stable-baselines3, non-optimized) in the environment governed by the MDP in Table 6. From an implementation perspective, we discretize both the state space using a spatial step of dx=0.01dx = 0.01 and further discretize each timestep of the MDP by dt=0.01dt = 0.01 as well. However, we note that the first-order PDE scheme in Section A.1 requires a much smaller timestep then spatial step and thus, the PDE is internally simulated at a timestep of dtPDE=0.0001dt_{PDE} = 0.0001 where it receives a new control input at every 0.01 second and maintains the same input for the higher resolution PDE simulation until the next 0.01 second is reached. Naturally, it is worth considering sampling of the controller at the same rate of the PDE - but this results in poor performance for the reinforcement learning models and is less indicative of a real-world application given it is challenging to provide control inputs at frequencies larger than 100Hz.

SAC ParameterValue
Learning Rate0.0003
Buffer Size1000000
Batch Size256
Soft Update Coefficient τ\tau0.005
Discount factor γ\gamma0.99
Action NoiseNone

Table 5. Parameters for SAC Model Trained

Case Study: Hyperbolic PDE

In this section, we show the results for two individual initial conditions for the Hyperbolic PDE problem, namely u(x,0)=0u(x,0)=0 and u(x,0)=10u(x,0)=10.

Initial ConditionControl AlgorithmTotal Episode Reward ↑\uparrowTotal Episode State Summed L2L_2 Norm (∑t=0T∥u(⋅,t)∥L2)↓(\sum_{t=0}^{T}\lVert u(\cdot,t)\rVert_{L_2})\downarrow
u(x,0)=1u(x,0)=1Backstepping289.8106.1
u(x,0)=1u(x,0)=1PPO246.0448.1
u(x,0)=1u(x,0)=1SAC212.9720.4
u(x,0)=10u(x,0)=10Backstepping198.41060.9
u(x,0)=10u(x,0)=10PPO32.72026.4
u(x,0)=10u(x,0)=10SAC133.71402.8

Table 7. Resulting control performance for backstepping, PPO, and SAC on the two PDE examples corresponding to Figure 6.

Reaction-Diffusion Partial Differential Equation

First-Order Finite Difference Scheme

We consider the benchmark reaction-diffusion PDE in the form

ut=uxx+λ(x)u(x,t),(x,t)∈(0,1)×R+,(24)u_t = u_{xx} + \lambda(x)u(x,t), \quad(x,t) \in(0,1) \times\mathbb{R}_+, \tag*{(24)}
u(0,t)=0,(25)u(0,t) = 0, \tag*{(25)}

with the control input again in either Dirichlet or Neumann form at x=1x = 1 (u(1,t)=U(t)u(1,t) = U(t)). As with the transport PDE, the recirculation term λ(x)u(x,t)\lambda(x)u(x,t) causes instability as otherwise the resulting PDE would reduce to the stable heat equation. For the numerical scheme, consider the first-order Taylor approximation for uu as

ujn+1=ujn+Δt(uj−1n−2ujn+uj+1n(Δx)2+λjujn),(26)u_j^{n+1} = u_j^n + \Delta t \left( \frac{u_{j-1}^n - 2u_j^n + u_{j+1}^n}{(\Delta x)^2} + \lambda_j u_j^n \right), \tag*{(26)}

where Δt\Delta t denotes the temporal timestep, Δx\Delta x denotes the spatial timestep, n=0,…,Ntn = 0,\ldots,N_t, j=0,…,Nxj = 0,\ldots,N_x where NtN_t and NxN_x are the total number of time steps respectively. Explicitly we set u0k=0 ∀k∈[0,Nt]u_0^k = 0 \ \forall k \in[0,N_t] for the first boundary condition and the second boundary condition follows as in the Hyperbolic PDE case with (21) for Neumann boundary conditions and uNxnu_{N_x}^n for Dirichlet boundary conditions. Again, we require extremely small spatial and temporal time-steps for stabilization of the scheme.

Details on numerical implementation for benchmark 1D reaction-diffusion PDE

We adopt the exact same, untuned, hyperparameters for PPO and SAC as in Section A.2.1 and adopt the same MDP as presented in Table 6 except that the dynamics are now governed by (24), (25), with u(1,t)=atu(1,t) = a_t and the time-horizon is shortened to 1 second as the algorithms are able to stabilize faster. We choose λ(x)=50cos⁡(γcos⁡−1(x))\lambda(x) = 50\cos(\gamma\cos^{-1}(x)) where γ\gamma is fixed to be 8. With the following configuration and initial conditions again sampled between 1 and 10, we see that the PDE is openloop unstable in Figure 8.

For training, we follow the same procedure as Section A.2 except that we require a finer simulation resolution of dt=0.001dt = 0.001, dx=0.005dx = 0.005, and dtPDE=0.00001dt_{PDE} = 0.00001 due to the approximation of the second spatial derivative in the reaction-diffusion PDE.

Case Study: Parabolic PDE

In this section, we show the results for two individual initial conditions for the parabolic PDE problem, namely u(x,0)=0u(x,0)=0 and u(x,0)=10u(x,0)=10.

Initial ConditionControl AlgorithmTotal Episode Reward ↑\uparrowTotal Episode Summed L2L_2 Norm (∑t=0T∥u(⋅,t)∥L2)↓(\sum_{t=0}^{T}\lVert u(\cdot,t)\rVert_{L_2})\downarrow
u(x,0)=1u(x,0)=1Backstepping299.81275.4
u(x,0)=1u(x,0)=1PPO295.11103.5
u(x,0)=1u(x,0)=1SAC239.81968.7
u(x,0)=10u(x,0)=10Backstepping298.212754.4
u(x,0)=10u(x,0)=10PPO283.223342.4
u(x,0)=10u(x,0)=10SAC140.59624.1

Table 8. Resulting control performance for backstepping, PPO, and SAC on the two PDE examples corresponding to Figure 9.

Example trajectories for $u(0,x)=1$ with backstepping, PPO, and SAC, and example trajectories for $u(0,x)=10$ with backstepping, PPO, and SAC

Figure 9. Examples of PDE system stabilization using backstepping, PPO, and SAC (left to right) under two different initial conditions u(x,0)=1,10u(x,0)=1,10. The PDE has functional recirculation coefficient using the Chebyshev polynomial defined as β(x)=50cos⁡(γcos⁡−1(x))\beta(x)=50\cos(\gamma\cos^{-1}(x)) with γ=8\gamma=8. The control signal for each approach is marked in red and in Figure 10.

Figure 10. The corresponding control signal given to the PDEs in Figure 9. The control signals on the left match the PDE with u(x,0)=1u(x,0)=1 (top-row of Figure 9) and the signals on the right correspond to the PDE with u(x,0)=10u(x,0)=10 (bottom-row of Figure 9) respectively.

Navier Stokes Equation

Second-Order Finite Difference Scheme

We consider 2D benchmark Navier Stokes PDE in the form:

∇⋅u=0,(x,y,t)∈([0,1]×[0,1]×R+),(27)\nabla\cdot u = 0, \qquad(x,y,t) \in([0,1] \times[0,1] \times\mathbb{R}_{+}), \tag*{(27)}
∂u∂t+u⋅∇u=1ρ∇p+ν∇2u,(x,y,t)∈((0,1)×(0,1)×R+),(28)\frac{\partial u}{\partial t} + u \cdot\nabla u = \frac{1}{\rho}\nabla p + \nu\nabla^{2}u, \qquad(x,y,t) \in((0,1) \times(0,1) \times\mathbb{R}_{+}), \tag*{(28)}
u(x,0,t)=U(x,t),(x,t)∈([0,1]×R+),(29)u(x,0,t) = U(x,t), \qquad(x,t) \in([0,1] \times\mathbb{R}_{+}), \tag*{(29)}
u(x,1,t)=u(1,y,t)=u(0,y,t)=0,(x,y,t)∈([0,1]×[0,1]×R+),(30)u(x,1,t) = u(1,y,t) = u(0,y,t) = 0, \qquad(x,y,t) \in([0,1] \times[0,1] \times\mathbb{R}_{+}), \tag*{(30)}

with boundary control input in the top boundary and the velocity at all other boundaries is set to be zero. We incorporate a predictor-corrector Le and Moin [26] scheme for the imcompressive Navier Stokes equations. In the scheme, we denote u=(u,v)u = (u,v) is the velocity vector, and spatial position is x=(x,y)x = (x,y).

We first present the predictor step that does not consider the pressure,

ui,j∗=ui,jn+Δt(ν(ui−1,jn−2ui,jn+ui+1,jn(Δx)2+ui,j−1n−2ui,jn+ui,j+1n(Δy)2)−Δt(ui,jnui+1,jn−ui−1,jn2Δx+vi,jnui,j+1n−ui,j−1n2Δy)),(31)u_{i,j}^{*} = u_{i,j}^{n} + \Delta t \left(\nu\left(\frac{u_{i-1,j}^{n} - 2u_{i,j}^{n} + u_{i+1,j}^{n}}{(\Delta x)^{2}} + \frac{u_{i,j-1}^{n} - 2u_{i,j}^{n} + u_{i,j+1}^{n}}{(\Delta y)^{2}}\right) - \Delta t \left(u_{i,j}^{n}\frac{u_{i+1,j}^{n} - u_{i-1,j}^{n}}{2\Delta x} + v_{i,j}^{n}\frac{u_{i,j+1}^{n} - u_{i,j-1}^{n}}{2\Delta y}\right)\right), \tag*{(31)}
vi,j∗=vi,jn+Δt(ν(vi−1,jn−2vi,jn+vi+1,jn(Δx)2+vi,j−1n−2vi,jn+vi,j+1n(Δy)2)−Δt(ui,jnvi+1,jn−vi−1,jn2Δx+vi,jnvi,j+1n−vi,j−1n2Δy)),(32)v_{i,j}^{*} = v_{i,j}^{n} + \Delta t \left(\nu\left(\frac{v_{i-1,j}^{n} - 2v_{i,j}^{n} + v_{i+1,j}^{n}}{(\Delta x)^{2}} + \frac{v_{i,j-1}^{n} - 2v_{i,j}^{n} + v_{i,j+1}^{n}}{(\Delta y)^{2}}\right) - \Delta t \left(u_{i,j}^{n}\frac{v_{i+1,j}^{n} - v_{i-1,j}^{n}}{2\Delta x} + v_{i,j}^{n}\frac{v_{i,j+1}^{n} - v_{i,j-1}^{n}}{2\Delta y}\right)\right), \tag*{(32)}

where Δt\Delta t denotes the temporal timestep, Δx\Delta x, Δy\Delta y denotes the spatial time step, n=0,…,Ntn = 0,\ldots,N_t, i=0,…,Nxi = 0,\ldots,N_x, j=0,…,Nyj = 0,\ldots,N_y where NtN_t, NxN_x, NyN_y are the total number of timesteps, respectively. ui,j∗,vi,j∗u_{i,j}^{*}, v_{i,j}^{*} are the resulted velocity field that does consider pressure.

To satisfy continuity equation ∇⋅u=0\nabla\cdot u = 0, we get the pressure Poisson equation:

∇2p=∂2p∂x2+∂2p∂y2=−ρ(∂2u∂x2+2∂u∂x∂v∂y+∂2v∂y2).(33)\nabla^{2}p = \frac{\partial^{2}p}{\partial x^{2}} + \frac{\partial^{2}p}{\partial y^{2}} = -\rho\left(\frac{\partial^{2}u}{\partial x^{2}} + 2\frac{\partial u}{\partial x}\frac{\partial v}{\partial y} + \frac{\partial^{2}v}{\partial y^{2}}\right). \tag*{(33)}

Then, we iteratively solve the Poisson equation for pressure:

pi,jiter+1←12(Δx2+Δy2)((pi+1,jiter+pi−1,jiter)Δy2+(pi,j+1iter+pi,j−1iter)Δx2)+ρ(ui+1,j∗−ui−1,j∗2Δx+vi+1,j∗−vi−1,j∗2Δy)Δx2Δy2,(34)p_{i,j}^{iter+1} \leftarrow\frac{1}{2(\Delta x^{2} + \Delta y^{2})}\left((p_{i+1,j}^{iter} + p_{i-1,j}^{iter})\Delta y^{2} + (p_{i,j+1}^{iter} + p_{i,j-1}^{iter})\Delta x^{2}\right) + \rho\left(\frac{u_{i+1,j}^{*} - u_{i-1,j}^{*}}{2\Delta x} + \frac{v_{i+1,j}^{*} - v_{i-1,j}^{*}}{2\Delta y}\right)\Delta x^{2}\Delta y^{2}, \tag*{(34)}

with iteriter denotes the iteration of the procedure for solving pressure. We denote the pressure solution as p∗p^{*}. Then we can perform the corrector step to compute the velocity vector at next time step:

ui,jn+1=ui,j∗−Δt⋅1ρpi+1,j∗−pi−1,j∗2Δx,(35)u_{i,j}^{n+1} = u_{i,j}^{*} - \Delta t \cdot\frac{1}{\rho}\frac{p_{i+1,j}^{*} - p_{i-1,j}^{*}}{2\Delta x}, \tag*{(35)}
vi,jn+1=vi,j∗−Δt⋅1ρpi,j+1∗−pi,j−1∗Δy.(36)v_{i,j}^{n+1} = v_{i,j}^{*} - \Delta t \cdot\frac{1}{\rho}\frac{p_{i,j+1}^{*} - p_{i,j-1}^{*}}{\Delta y}. \tag*{(36)}

We apply boundary conditions every time step after the predictor step and corrector step.

Control Algorithms Implemented for Navier Stokes PDE

Model-Based Optimization

In optimization algorithms, the optimal control is computed as solution of an optimization problem, where the partial differential equations appear as equality constraints. In the experiment, we consider to track the velocity vector to be the desired trajectory uref(x,t)u_{ref}(x,t) and to minimize the control cost with a reference control UrefU_{ref}. The optimization problem (i.e. optimal control) is formulated as follows,

min⁡U(t)J(U(t),u),(37)\begin{aligned} \min_{U(t)}\quad&J(U(t),\mathbf{u}), \tag*{(37)} \end{aligned}
s.t.∂u∂t=−u⋅∇u+1ρ∇p−ν∇2u,(38)\begin{aligned} \text{s.t.}\quad&\frac{\partial\mathbf{u}}{\partial t}=-\mathbf{u}\cdot\nabla\mathbf{u}+\frac{1}{\rho}\nabla p-\nu\nabla^{2}\mathbf{u}, \tag*{(38)} \end{aligned}
∇⋅u=0,(39)\nabla\cdot\mathbf{u}=0, \tag*{(39)}
u(x∈Γ,t)=U(t)(40)\mathbf{u}(x\in\Gamma,t)=U(t) \tag*{(40)}

where Γ\Gamma is the defined controllable boundary. The optimization problem can be solved using the augmented Lagrangian:

L(U,u,λ(x,t),ν(x,t))=J(U(t),u)+∬Aμ(∇⋅u)+∬AλT(∂u∂t+u⋅∇u=−1ρ∇p+ν∇2u).(41)\begin{aligned} L(U,\mathbf{u},\lambda(x,t),\nu(x,t))={}&J(U(t),\mathbf{u})+\iint_A\mu(\nabla\cdot\mathbf{u})\\ &+\iint_A\lambda^{T}\left(\frac{\partial\mathbf{u}}{\partial t}+\mathbf{u}\cdot\nabla\mathbf{u}=-\frac{1}{\rho}\nabla p+\nu\nabla^{2}\mathbf{u}\right). \tag*{(41)} \end{aligned}

A locally optimal solution is characterized by satisfying the first-order Karush-Kuhn-Tucker (KKT) conditions, which necessitate that the derivatives of the Lagrangian function LL with respect to variables UU, u\mathbf{u}, λ\lambda, μ\mu are all zero. Specifically, taking the derivatives with respect to λ\lambda and μ\mu yields the Navier-Stokes Equations. Additionally, the solution λ\lambda is required to be divergence-free, a condition that arises from the derivative of LL with respect to the pressure pp. The derivative of LL with respect to u\mathbf{u} leads to the differential equation for λ\lambda:

∂∂tλ=−(G+GT)u−ν∇2λ−∇μ+(u−uref),(42)\frac{\partial}{\partial t}\lambda=-(G+G^{T})\mathbf{u}-\nu\nabla^{2}\lambda-\nabla\mu+(\mathbf{u}-u_{ref}), \tag*{(42)}

where G=∂λ∂xG=\frac{\partial\lambda}{\partial x} represents the Jacobian of λ\lambda with respect to xx. The differential equation can be solve backwards in time with λ(x,T)=0\lambda(x,T)=0 and homogeneous boundary conditions Pyta et al. [34]. The derivative with respect to UU leads to

U=Uref−νγ∫Γ∂λ1∂x2.(43)U=U_{ref}-\frac{\nu}{\gamma}\int_{\Gamma}\frac{\partial\lambda_{1}}{\partial x_{2}}. \tag*{(43)}

Thus the (local) optimal control function U(t)U(t) can be computed by 1) Solving Navier Stokes equation to get the velocity vector u(x,y,t)\mathbf{u}(x,y,t) 2) Solving adjoint equation (42) for λ(x,y,t)\lambda(x,y,t) backwards from time TT to 0 with u(x,y,t)\mathbf{u}(x,y,t) 3) Solving equation (43) for U(t)U(t) forwards with λ(x,y,t)\lambda(x,y,t).

Reinforcement learning baselines: proximal policy optimization (PPO) and soft-actor critic (SAC)

We adopt the exact same, untuned, hyperparameters for PPO and SAC as in Section A.2.1.

Details on numerical implementation for benchmark 2D Navier Stokes PDE

We adopt the MDP as presented in Table 9. For training, we run each model with parameters listed in 4, 5 (standard defaults for stable-baselines3, non-optimized) in the environment governed by the MDP in Table 9. From an implementation perspective, we discretize both the state space using a spatial step of dx=0.05dx = 0.05 and further discretize each timestep of the MDP by dt=0.001dt = 0.001 as well. The PDE is internally simulated at a timestep of dtPDE=dt=0.001dt_{PDE} = dt = 0.001. We use the negative of the cost to be the reward for training. The reference velocity vector srefs_{ref} is the resulted velocity vector under the boundary control U(t)=3−5tU(t) = 3 - 5t, and the reference control uref=aref=2u_{ref} = a_{ref} = 2.

Markov Decision Process TupleParameter for Navier Stokes PDE Benchmark Example
SS - state spaceS⊆C([0,1]×[0,1])S \subseteq C([0,1] \times[0,1])
AA - action spaceA⊆{x∈R:−10≤x≤10}A \subseteq\{x \in\mathbb{R} : -10 \le x \le10\}
OO - observation spaceO=SO = S (can be modified to partially observable)
p0p_0 - initial samplep0⊆{f∈C[0,1]×[0,1]∣f(x,y)=(0,0) ∀x∈[0,1],y∈[0,1]}p_0 \subseteq\{f \in C[0,1] \times[0,1] \mid f(x,y) = (0,0)\ \forall x \in[0,1], y \in[0,1]\}
pf(⋅∣xt,at)p_f(\cdot\mid x_t, a_t) - state transitionDynamics described by PDE in (25) with boundary control u(x,0,t)=atu(x,0,t) = a_t and v(x,0,t)=0v(x,0,t) = 0
TT - time horizon0.2 seconds
ra(s,s′)r_a(s,s') - reward (s∈S)(s \in S)ra(s,s′)=−12∥s′−sref∥L2−γ12∥a−aref∥L2,γ=0.1,aref=2r_a(s,s') = -\frac{1}{2}\lVert s' - s_{ref}\rVert_{L_2} - \gamma\frac{1}{2}\lVert a - a_{ref}\rVert_{L_2}, \gamma= 0.1, a_{ref} = 2

Table 9. Markov Decision Process describing the PDE example in Section C.3

References

  1. [1]Kavosh Asadi, Dipendra Misra, and Michael L. Littman. Lipschitz continuity in model-based reinforcement learning, 2018.arxiv.org/abs/1804.07193
  2. [2]Julian Berberich, Carsten W. Scherer, and Frank Allgöwer. Combining prior knowledge and data for robust controller design. IEEE Transactions on Automatic Control, 68(8):4618–4633, 2023. doi: 10.1109/TAC.2022.3209342.DOI
  3. [3]Luke Bhan, Yuanyuan Shi, and Miroslav Krstic. Neural operators for bypassing gain and control computations in PDE backstepping. IEEE Transactions on Automatic Control, pages 1–16, 2023a. doi: 10.1109/TAC.2023.3347499.DOI
  4. [5]Yuexin Bian, Xiaohan Fu, Rajesh K Gupta, and Yuanyuan Shi. Ventilation and temperature control for energy-efficient and healthy buildings: A differentiable pde approach. arXiv preprint arXiv:2403.08996, 2024.arxiv.org/abs/2403.08996
  5. [6]Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5:411–444, 2022.arxiv.org/abs/2108.06266
  6. [7]Xin Chen, Guannan Qu, Yujie Tang, Steven Low, and Na Li. Reinforcement learning for selective key applications in power systems: Recent advances and future challenges. IEEE Transactions on Smart Grid, 13(4):2935–2958, 2022.DOI
  7. [8]Cenk Demir, Shumon Koga, and Miroslav Krstic. Neuron growth control and estimation by pde backstepping. Automatica, 165:111669, 2024. ISSN 0005-1098. doi: https://doi.org/10.1016/j.automatica.2024.111669. URL https://www.sciencedirect.com/science/article/pii/S0005109824001626.
  8. [9]Florent Di Meglio, Rafael Vazquez, Miroslav Krstic, and Nicolas Petit. Backstepping stabilization of an underactuated 3 × 3 linear hyperbolic system of fluid flow equations. In 2012 American Control Conference (ACC), pages 3365–3370, 2012.
  9. [10]Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pages 1329–1338. PMLR, 2016.arxiv.org/abs/1604.06778
  10. [11]Jie Feng, Yuanyuan Shi, Guannan Qu, Steven H. Low, Anima Anandkumar, and Adam Wierman. Stability constrained reinforcement learning for decentralized real-time voltage control. IEEE Transactions on Control of Network Systems, pages 1–12, 2023. doi: 10.1109/TCNS.2023.3338240.DOI
  11. [12]Max D Gunzburger. Perspectives in flow control and optimization. SIAM, 2002.
  12. [13]Jayesh K Gupta and Johannes Brandstetter. Towards multi-spatiotemporal-scale generalized PDE modeling. arXiv:2209.15616, 2022.arxiv.org/abs/2209.15616
  13. [14]Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. PMLR, 2018a.arxiv.org/abs/1801.01290
  14. [15]Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861–1870. PMLR, 10–15 Jul 2018b. URL https://proceedings.mlr.press/v80/haarnoja18b.html.arxiv.org/abs/1801.01290
  15. [16]Philipp Holl, Nils Thuerey, and Vladlen Koltun. Learning to control PDEs with differentiable physics. In International Conference on Learning Representations (ICLR), 2020.arxiv.org/abs/2001.07457
  16. [17]Miroslav Krstic Huan Yu. Traffic Congestion Control by PDE Backstepping. Birkhäuser Cham, 2023.DOI
  17. [18]Mojtaba Izadi, Javad Abdollahi, and Stevan S. Dubljevic. PDE backstepping control of one-dimensional heat equation with time-varying domain. Automatica, 54:41–48, 2015.DOI
  18. [19]A. Jameson, L. Martinelli, and N. A. Pierce. Optimum aerodynamic design using the Navier-Stokes equations. Theoretical and Computational Fluid Dynamics, 10(1):213–237, Jan 1998.
  19. [20]B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(6):4909–4926, 2021.arxiv.org/abs/2002.00444
  20. [21]Shumon Koga and Miroslav Krstic. Materials Phase Change PDE Control & Estimation. Birkhäuser, 2020.DOI
  21. [22]Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25, 2012.
  22. [23]Miroslav Krstic and Andrey Smyshlyaev. Backstepping boundary control for first-order hyperbolic PDEs and application to systems with actuator and sensor delays. Systems & Control Letters, 57 (9):750–758, 2008a.
  23. [24]Miroslav Krstic and Andrey Smyshlyaev. Boundary Control of PDEs. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2008b.
  24. [25]Miroslav Krstic, Luke Bhan, and Yuanyuan Shi. Neural operators of backstepping controller and observer gain functions for reaction–diffusion PDEs. Automatica, 164:111649, 2024. ISSN 0005-1098. doi: https://doi.org/10.1016/j.automatica.2024.111649. URL https://www.sciencedirect.com/science/article/pii/S0005109824001420.
  25. [26]Hung Le and Parviz Moin. An improvement of fractional step methods for the incompressible Navier-Stokes equations. Journal of computational physics, 92(2):369–379, 1991.DOI
  26. [27]Randall J. LeVeque. Numerical methods for conservation laws (2. ed.). Lectures in mathematics. Birkhäuser, 1992. ISBN 978-3-7643-2723-1.
  27. [28]Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations (ICLR), 2021.
  28. [30]Antoine McNamara, Adrien Treuille, Zoran Popović, and Jos Stam. Fluid control using the adjoint method. ACM Transactions On Graphics (TOG), 23(3):449–456, 2004.
  29. [31]Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.arxiv.org/abs/2003.08934
  30. [32]Scott J Moura, Nalin A Chaturvedi, and Miroslav Krstić. Adaptive partial differential equation observer for battery state-of-charge/state-of-health estimation via an electrochemical model. Journal of Dynamic Systems, Measurement, and Control, 136(1):011015, 2014.DOI
  31. [33]Saviz Mowlawi and Saleh Nabi. Optimal control of PDEs using physics-informed neural networks. Journal of Computational Physics, 473:111731, 2023.
  32. [34]Lorenz Pyta, Michael Herty, and Dirk Abel. Optimal feedback control of the incompressible Navier-Stokes-equations using reduced order models. In 2015 54th IEEE Conference on Decision and Control (CDC), pages 2519–2524. IEEE, 2015.DOI
  33. [35]Jie Qi, Jing Zhang, and Miroslav Krstic. Neural operators for delay-compensating control of hyperbolic PIDEs. arXiv:2307.11436, 2023.
  34. [36]Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-Baselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research, 22(268):1–8, 2021.
  35. [37]M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019.DOI
  36. [38]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
  37. [39]Yuanyuan Shi, Zongyi Li, Huan Yu, Drew Steeves, Anima Anandkumar, and Miroslav Krstic. Machine learning accelerated PDE backstepping observers. In 2022 IEEE 61st Conference on Decision and Control (CDC), pages 5423–5428, 2022.arxiv.org/abs/2211.15044
  38. [40]A. Smyshlyaev and M. Krstic. Closed-form boundary state feedbacks for a class of 1-D partial integro-differential equations. IEEE Transactions on Automatic Control, 49(12):2185–2202, 2004.DOI
  39. [41]A. Smyshlyaev and M. Krstic. Adaptive Control of Parabolic PDEs. Princeton University Press, 2010.DOI
  40. [42]Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. PDEBench: an extensive benchmark for scientific machine learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 1596–1611, 2022.arxiv.org/abs/2210.07182
  41. [43]Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. CoRR, abs/1801.00690, 2018.arxiv.org/abs/1801.00690
  42. [44]Rafael Vazquez, Guangwei Chen, Junfei Qiao, and Miroslav Krstic. The power series method to compute backstepping kernel gains: Theory and practice. In 2023 62nd IEEE Conference on Decision and Control (CDC), pages 8162–8169, 2023. doi: 10.1109/CDC49753.2023.10384080.DOI
  43. [45]Haiyan Wang, Feng Wang, and Kuai Xu. Modeling information diffusion in online social networks with partial differential equations, volume 7. Springer Nature, 2020.
  44. [46]Huan Yu, Saehong Park, Alexandre Bayen, Scott Moura, and Miroslav Krstic. Reinforcement learning versus PDE backstepping and PI control for congested freeway traffic. IEEE Transactions on Control Systems Technology, 30(4):1595–1611, 2022.arxiv.org/abs/1904.12957
  45. [47]Xiangyuan Zhang, Weichao Mao, Saviz Mowlavi, Mouhacine Benosman, and Tamer Başar. Controlgym: Large-scale control environments for benchmarking reinforcement learning algorithms, 2024.arxiv.org/abs/2311.18736

Paper details

vlnc:W_m4nupdty
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Engineering

Not recorded for this paper

View statement
Looking for related papers…
Contents