betaFeedback

WorksMathematics

Spatio-Temporal Differential Dynamic Programming for Control of Fields

Abstract

We consider the optimal control problem of a general nonlinear spatio-temporal system described by Partial Differential Equations (PDEs). Theory and algorithms for control of spatio-temporal systems are of rising interest among the automatic control community and exhibit numerous challenging characteristic from a control standpoint. Recent methods focus on finite-dimensional optimization techniques of a discretized finite dimensional ODE approximation of the infinite dimensional PDE system. In this paper, we derive a differential dynamic programming (DDP) framework for distributed and boundary control of spatio-temporal systems in infinite dimensions that is shown to generalize both the spatio-temporal LQR solution, and modern finite dimensional DDP frameworks. We analyze the convergence behavior and provide a proof of global convergence for the resulting system of continuous-time forward-backward equations. We explore and develop numerical approaches to handle sensitivities that arise during implementation, and apply the resulting STDDP algorithm to a linear and nonlinear spatio-temporal PDE system. Our framework is derived in infinite dimensional Hilbert spaces, and represents a discretization-agnostic framework for control of nonlinear spatio-temporal PDE systems.

Introduction

Many complex natural processes are governed by systems of equations with spatio-temporal dependence, and are typically described by Partial Differential Equation (PDE). These systems are ubiquitous in nature and can be found in most disciplines of engineering and applied physics. The range of natural processes includes fluid flow governed by the Navier-Stokes equation, sub-atomic particle systems governed by the Schrodinger equation, activation of neurons governed by the Nagumo equation [1], and flame front propagation in combustion systems governed by the Kuramoto-Sivashinsky equation [2].

Despite their ubiquity in nature and engineering, theory and numerical methods for control of spatio-temporal systems remains challenging due to the time-delay, dramatic under-actuation, high system dimensionality, and multi-modal bifurcations, which are often inherent in their dynamics. Furthermore, existence and uniqueness of solutions remains an open problem for many systems, and when they do exist, they typically only have a weak notion of differentiability. Analysis of their performance must be treated with calculus over functionals, and their state vectors are often described by vectors in an infinite-dimensional time-indexed Hilbert space even for scalar 1-dimensional (1D) PDEs. Put together, mathematically consistent and numerically realizable algorithms for control of spatio-temporal systems represents many of the largest current-day challenges facing the automatic control community.

The majority of recent methods for control of spatio-temporal systems typically reduce PDEs into a finite set of Ordinary Differential Equations (ODEs) through Reduced Order Models (ROMs), and apply standard finite-dimensional optimization methods which result in algorithms specific to the ROM used. Within this paradigm, deep learning methods have successfully been applied on policy networks in the finite dimensional setting for controlling Navier-Stokes systems [3, 4, 5, 6], for soft robotic systems [7, 8], as well as for many other systems [9]. These methods are often specific to a discretization scheme and represent a discretize-then-optimize approach. Some such methods can introduce new phenomena in the latent space representation, as in [10], where the resulting deep Koopman approach can be shown to violate linear stabilizability conditions of the latent space dynamics.

External to the machine-learning literature are infinite-dimensional methods found in the control theory literature [11, 12], which are dominated by linear or linearization-based approaches, which include Linear Quadratic Regulator (LQR) approaches for linear PDEs, and forward-backward approaches, which include approaches due to the Pontryagin Maximum Principle (PMP) [12, 13, 14]. Indeed local linearization methods allow for optimal solutions of an approximate problem, however require knowledge of linearization points a-priori. On the other hand, forward-backward schemes provide a nominal trajectory and optimization-based control update scheme at the expense of the backpropagation of a coupled system equation.

In contrast to Pontryagin methods which yield a state-independent backward equation and an open-loop controller, methods founded on the Bellman principle of optimality utilize backward equations that are state-dependent and yield closed-loop control solutions. Methods such as Differential Dynamic Programming (DDP) have decades of established history in the finite dimensional automatic control literature. Modern variations include control limits [15], state constraints [16], receding horizons [17], belief space control [18, 19], game-theoretic control [20], control on Lie groups [21], and using polynomial chaos variational integrators [22].

A previous attempt exists to extend the DDP framework to spatio-temporal systems in infinite dimensions [23], however this approach has several flaws and mathematical inconsistencies, as pointed out in [24]. Additionally, the DDP method has had significant growth since the early works [25]. Decades of advancement include linearization around the nominal trajectory as opposed to the optimal trajectory which decreases sensitivities of convergence behavior to the initial conditions, regularization in the second order backward equation to increase numerical stability, treatment of state and control constraints, and optimization over time horizon.

In light of the apparent literature gap, this manuscript is devoted to the development of DDP methods for spatio-temporal systems in infinite dimensions. Specifically, we derive the Spatio-Temporal DDP (STDDP) framework incorporating modern theoretical techniques, we demonstrate that the resulting system of forward-backward equations generalizes both the LQR solution in infinite dimensions and DDP in finite dimensions, we provide a proof of convergence for the resulting system of continuous-time forward-backward equations, we explore and develop numerical approaches to handle sensitivities that arise due to discretization, and apply the resulting algorithm to linear and nonlinear spatio-temporal PDE systems. In contrast to recent machine learning methods, our optimization is developed entirely in Hilbert spaces, and represents an optimize-then-discretize approach. As a result, the framework is a continuous-time formulation which is agnostic to discretization scheme during implementation.

Preliminaries and Problem Statement

Let D⊆RnD \subseteq\mathbb{R}^{n} denote a measurable connected open domain of Rn\mathbb{R}^{n} describing the space on which the system evolves. Let S⊆RnS \subseteq\mathbb{R}^{n} denote the boundary of DD, let D‾\overline{D} denote the closure of the domain, i.e. D‾=D∪S\overline{D} = D \cup S, and let T=[t0,tf]T = [t_{0},t_{f}] denote some arbitrary time domain. In fields representation, a general form of a deterministic PDE dynamical system is given by

∂tX(t,x)=F(t,x,X(t,x),Ud(t,x)),x∈D0=N(t,x,X(t,x),Ub(t,x)),x∈SX(t0,x)=X0(x),x∈D‾,(1)\begin{aligned} \partial_{t}X(t,x) &= F(t,x,X(t,x),U_{d}(t,x)), \qquad x \in D \\ 0 &= N(t,x,X(t,x),U_{b}(t,x)), \qquad x \in S \\ X(t_{0},x) &= X_{0}(x), \qquad x \in\overline{D}, \tag*{(1)} \end{aligned}

where X:T×D→RnX:T \times D \to\mathbb{R}^{n} is the state. This problem has two measurable control functions, Ub:T×S→RlU_{b}:T \times S \to\mathbb{R}^{l} which correspond to actuation on the boundary, and Ud:T×D→RkU_{d}:T \times D \to\mathbb{R}^{k} which corresponds to actuation distributed throughout the field excluding the boundary. The dynamics evolve by some measurable functional F:T×D×Rn×Rk→RnF:T \times D \times\mathbb{R}^{n} \times\mathbb{R}^{k} \to\mathbb{R}^{n} that is potentially nonlinear in the state function X(t,x)X(t,x) or the control function Ud(t,x)U_{d}(t,x), with a boundary condition functional N:T×S×Rn×Rl→RnN:T \times S \times\mathbb{R}^{n} \times\mathbb{R}^{l} \to\mathbb{R}^{n} that is also potentially a nonlinear functional of the state or control functions, and can be any type of boundary condition (e.g. Neumann, Dirichlet, etc.).

We can equivalently write eqs. (1) to (3) in the time-indexed Hilbert spaces perspective by first properly defining Hilbert spaces, as in [24]. Let L2n(D)L_{2}^{n}(D) denote the Hilbert space of nn-vector functions square integrable over DD with inner product

⟨X1,X2⟩D=∫DX1T(x)X2(x) dx,(2)\langle X_{1},X_{2}\rangle_{D} = \int_{D} X_{1}^{\mathsf{T}}(x)X_{2}(x)\,\mathrm{d}x, \tag*{(2)}

where dx=dx1dx2⋯dxn\mathrm{d}x = \mathrm{d}x_{1}\mathrm{d}x_{2}\cdots\mathrm{d}x_{n} is shorthand notation for the generalized volume integration over Rn\mathbb{R}^{n}. This is the Hilbert space of the domain, and we similarly define the Hilbert space over the boundary. Let L2n(S)L_{2}^{n}(S) denote the Hilbert space of nn-vector functions square integrable over SS with inner product

⟨X1,X2⟩S=∫SX1(ξ)X2(ξ) dSξ,(3)\langle X_{1},X_{2}\rangle_{S} = \int_{S} X_{1}(\xi)X_{2}(\xi)\,\mathrm{d}S_{\xi}, \tag*{(3)}

where dSξ\mathrm{d}S_{\xi} is an infinitesimal surface element of the boundary at a point ξ∈S\xi\in S. Let L(U,V)\mathcal{L}(U,V) denote the space of linear bounded operators from UU into VV. If we regard X(t,x)X(t,x) as an element of L2n(D)L_{2}^{n}(D), then we can rewrite eqs. (1) to (3) as

ddtX(t)=F(t,X(t),Ud(t)),X∈L2n(D), t∈T0=N(t,X(t),Ub(t)),X∈L2n(S), t∈TX(t0)=X0,(4)\begin{aligned} \frac{\mathrm{d}}{\mathrm{d}t}X(t) &= F(t,X(t),U_{d}(t)), \qquad X \in L_{2}^{n}(D),\ t \in T \\ 0 &= N(t,X(t),U_{b}(t)), \qquad X \in L_{2}^{n}(S),\ t \in T \\ X(t_{0}) &= X_{0}, \tag*{(4)} \end{aligned}

where X(t),X0∈L2n(D)X(t),X_{0} \in L_{2}^{n}(D) are respectively the Hilbert space state vector and initial conditions, Ud(t)∈L2k(D)U_{d}(t) \in L_{2}^{k}(D) is the Hilbert space distributed control vector, Ub∈L2l(S)U_{b} \in L_{2}^{l}(S), is the Hilbert space boundary control vector, F:T×L2n(D)×L2k(D)→L2n(D)F:T \times L_{2}^{n}(D) \times L_{2}^{k}(D) \to L_{2}^{n}(D) is a potentially nonlinear measurable function on the domain Hilbert space, and N:T×L2n(D)×L2l(S)→L2n(S)N:T \times L_{2}^{n}(D) \times L_{2}^{l}(S) \to L_{2}^{n}(S) is a potentially nonlinear measurable function on the boundary Hilbert space.

Remark 1. The field functional perspective of eqs. (1) to (3) and the time-indexed Hilbert space perspective of eqs. (4) to (8) are consistent in the sense that they share identical solutions up to the transformation between the perspectives used above.

Assumption 1. The PDE system in fields representation given by eqs. (1) to (3) is well posed in the sense of Hadamard, and admits a unique weak solution X(t,x)X(t,x), t∈Tt \in T, x∈Dˉx \in\bar{D} for each initial condition X0(x)∈RnX_0(x) \in\mathbb{R}^n.

Depending on the specific form of the PDE, this assumption can have varying degrees of severity, however in general it is a mild assumption. Please refer to [26] for more details on existence and uniqueness of various PDEs. Despite the potential severity, it is an assumption that is required henceforth.

Remark 2. If assumption 1 holds, then the PDE system in Hilbert space representation given by eqs. (4) to (8) is also well posed in the sense of Hadamard, and admits a unique weak Hilbert space solution X(t)∈L2n(Dˉ)X(t) \in L_2^n(\bar{D}), t∈Tt \in T for each Hilbert space initial condition X0∈L2n(Dˉ)X_0 \in L_2^n(\bar{D}).

This remark has an obvious proof (e.g. by contradiction) that is omitted. Throughout this work, we go back and forth between these two notational perspectives: the spatially varying fields perspective, and the time-indexed Hilbert space perspective. While the fields perspective demonstrates the spatial integration that is central to the Volterra-Taylor expansions more clearly, the time-indexed Hilbert space perspective will often yield a more compact notation that is easier to treat with familiar algebraic operations. Whenever we suppress the dependencies on the spatial variable xx, the variables are assumed to be in time-indexed Hilbert spaces.

In order to arrive at the optimal control problem, we first define the measurable cost functional in fields representation as

J(t,X(t,x),Ud(t,x),Ub(t,x)):=ϕ(tf,X(tf))+∫t0tfL(t,X(t,x),Ud(t,x),Ub(t,x)) dt,(5)J(t,X(t,x),U_d(t,x),U_b(t,x)) := \phi(t_f,X(t_f)) + \int_{t_0}^{t_f} L(t,X(t,x),U_d(t,x),U_b(t,x))\,\mathrm{d}t, \tag*{(5)}

where ϕ:T×Rn→R\phi: T \times\mathbb{R}^n \to\mathbb{R} is some measurable real-valued terminal cost functional, and L:T×Rn×Rk×Rl→RL : T \times\mathbb{R}^n \times\mathbb{R}^k \times\mathbb{R}^l \to\mathbb{R} is a measurable real-valued running cost functional. In time-indexed Hilbert spaces, the cost functional becomes

J(t,X(t),Ud(t),Ub(t))=ϕ(tf,X(tf))+∫t0tfL(t,X(t),Ud(t),Ub(t)) dt,(6)J(t,X(t),U_d(t),U_b(t)) = \phi(t_f,X(t_f)) + \int_{t_0}^{t_f} L(t,X(t),U_d(t),U_b(t))\,\mathrm{d}t, \tag*{(6)}

where J:T×L2n(D)×L2k(D)×L2l(S)→RJ : T \times L_2^n(D) \times L_2^k(D) \times L_2^l(S) \to\mathbb{R}, ϕ:T×L2n(D)→R\phi: T \times L_2^n(D) \to\mathbb{R}, and L:T×L2n(D)×L2k(D)×L2l(S)→RL : T \times L_2^n(D) \times L_2^k(D) \times L_2^l(S) \to\mathbb{R} are the equivalent measurable real-valued functionals in Hilbert spaces. The value functional is defined as

V(X(t),t):=min⁡Ud,Ub[J(X(t),Ud(t),Ub(t))].(7)V(X(t),t) := \min_{U_d,U_b} \left[J(X(t),U_d(t),U_b(t))\right]. \tag*{(7)}

Due to the Bellman Principle of Optimality, one can form the Hamilton-Jacobi-Bellman (HJB) equation as [23, 24]

References

  1. [1]G. J. Lord, C. E. Powell, and T. Shardlow, An Introduction to Computational Stochastic PDEs, ser. Cambridge Texts in Applied Mathematics. Cambridge University Press, 2014.DOI
  2. [2]S. N. Gomes, S. Kalliadasis, D. T. Papageorgiou, G. A. Pavliotis, and M. Pradas, “Controlling roughening processes in the stochastic kuramoto–sivashinsky equation,” Physica D: Nonlinear Phenomena, vol. 348, pp. 33–43, 2017.DOI
  3. [3]J. Rabault, M. Kuchta, A. Jensen, U. Rég lade, and N. Cerardi, “Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control,” Journal of Fluid Mechanics, vol. 865, pp. 281–302, 2019.DOI
  4. [4]K. Bieker, S. Peitz, S. L. Brunton, J. N. Kutz, and M. Dellnitz, “Deep model predictive control with online learning for complex physical systems,” arXiv preprint arXiv:1905.10094, 2019.DOI
  5. [5]A. T. Mohan and D. V. Gaitonde, “A deep learning based approach to reduced order modeling for turbulent flow control using lstm neural networks,” arXiv preprint arXiv:1804.09269, 2018.arxiv.org/abs/1804.09269
  6. [6]A. G. Nair, C.-A. Yeh, E. Kaiser, B. R. Noack, S. L. Brunton, and K. Taira, “Cluster-based feedback control of turbulent post-stall separated flows,” Journal of Fluid Mechanics, vol. 875, pp. 345–375, 2019.DOI
  7. [7]S. Satheeshbabu, N. K. Uppalapati, G. Chowdhary, and G. Krishnan, “Open loop position control of soft continuum arm using deep reinforcement learning,” in 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019, pp. 5133–5139.DOI
  8. [8]A. Spielberg, A. Zhao, Y. Hu, T. Du, W. Matusik, and D. Rus, “Learning-in-the-loop optimization: End-to-end control and co-design of soft robots through learned deep latent representations,” Advances in Neural Information Processing Systems, vol. 32, pp. 8284–8294, 2019.
  9. [9]A.-m. Farahmand, S. Nabi, and D. N. Nikovski, “Deep reinforcement learning for partial differential equation control,” in 2017 American Control Conference (ACC). IEEE, 2017, pp. 3120–3127.DOI
  10. [10]J. Morton, A. Jameson, M. J. Kochenderfer, and F. Witherden, “Deep dynamical modeling and control of unsteady fluid flows,” in Advances in Neural Information Processing Systems, 2018, pp. 9258–9268.arxiv.org/abs/1805.07472
  11. [11]I. Lasiecka and R. Triggiani, Control theory for partial differential equations: continuous and approximation theories. Cambridge University Press Cambridge, 2000, vol. 1.
  12. [12]F. Tröltzsch, Optimal control of partial differential equations: theory, methods, and applications. American Mathematical Soc., 2010, vol. 112.
  13. [13]M. I. Sumin, “The first variation and pontryagin’s maximum principle in optimal control for partial differential equations,” Computational Mathematics and Mathematical Physics, vol. 49, no. 6, pp. 958–978, 2009.DOI
  14. [14]J. M. Yong, “Pontryagin maximum principle for semilinear second order elliptic partial differential equations and variational inequalities with state constraints,” Differential and Integral Equations, vol. 5, no. 6, pp. 1307–1334, 1992.DOI
  15. [15]Y. Tassa, N. Mansard, and E. Todorov, “Control-limited differential dynamic programming,” in IEEE International Conference on Robotics and Automation (ICRA), May 2014, pp. 1168–1175.DOI
  16. [16]Y. Aoyama, G. Boutselis, A. Patel, and E. A. Theodorou, “Constrained differential dynamic programming revisited,” arXiv preprint arXiv:2005.00985, 2020.arxiv.org/abs/2005.00985
  17. [17]Y. Tassa, T. Erez, and W. D. Smart, “Receding horizon differential dynamic programming,” in NIPS. Citeseer, 2007, pp. 1465–1472.
  18. [18]Y. Pan and E. Theodorou, “Probabilistic differential dynamic programming,” Advances in Neural Information Processing Systems, vol. 27, pp. 1907–1915, 2014.
  19. [19]Y. Pan, G. I. Boutselis, and E. A. Theodorou, “Efficient reinforcement learning via probabilistic trajectory optimization,” IEEE transactions on neural networks and learning systems, vol. 29, no. 11, pp. 5459–5474, 2018.DOI
  20. [20]W. Sun, E. A. Theodorou, and P. Tsiotras, “Game theoretic continuous time differential dynamic programming,” in 2015 American Control Conference (ACC). IEEE, 2015, pp. 5593–5598.DOI
  21. [21]G. I. Boutselis, Y. Pan, and E. A. Theodorou, “Numerical trajectory optimization for stochastic mechanical systems,” SIAM Journal on scientific computing, vol. 41, no. 4, pp. A2065–A2087, 2019.DOI
  22. [22]G. I. Boutselis, G. De La Torre, and E. A. Theodorou, “Stochastic optimal control using polynomial chaos variational integrators,” in 2016 American Control Conference (ACC). IEEE, 2016, pp. 6586–6591.DOI
  23. [23]S. Tzafestas and J. Nightingale, “Differential dynamic-programming approach to optimal nonlinear distributed-parameter control systems,” in Proceedings of the Institution of Electrical Engineers, vol. 116, no. 6. IET, 1969, pp. 1079–1084.DOI
  24. [24]Y. Sakawa, “A matrix green’s formula and optimal control of linear distributed-parameter systems,” Journal of Optimization Theory and Applications, vol. 10, no. 5, pp. 290–299, 1972.DOI
  25. [25]D. H. Jacobson and D. Q. Mayne, Differential dynamic programming. New York: American Elsevier Pub. Co., 1970.
  26. [26]L. C. Evans, “Partial differential equations and monge-kantorovich mass transfer,” Current developments in mathematics, vol. 1997, no. 1, pp. 65–126, 1997.DOI
  27. [27]V. Volterra, Theory of functionals and of integral and integro-differential equations. Dover, 1959.
  28. [28]C. Shoemaker and L. Liao, “Proof of the quadratic convergence of differential dynamic programming,” Cornell University Operations Research and Industrial Engineering, Tech. Rep., 1990.
  29. [29]S. Yakowitz and B. Rutherford, “Computational aspects of discrete-time optimal control,” Applied Mathematics and Computation, vol. 15, no. 1, pp. 29–45, 1984.DOI
  30. [30]W. Sun, E. A. Theodorou, and P. Tsiotras, “Continuous-time differential dynamic programming with terminal constraints,” in 2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL). IEEE, 2014, pp. 1–6.
  31. [31]S. Tzafestas and J. Nightingale, “Optimal control of a class of linear stochastic distributed-parameter systems,” in Proceedings of the Institution of Electrical Engineers, vol. 115, no. 8. IET, 1968, pp. 1213–1220.DOI
  32. [32]A. Friedman, Partial differential equations of parabolic type. Courier Dover Publications, 2008.
  33. [33]J. D. O. Pantoja, “Algorithms for constrained optimization problems,” Differential Dynamic Programming and Newton’s Method. International Journal of Control, vol. 47, pp. 1539–1553, 1983.
  34. [34]D. Murray and S. Yakowitz, “Differential dynamic programming and newton’s method for discrete optimal control problems,” Journal of Optimization Theory and Applications, vol. 43, no. 3, pp. 395–414, 1984.DOI
  35. [35]G. I. Boutselis and E. Theodorou, “Differential dynamic programming on lie groups: Derivation, convergence analysis and numerical results,” arXiv preprint arXiv:1809.07883, 2018.arxiv.org/abs/1809.07883
  36. [36]Y. Tassa, T. Erez, and E. Todorov, “Synthesis and stabilization of complex behaviors through online trajectory optimization,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Oct 2012, pp. 4906–4913.
  37. [37]D. Butusov, T. Karimov, and V. Ostrovskii, “Semi-implicit ode solver for matrix riccati equation,” in 2016 IEEE NW Russia Young Researchers in Electrical and Electronic Engineering Conference (EICOnRusNW). IEEE, 2016, pp. 168–172.DOI
  38. [38]J. W. Demmel, S. C. Eisentat, J. R. Gilbert, X. S. Li, and J. W. Liu, “A supernodal approach to sparse partial pivoting,” SIAM Journal on Matrix Analysis and Applications, vol. 20, no. 3, pp. 720–755, 1999.DOI
  39. [39]K. Elamvazhuthi, H. Kuiper, and S. Berman, “Pde-based optimization for stochastic mapping and coverage strategies using robotic ensembles,” Automatica, vol. 95, pp. 356–367, 2018.arxiv.org/abs/1711.11018
  40. [40]I. V. Oseledets, “Tensor-train decomposition,” SIAM Journal on Scientific Computing, vol. 33, no. 5, pp. 2295–2317, 2011.DOI

Paper details

vlnc:W_t3e45us9
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents