Introduction
Many complex natural processes are governed by systems of equations with spatio-temporal dependence, and are typically described by Partial Differential Equation (PDE). These systems are ubiquitous in nature and can be found in most disciplines of engineering and applied physics. The range of natural processes includes fluid flow governed by the Navier-Stokes equation, sub-atomic particle systems governed by the Schrodinger equation, activation of neurons governed by the Nagumo equation [1 ] , and flame front propagation in combustion systems governed by the Kuramoto-Sivashinsky equation [2 ] .
Despite their ubiquity in nature and engineering, theory and numerical methods for control of spatio-temporal systems remains challenging due to the time-delay, dramatic under-actuation, high system dimensionality, and multi-modal bifurcations, which are often inherent in their dynamics. Furthermore, existence and uniqueness of solutions remains an open problem for many systems, and when they do exist, they typically only have a weak notion of differentiability. Analysis of their performance must be treated with calculus over functionals, and their state vectors are often described by vectors in an infinite-dimensional time-indexed Hilbert space even for scalar 1-dimensional (1D) PDEs. Put together, mathematically consistent and numerically realizable algorithms for control of spatio-temporal systems represents many of the largest current-day challenges facing the automatic control community.
The majority of recent methods for control of spatio-temporal systems typically reduce PDEs into a finite set of Ordinary Differential Equations (ODEs) through Reduced Order Models (ROMs), and apply standard finite-dimensional optimization methods which result in algorithms specific to the ROM used. Within this paradigm, deep learning methods have successfully been applied on policy networks in the finite dimensional setting for controlling Navier-Stokes systems [3 , 4 , 5 , 6 ] , for soft robotic systems [7 , 8 ] , as well as for many other systems [9 ] . These methods are often specific to a discretization scheme and represent a discretize-then-optimize approach. Some such methods can introduce new phenomena in the latent space representation, as in [10 ] , where the resulting deep Koopman approach can be shown to violate linear stabilizability conditions of the latent space dynamics.
External to the machine-learning literature are infinite-dimensional methods found in the control theory literature [11 , 12 ] , which are dominated by linear or linearization-based approaches, which include Linear Quadratic Regulator (LQR) approaches for linear PDEs, and forward-backward approaches, which include approaches due to the Pontryagin Maximum Principle (PMP) [12 , 13 , 14 ] . Indeed local linearization methods allow for optimal solutions of an approximate problem, however require knowledge of linearization points a-priori. On the other hand, forward-backward schemes provide a nominal trajectory and optimization-based control update scheme at the expense of the backpropagation of a coupled system equation.
In contrast to Pontryagin methods which yield a state-independent backward equation and an open-loop controller, methods founded on the Bellman principle of optimality utilize backward equations that are state-dependent and yield closed-loop control solutions. Methods such as Differential Dynamic Programming (DDP) have decades of established history in the finite dimensional automatic control literature. Modern variations include control limits [15 ] , state constraints [16 ] , receding horizons [17 ] , belief space control [18 , 19 ] , game-theoretic control [20 ] , control on Lie groups [21 ] , and using polynomial chaos variational integrators [22 ] .
A previous attempt exists to extend the DDP framework to spatio-temporal systems in infinite dimensions [23 ] , however this approach has several flaws and mathematical inconsistencies, as pointed out in [24 ] . Additionally, the DDP method has had significant growth since the early works [25 ] . Decades of advancement include linearization around the nominal trajectory as opposed to the optimal trajectory which decreases sensitivities of convergence behavior to the initial conditions, regularization in the second order backward equation to increase numerical stability, treatment of state and control constraints, and optimization over time horizon.
In light of the apparent literature gap, this manuscript is devoted to the development of DDP methods for spatio-temporal systems in infinite dimensions. Specifically, we derive the Spatio-Temporal DDP (STDDP) framework incorporating modern theoretical techniques, we demonstrate that the resulting system of forward-backward equations generalizes both the LQR solution in infinite dimensions and DDP in finite dimensions, we provide a proof of convergence for the resulting system of continuous-time forward-backward equations, we explore and develop numerical approaches to handle sensitivities that arise due to discretization, and apply the resulting algorithm to linear and nonlinear spatio-temporal PDE systems. In contrast to recent machine learning methods, our optimization is developed entirely in Hilbert spaces, and represents an optimize-then-discretize approach. As a result, the framework is a continuous-time formulation which is agnostic to discretization scheme during implementation.
Preliminaries and Problem Statement
Let D ⊆ R n D \subseteq\mathbb{R}^{n} D ⊆ R n denote a measurable connected open domain of R n \mathbb{R}^{n} R n describing the space on which the system evolves. Let S ⊆ R n S \subseteq\mathbb{R}^{n} S ⊆ R n denote the boundary of D D D , let D ‾ \overline{D} D denote the closure of the domain, i.e. D ‾ = D ∪ S \overline{D} = D \cup S D = D ∪ S , and let T = [ t 0 , t f ] T = [t_{0},t_{f}] T = [ t 0 , t f ] denote some arbitrary time domain. In fields representation, a general form of a deterministic PDE dynamical system is given by
∂ t X ( t , x ) = F ( t , x , X ( t , x ) , U d ( t , x ) ) , x ∈ D 0 = N ( t , x , X ( t , x ) , U b ( t , x ) ) , x ∈ S X ( t 0 , x ) = X 0 ( x ) , x ∈ D ‾ , (1) \begin{aligned}
\partial_{t}X(t,x) &= F(t,x,X(t,x),U_{d}(t,x)), \qquad x \in D \\
0 &= N(t,x,X(t,x),U_{b}(t,x)), \qquad x \in S \\
X(t_{0},x) &= X_{0}(x), \qquad x \in\overline{D}, \tag*{(1)}
\end{aligned} ∂ t X ( t , x ) 0 X ( t 0 , x ) = F ( t , x , X ( t , x ) , U d ( t , x )) , x ∈ D = N ( t , x , X ( t , x ) , U b ( t , x )) , x ∈ S = X 0 ( x ) , x ∈ D , (1)
where X : T × D → R n X:T \times D \to\mathbb{R}^{n} X : T × D → R n is the state. This problem has two measurable control functions, U b : T × S → R l U_{b}:T \times S \to\mathbb{R}^{l} U b : T × S → R l which correspond to actuation on the boundary, and U d : T × D → R k U_{d}:T \times D \to\mathbb{R}^{k} U d : T × D → R k which corresponds to actuation distributed throughout the field excluding the boundary. The dynamics evolve by some measurable functional F : T × D × R n × R k → R n F:T \times D \times\mathbb{R}^{n} \times\mathbb{R}^{k} \to\mathbb{R}^{n} F : T × D × R n × R k → R n that is potentially nonlinear in the state function X ( t , x ) X(t,x) X ( t , x ) or the control function U d ( t , x ) U_{d}(t,x) U d ( t , x ) , with a boundary condition functional N : T × S × R n × R l → R n N:T \times S \times\mathbb{R}^{n} \times\mathbb{R}^{l} \to\mathbb{R}^{n} N : T × S × R n × R l → R n that is also potentially a nonlinear functional of the state or control functions, and can be any type of boundary condition (e.g. Neumann, Dirichlet, etc.).
We can equivalently write eqs. (1) to (3) in the time-indexed Hilbert spaces perspective by first properly defining Hilbert spaces, as in [24 ] . Let L 2 n ( D ) L_{2}^{n}(D) L 2 n ( D ) denote the Hilbert space of n n n -vector functions square integrable over D D D with inner product
⟨ X 1 , X 2 ⟩ D = ∫ D X 1 T ( x ) X 2 ( x ) d x , (2) \langle X_{1},X_{2}\rangle_{D} = \int_{D} X_{1}^{\mathsf{T}}(x)X_{2}(x)\,\mathrm{d}x, \tag*{(2)} ⟨ X 1 , X 2 ⟩ D = ∫ D X 1 T ( x ) X 2 ( x ) d x , (2)
where d x = d x 1 d x 2 ⋯ d x n \mathrm{d}x = \mathrm{d}x_{1}\mathrm{d}x_{2}\cdots\mathrm{d}x_{n} d x = d x 1 d x 2 ⋯ d x n is shorthand notation for the generalized volume integration over R n \mathbb{R}^{n} R n . This is the Hilbert space of the domain, and we similarly define the Hilbert space over the boundary. Let L 2 n ( S ) L_{2}^{n}(S) L 2 n ( S ) denote the Hilbert space of n n n -vector functions square integrable over S S S with inner product
⟨ X 1 , X 2 ⟩ S = ∫ S X 1 ( ξ ) X 2 ( ξ ) d S ξ , (3) \langle X_{1},X_{2}\rangle_{S} = \int_{S} X_{1}(\xi)X_{2}(\xi)\,\mathrm{d}S_{\xi}, \tag*{(3)} ⟨ X 1 , X 2 ⟩ S = ∫ S X 1 ( ξ ) X 2 ( ξ ) d S ξ , (3)
where d S ξ \mathrm{d}S_{\xi} d S ξ is an infinitesimal surface element of the boundary at a point ξ ∈ S \xi\in S ξ ∈ S . Let L ( U , V ) \mathcal{L}(U,V) L ( U , V ) denote the space of linear bounded operators from U U U into V V V . If we regard X ( t , x ) X(t,x) X ( t , x ) as an element of L 2 n ( D ) L_{2}^{n}(D) L 2 n ( D ) , then we can rewrite eqs. (1) to (3) as
d d t X ( t ) = F ( t , X ( t ) , U d ( t ) ) , X ∈ L 2 n ( D ) , t ∈ T 0 = N ( t , X ( t ) , U b ( t ) ) , X ∈ L 2 n ( S ) , t ∈ T X ( t 0 ) = X 0 , (4) \begin{aligned}
\frac{\mathrm{d}}{\mathrm{d}t}X(t) &= F(t,X(t),U_{d}(t)), \qquad X \in L_{2}^{n}(D),\ t \in T \\
0 &= N(t,X(t),U_{b}(t)), \qquad X \in L_{2}^{n}(S),\ t \in T \\
X(t_{0}) &= X_{0}, \tag*{(4)}
\end{aligned} d t d X ( t ) 0 X ( t 0 ) = F ( t , X ( t ) , U d ( t )) , X ∈ L 2 n ( D ) , t ∈ T = N ( t , X ( t ) , U b ( t )) , X ∈ L 2 n ( S ) , t ∈ T = X 0 , (4)
where X ( t ) , X 0 ∈ L 2 n ( D ) X(t),X_{0} \in L_{2}^{n}(D) X ( t ) , X 0 ∈ L 2 n ( D ) are respectively the Hilbert space state vector and initial conditions, U d ( t ) ∈ L 2 k ( D ) U_{d}(t) \in L_{2}^{k}(D) U d ( t ) ∈ L 2 k ( D ) is the Hilbert space distributed control vector, U b ∈ L 2 l ( S ) U_{b} \in L_{2}^{l}(S) U b ∈ L 2 l ( S ) , is the Hilbert space boundary control vector, F : T × L 2 n ( D ) × L 2 k ( D ) → L 2 n ( D ) F:T \times L_{2}^{n}(D) \times L_{2}^{k}(D) \to L_{2}^{n}(D) F : T × L 2 n ( D ) × L 2 k ( D ) → L 2 n ( D ) is a potentially nonlinear measurable function on the domain Hilbert space, and N : T × L 2 n ( D ) × L 2 l ( S ) → L 2 n ( S ) N:T \times L_{2}^{n}(D) \times L_{2}^{l}(S) \to L_{2}^{n}(S) N : T × L 2 n ( D ) × L 2 l ( S ) → L 2 n ( S ) is a potentially nonlinear measurable function on the boundary Hilbert space.
Assumption 1. The PDE system in fields representation given by eqs. (1) to (3) is well posed in the sense of Hadamard, and admits a unique weak solution X ( t , x ) X(t,x) X ( t , x ) , t ∈ T t \in T t ∈ T , x ∈ D ˉ x \in\bar{D} x ∈ D ˉ for each initial condition X 0 ( x ) ∈ R n X_0(x) \in\mathbb{R}^n X 0 ( x ) ∈ R n .
Depending on the specific form of the PDE, this assumption can have varying degrees of severity, however in general it is a mild assumption. Please refer to [26 ] for more details on existence and uniqueness of various PDEs. Despite the potential severity, it is an assumption that is required henceforth.
In order to arrive at the optimal control problem, we first define the measurable cost functional in fields representation as
J ( t , X ( t , x ) , U d ( t , x ) , U b ( t , x ) ) : = ϕ ( t f , X ( t f ) ) + ∫ t 0 t f L ( t , X ( t , x ) , U d ( t , x ) , U b ( t , x ) ) d t , (5) J(t,X(t,x),U_d(t,x),U_b(t,x)) := \phi(t_f,X(t_f)) + \int_{t_0}^{t_f} L(t,X(t,x),U_d(t,x),U_b(t,x))\,\mathrm{d}t, \tag*{(5)} J ( t , X ( t , x ) , U d ( t , x ) , U b ( t , x )) := ϕ ( t f , X ( t f )) + ∫ t 0 t f L ( t , X ( t , x ) , U d ( t , x ) , U b ( t , x )) d t , (5)
where ϕ : T × R n → R \phi: T \times\mathbb{R}^n \to\mathbb{R} ϕ : T × R n → R is some measurable real-valued terminal cost functional, and L : T × R n × R k × R l → R L : T \times\mathbb{R}^n \times\mathbb{R}^k \times\mathbb{R}^l \to\mathbb{R} L : T × R n × R k × R l → R is a measurable real-valued running cost functional. In time-indexed Hilbert spaces, the cost functional becomes
J ( t , X ( t ) , U d ( t ) , U b ( t ) ) = ϕ ( t f , X ( t f ) ) + ∫ t 0 t f L ( t , X ( t ) , U d ( t ) , U b ( t ) ) d t , (6) J(t,X(t),U_d(t),U_b(t)) = \phi(t_f,X(t_f)) + \int_{t_0}^{t_f} L(t,X(t),U_d(t),U_b(t))\,\mathrm{d}t, \tag*{(6)} J ( t , X ( t ) , U d ( t ) , U b ( t )) = ϕ ( t f , X ( t f )) + ∫ t 0 t f L ( t , X ( t ) , U d ( t ) , U b ( t )) d t , (6)
where J : T × L 2 n ( D ) × L 2 k ( D ) × L 2 l ( S ) → R J : T \times L_2^n(D) \times L_2^k(D) \times L_2^l(S) \to\mathbb{R} J : T × L 2 n ( D ) × L 2 k ( D ) × L 2 l ( S ) → R , ϕ : T × L 2 n ( D ) → R \phi: T \times L_2^n(D) \to\mathbb{R} ϕ : T × L 2 n ( D ) → R , and L : T × L 2 n ( D ) × L 2 k ( D ) × L 2 l ( S ) → R L : T \times L_2^n(D) \times L_2^k(D) \times L_2^l(S) \to\mathbb{R} L : T × L 2 n ( D ) × L 2 k ( D ) × L 2 l ( S ) → R are the equivalent measurable real-valued functionals in Hilbert spaces. The value functional is defined as
V ( X ( t ) , t ) : = min U d , U b [ J ( X ( t ) , U d ( t ) , U b ( t ) ) ] . (7) V(X(t),t) := \min_{U_d,U_b} \left[J(X(t),U_d(t),U_b(t))\right]. \tag*{(7)} V ( X ( t ) , t ) := U d , U b min [ J ( X ( t ) , U d ( t ) , U b ( t )) ] . (7)
Due to the Bellman Principle of Optimality, one can form the Hamilton-Jacobi-Bellman (HJB) equation as [23 , 24 ]
− ∂ t V ( X ( t ) , t ) = min U d , U b [ L ( t , X , U d , U b ) + ⟨ V X ( t , X ) , F ( t , X , U d ) ⟩ ] , (8) -\partial_t V(X(t),t) = \min_{U_d,U_b} \left[L(t,X,U_d,U_b) + \left\langle V_X(t,X),F(t,X,U_d)\right\rangle\right], \tag*{(8)} − ∂ t V ( X ( t ) , t ) = U d , U b min [ L ( t , X , U d , U b ) + ⟨ V X ( t , X ) , F ( t , X , U d ) ⟩ ] , (8)
V ( X ( t f ) , t f ) = ϕ ( t f , X ( t f ) ) = : V f ∈ R , (9) V(X(t_f),t_f) = \phi(t_f,X(t_f)) =: V_f \in\mathbb{R}, \tag*{(9)} V ( X ( t f ) , t f ) = ϕ ( t f , X ( t f )) =: V f ∈ R , (9)
where we write ∂ t = ∂ ∂ t \partial_t = \frac{\partial}{\partial t} ∂ t = ∂ t ∂ to denote the normal partial derivative of a function with respect to a variable, and use subscript X X X , U b U_b U b , or U d U_d U d to denote the Gateaux partial derivative of a functional or operator with respect to an operator function. One can carry out the same derivation using Volterra’s notion of functional derivative [27 ] . Note that V ( X ( t ) , t ) V(X(t),t) V ( X ( t ) , t ) is a function of time, and a functional of X ( t ) X(t) X ( t ) . Also, it should be noted that the HJB equation in eq. (8) is a backwards nonlinear PDE.
Assumption 2. The backwards PDE in eq. (8) admits a unique viscosity solution V ( t , X ( t ) ) V(t,X(t)) V ( t , X ( t )) , t ∈ T t \in T t ∈ T , X ( t ) ∈ L 2 n ( D ‾ ) X(t) \in L_2^n(\overline{D}) X ( t ) ∈ L 2 n ( D ) for each terminal condition V ( X ( t f ) , t f ) = V f : = ϕ ( t f , X ( t f ) ) ∈ R V(X(t_f),t_f)=V_f:=\phi(t_f,X(t_f)) \in\mathbb{R} V ( X ( t f ) , t f ) = V f := ϕ ( t f , X ( t f )) ∈ R .
The DDP framework solves the HJB equation in eq. (8) iteratively via expansions of the value functional, cost functional, dynamics operator function, and boundary operator function to given order. Typically, the value functional and cost functional are expanded to second order so that the resulting HJB becomes a quadratic optimization problem with a unique optimal control minimizer.
Quadratic expansions also allow for proofs of global convergence and even proofs of quadratic convergence, that in finite dimensions, initially relied on well known convergence properties of the Newton method of optimization [28 , 29 ] for quadratic problems. Under similar reasoning, the dynamics are typically either expanded to first or second order.
Expansions of the Cost, Value, Field, and Boundary
The approach in this paper is a spatio-temporal DDP approach that is analogous to the finite dimensional DDP approach of [30 ] . Therein, the authors discuss the fundamental differences between their derivation, and the original derivation by Jacobson and Mayne [25 ] . The derivation by Jacobson and Mayne, of which a similar flavor is followed in [23 ] , is based on the restrictive assumption that the nominal control trajectory u ˉ \bar{u} u ˉ is sufficiently close to the optimal control solution u ∗ u^* u ∗ . This is circumvented by performing expansions around a nominal trajectory. Define a nominal state and control triple ( X ˉ , U ˉ d , U ˉ b ) (\bar{X},\bar{U}_d,\bar{U}_b) ( X ˉ , U ˉ d , U ˉ b ) and the variations δ X : = X − X ˉ \delta X := X-\bar{X} δ X := X − X ˉ , δ U d : = U d − U ˉ d \delta U_d := U_d-\bar{U}_d δ U d := U d − U ˉ d , and δ U b : = U b − U ˉ b \delta U_b := U_b-\bar{U}_b δ U b := U b − U ˉ b . In order to properly write the expansions, we require the following assumption
Assumption 3. The dynamics function F F F and boundary function N N N are differentiable almost everywhere, the running cost functional L L L and terminal cost functional ϕ \phi ϕ are twice differentiable almost everywhere, and the value functional V V V is three times differentiable almost everywhere. These stated derivatives are defined in the Gateaux sense with respect to the state and control triple ( X , U d , U b ) (X,U_d,U_b) ( X , U d , U b ) , and are square integrable in the Lebesgue sense. That is, the stated Gateaux derivative of each functional exists ∀ ( X , U d , U b ) ∈ ( L 2 n ( D ) , L 2 n ( D ) , L 2 n ( S ) ) \forall(X,U_d,U_b) \in(L_2^n(D),L_2^n(D),L_2^n(S)) ∀ ( X , U d , U b ) ∈ ( L 2 n ( D ) , L 2 n ( D ) , L 2 n ( S )) except on a properly defined set of measure zero.
As previously stated, the value functional is a function of time t t t , but a functional of the spacetime function X ( t , x ) X(t,x) X ( t , x ) . Thus the value functional is expanded via a Volterra-Taylor functional expansion [27 ]
V ( t , X ˉ ( t , x ) + δ X ( t , x ) ) = V ( t , X ˉ ( t , x ) ) + ∫ D V X T ( t , X ˉ ( t , x ) ) δ X ( t , x ) d x + 1 2 ∫ D ∫ D δ X T ( t , x ) V X X ( t , x , y ) δ X ( t , y ) d x d y + O ( δ 3 ) . (10) \begin{aligned}
V(t,\bar{X}(t,x)+\delta X(t,x))={}&V(t,\bar{X}(t,x))+\int_D V_X^{\mathsf{T}}(t,\bar{X}(t,x))\delta X(t,x)\,dx\\
&+\frac{1}{2}\int_D\int_D\delta X^{\mathsf{T}}(t,x)V_{XX}(t,x,y)\delta X(t,y)\,dx\,dy+O(\delta^3). \tag*{(10)}
\end{aligned} V ( t , X ˉ ( t , x ) + δ X ( t , x )) = V ( t , X ˉ ( t , x )) + ∫ D V X T ( t , X ˉ ( t , x )) δ X ( t , x ) d x + 2 1 ∫ D ∫ D δ X T ( t , x ) V X X ( t , x , y ) δ X ( t , y ) d x d y + O ( δ 3 ) . (10)
We maintain connection to the Hilbert space perspective by defining Hilbert space operators for each kernel function. Define the operator V X X ( t , X ) ∈ L ( L 2 n ( D ) , L 2 n ( D ) ) V_{XX}(t,X) \in\mathcal{L}(L_2^n(D),L_2^n(D)) V X X ( t , X ) ∈ L ( L 2 n ( D ) , L 2 n ( D )) as
V X X ( t , X ) W ( t ) : = ∫ D V X X ( t , x , y ) W ( t , y ) d y , (11) V_{XX}(t,X)W(t):=\int_D V_{XX}(t,x,y)W(t,y)\,dy, \tag*{(11)} V X X ( t , X ) W ( t ) := ∫ D V X X ( t , x , y ) W ( t , y ) d y , (11)
where V X X ( t , x , y ) V_{XX}(t,x,y) V X X ( t , x , y ) is the kernel function. In order to form the left-hand side of the HJB eq. (8) , we apply a re-arranged definition of the total differential [27 ] , given by
∂ t ( ⋅ ) = d d t ( ⋅ ) − ⟨ ( ⋅ ) X , F ( t , X , U d ) ⟩ , (12) \partial_t(\cdot)=\frac{d}{dt}(\cdot)-\langle(\cdot)_X,F(t,X,U_d)\rangle, \tag*{(12)} ∂ t ( ⋅ ) = d t d ( ⋅ ) − ⟨( ⋅ ) X , F ( t , X , U d )⟩ , (12)
which holds for any functional that explicitly depends on X X X and t t t . In order to simplify notation, we suppress arguments when functionals are evaluated on the nominal trajectory triple. We apply eq. (12) to each term on the right-hand side of eq. (10) , to yield the left-hand side of the HJB, which in Hilbert spaces, has the form
− ∂ t V ( t , X ˉ + δ X ) = − d d t ( V + ⟨ V X , δ X ⟩ + 1 2 ⟨ δ X , V X X δ X ⟩ ) + ⟨ V X , F ⟩ + ⟨ V X X F , δ X ⟩ (13) -\partial_t V(t,\bar{X}+\delta X)=-\frac{\mathrm{d}}{\mathrm{d}t}\left(V+\langle V_X,\delta X\rangle+\frac{1}{2}\langle\delta X,V_{XX}\delta X\rangle\right)+\langle V_X,F\rangle+\langle V_{XX}F,\delta X\rangle \tag*{(13)} − ∂ t V ( t , X ˉ + δ X ) = − d t d ( V + ⟨ V X , δ X ⟩ + 2 1 ⟨ δ X , V X X δ X ⟩ ) + ⟨ V X , F ⟩ + ⟨ V X X F , δ X ⟩ (13)
+ 1 2 ⟨ F V X X X δ X ′ , δ X ⟩ (14) +\frac{1}{2}\langle FV_{XXX}\delta X',\delta X\rangle \tag*{(14)} + 2 1 ⟨ F V X X X δ X ′ , δ X ⟩ (14)
where for the third order Gateaux derivative V X X X V_{XXX} V X X X , we have defined the tensor operator in time-indexed Hilbert spaces V X X X ( t , X ) ∈ L ( L 2 n ( D ) × L 2 n ( D ) , L 2 n ( D ) ) V_{XXX}(t,X)\in\mathcal{L}(L_2^n(D)\times L_2^n(D),L_2^n(D)) V X X X ( t , X ) ∈ L ( L 2 n ( D ) × L 2 n ( D ) , L 2 n ( D )) as
U ( t ) V X X X ( t , X ( t ) ) W ( t ) : = ∫ D ∫ D U T ( t , x ) V X X X ( t , x , y , z ) W ( t , y ) d x d y . (15) U(t)V_{XXX}(t,X(t))W(t):=\int_D\int_D U^{\mathsf{T}}(t,x)V_{XXX}(t,x,y,z)W(t,y)\,\mathrm{d}x\mathrm{d}y. \tag*{(15)} U ( t ) V X X X ( t , X ( t )) W ( t ) := ∫ D ∫ D U T ( t , x ) V X X X ( t , x , y , z ) W ( t , y ) d x d y . (15)
The 4-D kernel function V X X X ( t , x , y , z ) V_{XXX}(t,x,y,z) V X X X ( t , x , y , z ) is assumed to be symmetric about all three spatial axes for simplicity.
Next, we expand the cost functional with a Volterra-Taylor expansion to second order, which in time-indexed Hilbert spaces has the form
L ( t , X ˉ + δ X , U ˉ d + δ U d , U ˉ b + δ U b ) = L + ⟨ L X , δ X ⟩ + ⟨ L U d , δ U d ⟩ + ⟨ L U b , δ U b ⟩ S + 1 2 ⟨ δ X , L X X δ X ′ ⟩ + 1 2 ⟨ δ U d , ( L U d X + L X U d T ) δ X ′ ⟩ + 1 2 ⟨ δ U b , ( L U b X + L X U b T ) δ X ′ ⟩ S + 1 2 ⟨ δ U d , L U d U d δ U d ′ ⟩ + 1 2 ⟨ δ U b , L U b U b δ U b ′ ⟩ S + O ( δ 3 ) , (16) \begin{aligned}
L(t,\bar{X}+\delta X,\bar{U}_d+\delta U_d,\bar{U}_b+\delta U_b)
={}&L+\langle L_X,\delta X\rangle+\langle L_{U_d},\delta U_d\rangle+\langle L_{U_b},\delta U_b\rangle_S+\frac{1}{2}\langle\delta X,L_{XX}\delta X'\rangle\\
&+\frac{1}{2}\langle\delta U_d,(L_{U_dX}+L_{XU_d}^{\mathsf{T}})\delta X'\rangle+\frac{1}{2}\langle\delta U_b,(L_{U_bX}+L_{XU_b}^{\mathsf{T}})\delta X'\rangle_S\\
&+\frac{1}{2}\langle\delta U_d,L_{U_dU_d}\delta U_d'\rangle+\frac{1}{2}\langle\delta U_b,L_{U_bU_b}\delta U_b'\rangle_S+O(\delta^3), \tag*{(16)}
\end{aligned} L ( t , X ˉ + δ X , U ˉ d + δ U d , U ˉ b + δ U b ) = L + ⟨ L X , δ X ⟩ + ⟨ L U d , δ U d ⟩ + ⟨ L U b , δ U b ⟩ S + 2 1 ⟨ δ X , L X X δ X ′ ⟩ + 2 1 ⟨ δ U d , ( L U d X + L X U d T ) δ X ′ ⟩ + 2 1 ⟨ δ U b , ( L U b X + L X U b T ) δ X ′ ⟩ S + 2 1 ⟨ δ U d , L U d U d δ U d ′ ⟩ + 2 1 ⟨ δ U b , L U b U b δ U b ′ ⟩ S + O ( δ 3 ) , (16)
where we have defined the operators
L X X ( t , X ( t ) , U b ( t ) , U d ( t ) ) W ( t ) : = ∫ D L X X ( t , x , y ) W ( t , y ) d y L_{XX}(t,X(t),U_b(t),U_d(t))W(t):=\int_D L_{XX}(t,x,y)W(t,y)\,\mathrm{d}y L X X ( t , X ( t ) , U b ( t ) , U d ( t )) W ( t ) := ∫ D L X X ( t , x , y ) W ( t , y ) d y
L X U d ( t , X ( t ) , U b ( t ) , U d ( t ) ) W ( t ) : = ∫ D L X U d ( t , x , y ) W ( t , y ) d y L_{XU_d}(t,X(t),U_b(t),U_d(t))W(t):=\int_D L_{XU_d}(t,x,y)W(t,y)\,\mathrm{d}y L X U d ( t , X ( t ) , U b ( t ) , U d ( t )) W ( t ) := ∫ D L X U d ( t , x , y ) W ( t , y ) d y
L X U b ( t , X ( t ) , U b ( t ) , U d ( t ) ) W ( t ) : = ∫ S L X U b ( t , ξ , η ) W ( t , η ) d S η L_{XU_b}(t,X(t),U_b(t),U_d(t))W(t):=\int_S L_{XU_b}(t,\xi,\eta)W(t,\eta)\,\mathrm{d}S_\eta L X U b ( t , X ( t ) , U b ( t ) , U d ( t )) W ( t ) := ∫ S L X U b ( t , ξ , η ) W ( t , η ) d S η
L U d U d ( t , X ( t ) , U b ( t ) , U d ( t ) ) W ( t ) : = ∫ D L U d U d ( t , x , y ) W ( t , y ) d y L_{U_dU_d}(t,X(t),U_b(t),U_d(t))W(t):=\int_D L_{U_dU_d}(t,x,y)W(t,y)\,\mathrm{d}y L U d U d ( t , X ( t ) , U b ( t ) , U d ( t )) W ( t ) := ∫ D L U d U d ( t , x , y ) W ( t , y ) d y
L U b U b ( t , X ( t ) , U b ( t ) , U d ( t ) ) W ( t ) : = ∫ S L U b U b ( t , ξ , η ) W ( t , η ) d S η , L_{U_bU_b}(t,X(t),U_b(t),U_d(t))W(t):=\int_S L_{U_bU_b}(t,\xi,\eta)W(t,\eta)\,\mathrm{d}S_\eta, L U b U b ( t , X ( t ) , U b ( t ) , U d ( t )) W ( t ) := ∫ S L U b U b ( t , ξ , η ) W ( t , η ) d S η ,
and similarly defined operators for L U d X L_{U_dX} L U d X , L U b X L_{U_bX} L U b X .
Assumption 4. The measurable kernel functions L X X L_{XX} L X X , L X U d L_{XU_d} L X U d , L X U b L_{XU_b} L X U b are spatially symmetric and positive semi-definite. The measurable kernel functions V X X V_{XX} V X X , L U d U d L_{U_dU_d} L U d U d , L U b U b L_{U_bU_b} L U b U b are spatially symmetric and positive definite. The omitted cross term operators L U b U d L_{U_bU_d} L U b U d and L U d U b L_{U_dU_b} L U d U b are null operators.
Note the assumption that cross terms between boundary and distributed control (i.e. L U b U d L_{U_bU_d} L U b U d and L U d U b L_{U_dU_b} L U d U b ) are zero. This is a fairly benign assumption since cost functionals are often composed of pure quadratics in either U d U_d U d or U b U_b U b , but not both. Including these cross terms also yields optimal update equations for boundary and distributed control that are coupled to each other, and thus impose mathematical and implementation difficulties.
Next, the dynamics and boundary are expanded around the nominal trajectory. The dynamics functional F ( t , X ( t ) , U d ( t ) ) F(t,X(t),U_d(t)) F ( t , X ( t ) , U d ( t )) and boundary functional N ( t , X ( t ) , U b ( t ) ) N(t,X(t),U_b(t)) N ( t , X ( t ) , U b ( t )) map into L 2 n ( D ) L_2^n(D) L 2 n ( D ) and L 2 n ( S ) L_2^n(S) L 2 n ( S ) , respectively, and are not real-valued functionals, so it is appropriate to treat them as operator functions instead of as functionals despite having explicit dependence on functions X ˉ , U ˉ d , U ˉ b \bar{X},\bar{U}_d,\bar{U}_b X ˉ , U ˉ d , U ˉ b . In Hilbert space notation, the operator Taylor expansion of the dynamics and boundary have the form
F ( t , X ˉ + δ X , U ˉ d + δ U d ) = F ( t , X ˉ , U ˉ d ) + F X T ( t , X ˉ , U ˉ d ) δ X + F U d T ( t , X ˉ , U ˉ d ) δ U d + O ( δ 2 ) (17) F(t,\bar{X}+\delta X,\bar{U}_d+\delta U_d)=F(t,\bar{X},\bar{U}_d)+F_X^{\mathsf{T}}(t,\bar{X},\bar{U}_d)\delta X+F_{U_d}^{\mathsf{T}}(t,\bar{X},\bar{U}_d)\delta U_d+O(\delta^2) \tag*{(17)} F ( t , X ˉ + δ X , U ˉ d + δ U d ) = F ( t , X ˉ , U ˉ d ) + F X T ( t , X ˉ , U ˉ d ) δ X + F U d T ( t , X ˉ , U ˉ d ) δ U d + O ( δ 2 ) (17)
N ( t , X ˉ + δ X , U ˉ b + δ U b ) = N ( t , X ˉ , U ˉ b ) + N X T ( t , X ˉ , U ˉ b ) δ X + N U b T ( t , X ˉ , U ˉ b ) δ U b + O ( δ 2 ) (18) N(t,\bar{X}+\delta X,\bar{U}_b+\delta U_b)=N(t,\bar{X},\bar{U}_b)+N_X^{\mathsf{T}}(t,\bar{X},\bar{U}_b)\delta X+N_{U_b}^{\mathsf{T}}(t,\bar{X},\bar{U}_b)\delta U_b+O(\delta^2) \tag*{(18)} N ( t , X ˉ + δ X , U ˉ b + δ U b ) = N ( t , X ˉ , U ˉ b ) + N X T ( t , X ˉ , U ˉ b ) δ X + N U b T ( t , X ˉ , U ˉ b ) δ U b + O ( δ 2 ) (18)
We obtain the right-hand side of the HJB eq. (8) by plugging eqs. (16) to (18) , and a Volterra-Taylor expansion of V X V_X V X . After simplification, the right-hand side of the HJB eq. (8) becomes
min δ U d , δ U b [ L + ⟨ L X , δ X ⟩ + ⟨ L U d , δ U d ⟩ + ⟨ L U b , δ U b ⟩ S + 1 2 ⟨ δ X , L X X δ X ′ ⟩ + 1 2 ⟨ δ U d , ( L U d X + L X U d T ) δ X ⟩ + 1 2 ⟨ δ U b , ( L U b X + L X U b T ) δ X ′ ⟩ S + 1 2 ⟨ δ U d , L U d U d δ U d ′ ⟩ + 1 2 ⟨ δ U b , L U b U b δ U b ′ ⟩ S + ⟨ V X , F ⟩ + ⟨ V X , F X T δ X ⟩ + ⟨ V X , F U d T δ U d ⟩ + ⟨ δ X , V X X F ⟩ + ⟨ δ X , V X X F X T δ X ′ ⟩ + ⟨ δ X , V X X F U d T δ U d ⟩ + 1 2 ⟨ δ X , V X X X F δ X ′ ⟩ ] . (19) \min_{\delta U_d,\delta U_b}\left[L+\left\langle L_X,\delta X\right\rangle+\left\langle L_{U_d},\delta U_d\right\rangle+\left\langle L_{U_b},\delta U_b\right\rangle_S+\frac{1}{2}\left\langle\delta X,L_{XX}\delta X'\right\rangle+\frac{1}{2}\left\langle\delta U_d,\left(L_{U_dX}+L_{XU_d}^{\mathsf{T}}\right)\delta X\right\rangle+\frac{1}{2}\left\langle\delta U_b,\left(L_{U_bX}+L_{XU_b}^{\mathsf{T}}\right)\delta X'\right\rangle_S+\frac{1}{2}\left\langle\delta U_d,L_{U_dU_d}\delta U_d'\right\rangle+\frac{1}{2}\left\langle\delta U_b,L_{U_bU_b}\delta U_b'\right\rangle_S+\left\langle V_X,F\right\rangle+\left\langle V_X,F_X^{\mathsf{T}}\delta X\right\rangle+\left\langle V_X,F_{U_d}^{\mathsf{T}}\delta U_d\right\rangle+\left\langle\delta X,V_{XX}F\right\rangle+\left\langle\delta X,V_{XX}F_X^{\mathsf{T}}\delta X'\right\rangle+\left\langle\delta X,V_{XX}F_{U_d}^{\mathsf{T}}\delta U_d\right\rangle+\frac{1}{2}\left\langle\delta X,V_{XXX}F\delta X'\right\rangle\right]. \tag*{(19)} δ U d , δ U b min [ L + ⟨ L X , δ X ⟩ + ⟨ L U d , δ U d ⟩ + ⟨ L U b , δ U b ⟩ S + 2 1 ⟨ δ X , L X X δ X ′ ⟩ + 2 1 ⟨ δ U d , ( L U d X + L X U d T ) δ X ⟩ + 2 1 ⟨ δ U b , ( L U b X + L X U b T ) δ X ′ ⟩ S + 2 1 ⟨ δ U d , L U d U d δ U d ′ ⟩ + 2 1 ⟨ δ U b , L U b U b δ U b ′ ⟩ S + ⟨ V X , F ⟩ + ⟨ V X , F X T δ X ⟩ + ⟨ V X , F U d T δ U d ⟩ + ⟨ δ X , V X X F ⟩ + ⟨ δ X , V X X F X T δ X ′ ⟩ + ⟨ δ X , V X X F U d T δ U d ⟩ + 2 1 ⟨ δ X , V X X X F δ X ′ ⟩ ] . (19)
Equating eq. (13) to eq. (19) and canceling common terms yields
− d d t ( V + ⟨ V X , δ X ⟩ + 1 2 ⟨ δ X , V X X δ X ′ ⟩ ) = min δ U d , δ U b [ L + ⟨ L X , δ X ⟩ + ⟨ L U d , δ U d ⟩ + ⟨ L U b , δ U b ⟩ S + 1 2 ⟨ δ X , L X X δ X ′ ⟩ + 1 2 ⟨ δ U d , ( L U d X + L X U d T ) δ X ⟩ + 1 2 ⟨ δ U b , ( L U b X + L X U b T ) δ X ′ ⟩ S + 1 2 ⟨ δ U d , L U d U d δ U d ′ ⟩ + 1 2 ⟨ δ U b , L U b U b δ U b ′ ⟩ S + ⟨ V X , F X T δ X ⟩ + ⟨ V X , F U d T δ U d ⟩ + ⟨ δ X , V X X F X T δ X ′ ⟩ + ⟨ δ X , V X X F U d T δ U d ⟩ ] . (20) -\frac{\mathrm{d}}{\mathrm{d}t}\left(V+\left\langle V_X,\delta X\right\rangle+\frac{1}{2}\left\langle\delta X,V_{XX}\delta X'\right\rangle\right)=\min_{\delta U_d,\delta U_b}\left[L+\left\langle L_X,\delta X\right\rangle+\left\langle L_{U_d},\delta U_d\right\rangle+\left\langle L_{U_b},\delta U_b\right\rangle_S+\frac{1}{2}\left\langle\delta X,L_{XX}\delta X'\right\rangle+\frac{1}{2}\left\langle\delta U_d,\left(L_{U_dX}+L_{XU_d}^{\mathsf{T}}\right)\delta X\right\rangle+\frac{1}{2}\left\langle\delta U_b,\left(L_{U_bX}+L_{XU_b}^{\mathsf{T}}\right)\delta X'\right\rangle_S+\frac{1}{2}\left\langle\delta U_d,L_{U_dU_d}\delta U_d'\right\rangle+\frac{1}{2}\left\langle\delta U_b,L_{U_bU_b}\delta U_b'\right\rangle_S+\left\langle V_X,F_X^{\mathsf{T}}\delta X\right\rangle+\left\langle V_X,F_{U_d}^{\mathsf{T}}\delta U_d\right\rangle+\left\langle\delta X,V_{XX}F_X^{\mathsf{T}}\delta X'\right\rangle+\left\langle\delta X,V_{XX}F_{U_d}^{\mathsf{T}}\delta U_d\right\rangle\right]. \tag*{(20)} − d t d ( V + ⟨ V X , δ X ⟩ + 2 1 ⟨ δ X , V X X δ X ′ ⟩ ) = δ U d , δ U b min [ L + ⟨ L X , δ X ⟩ + ⟨ L U d , δ U d ⟩ + ⟨ L U b , δ U b ⟩ S + 2 1 ⟨ δ X , L X X δ X ′ ⟩ + 2 1 ⟨ δ U d , ( L U d X + L X U d T ) δ X ⟩ + 2 1 ⟨ δ U b , ( L U b X + L X U b T ) δ X ′ ⟩ S + 2 1 ⟨ δ U d , L U d U d δ U d ′ ⟩ + 2 1 ⟨ δ U b , L U b U b δ U b ′ ⟩ S + ⟨ V X , F X T δ X ⟩ + ⟨ V X , F U d T δ U d ⟩ + ⟨ δ X , V X X F X T δ X ′ ⟩ + ⟨ δ X , V X X F U d T δ U d ⟩ ] . (20)
Remark 3. The exact singleton Newton minimizer δ U b ∗ \delta U_b^* δ U b ∗ of the approximate HJB equation eq. (20) does not incorporate the value functional V ( t , X ˉ ) V(t,\bar{X}) V ( t , X ˉ ) or its derivatives V X ( t , X ˉ ) , V X X ( t , X ˉ ) V_X(t,\bar{X}), V_{XX}(t,\bar{X}) V X ( t , X ˉ ) , V X X ( t , X ˉ ) .
This is an important point. The value functional is defined as the minimization surface of the original problem in eq. (7) and the apparent decoupling between the optimal update δ U b ∗ \delta U_b^* δ U b ∗ and the value functional and/or its derivatives within the resulting approximate HJB eq. (20) yields a naive update. The authors in [23 ] and [31 ] realize this fact, and use the Green’s theorem in order to incorporate boundary information into specific terms in eq. (20) . However, there are errors in their application of Green’s theorem in the multivariate case, as noted in [24 ] .
Green's Theorem in Hilbert Spaces
Green’s theorem is used widely in calculus to relate the volume integral of the interior of a region to a surface integral of its boundary. In the context of STDDP, it allows us to capture pertinent effects of the value function on the boundary.
Assumption 5. F X F_X F X is a linear operator with standard form given by A x ( t , x ) A_x(t,x) A x ( t , x ) in [24 ] , and N X N_X N X is a linear operator with standard form given by β A ( t , ξ ) \beta_A(t,\xi) β A ( t , ξ ) in [24 ] .
Theorem 4.1. Let Y ( t ) , Z ( t ) ∈ L 2 n ( D ˉ ) Y(t),Z(t) \in L_2^n(\bar{D}) Y ( t ) , Z ( t ) ∈ L 2 n ( D ˉ ) . Under assumption 5 5 5 , the following holds:
⟨ Y ( t ) , F X ( t , X ( t ) , U ˉ d ( t ) ) Z ( t ) ⟩ − ⟨ Z ( t ) , F X ∗ ( t , X ( t ) , U ˉ d ( t ) ) Y ( t ) ⟩ = ⟨ Y ( t ) , N X ( t , X ( t ) , U ˉ b ( t ) ) Z ( t ) ⟩ S − ⟨ Z ( t ) , N X ∗ ( t , X ( t ) , U ˉ b ( t ) ) Y ( t ) ⟩ S (21) \left\langle Y(t),F_X\left(t,X(t),\bar{U}_d(t)\right)Z(t)\right\rangle-\left\langle Z(t),F_X^*\left(t,X(t),\bar{U}_d(t)\right)Y(t)\right\rangle
=
\left\langle Y(t),N_X\left(t,X(t),\bar{U}_b(t)\right)Z(t)\right\rangle_S-\left\langle Z(t),N_X^*\left(t,X(t),\bar{U}_b(t)\right)Y(t)\right\rangle_S \tag*{(21)} ⟨ Y ( t ) , F X ( t , X ( t ) , U ˉ d ( t ) ) Z ( t ) ⟩ − ⟨ Z ( t ) , F X ∗ ( t , X ( t ) , U ˉ d ( t ) ) Y ( t ) ⟩ = ⟨ Y ( t ) , N X ( t , X ( t ) , U ˉ b ( t ) ) Z ( t ) ⟩ S − ⟨ Z ( t ) , N X ∗ ( t , X ( t ) , U ˉ b ( t ) ) Y ( t ) ⟩ S (21)
The equivalent fields representation can be found in [24 ] , and the proof is a standard result (c.f. [32 ] ). The following corollary is a direct application of theorem 4.1 4.1 4.1 to the applicable terms of the HJB in eq. (20) .
Corollary 4.1. If assumption 5 5 5 holds, then
⟨ V X , F X T δ X ⟩ = ⟨ δ X , F X ∗ V X ⟩ − ⟨ V X , Δ N ⟩ S − ⟨ V X , N U b T δ U b ⟩ S − ⟨ δ X , N X ∗ V X ⟩ S (22) \left\langle V_X,F_X^\mathsf{T}\delta X\right\rangle
=
\left\langle\delta X,F_X^*V_X\right\rangle
-\left\langle V_X,\Delta N\right\rangle_S
-\left\langle V_X,N_{U_b}^\mathsf{T}\delta U_b\right\rangle_S
-\left\langle\delta X,N_X^*V_X\right\rangle_S \tag*{(22)} ⟨ V X , F X T δ X ⟩ = ⟨ δ X , F X ∗ V X ⟩ − ⟨ V X , Δ N ⟩ S − ⟨ V X , N U b T δ U b ⟩ S − ⟨ δ X , N X ∗ V X ⟩ S (22) and
⟨ V X X δ X , F X T δ X ′ ⟩ = ⟨ δ X , F X ∗ V X X δ X ′ ⟩ − ⟨ V X X δ X , Δ N ⟩ S − ⟨ V X X δ X , N U b T δ U b ⟩ S − ⟨ δ X , N X ∗ V X X δ X ′ ⟩ S (23) \left\langle V_{XX}\delta X,F_X^\mathsf{T}\delta X'\right\rangle
=
\left\langle\delta X,F_X^*V_{XX}\delta X'\right\rangle
-\left\langle V_{XX}\delta X,\Delta N\right\rangle_S
-\left\langle V_{XX}\delta X,N_{U_b}^\mathsf{T}\delta U_b\right\rangle_S
-\left\langle\delta X,N_X^*V_{XX}\delta X'\right\rangle_S \tag*{(23)} ⟨ V X X δ X , F X T δ X ′ ⟩ = ⟨ δ X , F X ∗ V X X δ X ′ ⟩ − ⟨ V X X δ X , Δ N ⟩ S − ⟨ V X X δ X , N U b T δ U b ⟩ S − ⟨ δ X , N X ∗ V X X δ X ′ ⟩ S (23)
where Δ N = N ( X + δ X , U b + δ U b ) − N ( X , U b ) \Delta N=N(X+\delta X,U_b+\delta U_b)-N(X,U_b) Δ N = N ( X + δ X , U b + δ U b ) − N ( X , U b ) , F X ∗ F_X^* F X ∗ is the adjoint operator of F X F_X F X , N X ∗ N_X^* N X ∗ is the adjoint operator of N X N_X N X , and we have suppressed explicit time dependencies for simplicity.
Plugging equations eqs. (22) and (23) into eq. (20) yields
− d d t ( V + ⟨ V X , δ X ⟩ + 1 2 ⟨ δ X , V X X δ X ′ ⟩ ) = min δ U d , δ U b [ L + ⟨ L X , δ X ⟩ + ⟨ L U d , δ U d ⟩ + ⟨ L U b , δ U b ⟩ S + 1 2 ⟨ δ X , L X X δ X ′ ⟩ -\frac{\mathrm{d}}{\mathrm{d}t}\left(V+\left\langle V_X,\delta X\right\rangle+\frac{1}{2}\left\langle\delta X,V_{XX}\delta X'\right\rangle\right)
=
\min_{\delta U_d,\delta U_b}\left[
L+\left\langle L_X,\delta X\right\rangle+\left\langle L_{U_d},\delta U_d\right\rangle+\left\langle L_{U_b},\delta U_b\right\rangle_S+\frac{1}{2}\left\langle\delta X,L_{XX}\delta X'\right\rangle
\right. − d t d ( V + ⟨ V X , δ X ⟩ + 2 1 ⟨ δ X , V X X δ X ′ ⟩ ) = δ U d , δ U b min [ L + ⟨ L X , δ X ⟩ + ⟨ L U d , δ U d ⟩ + ⟨ L U b , δ U b ⟩ S + 2 1 ⟨ δ X , L X X δ X ′ ⟩
+ 1 2 ⟨ δ U d , ( L U d X + L X U d T ) δ X ′ ⟩ + 1 2 ⟨ δ U b , ( L U b X + L X U b T ) δ X ′ ⟩ S + 1 2 ⟨ δ U d , L U d U d δ U d ′ ⟩ \left.
+\frac{1}{2}\left\langle\delta U_d,\left(L_{U_dX}+L_{XU_d}^{\mathsf{T}}\right)\delta X'\right\rangle
+\frac{1}{2}\left\langle\delta U_b,\left(L_{U_bX}+L_{XU_b}^{\mathsf{T}}\right)\delta X'\right\rangle_S
+\frac{1}{2}\left\langle\delta U_d,L_{U_dU_d}\delta U_d'\right\rangle
\right. + 2 1 ⟨ δ U d , ( L U d X + L X U d T ) δ X ′ ⟩ + 2 1 ⟨ δ U b , ( L U b X + L X U b T ) δ X ′ ⟩ S + 2 1 ⟨ δ U d , L U d U d δ U d ′ ⟩
+ 1 2 ⟨ δ U b , L U b U b δ U b ′ ⟩ S + ⟨ δ X , F X ∗ V X ⟩ − ⟨ V X , Δ N ⟩ S − ⟨ V X , N U b T δ U b ⟩ S − ⟨ δ X , N X ∗ V X ⟩ S \left.
+\frac{1}{2}\left\langle\delta U_b,L_{U_bU_b}\delta U_b'\right\rangle_S
+\left\langle\delta X,F_X^*V_X\right\rangle
-\left\langle V_X,\Delta N\right\rangle_S
-\left\langle V_X,N_{U_b}^{\mathsf{T}}\delta U_b\right\rangle_S
-\left\langle\delta X,N_X^*V_X\right\rangle_S
\right. + 2 1 ⟨ δ U b , L U b U b δ U b ′ ⟩ S + ⟨ δ X , F X ∗ V X ⟩ − ⟨ V X , Δ N ⟩ S − ⟨ V X , N U b T δ U b ⟩ S − ⟨ δ X , N X ∗ V X ⟩ S
+ ⟨ V X , F U d T δ U d ⟩ + ⟨ δ X , F X ∗ V X X δ X ′ ⟩ − ⟨ δ X , V X X Δ N ⟩ − ⟨ δ X , V X X N U b T δ U b ⟩ S \left.
+\left\langle V_X,F_{U_d}^{\mathsf{T}}\delta U_d\right\rangle
+\left\langle\delta X,F_X^*V_{XX}\delta X'\right\rangle
-\left\langle\delta X,V_{XX}\Delta N\right\rangle
-\left\langle\delta X,V_{XX}N_{U_b}^{\mathsf{T}}\delta U_b\right\rangle_S
\right. + ⟨ V X , F U d T δ U d ⟩ + ⟨ δ X , F X ∗ V X X δ X ′ ⟩ − ⟨ δ X , V X X Δ N ⟩ − ⟨ δ X , V X X N U b T δ U b ⟩ S
− ⟨ δ X , N X ∗ V X X δ X ′ ⟩ S + ⟨ δ X , V X X F U d T δ U d ⟩ ] (24) \left.
-\left\langle\delta X,N_X^*V_{XX}\delta X'\right\rangle_S
+\left\langle\delta X,V_{XX}F_{U_d}^{\mathsf{T}}\delta U_d\right\rangle
\right] \tag*{(24)} − ⟨ δ X , N X ∗ V X X δ X ′ ⟩ S + ⟨ δ X , V X X F U d T δ U d ⟩ ] (24)
The form of the HJB equation in eq. (24) now properly incorporates boundary information of the value functional. As shown in the subsequent section, the resulting optimal update δ U b ∗ \delta U_b^* δ U b ∗ leverages the first and second derivative of the value functional, which is expected in the context of the established DDP method in finite dimensions.
We note that the form of the HJB eq. (24) is remarkably different than that of [23 ] . The fundamental differences arise due to a) improper application of Green’s theorem, as discussed in [24 ] , and b) terms that are a result of a fundamental difference of reasoning followed in their derivation. For example, the expansions in [23 ] are quite different than the ones computed here, and may reflect an evolution in the DDP approach over decades of research.
Optimal Distributed and Boundary Control Solutions
We find two singleton Newton solutions to the HJB eq. (24) ; one for the optimal distributed control update δ U d ∗ \delta U_d^* δ U d ∗ , and one for the optimal boundary control update δ U b ∗ \delta U_b^* δ U b ∗ .
Theorem 5.1. Under the stated assumptions, the optimal distributed update δ U d ∗ \delta U_d^* δ U d ∗ and the optimal boundary update δ U b ∗ \delta U_b^* δ U b ∗ are given in Hilbert spaces by
δ U d ∗ = − L U d U d − 1 ( F U d T V X + L U d ) − 1 2 L U d U d − 1 ( L U d X + L X U d T + 2 F U d T V X X ) δ X (25) \delta U_d^*=-L_{U_dU_d}^{-1}\left(F_{U_d}^{\mathsf{T}}V_X+L_{U_d}\right)-\frac{1}{2}L_{U_dU_d}^{-1}\left(L_{U_dX}+L_{XU_d}^{\mathsf{T}}+2F_{U_d}^{\mathsf{T}}V_{XX}\right)\delta X \tag*{(25)} δ U d ∗ = − L U d U d − 1 ( F U d T V X + L U d ) − 2 1 L U d U d − 1 ( L U d X + L X U d T + 2 F U d T V X X ) δ X (25) δ U b ∗ = − L U b U b − 1 ( L U b − N U b T V X ) − 1 2 L U b U b − 1 ( L U b X + L X U b T − 2 N U b T V X X ) δ X (26) \delta U_b^*=-L_{U_bU_b}^{-1}\left(L_{U_b}-N_{U_b}^{\mathsf{T}}V_X\right)-\frac{1}{2}L_{U_bU_b}^{-1}\left(L_{U_bX}+L_{XU_b}^{\mathsf{T}}-2N_{U_b}^{\mathsf{T}}V_{XX}\right)\delta X \tag*{(26)} δ U b ∗ = − L U b U b − 1 ( L U b − N U b T V X ) − 2 1 L U b U b − 1 ( L U b X + L X U b T − 2 N U b T V X X ) δ X (26) where we have defined the inverse operators L U d U d − 1 ∈ L ( L 2 k ( D ) , L 2 k ( D ) ) L_{U_dU_d}^{-1}\in\mathcal{L}(L_2^k(D),L_2^k(D)) L U d U d − 1 ∈ L ( L 2 k ( D ) , L 2 k ( D )) and L U b U b − 1 ∈ L ( L 2 l ( S ) , L 2 l ( S ) ) L_{U_bU_b}^{-1}\in\mathcal{L}(L_2^l(S),L_2^l(S)) L U b U b − 1 ∈ L ( L 2 l ( S ) , L 2 l ( S )) by their respective inverse kernels, given by
L U d U d − 1 ( t ) W ( t ) = ∫ D L ˉ U d U d ( t , x , y ) W ( t , y ) d y (27) L_{U_dU_d}^{-1}(t)W(t)=\int_D\bar{L}_{U_dU_d}(t,x,y)W(t,y)\,dy \tag*{(27)} L U d U d − 1 ( t ) W ( t ) = ∫ D L ˉ U d U d ( t , x , y ) W ( t , y ) d y (27) L U b U b − 1 W ( t ) = ∫ S L ˉ U b U b ( t , ξ , η ) W ( t , η ) d S η (28) L_{U_bU_b}^{-1}W(t)=\int_S\bar{L}_{U_bU_b}(t,\xi,\eta)W(t,\eta)\,dS_\eta \tag*{(28)} L U b U b − 1 W ( t ) = ∫ S L ˉ U b U b ( t , ξ , η ) W ( t , η ) d S η (28) with L U d U d ( t , x , y ) L_{U_dU_d}(t,x,y) L U d U d ( t , x , y ) and L ˉ U b U b ( t , ξ , η ) \bar{L}_{U_bU_b}(t,\xi,\eta) L ˉ U b U b ( t , ξ , η ) denoting the kernel function of the operator L U d U d − 1 ( t ) L_{U_dU_d}^{-1}(t) L U d U d − 1 ( t ) and L U d U d − 1 ( t ) L_{U_dU_d}^{-1}(t) L U d U d − 1 ( t ) (resp.), and satisfying the property of inverses for kernels
∫ D L U d U d ( t , x , y ) L ˉ U d U d ( t , y , x ′ ) d y = I δ ( x − x ′ ) (29) \int_D L_{U_dU_d}(t,x,y)\bar{L}_{U_dU_d}(t,y,x')\,dy=I\delta(x-x') \tag*{(29)} ∫ D L U d U d ( t , x , y ) L ˉ U d U d ( t , y , x ′ ) d y = I δ ( x − x ′ ) (29) ∫ S L U b U b ( t , ξ , η ) L ˉ U d U d ( t , η , ξ ′ ) d S η = I δ ( ξ − ξ ′ ) (30) \int_S L_{U_bU_b}(t,\xi,\eta)\bar{L}_{U_dU_d}(t,\eta,\xi')\,dS_\eta=I\delta(\xi-\xi') \tag*{(30)} ∫ S L U b U b ( t , ξ , η ) L ˉ U d U d ( t , η , ξ ′ ) d S η = I δ ( ξ − ξ ′ ) (30) Proof. The result can be found by applying a Newton step (e.g. taking the respective partial derivative and setting equal to zero) of the HJB eq. (24) .
The equivalent expressions in fields notation expose the spatial integration that takes place in these calculations, and are provided for completeness
δ U d = − ∫ D L ˉ U d U d ( t , x , y ) ( F U d T ( t , y , y ′ ) V X ( t , y ′ ) + L U d ( t , y ) ) d y − 1 2 ∫ D ∫ D L ˉ U d U d ( t , x , y ) ( L U d X ( t , y , y ′ ) + L X U d T ( t , y , y ′ ) + 2 F U d ( t , y , y ′ ′ ) V X X ( t , y ′ ′ , y ′ ) ) δ X ( t , y ′ ) d y d y ′ (31) \delta U_d=-\int_D \bar{L}_{U_dU_d}(t,x,y)\left(F_{U_d}^{\mathsf{T}}(t,y,y')V_X(t,y')+L_{U_d}(t,y)\right)\,\mathrm{d}y
-\frac{1}{2}\int_D\int_D\bar{L}_{U_dU_d}(t,x,y)\left(L_{U_dX}(t,y,y')+L_{XU_d}^{\mathsf{T}}(t,y,y')+2F_{U_d}(t,y,y'')V_{XX}(t,y'',y')\right)\delta X(t,y')\,\mathrm{d}y\,\mathrm{d}y' \tag*{(31)} δ U d = − ∫ D L ˉ U d U d ( t , x , y ) ( F U d T ( t , y , y ′ ) V X ( t , y ′ ) + L U d ( t , y ) ) d y − 2 1 ∫ D ∫ D L ˉ U d U d ( t , x , y ) ( L U d X ( t , y , y ′ ) + L X U d T ( t , y , y ′ ) + 2 F U d ( t , y , y ′′ ) V X X ( t , y ′′ , y ′ ) ) δ X ( t , y ′ ) d y d y ′ (31)
δ U b = − ∫ S L ˉ U b U b ( t , x , η ) ( L U b ( t , η ) − N U b T ( t , η , η ′ ) V X ( t , η ′ ) ) d S η − 1 2 ∫ S ∫ D L ˉ U b U b ( t , x , η ) ( L U b X ( t , η , y ) + L X U b ( t , η , y ) − 2 N U b T ( t , η , η ′ ) V X X ( t , η ′ , y ) ) δ X ( t , y ) d y d S η (32) \delta U_b=-\int_S\bar{L}_{U_bU_b}(t,x,\eta)\left(L_{U_b}(t,\eta)-N_{U_b}^{\mathsf{T}}(t,\eta,\eta')V_X(t,\eta')\right)\,\mathrm{d}S_\eta
-\frac{1}{2}\int_S\int_D\bar{L}_{U_bU_b}(t,x,\eta)\left(L_{U_bX}(t,\eta,y)+L_{XU_b}(t,\eta,y)-2N_{U_b}^{\mathsf{T}}(t,\eta,\eta')V_{XX}(t,\eta',y)\right)\delta X(t,y)\,\mathrm{d}y\,\mathrm{d}S_\eta \tag*{(32)} δ U b = − ∫ S L ˉ U b U b ( t , x , η ) ( L U b ( t , η ) − N U b T ( t , η , η ′ ) V X ( t , η ′ ) ) d S η − 2 1 ∫ S ∫ D L ˉ U b U b ( t , x , η ) ( L U b X ( t , η , y ) + L X U b ( t , η , y ) − 2 N U b T ( t , η , η ′ ) V X X ( t , η ′ , y ) ) δ X ( t , y ) d y d S η (32)
The Backward Value Functional Equations
The value functional is a backward equation according to the HJB eq. (8) , and is separated by order into zeroth, first, and second order derivative of the value functional. We present the fields representations of these backward equations without cross terms for simplicity. The more general forms of the backward equations with cross terms have been derived, but are lengthy and are omitted due to length considerations.
Theorem 6.1. Under the above stated assumptions, and with optimal control in fields representation given by eqs. (31) and (32) , the zeroth-order backward value functional equation is given by
− d d t V ( t , X ( t , x ) ) = L − ∫ S V X ( t , ξ ) T Δ N ( t , ξ ) d S ξ − 1 2 ∫ D ∫ D ( L U d T ( t , y ) + V X T ( t , x ) F U d ( t , x , y ) ) L ˉ U d U d ( t , y , y ′ ) ( L U d ( t , y ′ ) + F U d T ( t , y ′ , y ′ ′ ) V X ( t , y ′ ′ ) ) d y d y ′ − 1 2 ∫ S ∫ S ( L U b T ( t , ξ ′ ) − V X T ( t , ξ ) N U b ( t , ξ , ξ ′ ) ) L ˉ U b U b ( t , ξ ′ , η ) ( L U b ( t , η ) − N U b T ( t , η , η ′ ) V X ( t , η ′ ) ) d S ξ ′ d S η (33) -\frac{\mathrm{d}}{\mathrm{d}t}V(t,X(t,x))
=L-\int_S V_X(t,\xi)^{\mathsf{T}}\Delta N(t,\xi)\,\mathrm{d}S_\xi
-\frac{1}{2}\int_D\int_D\left(L_{U_d}^{\mathsf{T}}(t,y)+V_X^{\mathsf{T}}(t,x)F_{U_d}(t,x,y)\right)\bar{L}_{U_dU_d}(t,y,y')\left(L_{U_d}(t,y')+F_{U_d}^{\mathsf{T}}(t,y',y'')V_X(t,y'')\right)\,\mathrm{d}y\,\mathrm{d}y'
-\frac{1}{2}\int_S\int_S\left(L_{U_b}^{\mathsf{T}}(t,\xi')-V_X^{\mathsf{T}}(t,\xi)N_{U_b}(t,\xi,\xi')\right)\bar{L}_{U_bU_b}(t,\xi',\eta)\left(L_{U_b}(t,\eta)-N_{U_b}^{\mathsf{T}}(t,\eta,\eta')V_X(t,\eta')\right)\,\mathrm{d}S_{\xi'}\,\mathrm{d}S_\eta \tag*{(33)} − d t d V ( t , X ( t , x )) = L − ∫ S V X ( t , ξ ) T Δ N ( t , ξ ) d S ξ − 2 1 ∫ D ∫ D ( L U d T ( t , y ) + V X T ( t , x ) F U d ( t , x , y ) ) L ˉ U d U d ( t , y , y ′ ) ( L U d ( t , y ′ ) + F U d T ( t , y ′ , y ′′ ) V X ( t , y ′′ ) ) d y d y ′ − 2 1 ∫ S ∫ S ( L U b T ( t , ξ ′ ) − V X T ( t , ξ ) N U b ( t , ξ , ξ ′ ) ) L ˉ U b U b ( t , ξ ′ , η ) ( L U b ( t , η ) − N U b T ( t , η , η ′ ) V X ( t , η ′ ) ) d S ξ ′ d S η (33) with terminal condition
V ( t f , X ( t f , x ) ) = ϕ ( t f , X ( t f , x ) ) , (34) V(t_f,X(t_f,x))=\phi(t_f,X(t_f,x)), \tag*{(34)} V ( t f , X ( t f , x )) = ϕ ( t f , X ( t f , x )) , (34) the first-order backward value functional equation is given by
− d d t V X ( t , X ( t , x ) ) = L X ( t , x ) + F X T ( t , x , y ) V X ( t , y ) − ∫ D ∫ D V X X ( t , x , x ′ ) F U d ( t , x ′ , y ) L ˉ U d U d ( t , y , y ′ ) ( L U d ( t , y ′ ) + F U d T ( t , y ′ , y ′ ′ ) V X ( t , y ′ ′ ) ) d y d y ′ + ∫ S ∫ S V X X ( t , ξ , ξ ′ ) N U b ( t , ξ ′ , η ) L ˉ U b U b ( t , η , η ′ ) ( L U b ( t , η ′ ) − N U b T ( t , η ′ , η ′ ′ ) V X ( t , η ′ ′ ) ) d S η d S η ′ (35) -\frac{\mathrm{d}}{\mathrm{d}t}V_X(t,X(t,x))
=L_X(t,x)+F_X^{\mathsf{T}}(t,x,y)V_X(t,y)
-\int_D\int_D V_{XX}(t,x,x')F_{U_d}(t,x',y)\bar{L}_{U_dU_d}(t,y,y')\left(L_{U_d}(t,y')+F_{U_d}^{\mathsf{T}}(t,y',y'')V_X(t,y'')\right)\,\mathrm{d}y\,\mathrm{d}y'
+\int_S\int_S V_{XX}(t,\xi,\xi')N_{U_b}(t,\xi',\eta)\bar{L}_{U_bU_b}(t,\eta,\eta')\left(L_{U_b}(t,\eta')-N_{U_b}^{\mathsf{T}}(t,\eta',\eta'')V_X(t,\eta'')\right)\,\mathrm{d}S_\eta\,\mathrm{d}S_{\eta'} \tag*{(35)} − d t d V X ( t , X ( t , x )) = L X ( t , x ) + F X T ( t , x , y ) V X ( t , y ) − ∫ D ∫ D V X X ( t , x , x ′ ) F U d ( t , x ′ , y ) L ˉ U d U d ( t , y , y ′ ) ( L U d ( t , y ′ ) + F U d T ( t , y ′ , y ′′ ) V X ( t , y ′′ ) ) d y d y ′ + ∫ S ∫ S V X X ( t , ξ , ξ ′ ) N U b ( t , ξ ′ , η ) L ˉ U b U b ( t , η , η ′ ) ( L U b ( t , η ′ ) − N U b T ( t , η ′ , η ′′ ) V X ( t , η ′′ ) ) d S η d S η ′ (35) with boundary and terminal conditions
0 = N X ∗ ( t , ξ , η ) V X ( t , η ) − ∫ S V X X ( t , ξ , η ) Δ N ( t , η ) d S η (36) 0=N_X^*(t,\xi,\eta)V_X(t,\eta)-\int_S V_{XX}(t,\xi,\eta)\Delta N(t,\eta)\,\mathrm{d}S_\eta \tag*{(36)} 0 = N X ∗ ( t , ξ , η ) V X ( t , η ) − ∫ S V X X ( t , ξ , η ) Δ N ( t , η ) d S η (36) V X ( t f , X ( t f , x ) ) = ϕ X ( t f , X ( t f , x ) ) , (37) V_X(t_f,X(t_f,x))=\phi_X(t_f,X(t_f,x)), \tag*{(37)} V X ( t f , X ( t f , x )) = ϕ X ( t f , X ( t f , x )) , (37)
and the second-order backward value functional equation is given by
− d d t V X X ( t , x , y ) = L X X ( t , x , y ) + F X ∗ ( t , x , y ′ ) V X X ( t , y ′ , y ) + [ F X ∗ ( t , x , y ′ ) V X X ( t , y ′ , y ) ] T − ∫ D ∫ D V X X ( t , x , x ′ ) F U d ( t , x ′ , y ′ ) L ˉ U d U d ( t , y ′ , y ′ ′ ) F U d T ( t , y ′ ′ , z ) V X X ( t , z , y ) d y ′ d y ′ ′ − ∫ S ∫ S V X X ( t , x , ξ ) N U b ( t , ξ , ξ ′ ) L ˉ U b U b ( t , ξ ′ , η ) N U b T ( t , η , η ′ ) V X X ( t , η ′ , y ) d S ξ ′ d S η (38) -\frac{\mathrm{d}}{\mathrm{d}t}V_{XX}(t,x,y)=L_{XX}(t,x,y)+F_X^{*}(t,x,y')V_{XX}(t,y',y)+\left[F_X^{*}(t,x,y')V_{XX}(t,y',y)\right]^{\mathsf{T}}
-\int_D\int_D V_{XX}(t,x,x')F_{U_d}(t,x',y')\bar{L}_{U_dU_d}(t,y',y'')F_{U_d}^{\mathsf{T}}(t,y'',z)V_{XX}(t,z,y)\,\mathrm{d}y'\mathrm{d}y''
-\int_S\int_S V_{XX}(t,x,\xi)N_{U_b}(t,\xi,\xi')\bar{L}_{U_bU_b}(t,\xi',\eta)N_{U_b}^{\mathsf{T}}(t,\eta,\eta')V_{XX}(t,\eta',y)\,\mathrm{d}S_{\xi'}\mathrm{d}S_{\eta} \tag*{(38)} − d t d V X X ( t , x , y ) = L X X ( t , x , y ) + F X ∗ ( t , x , y ′ ) V X X ( t , y ′ , y ) + [ F X ∗ ( t , x , y ′ ) V X X ( t , y ′ , y ) ] T − ∫ D ∫ D V X X ( t , x , x ′ ) F U d ( t , x ′ , y ′ ) L ˉ U d U d ( t , y ′ , y ′′ ) F U d T ( t , y ′′ , z ) V X X ( t , z , y ) d y ′ d y ′′ − ∫ S ∫ S V X X ( t , x , ξ ) N U b ( t , ξ , ξ ′ ) L ˉ U b U b ( t , ξ ′ , η ) N U b T ( t , η , η ′ ) V X X ( t , η ′ , y ) d S ξ ′ d S η (38)
with boundary and terminal conditions
0 = N X ∗ ( t , ξ , η ) V X X ( t , η , y ) (39) 0=N_X^{*}(t,\xi,\eta)V_{XX}(t,\eta,y) \tag*{(39)} 0 = N X ∗ ( t , ξ , η ) V X X ( t , η , y ) (39)
V X X ( t f , X ( t f , x ) ) = ϕ X X ( t f , X ( t f , x ) ) (40) V_{XX}(t_f,X(t_f,x))=\phi_{XX}(t_f,X(t_f,x)) \tag*{(40)} V X X ( t f , X ( t f , x )) = ϕ X X ( t f , X ( t f , x )) (40)
Proof. These equations are obtained by plugging in eqs. (25) and (26) into the eq. (24) and grouping terms by order of δ X \delta X δ X .
The iterative forward-backward system is completed by the approximate variation of the field and boundary dynamics, which are found by rearranging eqs. (17) and (18) as
d δ X ( t , x ) d t = F ( X ˉ + δ X , U ˉ d + δ U d , t , x ) − F ( X ˉ , U ˉ d , t , x ) = F X T ( X ˉ , U ˉ d , t , x ) δ X + F U d T ( X ˉ , U ˉ d , t , x ) δ U d , x ∈ D (41) \frac{\mathrm{d}\delta X(t,x)}{\mathrm{d}t}=F(\bar{X}+\delta X,\bar{U}_d+\delta U_d,t,x)-F(\bar{X},\bar{U}_d,t,x)
=F_X^{\mathsf{T}}(\bar{X},\bar{U}_d,t,x)\delta X+F_{U_d}^{\mathsf{T}}(\bar{X},\bar{U}_d,t,x)\delta U_d,\qquad x\in D \tag*{(41)} d t d δ X ( t , x ) = F ( X ˉ + δ X , U ˉ d + δ U d , t , x ) − F ( X ˉ , U ˉ d , t , x ) = F X T ( X ˉ , U ˉ d , t , x ) δ X + F U d T ( X ˉ , U ˉ d , t , x ) δ U d , x ∈ D (41)
0 = N ( X ˉ + δ X , U ˉ b + δ U b , t , ξ ) − N ( X ˉ , U ˉ b , t , ξ ) = N X T ( X ˉ , U ˉ b , t , ξ ) δ X + N U b T ( X ˉ , U ˉ b , t , ξ ) δ U b , ξ ∈ S . (42) 0=N(\bar{X}+\delta X,\bar{U}_b+\delta U_b,t,\xi)-N(\bar{X},\bar{U}_b,t,\xi)
=N_X^{\mathsf{T}}(\bar{X},\bar{U}_b,t,\xi)\delta X+N_{U_b}^{\mathsf{T}}(\bar{X},\bar{U}_b,t,\xi)\delta U_b,\qquad\xi\in S. \tag*{(42)} 0 = N ( X ˉ + δ X , U ˉ b + δ U b , t , ξ ) − N ( X ˉ , U ˉ b , t , ξ ) = N X T ( X ˉ , U ˉ b , t , ξ ) δ X + N U b T ( X ˉ , U ˉ b , t , ξ ) δ U b , ξ ∈ S . (42)
Finally, the control updates of iteration k + 1 k+1 k + 1 are given by
U d k + 1 = U d k + γ d δ U d k (43) U_d^{k+1}=U_d^k+\gamma_d\delta U_d^k \tag*{(43)} U d k + 1 = U d k + γ d δ U d k (43)
U b k + 1 = U b k + γ b δ U b k . (44) U_b^{k+1}=U_b^k+\gamma_b\delta U_b^k. \tag*{(44)} U b k + 1 = U b k + γ b δ U b k . (44)
Recovering Standard Results
The optimal distributed and boundary control and resulting backward value functional equations represent a generalization of a) DDP in finite dimensions and b) the LQR for PDEs. These results are standard results in the control literature, and as such it is important to the validity of our approach to clearly demonstrate that these standard results can be recovered from the equations detailed in the previous sections.
Differential Dynamic Programming in Finite Dimensions
We begin by roughly outlining an analogous derivation of DDP in finite dimensions. There are many different formulations of DDP in finite dimensions. Our approach specifically follows a body of literature that expands the pertinent functionals around a nominal trajectory. Despite having an extra term for a terminal constraint, we refer to [30 ] as they present a clean derivation that represents a finite dimensional analogue to the derivation in this document. We ignore the terms having to do with the terminal constraint. and the terms that come from second order expansions of the dynamics for ease of comparison. Therein they consider a finite dimensional system of the general form
d d t x = F ( x , u , t ) , x ( t 0 ) = x 0 (45) \frac{\mathrm{d}}{\mathrm{d}t}x = F(x,u,t), \qquad x(t_0)=x_0 \tag*{(45)} d t d x = F ( x , u , t ) , x ( t 0 ) = x 0 (45)
The optimization problem is formulated as
V ( x 0 , t 0 ) = min u J ( x , u ) = min u [ ϕ ( x ( t f ) , t f ) + ∫ t 0 t f L ( x , u , t ) d t ] (46) V(x_0,t_0)=\min_{u}J(x,u)
=\min_{u}\left[\phi(x(t_f),t_f)+\int_{t_0}^{t_f}L(x,u,t)\,\mathrm{d}t\right] \tag*{(46)} V ( x 0 , t 0 ) = u min J ( x , u ) = u min [ ϕ ( x ( t f ) , t f ) + ∫ t 0 t f L ( x , u , t ) d t ] (46)
After applying standard Taylor expansions of the value functional, its first and second derivative, the dynamics, and the cost functional, plugging them into the HJB equation and performing Newton minimization, they obtain the optimal control update as
δ u = − L u u − 1 ( L u + F u V x ) − 1 2 L u u − 1 ( L u x + L x u T + 2 F u V X X ) δ x (47) \delta u=-L_{uu}^{-1}(L_u+F_uV_x)-\frac{1}{2}L_{uu}^{-1}(L_{ux}+L_{xu}^{\mathsf{T}}+2F_uV_{XX})\delta x \tag*{(47)} δ u = − L uu − 1 ( L u + F u V x ) − 2 1 L uu − 1 ( L ux + L xu T + 2 F u V X X ) δ x (47)
This is equivalent in form to the optimal distributed and boundary control update in Hilbert spaces given in eqs. (25) and (26) . The resulting backward equations of the value functional in [30 ] are given by
− d d t V = L − 1 2 k T L u u k (48) -\frac{\mathrm{d}}{\mathrm{d}t}V=L-\frac{1}{2}k^{\mathsf{T}}L_{uu}k \tag*{(48)} − d t d V = L − 2 1 k T L uu k (48)
− d d t V x = L x + F x V x − K T L u u k (49) -\frac{\mathrm{d}}{\mathrm{d}t}V_x=L_x+F_xV_x-K^{\mathsf{T}}L_{uu}k \tag*{(49)} − d t d V x = L x + F x V x − K T L uu k (49)
− d d t V X X = L x x − K T L u u K + V x x F x T + F x V x x (50) -\frac{\mathrm{d}}{\mathrm{d}t}V_{XX}=L_{xx}-K^{\mathsf{T}}L_{uu}K+V_{xx}F_x^{\mathsf{T}}+F_xV_{xx} \tag*{(50)} − d t d V X X = L xx − K T L uu K + V xx F x T + F x V xx (50)
where k ∈ R k k\in\mathbb{R}^{k} k ∈ R k , and K ∈ R k × n K\in\mathbb{R}^{k\times n} K ∈ R k × n are given by
k = − L u u − 1 ( L u + F u V x ) (51) k=-L_{uu}^{-1}(L_u+F_uV_x) \tag*{(51)} k = − L uu − 1 ( L u + F u V x ) (51)
K = − 1 2 L u u − 1 ( L u x + L x u T + 2 F u V X X ) (52) K=-\frac{1}{2}L_{uu}^{-1}(L_{ux}+L_{xu}^{\mathsf{T}}+2F_uV_{XX}) \tag*{(52)} K = − 2 1 L uu − 1 ( L ux + L xu T + 2 F u V X X ) (52)
In order to make the same comparison for the backward equations of the zeroth, first, and second-order value functional in fields, we first define the kernel functions k d : T × R n × R n → R k k_d:T\times\mathbb{R}^{n}\times\mathbb{R}^{n}\to\mathbb{R}^{k} k d : T × R n × R n → R k , k b : T × R n × R n → R l k_b:T\times\mathbb{R}^{n}\times\mathbb{R}^{n}\to\mathbb{R}^{l} k b : T × R n × R n → R l , K d : T × R n × R n × R n → T × R k × R n K_d:T\times\mathbb{R}^{n}\times\mathbb{R}^{n}\times\mathbb{R}^{n}\to T\times\mathbb{R}^{k}\times\mathbb{R}^{n} K d : T × R n × R n × R n → T × R k × R n , and K d : T × R n × R n × R n → T × R l × R n K_d:T\times\mathbb{R}^{n}\times\mathbb{R}^{n}\times\mathbb{R}^{n}\to T\times\mathbb{R}^{l}\times\mathbb{R}^{n} K d : T × R n × R n × R n → T × R l × R n , which are defined analogously to k k k and K K K in [30 ] , and given by
k d ( t , x , y ) = − L ˉ U d U d ( t , x , y ) ( L U d ( t , y ) + F U d T ( t , y , y ′ ) V X ( t , y ′ ) ) (53) k_d(t,x,y)=-\bar{L}_{U_dU_d}(t,x,y)\left(L_{U_d}(t,y)+F_{U_d}^{\mathsf{T}}(t,y,y')V_X(t,y')\right) \tag*{(53)} k d ( t , x , y ) = − L ˉ U d U d ( t , x , y ) ( L U d ( t , y ) + F U d T ( t , y , y ′ ) V X ( t , y ′ ) ) (53)
k b ( t , x , η ) = − L ˉ U b U b ( t , x , η ) ( L U b ( t , η ) − N U b T ( t , η , η ′ ) V X ( t , η ′ ) ) (54) k_b(t,x,\eta)=-\bar{L}_{U_bU_b}(t,x,\eta)\left(L_{U_b}(t,\eta)-N_{U_b}^{\mathsf{T}}(t,\eta,\eta')V_X(t,\eta')\right) \tag*{(54)} k b ( t , x , η ) = − L ˉ U b U b ( t , x , η ) ( L U b ( t , η ) − N U b T ( t , η , η ′ ) V X ( t , η ′ ) ) (54)
K d ( t , x , y , y ′ ) = − 1 2 L ˉ U d U d ( t , x , y ) ( L U d X ( t , y , y ′ ) + L X U d T ( t , y , y ′ ) + 2 F U d ( t , y , y ′ ′ ) V X X ( t , y ′ , y ′ ′ ) ) (55) K_d(t,x,y,y')=-\frac{1}{2}\bar{L}_{U_dU_d}(t,x,y)\left(L_{U_dX}(t,y,y')+L_{XU_d}^{\mathsf{T}}(t,y,y')+2F_{U_d}(t,y,y'')V_{XX}(t,y',y'')\right) \tag*{(55)} K d ( t , x , y , y ′ ) = − 2 1 L ˉ U d U d ( t , x , y ) ( L U d X ( t , y , y ′ ) + L X U d T ( t , y , y ′ ) + 2 F U d ( t , y , y ′′ ) V X X ( t , y ′ , y ′′ ) ) (55)
K b ( t , ξ , η , y ) = − 1 2 L ˉ U b U b ( t , x , η ) ( L U b X ( t , η , y ) + L X U b ( t , η , y ) − 2 N U b T ( t , η , η ′ ) V X X ( t , η ′ , y ) ) (56) K_b(t,\xi,\eta,y)=-\frac{1}{2}\bar{L}_{U_bU_b}(t,x,\eta)\left(L_{U_bX}(t,\eta,y)+L_{XU_b}(t,\eta,y)-2N_{U_b}^{\mathsf{T}}(t,\eta,\eta')V_{XX}(t,\eta',y)\right) \tag*{(56)} K b ( t , ξ , η , y ) = − 2 1 L ˉ U b U b ( t , x , η ) ( L U b X ( t , η , y ) + L X U b ( t , η , y ) − 2 N U b T ( t , η , η ′ ) V X X ( t , η ′ , y ) ) (56)
Thus eqs. (33) , (35) and (38) in fields representation take the form
− d d t V ( t , X ( t , x ) ) = L − ∫ S V X ( t , ξ ) T Δ N ( t , ξ ) d S ξ − 1 2 ∫ D ∫ D ∫ D k d T ( t , x , y ) L U d U d ( t , y , y ′ ′ ) k d ( t , y ′ ′ , y ′ ) d x d y d y ′ − 1 2 ∫ S ∫ S ∫ S k b T ( t , ξ , ξ ′ ) L U b U b ( t , ξ ′ , η ′ ) k b ( t , η ′ , η ) d S ξ d S ξ ′ d S η (57) \begin{aligned}
-\frac{\mathrm{d}}{\mathrm{d}t}V(t,X(t,x))={}&L-\int_S V_X(t,\xi)^{\mathsf{T}}\Delta N(t,\xi)\,\mathrm{d}S_\xi\\
&-\frac{1}{2}\int_D\int_D\int_D k_d^{\mathsf{T}}(t,x,y)L_{U_dU_d}(t,y,y'')k_d(t,y'',y')\,\mathrm{d}x\mathrm{d}y\mathrm{d}y'\\
&-\frac{1}{2}\int_S\int_S\int_S k_b^{\mathsf{T}}(t,\xi,\xi')L_{U_bU_b}(t,\xi',\eta')k_b(t,\eta',\eta)\,\mathrm{d}S_\xi\mathrm{d}S_{\xi'}\mathrm{d}S_\eta \tag*{(57)}
\end{aligned} − d t d V ( t , X ( t , x )) = L − ∫ S V X ( t , ξ ) T Δ N ( t , ξ ) d S ξ − 2 1 ∫ D ∫ D ∫ D k d T ( t , x , y ) L U d U d ( t , y , y ′′ ) k d ( t , y ′′ , y ′ ) d x d y d y ′ − 2 1 ∫ S ∫ S ∫ S k b T ( t , ξ , ξ ′ ) L U b U b ( t , ξ ′ , η ′ ) k b ( t , η ′ , η ) d S ξ d S ξ ′ d S η (57)
− d d t V X ( t , X ( t , x ) ) = L X + F X ∗ V X − ∫ D ∫ D ∫ D K d T ( t , x , y , y ′ ) L U d U d ( t , y ′ , z ) k d ( t , z , y ′ ′ ) d y d y ′ d y ′ ′ − ∫ S ∫ S ∫ S K b T ( t , x , ξ , η ) L U b U b ( t , η , φ ) k b ( t , φ , η ′ ) d S ξ d S η d S η ′ (58) -\frac{\mathrm{d}}{\mathrm{d}t}V_X(t,X(t,x))=L_X+F_X^*V_X-\int_D\int_D\int_D K_d^{\mathsf{T}}(t,x,y,y')L_{U_dU_d}(t,y',z)k_d(t,z,y'')\,\mathrm{d}y\,\mathrm{d}y'\,\mathrm{d}y''
-\int_S\int_S\int_S K_b^{\mathsf{T}}(t,x,\xi,\eta)L_{U_bU_b}(t,\eta,\varphi)k_b(t,\varphi,\eta')\,\mathrm{d}S_\xi\,\mathrm{d}S_\eta\,\mathrm{d}S'_\eta \tag*{(58)} − d t d V X ( t , X ( t , x )) = L X + F X ∗ V X − ∫ D ∫ D ∫ D K d T ( t , x , y , y ′ ) L U d U d ( t , y ′ , z ) k d ( t , z , y ′′ ) d y d y ′ d y ′′ − ∫ S ∫ S ∫ S K b T ( t , x , ξ , η ) L U b U b ( t , η , φ ) k b ( t , φ , η ′ ) d S ξ d S η d S η ′ (58)
− d d t V X X ( t , x , y ) = L X X ( t , x , y ) + F X ∗ ( t , x , y ′ ) V X X ( t , y ′ , y ) + [ F X ∗ ( t , x , y ′ ) V X X ( t , y ′ , y ) ] T − ∫ D ∫ D ∫ D K d T ( t , x , y , y ′ ) L U d U d ( t , y ′ , z ) K d ( t , z , y ′ ′ , y ′ ′ ′ ) d y ′ d y ′ ′ d y ′ ′ ′ − ∫ S ∫ S ∫ S K b T ( t , x , ξ , η ) L U b U b ( t , η , φ ) K b ( t , φ , η ′ , y ) d S ξ d S η d S η ′ (59) -\frac{\mathrm{d}}{\mathrm{d}t}V_{XX}(t,x,y)=L_{XX}(t,x,y)+F_X^*(t,x,y')V_{XX}(t,y',y)+\left[F_X^*(t,x,y')V_{XX}(t,y',y)\right]^{\mathsf{T}}
-\int_D\int_D\int_D K_d^{\mathsf{T}}(t,x,y,y')L_{U_dU_d}(t,y',z)K_d(t,z,y'',y''')\,\mathrm{d}y'\,\mathrm{d}y''\,\mathrm{d}y'''
-\int_S\int_S\int_S K_b^{\mathsf{T}}(t,x,\xi,\eta)L_{U_bU_b}(t,\eta,\varphi)K_b(t,\varphi,\eta',y)\,\mathrm{d}S_\xi\,\mathrm{d}S_\eta\,\mathrm{d}S'_\eta \tag*{(59)} − d t d V X X ( t , x , y ) = L X X ( t , x , y ) + F X ∗ ( t , x , y ′ ) V X X ( t , y ′ , y ) + [ F X ∗ ( t , x , y ′ ) V X X ( t , y ′ , y ) ] T − ∫ D ∫ D ∫ D K d T ( t , x , y , y ′ ) L U d U d ( t , y ′ , z ) K d ( t , z , y ′′ , y ′′′ ) d y ′ d y ′′ d y ′′′ − ∫ S ∫ S ∫ S K b T ( t , x , ξ , η ) L U b U b ( t , η , φ ) K b ( t , φ , η ′ , y ) d S ξ d S η d S η ′ (59)
where in each equation, one of the integrals cancels due to the inverse kernel property in eqs. (29) and (30) . Thus one can recover equations eqs. (48) to (50) by considering an ODE system that a) does not have a spatial state vector so the Volterra-Taylor expansion becomes a Taylor expansion and the volume integrals are equal to their integrand, b) does not have a spatial boundary so surface integrals over the boundary are zero, and c) has real-valued finite dimensional Jacobians defined on an orthonormal basis (with an orthonormal dual basis) so that the adjoint is equal to the transpose.
Thus, the DDP equations for PDEs are a generalization of the DDP equations for finite ODE systems. In the following section we demonstrate a similar generalization of the LQR solution for PDEs.
The Linear Quadratic Regulator of Fields
The linear quadratic regulator equations are obtained in [24 ] . Therein, they consider a linear PDE of the form
∂ t X ( t , x ) = A x ( t ) X ( t , x ) + B d ( t , x ) U d ( t , x ) , x ∈ D (60) \partial_tX(t,x)=A_x(t)X(t,x)+B_d(t,x)U_d(t,x),\qquad x\in D \tag*{(60)} ∂ t X ( t , x ) = A x ( t ) X ( t , x ) + B d ( t , x ) U d ( t , x ) , x ∈ D (60)
X ( t 0 , x ) = X 0 (61) X(t_0,x)=X_0 \tag*{(61)} X ( t 0 , x ) = X 0 (61)
where A x A_x A x is a linear differential operator that has standard form
A x ( t ) = ∑ i , j = 1 n A i j ( t , x ) ∂ 2 ∂ x i ∂ x j + ∑ i = 1 n B i ( t , x ) ∂ ∂ x i + C ( t , x ) . (62) A_x(t)=\sum_{i,j=1}^{n}A_{ij}(t,x)\frac{\partial^2}{\partial x_i\partial x_j}+\sum_{i=1}^{n}B_i(t,x)\frac{\partial}{\partial x_i}+C(t,x). \tag*{(62)} A x ( t ) = i , j = 1 ∑ n A ij ( t , x ) ∂ x i ∂ x j ∂ 2 + i = 1 ∑ n B i ( t , x ) ∂ x i ∂ + C ( t , x ) . (62)
The boundary condition is given by
B b ( t , ξ ) U b ( t , ξ ) = F ( t , ξ ) X ( t , ξ ) + ∑ j = 1 n A j ( t , ξ ) ∂ X ∂ x j , ξ ∈ S (63) B_b(t,\xi)U_b(t,\xi)=F(t,\xi)X(t,\xi)+\sum_{j=1}^{n}A_j(t,\xi)\frac{\partial X}{\partial x_j},\qquad\xi\in S \tag*{(63)} B b ( t , ξ ) U b ( t , ξ ) = F ( t , ξ ) X ( t , ξ ) + j = 1 ∑ n A j ( t , ξ ) ∂ x j ∂ X , ξ ∈ S (63)
where the operator A j A_j A j is given by
A j ( t , ξ ) = ∑ i = 1 n A i j ( t , ξ ) cos ( n ξ , x i ) , (64) A_j(t,\xi)=\sum_{i=1}^{n}A_{ij}(t,\xi)\cos(n_\xi,x_i), \tag*{(64)} A j ( t , ξ ) = i = 1 ∑ n A ij ( t , ξ ) cos ( n ξ , x i ) , (64)
where ( n ξ , x i ) (n_\xi,x_i) ( n ξ , x i ) is the angle between the outward normal n ξ n_\xi n ξ at a boundary point ξ ∈ S \xi\in S ξ ∈ S and the x i x_i x i -axis. The dynamics are equivalently described in Hilbert spaces as
d d t X ( t ) = A ( t ) X ( t ) + B d ( t ) U d ( t ) , X ∈ L 2 n ( D ) (65) \frac{\mathrm{d}}{\mathrm{d}t}X(t)=A(t)X(t)+B_d(t)U_d(t),\qquad X\in L_2^n(D) \tag*{(65)} d t d X ( t ) = A ( t ) X ( t ) + B d ( t ) U d ( t ) , X ∈ L 2 n ( D ) (65)
B b ( t ) U b ( t ) = F ( t ) X ( t ) + A ( t ) ⋅ ∇ x X ( t ) , X ∈ L 2 n ( S ) (66) B_b(t)U_b(t)=F(t)X(t)+A(t)\cdot\nabla_xX(t),\qquad X\in L_2^n(S) \tag*{(66)} B b ( t ) U b ( t ) = F ( t ) X ( t ) + A ( t ) ⋅ ∇ x X ( t ) , X ∈ L 2 n ( S ) (66)
Table 1. Corresponding Hilbert space operators between LQR of fields and DDP of fields.
which is a familiar control affine linear system form in Hilbert spaces. The optimization problem is formulated as
J ( t 0 , X 0 , U d , U b ) = ϕ ( X ( t f ) ) + ∫ t 0 t f L ( t , X ( t ) , U d ( t ) , U b ( t ) ) d t (67) J(t_0,X_0,U_d,U_b)=\phi\left(X(t_f)\right)+\int_{t_0}^{t_f}L\left(t,X(t),U_d(t),U_b(t)\right)\mathrm{d}t \tag*{(67)} J ( t 0 , X 0 , U d , U b ) = ϕ ( X ( t f ) ) + ∫ t 0 t f L ( t , X ( t ) , U d ( t ) , U b ( t ) ) d t (67)
where the running cost L L L has the form
L ( t , X ( t ) , U d ( t ) , U b ( t ) ) = 1 2 ∫ D ∫ D X ( t , x ) T Q ( t , x , y ) X ( t , y ) d x d y + 1 2 ∫ D ∫ D U d ( t , x ) T R d ( t , x , y ) U d ( t , y ) d x d y + 1 2 ∫ S ∫ S U b ( t , ξ ) T R b ( t , ξ , η ) U b ( t , η ) d S ξ d S η (68) \begin{aligned}
L\left(t,X(t),U_d(t),U_b(t)\right)={}&\frac{1}{2}\int_D\int_D X(t,x)^\mathsf{T}Q(t,x,y)X(t,y)\mathrm{d}x\mathrm{d}y+\frac{1}{2}\int_D\int_D U_d(t,x)^\mathsf{T}R_d(t,x,y)U_d(t,y)\mathrm{d}x\mathrm{d}y\\
&+\frac{1}{2}\int_S\int_S U_b(t,\xi)^\mathsf{T}R_b(t,\xi,\eta)U_b(t,\eta)\mathrm{d}S_\xi\mathrm{d}S_\eta \tag*{(68)}
\end{aligned} L ( t , X ( t ) , U d ( t ) , U b ( t ) ) = 2 1 ∫ D ∫ D X ( t , x ) T Q ( t , x , y ) X ( t , y ) d x d y + 2 1 ∫ D ∫ D U d ( t , x ) T R d ( t , x , y ) U d ( t , y ) d x d y + 2 1 ∫ S ∫ S U b ( t , ξ ) T R b ( t , ξ , η ) U b ( t , η ) d S ξ d S η (68)
where the kernels Q ≥ 0 Q \ge0 Q ≥ 0 , R d > 0 R_d > 0 R d > 0 and R b > 0 R_b > 0 R b > 0 are all assumed to be symmetric about all spatial axes. The resulting optimal distributed and boundary control equations are obtained after applying Green’s theorem to the HJB equation and performing Newton minimization. They are given by
U d ∗ ( t ) = − R d − 1 ( t ) B d ( t ) T P ( t ) X ( t ) (69) U_d^*(t)=-R_d^{-1}(t)B_d(t)^\mathsf{T}P(t)X(t) \tag*{(69)} U d ∗ ( t ) = − R d − 1 ( t ) B d ( t ) T P ( t ) X ( t ) (69)
U b ∗ ( t ) = − R b − 1 ( t ) B b ( t ) T P ( t ) X ( t ) (70) U_b^*(t)=-R_b^{-1}(t)B_b(t)^\mathsf{T}P(t)X(t) \tag*{(70)} U b ∗ ( t ) = − R b − 1 ( t ) B b ( t ) T P ( t ) X ( t ) (70)
which are equivalently written in fields representation as
U d ∗ ( t , x ) = − ∫ D ∫ D R ˉ d ( t , x , y ) B d T ( t , y ) P ( t , y , y ′ ) X ( t , y ′ ) d y d y ′ (71) U_d^*(t,x)=-\int_D\int_D \bar{R}_d(t,x,y)B_d^\mathsf{T}(t,y)P(t,y,y')X(t,y')\mathrm{d}y\mathrm{d}y' \tag*{(71)} U d ∗ ( t , x ) = − ∫ D ∫ D R ˉ d ( t , x , y ) B d T ( t , y ) P ( t , y , y ′ ) X ( t , y ′ ) d y d y ′ (71)
U b ∗ ( t , x ) = − ∫ S ∫ D R ˉ b ( t , ξ , η ) B b T ( t , η ) P ( t , η , y ) X ( t , y ) d y d S η (72) U_b^*(t,x)=-\int_S\int_D \bar{R}_b(t,\xi,\eta)B_b^\mathsf{T}(t,\eta)P(t,\eta,y)X(t,y)\mathrm{d}y\mathrm{d}S_\eta \tag*{(72)} U b ∗ ( t , x ) = − ∫ S ∫ D R ˉ b ( t , ξ , η ) B b T ( t , η ) P ( t , η , y ) X ( t , y ) d y d S η (72)
In order to make the generalization clear, we rewrite eqs. (69) and (70) in our notation using the correspondences listed in table 1
U d ∗ ( t ) = − L U d U d − 1 ( t ) F U d T ( t ) V X ( t , X ) (73) U_d^*(t)=-L_{U_dU_d}^{-1}(t)F_{U_d}^\mathsf{T}(t)V_X(t,X) \tag*{(73)} U d ∗ ( t ) = − L U d U d − 1 ( t ) F U d T ( t ) V X ( t , X ) (73)
U b ∗ ( t ) = L U b U b − 1 ( t ) N U b T ( t ) V X ( t , X ) (74) U_b^*(t)=L_{U_bU_b}^{-1}(t)N_{U_b}^\mathsf{T}(t)V_X(t,X) \tag*{(74)} U b ∗ ( t ) = L U b U b − 1 ( t ) N U b T ( t ) V X ( t , X ) (74)
and repeat eqs. (25) and (26) here for clarity
δ U d ∗ = − L U d U d − 1 ( L U d + F U d T V X ) − 1 2 L U d U d − 1 ( L U d X + L X U d T + 2 F U d T V X X ) δ X \delta U_d^*=-L_{U_dU_d}^{-1}\left(L_{U_d}+F_{U_d}^\mathsf{T}V_X\right)-\frac{1}{2}L_{U_dU_d}^{-1}\left(L_{U_dX}+L_{XU_d}^\mathsf{T}+2F_{U_d}^\mathsf{T}V_{XX}\right)\delta X δ U d ∗ = − L U d U d − 1 ( L U d + F U d T V X ) − 2 1 L U d U d − 1 ( L U d X + L X U d T + 2 F U d T V X X ) δ X
δ U b ∗ = − L U b U b − 1 ( L U b − N U b T V X ) − 1 2 L U b U b − 1 ( L U b X + L X U b T − 2 N U b T V X X ) δ X \delta U_b^*=-L_{U_bU_b}^{-1}\left(L_{U_b}-N_{U_b}^\mathsf{T}V_X\right)-\frac{1}{2}L_{U_bU_b}^{-1}\left(L_{U_bX}+L_{XU_b}^\mathsf{T}-2N_{U_b}^\mathsf{T}V_{XX}\right)\delta X δ U b ∗ = − L U b U b − 1 ( L U b − N U b T V X ) − 2 1 L U b U b − 1 ( L U b X + L X U b T − 2 N U b T V X X ) δ X
Thus, we can recover eqs. (77) and (78) by a) assuming the cost functional is a pure quadratic without cross terms so that the terms L U d L_{U_d} L U d , L U b L_{U_b} L U b , L X U d L_{XU_d} L X U d , L U d X L_{U_dX} L U d X , L X U b L_{XU_b} L X U b , and L U b X L_{U_bX} L U b X are null and b) using only gradient information of the value functional so that V X X V_{XX} V X X terms are ignored.
The resulting second-order backward value functional equation of Riccati type for LQR is given in fields representation by
∂ P ( t , x , y ) ∂ t = − A X ∗ ( t ) P ( t , x , y ) − [ A X ∗ ( t ) P ( t , x , y ) ] T − Q ( t , x , y ) + ∫ D ∫ D P ( t , x , x ′ ) B d ( t , x ′ ) R ˉ d ( t , x ′ , x ′ ′ ) B d T ( t , x ′ ′ ) P ( t , x ′ ′ , y ) d x ′ d x ′ ′ + ∫ S ∫ S P ( t , x , ξ ) B b ( t , ξ ) R ˉ b ( t , ξ , η ) B b T ( t , η ) P ( t , η , y ) d S ξ d S η . (75) \frac{\partial P(t,x,y)}{\partial t}=-A_X^{*}(t)P(t,x,y)-\left[A_X^{*}(t)P(t,x,y)\right]^{\mathsf{T}}-Q(t,x,y)
+\int_D\int_D P(t,x,x')B_d(t,x')\bar{R}_d(t,x',x'')B_d^{\mathsf{T}}(t,x'')P(t,x'',y)\,dx'\,dx''
+\int_S\int_S P(t,x,\xi)B_b(t,\xi)\bar{R}_b(t,\xi,\eta)B_b^{\mathsf{T}}(t,\eta)P(t,\eta,y)\,dS_\xi\,dS_\eta. \tag*{(75)} ∂ t ∂ P ( t , x , y ) = − A X ∗ ( t ) P ( t , x , y ) − [ A X ∗ ( t ) P ( t , x , y ) ] T − Q ( t , x , y ) + ∫ D ∫ D P ( t , x , x ′ ) B d ( t , x ′ ) R ˉ d ( t , x ′ , x ′′ ) B d T ( t , x ′′ ) P ( t , x ′′ , y ) d x ′ d x ′′ + ∫ S ∫ S P ( t , x , ξ ) B b ( t , ξ ) R ˉ b ( t , ξ , η ) B b T ( t , η ) P ( t , η , y ) d S ξ d S η . (75)
which is rewritten in the DDP notation by again applying the correspondences listed in table 1 as
∂ V X X ( t , x , y ) ∂ t = − F X ∗ ( t , x , y ) V X X ( t , x , y ) − [ F X ∗ ( t , x , y ) V X X ( t , x , y ) ] T − L X X ( t , x , y ) + ∫ D ∫ D V X X ( t , x , x ′ ) F U d ( t , x ′ ) L ˉ U d U d ( t , x ′ , x ′ ′ ) F U d T ( t , x ′ ′ ) V X X ( t , x ′ ′ , y ) d x ′ d x ′ ′ + ∫ S ∫ S V X X ( t , x , ξ ) F U b ( t , ξ ) L ˉ U b U b ( t , ξ , η ) F U b T ( t , η ) V X X ( t , η , y ) d S ξ d S η . (76) \frac{\partial V_{XX}(t,x,y)}{\partial t}=-F_X^{*}(t,x,y)V_{XX}(t,x,y)-\left[F_X^{*}(t,x,y)V_{XX}(t,x,y)\right]^{\mathsf{T}}-L_{XX}(t,x,y)
+\int_D\int_D V_{XX}(t,x,x')F_{U_d}(t,x')\bar{L}_{U_dU_d}(t,x',x'')F_{U_d}^{\mathsf{T}}(t,x'')V_{XX}(t,x'',y)\,dx'\,dx''
+\int_S\int_S V_{XX}(t,x,\xi)F_{U_b}(t,\xi)\bar{L}_{U_bU_b}(t,\xi,\eta)F_{U_b}^{\mathsf{T}}(t,\eta)V_{XX}(t,\eta,y)\,dS_\xi\,dS_\eta. \tag*{(76)} ∂ t ∂ V X X ( t , x , y ) = − F X ∗ ( t , x , y ) V X X ( t , x , y ) − [ F X ∗ ( t , x , y ) V X X ( t , x , y ) ] T − L X X ( t , x , y ) + ∫ D ∫ D V X X ( t , x , x ′ ) F U d ( t , x ′ ) L ˉ U d U d ( t , x ′ , x ′′ ) F U d T ( t , x ′′ ) V X X ( t , x ′′ , y ) d x ′ d x ′′ + ∫ S ∫ S V X X ( t , x , ξ ) F U b ( t , ξ ) L ˉ U b U b ( t , ξ , η ) F U b T ( t , η ) V X X ( t , η , y ) d S ξ d S η . (76)
eq. (80) is identical to the second order backward value functional of DDP of fields without cross terms in the running cost, given in eq. (42).
Thus we conclude that the equations of STDDP are a generalization of LQR of fields. This generalization is analogous to the similar generalization of LQR of ODE systems to DDP of ODE systems. Whereas LQR is the analytically optimal controller for linear systems, it cannot be applied directly to a nonlinear system, nor can it be applied directly to a linear system whose running cost functional is not purely quadratic. In contrast, the iterative approximate optimal control method provided by DDP of fields was constructed for such systems.
Continuous-Time Convergence Analysis
Global convergence of the discrete finite dimensional DDP algorithm defined for discrete ODE systems was first provided by Yakowitz and Rutherford [29 ] . Later the proof that discrete finite dimensional DDP converges quadratically in the number of iterations was proved independently by Pantoja [33 ] and Murray and Yakowitz [34 ] . This quadratic convergence proof relied on convergence of Newton’s method, but later an independent proof relying only on the dynamic programming principle was given by Liao and Shoemaker [28 ] . Through decades of application of the DDP algorithm, there have been numerous extensions of the proof of global convergence, for example for DDP on Lie groups in [35 ] and for DDP with generalized Polynomail Chaos expansions in [21 ] . However it appears to the best knowledge of the authors that most if not all proofs of global convergence are for DDP and its extensions in discrete time, and not in continuous time.
Typically, one determines provable convergence characteristics by investigating the behavior of the derivative
d J i ( t , X ( t ) , U ( t ) ) d i = d J i ( t , X ( t ) , U ( t ) ) d U t 0 : t f i d U t 0 : t f i d i (77) \frac{dJ^i(t,X(t),U(t))}{di}=\frac{dJ^i(t,X(t),U(t))}{dU_{t_0:t_f}^i}\frac{dU_{t_0:t_f}^i}{di} \tag*{(77)} d i d J i ( t , X ( t ) , U ( t )) = d U t 0 : t f i d J i ( t , X ( t ) , U ( t )) d i d U t 0 : t f i (77)
where, due to the decoupled nature of the distributed and boundary control updates, we have defined the Hilbert space control vector U ( t ) ∈ L 2 k + l ( D ˉ ) U(t)\in L_2^{k+l}(\bar{D}) U ( t ) ∈ L 2 k + l ( D ˉ ) as the direct product Hilbert space analog of the stacked distributed and boundary control vectors in fields representation U ( t , x ) = [ U d ( t , x ) , U b ( t , x ) ] T U(t,x)=[U_d(t,x),U_b(t,x)]^{\mathsf{T}} U ( t , x ) = [ U d ( t , x ) , U b ( t , x ) ] T . This notation simplifies our analysis significantly. We have also introduced the trajectory notation, where subscript t 0 : t f t_0:t_f t 0 : t f represents the entire trajectory in time of the associated variable, and used the superscript i i i for the STDDP iteration index. Similarly δ U t 0 : t f i \delta U^{i}_{t_0:t_f} δ U t 0 : t f i denotes the control update trajectory for control trajectory U t 0 : t f i U^{i}_{t_0:t_f} U t 0 : t f i . This trajectory notation defines a temporal Hilbert space over time-indexed spatial Hilbert spaces. Let L 2 n ( T ) L_2^n(T) L 2 n ( T ) denote the Hilbert space of L 2 n ( D ‾ ) L_2^n(\overline{D}) L 2 n ( D ) -vector functions square integrable over T T T with inner product
⟨ X 1 , t 0 : t f , X 2 , t 0 : t f ⟩ T = ∫ t 0 t f ⟨ X 1 ( s ) , X 2 ( s ) ⟩ d s . (78) \left\langle X_{1,t_0:t_f},X_{2,t_0:t_f}\right\rangle_T=\int_{t_0}^{t_f}\left\langle X_1(s),X_2(s)\right\rangle\,\mathrm{d}s. \tag*{(78)} ⟨ X 1 , t 0 : t f , X 2 , t 0 : t f ⟩ T = ∫ t 0 t f ⟨ X 1 ( s ) , X 2 ( s ) ⟩ d s . (78)
This allows us to write time integrals over trajectory variables as inner product tensor contractions, and treat continuous trajectories as objects in a similar way to the continuum of the PDE variables. We begin by stating the following lemma, assumption, and proposition that will be used in our analysis.
Lemma 8.1. Assume the cost functional has the form of eq. (6) and define the measurable backward recursive functional ψ ∈ L ( L 2 n ( D ‾ ) ) \psi\in\mathcal{L}(L_2^n(\overline{D})) ψ ∈ L ( L 2 n ( D )) for some ε > 0 \varepsilon>0 ε > 0 as
ψ ( t , X ( t ) , U ( t ) ) = ∫ t t + ε L X ( s , X ( s ) , U ( s ) ) d s + Φ ( t , s ) ψ ( t + ε , X ( t + ε ) , U ( t + ε ) ) (79) \psi(t,X(t),U(t))=\int_t^{t+\varepsilon}L_X(s,X(s),U(s))\,\mathrm{d}s+\Phi(t,s)\psi(t+\varepsilon,X(t+\varepsilon),U(t+\varepsilon)) \tag*{(79)} ψ ( t , X ( t ) , U ( t )) = ∫ t t + ε L X ( s , X ( s ) , U ( s )) d s + Φ ( t , s ) ψ ( t + ε , X ( t + ε ) , U ( t + ε )) (79) ψ ( t f , X ( t f ) , U ( t f ) ) = ϕ X ( t , X ( t ) ) (80) \psi(t_f,X(t_f),U(t_f))=\phi_X(t,X(t)) \tag*{(80)} ψ ( t f , X ( t f ) , U ( t f )) = ϕ X ( t , X ( t )) (80) where Φ ( ⋅ , ⋅ ) \Phi(\cdot,\cdot) Φ ( ⋅ , ⋅ ) is a contractive linear semigroup generated by the approximate variation dynamics in eqs. (41) and (42) , and is assumed to be positive definite almost everywhere. Then the cost functional satisfies
d J i ( t , X ( t ) , U ( t ) ) d U t 0 : t f i = ⟨ L U , t 0 : t f , 1 t 0 : t f ⟩ T + ⟨ F U , t 0 : t f , ψ t 0 : t f ⟩ T (81) \frac{\mathrm{d}J^i(t,X(t),U(t))}{\mathrm{d}U^i_{t_0:t_f}}=\left\langle L_{U,t_0:t_f},\mathbf{1}_{t_0:t_f}\right\rangle_T+\left\langle F_{U,t_0:t_f},\psi_{t_0:t_f}\right\rangle_T \tag*{(81)} d U t 0 : t f i d J i ( t , X ( t ) , U ( t )) = ⟨ L U , t 0 : t f , 1 t 0 : t f ⟩ T + ⟨ F U , t 0 : t f , ψ t 0 : t f ⟩ T (81) where 1 t 0 : t f \mathbf{1}_{t_0:t_f} 1 t 0 : t f is the trajectory of ones.
Proof. The proof is in the Supplementary Material, section S1
Assumption 6. The search space of control trajectories U ⊇ U t 0 : t f i \mathcal{U}\supseteq U^i_{t_0:t_f} U ⊇ U t 0 : t f i is compact.
Our analysis is simplified by the Q Q Q functional notation defined as follows
Q U U = L U U Q U = L U + F U T V X Q U X = L U X + F U T V X X Q X = L X + F X T V X Q X X = L X X + F X ∗ V X X + [ F X ∗ V X X ] T \begin{aligned}
Q_{UU}&=L_{UU}\\
Q_U&=L_U+F_U^{\mathsf{T}}V_X\\
Q_{UX}&=L_{UX}+F_U^{\mathsf{T}}V_{XX}\\
Q_X&=L_X+F_X^{\mathsf{T}}V_X\\
Q_{XX}&=L_{XX}+F_X^*V_{XX}+\left[F_X^*V_{XX}\right]^{\mathsf{T}}
\end{aligned} Q U U Q U Q U X Q X Q X X = L U U = L U + F U T V X = L U X + F U T V X X = L X + F X T V X = L X X + F X ∗ V X X + [ F X ∗ V X X ] T
Proposition 8.1. Let the PDE D ( t ) ∈ L 2 n ( D ‾ ) D(t)\in L_2^n(\overline{D}) D ( t ) ∈ L 2 n ( D ) have dynamics
d d t D ( t ) = − F X T D ( t ) + Q U X T Q U U − 1 Q U (82) \frac{\mathrm{d}}{\mathrm{d}t}D(t)=-F_X^{\mathsf{T}}D(t)+Q_{UX}^{\mathsf{T}}Q_{UU}^{-1}Q_U \tag*{(82)} d t d D ( t ) = − F X T D ( t ) + Q U X T Q U U − 1 Q U (82) D ( T ) = 0 (83) D(T)=0 \tag*{(83)} D ( T ) = 0 (83) Then D ( t ) D(t) D ( t ) has weak backwards solutions defined in the Hadamard sense, and given by
D ( t ) = ∫ T t Φ T ( t , τ ) Q U X T ( τ ) Q U U − 1 Q U ( τ ) d τ (84) D(t)=\int_T^t\Phi^{\mathsf{T}}(t,\tau)Q_{UX}^{\mathsf{T}}(\tau)Q_{UU}^{-1}Q_U(\tau)\,\mathrm{d}\tau \tag*{(84)} D ( t ) = ∫ T t Φ T ( t , τ ) Q U X T ( τ ) Q U U − 1 Q U ( τ ) d τ (84)
Proof. The existence of weak solutions is given by the assumption that solutions to ψ \psi ψ and V X V_X V X exist. The rest of the proof is immediate given that the dynamics are of semilinear form and have a zero terminal condition. □ \square □
Theorem 8.1. Consider the continuous-time optimal control problem in eq. (7) subject to the dynamics in eqs. (4) and (7) with cost functional J J J having no cross terms for simplicity. Let U ‾ t 0 : t f ∈ L 2 k + l ( T ) \overline{U}_{t_0:t_f} \in\mathcal{L}^{k+l}_2(T) U t 0 : t f ∈ L 2 k + l ( T ) be a nominal control trajectory and let δ U t 0 : t f ∈ L 2 k + l ( T ) \delta U_{t_0:t_f} \in\mathcal{L}^{k+l}_2(T) δ U t 0 : t f ∈ L 2 k + l ( T ) be the trajectory of control updates from eqs. (25) and (26) . Then the following holds:
d J i ( t , X ( t ) , U ( t ) ) d i = − γ ⟨ Q U , t 0 : t f , M t 0 : t f Q U , t 0 : t f ⟩ T + O ( γ 2 ) (85) \frac{\mathrm{d}J^i(t,X(t),U(t))}{\mathrm{d}i}=-\gamma\left\langle Q_{U,t_0:t_f},M_{t_0:t_f}Q_{U,t_0:t_f}\right\rangle_T+O(\gamma^2) \tag*{(85)} d i d J i ( t , X ( t ) , U ( t )) = − γ ⟨ Q U , t 0 : t f , M t 0 : t f Q U , t 0 : t f ⟩ T + O ( γ 2 ) (85) where the trajectory operator M t 0 : t f ∈ L ( L 2 l + k ( T ) , L 2 l + k ( T ) ) M_{t_0:t_f}\in\mathcal{L}\left(\mathcal{L}^{l+k}_2(T),\mathcal{L}^{l+k}_2(T)\right) M t 0 : t f ∈ L ( L 2 l + k ( T ) , L 2 l + k ( T ) ) has a positive definite kernel ∀ t ∈ T \forall t\in T ∀ t ∈ T .
Proof. Observe that due to the iterative updates in eqs. (43) and (44) , we have
d U ‾ t 0 : t f d i = γ δ U t 0 : t f . (86) \frac{\mathrm{d}\overline{U}_{t_0:t_f}}{\mathrm{d}i}=\gamma\delta U_{t_0:t_f}. \tag*{(86)} d i d U t 0 : t f = γ δ U t 0 : t f . (86)
Also at our disposal is the identity
L U , t 0 : t f + F U , t 0 : t f T ψ t 0 : t f = Q U , t 0 : t f − F U , t 0 : t f T ( V X , t 0 : t f − ψ t 0 : t f ) . (87) L_{U,t_0:t_f}+F^\mathsf{T}_{U,t_0:t_f}\psi_{t_0:t_f}=Q_{U,t_0:t_f}-F^\mathsf{T}_{U,t_0:t_f}(V_{X,t_0:t_f}-\psi_{t_0:t_f}). \tag*{(87)} L U , t 0 : t f + F U , t 0 : t f T ψ t 0 : t f = Q U , t 0 : t f − F U , t 0 : t f T ( V X , t 0 : t f − ψ t 0 : t f ) . (87)
Plugging eq. (87) into eq. (81) yields
d J i ( t , X ( t ) , U ( t ) ) d i = − γ ⟨ Q U , t 0 : t f , Q U U , t 0 : t f − 1 Q U , t 0 : t f ⟩ T + γ ⟨ F U , t 0 : t f T ( V X , t 0 : t f − ψ t 0 : t f ) , Q U U , t 0 : t f − 1 Q U , t 0 : t f ⟩ T − γ ⟨ Q U , t 0 : t f , Q U U , t 0 : t f − 1 Q U X , t 0 : t f δ X t 0 : t f ⟩ T + γ ⟨ F U , t 0 : t f T ( V X , t 0 : t f − ψ t 0 : t f ) , Q U U , t 0 : t f − 1 Q U X , t 0 : t f δ X t 0 : t f ⟩ T (88) \begin{aligned}
\frac{\mathrm{d}J^i(t,X(t),U(t))}{\mathrm{d}i}
={}&-\gamma\left\langle Q_{U,t_0:t_f},Q^{-1}_{UU,t_0:t_f}Q_{U,t_0:t_f}\right\rangle_T\\
&+\gamma\left\langle F^\mathsf{T}_{U,t_0:t_f}(V_{X,t_0:t_f}-\psi_{t_0:t_f}),Q^{-1}_{UU,t_0:t_f}Q_{U,t_0:t_f}\right\rangle_T\\
&-\gamma\left\langle Q_{U,t_0:t_f},Q^{-1}_{UU,t_0:t_f}Q_{UX,t_0:t_f}\delta X_{t_0:t_f}\right\rangle_T\\
&+\gamma\left\langle F^\mathsf{T}_{U,t_0:t_f}(V_{X,t_0:t_f}-\psi_{t_0:t_f}),Q^{-1}_{UU,t_0:t_f}Q_{UX,t_0:t_f}\delta X_{t_0:t_f}\right\rangle_T \tag*{(88)}
\end{aligned} d i d J i ( t , X ( t ) , U ( t )) = − γ ⟨ Q U , t 0 : t f , Q U U , t 0 : t f − 1 Q U , t 0 : t f ⟩ T + γ ⟨ F U , t 0 : t f T ( V X , t 0 : t f − ψ t 0 : t f ) , Q U U , t 0 : t f − 1 Q U , t 0 : t f ⟩ T − γ ⟨ Q U , t 0 : t f , Q U U , t 0 : t f − 1 Q U X , t 0 : t f δ X t 0 : t f ⟩ T + γ ⟨ F U , t 0 : t f T ( V X , t 0 : t f − ψ t 0 : t f ) , Q U U , t 0 : t f − 1 Q U X , t 0 : t f δ X t 0 : t f ⟩ T (88)
Note that the total variation δ X ∈ L 2 n ( D ‾ ) \delta X\in\mathcal{L}^n_2(\overline{D}) δ X ∈ L 2 n ( D ) , with dynamics given in semilinear form by eqs. (41) and (42) , has a solution given by
δ X ( t ) = Φ ( t , t 0 ) δ X 0 + ∫ 0 t Φ ( t , s ) F U ( s , ⋅ ) δ U ( s ) d s (89) \delta X(t)=\Phi(t,t_0)\delta X_0+\int_0^t\Phi(t,s)F_U(s,\cdot)\delta U(s)\,\mathrm{d}s \tag*{(89)} δ X ( t ) = Φ ( t , t 0 ) δ X 0 + ∫ 0 t Φ ( t , s ) F U ( s , ⋅ ) δ U ( s ) d s (89)
Thus, since δ X 0 = 0 \delta X_0=0 δ X 0 = 0 , and δ U ( t ) = O ( γ ) \delta U(t)=O(\gamma) δ U ( t ) = O ( γ ) , it follows that δ X = O ( γ ) \delta X=O(\gamma) δ X = O ( γ ) , so we have
d J i ( t , X ( t ) , U ( t ) ) d i = − γ ⟨ Q U , t 0 : t f , Q U U , t 0 : t f − 1 Q U , t 0 : t f ⟩ T + γ ⟨ F U , t 0 : t f T ( V X , t 0 : t f − ψ t 0 : t f ) , Q U U , t 0 : t f − 1 Q U , t 0 : t f ⟩ T + O ( γ 2 ) (90) \begin{aligned}
\frac{\mathrm{d}J^i(t,X(t),U(t))}{\mathrm{d}i}
={}&-\gamma\left\langle Q_{U,t_0:t_f},Q^{-1}_{UU,t_0:t_f}Q_{U,t_0:t_f}\right\rangle_T\\
&+\gamma\left\langle F^\mathsf{T}_{U,t_0:t_f}(V_{X,t_0:t_f}-\psi_{t_0:t_f}),Q^{-1}_{UU,t_0:t_f}Q_{U,t_0:t_f}\right\rangle_T+O(\gamma^2) \tag*{(90)}
\end{aligned} d i d J i ( t , X ( t ) , U ( t )) = − γ ⟨ Q U , t 0 : t f , Q U U , t 0 : t f − 1 Q U , t 0 : t f ⟩ T + γ ⟨ F U , t 0 : t f T ( V X , t 0 : t f − ψ t 0 : t f ) , Q U U , t 0 : t f − 1 Q U , t 0 : t f ⟩ T + O ( γ 2 ) (90)
Next, notice that D ( t ) = V X ( t ) − ψ ( t ) D(t)=V_X(t)-\psi(t) D ( t ) = V X ( t ) − ψ ( t ) has dynamics of the form of eq. (82) , with an equivalent terminal condition. Thus, plugging in eq. (84) into eq. (90) and reducing yields
d J i ( t , X ( t ) , U ( t ) ) d i = − γ ⟨ Q U , t 0 : t f , Q U U , t 0 : t f − 1 Q U , t 0 : t f ⟩ T − γ ⟨ Q U , t 0 : t f , A 1 , t 0 : t f Q U , t 0 : t f ⟩ T + O ( γ 2 ) (91) \frac{\mathrm{d}J^i(t,X(t),U(t))}{\mathrm{d}i}=-\gamma\left\langle Q_{U,t_0:t_f},Q^{-1}_{UU,t_0:t_f}Q_{U,t_0:t_f}\right\rangle_T-\gamma\left\langle Q_{U,t_0:t_f},A_{1,t_0:t_f}Q_{U,t_0:t_f}\right\rangle_T+O(\gamma^2) \tag*{(91)} d i d J i ( t , X ( t ) , U ( t )) = − γ ⟨ Q U , t 0 : t f , Q U U , t 0 : t f − 1 Q U , t 0 : t f ⟩ T − γ ⟨ Q U , t 0 : t f , A 1 , t 0 : t f Q U , t 0 : t f ⟩ T + O ( γ 2 ) (91)
where A 1 , t 0 : t f ∈ L ( L 2 k + l ( T ) , L 2 k + l ( T ) ) A_{1,t_0:t_f} \in\mathcal{L}(L_2^{k+l}(T),L_2^{k+l}(T)) A 1 , t 0 : t f ∈ L ( L 2 k + l ( T ) , L 2 k + l ( T )) has a positive definite kernel ∀ t ∈ T \forall t \in T ∀ t ∈ T due to the positive definiteness of Φ ( ⋅ , ⋅ ) \Phi(\cdot,\cdot) Φ ( ⋅ , ⋅ ) by definition, and the positive definiteness of V X X V_{XX} V X X by assumption. Thus, due to the positive definiteness of the kernels of L U d U d ( t ) L_{U_dU_d}(t) L U d U d ( t ) and L U b U b ( t ) L_{U_bU_b}(t) L U b U b ( t ) by assumption, one can form M ∈ L ( L 2 k + l ( D ˉ ) , L 2 k + l ( D ˉ ) ) M \in\mathcal{L}(L_2^{k+l}(\bar{D}),L_2^{k+l}(\bar{D})) M ∈ L ( L 2 k + l ( D ˉ ) , L 2 k + l ( D ˉ )) with positive definite kernel ∀ t ∈ T \forall t \in T ∀ t ∈ T such that
d J i ( t , X ( t ) , U ( t ) ) d i = − γ ⟨ Q U , t 0 : t f , M t 0 : t f Q U , t 0 : t f ⟩ T + O ( γ 2 ) (92) \frac{\mathrm{d}J^i(t,X(t),U(t))}{\mathrm{d}i}
= -\gamma\left\langle Q_{U,t_0:t_f},M_{t_0:t_f}Q_{U,t_0:t_f}\right\rangle_T + O(\gamma^2) \tag*{(92)} d i d J i ( t , X ( t ) , U ( t )) = − γ ⟨ Q U , t 0 : t f , M t 0 : t f Q U , t 0 : t f ⟩ T + O ( γ 2 ) (92)
which concludes the proof
Corollary 8.1. Suppose assumption 6 holds. Then the iterative eqs. (4) , (7), (25) , (26) , (33) to (40) , (43) and (44) will converge to a stationary solution of eq. (7) .
Proof. The proof is in the Supplementary Material, § 15
STDDP Algorithm
The resulting STDDP algorithm can be applied for control of any nonlinear forward spatio-temporal PDE system satisfying the stated assumptions. It is an iterative forward-backward approach, wherein each iteration forward propagates the dynamics, backward propagates the value functional and its derivatives, and updates the control based on approximate variation dynamics. The resulting procedure is described in greater detail in algorithm 1.
Note that algorithm 1 has a forward process, a backward process, and another forward process. While this is a simpler algorithmic exposition, the runtime performance can be improved by simply combining the two forward time loops. While the numerical experiments in this manuscript were performed with a fixed learning rate for demonstration purposes, it can be numerically advantageous to apply line search methods to adapt the learn rate. Some such methods are described in [36 ] and [15 ] , and typically evaluate the best learning rate based on the best improvement in the cost functional. However, since the value functional typically encodes problem information beyond the cost metric, one may also evaluate learning rate based on improvements in the value functional.
The inputs of the STDDP algorithm can change depending on the specific problem but in most cases contain time interval (T T T ), number of iterations (K K K ), initial state (X 0 X_0 X 0 ), time discretization (Δ t \Delta t Δ t ), distributed control learn rate (γ d \gamma_d γ d ), and boundary control learn rate (γ b \gamma_b γ b ). One may also include a number of rollouts (R R R ) for a parallelized line search. Instead of a fixed number of iterations, one may also check for convergence using relative or absolute convergence criteria in either the cost functional or the value functional [15 ] .
Forward & Backward PDE Discretization Methods
In order to implement the forward spatio-temporal system dynamics in eqs. (4) and (7) and the backward value functional system in eqs. (33) to (40) on a digital computer, these forward and backward PDEs must be spatially and temporally discretized.
Nonlinear PDEs in the Eulerian formalism are often spatially discretized using either finite difference methods, Galerkin methods, or finite element methods. In this work we apply a spatial central finite difference discretization, which yields a fixed 1D grid of length a a a , with J J J elements. We note that through the above derivation, any discretization can be used in place of the central difference.
While there are numerous works describing temporal discretization methods for a multitude of forward PDEs, there are relatively few that describe temporal discretization schemes for backward PDEs of Riccati
Algorithm 1 STDDP
Function: ( U d ∗ , U b ∗ ) = S T D D P ( T , K , X 0 , U ˉ d , U ˉ b , Δ t , γ d , γ b ) (U_d^{*},U_b^{*}) = \mathit{STDDP}(T,K,X_0,\bar{U}_d,\bar{U}_b,\Delta t,\gamma_d,\gamma_b) ( U d ∗ , U b ∗ ) = STDDP ( T , K , X 0 , U ˉ d , U ˉ b , Δ t , γ d , γ b )
for k = 1 k = 1 k = 1 to K K K do
Forward propagate PDE dynamics in eq. (4)
Evaluate running cost L L L and its partial derivatives
Evaluate terminal cost ϕ \phi ϕ and its partial derivatives
Backward propagate value functional via eqs. (33) , (35) and (38)
Forward propagate approximate variation dynamics via eqs. (41) and (42)
Compute updates δ U d k \delta U_d^k δ U d k and δ U b k \delta U_b^k δ U b k via eqs. (25) and (26)
Update control U d k + 1 , U b k + 1 U_d^{k+1}, U_b^{k+1} U d k + 1 , U b k + 1 via eqs. (43) and (44)
end for
type. In finite dimensions, these are typically referred to as Riccati Differential Equations (RDEs), and their discretization presents several difficulties which stem from a matrix-valued variable that cannot be analytically isolated without using a Kronecker scheme. Furthermore, RDEs are known to be quite stiff in many contexts due to a fast transient response [37 ] .
The most straightforward method is the explicit time Euler discretization method, which has a fast implementation, yet is sensitive to discretization step size for stiff dynamics. This sensitivity can be reduced by applying Runge-Kutta time-integration techniques, however one must either super-sample the dynamics or apply an equivalent Runge-Kutta integration for the dynamics and value functionals.
Semi-implicit time discretization and implicit time discretization are well known to handle stiff dynamics, yet require isolation of the value functional. This in turn yields an update with a very large Kronecker sum matrix inversion. To elucidate, consider the discretized 1D Hilbert space representation of eq. (38) , where F X ∗ = F X T F_X^{*}=F_X^{\mathsf{T}} F X ∗ = F X T , given by
− d d t V X X ( t ) = L X X + 1 Δ x F X T V X X ( t ) + 1 Δ x V X X ( t ) F X − 1 Δ x 2 V X X ( t ) F U d L U d U d − 1 F U d T V X X ( t ) − 1 Δ x 2 V X X ( t ) N U b L U b U b − 1 N U b T V X X ( t ) . (93) -\frac{\mathrm{d}}{\mathrm{d}t}V_{XX}(t)=L_{XX}+\frac{1}{\Delta x}F_X^{\mathsf{T}}V_{XX}(t)+\frac{1}{\Delta x}V_{XX}(t)F_X-\frac{1}{\Delta x^2}V_{XX}(t)F_{U_d}L_{U_dU_d}^{-1}F_{U_d}^{\mathsf{T}}V_{XX}(t)-\frac{1}{\Delta x^2}V_{XX}(t)N_{U_b}L_{U_bU_b}^{-1}N_{U_b}^{\mathsf{T}}V_{XX}(t). \tag*{(93)} − d t d V X X ( t ) = L X X + Δ x 1 F X T V X X ( t ) + Δ x 1 V X X ( t ) F X − Δ x 2 1 V X X ( t ) F U d L U d U d − 1 F U d T V X X ( t ) − Δ x 2 1 V X X ( t ) N U b L U b U b − 1 N U b T V X X ( t ) . (93)
Clearly, the desired variable V X X V_{XX} V X X cannot be completely isolated in this form. However, one can equivalently write a vector form by application of the vec operator
− vec ( d d t V X X ( t ) ) = vec ( L X X ) + 1 Δ x F X T ⊕ F X T vec ( V X X ( t ) ) − 1 Δ x 2 vec ( V X X ( t ) F U d L U d U d − 1 F U d T V X X ( t ) ) − 1 Δ x 2 vec ( V X X ( t ) N U b L U b U b − 1 N U b T V X X ( t ) ) . (94) -\operatorname{vec}\left(\frac{\mathrm{d}}{\mathrm{d}t}V_{XX}(t)\right)=\operatorname{vec}(L_{XX})+\frac{1}{\Delta x}F_X^{\mathsf{T}}\mathbin{\oplus}F_X^{\mathsf{T}}\operatorname{vec}\left(V_{XX}(t)\right)-\frac{1}{\Delta x^2}\operatorname{vec}\left(V_{XX}(t)F_{U_d}L_{U_dU_d}^{-1}F_{U_d}^{\mathsf{T}}V_{XX}(t)\right)-\frac{1}{\Delta x^2}\operatorname{vec}\left(V_{XX}(t)N_{U_b}L_{U_bU_b}^{-1}N_{U_b}^{\mathsf{T}}V_{XX}(t)\right). \tag*{(94)} − vec ( d t d V X X ( t ) ) = vec ( L X X ) + Δ x 1 F X T ⊕ F X T vec ( V X X ( t ) ) − Δ x 2 1 vec ( V X X ( t ) F U d L U d U d − 1 F U d T V X X ( t ) ) − Δ x 2 1 vec ( V X X ( t ) N U b L U b U b − 1 N U b T V X X ( t ) ) . (94)
Semi-implicit time discretization schemes typically evaluate terms that are linear in V X X ( t ) V_{XX}(t) V X X ( t ) at the current time step and non-linear terms in V X X ( t ) V_{XX}(t) V X X ( t ) at the next time step [1 ] , which is the previous time step in the case of backward PDEs. The resulting semi-implicit update is given by
vec ( V X X ( t k − 1 ) ) = [ I − F X T ⊗ F X T Δ t ] − 1 [ vec ( V X X ( t k ) ) + vec ( L X X ) Δ t − vec ( V X X ( t k ) F U d L U d U d − 1 F U d T V X X ( t k ) ) Δ t ] . (95) \operatorname{vec}\left(V_{XX}(t_{k-1})\right)=\left[I-F_X^{\mathsf{T}}\otimes F_X^{\mathsf{T}}\Delta t\right]^{-1}\left[\operatorname{vec}\left(V_{XX}(t_k)\right)+\operatorname{vec}(L_{XX})\Delta t-\operatorname{vec}\left(V_{XX}(t_k)F_{U_d}L_{U_dU_d}^{-1}F_{U_d}^{\mathsf{T}}V_{XX}(t_k)\right)\Delta t\right]. \tag*{(95)} vec ( V X X ( t k − 1 ) ) = [ I − F X T ⊗ F X T Δ t ] − 1 [ vec ( V X X ( t k ) ) + vec ( L X X ) Δ t − vec ( V X X ( t k ) F U d L U d U d − 1 F U d T V X X ( t k ) ) Δ t ] . (95)
The resulting update equation is less sensitive to time discretization step size Δ t \Delta t Δ t , however it requires the inversion of a large matrix of size J 2 × J 2 J^2 \times J^2 J 2 × J 2 for each time step of each iteration, where J J J is the spatial discretization size of the 1D PDE. A key observation is that the matrix M : = I − F X T ⊗ F X T Δ t M := I - F_X^{\mathsf{T}} \otimes F_X^{\mathsf{T}} \Delta t M := I − F X T ⊗ F X T Δ t typically only has as many diagonals as the order of the spatial discretization, and is zeros elsewhere except for the boundary conditions, thus it is a sparse matrix. For example, in the case of a second order spatial central difference discretization of the Burgers equation with Homogeneous Dirichlet boundary conditions, M M M is tridiagonal. Thus the inverse can be efficiently computed with sparse linear equation solvers such as SuperLU [38 ] .
In [37 ] , the authors describe so called D-methods, which reduce computational complexity inherent to semi-implicit methods by applying explicit Euler discretization to some subset of the variables, and applies semi-implicit discretization to the rest. This could dramatically reduce complexity; if J e ≤ J J_e \leq J J e ≤ J is the number of points treated with explicit discretization, then the resulting semi-implicit inverse is of size ( J − J e ) 2 × ( J − J e ) 2 (J-J_e)^2 \times(J-J_e)^2 ( J − J e ) 2 × ( J − J e ) 2 . This may have dramatic benefit for ODEs systems where one may have slower and faster channels, However it is not clear how to select grid elements for the associated D-method for Riccati PDEs.
Simulated Experiments
We applied the STDDP algorithm to two simulated PDE experiments to optimally control the system to a prescribed desired behavior. Each experiment used less than 32 GB RAM, and was run on a desktop computer with an Intel Xeon 12-core CPU with a NVIDIA GeForce GTX 980 GPU. The computations did not utilize GPU parallelization, however many operations, such as cost and partial derivative computations, can be parallelized for greater computational efficiency.
The simulated experiments involve reaching tasks, where the PDE is initialized at a zero initial condition over the spatial region, and must reach certain field values at prescribed regions of the spatial domain. As discussed in the previous section, each PDE was spatially discretized by a spatial central difference discretization, and an explicit-time Euler discretization. The first and second derivative of the value functional were spatially discretized on the same spatial central difference grid as the forward dynamical PDE, and all three backward equations were temporally discretized with an explicit Euler discretization. Regularization was added to the second derivative of the value functional in order to aid in numerical stability.
Each experiment considered a pure quadratic cost functional of the form
J ( t , h ( t , x ) , U d ( t ) , U b ( t ) ) : = ⟨ h ( t f , x ) − h d e s ( t f , x ) , Q f ( h ( t f , x ) − h d e s ( t f , x ) ) ⟩ D d e s + ∫ t 0 t f ( ⟨ h ( t , x ) − h d e s ( t , x ) , Q ( h ( t , x ) − h d e s ( t , x ) ) ⟩ D d e s + ⟨ U d ( t , x ) , R d U d ( t , x ) ⟩ S + ⟨ U b ( t , x ) , R b U b ( t , x ) ⟩ S ) d t , (96) \begin{aligned}
J(t,h(t,x),U_d(t),U_b(t)) := {} & \left\langle h(t_f,x)-h_{\mathrm{des}}(t_f,x),Q_f\left(h(t_f,x)-h_{\mathrm{des}}(t_f,x)\right)\right\rangle_{D_{\mathrm{des}}} \\
&+ \int_{t_0}^{t_f} \left(\left\langle h(t,x)-h_{\mathrm{des}}(t,x),Q\left(h(t,x)-h_{\mathrm{des}}(t,x)\right)\right\rangle_{D_{\mathrm{des}}} \right. \\
&\qquad\qquad\left. + \left\langle U_d(t,x),R_d U_d(t,x)\right\rangle_S + \left\langle U_b(t,x),R_b U_b(t,x)\right\rangle_S \right) \, \mathrm{d}t , \tag*{(96)}
\end{aligned} J ( t , h ( t , x ) , U d ( t ) , U b ( t )) := ⟨ h ( t f , x ) − h des ( t f , x ) , Q f ( h ( t f , x ) − h des ( t f , x ) ) ⟩ D des + ∫ t 0 t f ( ⟨ h ( t , x ) − h des ( t , x ) , Q ( h ( t , x ) − h des ( t , x ) ) ⟩ D des + ⟨ U d ( t , x ) , R d U d ( t , x ) ⟩ S + ⟨ U b ( t , x ) , R b U b ( t , x ) ⟩ S ) d t , (96)
where the inner product ⟨ ⋅ , ⋅ ⟩ D d e s \langle\cdot,\cdot\rangle_{D_{\mathrm{des}}} ⟨ ⋅ , ⋅ ⟩ D des is defined on the desired subregion D d e s ⊆ D D_{\mathrm{des}} \subseteq D D des ⊆ D .
The first experiment was a temperature reaching task on the 1D Heat equation with homogeneous Dirichlet boundary conditions, given in fields representation by
∂ t h ( t , x ) = ε ∂ x x h ( t , x ) + m ( x ) T U d ( t , x ) , h ( t , 0 ) = h ( t , a ) = 0 , h ( 0 , x ) = h 0 ( x ) , (97) \begin{aligned}
\partial_t h(t,x) &= \varepsilon\partial_{xx}h(t,x) + \mathbf{m}(x)^{\mathsf{T}}U_d(t,x), \\
h(t,0) &= h(t,a) = 0, \\
h(0,x) &= h_0(x), \tag*{(97)}
\end{aligned} ∂ t h ( t , x ) h ( t , 0 ) h ( 0 , x ) = ε ∂ xx h ( t , x ) + m ( x ) T U d ( t , x ) , = h ( t , a ) = 0 , = h 0 ( x ) , (97)
where ε \varepsilon ε is the thermal diffusivity parameter. The heat equation is a pure diffusion equation, and validates the approach’s ability to achieve high quality distributed control solutions in the linear PDEs regime. The
STDDP algorithm was run until convergence, and the results of which are depicted in Figure 1 . Starting from a zero initial condition, the PDE was tasked with raising the temperature to T = 1.0 T = 1.0 T = 1.0 at the outer regions, and raising the temperature to T = 0.5 T = 0.5 T = 0.5 at the central region.
Figure 1. Heat Equation Temperature Reaching Task. (left) controlled contour plot where color represents temperature, (right) final time snapshot of the uncontrolled and controlled systems, (bottom) convergence plot of the heat equation temperature reaching task on a log-log scale, where the value integral depicted in red is the time integral of the value functional.
The system was temporally discretized into 1200 time steps and spatially discretized into 64 grid points. The typical convergence behavior for the STDDP algorithm applied to the heat equation is depicted in the bottom subfigure of Figure 1 . In this case, the weight values were R d = 0.4 R_d = 0.4 R d = 0.4 , Q = 300 Q = 300 Q = 300 , and Q f = 300 Q_f = 300 Q f = 300 . Depicted is a log-log plot of the cost functional J t , X ( t ) J_{t,X(t)} J t , X ( t ) , its state cost functional and control cost functional components, and the time integral of the value functional, which is concisely termed the value integral. The convergence behavior of the value integral demonstrates super-quadratic convergence in the first 50 iterations.
The second experiment was a velocity reaching task on the 1D Burgers equation with non-homogenous Dirichlet boundary conditions, given in fields representation by
∂ t h ( t , x ) = − h ( t , x ) ∂ x h ( t , x ) + ε ∂ x x h ( t , x ) + m ( x ) T U d ( t , x ) , h ( t , 0 ) = h ( t , a ) = 1.0 , h ( 0 , x ) = h 0 ( x ) , (98) \begin{aligned}
\partial_t h(t,x) &= -h(t,x)\partial_x h(t,x) + \varepsilon\partial_{xx}h(t,x) + \mathbf{m}(x)^{\mathsf{T}} U_d(t,x), \\
h(t,0) &= h(t,a) = 1.0, \\
h(0,x) &= h_0(x), \tag*{(98)}
\end{aligned} ∂ t h ( t , x ) h ( t , 0 ) h ( 0 , x ) = − h ( t , x ) ∂ x h ( t , x ) + ε ∂ xx h ( t , x ) + m ( x ) T U d ( t , x ) , = h ( t , a ) = 1.0 , = h 0 ( x ) , (98)
where the parameter ε \varepsilon ε is the viscosity of the medium. The Burgers equation is a nonlinear PDE, and demonstrates the efficacy of the approach on nonlinear PDEs. Starting from a zero initial condition, the PDE is tasked with raising the velocity to v = 2.0 v = 2.0 v = 2.0 on the outer regions, and v = 1.0 v = 1.0 v = 1.0 on the central region. The Burgers equation is often used as a simplified model of fluid flow, however also has applications in describing the dynamics of swarms for robotic systems [39 ] . The STDDP was applied to the Burgers PDE and was run until convergence. The results are depicted in Figure 2 .
Figure 2. Burgers Equation Velocity Reaching Task. (left) controlled contour plot where color represents velocity, (right) final time snapshot comparing to the uncontrolled system.
The nonlinear advection present in the Burgers equation produces an apparent rightward motion that builds over the spatial domain to create an apparent wavefront towards the right endpoint. The system is provided with 5 actuators, and must overcome this nonlinearity in order to minimize the state cost. Despite the added actuators, the task remains severely under-actuated. In this case, the weight values were R d = 0.4 R_d = 0.4 R d = 0.4 , Q = 30 Q = 30 Q = 30 , and Q f = 30 Q_f = 30 Q f = 30 . As depicted, the provided values of state and control cost weighting provide a balancing between the state and control performance metrics.
In both of the experiments, the various discretization schemes described in § 9.1 were tested, namely the explicit Euler discretization, a Runge-Kutta 2-point discretization, and the semi-implicit. The authors report that while the semi-implicit method had slightly lower sensitivity to the time-step increment Δ t \Delta t Δ t compared to the explicit Euler and Runge-Kutta methods, the large matrix inversion caused dramatically slower per-iteration run-time. The Runge-Kutta method had higher accuracy than the Euler method, but required super-sampling (i.e. sampling the midpoint of a time-increment) thus doubling the total time steps on forward and backward passes. The explicit Euler discretization had the fastest per-iteration run time at about 0.4 seconds per iteration, and was stabilized using regularization methods, akin to [36 ] .
Common to finite and infinite dimensional DDP methods are parameter sensitivities that may limit choice of the control cost weighting and the state cost weightings. When these limits arise, they are typically due to the numerically stiff and sensitive dynamics found in the backwards Ricatti equation eq. (38) , and present a limitation in the ability of DDP approaches to use arbitrary ratios of state performance and control effort. This can be especially limiting in systems with under-actuation as control signals can often be much larger for task completion, thus requiring a larger ratio between state cost weight and control cost weight. Without a "warm start", the operational initialization window for control weights may limit the use of an arbitrary desired set of parameters, thus changing the task specifications to meet numerical requirements.
In Figure 3 , we demonstrate that this can be overcome with a simple simulated annealing scheme. In this simulated experiment, the simulated annealing scheme was adopted in order to reach an arbitrarily large weight ratio W d : = Q / R d = 4.8 × 10 6 W_d := Q/R_d = 4.8 \times10^6 W d := Q / R d = 4.8 × 1 0 6 starting from a nominal weight ratio of W d = 25 W_d = 25 W d = 25 . This approach allows one to arbitrarily choose the relative importance of state performance and control effort. Depicted is a contour plot that demonstrates that the desired regions are quickly reached, and the system remains at the desired region for the duration of the simulation. Also depicted is a final time snapshot with dramatically smaller deviation from the desired region as compared to the solution in Figure 2 , albeit at the expense of larger control effort.
Figure 3. Burgers Equation Velocity Reaching Task with Simulated Annealing. (left) controlled contour plot where color represents velocity, (right) final time snapshot comparing the optimized solution to the uncontrolled system.
Discussion & Conclusion
We address the optimal control on nonlinear spatio-temporal systems through the lens of the Bellman principle of optimality, and develop the STDDP framework. We demonstrate that the resulting forward-backward system of equations can recover standard results, including the LQR solution for linear PDEs and the DDP solution for finite nonlinear ODEs. We analyze the convergence behavior and emerge with provable global convergence of the resulting forward-backward system. We discuss and develop discretization schemes for the backward second derivative of the value functional, and implement the resulting algorithm on a linear PDE system and a nonlinear PDE system.
The numerical results demonstrate the utility of the STDDP framework. It has the capability of obtaining high quality control solutions in the linear and nonlinear regime for spatio-temporal PDE systems. It has flexibility with respect to discretization schemes due to the optimize-then-discretize approach. It exhibits computationally efficiency for 1D PDEs with a typical 0.5 second time-per-iteration without any parallelization.
Overall, the results presented in this manuscript are encouraging to the authors for future work on extending the approach to 2D and 3D problem spaces. Such scaling will result in large tensors, however one can leverage the sparsity inherent in PDE discretizations and utilize common tensor decompositions such as the tensor train decomposition [40 ] for a dramatic computational speed-up. Other future directions include extensions to the case of a system with additive Gaussian noise, second order expansions of the dynamics, and novel methods to handle the sensitivities that arise in the discretization of the backward process.
Acknowledgements
This work was supported by the Army Research Office contract W911NF2010151. Ethan N. Evans was supported by the SMART scholarship.
Proof of Lemma 9.1
Proof. We start with the total derivative for the cost functional
d J i ( t , X ( t ) , U ( t ) ) d U t 0 : t f i = ∂ J i ( t , X ( t ) , U ( t ) ) ∂ U t 0 : t f i + ∂ J i ( t , X ( t ) , U ( t ) ) ∂ X t 0 : t f i ∂ X t 0 : t f i ∂ U t 0 : t f i (99) \frac{\mathrm{d}J^{i}\left(t,X(t),U(t)\right)}{\mathrm{d}U^{i}_{t_{0}:t_{f}}}
=
\frac{\partial J^{i}\left(t,X(t),U(t)\right)}{\partial U^{i}_{t_{0}:t_{f}}}
+
\frac{\partial J^{i}\left(t,X(t),U(t)\right)}{\partial X^{i}_{t_{0}:t_{f}}}
\frac{\partial X^{i}_{t_{0}:t_{f}}}{\partial U^{i}_{t_{0}:t_{f}}} \tag*{(99)} d U t 0 : t f i d J i ( t , X ( t ) , U ( t ) ) = ∂ U t 0 : t f i ∂ J i ( t , X ( t ) , U ( t ) ) + ∂ X t 0 : t f i ∂ J i ( t , X ( t ) , U ( t ) ) ∂ U t 0 : t f i ∂ X t 0 : t f i (99)
= ⟨ L U , t 0 : t f , 1 t 0 : t f ⟩ T + ⟨ L X , t 0 : t f , ∂ X t 0 : t f i ∂ U t 0 : t f i ⟩ T + ⟨ ϕ X T ( t f , X ( t f ) ) , ∂ X t f i ∂ U t 0 : t f i ⟩ T (100) =
\left\langle L_{U,t_{0}:t_{f}},\mathbbm{1}_{t_{0}:t_{f}}\right\rangle_{T}
+
\left\langle L_{X,t_{0}:t_{f}},\frac{\partial X^{i}_{t_{0}:t_{f}}}{\partial U^{i}_{t_{0}:t_{f}}}\right\rangle_{T}
+
\left\langle\phi_{X}^{\mathsf{T}}\left(t_{f},X(t_{f})\right),\frac{\partial X^{i}_{t_{f}}}{\partial U^{i}_{t_{0}:t_{f}}}\right\rangle_{T} \tag*{(100)} = ⟨ L U , t 0 : t f , 1 t 0 : t f ⟩ T + ⟨ L X , t 0 : t f , ∂ U t 0 : t f i ∂ X t 0 : t f i ⟩ T + ⟨ ϕ X T ( t f , X ( t f ) ) , ∂ U t 0 : t f i ∂ X t f i ⟩ T (100)
The state trajectory X t 0 : t f X_{t_{0}:t_{f}} X t 0 : t f is due to the approximate state evolution given by eqs. (17) and (18) , which has linear affine form with solution
X ( t ) = Φ ( t , t 0 ) X 0 + ∫ t 0 t Φ ( t , s ) F U ( s ) U ( s ) d s (101) X(t)=\Phi(t,t_{0})X_{0}+\int_{t_{0}}^{t}\Phi(t,s)F_{U}(s)U(s)\,\mathrm{d}s \tag*{(101)} X ( t ) = Φ ( t , t 0 ) X 0 + ∫ t 0 t Φ ( t , s ) F U ( s ) U ( s ) d s (101)
Thus, one has
d J i ( t , X ( t ) , U ( t ) ) d U t 0 : t f i = ⟨ L U , t 0 : t f , 1 t 0 : t f ⟩ T + ⟨ Φ T ( t , t 0 : t f ) L X , t 0 : t f , F U , t 0 : t f ⟩ T + ⟨ Φ T ( T , t 0 : t f ) ϕ X T ( t f , X ( t f ) ) , F U , t 0 : t f ⟩ T (102) \frac{\mathrm{d}J^{i}\left(t,X(t),U(t)\right)}{\mathrm{d}U^{i}_{t_{0}:t_{f}}}
=
\left\langle L_{U,t_{0}:t_{f}},\mathbbm{1}_{t_{0}:t_{f}}\right\rangle_{T}
+
\left\langle\Phi^{\mathsf{T}}\left(t,t_{0}:t_{f}\right)L_{X,t_{0}:t_{f}},F_{U,t_{0}:t_{f}}\right\rangle_{T}
+
\left\langle\Phi^{\mathsf{T}}\left(T,t_{0}:t_{f}\right)\phi_{X}^{\mathsf{T}}\left(t_{f},X(t_{f})\right),F_{U,t_{0}:t_{f}}\right\rangle_{T} \tag*{(102)} d U t 0 : t f i d J i ( t , X ( t ) , U ( t ) ) = ⟨ L U , t 0 : t f , 1 t 0 : t f ⟩ T + ⟨ Φ T ( t , t 0 : t f ) L X , t 0 : t f , F U , t 0 : t f ⟩ T + ⟨ Φ T ( T , t 0 : t f ) ϕ X T ( t f , X ( t f ) ) , F U , t 0 : t f ⟩ T (102)
Now, due to the terminal condition on ψ \psi ψ given in eq. (80) , one has
d J i ( t , X ( t ) , U ( t ) ) d U t 0 : t f i = ⟨ L U , t 0 : t f , 1 t 0 : t f ⟩ T + ⟨ Φ T ( t , t 0 : t f ) L X , t 0 : t f , F U , t 0 : t f ⟩ T + ⟨ Φ T ( T , t 0 : t f ) ψ ( t f , X ( t f ) , U ( t f ) ) , F U , t 0 : t f ⟩ T . (103) \frac{\mathrm{d}J^{i}\left(t,X(t),U(t)\right)}{\mathrm{d}U^{i}_{t_{0}:t_{f}}}
=
\left\langle L_{U,t_{0}:t_{f}},\mathbbm{1}_{t_{0}:t_{f}}\right\rangle_{T}
+
\left\langle\Phi^{\mathsf{T}}\left(t,t_{0}:t_{f}\right)L_{X,t_{0}:t_{f}},F_{U,t_{0}:t_{f}}\right\rangle_{T}
+
\left\langle\Phi^{\mathsf{T}}\left(T,t_{0}:t_{f}\right)\psi\left(t_{f},X(t_{f}),U(t_{f})\right),F_{U,t_{0}:t_{f}}\right\rangle_{T}. \tag*{(103)} d U t 0 : t f i d J i ( t , X ( t ) , U ( t ) ) = ⟨ L U , t 0 : t f , 1 t 0 : t f ⟩ T + ⟨ Φ T ( t , t 0 : t f ) L X , t 0 : t f , F U , t 0 : t f ⟩ T + ⟨ Φ T ( T , t 0 : t f ) ψ ( t f , X ( t f ) , U ( t f ) ) , F U , t 0 : t f ⟩ T . (103)
Finally, due to the backward recursion over the trajectory given by eq. (79) , one has
d J i ( t , X ( t ) , U ( t ) ) d U t 0 : t f i = ⟨ L U , t 0 : t f , 1 t 0 : t f ⟩ T + ⟨ ψ t 0 : t f , F U , t 0 : t f ⟩ T , (104) \frac{\mathrm{d}J^{i}\left(t,X(t),U(t)\right)}{\mathrm{d}U^{i}_{t_{0}:t_{f}}}
=
\left\langle L_{U,t_{0}:t_{f}},\mathbbm{1}_{t_{0}:t_{f}}\right\rangle_{T}
+
\left\langle\psi_{t_{0}:t_{f}},F_{U,t_{0}:t_{f}}\right\rangle_{T}, \tag*{(104)} d U t 0 : t f i d J i ( t , X ( t ) , U ( t ) ) = ⟨ L U , t 0 : t f , 1 t 0 : t f ⟩ T + ⟨ ψ t 0 : t f , F U , t 0 : t f ⟩ T , (104)
which concludes the proof.
Proof of Corollary 9.1
Proof. For simplicity of notation, we will use the shorthand J i : = J i ( t , X i ( t ) , U i ( t ) ) J^{i}:=J^{i}\left(t,X^{i}(t),U^{i}(t)\right) J i := J i ( t , X i ( t ) , U i ( t ) ) . Theorem 8.1 and assumption 6 together imply that ∃ γ ∈ [ 0 , 1 ) \exists\gamma\in[0,1) ∃ γ ∈ [ 0 , 1 ) such that the change in the cost over iterations Δ J i : = J i − J i − 1 < 0 \Delta J^{i}:=J^{i}-J^{i-1}<0 Δ J i := J i − J i − 1 < 0 , ∀ i ∈ N + \forall i\in\mathbb{N}_{+} ∀ i ∈ N + . The cost functional J ( ⋅ , ⋅ , ⋅ ) J(\cdot,\cdot,\cdot) J ( ⋅ , ⋅ , ⋅ ) is continuous in its arguments since it is differentiable by assumption, therefore it is also continuous with respect to iterations i i i . Thus lim i → ∞ Δ J i = 0 \lim_{i\to\infty}\Delta J^{i}=0 lim i → ∞ Δ J i = 0 and ∃ \exists ∃ a pair ( X t 0 : t f ∗ , U t 0 : t f ∗ ) \left(X^{*}_{t_{0}:t_{f}},U^{*}_{t_{0}:t_{f}}\right) ( X t 0 : t f ∗ , U t 0 : t f ∗ ) such that lim i → ∞ J ( t , X i ( t ) , U i ( t ) ) = J ( t , X ∗ ( t ) , U ∗ ( t ) ) = : J ∗ \lim_{i\to\infty}J\left(t,X^{i}(t),U^{i}(t)\right)=J\left(t,X^{*}(t),U^{*}(t)\right)=:J^{*} lim i → ∞ J ( t , X i ( t ) , U i ( t ) ) = J ( t , X ∗ ( t ) , U ∗ ( t ) ) =: J ∗ .
Next, lim i → ∞ Δ J i = 0 \lim_{i\to\infty}\Delta J^{i}=0 lim i → ∞ Δ J i = 0 together with eq. (85) and the positive definiteness of M t 0 : t f M_{t_{0}:t_{f}} M t 0 : t f imply that lim i → ∞ Q U , t 0 : t f i = 0 t 0 : t f \lim_{i\to\infty}Q^{i}_{U,t_{0}:t_{f}}=0_{t_{0}:t_{f}} lim i → ∞ Q U , t 0 : t f i = 0 t 0 : t f . By this notation we mean that lim i → ∞ Q U i ( t , X ( t ) , U ( t ) ) = 0 ∀ t ∈ T \lim_{i\to\infty}Q^{i}_{U}\left(t,X(t),U(t)\right)=0\ \forall t\in T lim i → ∞ Q U i ( t , X ( t ) , U ( t ) ) = 0 ∀ t ∈ T . Recall also that δ X 0 i \delta X^{i}_{0} δ X 0 i . We seek the intermediate result that δ X t 0 : t f ∗ : = lim i → ∞ X t 0 : t f i = 0 t 0 : t f \delta X^{*}_{t_{0}:t_{f}}:=\lim_{i\to\infty}X^{i}_{t_{0}:t_{f}}=0_{t_{0}:t_{f}} δ X t 0 : t f ∗ := lim i → ∞ X t 0 : t f i = 0 t 0 : t f . This can be observed in the coupled solutions of δ X ( t ) \delta X(t) δ X ( t ) and δ U ( t ) \delta U(t) δ U ( t ) , but is more clear by inspection of the closed-loop variation dynamics, which have generalized form
d δ X ( t ) d t = F X T ( t ) δ X ( t ) − F U T ( t ) Q U U − 1 ( t ) ( Q U ( t ) + Q U X ( t ) δ X ( t ) ) = ( F X T ( t ) − F U T ( t ) Q U U − 1 ( t ) Q U X ( t ) ) δ X ( t ) − F U T ( t ) Q U U − 1 ( t ) Q U ( t ) \begin{aligned}
\frac{\mathrm{d}\delta X(t)}{\mathrm{d}t} = F_X^{\mathsf{T}}(t)\delta X(t) - F_U^{\mathsf{T}}(t)Q_{UU}^{-1}(t)\left(Q_U(t) + Q_{UX}(t)\delta X(t)\right) \\
&= \left(F_X^{\mathsf{T}}(t) - F_U^{\mathsf{T}}(t)Q_{UU}^{-1}(t)Q_{UX}(t)\right)\delta X(t) - F_U^{\mathsf{T}}(t)Q_{UU}^{-1}(t)Q_U(t)
\end{aligned} d t d δ X ( t ) = F X T ( t ) δ X ( t ) − F U T ( t ) Q U U − 1 ( t ) ( Q U ( t ) + Q U X ( t ) δ X ( t ) ) = ( F X T ( t ) − F U T ( t ) Q U U − 1 ( t ) Q U X ( t ) ) δ X ( t ) − F U T ( t ) Q U U − 1 ( t ) Q U ( t )
with solution of the form
δ X ( t ) = Φ ~ ( t , t 0 ) δ X 0 − ∫ t 0 t Φ ~ ( t , s ) F U T ( s ) Q U U − 1 ( s ) Q U ( s ) d s , \delta X(t) = \widetilde{\Phi}(t,t_0)\delta X_0 - \int_{t_0}^{t}\widetilde{\Phi}(t,s)F_U^{\mathsf{T}}(s)Q_{UU}^{-1}(s)Q_U(s)\,\mathrm{d}s, δ X ( t ) = Φ ( t , t 0 ) δ X 0 − ∫ t 0 t Φ ( t , s ) F U T ( s ) Q U U − 1 ( s ) Q U ( s ) d s ,
where Φ ~ ( ⋅ , ⋅ ) \widetilde{\Phi}(\cdot,\cdot) Φ ( ⋅ , ⋅ ) is a contractive linear semigroup. Thus, since δ X 0 = 0 \delta X_0 = 0 δ X 0 = 0 , and lim i → ∞ Q U , t 0 : t f i = 0 \lim_{i\to\infty} Q_{U,t_0:t_f}^{i} = 0 lim i → ∞ Q U , t 0 : t f i = 0 , we have that δ X t 0 : t f ∗ = 0 t 0 : t f \delta X_{t_0:t_f}^{*} = 0_{t_0:t_f} δ X t 0 : t f ∗ = 0 t 0 : t f and thus δ U t 0 : t f ∗ : = lim i → ∞ δ U t 0 : t f i = 0 t 0 : t f \delta U_{t_0:t_f}^{*} := \lim_{i\to\infty}\delta U_{t_0:t_f}^{i} = 0_{t_0:t_f} δ U t 0 : t f ∗ := lim i → ∞ δ U t 0 : t f i = 0 t 0 : t f , which implies that lim i → ∞ U t 0 : t f i = U t 0 : t f ∗ \lim_{i\to\infty} U_{t_0:t_f}^{i} = U_{t_0:t_f}^{*} lim i → ∞ U t 0 : t f i = U t 0 : t f ∗ .
Finally, we must show that the converged control trajectory U t 0 : t f ∗ U_{t_0:t_f}^{*} U t 0 : t f ∗ is stationary. To show this, consider the rate of change of the cost functional with respect to control over iterations in the limit, namely
lim i → ∞ d J i d U t 0 : t f i = lim i → ∞ ⟨ Q U , t 0 : t f i , 1 t 0 : t f ⟩ T − lim i → ∞ ⟨ F U , t 0 : t f i , V X , t 0 : t f i − ψ t 0 : t f ⟩ T \lim_{i\to\infty}\frac{\mathrm{d}J^{i}}{\mathrm{d}U_{t_0:t_f}^{i}} = \lim_{i\to\infty}\left\langle Q_{U,t_0:t_f}^{i},\mathbb{1}_{t_0:t_f}\right\rangle_T - \lim_{i\to\infty}\left\langle F_{U,t_0:t_f}^{i},V_{X,t_0:t_f}^{i}-\psi_{t_0:t_f}\right\rangle_T i → ∞ lim d U t 0 : t f i d J i = i → ∞ lim ⟨ Q U , t 0 : t f i , 1 t 0 : t f ⟩ T − i → ∞ lim ⟨ F U , t 0 : t f i , V X , t 0 : t f i − ψ t 0 : t f ⟩ T
Since we already showed that lim i → ∞ Q U , t 0 : t f i = 0 \lim_{i\to\infty}Q_{U,t_0:t_f}^{i}=0 lim i → ∞ Q U , t 0 : t f i = 0 , one can easily apply the dominated convergence theorem to show that the first term is a zero trajectory. We must only prove that the second term is also a zero trajectory. By proposition 8.1, we have
lim i → ∞ ( V X i ( t ) − ψ ( t ) ) = lim i → ∞ ∫ T t Φ T ( t , τ ) Q U X i T ( τ ) Q U U i − 1 ( τ ) Q U i ( τ ) d τ = ∫ T t lim i → ∞ Φ T ( t , τ ) Q U X i T ( τ ) Q U U i − 1 ( τ ) Q U i ( τ ) d τ = 0 \begin{aligned}
\lim_{i\to\infty}\left(V_X^{i}(t)-\psi(t)\right) = \lim_{i\to\infty}\int_T^t\Phi^{\mathsf{T}}(t,\tau)Q_{UX}^{i\mathsf{T}}(\tau)Q_{UU}^{i-1}(\tau)Q_U^{i}(\tau)\,\mathrm{d}\tau\\
&= \int_T^t\lim_{i\to\infty}\Phi^{\mathsf{T}}(t,\tau)Q_{UX}^{i\mathsf{T}}(\tau)Q_{UU}^{i-1}(\tau)Q_U^{i}(\tau)\,\mathrm{d}\tau\\
&= 0
\end{aligned} i → ∞ lim ( V X i ( t ) − ψ ( t ) ) = i → ∞ lim ∫ T t Φ T ( t , τ ) Q U X i T ( τ ) Q U U i − 1 ( τ ) Q U i ( τ ) d τ = ∫ T t i → ∞ lim Φ T ( t , τ ) Q U X i T ( τ ) Q U U i − 1 ( τ ) Q U i ( τ ) d τ = 0
where we have again applied a properly formulated dominated convergence argument due to boundedness of V X X , t 0 : t f V_{XX,t_0:t_f} V X X , t 0 : t f and X t 0 : t f X_{t_0:t_f} X t 0 : t f ∀ i ∈ N + \forall i \in\mathbb{N}_{+} ∀ i ∈ N + by assumption, and due to Q U , t 0 : t f i Q_{U,t_0:t_f}^{i} Q U , t 0 : t f i being a decreasing function over iterations. The limit is again zero since lim i → ∞ Q U , t 0 : t f i = 0 t 0 : t f \lim_{i\to\infty}Q_{U,t_0:t_f}^{i}=0_{t_0:t_f} lim i → ∞ Q U , t 0 : t f i = 0 t 0 : t f . Thus lim i → ∞ d J i d U t 0 : t f i = 0 t 0 : t f \lim_{i\to\infty}\frac{\mathrm{d}J^{i}}{\mathrm{d}U_{t_0:t_f}^{i}}=0_{t_0:t_f} lim i → ∞ d U t 0 : t f i d J i = 0 t 0 : t f , and the converged trajectory U t 0 : t f ∗ U_{t_0:t_f}^{*} U t 0 : t f ∗ is indeed stationary, which concludes the proof.