betaFeedback

WorksEngineering

Reinforcement Learning Optimized Flight Control for Interception of High-Altitude Maneuvering Targets

Abstract

We implement and compare two approaches to improving flight control system (FCS) performance using reinforcement meta-learning (meta-RL). Both approaches are evaluated in a terminal phase hypersonic intercept engagement requiring hit-to-kill accuracy against a highly agile maneuvering target. In the first approach (FCS-RL), meta-RL is used to optimize a full FCS that maps acceleration tracking error and other measurements directly to commanded channel deflection rates of change, combining pitch, yaw, and roll control into a single policy. In contrast, the second approach (FCS-3L-RL) starts with a standard FCS (FCS-3L) implementing a three-loop autopilot for pitch and yaw control, but adapts the scheduled autopilot gains in real time using a policy optimized with meta-RL. Both approaches are compared to the FCS-3L in a 6-DOF simulator that captures most factors relevant to guidance and flight control. Simulations show that under ideal conditions the three systems have comparable performance. However, under conditions of aerodynamic uncertainty, radome refraction, or target evasive maneuvers, the FCS-RL significantly outperforms the FCS-3L-RL with improved intercept probability and less control effort. In turn, the FCS-3L-RL outperforms the FCS-3L by a similar margin. Importantly, we demonstrate that both FCS-RL and FCS-3L-RL generalize well to novel engagement scenarios, flight conditions, and target maneuvers that are not experienced during the optimization process.

Introduction

Flight control for control surface actuated interceptors tasked with intercepting actively evading targets in the high altitude hypersonic flight regime with hit-to-kill accuracy is a significant challenge. Consider that strike weapons fielded by a peer adversary will likely actively evade interception, requiring the interceptor to be capable of significant terminal phase maneuvering. These maneuvers can result in unsteady effects, boundary layer separation, and shock–boundary layer interaction [1, 2]. Aerodynamic modeling is further complicated by the extreme aero-thermal environment associated with hypersonic flight, where vehicle structures undergo stress caused by coupling of Fluid-Thermal-Structural Interactions (FTSI) and significant aerodynamic heating typical in hypersonic regimes [3]. This results in time-varying airframe and control surface deformation that are functions of both peak and integrated heating, which leads to vehicle aerodynamics differing substantially from that assumed during the design of the FCS [4, 5]. This aerodynamic uncertainty will require the flight control system (FCS) to adapt to differences between the simulated and deployment environments. The design of the FCS is further complicated by parasitic interaction between the guidance and flight control systems due to radome refraction [6, 7, 8]. Finally, high altitude engagements using only control surfaces actuators result in limited acceleration capability due to the low dynamic pressure.

Most currently deployed flight control systems for cruciform geometry skid-to-turn interceptors use the three-loop autopilot (TLA) [9, 10, 11] for pitch and yaw control, and a two-loop proportional-integral (roll rate and roll) controller for roll control. The TLA uses acceleration and rotational velocity measurements to track a commanded acceleration in the given channel (pitch and yaw). The FCS gains are optimized over a group of design points, typically angle of attack, sideslip angle, Mach, and altitude, and during flight the gains are interpolated between these design points to give optimal performance over the flight envelope. This approach gives both fast response and robust tracking to commanded acceleration. However, the standard TLA has limitations, including the assumptions of linear dynamics and no cross channel coupling, and simplified second order actuator model that does not account for fin limits and slew rate. Moreover, the three-loop architecture is a linear controller, which might constrain performance. Finally, as we show in the following, performance can degrade under high levels of aerodynamic uncertainty.

Consequently, considerable research has been focused on augmenting the standard three-loop autopilot, as well as exploring alternative approaches to interceptor flight control. Model-reference adaptive control (MRAC) methods [12, 13] have been used to augment conventional three-loop autopilots through supplemental control commands that drive the missile acceleration response toward that of a desired reference model, improving tracking performance under aerodynamic model uncertainty. Despite these benefits, MRAC has several limitations. Conventional MRAC stability and tracking guarantees rely on plant-model matching conditions, which may be violated under substantial aerodynamic uncertainty or changes in control effectiveness [14]. Furthermore, the reference model may demand a response that exceeds the actuator constraints, resulting in saturation and degraded tracking performance [15]. Adaptive controllers can also be sensitive to unmodeled high-frequency dynamics and measurement disturbances, potentially compromising stability, particularly at high adaptation gains [16]. Finally, asymptotic tracking guarantees do not necessarily ensure satisfactory transient performance, motivating modified adaptive architectures that explicitly address transient response [17].

More recently, reinforcement learning (RL) has been investigated both for optimizing conventional missile autopilots and for replacing their prescribed feedback architectures. Shin et al. [18] employed deep reinforcement learning to optimize the gains of a conventional three-loop autopilot, preserving its underlying control structure while allowing state-dependent gain adaptation. Taligatos et al. [19] compared RL-based optimization of conventional controller gains against a fully learned controller that directly commands control surface deflections, highlighting tradeoffs in control effort versus overshoot between the two methods. Similarly, Candeli et al. [20] employed RL to directly synthesize a missile autopilot, demonstrating its effectiveness under nonlinear aerodynamic dynamics and model uncertainty. Wang et al. [21] extended this approach to integrated guidance and control for interception of highly maneuverable targets, with an RL policy directly commanding pitch and yaw control surfaces. Gaudet et al. [22] used a hypersonic point defense intercept application to compare the effectiveness of using reinforcement meta-learning (meta-RL) to optimize a separate FCS policy as compared to optimizing an integrated guidance and control policy, and in [23] investigated meta-RL for an integrated guidance and control system in an air-to-air intercept application. Collectively, these studies illustrate the potential of RL to improve conventional FCS performance.

In the following, we explore two approaches to apply reinforcement meta-learning (meta-RL) to optimize an adaptive FCS capable of hit-to-kill (miss less than 0.25 m) terminal-phase performance against maneuvering targets over a high-altitude hypersonic flight envelope using only control surface actuation. In the first approach (FCS-RL), meta-RL is used to optimize a full FCS mapping acceleration tracking error and other measurements directly to channel deflection rates of change, combining pitch, yaw, and roll control into a single policy. In contrast, the second approach (FCS-3L-RL) starts with a standard FCS (FCS-3L) implementing a three-loop autopilot for pitch and yaw control, but adapts the scheduled autopilot gains in real time using a policy optimized with meta-RL. This work has some similarity to [19] in that we explore using RL to augment a standard FCS as well as optimizing a full meta-RL FCS. However, our work uses a much higher fidelity simulator, uses proximal policy optimization in place of deep deterministic policy gradient, includes testing over full trajectories, evaluates performance under aerodynamic uncertainty and a large flight envelope, and uses meta-RL instead of RL. The recurrent policy used in our implementation of meta-RL allows the FCS-RL to implement an integrator, which we later show is essential for the RL optimized FCS to operate over a large flight envelope.

The paper is organized as follows. § 2 describes the simulation environment and engagement scenarios, § 8 covers the development of the FCS-3L benchmark, IV.A gives background for RL and meta-RL and § 10 and § 10.A formulate the FCS optimization problems in the meta-RL framework. § 12 describes the benchmark FCS and compares the performance of the meta-RL and benchmark flight control systems. § 13 gives a brief discussion of certification issues. Conclusions are given in § 14

Simulation Environment

In this section, unless a variable has the "B" (body frame) or "S" (LOS stabilized frame) superscript, it is assumed to be in the inertial reference frame. The simulator, taken from [24], has sufficient fidelity to capture most factors relevant to flight control. Specifically, our simulator is 6-DOF, includes flight condition estimation, a guidance system, a non-linear radome model, an actuator model that accounts for actuator bandwidth, fin limits, and fin rate limits, an inertial measurement unit (IMU) that models noise, bias, and placement distance from center of gravity, a radar receiver with line-of-sight (LOS) and range dependent noise, delays between IMU and radar measurements, modeling of aerodynamic uncertainty, and explicit modeling of LOS recovery using a strap down seeker. Importantly, we optimize and test the flight control systems taking into account aerodynamic model uncertainty, which would be expected in the hypersonic flight regime.

The simulator is shown in Figure 1, where all systems run at 500 Hz. Modeling relative velocity estimation in this scenario has a significant performance impact [24]. Consequently, to better compare the effectiveness of the three flight control systems, we assume ground truth knowledge of relative velocity, and leave full relative state estimation to a separate paper [24]. However, we model flight condition estimation, including angle of attack, sideslip angle, Mach, altitude, and change in attitude from sensor measurements, as described later in § 2.C.5 and II.C.7.

Simulator diagram showing the interceptor guidance and control system, target model, engagement initializer, and 6-DOF EOM

Figure 1. Simulator Diagram

The interceptor G&C system must satisfy a load constraint, where the norm of the yy and zz body frame forces acting on the interceptor (see § 2.D) must be less than 75 g: ∥[FyB,FzB]∥<75g\lVert[F_y^B,F_z^B]\rVert< 75g, where g=9.81 m/s2g=9.81\ \mathrm{m/s^2} is the magnitude of the gravitational acceleration at the Earth’s surface. The look angle constraint requires the maximum angle between the body frame x-axis and the body frame LOS vector to be less than 45∘45^\circ. Rather than directly impose the look angle constraint by terminating an episode when the constraint is violated, we instead latch the G&C system observables at their value prior to the violation for as long as the constraint is violated. Consequently, it is possible to violate the constraint for short periods and maintain good performance, noting that this constraint is always violated close to the end of the engagement. Note that if we had included relative velocity estimation using a Kalman filter, the filter would propagate the state open loop (no measurement update) when the target is out of the field of view.

Engagement Scenario

Referring to Figure 21, the missile position vector, missile velocity vector, target position vector, and target velocity vector are shown as rM\mathbf{r}_{M}, vM\mathbf{v}_{M}, rT\mathbf{r}_{T}, vT\mathbf{v}_{T}. We can also define the relative position and velocity vectors rTM=rT−rM\mathbf{r}_{TM} = \mathbf{r}_{T} - \mathbf{r}_{M} and vTM=vT−vM\mathbf{v}_{TM} = \mathbf{v}_{T} - \mathbf{v}_{M}. The elevation angle θE\theta_{E} is the angle between rTM\mathbf{r}_{TM} and its projection onto the x-y plane. We randomly generate the target’s initial velocity vector such that vT\mathbf{v}_{T} lies within a cone with axis rTM\mathbf{r}_{TM} and half apex angle θvT\theta_{v_{T}}. In R3\mathbb{R}^{3} we define a heading error (HEHE) as the angle between the interceptor’s initial velocity vector and the velocity vector associated with the lead angle required to put the interceptor on a collision heading with the target. The interceptor initial conditions are listed in Table 1.

Engagement geometry showing missile and target positions, velocities, and line of sight

Figure 2. Engagement

Parameters Drawn UniformlyMinMax
Elevation Angle to Target θE\theta_{E} (degrees)010
Azimuth Angle to Target θA\theta_{A} (degrees)-180180
Altitude (m)1500025000
Missile Velocity Magnitude ∥vM∥\lVert\mathbf{v}_{M}\rVert (m/s)23002600
Heading Error (degrees)05
Initial Angle of Attack α\alpha (degrees)-5-5
Initial Side Slip Angle β\beta (degrees)-55
Initial Range to Targeted Asset ∥rAT∥\lVert\mathbf{r}_{AT}\rVert (m)2000025000
Target Velocity Magnitude ∥vT∥\lVert\mathbf{v}_{T}\rVert (m/s)30004000
Target Velocity Cone Half Apex Angle θvT\theta_{v_{T}} (degrees)015
Interceptor Maximum Acceleration (@ 2500 m/s 20000 m, applied to each component) (g)-20
Target Evasion Acceleration (@ 2500 m/s 20000 m, applied to each component) (g)04
Target Weave Period (s)0.53.0

Table 1. Simulation Initial Conditions

Target Evasive Acceleration Model

During optimization, we consider three types of evasive maneuvers, a step maneuver, a weaving maneuver, and a jinking maneuver, and add different maneuver types to test the generalization capability of the optimized policies. The weaving maneuver has frequency shared for each axis and phase independent for each axis in R3\mathbb{R}^{3}, and randomized phase offset and frequency. In contrast, the jinking maneuver is only randomized in frequency, shared for each axis. Each of these maneuvers has a 1/3 probability of being selected at the start of each maneuver. At each simulation step, the target’s evasive acceleration vector is modified by removing the component parallel with the target velocity vector, and then adjusted for dynamic pressure, with a nominal acceleration capability of 4 g at an altitude of 20 km and a speed of 2500 m/s. Thus, the target acceleration vector is always orthogonal to the target velocity. The maneuvers have varying acceleration levels and random start time, duration, and switching time. At the start of each episode, with probability 0.5 the maneuvers in that episode use the full evasion acceleration capability, and with probability 0.5 the full acceleration is scaled uniformly by a factor uniformly distributed between 0 and 1. Note that the target drag orthogonal to the interceptor LOS is also perceived as a maneuver; the target drag model is shown in § 2.E. Sample target maneuvers are shown in Figure 3, which show evasive acceleration only (not drag orthogonal to LOS).

Sample target maneuvers showing jinking, step, and weave evasive acceleration

Figure 3. Sample Target Maneuvers (only evasive acceleration shown)

Interceptor Components

Radome Model

We calculate the refracted azimuth θ~u\tilde{\theta}_u and elevation θ~v\tilde{\theta}_v angles as shown in Eqs. (1a) through (1b), where our nominal value of AA is 0.5 mrad, and KuK_u, KvK_v are set to 3.0. Note we assume that there is a combination of software (such as compensation lookup tables) and hardware (such as radome shaping) that reduce the effective radome refraction to these values.

θ~u=θu+A(0.75θuπ/2+0.25cos⁡(2πkuθu))(1)\tilde{\theta}_u = \theta_u + A\left(0.75\frac{\theta_u}{\pi/2} + 0.25\cos\left(\frac{2\pi}{k_u}\theta_u\right)\right) \tag*{(1)}
θ~v=θv+A(0.75θvπ/2+0.25cos⁡(2πkvθv))(2)\tilde{\theta}_v = \theta_v + A\left(0.75\frac{\theta_v}{\pi/2} + 0.25\cos\left(\frac{2\pi}{k_v}\theta_v\right)\right) \tag*{(2)}

The refraction angle θR\theta_R is shown in Figure 4 for various values of kk. Note that the radome slope ∂θR∂θL\frac{\partial\theta_R}{\partial\theta_L} is given by the slope of the curves in the figure.

Radome refraction angle as a function of look angle

Figure 4. Radome Refraction Angle as Function of Look Angle

Seeker Model

We model sensor noise (elevation, azimuth, and range) as Gaussian noise, using a standard deviation of 1% of range for range dependent range noise and 0.2 mrad for angle noise. Obtaining this level of angle noise requires a dual model seeker (radar for range, infrared for angles). This improves angular resolution, target discrimination, and resistance to countermeasures. Note that our assumed noise level is post signal processing. We model the lag of the seeker signal processor as a pure delay of 4 ms.

Rate Gyro Model

Our rate gyro model corrupts the ground truth rotational velocity ωB\omega^{\mathrm{B}} with both scale factor bias and Gaussian noise, as shown in Eq. (2a), where U(a,b,n)\mathcal{U}(a,b,n) denotes an nn dimensional uniformly distributed random variable bounded by (a,b)(a,b) that is generated at the start of each episode, and each dimension of the random variable is independent. We use ϵω=0.001\epsilon_\omega= 0.001 and σω=0.8∘\sigma_\omega= 0.8^\circ. The rotational velocity is passed through a two pole low pass filter with a 100 Hz natural frequency and 0.707 damping, as shown in Eq. 2b.

ω1B=ω(1+U(−ϵω,ϵω,3))(3)\omega^{\mathrm{B}}_1 = \omega(1+\mathcal{U}(-\epsilon_\omega,\epsilon_\omega,3)) \tag*{(3)}
ωobsB=LPF⁡(ω1B)+N(0,σω,3)(4)\omega^{\mathrm{B}}_{\mathrm{obs}} = \operatorname{LPF}(\omega^{\mathrm{B}}_1) + \mathcal{N}(0,\sigma_\omega,3) \tag*{(4)}

Accelerometer Model

We account for accelerometer placement using Eq. (3a), with c=rcg−racc\mathbf{c}=\mathbf{r}_{\mathrm{cg}}-\mathbf{r}_{\mathrm{acc}}, where racc\mathbf{r}_{\mathrm{acc}} is the accelerometer location, which we assume to be -25 cm using the reference frame shown in Figure 5. Similarly to the rate gyro model, the body frame acceleration is corrupted with a scale factor error as shown in Eqs. (3b) and (3c), where we use ϵacc=0.001\epsilon_{\mathrm{acc}}=0.001, and "LPF" models the accelerometer dynamics as a two pole low pass filter with a 100 Hz natural frequency and 0.707 damping, and σacc=0.1 m/s2\sigma_{\mathrm{acc}}=0.1\ \mathrm{m/s}^{2}.

Interceptor Geometry

Figure 5. Interceptor Geometry

a1B=aB+[ω×]c+[ω×][ω×](c+δc)(5)\mathbf{a}_{1}^{\mathrm{B}}=\mathbf{a}^{\mathrm{B}}+[\boldsymbol{\omega}\times]\mathbf{c}+[\boldsymbol{\omega}\times][\boldsymbol{\omega}\times](\mathbf{c}+\delta\mathbf{c}) \tag*{(5)}
a2B=LPF⁡(a1B)(6)\mathbf{a}_{2}^{\mathrm{B}}=\operatorname{LPF}(\mathbf{a}_{1}^{\mathrm{B}}) \tag*{(6)}
aobsB=a2B(1+U(−ϵacc,ϵacc,3))+N(0,σacc,3)(7)\mathbf{a}_{\mathrm{obs}}^{\mathrm{B}}=\mathbf{a}_{2}^{\mathrm{B}}\left(1+\mathcal{U}(-\epsilon_{\mathrm{acc}},\epsilon_{\mathrm{acc}},3)\right)+\mathcal{N}(0,\sigma_{\mathrm{acc}},3) \tag*{(7)}

Line of Sight Recovery

Referring to Figure 1, the refracted body frame line of sight unit vector r^TMB\hat{\mathbf{r}}_{\mathrm{TM}}^{\mathrm{B}} from the dual-mode receiver block is input to the LOS recovery block. This block uses the estimated change of attitude since the start of the engagement dqobs\mathbf{dq}_{\mathrm{obs}} to rotate r^TMB\hat{\mathbf{r}}_{\mathrm{TM}}^{\mathrm{B}} into the reference frame corresponding to the interceptor’s attitude at the start of the engagement. Specifically, at each guidance step we map dqobs\mathbf{dq}_{\mathrm{obs}} to a direction cosine matrix C(dqobs)\mathbf{C}(\mathbf{dq}_{\mathrm{obs}}), and compute the recovered (stabilized) LOS as r^TMS=[C(dqobs)]Tr^TMB\hat{\mathbf{r}}_{\mathrm{TM}}^{\mathrm{S}}=[\mathbf{C}(\mathbf{dq}_{\mathrm{obs}})]^{\mathsf{T}}\hat{\mathbf{r}}_{\mathrm{TM}}^{\mathrm{B}}.

The estimated change in attitude dqobs\mathbf{dq}_{\mathrm{obs}} is parameterized as a quaternion, and is computed by integrating ω\boldsymbol{\omega} as shown in Equation (8), where dqobs\mathbf{dq}_{\mathrm{obs}} is reset at the start of each episode to [1,0,0,0][1,0,0,0]. In our simulation model, we approximate this integration using fourth-order Runge-Kutta integration. In the unrealistic case where we have access to the ground truth value of ω\boldsymbol{\omega}, this completely decouples r^TM\hat{\mathbf{r}}_{\mathrm{TM}} from body rotation. However, since ωobs\boldsymbol{\omega}_{\mathrm{obs}} is corrupted with a scale factor error and Gaussian noise, in general dqobs≠dq\mathbf{dq}_{\mathrm{obs}}\ne\mathbf{dq}, and the LOS recovery will be imperfect. The combination of imperfect LOS recovery and radome refraction result in a false indication of target motion. This results in increased miss distance, and can potentially destabilize the G&C system. This is discussed in more detail in [6, 7, 25].

[dq˙0dq˙1dq˙2dq˙3]=12[dq0−dq1−dq2−dq3dq1dq0−dq3dq2dq2dq3dq0−dq1dq3−dq2dq1dq0][0ωobs0ωobs1ωobs2](8)\begin{bmatrix} \dot{dq}_{0}\\ \dot{dq}_{1}\\ \dot{dq}_{2}\\ \dot{dq}_{3} \end{bmatrix} = \frac{1}{2} \begin{bmatrix} dq_{0} & -dq_{1} & -dq_{2} & -dq_{3}\\ dq_{1} & dq_{0} & -dq_{3} & dq_{2}\\ dq_{2} & dq_{3} & dq_{0} & -dq_{1}\\ dq_{3} & -dq_{2} & dq_{1} & dq_{0} \end{bmatrix} \begin{bmatrix} 0\\ \omega_{\mathrm{obs}0}\\ \omega_{\mathrm{obs}1}\\ \omega_{\mathrm{obs}2} \tag*{(8)} \end{bmatrix}

Guidance System

The guidance system implements true proportional navigation (TPN) [26] to calculate a commanded acceleration acomS\mathbf{a}_{\mathrm{com}}^{\mathrm{S}} from the LOS rotation rate, as shown in Eqs. (5a) through (5c), where vc=−rTMS⋅vTMS∥rTMS∥\mathbf{v}_{c}=-\frac{\mathbf{r}_{\mathrm{TM}}^{\mathrm{S}}\cdot\mathbf{v}_{\mathrm{TM}}^{\mathrm{S}}}{\lVert\mathbf{r}_{\mathrm{TM}}^{\mathrm{S}}\rVert}. This commanded acceleration is then rotated into the body frame using the estimated change in attitude dq\mathbf{dq} and adjusted using g-limits. These g-limits assume a maximum acceleration of 25 g at an altitude of 20 km and a speed of 2500 m/s, and are adjusted for dynamic pressure. This body frame commanded acceleration is then used to compute the acceleration tracking error according to aerrB=acom[y,z]B−aobs[y,z]B\mathbf{a}_{\mathrm{err}}^{\mathrm{B}}=\mathbf{a}_{\mathrm{com}}[\mathbf{y},z]^{\mathrm{B}}-\mathbf{a}_{\mathrm{obs}}[\mathbf{y},z]^{\mathrm{B}}, where aobsB\mathbf{a}_{\mathrm{obs}}^{\mathrm{B}} is the accelerometer measurement and [y,z][\mathbf{y},z] indicates we are computing the error over the body frame y and z components. We use N=4N=4 for the navigation ratio.

Ω=rTMS×vTMSrTMS⋅rTMS(9)\boldsymbol{\Omega}=\frac{\mathbf{r}_{\mathrm{TM}}^{\mathrm{S}}\times\mathbf{v}_{\mathrm{TM}}^{\mathrm{S}}}{\mathbf{r}_{\mathrm{TM}}^{\mathrm{S}}\cdot\mathbf{r}_{\mathrm{TM}}^{\mathrm{S}}} \tag*{(9)}
acomS=−Nvc(rTMS∥rTMS∥×Ω)(10)\mathbf{a}_{\mathrm{com}}^{\mathrm{S}}=-N\mathbf{v}_{c}\left(\frac{\mathbf{r}_{\mathrm{TM}}^{\mathrm{S}}}{\lVert\mathbf{r}_{\mathrm{TM}}^{\mathrm{S}}\rVert}\times\boldsymbol{\Omega}\right) \tag*{(10)}
acomB=[C(dqobs)]acomS(11)\mathbf{a}_{\mathrm{com}}^{\mathrm{B}}=[\mathbf{C}(\mathbf{dq}_{\mathrm{obs}})]\mathbf{a}_{\mathrm{com}}^{\mathrm{S}} \tag*{(11)}

Flight Condition Estimation

The FCS requires an estimate of angle of attack, α\alpha, sideslip angle β\beta, altitude hh, and Mach MM. These observations are generated by integrating aobsB\mathbf{a}_{\mathrm{obs}}^{\mathrm{B}} to obtain an estimate of body frame velocity and position, allowing computation of estimated interceptor velocity vobsBv_{\mathrm{obs}}^{\mathrm{B}} and position robsB\mathbf{r}_{\mathrm{obs}}^{\mathrm{B}}, with observed angle of attack αobs\alpha_{\mathrm{obs}} and sideslip angle βobs\beta_{\mathrm{obs}} computed from vobsBv_{\mathrm{obs}}^{\mathrm{B}}. Estimated body frame position robsB\mathbf{r}_{\mathrm{obs}}^{\mathrm{B}} is rotated into the inertial frame using estimated attitude, allowing computation of altitude hobsh_{\mathrm{obs}}. Mach MM is computed from estimated altitude and speed using an interpolated Earth atmospheric model table. Finally, for roll stabilization, the FCS requires an estimate of roll (θrollobs\theta_{\mathrm{rollobs}}), which is computed by integrating ωobsx\omega_{\mathrm{obs}x}.

Actuator Model

The channel commanded deflections δC=[δP,δY,δA,δSB]\delta_{\mathrm{C}} = [\delta_{\mathrm{P}}, \delta_{\mathrm{Y}}, \delta_{\mathrm{A}}, \delta_{\mathrm{SB}}] (pitch, yaw, roll, and speed brake channel commands) from the FCS are converted to physical fin deflections δF=[δ1,δ2,δ3,δ4]\delta_{\mathrm{F}} = [\delta_{1}, \delta_{2}, \delta_{3}, \delta_{4}] using the fin mix matrix shown in Eq. (12) and the transformations shown in Eqs. (7a) and (14). Note that the speed brake is unused.

Mfin=[101−10111−101−10−111](12)\mathbf{M}_{\mathrm{fin}} = \begin{bmatrix} 1 & 0 & 1 & -1 \\ 0 & 1 & 1 & 1 \\ -1 & 0 & 1 & -1 \\ 0 & -1 & 1 & 1 \end{bmatrix} \tag*{(12)}
δC=Mfin−1δF(13)\delta_{\mathrm{C}} = \mathbf{M}_{\mathrm{fin}}^{-1}\delta_{\mathrm{F}} \tag*{(13)}
δF=MfinδC(14)\delta_{\mathrm{F}} = \mathbf{M}_{\mathrm{fin}}\delta_{\mathrm{C}} \tag*{(14)}

δF\delta_{\mathrm{F}} is then clipped to remain within the maximum deflection of 30∘30^\circ, and the slew rate is limited to 500∘/s500^\circ/\mathrm{s}. Each element of δF\delta_{\mathrm{F}} is then passed through second order actuator dynamics corresponding to the transfer function shown in Eq. (15), where we use ζACT=0.7\zeta_{\mathrm{ACT}} = 0.7 and ωACT=314\omega_{\mathrm{ACT}} = 314 rad/s (50Hz). The output of the actuator dynamics is then referred to as δF\delta_{\mathrm{F}} in the remainder of the document.

11+2ζACTωACTs+s2ωACT2(15)\frac{1}{1 + \frac{2\zeta_{\mathrm{ACT}}}{\omega_{\mathrm{ACT}}}s + \frac{s^{2}}{\omega_{\mathrm{ACT}}^{2}}} \tag*{(15)}

Aerodynamic Model

The interceptor geometry is taken from [27] and is shown in Figure 5, with the nominal center of gravity set to -0.54 m. Our nominal aerodynamic model is derived from slender body and Newtonian impact theory [28, 29]. The method in [29] is extended to 6-DOF using a method similar to that in [22], but with a higher fidelity axial force model, rolling moment model, and the addition of a cross channel interference term for control effectiveness. The vehicle diameter is d=0.105d = 0.105 m. The length is xL=1.05x_{\mathrm{L}} = 1.05 m, and the inertia tensor is given in Figure 9. The vehicle mass is 13.5 kg, the reference length is Lref=dL_{\mathrm{ref}} = d, and the reference area is Sref=π(d/2)2S_{\mathrm{ref}} = \pi(d/2)^2. The open loop response to flight control commanded deflections are shown in Figure 6 and Figure 7, where δC=δP,δY,δA\delta_{\mathrm{C}} = \delta_{\mathrm{P}}, \delta_{\mathrm{Y}}, \delta_{\mathrm{A}} are the FCS commanded channel deflections as described in § 2.C.8. We see that the vehicle is slightly open-loop unstable.

Open Loop Airframe Response plots

Figure 6. Open Loop Airframe Response δC=[30,30,0]\delta_C = [30, 30, 0], Altitude=15 km, speed=2500 m/s

Open Loop Airframe Response plots

Figure 7. Open Loop Airframe Response δC=[30,30,0]\delta_C = [30, 30, 0], Altitude=25 km, speed=2500 m/s

Environment-Agent Interface

Figure 9. Environment-Agent Interface

J=[0.020000.950000.95](16)\mathbf{J} = \begin{bmatrix} 0.02 & 0 & 0 \\ 0 & 0.95 & 0 \\ 0 & 0 & 0.95 \end{bmatrix} \tag*{(16)}

We model aerodynamic uncertainty by randomly modifying the shape of the aerodynamic coefficient curves for each simulated trajectory. To do so, we first bias the coefficients returned from the nominal aerodynamic model as follows. At the start of each episode with probability p=0.5p = 0.5 we uniformly sample an aerodynamic coefficient bias ϵcoeff∈R4\epsilon_{\mathrm{coeff}} \in\mathbb{R}^{4} between some bounds +/−ϵ+/-\epsilon, otherwise a coefficient bias is (independently) set to either −ϵ-\epsilon or ϵ\epsilon with equal probability, noting that in both cases the bias applied to the coefficient groups (Cz,Cy)(C_z,C_y), (Cm,Cn)(C_m,C_n), CxC_x, and ClC_l are independent. We assume that the bias on CzC_z and CyC_y is identical to symmetry (and similarly for CmC_m and CnC_n). Then, for the remainder of the episode, the force and moment coefficients are biased using the sampled value of ϵcoeff\epsilon_{\mathrm{coeff}}. We account for CP/CG shifting by modifying the moments as shown in Eqs. (10a) through (10d). Here ΔCP\Delta_{\mathrm{CP}} is uniformly generated at the start of each episode and MRP is the moment reference point, which in this work is set to the CG. Due to symmetry, errors in ΔCPy\Delta_{\mathrm{CP}_y} and ΔCPz\Delta_{\mathrm{CP}_z} will mostly cancel.

ΔCCGCP=(MRP−CG)+ΔCPLref×[Cx,Cy,Cz](17)\Delta C_{\mathrm{CGCP}} = \frac{(\mathrm{MRP}-\mathrm{CG})+\Delta_{\mathrm{CP}}}{L_{\mathrm{ref}}}\times[C_x,C_y,C_z] \tag*{(17)}
Cladj=Cl+ΔCCGCPx(18)C_{l_{\mathrm{adj}}}=C_l+\Delta C_{\mathrm{CGCP}_x} \tag*{(18)}
Cmadj=Cm+ΔCCGCPy(19)C_{m_{\mathrm{adj}}}=C_m+\Delta C_{\mathrm{CGCP}_y} \tag*{(19)}
Cnadj=Cn+ΔCCGCPz(20)C_{n_{\mathrm{adj}}}=C_n+\Delta C_{\mathrm{CGCP}_z} \tag*{(20)}

Equations of Motion

Let vB≡[u,v,w]\mathbf{v}^{\mathrm{B}}\equiv[u,v,w] denote the body frame interceptor velocity. The angle of attack α\alpha and sideslip angle β\beta are then computed as shown in Eqs. (11a) through (11b).

α=arctan⁡wu(21)\alpha=\arctan\frac{w}{u} \tag*{(21)}
β=arctan⁡vu(22)\beta=\arctan\frac{v}{u} \tag*{(22)}
Equation not extracted
(23)

The atmospheric density ρ\rho and Mach number MM are calculated using the 1976 standard Earth atmospheric temperature model. The interceptor control surface deflections θctrl=[θLT,θRT,θBT,θUT]\boldsymbol{\theta}_{\mathrm{ctrl}}=[\theta_{\mathrm{LT}},\theta_{\mathrm{RT}},\theta_{\mathrm{BT}},\theta_{\mathrm{UT}}], dynamic pressure qo=12ρV2q_o=\frac{1}{2}\rho V^2 with V=∥vB∥V=\lVert\mathbf{v}^{\mathrm{B}}\rVert, rotational velocity vector ω\boldsymbol{\omega}, MM, α\alpha and β\beta are then input to the aerodynamic model described in § 2.D to compute the body frame force FB\mathbf{F}^{\mathrm{B}} and torque LB\mathbf{L}^{\mathrm{B}}.

The rotational velocity ω\boldsymbol{\omega} is updated by integrating the Euler rotational equations of motion, as shown in Equation (24), where J\mathbf{J} is the interceptor’s inertia tensor, and with the skew symmetric operator [a×][\mathbf{a}\times] defined in Eq. (25).

Jω˙=−[ω×]Jω+LB(24)\mathbf{J}\dot{\boldsymbol{\omega}}=-[\boldsymbol{\omega}\times]\mathbf{J}\boldsymbol{\omega}+\mathbf{L}^{\mathrm{B}} \tag*{(24)}
[a×]≡[0−a3a2a30−a1−a2a10](25)[\mathbf{a}\times]\equiv \begin{bmatrix} 0 & -a_3 & a_2\\ a_3 & 0 & -a_1\\ -a_2 & a_1 & 0 \end{bmatrix} \tag*{(25)}

The interceptor’s attitude q\mathbf{q} is updated by integrating the differential kinematic equations shown in Equation (26), where the interceptor’s attitude is parameterized using the quaternion representation and ωi\omega_i denotes the ithi^{th} component of the rotational velocity vector ωB\boldsymbol{\omega}_{\mathrm{B}}.

[q˙0q˙1q˙2q˙3]=12[q0−q1−q2−q3q1q0−q3q2q2q3q0−q1q3−q2q1q0][0ω0ω1ω2](26)\begin{bmatrix} \dot{q}_0\\ \dot{q}_1\\ \dot{q}_2\\ \dot{q}_3 \end{bmatrix} = \frac{1}{2} \begin{bmatrix} q_0 & -q_1 & -q_2 & -q_3\\ q_1 & q_0 & -q_3 & q_2\\ q_2 & q_3 & q_0 & -q_1\\ q_3 & -q_2 & q_1 & q_0 \end{bmatrix} \begin{bmatrix} 0\\ \omega_0\\ \omega_1\\ \omega_2 \tag*{(26)} \end{bmatrix}

The interceptor body frame velocity vB\mathbf{v}^{\mathrm{B}} is updated by integrating the differential equation given in Eq. (27), where mm is the interceptor mass, CBN(q)C_{\mathrm{BN}}(q) is the DCM mapping from the inertial to body frame given the interceptor attitude q\mathbf{q}, and g=−9.81 m/s2\mathbf{g}=-9.81\ \mathrm{m/s^2}.

v˙B=−[ω×]vB+FBm+[CBN(q)]T[00g](27)\dot{\mathbf{v}}^{\mathrm{B}}=-[\boldsymbol{\omega}\times]\mathbf{v}^{\mathrm{B}}+\frac{\mathbf{F}^{\mathrm{B}}}{m}+[C_{\mathrm{BN}}(q)]^{T} \begin{bmatrix} 0 & 0 & g \end{bmatrix} \tag*{(27)}

vB\mathbf{v}^{\mathrm{B}} is then rotated into the North-East-Down (NED) inertial frame as shown in Eq. (16a), and vM\mathbf{v}_{\mathrm{M}} is calculated by rotating the vector into the East-North-Up (ENU) reference frame depicted in Figure 2.

vN=CBN(q)vB(28)\mathbf{v}^{\mathrm{N}}=C_{\mathrm{BN}}(q)\mathbf{v}^{\mathrm{B}} \tag*{(28)}
vM=CNEvN(29)\mathbf{v}_{\mathrm{M}}=C_{\mathrm{NE}}\mathbf{v}^{\mathrm{N}} \tag*{(29)}

The interceptor inertial frame position rM\mathbf{r}_{\mathrm{M}} is then updated by integrating Eq. (30)

r˙M=vM(30)\dot{\mathbf{r}}_{\mathrm{M}} = \mathbf{v}_{\mathrm{M}} \tag*{(30)}

The target inertial frame position rT\mathbf{r}_{\mathrm{T}} and velocity unadjusted for drag v~T\tilde{\mathbf{v}}_{\mathrm{T}} are then updated by integrating Eqs. (18a) and (32). The target speed VT=∥vT∥V_{\mathrm{T}} = \lVert\mathbf{v}_{\mathrm{T}}\rVert is adjusted for drag by integrating Eq. (18c), where at the start of each episode we randomly generate kTk_{\mathrm{T}} from 0.1 to 0.25 and cd0c_{\mathrm{d0}} from 0.05 to 0.10. We use mT=450m_{\mathrm{T}} = 450 kg.

r˙T=vT(31)\dot{\mathbf{r}}_{\mathrm{T}} = \mathbf{v}_{\mathrm{T}} \tag*{(31)}
v~˙T=aT(32)\dot{\tilde{\mathbf{v}}}_{\mathrm{T}} = \mathbf{a}_{\mathrm{T}} \tag*{(32)}
V˙T=−ρ(hT)VT2cd02mT−kT∥aT∥(33)\dot{V}_{\mathrm{T}} = -\frac{\rho(h_{\mathrm{T}})V_{\mathrm{T}}^{2}c_{\mathrm{d0}}}{2m_{\mathrm{T}}} - k_{\mathrm{T}}\lVert\mathbf{a}_{\mathrm{T}}\rVert \tag*{(33)}

The target velocity is then computed as shown in Eq. (34)

vT=VTv~T∥v~T∥(34)\mathbf{v}_{\mathrm{T}} = V_{\mathrm{T}}\frac{\tilde{\mathbf{v}}_{\mathrm{T}}}{\lVert\tilde{\mathbf{v}}_{\mathrm{T}}\rVert} \tag*{(34)}

The equations of motion are updated using fourth order Runge-Kutta integration. For ranges greater than 50 m, a timestep of 5 ms is used, and for the final 50 m of homing, a timestep of 25 us is used in order to more accurately (within 0.125 m) measure miss distance; this technique is borrowed from [30].

3-Loop Autopilot FCS Benchmark

The benchmark FCS (FCS-3L) for this application uses a three-loop autopilot for pitch and yaw control. We assume perfect roll control for the benchmark (and for the augmented benchmark, the FCS-3L-RL), obtained by forcing the xx-component of rotational velocity to zero, making this a more difficult benchmark to beat. The three-loop autopilot design and gain scheduling are taken from [9], with the modification for an unstable airframe (the linearized airframe is unstable) taken from [31], and modified for our vehicle geometry. This gain scheduling was suboptimal for altitudes above 15 km. Therefore, we performed a series of grid searches on the open-loop crossover frequency ωCR\omega_{\mathrm{CR}}, damping ζ\zeta, and time constant τ\tau at three design points: 2500 m/s 15 km, 2500 m/s 20 km, and 2500 m/s 25 km. For a given design point, each design (a tuple of ωCR\omega_{\mathrm{CR}}, ζ\zeta, τ\tau) was simulated for 5000 episodes with up to 10% aerodynamic coefficient bias and 1% center of pressure variation (see § 2.D). We considered a design stable if it resulted in no load constraint violations. We chose this approach as traditional stability metrics such as phase margin with a linearized system can be overly conservative, as a large gain margin can be required to compensate for differences between the linearized and actual system. This approach also gives a better comparison to the FCS-RL, which being inherently nonlinear, cannot be analyzed using linear system metrics. For each FCS parameter tuple at a given design point, we chose the best performing set of parameters that resulted in a stable design. During FCS operation, we interpolate between the parameters at each of the three design points using linear interpolation. This approach is different than what is typically used for interceptor flight control systems, where the gains are optimized at a given design point using step response metrics, but has the advantage that the gains can be optimized considering the dynamics of both the guidance and flight control systems, which are more directly comparable to the approach used for FCS-RL optimization. A block diagram of the pitch and yaw channels for the FCS-3L is illustrated in Figure 8, where acoma_{\mathrm{com}}, aobsa_{\mathrm{obs}}, ωobs\omega_{\mathrm{obs}} coming from either the pitch or yaw channel from the guidance system, accelerometer, and rate gyro, respectively, and δcom\delta_{\mathrm{com}} being either the pitch or yaw commanded channel deflection. The relationship between the block diagram gains and ωCR\omega_{\mathrm{CR}}, ζ\zeta, τ\tau can be found in [9]. Step response plots for this benchmark can be found in § 12.A. Figure 13 We did not add adaptive control, as previous work [32] showed that MRCAS was ineffective in modeling this intercept scenario due to the short terminal phase homing time (which is unavoidable with high closing speeds and a realistic seeker range) and the other factors discussed in § 1.

Three-Loop Autopilot Block Diagram

Figure 8. Three-Loop Autopilot Block Diagram

Step response comparison plots for Meta-RL and benchmark systems under nominal, fast, and slow aerodynamics

Figure 13. Step Response Comparison: 25 km, 2500 m/s

Meta-RL Optimization Framework

Note that in this section, unless a variable has the "B" (body frame) superscript, it is assumed to be in the LOS stabilized ("S") reference frame.

Overview of Reinforcement Learning

In the reinforcement learning framework, an agent learns through episodic interaction with an environment how to successfully complete a task using a policy π\pi that maps observations o\mathbf{o} to actions u\mathbf{u}, i.e., u=π(o)\mathbf{u} = \pi(\mathbf{o}). In actor-critic RL algorithms, policy optimization makes use of a value function v=Vπ(x)v = V^{\pi}(\mathbf{x}) that estimates the estimated value of being in state x\mathbf{x} while following policy π\pi. Note that the value function can take either ground truth states x\mathbf{x}, observations o\mathbf{o}, or some combination of both, as only the policy is deployed. The environment initializes an episode by generating a ground truth state x\mathbf{x}, mapping this state to an observation, and passing the observation to the agent. The agent uses this observation to generate an action that is sent to the environment; the environment then uses the action and the current ground truth state to generate the next state and a scalar reward signal r(x,u)r(\mathbf{x}, \mathbf{u}). The reward and the observation corresponding to the next state are then passed to the agent. The process repeats until the environment terminates the episode, with the termination signaled to the agent via a done signal. In this work, an episode terminates when the closing velocity vcv_c turns negative, or a path constraint is violated. Trajectories collected over a set of episodes (referred to as rollouts) are collected during interaction between the agent and environment, and used to update the policy and value functions. The interface between agent and environment is depicted in Figure 9, where the environment instantiates the simulator and the system model shown in Figure 1.

Meta-RL differs from generic reinforcement learning in that the agent learns over an ensemble of environments. For example, each environment can contain a unique aerodynamic model, target maneuver model, actuator model, and other peripheral components. Similar to [33], we implement both the policy and value function as a deep recurrent neural network, with each network having a hidden state h\mathbf{h} that is a function of the history of observations o\mathbf{o}. Thus, the policy and value function become u=πθ(o,h)\mathbf{u} = \pi_{\theta}(\mathbf{o}, \mathbf{h}) and v=Vπ(x,h)v = V^{\pi}(\mathbf{x}, \mathbf{h}). We optimize the policy using a customized version of the proximal policy optimization algorithm (PPO) [34]. Maximizing the PPO objective function requires learning hidden layer parameters θh\boldsymbol{\theta}_h that result in h\mathbf{h} evolving in response to the history of o\mathbf{o} and u\mathbf{u} in a manner that facilitates fast adaptation to environments encountered during optimization, as well as novel environments outside of the training distribution. The optimized policy then adapts real-time to conditions experienced during deployment. Importantly, the network parameters remain fixed during deployment with adaptation occurring through the evolution of h\mathbf{h}.

The network architectures used in this work are shown in Table 2, where nhin_{hi} is the number of units in layer ii, obs_dim is the observation dimension, and act_dim is the action dimension. The policy and value functions are periodically updated during optimization after accumulating trajectory rollouts of 30 simulated episodes.

Policy NetworkValue Network
Layer# unitsactivation# unitsactivation
hidden 110∗obs_dim10 * \mathrm{obs\_dim}tanh10∗obs_dim10 * \mathrm{obs\_dim}tanh
hidden 2nh1∗nh3\sqrt{n_{h1} * n_{h3}}tanhnh1∗nh3\sqrt{n_{h1} * n_{h3}}tanh
hidden 310∗act_dim10 * \mathrm{act\_dim}tanh5tanh
outputact_dim\mathrm{act\_dim}linear1linear

Table 2. Policy and Value Function network architecture

Meta-RL Problem Formulation for Gain Adaptation (FCS-3L-RL)

The agent observation o\mathbf{o} is shown in Eqs. (35), where αobs\alpha_{\mathrm{obs}}, βobs\beta_{\mathrm{obs}}, MobsM_{\mathrm{obs}}, hobsh_{\mathrm{obs}}, ωobs\omega_{\mathrm{obs}}, and aerr\mathbf{a}_{\mathrm{err}} are generated as shown Sections II.C.3, II.C.4, and II.C.6. δC\delta_C are the commanded channel deflections from Section II.C.8 and derived from policy actions as described in the following. GBFCS\mathbf{G}_{\mathrm{BFCS}} are the adaptive gain biases that modify the scheduled gains for the three-loop FCS. These biases are integrated from the policy actions, and must be included in the observation space. The observations are scaled to have zero mean and unit variance.

o=[αobsβobsMobshobsωobsaerrBδCGBFCS](35)\mathbf{o} = \left[\alpha_{\mathrm{obs}}\quad\beta_{\mathrm{obs}}\quad M_{\mathrm{obs}}\quad h_{\mathrm{obs}}\quad\omega_{\mathrm{obs}}\quad\mathbf{a}^{\mathrm{B}}_{\mathrm{err}}\quad\delta_C\quad\mathbf{G}_{\mathrm{BFCS}}\right] \tag*{(35)}

The output of the policy u=π(o)∈R12\mathbf{u} = \pi(\mathbf{o}) \in\mathbb{R}^{12} is scaled through multiplication by uscaleu_{\mathrm{scale}} and input to an integrator that saturates at ulimitu_{\mathrm{limit}}. The integrator output is then split into two vector components, each in R6\mathbb{R}^{6}, with each set biasing the pitch and yaw channel three loop autopilot gains. The integration improves performance by smoothing the action. We use uscale=18u_{\mathrm{scale}} = 18 and ulimit=0.3u_{\mathrm{limit}} = 0.3

The reward function is shown below in Equations (21a) through (40). rshapingr_{\mathrm{shaping}} is a shaping reward given at each step in an episode. These shaping rewards take the form of a Gaussian-like function of the norm of the scaled acceleration tracking error as shown in Eq. (21a). rctrlr_{\mathrm{ctrl}} is a control effort penalty, again given at each step in an episode, and rbonusr_{\mathrm{bonus}} is a bonus given at the end of an episode if the miss distance is below the threshold rlimr_{\mathrm{lim}}. Importantly, the current episode is terminated if a path constraint is violated, in which case the stream of positive shaping rewards is terminated, and the agent receives a negative reward. We use α=2\alpha= 2, β=−0.05\beta= -0.05, δ=−0.10\delta= -0.10, ϵ=10\epsilon= 10, ζ=−10\zeta= -10, rlim=3r_{\mathrm{lim}} = 3 m, and σacc=0.2\sigma_{\mathrm{acc}} = 0.2. We use a discount rate of 0.90 for shaping rewards and 0.995 for the terminal reward.

rshaping=αexp⁡(−(∥aerrB∥/amax⁡)2σacc2)(36)r_{\mathrm{shaping}} = \alpha\exp\left(-\frac{\left(\lVert\mathbf{a}^{\mathrm{B}}_{\mathrm{err}}\rVert/a_{\max}\right)^2}{\sigma_{\mathrm{acc}}^2}\right) \tag*{(36)}
rctrl=δu(37)r_{\mathrm{ctrl}} = \delta\mathbf{u} \tag*{(37)}
rbonus={ϵ,if rTM<rlim and done0,otherwise(38)r_{\mathrm{bonus}} = \begin{cases} \epsilon, & \text{if } r_{\mathrm{TM}} < r_{\mathrm{lim}}\text{ and done} \\ 0, & \text{otherwise} \end{cases} \tag*{(38)}
rpenalty={ζ,if any path constraint violated0,otherwise(39)r_{\mathrm{penalty}} = \begin{cases} \zeta, & \text{if any path constraint violated} \\ 0, & \text{otherwise} \end{cases} \tag*{(39)}
r=rshaping+rctrl+rbonus+rpenalty(40)r = r_{\mathrm{shaping}} + r_{\mathrm{ctrl}} + r_{\mathrm{bonus}} + r_{\mathrm{penalty}} \tag*{(40)}

The gain definitions are from [9], and defined in Table 3. Figure 10 illustrates the gain adaptation during an engagement with a jinking target. Note that the occasional bias saturation indicates that further performance improvement may have been possible with further hyperparameter tuning.

GainDefinition
KDCK_{\mathrm{DC}}Steady State
KAK_{\mathrm{A}}Acceleration feedback
ωI\omega_{\mathrm{I}}Forward path integrator
KRK_{\mathrm{R}}Rate feedback
K1K_{1}Aerodynamic acceleration
K3K_{3}Aerodynamic body rate

Table 3. Gain Definitions

Adaptive gain bias plots over engagement

Figure 10. Adaptive Gain Biases over Engagement

For optimization, aerodynamic uncertainty is modeled using a maximum coefficient bias of 10% and a maximum center of pressure shift equal to 1% (see § 2.D). A learning curve plotting the mean reward over rollouts versus episode is shown in Figure 11. Here each optimization rollout consists of 30 simulated episodes. The reward history is slightly noisy due to the fact that the exploration variance does not fall to zero during the optimization (it would be destabilizing to the PPO algorithm if it did fall to zero). Note that for testing, the exploration variance is disabled, and we return the mean value of the policy distribution. The maneuver distribution used for both optimization and testing is 33% step, 33% weave, and 34% jinking.

Optimization reward history for FCS-3L-RL

Figure 11. Optimization Reward History for FCS-3L-RL

Meta-RL Problem Formulation, Full FCS (FCS-RL)

The agent observation o\mathbf{o} is shown in Eqs. (41), where αobs\alpha_{\mathrm{obs}}, βobs\beta_{\mathrm{obs}}, MobsM_{\mathrm{obs}}, hobsh_{\mathrm{obs}}, ωobs\omega_{\mathrm{obs}}, and aerrB\mathbf{a}_{\mathrm{err}}^{\mathrm{B}} are generated as shown Sections II.C.3, II.C.4, and II.C.6. θroll\theta_{\mathrm{roll}} is obtained through integration of the xx component of the measured rotational velocity (ωobs\omega_{\mathrm{obs}}) and δC\delta_{\mathrm{C}} are the commanded channel deflections from Section II.C.8 and derived from policy actions as described in the following. The observations are scaled to have zero mean and unit variance.

o=[αobsβobsMobshobsωobsaerrBδCθroll](41)\mathbf{o} = \left[ \alpha_{\mathrm{obs}}\quad\beta_{\mathrm{obs}}\quad M_{\mathrm{obs}}\quad h_{\mathrm{obs}}\quad\omega_{\mathrm{obs}}\quad\mathbf{a}_{\mathrm{err}}^{\mathrm{B}}\quad\delta_{\mathrm{C}}\quad\theta_{\mathrm{roll}} \right] \tag*{(41)}

The output of the policy u=π(o)∈R3\mathbf{u} = \pi(\mathbf{o}) \in\mathbb{R}^{3} is split into three scalar components, a pitch channel deflection rate δ˙P\dot{\delta}_{P}, a yaw channel deflection rate δ˙Y\dot{\delta}_{Y}, and a roll channel deflection rate δ˙A\dot{\delta}_{A}. These are scaled by multiplication with the maximum allowed deflection rate and then clipped to fall within the allowed commanded deflection rates. The maximum deflection rates are 500°/s. These commanded deflection rates are then integrated, with the integrator saturating at the maximum allowed deflection, giving the commanded deflections δP\delta_{\mathrm{P}}, δY\delta_{\mathrm{Y}}, and δA\delta_{\mathrm{A}}

The reward function is shown below in Equations (23a) through (23f). rshapingr_{\mathrm{shaping}} is a shaping reward given at each step in an episode. These shaping rewards take the form of a Gaussian-like function of the norm of the scaled acceleration tracking error as shown in Eq. (36). rrollr_{\mathrm{roll}} encourages the agent to minimize deviations of body roll from zero, with differential fin deflections only used to counteract the rolling moment induced by angle of attack and sideslip (See § 2.D). rctrlr_{\mathrm{ctrl}} is a control effort penalty, again given at each step in an episode, and rbonusr_{\mathrm{bonus}} is a bonus given at the end of an episode if the miss distance is below the threshold rlimr_{\mathrm{lim}}. Importantly, the current episode is terminated if a path constraint is violated, in which case the stream of positive shaping rewards is terminated, and the agent receives a negative reward. We use α=2\alpha= 2, β=−0.05\beta= -0.05, δ=−0.10\delta= -0.10, ϵ=10\epsilon= 10, ζ=−10\zeta= -10, rlim=3r_{\mathrm{lim}} = 3 m, σΩ=0.02\sigma_{\Omega} = 0.02, and σacc=0.2\sigma_{\mathrm{acc}} = 0.2. We use a discount rate of 0.90 for shaping rewards and 0.995 for the terminal reward.

rshaping=αexp⁡(−(∥aerrB∥/amax⁡)2σacc2)(42)r_{\mathrm{shaping}} = \alpha\exp\left(-\frac{\left(\lVert\mathbf{a}_{\mathrm{err}}^{\mathrm{B}} \rVert/ a_{\max}\right)^2}{\sigma_{\mathrm{acc}}^2}\right) \tag*{(42)}
rroll=β∣θroll∣(43)r_{\mathrm{roll}} = \beta\lvert\theta_{\mathrm{roll}} \rvert \tag*{(43)}
rctrl=δ∥∥ddt(δP)ddt(δY)ddt(δA)∥∥(44)r_{\mathrm{ctrl}} = \delta\left\lVert\left\lVert\frac{\mathrm{d}}{\mathrm{d}t}(\delta_{\mathrm{P}})\quad\frac{\mathrm{d}}{\mathrm{d}t}(\delta_{\mathrm{Y}})\quad\frac{\mathrm{d}}{\mathrm{d}t}(\delta_{\mathrm{A}}) \right\rVert\right\rVert \tag*{(44)}
rbonus={ϵ,if rTM<rlim and done0,otherwise(45)r_{\mathrm{bonus}} = \begin{cases} \epsilon, & \text{if } r_{\mathrm{TM}} < r_{\mathrm{lim}} \text{ and done} \\ 0, & \text{otherwise} \end{cases} \tag*{(45)}
rpenalty={ζ,if any path constraint violated0,otherwise(46)r_{\mathrm{penalty}} = \begin{cases} \zeta, & \text{if any path constraint violated} \\ 0, & \text{otherwise} \end{cases} \tag*{(46)}
r=rshaping+rctrl+rroll+rbonus+rpenalty(47)r = r_{\mathrm{shaping}} + r_{\mathrm{ctrl}} + r_{\mathrm{roll}} + r_{\mathrm{bonus}} + r_{\mathrm{penalty}} \tag*{(47)}

We use curriculum learning [35] to optimize a policy that adapts to high levels of aerodynamic uncertainty. Specifically, we first optimize the policy using coefficient bias bounds (CB) of 10% and center of pressure location bounds (CP) of 1% (see § 2.D). The resulting policy is then re-optimized using CB=20% and CP=2%, but with the policy initial exploration standard deviation set to 0.30 (as opposed to 1.00 for the initial optimization), allowing the second optimization run to retain learning from the first optimization, but giving the policy enough exploration ability to improve performance under the higher aerodynamic uncertainty. We then repeat this incremental optimization a third time, using CB=40% and CP=2%. This results in improved adaptability to aerodynamic uncertainty compared to trying a single full optimization using CB=40% and CP=2%. The reward learning curve for the initial optimization is shown in Figure 12, where "Mean" is the mean reward over a set of rollouts. Before episode 10000, the agent is primarily learning to avoid constraint violations, which result in early termination of an episode. The maneuver distribution used for both optimization and testing is 33% step, 33% weave, and 34% jinking.

Optimization reward history

Figure 12. Optimization Reward History

Experiments

FCS Relative Performance

Relative performance is measured by running 5000 simulations for each case in the following tables. In the following tables, "AREF" (in g) is the reference acceleration (see § 2.B) for the target, "CB" is the aerodynamic coefficient bias, "CP" is the aerodynamic center of pressure shift (see § 2.D), and "R" is the maximum radome refraction in radians (see § 2.C.1). The FCS-RL, FCS-3L, and FCS-3L-RL use the labels ’RL’, ’3L’, and ’3L-A’, respectively.

Relative performance is quantified using several metrics. Accuracy is quantified by "Miss < 25 cm", which indicates the percentage of episodes that resulted in the interceptor achieving a miss distance less than or equal to 25 cm. "IR" is the interceptor ratio, which denotes the number of interceptors using the FCS-3L and FCS-3L-RL that are required to match the accuracy of an interceptor using the FCS-RL, according to the equation

IR=log⁡m1log⁡m2\mathrm{IR} = \frac{\log m_1}{\log m_2}

where m1m_1 and m2m_2 are the probability of a miss distance greater than 25 cm for the FCS-RL and either FCS-3L or FCS-3L-RL, respectively. Control activity is measured using the root-mean-square of the magnitude of the interceptor acceleration "RMS ∥aM∥\lVert\mathbf{a}_M\rVert". Finally, "Vio (%)" gives the percentage of episodes that resulted in a load constraint violation. Table 4 shows that under conditions of no radome refraction, no aerodynamic uncertainty, and no target evasive acceleration, there is essentially no performance difference between the three flight control systems. To demonstrate the need for a recurrent policy in this application, we optimized a version of FCS-RL with the recurrent layer replaced with a standard feed forward layer (RL-MLP). The results clearly demonstrate the need for a recurrent policy (which allows realization of an integrator) when applying RL to flight control over a large flight envelope.

CaseFCSMiss < 25 cm (%)IRRMS ∥aM∥\lVert\mathbf{a}_M\rVert (m/s2^2)Vio (%)
AREF=0 CB=0.0-CP=0.0 R=03L1001.0410.0
AREF=0 CB=0.0-CP=0.0 R=03L-RL1001.0410.0
AREF=0 CB=0.0-CP=0.0 R=0RL100-400.0
AREF=0 CB=0.0-CP=0.0 R=0RL-MLP6-14434.1

Table 4. Ideal Conditions

Table 5 tabulates performance when we consider radome refraction (§ 2.C.1). Radome refraction results in parasitic interaction between the guidance and flight control systems that reduces performance. Except for the special case of a hemispherical radome, the actual LOS refraction depends on where the LOS passes through the radome. Furthermore, in order for the FCS to track a commanded acceleration, an interceptor using control surface actuation must rotate. This rotation results in the LOS to the target passing through a different part of the radome, causing the refraction angle to change, resulting in a false sense of target motion. The guidance system then reacts to this by giving a commanded acceleration that takes into account the false target motion, completing the parasitic feedback loop. We see that even with small radome refraction a performance differential appears between the flight control systems, with the differential growing as the radome refraction is increased.

CaseFCSMiss < 25 cm (%)IRRMS ∥aM∥\lVert a_M \rVert (m/s2^2)Vio (%)
AREF=4 CB=0.10-CP=0.01 R=0.00053L702.5610.0
AREF=4 CB=0.10-CP=0.01 R=0.00053L-A801.9590.0
AREF=4 CB=0.10-CP=0.01 R=0.0005RL95-560.0
AREF=4 CB=0.10-CP=0.01 R=0.0013L602.5600.0
AREF=4 CB=0.10-CP=0.01 R=0.0013L-A701.9590.0
AREF=4 CB=0.10-CP=0.01 R=0.001RL90-560.0
AREF=4 CB=0.10-CP=0.01 R=0.0023L352.7630.0
AREF=4 CB=0.10-CP=0.01 R=0.0023L-A452.0620.0
AREF=4 CB=0.10-CP=0.01 R=0.002RL69-570.0
AREF=4 CB=0.10-CP=0.01 R=0.0053L112.7660.0
AREF=4 CB=0.10-CP=0.01 R=0.0053L-A151.9650.0
AREF=4 CB=0.10-CP=0.01 R=0.005RL27-630.0

Table 5. Radome Sensitivity

For the simulations used to create Table 6 we increase AREF from 4 to 8, doubling the target’s acceleration at a given dynamic pressure. Again, there is a clear difference between the performance of the three flight control systems. Next, Table 7 explores which target maneuvers are more difficult for the flight control systems. Whereas the performance for all flight control systems is nearly identical for the step maneuver, we see that a large performance differential exists for weaving and jinking maneuvers. Importantly, these types of maneuvers are likely to be employed by a target that is actively maneuvering to avoid interception, as it keeps the threat from deviating too far from the trajectory required to hit its target. Moreover, it is possible for certain types of reentry vehicles to spiral even without the capability of actively evading interception [36, 37].

CaseFCSMiss < 25 cm (%)IRRMS ∥aM∥\lVert a_M \rVert (m/s2^2)Vio (%)
AREF=8 CB=0.10-CP=0.013L522.4950.0
AREF=8 CB=0.10-CP=0.013L-A532.3960.0
AREF=8 CB=0.10-CP=0.01RL83-910.0

Table 6. Increased Target Evasive Acceleration (R=0.0005)

CaseFCSMiss < 25 cm (%)IRRMS ∥aM∥\lVert a_M \rVert (m/s2^2)Vio (%)
Step3L981.2650.0
Step3L-A971.3650.0
StepRL99-460.0
Weave3L474.7540.0
Weave3L-A652.9520.0
WeaveRL95-470.0
Jink3L662.7630.0
Jink3L-A751.7610.0
JinkRL90-570.0

Table 7. Target Maneuver Sensitivity (AREF=4, CB=0.10, CP=0.01, R=0.0005)

Table 8 reports performance with various target weave periods, ranging from 0.1 to 1.0 seconds. We see that the weave period that maximizes miss distance is different between the three flight control systems. To see why, consider that in [38] an expression is derived showing peak miss distance as a function of normalized target weave frequency, specifically ωTT\omega_T T, where ωT\omega_T and TT are the target weave frequency and guidance system time constant, respectively. When taking into account FCS dynamics, the peak miss distance becomes a function of the effective time constant of the combined guidance and control system. Indeed, for a given guidance system, a decrease in flight control response time can improve effectiveness against weaving targets [38]. Importantly, although all three flight control system’s performance are impacted by weaving targets, there is again a clear performance gap between the three systems. The performance difference is at least partly due to the improved flight control response time of the FCS-RL (See Figure 13 in the following). It is also possible that the FCS-RL is dynamically adapting its effective bandwidth to increase performance.

Weave Period (s)FCSMiss < 25 cm (%)IRRMS ∥aM∥\lVert a_M \rVert (m/s2^2)Vio (%)
0.13L1001.0420.0
0.13L-A1001.0410.0
0.1RL100-400.0
0.503L501.4450.0
0.503L-A451.7480.0
0.50RL63-450.0
1.003L247.7530.0
1.003L-A305.9530.0
1.00RL88-450.0

Table 8. Target Weave Period (AREF=4, CB=0.10, CP=0.01, R=0.0005)

Table 9 illustrates the impact of increasing the level of aerodynamic uncertainty. We see that although all three FCS are robust to aerodynamic uncertainty, the performance advantage of both the FCS-RL and FCS-3L-RL are maintained as compared to FCS-3L, with a higher hit rate, lower control effort, and fewer constraint violations Finally, Table 10 gives interceptor and target statistics computed over 5000 trajectories, where we see that the acceleration of the interceptor and target are quite close at the end of the engagement, making this a highly agile target. Since the target starts at a much higher altitude than the interceptor, the lower dynamic pressure gives the target less maneuverability earlier in the engagement.

Cumulative RMS ∥aMB[y,z]∥\lVert\mathbf{a}^{\mathrm{B}}_{\mathrm{M}}[y,z]\rVert (m/s2^2)40
Cumulative RMS ∥aT∥⊥λTM\lVert\mathbf{a}_{\mathrm{T}}\rVert\perp\lambda_{\mathrm{TM}} (m/s2^2)12
Terminal RMS ∥aMB[y,z]∥\lVert\mathbf{a}^{\mathrm{B}}_{\mathrm{M}}[y,z]\rVert (m/s2^2)58
Terminal RMS ∥aT∥⊥λTM\lVert\mathbf{a}_{\mathrm{T}}\rVert\perp\lambda_{\mathrm{TM}} (m/s2^2)51
Maximum Terminal Altitude hh (m)31516
Minimum Terminal Altitude hh (m)12838

Table 10. Typical High Altitude Engagement Statistics

CaseFCSMiss < 25 cm (%)IRRMS ∥aM∥\lVert a_M \rVert (m/s2^2)Vio (%)
AREF=4 CB=0.10-CP=0.013L702.5610.0
AREF=4 CB=0.10-CP=0.013L-A801.9590.0
AREF=4 CB=0.10-CP=0.01RL95-560.0
AREF=4 CB=0.20-CP=0.023L662.5620.6
AREF=4 CB=0.20-CP=0.023L-A771.8600.4
AREF=4 CB=0.20-CP=0.02RL93-570.0
AREF=4 CB=0.40-CP=0.033L601.9653.4
AREF=4 CB=0.40-CP=0.033L-A701.5633.2
AREF=4 CB=0.40-CP=0.03RL84-600.8

Table 9. Aerodynamic Uncertainty (R=0.0005)

Step response plots at 25 km altitude and 2500 m/s comparing the FCS-RL and FCS-3L under nominal, "slow", and "fast" aerodynamic models are shown in Figure 13. The slow model has a reduction in aerodynamic force and moment coefficients and a shift in CP giving increased static margin, whereas the fast model has an increase in aerodynamic coefficients and reduced static margin. Although this is modeled as a shift in CP, it would be equivalent to a shift in CG in the opposite direction. We see that the FCS-RL exhibits a significant reduction in flight control response time as compared to the benchmark. A full trajectory for the FCS-RL starting at 25 km altitude with 0∘0^\circ inclination to target is shown in Figure 14.

FCS-RL sample trajectory

Figure 14. FCS-RL sample trajectory

Policy Generalization Testing

We demonstrate the generalization capability of the three flight control systems by modifying the engagement initial conditions, as shown in Table 11. Note that in any real application, the homing phase is effective over some "basket" of initial conditions consistent with the end of the mid-course guidance phase. This basket is limited by a minimum range-to-target that gives the interceptor enough time to complete a successful terminal phase intercept, with the minimum time being a function of target maneuver, closing speed, and initial heading error. Using these modified initial conditions, we test the same target maneuver distribution as in § 12.A, and also test against two target maneuver types not seen during optimization, a vertical S maneuver [39] (vert-S) and frequency modulated weave (FM-weave). The vertical S maneuver is identical to the weave maneuver except that the target acceleration instantaneously transitions between minimum and maximum acceleration at the weave maneuver zero acceleration crossings. In our simulator the vertical S differs from the jinking maneuver in that the jinking maneuver does not switch sign with a fixed period, and the jinking is independent for each axis. The frequency modulated weave has a random variation in the weave frequency over each engagement. This would be a relatively simple maneuver to execute for a maneuverable re-entry vehicle that is already spiraling with the fin deflections used to modulate the spiraling frequency.

Elevation Angle to Target θE\theta_E (degrees)3060
Azimuth Angle to Target θA\theta_A (degrees)-180180
Altitude (m)1500022000
Missile Velocity Magnitude ∥vM∥\lVert\mathbf{v}_M\rVert (m/s)22002800
Initial Range to Targeted Asset ∥rAT∥\lVert\mathbf{r}_{AT}\rVert (m)1500030000
Target Velocity Magnitude ∥vT∥\lVert\mathbf{v}_T\rVert (m/s)25004500
Target Velocity Cone Half Apex Angle θvT\theta_{v_T} (degrees)020

Table 11. Simulation Modified Initial Conditions)

Example target acceleration trajectories for the FM-weave maneuver are illustrated in Figure 15.

Frequency Modulated Weave Sample Trajectories

Figure 15. Frequency Modulated Weave Sample Trajectories

From Table 12 we see that all three flight control systems are robust to an increase in both the range of initial conditions and uncertainty in the initial state estimate. The vertical-S maneuver is challenging for all three flight control systems, but more so for the FCS-3L and FCS-3L-RL. The key takeaway is that the RL FCS policy generalizes well to both initial conditions and target maneuvers outside of the optimization distribution. The FCS-RL performance is not affected by the FM-weave maneuver, whereas the FCS-3L and FCS-3L-RL are significantly reduced.

CaseUKFMiss < 25cm (%)RMS ∥aM∥\lVert\mathbf{a}_M\rVert (m/s2^2)IRVio (%)
Mod-IC3L73491.90.0
Mod-IC3L-A83491.40.0
Mod-ICRL9246-0.0
Mod-IC vert-S3L42492.20.0
Mod-IC vert-S3L-A47581.90.0
Mod-IC vert-SRL7045-0.0
Mod-IC FM-weave3L58472.90.0
Mod-IC FM-weave3L-A54523.30.0
Mod-IC FM-weaveRL9243-0.0

Table 12. Generalization Cases

Flight Certification

Linear system theory can provide theoretical stability guarantees for the FCS-3L in the simplified case of a linearized system. However, optimal gain scheduling requires evaluating performance in a more realistic non-linear simulator [40]. Importantly, it is possible that for the case of a non-linear system, the system can be unstable at some flight condition while at the same time passing stability checks using the linearized system. This becomes more likely as the aerodynamics become more non-linear (which can be the case in the hypersonic flight regime). In addition, the actual deployment aerodynamics can be quite different from the simulated aerodynamics, weakening stability arguments based on sufficient gain or phase margin. Consequently, the FCS is not actually provably stable using linear systems theory when coupled with non-linear aerodynamics and significant aerodynamic uncertainty. Although a sufficiently large gain margin could compensate for differences between the actual and linearized systems, this comes with a performance cost. Consequently, extensive Monte Carlo simulation is required before moving to a flight test, which is also possible using the FCS-RL optimization process. Importantly, there has been recent work on the certification of stability for a meta-RL policy [41]. Finally, although the FCS-3L-RL underperformed the FCS-RL, it has the advantage of allowing flight certification using linear stability metrics, which could easily be implemented by limiting the gain bias to ensure that the resulting gains result in adequate margin for the linear system. This would allow a similar flight certification procedure as current systems.

Conclusion

We applied reinforcement meta-learning (meta-RL) to two approaches to flight control system (FCS) optimization. In the first approach (FCS-RL) we optimize an adaptive policy directly mapping commanded acceleration tracking error and sensor outputs to commanded fin deflection rates in the pitch, yaw, and roll channels. In the second approach (FCS-3L-RL), the scheduled gains for a standard FCS (FCS-3L) using the three-loop autopilot for pitch and yaw control are adaptively adjusted during a trajectory using a meta-RL gain adaptation policy. The three flight control systems were then extensively tested in a 6-DOF simulator that modeled most factors relevant to flight control. Under ideal conditions, the performance of the three flight control systems was nearly identical. However, under conditions of radome refraction, target evasive maneuvers, and aerodynamic uncertainty, we found a significant performance differential appear in favor of the FCS-RL, with higher interception rates and lower control effort. We believe that there are two primary factors behind the FCS-RL performance. The first is the higher expressive power of the recurrent policy compared to a fixed three-loop topology, which allows for faster flight control response time, as measured by step response rise time. The second is the adaptability of the FCS-RL achieved by optimizing over an ensemble of aerodynamic models, and improved through curriculum learning. It is likely that the integrated pitch, yaw, and roll channel control has a smaller effect, but would have a much greater impact in bank-to-turn applications where aerodynamic coupling is higher. To put the performance differential in perspective, facing an actively evading target under small levels of radome refraction and aerodynamic uncertainty, the interception rate for the FCS-RL and FCS-3L are 95% and 70% respectively. This implies that 2.5 interceptors using the benchmark system would be required to reach the FCS-RL hit rate for a single interceptor. In other more challenging scenarios, the FCS-3L required more than 5 interceptors to match the FCS-RL performance. In addition to the obvious cost advantage, this increased accuracy can be critical in scenarios with a limited number of interceptors and/or launchers. Although FCS-3L-RL also demonstrates improved performance compared to FCS-3L-RL, the performance differential is smaller than FCS-RL. Finally, we demonstrate that both the FCS-3L-RL and the FCS-RL generalize well to initial conditions and target maneuvers that were not experienced during optimization.

References

  1. [1]Alviani, R., Poggie, J., and Blaisdell, G., “Unsteady Aspects of Shock-Wave/Boundary-Layer Interaction Resulting from Control Surface Deflection,” AIAA Journal, Vol. 60, No. 8, 2022, pp. 4649–4659. https://doi.org/10.2514/1.J061337
  2. [2]Edwards, J. R., “Numerical Simulations of Shock/Boundary Layer Interactions Using Time-Dependent Modeling Techniques: A Survey of Recent Results,” Progress in Aerospace Sciences, Vol. 44, No. 6, 2008, pp. 447–465. https://doi.org/10.1016/j.paerosci.2008.06.004
  3. [3]Witeof, Z., and Pasilaio, C. L., “Fluid-Thermal-Structural Interaction Effects in Preliminary Design of High Speed Vehicles,” 56th AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, 2015. https://doi.org/10.2514/6.2015-1631
  4. [4]Falkiewicz, N. J., Cesnik, C. E. S., Bolender, M. A., and Doman, D. B., “Thermoelastic Formulation of a Hypersonic Vehicle Control Surface for Control-Oriented Simulation,” AIAA Guidance, Navigation, and Control Conference, 2009. https://doi.org/10.2514/6.2009-6284
  5. [5]Culler, A. J., and McNamara, J. J., “Impact of Fluid-Thermal-Structural Coupling on Response Prediction of Hypersonic Skin Panels,” AIAA Journal, Vol. 49, No. 11, 2011, pp. 2393–2406. https://doi.org/10.2514/1.J050617
  6. [6]Zarchan, P., “Tactical and strategic missile guidance,” American Institute of Aeronautics and Astronautics, Inc., 2012, pp. 126–132. https://doi.org/10.2514/4.868948
  7. [7]Siouris, G. M., “Missile guidance and control systems,” Springer Science & Business Media, 2004, pp. 142–143. https://doi.org/10.1007/b97614
  8. [8]Shneydor, N. A., “Missile guidance and pursuit: kinematics, dynamics and control,” Elsevier, 1998, pp. 151–155. https://doi.org/10.1533/9781782420590.
  9. [9]Zarchan, P., “Tactical and strategic missile guidance,” American Institute of Aeronautics and Astronautics, Inc., 2012, pp. 529–564. https://doi.org/10.2514/4.868948
  10. [10]Stallard, D. V., “An Approach to Autopilot Design for Homing Interceptor Missiles,” AIAA Guidance, Navigation, and Control Conference, AIAA, 1991, pp. 99–113.DOI
  11. [11]Garnell, P., Guided Weapon Control Systems, 2nd ed., Pergamon Press, Oxford, 1980.DOI
  12. [12]Lee, H., Snyder, S., and Hovakimyan, N., “L₁ Adaptive Output Feedback Augmentation for Missile Systems,” IEEE Transactions on Aerospace and Electronic Systems, Vol. 54, No. 2, 2018, pp. 680–692. https://doi.org/10.1109/TAES.2017.2764218
  13. [13]Rusnak, I., Weiss, H., and Barkana, I., “Improving the Performance of Existing Missile Autopilot Using Simple Adaptive Control,” International Journal of Adaptive Control and Signal Processing, Vol. 28, No. 7-8, 2014, pp. 732–749. https://doi.org/10.1002/acs.2457
  14. [14]Joshi, S. M., Tao, G., and Khong, T., “A Direct Adaptive Control Approach in the Presence of Model Mismatch,” AIAA Guidance, Navigation, and Control Conference, AIAA, Chicago, Illinois, 2009. https://doi.org/10.2514/6.2009-5620
  15. [15]Kannan, S. K., and Johnson, E. N., “Model Reference Adaptive Control with a Constrained Linear Reference Model,” 49th IEEE Conference on Decision and Control, IEEE, 2010, pp. 48–53. https://doi.org/10.1109/CDC.2010.5717275
  16. [16]Rohrs, C. E., Valavani, L., Athans, M., and Stein, G., “Robustness of Continuous-Time Adaptive Control Algorithms in the Presence of Unmodeled Dynamics,” IEEE Transactions on Automatic Control, Vol. 30, No. 9, 1985, pp. 881–889. https://doi.org/10.1109/TAC.1985.1104070
  17. [17]Cao, C., and Hovakimyan, N., “Design and Analysis of a Novel L₁ Adaptive Control Architecture with Guaranteed Transient Performance,” IEEE Transactions on Automatic Control, Vol. 53, No. 2, 2008, pp. 586–591. https://doi.org/10.1109/TAC.2007.914282
  18. [18]Shin, H.-S., He, S., and Tsourdос, A., “A Domain-Knowledge-Aided Deep Reinforcement Learning Approach for Flight Control Design,”, 2019. URL https://arxiv.org/abs/1908.06884
  19. [19]Taligatos, R., Nguyen, D. H., and Lowenberg, M. H., “Investigating Missile Feedback Control: Reinforcement Learning, PID, or Both?” AIAA SCITECH 2026 Forum, AIAA, 2026. https://doi.org/10.2514/6.2026-1162
  20. [20]Candeli, A., De Tommasi, G., Lui, D. G., Mele, A., Santini, S., and Tartaglione, G., “A Deep Deterministic Policy Gradient Learning Approach to Missile Autopilot Design,” IEEE Access, Vol. 10, 2022, pp. 19685–19696. https://doi.org/10.1109/ACCESS.2022.3150926
  21. [21]Wang, W., Wu, M., Chen, Z., and Liu, X., “Integrated Guidance-and-Control Design for Three-Dimensional Interception Based on Deep-Reinforcement Learning,” Aerospace, Vol. 10, No. 2, 2023, p. 167. https://doi.org/10.3390/aerospace10020167
  22. [22]Gaudet, B., and Furfarо, R., “A Comparison of Partially and Fully Integrated Guidance and Flight Control Optimized with Reinforcement Meta-Learning,” AIAA SCITECH 2023 Forum, 2023, p. 1628.DOI
  23. [23]Gaudet, B., Charcos, I., and Furfaro, R., “Integrated and Adaptive Guidance and Control for Endoatmospheric Missiles via Reinforcement Learning,” arXiv preprint arXiv:2109.03880, 2021.DOI
  24. [24]Gaudet, B., Durant, D., and Furfaro, R., “Reinforcement Learning for Target Jerk Estimation in Interceptor Homing Phase,”, 2026. URL https://arxiv.org/abs/2611.15923
  25. [25]Gaudet, B., Furfaro, R., Linares, R., and Scorsoglio, A., “Reinforcement Metalearning for Interception of Maneuvering Exoatmospheric Targets with Parasitic Attitude Loop,” Journal of Spacecraft and Rockets, Vol. 58, No. 2, 2021, pp. 386–399.DOI
  26. [26]Shneydor, N. A., “Missile guidance and pursuit: kinematics, dynamics and control,” Elsevier, 1998, p. 118. https://doi.org/10.1533/9781782420590
  27. [27]Vasile, J. D., Bryson, J. T., Sahu, J., Paul, J. L., and Gruenwald, B. C., “Aerodynamic Dataset Generation of a Long-Range Projectile,” Tech. rep., ARMY RESEARCH LAB ABERDEEN PROVING GROUND MD ABERDEEN PROVING GROUND United . . ., 2020.
  28. [28]Jorgensen, L. H., “A method for estimating static aerodynamic characteristics for slender bodies of circular and noncircular cross section alone and with lifting surfaces at angles of attack from 0 deg to 90 deg,” 1973.
  29. [29]Zarchan, P., “Tactical and strategic missile guidance,” American Institute of Aeronautics and Astronautics, Inc., 2012, pp. 473–482. https://doi.org/10.2514/4.868948
  30. [30]Zarchan, P., “Tactical and strategic missile guidance,” American Institute of Aeronautics and Astronautics, Inc., 2012, pp. 18–21. https://doi.org/10.2514/4.868948
  31. [31]Zarchan, P., “Advanced Tactical and Strategic Missile Guidance, Volume 2,” American Institute of Aeronautics and Astronautics, Inc., 2012, pp. 1–29. https://doi.org/10.2514/4.105388
  32. [32]Furfaro, R., and Gaudet, B., “Deep Learning Framework for Rapid development of High Performance Hypersonic Weapons, 2021-002 T2, GN&C Working Group Final Report,” University Consortium for Applied Hypersonics, Contract: HQ0034229CA10, 2026. Final report is CUI, contact UCAH for access. Report will appear in DTIC. PI: Furfaro, Working group lead: Gaudet.
  33. [33]Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., and Botvinick, M., “Learning to reinforcement learn,” arXiv preprint arXiv:1611.05763, 2016.arxiv.org/abs/1611.05763
  34. [34]Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O., “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
  35. [35]Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M. E., and Stone, P., “Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey,” Journal of Machine Learning Research, Vol. 21, No. 181, 2020, pp. 1–50.arxiv.org/abs/2003.04960
  36. [36]Chadwick, W. R., and Zarchan, P., “Interception of Spiraling Ballistic Missiles,” Proceedings of the American Control Conference, IEEE, Seattle, Washington, USA, 1995, pp. —. Discusses guidance and interception performance against spiraling ballistic reentry vehicles.
  37. [37]Platus, D. H., “Ballistic Re-entry Vehicle Flight Dynamics,” Journal of Guidance and Control, Vol. 5, No. 1, 1982, pp. 4–16. https://doi.org/10.2514/3.56131, analysis of ballistic reentry vehicle aerodynamic and rotational dynamics during atmospheric reentry.
  38. [38]Zarchan, P., “Tactical and strategic missile guidance,” American Institute of Aeronautics and Astronautics, Inc., 2012, pp. 439–471. https://doi.org/10.2514/4.868948
  39. [39]Zarchan, P., “Tactical and strategic missile guidance,” American Institute of Aeronautics and Astronautics, Inc., 2012, pp. 120–121. https://doi.org/10.2514/4.868948
  40. [40]Jackson, P. B., “Overview of Missile Flight Control Systems,” Johns Hopkins APL Technical Digest, Vol. 29, No. 1, 2010, pp. 9–24. URL https://www.jhuapl.edu/sites/default/files/2024-09/29-01-Jackson.pdf
  41. [41]Scorsoglio, A., D’Ambrosio, A., and Furfaro, R., “[Article Title],” Journal of Guidance, Control, and Dynamics, 2026, pp. 1–17. https://doi.org/10.2514/1.G009769 URL https://doi.org/10.2514/1.G009769 articles in Advance.

Paper details

vlnc:W_54dtavte
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Engineering

Not recorded for this paper

View statement
Looking for related papers…
Contents