WorksEngineering
Reinforcement Learning for Null-Space Exploitation in Pseudo-Inverse Control Allocation
Abstract
Pseudo-inverse methods are widely used to solve control allocation problems for overactuated aircraft with redundant control effectors. These methods reliably satisfy the primary command tracking objective, while the remaining null-space degrees of freedom are in principle available for secondary objectives such as drag reduction or actuator wear balancing. However, exploiting this freedom in practice requires hand-tuned weighting matrices or a priori preferred deflections that are difficult to specify across a wide flight envelope with nonlinear, state-dependent dynamics. This paper presents a hybrid architecture in which a reinforcement learning agent operates exclusively within the null space of the control effectiveness matrix, learning secondary objective strategies directly from interaction with the flight dynamics. The null-space projector guarantees by construction that primary command tracking remains invariant regardless of the agent output, providing a principled separation between certified primary control and adaptive secondary optimization. The approach is demonstrated on a nonlinear six-degree-of-freedom distributed electric propulsion aircraft model with aero-propulsive effects across diverse mission scenarios, achieving measurable drag reduction compared to the baseline allocation while preserving tracking integrity and respecting actuator constraints. The proposed architecture forms a foundation for future extensions toward multi-objective, mission-conditioned policies.
Introduction
Modern engineered systems are increasingly designed with more control effectors than independent degrees of freedom to be controlled, resulting in overactuated configurations. Examples range from autonomous robots with several effectors to satellites equipped with clusters of reaction wheels and fighter aircraft with redundant aerodynamic surfaces [1], [2]. This redundancy is purposeful by design: a larger actuator set improves fault tolerance and reconfiguration capability, and is often preferred for reasons of safety, accuracy, bandwidth, and maintainability. In complex platforms, effector groups may serve multiple control objectives simultaneously; for example, in aircraft, engines are primarily designed for thrust yet can be commanded to generate a yawing moment when conventional aerodynamic surfaces are insufficient or unavailable [3], [4]. Control allocation constitutes the intermediate layer between the high-level controller and the individual actuators, translating a commanded virtual input such as a desired moment vector into a feasible set of effector deflections while respecting physical position and rate constraints as illustrated in Figure 1.

Figure 1. Control system architecture with control allocation. The high level motion controller commands a virtual effort vector ; the allocation layer distributes this demand across individual effectors
When the actuator set is redundant, the allocation problem admits infinitely many solutions, and the surplus degrees of freedom can in principle be directed toward secondary objectives such as minimizing drag, reducing power consumption, limiting actuator wear, or preserving a low radar cross-section. Classical methods including pseudoinverse-based allocation [3], null-space augmentation, and iterative redistribution strategies [5] reliably solve the primary task, but secondary objective optimization remains dependent on fixed weighting schemes or manual tuning that do not generalize across flight conditions and mission requirements. When multiple competing objectives must be balanced under nonlinear, state dependent dynamics, a learning based approach that discovers effective allocation strategies directly from interaction with the flight dynamics becomes attractive.
Related Work
The decomposition of the control allocation solution into a pseudo-inverse particular solution and a null-space component was first formalized by Bordignon and Durham [6], [7], who introduced the null-space intersection method for constrained allocation of overactuated aircraft. Durham et al. [8] subsequently exploited this freedom for drag minimization by projecting cost gradients into the null space of the control effectiveness matrix. The comprehensive treatment by Durham, Bordignon, and Beck [3] established the geometric theory relating the null-space structure to the Attainable Moment Set.
Several authors have applied null-space methods to specific secondary objectives in flight control. Stolk and de Visser [9] developed minimum-drag control allocation for the Innovative Control Effector (ICE) tailless fighter, combining weighted pseudo-inverse prioritization with gradient descent in the null space toward minimum-drag deflections. Pfeifle and Fichter [10] derived a closed-form null-space increment for minimum propeller shaft power in over-actuated transition aircraft. Pollack and van Kampen [11] extended incremental control allocation for secondary objectives using the null-space degrees of freedom.
Data-driven methods have been increasingly applied to control allocation, with neural networks trained to approximate or replace classical allocators. Kang et al. [12] train a neural allocator for a tailless aircraft that replicates a dynamic allocation scheme while avoiding heavy online optimization. Madruga et al. [13] use offline-trained networks to substitute the fixed allocation matrix on a quadrotor, compensating for aerodynamic effects without explicit inflow models. Chen et al. [14] combine an adaptive neural attitude controller with a constrained allocation layer to handle uncertainties and time-varying disturbances. Reinforcement learning extends this direction by learning allocation policies directly from interaction with the environment: de Visser et al. [15] apply RL to distribute control commands across 13 effectors for longitudinal control without a preidentified effectiveness matrix. However, these approaches are fundamentally black-box function approximators in which the mapping from state to actuator command is not transparent. Secondary objectives and constraints are enforced indirectly through reward or loss shaping, behavior under distribution shift is difficult to bound, and failure diagnosis is hindered by limited interpretability. In safety-critical applications such as flight control, these limitations present significant barriers to verification and certification that often outweigh empirical performance gains.
While the present work was in preparation, Isci and Koyuncu [16] recently proposed a hybrid allocation scheme for the ICE aircraft in which a reinforcement-learning agent adjusts the effector weights of an incremental quadratic programming allocator. Since a weight change only redistributes effort among solutions that satisfy the demanded moment, primary tracking is preserved while the actuator constraints remain inactive, and the agent is trained to reduce allocation error. The present work differs in the nature of the secondary objective. In their scheme the objective is internal to the allocator, the allocation error itself, whereas the secondary objectives considered here are external and mission-dependent, such as energy, actuator wear, and acoustic noise, which are nonlinear functions of the operating point rather than quantities the allocator already represents. The two directions are therefore complementary: their weight adaptation refines the allocator’s internal behavior, while the present approach exploits the null space to optimize external, mission-level objectives.
This paper proposes a hybrid architecture that preserves the reliability of classical pseudo-inverse allocation for primary command tracking while employing a reinforcement learning agent to optimize the null-space component for secondary objectives. The null-space projector guarantees by construction that the learned policy cannot degrade the primary control response, regardless of the agent output. This architectural separation provides a principled safety boundary between certified primary control and adaptive secondary optimization, addressing the interpretability concerns of purely data-driven allocators while retaining the capacity to learn complex, state-dependent strategies. The approach is demonstrated on a nonlinear six-degree-of-freedom model of a distributed-electric-propulsion transport aircraft including aeropropulsive effects in a high-fidelity simulation environment.
Problem Formulation
Consider an overactuated aircraft with control effectors and controlled axes, where . The flight control system generates a required command based on the flight computer demands [3]. In the linear control allocation case, the relationship between the control effector deflections and the achieved command is expressed as
where is the control effectiveness matrix, whose entries represent the partial derivatives of the achieved command with respect to each control effector deflection. The primary objective of control allocation is to find such that
Since , the system (2) is underdetermined and admits infinitely many solutions. A common approach is the Weighted Pseudo-Inverse (WPI), which yields the minimum-norm solution
where is a positive-definite weighting matrix and denotes the weighted pseudo-inverse.
The weighted pseudo-inverse commits to a single minimum-norm allocation and thereby leaves the redundancy of the overactuated system unused. This surplus freedom becomes explicit when the solution set is expressed through the null space of . Any solution of (2) can be written as the sum of a particular solution and a homogeneous component lying in the null space of [3],
where is the particular solution from (3), and the homogeneous component satisfies , i.e. , so it does not change the achieved command . When has full row rank , the null space has dimension , corresponding to the degrees of freedom that remain once the primary objective is met. This component is parameterized by projecting an arbitrary preferred deflection onto ,
where is the null-space projector. Since is a right inverse of , the projector satisfies by construction, so the primary command-tracking objective (2) remains invariant for any choice of . The complete allocation law therefore becomes
where the first term produces the required command and the second provides the free parameters that can be exploited for secondary objectives without affecting the primary control task.

Figure 2. Overview of the hybrid control allocation architecture. The upper chain represents the conventional control loop: the flight controller generates incremental moment commands, which the Weighted Pseudo-Inverse allocator maps to a baseline actuator command. The null-space injection block modifies this baseline by adding a correction that lies entirely within the null space of the control effectiveness matrix, producing the final actuator command sent to the actuator dynamics.
In (6) the preferred deflection is the only free quantity: it selects one point in the -dimensional null space and thereby fixes the secondary behavior, while the achieved command stays invariant. Classical schemes set it a priori through fixed weights or scheduled targets, which do not generalize across the flight envelope and cannot capture secondary objectives such as energy that are nonlinear and accumulate over the mission. We therefore let a learned policy generate the preference from the current state,
where encodes the flight condition and mission, and the parameters are trained by reinforcement learning to maximize a cumulative return rather than an instantaneous cost. The following section formulates this learning problem.
Reinforcement Learning
Reinforcement learning (RL) formulates sequential decision-making as a Markov Decision Process (MDP), defined by the tuple , where is the state space, the action space, the transition probability, the reward function, and the discount factor [17]. An agent interacts with the environment by observing a state , selecting an action according to a policy , and receiving a scalar reward . The objective is to find the parameters that maximize the expected discounted return
Policy gradient methods optimize (8) by estimating the gradient of the expected return with respect to the policy parameters. Among these, Proximal Policy Optimization (PPO) has emerged as a robust and sample-efficient algorithm for continuous control tasks [18]. PPO constrains the policy update to a trust region by clipping the probability ratio between the new and old policies,
where is the importance sampling ratio, the estimated advantage function, and the clipping parameter that controls the maximum allowed policy deviation per update.
The null-space allocation problem introduced in Section II maps naturally onto this MDP framework. The state is an observation of the information available to the allocator at time ,
where are the measured flight variables, the required command, and the current effector configuration. The observation map selects and normalizes these signals, together with quantities derived from them such as the baseline solution or the null-space coordinates. The specific observation used in this work is given in Section IV-A. The action is the preferred deflection from (7),
which the null-space projector maps to the moment-invariant correction before it is added to the baseline solution (6). which the null-space projector maps to the moment-invariant correction before it is added to the baseline solution as in (??). The projection enforces regardless of the agent output, effectively constraining the action space to without requiring explicit enforcement during training. The state transition
is governed by the nonlinear six-degree-of-freedom aircraft dynamics , the actuator models, and the flight controller response. A mission-conditioned reward
evaluates the resulting allocation with respect to the active mission profile and provides the scalar signal for the PPO policy update (9). The agent thereby learns to select the preferred deflection so that the secondary objective is optimized over the course of a mission.
Training Setup
Aircraft Model
Having established the general control allocation framework for an arbitrary overactuated system with effectors and controlled axes, the formulation is now applied to a specific aircraft configuration. The SynTrac Configuration 2a distributed electric propulsion (DEP) aircraft serves as the application platform [19]. It is a commuter-class transport whose overactuation arises from the combination of conventional aerodynamic control surfaces and ten wing-mounted propellers, giving a large actuator redundancy relative to the three controlled moment axes. The rigid-body dynamics are modeled as a six-degree-of-freedom (6-DOF) nonlinear system with state vector

Figure 3. The SynTrac Configuration 2a distributed electric propulsion aircraft. The control effectors comprise four aerodynamic surfaces (left and right ailerons, elevator, and rudder) and the blade pitch angles of ten wing-mounted propellers.
where is the true airspeed, the angle of attack, the sideslip angle, , , the body-axis roll, pitch, and yaw rates, , , the Euler angles, and , , the position coordinates in the Earth-fixed frame. The principal geometric and inertial properties of the Configuration 2a aircraft are listed in Table I.
The aircraft is controlled through fourteen effectors: four aerodynamic control surfaces, namely a left and right aileron (, ), an elevator (), and a rudder (), together with the blade pitch angles of ten wing-mounted, constant-speed propellers (), whose thrust is modulated through pitch. These are collected in the effector command vector
subject to the position and rate constraints summarized in Table II. Each actuator is modeled as a second-order system with natural frequency and damping ratio (critically damped).
| Parameter | Value | Unit |
| Wing area, | 58.4 | |
| Wing span, | 27.00 | |
| Mean wing chord, | 2.16 | |
| Mass, | 24533 | |
| 217205 | ||
| 612834 | ||
| 705546 | ||
| 0 |
Table 1. Configuration parameters of the SynTrac Configuration 2a aircraft (MTOM case).
| [°] | [°] | [°/s] | [rad/s] | |
| 14 | 20 | 30 | ||
| 14 | 20 | 30 | ||
| 13 | 20 | 30 | ||
| 25 | 20 | 30 | ||
| 15 | 20 | 10 |
Table 2. SynTrac Configuration 2a effector constraints and dynamics.
SynTrac Flight Controller and Control Allocation
The inner loop is based on incremental nonlinear dynamic inversion (INDI). An outer attitude and rate controller generates the required angular acceleration from the tracking error, and the INDI law realizes it incrementally about the current operating point.
where is the current measured effector configuration and the angular acceleration measured at that configuration. The first two terms form the weighted pseudo-inverse increment that tracks the required command, and the third injects the learned null-space preference. Equation (16) is the INDI realization of the general allocation law (6), in which the particular solution is taken incrementally about the current state.
The control effectiveness matrix is evaluated at the current operating point, so it captures the state dependence of both the aerodynamic surfaces and the propeller pitch, including the aeropulsive coupling of the distributed propulsion system. Each column is obtained from the onboard model by perturbing effector about its nominal value, which avoids precomputed gain tables and keeps the allocation valid across the flight envelope.
Training
The policy and value function are represented by the two feed-forward networks summarized in the upper part of Table III. The actor is a continuous Gaussian policy that maps the observation to the mean and standard deviation of the action . It uses three hidden layers of 64, 64 and 32 units with ReLU and tanh activations. A scaling layer normalizes the observation at the input; the mean head is passed through a tanh and a scaling layer so that the commanded preference stays within the admissible effector range, and the standard-deviation head uses a softplus to remain positive. The critic is a smaller network, with hidden layers of 64 and 32 units, that estimates the state value and serves as the baseline in the advantage computation.
| Actor Network | |
| Normalization | Scaling layer |
| Hidden layers | 64, 64, 32 |
| Activations | ReLU, ReLU, tanh |
| Mean output | Fully Connected → tanh → scaling |
| Std. output | Fully Connected → softplus → scaling |
| Critic Network | |
| Normalization | Scaling layer |
| Hidden layers | 64, 32 |
| Activations | ReLU, ReLU |
| Output | Fully Connected |
| PPO Hyperparameters | |
| Clip factor | 0.15 |
| Discount factor | 0.995 |
| Mini-batch size | 64 |
| Entropy loss weight | 0.01 |
| Actor learning rate | |
| Critic learning rate | |
| Gradient threshold | 0.5 |
Table III. PPO agent architecture and training hyperparameters.
Each training episode samples a mission from a library that spans the flight envelope and the operational mission set, with every mission initialized from trimmed flight. The reward is scaled to order one so that the returns stay in a numerically favorable range, and the experience horizon is matched to the episode length, which keeps the advantage estimates consistent along the trajectory. At each step the policy proposes a preferred deflection, the null-space projector maps it to the moment-invariant correction, and the resulting allocation is evaluated by the mission-conditioned reward to form the training signal.
Results
The trained PPO agent is evaluated against the baseline WPI allocation to assess the effect of the learned null-space policy on secondary objective performance.
Drag Minimization
The trained drag-minimization policy is evaluated against the weighted pseudo-inverse baseline on a representative mission that combines a lateral-directional command at s with an angle-of-attack command at s. Each case is run under identical initial conditions and reference commands, once with the agent disabled (RL off, baseline allocation) and once with the agent active (RL on), so that any difference is attributable solely to the learned null-space correction.
Figure 4 shows the aerodynamic drag force over the mission. The mean drag falls from 10212 N for the baseline to 9284 N with the agent active, a reduction of 928 N or

Figure 4. Aerodynamic drag force over the mission for the baseline (RL off, WPI) and the learned allocation (RL on). The mean drag is reduced from 10212 N to 9284 N, a 9.1% reduction.
9.1%. The benefit is sustained rather than transient: the RL-on trace lies at or below the baseline throughout, with the largest relative reductions in the quasi-steady segments before 25 s and after 30 s, and a lowered peak during the high-load event near 27 s, where the drag maximum drops from about N to N.
That the reduction is achieved without disturbing the primary task is confirmed by Figure 6 and Figure 7. The roll, pitch and yaw rates (Figure 6) and the bank, angle-of-attack and sideslip responses (Figure 7) are essentially indistinguishable between RL off and RL on, and both track the reference commands equally. This is the empirical counterpart of the invariance established in § 2: because the learned correction lies in the null space of , it leaves the achieved moments, and therefore the closed-loop attitude and rate response, unchanged.

Figure 6. Body-axis roll, pitch and yaw rates for the reference, the baseline (RL off) and the learned allocation (RL on). The rates are essentially identical with and without the agent, confirming moment-tracking invariance.

Figure 7. Bank angle , angle of attack and sideslip for the reference, the baseline (RL off) and the learned allocation (RL on). The attitude response is unchanged by the null-space correction.
The source of the drag reduction is visible in the effector histories of Figure 5. The baseline deploys the left and right ailerons close to their limits and spreads the ten propeller pitch angles over a wide range, whereas the learned policy holds the ailerons near zero and keeps the propeller pitch tightly clustered. The elevator and rudder, which carry the commanded pitch and yaw, are nearly identical in both cases and retain little null-space freedom. The agent therefore unloads the aerodynamic surfaces and redistributes the trim onto the distributed propulsion, reaching a less draggy configuration that produces the same moments. This is exactly the redundancy that the baseline leaves unused and that the null-space policy is designed to exploit.

Figure 5. Effector deflections for the baseline (RL off) and the learned allocation (RL on): the four aerodynamic surfaces and the distributed-propulsion pitch angles. The agent holds the ailerons near zero and keeps the propeller pitch clustered, while the elevator and rudder, which carry the commanded pitch and yaw, are nearly unchanged.
Conclusion and Outlook
This work presented a hybrid control allocation architecture in which a deterministic pseudo-inverse resolves the primary command while a reinforcement-learning policy acts within the null space of the control effectiveness matrix to optimize a secondary objective. Because the learned correction enters through the null-space projection, it cannot alter the achieved moments, so command tracking is preserved by construction and the certification argument is confined to the deterministic core.
Future work will extend the approach along two directions. The first is a broader set of mission profiles, with a single mission-conditioned policy trained across the flight envelope and the operational mission set, so that the secondary objective can be adapted online and evaluated on further objectives such as acoustic noise. The second addresses the path dependency introduced by the incremental INDI allocation. Because the effectors retain their previous positions, these null-space increments accumulate from step to step. The configuration reached at any instant is therefore the sum of all past corrections rather than a function of the current command, which lets it drift toward the actuator limits and makes the achieved secondary objective depend on the path taken rather than on the instantaneous state, something an instantaneous allocation rule cannot account for.
Acknowledgment
This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - Project ID 498601949 - TRR 364 SynTrac.
References
- [1]S. Yin, B. Xiao, S. X. Ding, and D. Zhou, “A review on recent development of spacecraft attitude fault tolerant control system,” IEEE Transactions on Industrial Electronics, vol. 63, no. 5, pp. 3311–3320, 2016.DOI
- [2]T. I. Fossen and T. A. Johansen, “A survey of control allocation methods for ships and underwater vehicles,” in 2006 14th Mediterranean Conference on Control and Automation, 2006, pp. 1–6.DOI
- [3]W. Durham, K. A. Bordignon, and R. Beck, Aircraft control allocation. John Wiley & Sons, 2017.DOI
- [4]T. A. Johansen and T. I. Fossen, “Control allocation—a survey,” Automatica, vol. 49, no. 5, pp. 1087–1103, 2013.DOI
- [5]J. M. Buffington and D. F. Enns, “Lyapunov stability analysis of daisy chain control allocation,” Journal of Guidance, Control, and Dynamics, vol. 19, no. 6, pp. 1226–1230, 1996.DOI
- [6]K. Bordignon and W. Durham, “Null-space augmented solutions to constrained control allocation problems,” in Guidance, navigation, and control conference, 1995, p. 3209.DOI
- [7]W. C. Durham, “Constrained control allocation,” Journal of Guidance, control, and Dynamics, vol. 16, no. 4, pp. 717–725, 1993.DOI
- [8]W. C. Durham, J. G. Bolling, and K. A. Bordignon, “Minimum drag control allocation,” Journal of Guidance, Control, and Dynamics, vol. 20, no. 1, pp. 190–193, 1997.
- [9]R. Stolk and C. de Visser, “Minimum drag control allocation for the innovative control effector aircraft,” in CEAS EuroGNC 2019, 2019.
- [10]O. Pfeifle and W. Fichter, “Minimum power control allocation for incremental control of over-actuated transition aircraft,” Journal of Guidance, Control, and Dynamics, vol. 46, no. 2, pp. 286–300, 2023.DOI
- [11]T. Pollack and E.-J. Van Kampen, “Multi-objective design and performance analysis of incremental control allocation-based flight control laws,” in AIAA SCITECH 2023 Forum, 2023, p. 1249.DOI
- [12]J. Kang and K. Choi, “Development of an artificial neural network control allocation algorithm for small tailless aircraft based on dynamic allocation method,” International Journal of Aeronautical and Space Sciences, vol. 23, no. 2, pp. 363–378, 2022.DOI
- [13]S. P. Madruga, A. H. Tavares, S. O. Luiz, T. P. do Nascimento, and A. M. N. Lima, “Aerodynamic effects compensation on multirotor uavs based on a neural network control allocation approach,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 2, pp. 295–312, 2021.DOI
- [14]M. Chen, “Constrained control allocation for overactuated aircraft using a neuromydamic model,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 46, no. 12, pp. 1630–1641, 2015.DOI
- [15]P. S. de Vries and E.-J. Van Kampen, “Reinforcement learning-based control allocation for the innovative control effectors aircraft,” in AIAA Scitech 2019 Forum, 2019, p. 0144.DOI
- [16]H. Isci and E. Koyuncu, “Safe, hybrid control allocation for the innovative control effectors (ICE) aircraft,” in CEAS Conference on Guidance, Navigation and Control (EuroGNC), Madrid, Spain, May 2026, paper CEAS-GNC-2026-033.
- [17]R. S. Sutton, A. G. Barto, et al., Reinforcement learning: An introduction. MIT press Cambridge, 1998, vol. 1, no. 1.DOI
- [18]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
- [19]J. Schlittner, Y. Freund, J. Mangold, R. Hanke-Rauschenbach, and A. Strohmayer, “Preliminary design of regional aircraft—integration of a fuel cell-electric energy network in suave,” Aerospace, vol. 12, no. 3, p. 249, 2025.