Application of Reinforcement Learning to the Flight Control of Fixed-Wing Uncrewed Aerial Vehicles

Abstract

Fixed-wing Uncrewed Aerial Vehicles (UAVs) are capable of greater flight range and endurance than rotary-wing platforms for a given Maximum Takeoff Weight (MTOW), yet
the need to maintain a minimum airspeed hinders their manoeuvrability. As such, they
are generally limited in their ability to operate in complex urban environments. One method
to improve the agility of a fixed-wing aircraft is to fly at more extreme flight attitudes and use
dynamic, post-stall manoeuvres. However, conventional flight controllers generally cannot operate in such flight regimes. Reinforcement Learning (RL) is a Machine Learning (ML) paradigm
that trains an agent through interaction with an environment. RL methods have demonstrated
success across several domains, including the control of real-world robotic systems.
Prior work by the research group showed the potential for RL models to control a bio-inspired
perched landing manoeuvre for a sweep-wing aircraft. The first part of this thesis investigates
approaches to improve the performance of RL models for this scenario by minimising errors in
the desired final states. This research focuses on training models that perform well in simulation
and the real world. Initial work explored not just RL improvements, such as choice of algorithm,
but also improvements to the numerical model to reduce the gap between simulation and reality.
Trained models were tested in the real world using an automated experimental process, with
models trained to account for atmospheric disturbances showing reduced final state error. Further
in-depth optimisation of the RL training process demonstrated a subsequent improvement in
performance in reality, with a reduction in final landing position error and velocity compared
to the baseline. This work concluded by investigating methods to alter the perched landing
scenario using the throttle in high wind conditions. This first part of the thesis demonstrated the
challenges of learning with an incomplete or inaccurate simulation environment, with much of
the work focused on overcoming those limitations.
The next part of the thesis introduces a simulation environment for a new, aerobatic fixedwing aircraft and a pair of agile flight scenarios - rapid obstacle avoidance and constrained
space landing. Several learning approaches are investigated, including modern RL algorithms
that demonstrate superior sample efficiency and performance in this work and other continuous
control tasks in the literature.
Finally, a system architecture for learning aerobatic flight manoeuvres exclusively from
real-world experience is introduced. This work provides results from a simulated proof of concept
and describes the steps to use the system in the real world.
This thesis identifies and investigates many practical aspects and challenges of applying
RL methods to fixed-wing UAV flight control. It concludes with future research directions to
advance this field, identifying the need for accurate and fast, highly parallelisable simulation
environments to match the paradigms that have driven progress in other fields of robotics.

Date of Award	4 Feb 2025
Original language	English
Awarding Institution	University of Bristol
Supervisor	Tom S Richardson (Supervisor) & Mark Hansen (Supervisor)