Human Pilots Research Articles

Multi-Unmanned Aerial Vehicle-Assisted Flood Navigation of Waterborne Vehicles Using Deep Reinforcement Learning

Abstract During disasters, such as floods, it is crucial to get real-time ground information for planning rescue and response operations. With the advent of technology, unmanned aerial vehicles (UAVs) are being deployed for real-time path planning to provide support to evacuation teams. However, their dependency on expert human pilots for command and control limits their operational capacity to the line-of-sight range. In this article, we utilize a deep reinforcement learning algorithm to autonomously control multiple UAVs for area coverage. The objective is to identify serviceable paths for safe navigation of waterborne evacuation vehicles (WBVs) to reach critical location(s) during floods. The UAVs are tasked to capture the obstacle-related data and identify shallow water regions for unrestricted motion of the WBV(s). The data gathered by UAVs is used by the minimum expansion A* (MEA*) algorithm for path planning to assist WBV(s). MEA* addresses the node expansion issue with the standard A* algorithm, by pruning the unserviceable nodes/locations based on the captured information, hence expediting the path planning process. The proposed approach, MEA*MADDPG, is compared with other prevalent techniques from the literature over simulated flood environments with moving obstacles. The results highlight the significance of the proposed model as it outperforms other techniques when compared over various performance metrics.

UAV air combat autonomous trajectory planning method based on robust adversarial reinforcement learning

The poor robustness of the air combat autonomous trajectory planning strategy (ATP) trained through vanilla reinforcement learning (RL) methods is attributed to its dependence on the source environment. When this ATP is transformed from the source to the target environment, it may fail to lead to an effective dogfight victory due to disturbances, which can potentially threaten the flight safety of the unmanned combat aerial vehicle (UCAV). In order to enhance the robustness of ATP, we propose a robust RL-based method for generating an ATP strategy. This method can effectively consider uncertainties in the state and potential model misspecifications. First, a jointly trained adversary is designed to apply disturbances in the environment, forming a two-player zero-sum game with the ATP agent. To solve this game problem and learn a robust ATP, the robust adversarial RL method (RARL) is applied. Additionally, in order to prevent non-convergence of strategies resulting from the introduction of the RARL algorithm, a curriculum learning method is proposed that emulates the way human pilots learn, gradually progressing from simpler to more challenging courses. Finally, a quantitative method for evaluating the robustness of the ATP is proposed, taking into consideration the differences in ATP performance between the source and target combat environments. The results of robustness estimations demonstrate that the RARL algorithm, which is proposed herein, effectively enhances the ATP's robustness.

Imitation learning for aerobatic maneuvering in fixed-wing aircraft

This study focuses on the task of developing automated models for complex aerobatic aircraft maneuvers. The approach employed here utilizes Behavioral Cloning, a technique in which human pilots supply a series of sample maneuvers. These maneuvers serve as training data for a Machine Learning algorithm, enabling the system to generate control models for each maneuver. The optimal instances for each maneuver were chosen based on a set of objective evaluation criteria. By utilizing these selected sets of examples, resilient models were developed, capable of reproducing the maneuvers performed by the human pilots who supplied the examples. In certain instances, these models even exhibited superior performance compared to the pilots themselves, a phenomenon referred to as the “clean-up effect”. We also explore the application of transfer learning to adapt the developed controllers to various airplane models, revealing compelling evidence that transfer learning is effective for refining them for targeted aircraft. A comprehensive set of intricate maneuvers was executed through a meta-controller capable of orchestrating the fundamental maneuvers acquired through imitation. This undertaking yielded promising outcomes, demonstrating the proficiency of several Machine Learning models in successfully executing highly intricate aircraft maneuvers. This paper is an extended version of the previously ICCS 2023 published conference paper [1] .

Autonomous robotic drone system for mapping forest interiors

Abstract. During the last decade, the use of drones in forest monitoring and remote sensing has become highly popular. While most of the monitoring tasks take place in high altitudes and open air, in the last few years drones have also gained interest in under-canopy data collection. However, flying under the forest canopy is a complex task since the drone can not use Global Navigation Satellite Systems (GNSS) for positioning and it has to continually avoid obstacles, such as trees, branches, and rocks, on its path. For that reason, drone-based data collection under the forest canopy is still mainly based on manual control by human pilots. Autonomous flying in GNSS-denied obstacle-rich environment has been an actively researched topic in the field of robotics during the last years and various open-sourced methods have been published in the literature. However, most of the research is done purely from the point-of-view of robotics and only a few studies have been published in the boundary of forest sciences and robotics aiming to take steps towards autonomous forest data collection. In this study, a prototype of an autonomous under-canopy drone is developed and implemented utilizing state-of-the-art open-source methods. The prototype is utilizing the EGO-Planner-v2 trajectory planner for autonomous obstacle avoidance and VINS-Fusion for Visual-inertial-odometry based GNSS-free pose estimation. The flying performance of the prototype is evaluated by performing multiple test flights with real hardware in two different boreal forest test plots with medium and difficult densities. Furthermore, the first results of the forest data collecting performance are obtained by post-processing the data collected with a low-cost stereo camera during one test flight to a 3D point cloud and by performing diameter breast at height (DBH) estimation. In the medium-density forest, all seven test flights were successful, but in the difficult test forest, one of eight test flights failed. The RMSE of the DBH estimation was 3.86 cm (12.98 %).

Explainable Basic-Fighter-Maneuver Decision Support Scheme for Piloting Within-Visual-Range Air Combat

In within-visual-range (WVR) air combat, basic fighter maneuvers (BFMs) are widely used. A BFM decision support scheme has been proposed to aid human pilots in the complex air combat engagement. Recent artificial intelligence advances provide novel opportunities for the development of BFM decision support research. This paper commences by establishing an air-combat-engagement database. Key features that pilots rely on for BFM decision-making in WVR air combat are analyzed, which identifies the input and output data essential for the development of the BFM decision support scheme. A Long Short-Term-Memory (LSTM)-based BFM decision support scheme is then proposed to map input (i.e., combat situations) to output (i.e., BFM decision). Additionally, Shapley-Additive-Explanations-based explainability analysis is also employed to assess the importance of each input feature in the LSTM blocks, and to explain the contribution of each feature to the BFM decision. To evaluate the effectiveness of the proposed BFM decision support scheme, WVR air-combat tests are conducted, which justify the effectiveness of the proposed scheme.

UAV Head-On Situation Maneuver Generation Using Transfer-Learning-Based Deep Reinforcement Learning

AbstractRecently, the demand for unmanned aerial vehicle technology has increased. In particular, AI pilots through reinforcement learning (RL) are more flexible than those using rule-based methods. Further, AI pilots with RL are expected to replace human pilots in the future. In a recent study, rather than completely replacing humans, studies on AI pilots are conducted toward the collaboration between man and unmanned aircraft. AI pilots have several advantages over humans. For example, on the one hand, human pilots avoid head-on situations due to collision. On the other hand, AI pilots may prefer head-on situations to finish the episode quickly. This study proposes a two-circle-based transfer learning method to demonstrate excellent performance in head-on situations. Based on the experimental results, the proposed two-circle-based multi-task transfer learning model outperforms the model without transfer learning-based RL. A study on transfer-learning-based learning technique has been conducted. However, it had a one-circle-based learning technique was specialized only for tail-chasing, limiting its application (Bae et al. in IEEE Access 11:26427–26440, 2023). Practically, the proposed two-circle-based learning technique outperforms the one-circle-based transfer learning technique in head-on situations.

Hierarchical Reinforcement Learning for Air Combat at DARPA's AlphaDogfight Trials

Autonomous control in high-dimensional, continuous state spaces is a persistent and important challenge in the fields of robotics and artificial intelligence. Because of high risk and complexity, the adoption of AI for autonomous combat systems has been a long-standing difficulty. In order to address these issues, DARPA's AlphaDogfight Trials (ADT) program sought to vet the feasibility of and increase trust in AI for autonomously piloting an F-16 in simulated air-to-air combat. Our submission to ADT solves the high-dimensional, continuous control problem using a novel hierarchical deep reinforcement learning approach consisting of a high-level policy selector and a set of separately trained low-level policies specialized for excelling in specific regions of the state space. Both levels of the hierarchy are trained using off-policy, maximum entropy methods with expert knowledge integrated through reward shaping. Our approach outperformed human expert pilots and achieved a second-place rank in the ADT championship event. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Impact Statement–</i> Significant performance milestones in reinforcement learning have been achieved in recent years, with autonomous agents demonstrating super-human performance across a wide variety of tasks. Before these algorithms can be extensively deployed in real-world defense applications, a greater level of trust must first be achieved. ADT was an important step towards developing the trust necessary to operationalize these algorithms, by demonstrating their effectiveness on a foundational yet relevant problem in a high-fidelity simulation environment. Developed for the program, our hierarchical reinforcement learning agent was designed alongside of and competed against active fighter pilots, and ultimately defeated a graduate of the United States Air Force's F-16 Weapons Instructor Course in match play.

Effective training to improve DeepPilot

We present an approach to autonomous drone racing inspired by how a human pilot learns a race track. Human pilots drive around the track multiple times to familiarise themselves with the track and find key points that allow them to complete the track without the risk of collision. This paper proposes a three-stage approach: exploration, navigation, and refinement. Our approach does not require prior knowledge about the race track, such as the number of gates, their positions, and their orientations. Instead, we use a trained neural pilot called DeepPilot to return basic flight commands from camera images where a gate is visible to navigate an unknown race track and a Single Shot Detector to visually detect the gates during the exploration stage to identify points of interest. These points are then used in the navigation stage as waypoints in a flight controller to enable faster flight and navigate the entire race track. Finally, in the refinement stage, we use the methodology developed in stages 1 and 2, to generate novel data to re-train DeepPilot, which produces more realistic manoeuvres for when the drone has to cross a gate. In this sense, similar to the original work, rather than generating examples by flying in a full track, we use small tracks of three gates to discover effective waypoints to be followed by the waypoint controller. This produces novel training data for DeepPilot without human intervention. By training with this new data, DeepPilot significantly improves its performance by increasing its flight speed twice w.r.t. its original version. Also, for this stage 3, we required 66 % less training data than in the original DeepPilot without compromising the effectiveness of DeepPilot to enable a drone to autonomously fly in a racetrack.

Manual-Based Automated Maneuvering Decisions for Air-to-Air Combat

A novel new air combat algorithm is proposed, which is based on the knowledge extracted from the experience of human pilots. First, to implement a fighter that maneuvers based on manual control, the maneuver form of the fighter is analyzed and represented as a block. Second, the blocks for each function are connected based on their relationship, and a flow diagram is presented according to the engagement situation of the adversary and ownship. Third, a behavior tree model is applied as a decision-making model to implement the flow diagram as a simulation program. The behavior tree offers good scalability because nonleaf nodes can be added when sophisticated and complex decision-making is required. The proposed method has the advantage of making all maneuvers performed by the algorithm understandable and interpretable. Additionally, it can replace expensive and dangerous dogfighting training for student pilots because the proposed model can emulate maneuvers that manned pilots would perform. To verify the proposed method, the evaluation criteria from the AlphaDogfight Trials are equally applied in the simulation. The experimental results demonstrate that the proposed method has superior engagement capability as compared to the existing air-to-air combat models.

Two Circle-based Aircraft Head-on Reinforcement Learning Technique using Curriculum

Recently, AI pilots using reinforcement learning are developing to a level that is more flexible than rule-based methods and can replace human pilots. In this paper, a curriculum was used to help head-on combat with reinforcement learning. It is not easy to learn head-on with a reinforcement learning method without a curriculum, but in this paper, through the two circle-based head-on air combat learning technique, ownship gradually increase the difficulty and become good at head-on combat. On the two-circle, the ATA angle between the ownship and target gradually increased and the AA angle gradually decreased while learning was conducted. By performing reinforcement learning with and w/o curriculum, it was engaged with the rule-based model. And as the win ratio of the curriculum based model increased to close to 100 %, it was confirmed that the performance was superior.

Military Based Fixed Wing Scouting of an Unmanned Aerial Vehicle

Abstract: Unmanned Aerial Vehicles (UAV) is in constant development since First World War. As the technology became complex and compact, the UAVs are also being utilized for commercial purposes. The main purpose of a UAV was for scouting, to perform certain operations in remote areas with minimal human interference and to eliminate the risk of human pilots. UAVs are classified based on their weight, range of operation, type of model, and fuel consumption. At present, multi-rotor drones with high mobility are widely used for commercial sectors especially, aerial photography, sports, surveys, etc. Most of the drones are electric-based which provide a maximum of 30 minutes flight time that is sufficient for small scale applications. There are combustible fuel based drones in development, but they are very expensive, also multi-rotor drones are inefficient and provide less output per input.

Knowledge-Driven Accurate Opponent Trajectory Prediction for Gun-Dominated Autonomous Air Combat

The autonomous air combat (AAC) technique has been a lasting topic for decades. Accurate opponent trajectory prediction provides a fundamental basis for decision making in AAC. In this work, we propose a knowledge-driven scheme for the opponent trajectory prediction problem in one-versus-one gun-dominated within-visual-range (WVR) air combat. A dedicated air combat engagement database is first constructed via skilled human pilots flying WVR air combat. Two baseline algorithms following rule-based and learning-based paradigms are developed and optimized. The knowledge-driven scheme begins with defining handcrafted features extracted from the opponent history movements, and principal components analysis is adopted to compress/refine the features. A generalized regression neural network is then developed to compensate for the residual of the rule-based trajectory prediction method, wherein the refined features are used as inputs, and the residual compensation are used as outputs. Via extensive simulation tests, the proposed scheme shows a more accurate performance as compared with the two other baseline algorithms. To demonstrate the applicability of the proposed scheme, an automatic gun-firing strategy for commencing gun attack in AAC is also illustrated, which justifies the proposed scheme.

The Development of Rule-based AI Engagement Model for Air-to-Air Combat Simulation

Since the concept of Manned-UnManned Teaming(MUM-T) and Unmanned Aircraft System(UAS) can efficiently respond to rapidly changing battle space, many studies are being conducted as key components of the mosaic warfare environment. In this paper, we propose a rule-based AI engagement model based on Basic Fighter Maneuver(BFM) capable of Within-Visual-Range(WVR) air-to-air combat and a simulation environment in which human pilots can participate. In order to develop a rule-based AI engagement model that can pilot a fighter with a 6-DOF dynamics model, tactical manuals and human pilot experience were configured as knowledge specifications and modeled as a behavior tree structure. Based on this, we improved the shortcomings of existing air combat models. The proposed model not only showed a 100 % winning rate in engagement with human pilots, but also visualized decision-making processes such as tactical situations and maneuvering behaviors in real time. We expect that the results of this research will serve as a basis for development of various AI-based engagement models and simulators for human pilot training and embedded software test platform for fighter.

Scalable Autonomous Separation Assurance With Heterogeneous Multi-Agent Reinforcement Learning

In this article, a scalable autonomous separation assurance framework is proposed for high-density en route airspace sectors with heterogeneous aircraft objectives. To handle the complex dynamic decision making under uncertainty, multi-agent reinforcement learning is used in a decentralized approach with each aircraft being represented as an agent. Based on this, each agent locally solves the separation assurance problem, allowing the framework to scale to a large number of aircraft. In addition, each agent has the ability to learn the intention of the intruder aircraft, which is essential in environments with heterogeneous agents. Numerical experiments are performed in a real-time air traffic simulator. The results demonstrate that the proposed framework is able to effectively ensure the safe separation of heterogeneous agents, while also optimizing the intrinsic agent objectives in high-density en route airspace sectors. In addition, the efficiency of the proposed framework is demonstrated and shown to provide real-time decision making for separation assurance. Note to Practitioners—In commercial aviation, the workload of human air traffic controllers increases with the growth of air traffic density. Robust decision making systems to augment human air traffic controllers allows for increased air traffic without increased workload, resulting in a safer airspace environment. In addition, advanced air mobility (AAM) is concerned with low-altitude airspace operations with both human and autonomous pilots. In this environment, autonomous real-time separation assurance systems are required. Most traditional separation assurance approaches fail to handle stochastic and high-density environments, rendering them inapplicable to future high-density traditional airspace and the envisioned low-altitude AAM operations. Therefore, it is important to study decentralized approaches that can place the separation assurance problem as an intrinsic objective of each aircraft to ensure cooperation in high-density airspace. With this consideration, a scalable, decentralized autonomous separation assurance framework capable of handling heterogeneous agents is proposed in this article. This framework is able to perceive the current air traffic environment and select speed advisories to ensure safe separation requirements, while balancing intrinsic objectives such as minimizing delay. While one limitation of multi-agent reinforcement learning is the long training time, this article demonstrates how the framework also can leverage modern computing clusters to significantly reduce training time without sacrificing performance.

An Autonomous Control Framework of Unmanned Helicopter Operations for Low-Altitude Flight in Mountainous Terrains

Low-altitude flight in mountainous terrains is a difficult flight task applied in both military and civilian fields. The helicopter has to maintain low altitude to realize search and rescue, reconnaissance, penetration, and strike operations. It contains complex environment perception, multilevel decision making, and multi-objective flight control; thus, flight is currently mainly conducted by human pilots. In this work, a control framework is implemented to realize autonomous flight for unmanned helicopter operations in an unknown mountainous environment. The identification of targets and threats is introduced using a deep neural network. A 3D vector field histogram method is adopted for local terrain avoidance based on airborne Lidar sensors. In particular, we propose an intuitive direct-viewing method to judge and change the visibilities of the helicopter. On this basis, a finite state machine is built for decision making of the autonomous flight. A highly realistic simulation environment is established to verify the proposed control framework. The simulation results demonstrate that the helicopter can autonomously complete flight missions including a fast approach, threat avoidance, cover concealment, and circuitous flight operations similar to human pilots. The proposed control framework provides an effective solution for complex flight tasks and expands the flight control technologies for high-level unmanned helicopter operations.

The Virtual Top Gun [Aerospace - defence

Implementing autonomous technology into fighter jets will mean convincing human pilots to cede control.

The sensing, state-estimation, and control behind the winning entry to the 2019 Artificial Intelligence Robotic Racing Competition

Autonomous drone racing currently forms an extreme challenge in robotics. While human drone racers can fly through complex tracks at speeds of up to 190 km/h (53 m/s), autonomous drones still need to tackle several fundamental problems in AI under severe restrictions in terms of resources before they reach the same adaptability and speed. In this article, we present the winning solution of the first AI Robotic Racing (AIRR) Circuit, an autonomous drone race competition in which all participating teams used the same drone, to which they had limited access. The core of our approach is inspired by how human pilots combine noisy observations of the race gates with a mental model of the drone’s dynamics. The navigation is based on gate detection with an efficient deep neural segmentation network and active vision. Combined with contributions to robust state estimation and risk-based control, our solution was able to reach speeds of ≈33 km/h (9.2m/s) and hereby more than triple the speeds seen in previous autonomous drone race competitions. This work analyses the performance of each component and discusses the implications for high-performance real-world AI applications with limited training time.

Visual attention prediction improves performance of autonomous drone racing agents.

Humans race drones faster than neural networks trained for end-to-end autonomous flight. This may be related to the ability of human pilots to select task-relevant visual information effectively. This work investigates whether neural networks capable of imitating human eye gaze behavior and attention can improve neural networks' performance for the challenging task of vision-based autonomous drone racing. We hypothesize that gaze-based attention prediction can be an efficient mechanism for visual information selection and decision making in a simulator-based drone racing task. We test this hypothesis using eye gaze and flight trajectory data from 18 human drone pilots to train a visual attention prediction model. We then use this visual attention prediction model to train an end-to-end controller for vision-based autonomous drone racing using imitation learning. We compare the drone racing performance of the attention-prediction controller to those using raw image inputs and image-based abstractions (i.e., feature tracks). Comparing success rates for completing a challenging race track by autonomous flight, our results show that the attention-prediction based controller (88% success rate) outperforms the RGB-image (61% success rate) and feature-tracks (55% success rate) controller baselines. Furthermore, visual attention-prediction and feature-track based models showed better generalization performance than image-based models when evaluated on hold-out reference trajectories. Our results demonstrate that human visual attention prediction improves the performance of autonomous vision-based drone racing agents and provides an essential step towards vision-based, fast, and agile autonomous flight that eventually can reach and even exceed human performances.

An Adaptive Human Pilot Model for Adaptively Controlled Systems

Despite their success in handling uncertain dynamical systems that are prone to failure, adaptive controllers are observed to have unfavorable interactions with human pilots in certain applications. To alleviate this problem, we need to evaluate the safety and performance of adaptive controllers in the simulation environment using realistic pilot models before conducting flight tests. While many useful human pilot models exist in the literature, models that are adequate for the prediction of adaptive human-adaptive controller interactions are yet to be available. In this letter, we fill this gap by proposing an adaptive human pilot model suited for the prediction of human behavior in the loop with an adaptive controller. The model can serve as a valuable tool guiding the design of adaptive controllers so as to ensure smooth pilot-controller interactions.

Perception Action Aware-Based Autonomous Drone Race in a Photorealistic Environment

The development of autonomous, fast, agile small Unmanned Aerial Vehicles (UAVs) brings up fundamental challenges in dynamic environments with fast and agile maneuvers, unreliable state estimation, imperfect sensing, coupling action, and perception in real-time under severe resource constraints. However, autonomous drone racing is a challenging research problem at the intersection of computer vision, planning, state estimation, and control. To bridge this, we propose an approach in the context of autonomous, perception-action aware vision-based drone racing in a photorealistic environment. Our approach integrates a deep convolutional neural network (CNN) with state-of-the-art path planning, state estimation, and control algorithms. The developed deep learning method is based on computer vision approaches to detecting the gates and estimating the flyable area. The planner and controller then use this information to generate a short, minimum-snap trajectory segment and send corresponding motor commands to reach the desired goal. A thorough evaluation of our proposed methodology has been carried out using the Gazebo and FlightGoggles (photorealistic sensor) environments. Extensive experiments demonstrate that the proposed approach outperforms state-of-the-art methods and flies the drone more consistently than many human pilots. Moreover, we demonstrated that our proposed system successfully guided the drone through tight race courses, reaching speeds up to 7m/s of the 2019 AlphaPilot Challenge.