Maneuver Decision Research Articles

Autonomous Maneuver Decision Making of Dual-UAV Cooperative Air Combat Based on Deep Reinforcement Learning

Autonomous maneuver decision making is the core of intelligent warfare, which has become the main research direction to enable unmanned aerial vehicles (UAVs) to independently generate control commands and complete air combat tasks according to environmental situation information. In this paper, an autonomous maneuver decision making method is proposed for air combat by two cooperative UAVs, which is showcased by using the typical olive formation strategy as a practical example. First, a UAV situation assessment model based on the relative situation is proposed, which uses the real-time target and UAV location information to assess the current situation or threat. Second, the continuous air combat state space is discretized into a 13 dimensional space for dimension reduction and quantitative description, and 15 typical action commands instead of a continuous control space are designed to reduce the difficulty of UAV training. Third, a reward function is designed based on the situation assessment which includes the real-time gain due to maneuver and the final combat winning/losing gain. Fourth, an improved training data sampling strategy is proposed, which samples the data in the experience pool based on priority to accelerate the training convergence. Fifth, a hybrid autonomous maneuver decision strategy for dual-UAV olive formation air combat is proposed which realizes the UAV capability of obstacle avoidance, formation and confrontation. Finally, the air combat task of dual-UAV olive formation is simulated and the results show that the proposed method can help the UAVs defeat the enemy effectively and outperforms the deep Q network (DQN) method without priority sampling in terms of the convergence speed.

Vision-Based 3D Aerial Target Detection and Tracking for Maneuver Decision in Close-Range Air Combat

Automatic maneuver decision in close-range air combat depends on the situation awareness of the 3D aerial space. Optimal decision could only be made when the 3D state (e.g. 3D position, orientation and velocity) of the target aircraft is accurately provided. Together with the state of the aircraft in our side, optimal maneuver decision could be made by maximizing the situation advantage or utilizing deep reinforcement learning. On the other hand, vision-based 3D sensing methods are ideal for acquiring the 3D state of the target aircraft in close-range air combat, since radar and other sensors work badly in such short range. In this paper, we propose a novel pipeline for vision-based maneuver decision in close-range air combat. The proposed pipeline contains three main modules: 3D target detection based on Augmented Autoencoder, 3D target tracking based on segmentation and optimization, and maneuver decision based on advantage maximization and Deep Q Networks (DQN). The proposed method effectively handles the difficulties in air combat environment, such as fast movement, occlusion from cloud, etc. Experiments demonstrate that our method could robustly detect and track the target aircraft in complex environment, which provides strong priors for maneuver decision and helps to significantly improve the winning rate of short-range air combat.

Research on Maneuvering Decision Algorithm Based on Improved Deep Deterministic Policy Gradient

Autonomous maneuvering decisions of unmanned aerial vehicle (UAV) in short-range air combat remain a challenging research topic, and a decision method based on an improved deep deterministic policy gradient (DDPG) is proposed. First, the problem model is improved from the perspective of energy–air combat, and a decision model with engine thrust, angle of attack, and roll angle as control variables is established. The normal and tangential overloads are determined by these control variables, and the decision is constrained by the flight stability and threshold range. Subsequently, the decision learning algorithm of the maneuver command is designed based on the DDPG framework. According to the energy air combat, speed is introduced into the return function in some states to make the return value more in line with reality. In view of the slow learning speed of the DDPG algorithm, the winning rate is introduced into the ε-greedy strategy to adjust the exploration and application probabilities in real time. In view of the decrease in computational efficiency caused by the large amount of empirical data, a similar empirical exclusion was carried out based on the vector distance. The simulation results show that the DDPG-based algorithm realizes autonomous decisions of engine thrust, roll angle, and attack angle under constraints, and the comparative simulation shows that the improvement measures are effective.

UAV maneuvering decision -making algorithm based on Twin Delayed Deep Deterministic Policy Gradient Algorithm

Aiming at intelligent decision-making of UAV based on situation information in air combat, a novel maneuvering decision method based on deep reinforcement learning is proposed in this paper. The autonomous maneuvering model of UAV is established by Markov Decision Process. The Twin Delayed Deep Deterministic Policy Gradient(TD3) algorithm and the Deep Deterministic Policy Gradient (DDPG) algorithm in deep reinforcement learning are used to train the model, and the experimental results of the two algorithms are analyzed and compared. The simulation experiment results show that compared with the DDPG algorithm, the TD3 algorithm has stronger decision-making performance and faster convergence speed, and is more suitable forsolving combat problems. The algorithm proposed in this paper enables UAVs to autonomously make maneuvering decisions based on situation information such as position, speed, and relative azimuth, adjust their actions to approach and successfully strike the enemy, providing a new method for UAVs to make intelligent maneuvering decisions during air combat.

UAV cooperative air combat maneuver decision based on multi-agent reinforcement learning

In order to improve the autonomous ability of unmanned aerial vehicles (UAV) to implement air combat mission, many artificial intelligence-based autonomous air combat maneuver decision-making studies have been carried out, but these studies are often aimed at individual decision-making in 1v1 scenarios which rarely happen in actual air combat. Based on the research of the 1v1 autonomous air combat maneuver decision, this paper builds a multi-UAV cooperative air combat maneuver decision model based on multi-agent reinforcement learning. Firstly, a bidirectional recurrent neural network (BRNN) is used to achieve communication between UAV individuals, and the multi-UAV cooperative air combat maneuver decision model under the actor-critic architecture is established. Secondly, through combining with target allocation and air combat situation assessment, the tactical goal of the formation is merged with the reinforcement learning goal of every UAV, and a cooperative tactical maneuver policy is generated. The simulation results prove that the multi-UAV cooperative air combat maneuver decision model established in this paper can obtain the cooperative maneuver policy through reinforcement learning, the cooperative maneuver policy can guide UAVs to obtain the overall situational advantage and defeat the opponents under tactical cooperation.

Autonomous maneuver strategy of swarm air combat based on DDPG

Unmanned aerial vehicles (UAVs) have been found significantly important in the air combats, where intelligent and swarms of UAVs will be able to tackle with the tasks of high complexity and dynamics. The key to empower the UAVs with such capability is the autonomous maneuver decision making. In this paper, an autonomous maneuver strategy of UAV swarms in beyond visual range air combat based on reinforcement learning is proposed. First, based on the process of air combat and the constraints of the swarm, the motion model of UAV and the multi-to-one air combat model are established. Second, a two-stage maneuver strategy based on air combat principles is designed which include inter-vehicle collaboration and target-vehicle confrontation. Then, a swarm air combat algorithm based on deep deterministic policy gradient strategy (DDPG) is proposed for online strategy training. Finally, the effectiveness of the proposed algorithm is validated by multi-scene simulations. The results show that the algorithm is suitable for UAV swarms of different scales.

Near-Field Route Optimization-Supported Polar Ice Navigation via Maritime Radar Videos

The accurate design of ship routing plans in arctic areas is not easy, considering that navigation conditions (e.g., weather, visibility, and ice thickness) may change frequently. A ship’s crew identifies sea ice in arctic channels with the help of radar echoes, and ship maneuvering decisions are made to avoid navigation interference. Ship officials must manually and consistently change the ship’s route of travel, which is time-consuming and tedious. To address this issue, we propose a near-field route optimization model for the purpose of automatically selecting an optimal route with the help of radar echo images. The ship near-field route optimization model uses a multiobjective optimal strategy considering factors of minimum navigation risk and steaming distance. We verified the model’s performance with the support of the Xuelong voyage dataset. This research finding can help a ship’s crew to design more reasonable navigation routes in polar channels.

Human-like Decision-Making System for Overtaking Stationary Vehicles Based on Traffic Scene Interpretation.

There are multifarious stationary vehicles in urban driving environments. Autonomous vehicles need to make appropriate overtaking maneuver decisions to navigate through the stationary vehicles. In literature, overtaking maneuver decision problems have been addressed in the perspective of either discretionary lane-change or parked vehicle classification. While the former approaches are prone to generating undesired overtaking maneuvers in urban traffic scenarios, the latter approaches induce deadlock situations behind a stationary vehicle which is not distinctly classified as a parked vehicle. To overcome the limitations, we analyzed the significant decision factors in the traffic scenes and designed a Deep Neural Network (DNN) model to make human-like overtaking maneuver decisions. The significant traffic-related and intention-related decision factors were harmoniously extracted in the traffic scene interpretation process and were utilized as the inputs of the model to generate overtaking maneuver decisions in the same manner with the human driver. The overall validation results convinced that the extracted decision factors contributed to increasing the learning performance of the model, and consequently, the proposed decision-making system enabled the autonomous vehicles to generate more human-like overtaking maneuver decisions in various urban traffic scenarios.

Visual Range Maneuver Decision of Unmanned Combat Aerial Vehicle Based on Fuzzy Reasoning

Visual Range Maneuver Decision of Unmanned Combat Aerial Vehicle Based on Fuzzy Reasoning

Indirect Adaptive Robust Control Design for Course Tracking of Ships Subject to Unknown Control Coefficient and Disturbances

For course control of ships with unknown control coefficient and model parameters, an indirect adaptive robust controller, in which the parameter estimation law and the control law are designed separately, is proposed. This design method can achieve not only excellent course control performance but also accurate parameter estimates for secondary purposes such as assisting in ship maneuvering decision. Firstly, a Nussbaum function is combined with the adaptive dynamic surface control method to design a strong robust controller which can ensure the stability of the closed-loop ship course control system in spite of parameter uncertainties, unknown control coefficient and disturbances. Secondly, the nonlinear model for ship steering is converted into linear form by using the X-swapping technique. And a modified least-squares identification algorithm is then proposed to estimate the unknown model parameters. The global uniform ultimate boundedness of all signals of the resulting closed-loop system is guaranteed via Lyapunov stability theory. Lastly, simulation results are executed to demonstrate the effectiveness of the proposed design method.

Risk assessment of recent high-interest conjunctions

There have been several recent high-profile space events, notably the conjunctions of the Aeolus and Starlink spacecraft (both maneuverable) and the IRAS and GGSE-4 spacecraft (both dead). Here, we examine the timeline, activities, and risks associated with each of these conjunction events. To accomplish this, we fused observations from the Space Surveillance Network (SSN) to produce a sequence of high-accuracy orbit solutions and used that sequence to generate three-dimensional, dynamic progressions of each conjunction's miss distance, collision probability and its relation to the dilution threshold. Probability of Collision (Pc) is an essential tool for assessing conjunction threats. The Pc Topology tool characterizes the evolution of a conjunction's miss distance and collision probability, both of which are critical inputs to the maneuver avoidance decision-making process and exposing SSA deficiencies. We show the effect of combined hardbody radius (CHBR) on Pc as well as modeling with different shapes. Additionally, the Pc Topology tool reveals the calculation's placement in/out of the dilution region.The European Space Agency commanded the Aeolus spacecraft to perform an emergency maneuver to avoid this high-probability collision hazard. Our results indicate that this maneuver was warranted based upon ESA's decision threshold criteria and the Pc Topology trending analysis. While both collision events fortunately did not occur, we estimated the potential debris fields that these events may have generated had they occurred. Realizing that large debris production might be a factor in maneuver decision making, we then assess the impact of these fragmentation fields upon estimated operator workload and subsequent collision risk.

Moving Time UCAV Maneuver Decision Based on the Dynamic Relational Weight Algorithm and Trajectory Prediction

To improve the accuracy and real-time performance of autonomous decision-making by the unmanned combat aerial vehicle (UCAV), a decision-making method combining the dynamic relational weight algorithm and moving time strategy is proposed, and trajectory prediction is added to maneuver decision-making. Considering the lack of continuity and diversity of air combat situation reflected by the constant weight in situation assessment, a dynamic relational weight algorithm is proposed to establish an air combat situation system and adjust the weight according to the current situation. Based on the dominance function, this method calculates the correlation degree of each subsituation and the total situation. According to the priority principle and information entropy theory, the hierarchical fitting function is proposed, the association expectation is calculated by using if-then rules, and the weight is dynamically adjusted. In trajectory prediction, the online sliding input module is introduced, and the long- and short-term memory (LSTM) network is used for real-time prediction. To further improve the prediction accuracy, the adaptive boosting (Ada) method is used to build the outer frame and compare with three traditional prediction networks. The results show that the prediction accuracy of Ada-LSTM is better. In the decision-making method, the moving time optimization strategy is adopted. To solve the problem of timeliness and optimization, each control variable is divided into 9 gradients, and there are 729 control schemes in the control sequence. Through contrast pursuit simulation experiments, it is verified that the maneuver decision method combining the dynamic relational weight algorithm and moving time strategy has a better accuracy and real-time performance. In the case of using prediction and not using prediction, the adaptive countermeasure simulation is carried out with the current more advanced Bayesian inference maneuvering decision-making scheme. The results show that the UCAV maneuvering decision-making ability combined with accurate prediction is better.

A Network-Based Conflict Resolution Approach for Unmanned Aerial Vehicle Operations in Dense Nonsegregated Airspace

To accommodate the growing demand for airspace access by various entrants, the worldwide air traffic-management system is undergoing the process of airspace integration, where unmanned aerial vehicles are at the forefront of change. As heterogeneous aircraft operate in nonsegregated airspace with distinct performances and separation requirements, numerous interactions and conflicts are expected, as is the conflict resolution (CR) difficulty that follows. Considering the domino effect, where one maneuver decision made on an individual may generate a cascading influence on its neighbors, an optimal resolution order that adjusts the fewest aircraft possible is crucial to resolution efficiency. Therefore, an in-depth investigation into couplings among aircraft is essential, which is not widely seen in the literature. To fill this gap, we propose a network-based CR approach to adjust a minimal number of conflict-critical aircraft while guaranteeing a global conflict-free environment. Specifically, a conflict network is first developed to analyze the pairwise conflict relations among aircraft, where the detection of a certain aircraft is termed as an edge, and conflict severity is measured as the weight of this edge. Moreover, an improved PageRank algorithm is designed to identify key aircraft that are bottlenecks of system safety. A centralized CR sequence allocation (SA) is further implemented to ensure that these key aircraft assume major resolution responsibility and deconflict first. Simulation results show that the approach maintains high resolution efficiency in terms of fewer aircraft adjustments and computing time as well as traffic efficiency in terms of reduced velocity and path deviations under mixed-operation scenarios.

Semantic-Level Maneuver Sampling and Trajectory Planning for On-Road Autonomous Driving in Dynamic Scenarios

Maneuver decision-making and trajectory planning play important roles in autonomous driving since a safe and flexible decision module is indispensable for navigation. Typical algorithms apply sampling methods to generate feasible trajectories. However, the fixed sampling distance and maneuver execution time in a sampling approach sacrifice the flexibility of algorithm. Moreover, since motion planning can be represented as a high-dimensional problem, it usually results in unnecessary samples that require additional resources to search the solution. Therefore, a semantic-level maneuver sampling and trajectory planning algorithm is proposed to solve the above problems. In the upper-level maneuver decision, the decision-making problem is formulated as a selection of the forward leading object. A semantic-level decision tree is built to sample long-term maneuver sequences, and the safety corridor of each maneuver sequence is calculated according to the surrounding environment. In the lower-level trajectory planning, the process is decoupled into longitudinal and lateral directions. First, a heuristic search method is proposed to generate longitudinal trajectory candidates for each maneuver sequence. Then, an exhaustive search algorithm is employed to synchronously generate the lateral trajectory within safety corridor. Among the generated trajectory candidates, the one with minimum cost will be chosen as the searching result. Furthermore, in order to improve the driving comfort, numerical optimization is adopted to refine the result by accounting for the constraints of kinematics and safety. Finally, the proposed method was evaluated through simulations of typical on-road dynamic scenarios, which help verify its performance with desirable computation efficiency of less than 32 ms.

Fighter Equipment Contribution Evaluation Based on Maneuver Decision

With the development of integrated technology and equipment systems, decision-making support has been increasingly applied in equipment research and development, especially in air combat. The development of airborne equipment can significantly improve the confrontation ability of manned fighters. Contribution evaluations restrict decision making in equipment research and development and are important in beyond-visual-range (BVR) air combat, which involves dynamic and uncertain enemy positions. This paper proposes a contribution evaluation model for BVR air combat based on autonomous maneuver decision making; this approach mainly includes a situation evaluation model, a maneuver decision model, and a one-to-one BVR air combat evaluation model. However, such a model includes a high-dimensional state and action space, which require many computations, especially if equipment changes occur. Then, an autonomous maneuver decision algorithm based on the influence diagram method is proposed; this algorithm uses a rolling time domain concept to improve combat fidelity and obtain useful equipment contribution evaluation results. Finally, one-to-one BVR air combat is simulated based on different aircraft models and airborne equipment. The simulation results show that the proposed maneuver decision model and countermeasure evaluation model can help experts quantify the contributions of equipment and provide adequate decision-making support to improve equipment systems.

Application of Deep Reinforcement Learning in Maneuver Planning of Beyond-Visual-Range Air Combat

Beyond-visual-range (BVR) engagement becomes more and more popular in the modern air battlefield. The key and difficulty for pilots in the fight is maneuver planning, which reflects the tactical decision-making capacity of the both sides and determinates success or failure. In this paper, we propose an intelligent maneuver planning method for BVR combat with using an improved deep Q network (DQN). First, a basic combat environment builds, which mainly includes flight motion model, relative motion model and missile attack model. Then, we create a maneuver decision framework for agent interaction with the environment. Basic perceptive variables are constructed for agents to form continuous state space. Also, considering the threat of each side missile and the constraint of airfield, the reward function is designed for agents to training. Later, we introduce a training algorithm and propose perceptional situation layers and value fitting layers to replace policy network in DQN. Based on long short-term memory (LSTM) cell, the perceptional situation layer can convert basic state to high-dimensional perception situation. The fitting layer does well in mapping action. Finally, three combat scenarios are designed for agent training and testing. Simulation result shows the agent can avoid the threat of enemy and gather own advantages to threat the target. It also proves the models and methods of agents are valid and intelligent air combat can be realized.

Air Combat Maneuver Decision Based on Reinforcement Genetic Algorithm

With the continuous development of UAV technology, the trend of using UAV in the military battlefield is increasingly obvious, but the autonomous air combat capability of UAV needs to be further improved. The air combat maneuvering decision is the key link to realize the UAV autonomous air combat, and the genetic algorithm has good robustness and global searching ability which is suitable for solving large-scale optimization problems. This paper uses an improved genetic algorithm to model UAV air combat maneuvering decisions. Based on engineering application requirements, a typical simulation test scenario is established. The simulation results show that the air combat maneuvering decision model based on reinforcement genetic algorithm in this paper can obtain the correct maneuvering decision sequence and gain a position advantage in combat.

Modeling lateral movement decisions of powered two wheelers in disordered heterogeneous traffic conditions

ABSTRACT This article reports a systematic investigation carried out to model the lateral movement decisions of Powered-Two-Wheelers (PTWs) in disordered heterogeneous traffic conditions. The lateral maneuvering decisions of PTWs were framed as a typical multiclass classification problem, and significant factors governing such decisions were identified. Four machine learning models, along with the statistical model, were built to achieve this objective. The comparative analysis regarding the predictive performance of these models shown that the random forest model outperforms the rest of the considered models in terms of its classification power. To this end, the results from this investigation revealed that the lateral movement pattern of PTWs could be predicted using speed and spatial information of its surrounding vehicles. Interestingly, this information can be seamlessly collected with some sensors presently deployed in the advanced vehicles. Thus, the developed models would help in the design of active safety and driving assistance systems for such vehicles.

Understanding and overcoming horizontal separation complexity in air traffic control: an expert/novice comparison

Humans still play a key role in air traffic control but their performances limit the capacity of the airspace and are responsible for delays. At the tactical level, even though air traffic controllers (ATCO) are trained for years, their performances are limited. In this article, we first isolated the tactical horizontal deconfliction task and explained its mathematical complexity. We observed through a simple experiment conducted on trainee and experienced ATCOs its complexity on random traffic in a part-task trainer displaying two to five aircraft trajectories at the same altitude. We compared performances of trainee ATCOs with experienced ATCOs using two different displays: a basic display showing information on aircraft positions and a dynamic visualization tool that represents the conflicting portions of aircraft trajectories and the evolution of the conflict zone when the user adds a maneuver to an aircraft. The tool allows the user to dynamically check the potential conflicting zones with the computer mouse before making a maneuver decision. Results showed that in easy situations (two aircraft), performance was similar with both displays and groups. However, as the complexity of the situations grows (from three to five aircraft), the dynamic visualization tool enables users to solve the conflicts more efficiently. Using the tool leads to fewer unsolved conflicts. Even if experienced ATCOs performed much better than trainee ATCOs on complex situations, they also performed much better with the conflict visualization tool than without on such situations.

Game Control Methods Comparison when Avoiding Collisions with Multiple Objects Using Radar Remote Sensing

This article formulates the concept of games in the field of process control theory in marine sciences and reviews the literature on the possible applications of games. The possible types of game control processes for moving objects are presented. A computer-aided object safe control in the game environment, with an appropriate steering system, is described based on radar remote sensing in order to avoid collisions with many other objects that are encountered. First, the basic model of object movement in the game environment is presented as a differential game with many objects, described by appropriate game state equations, state and steering restrictions, and a quality control index in the form of an integral and final payment of the game. Next, the surrogate models of the differential game are described in detail for the development of practical computer control programs using positional and matrix game models. Particular attention, in each type of game, is paid to the aspect of cooperation or lack of cooperation between objects in making maneuvering decisions. A computer simulation illustrates these considerations with game control programs at a sea-crossing situation where multiple objects were encountered. Safe object trajectories are compared using two methods of game control using positional and matrix game models while also considering cases with cooperation or non-cooperation of objects.