Combat Maneuvering Research Articles

UAV swarm air combat maneuver decision-making method based on multi-agent reinforcement learning and transferring

UAV swarm air combat maneuver decision-making method based on multi-agent reinforcement learning and transferring

Air combat maneuver decision based on deep reinforcement learning with auxiliary reward

Air combat maneuver decision based on deep reinforcement learning with auxiliary reward

Deep reinforcement learning-based air combat maneuver decision-making: literature review, implementation tutorial and future direction

Deep reinforcement learning-based air combat maneuver decision-making: literature review, implementation tutorial and future direction

An air combat maneuver pattern extraction based on time series segmentation and clustering analysis

Target maneuver recognition is a prerequisite for air combat situation awareness, trajectory prediction, threat assessment and maneuver decision. To get rid of the dependence of the current target maneuver recognition method on empirical criteria and sample data, and automatically and adaptively complete the task of extracting the target maneuver pattern, in this paper, an air combat maneuver pattern extraction based on time series segmentation and clustering analysis is proposed by combining autoencoder, G-G clustering algorithm and the selective ensemble clustering analysis algorithm. Firstly, the autoencoder is used to extract key features of maneuvering trajectory to remove the impacts of redundant variables and reduce the data dimension; Then, taking the time information into account, the segmentation of Maneuver characteristic time series is realized with the improved FSTS-AEGG algorithm, and a large number of maneuver primitives are extracted; Finally, the maneuver primitives are grouped into some categories by using the selective ensemble multiple time series clustering algorithm, which can prove that each class represents a maneuver action. The maneuver pattern extraction method is applied to small scale air combat trajectory and can recognize and correctly partition at least 71.3% of maneuver actions，indicating that the method is effective and satisfies the requirements for engineering accuracy. In addition, this method can provide data support for various target maneuvering recognition methods proposed in the literature, greatly reduce the workload and improve the recognition accuracy.

Beyond visual range maneuver intention recognition based on attention enhanced tuna swarm optimization parallel BiGRU

This paper researches the problem of Beyond Visual Range (BVR) air combat maneuver intention recognition. To achieve efficient and accurate intention recognition, an Attention enhanced Tuna Swarm Optimization-Parallel Bidirectional Gated Recurrent Unit network (A-TSO-PBiGRU) is proposed, which constructs a novel Parallel BiGRU (PBiGRU). Firstly, PBiGRU has a parallel network structure, whose proportion of forward and backward network can be adjusted by forward coefficient and backward coefficient. Secondly, to achieve object-oriented adjustment of forward and backward coefficients, the tuna swarm optimization algorithm is introduced and the negative log-likelihood estimation loss function is used as the objective function, it realizes the dynamic combination of sequence guidance and reverse correction. Finally, the attention mechanism is used to obtain more useful information to improve the recognition accuracy. Through offline recognition experiment, it is proved that A-TSO-PBiGRU can effectively improve the convergence speed and recognition accuracy compared with GRU-related networks. Compared with the other six comparison algorithms, maneuver intention recognition accuracy also has significant advantages. In the online recognition experiment, maneuver intention recognition accuracy of A-TSO-PBiGRU is 93.7%, it shows excellent maneuver intention recognition ability.

UCAV situation assessment method based on C-LSHADE-Means and SAE-LVQ

The unmanned combat aerial vehicle (UCAV) is a research hot issue in the world, and the situation assessment is an important part of it. To overcome shortcomings of the existing situation assessment methods, such as low accuracy and strong dependence on prior knowledge, a data-driven situation assessment method is proposed. The clustering and classification are combined, the former is used to mine situational knowledge, and the latter is used to realize rapid assessment. Angle evaluation factor and distance evaluation factor are proposed to transform multi-dimensional air combat information into two-dimensional features. A convolution success-history based adaptive differential evolution with linear population size reduction-means (C-LSHADE-Means) algorithm is proposed. The convolutional pooling layer is used to compress the size of data and preserve the distribution characteristics. The LSHADE algorithm is used to initialize the center of the mean clustering, which overcomes the defect of initialization sensitivity. Comparing experiment with the seven clustering algorithms is done on the UCI data set, through four clustering indexes, and it proves that the method proposed in this paper has better clustering performance. A situation assessment model based on stacked autoencoder and learning vector quantization (SAE-LVQ) network is constructed, and it uses SAE to reconstruct air combat data features, and uses the self-competition layer of the LVQ to achieve efficient classification. Compared with the five kinds of assessments models, the SAE-LVQ model has the highest accuracy. Finally, three kinds of confrontation processes from air combat maneuvering instrumentation (ACMI) are selected, and the model in this paper is used for situation assessment. The assessment results are in line with the actual situation.

Short-range air combat maneuver decision of UAV swarm based on multi-agent Transformer introducing virtual objects

With the development of Unmanned Aerial Vehicle (UAV) swarm technology, there has been a growing interest in using Artificial Intelligence (AI) to drive UAV swarms for short-range air combat. However, due to the complexity of situation information in UAV swarm air combat, making accurate air combat decisions based on air combat situation information has become a challenge. In this paper, we propose the multi-agent Transformer introducing virtual objects (MTVO) to address this issue. First, the proposed approach designs a multi-agent Transformer network structure by exploiting the homogeneity feature of the UAV state information in the swarm. This structure enables the structured processing of complex situation information. Specifically, the local situation information of each UAV is calculated by self-attention, which reduces the size of the swarm situation information while retaining the information of key UAVs. This approach reduces the difficulty of processing UAV swarm situation information. Moreover, we add a virtual object to the UAV swarm information to assist in calculating the weight distribution of the local situation. The weighted fusion of local situations allows us to obtain a more effective representation of the global situation, which serves as the basis for more accurate air combat maneuver decisions. We demonstrate the performance of the proposed method through air combat simulation results using Reinforcement Learning (RL) methods, and validate the applicability and effectiveness of the MTVO network for the short-range air combat problem of UAV swarms.

A Game-Theoretic Model for One-on-One Air Combat

The primary goal of air combat maneuvering is to neutralize the enemy under any circumstances. A critical step in achieving this goal is to gain and maintain aerial advantage by choosing and executing appropriate flighter maneuvers. In this paper, one-on-one air combat is modeled as a two-player zero-sum game to obtain optimal strategies for the players to select the flighter maneuvers. A new approach for constructing the game payoff function that accounts for the loss/gain of aerial advantage during the execution of a maneuver is proposed. A case study to evaluate the proposed approach, by comparing it with an existing approach, is performed through computer simulations.

Hierarchical reinforcement learning from competitive self-play for dual-aircraft formation air combat

AbstractThe recent development of technology helps in the revolutionary war and it controls the war which is influenced by brilliant planning. The maneuver aircraft of intelligent algorithm aids the pilot to decide the particular position on the battlefield. Nowadays the hardware components of radar and missiles are widely used and the beyond visual range is the most popular method applied in air combat. The introduction of close-range air combat maneuver decisions generates the attention of researchers in artificial intelligence. Most of the existing methods are based on autonomous aircraft focused in air combat scenario but manual air combats are widely applied in dual aircraft. Based on the factors mentioned above, a novel hierarchical maneuver decision architecture is applied to a dual-aircraft close-range air combat scenario. Subsequently, the soft actor-critic algorithm is merged with competitive self-play which integrates the knowledge of sub-strategies. Further, the reinforcement learning technique is employed to achieve an approximate Nash equilibrium master strategy. The experimental results show that the hierarchical architecture exhibits good performance, symmetry, and robustness. The research generates a solution for intelligent formation of air combat in the future and guidance for manned or unmanned aircraft cooperative combat.

UAV's air combat decision-making based on deep deterministic policy gradient and prediction

To solve the enemy uncertain manipulation problem during a UAV's autonomous air combat maneuver decision-making, this paper proposes an autonomous air combat maneuver decision-making method that combines target maneuver command prediction with the deep deterministic policy algorithm. The situation data of both sides of air combat are effectively fused and processed, the UAV's six-degree-of-freedom model and maneuver library are built. In air combat, the target generates its corresponding maneuver library instructions through the deep Q network algorithm; at the same time, the UAV on our side gives the target maneuver prediction results through the probabilistic neural network. A deep deterministic policy gradient reinforcement learning method that considers both the situation information of two aircraft and the prediction results of enemy aircraft is proposed, so that the UAV can choose the appropriate maneuver decision according to the current air combat situation. The simulation results show that the method can effectively use the air combat situation information and target maneuver prediction information so that it can improve the effectiveness of the reinforcement learning method for UAV's autonomous air combat decision-making on the premise of ensuring convergence.

UAV maneuver decision-making via deep reinforcement learning for short-range air combat

The unmanned aerial vehicle (UAV) has been applied in unmanned air combat because of its flexibility and practicality. The short-range air combat situation is rapidly changing, and the UAV has to make the autonomous maneuver decision as quickly as possible. In this paper, a type of short-range air combat maneuver decision method based on deep reinforcement learning is proposed. Firstly, the combat environment, including UAV motion model and the position and velocity relationships, is described. On this basic, the combat process is established. Secondly, some improved points based on proximal policy optimization (PPO) are proposed to enhance the maneuver decision-making ability. The gate recurrent unit (GRU) can help PPO make decisions with continuous timestep data. The actor network's input is the observation of UAV, however, the input of the critic network, named state, includes the blood values which cannot be observed directly. In addition, the action space with 15 basic actions and well-designed reward function are proposed to combine the air combat environment and PPO. In particular, the reward function is divided into dense reward, event reward and end-game reward to ensure the training feasibility. The training process is composed of three phases to shorten the training time. Finally, the designed maneuver decision method is verified through the ablation study and confrontment tests. The results show that the UAV with the proposed maneuver decision method can obtain an effective action policy to make a more flexible decision in air combat.

Air Combat Maneuver Decision Method Based on A3C Deep Reinforcement Learning

To solve the maneuvering decision problem in air combat of unmanned combat aircraft vehicles (UCAVs), in this paper, an autonomous maneuver decision method is proposed for a UCAV based on deep reinforcement learning. Firstly, the UCAV flight maneuver model and maneuver library of both opposing sides are established. Then, considering the different state transition effects of various actions when the pitch angles of the UCAVs are different, the 10 state variables including the pitch angle, are taken as the state space. Combined with the air combat situation threat assessment index model, a two-layer reward mechanism combining internal reward and sparse reward is designed as the evaluation basis of reinforcement learning. Then, the neural network model of the full connection layer is built according to an Asynchronous Advantage Actor–Critic (A3C) algorithm. In the way of multi-threading, our UCAV keeps interactively learning with the environment to train the model and gradually learns the optimal air combat maneuver countermeasure strategy, and guides our UCAV to conduct action selection. The algorithm reduces the correlation between samples through multi-threading asynchronous learning. Finally, the effectiveness and feasibility of the method are verified in three different air combat scenarios.

Autonomous Maneuver Decision of Air Combat Based on Simulated Operation Command and FRV-DDPG Algorithm

With the improvement of UAV performance and intelligence in recent years, it is particularly important for unmanned aerial vehicles (UAVs) to improve the ability of autonomous air combat. Aiming to solve the problem of how to improve the autonomous air combat maneuver decision ability of UAVs so that it can be close to manual manipulation, this paper proposes an autonomous air combat maneuvering decision method based on the combination of simulated operation command and the final reward value deep deterministic policy gradient (FRV-DDPG) algorithm. Firstly, the six-degree-of-freedom (6-DOF) model is established based on the air combat process, UAV motion, and missile motion. Secondly, a prediction method based on the Particle swarm optimization radial basis function (PSO-RBF) is designed to simulate the operation command of the enemy aircraft, which makes the training process more realistic, and then an improved DDPG strategy is proposed, which returns the final reward value to the previous reward value in a certain proportion of time for offline training, which can improve the convergence speed of the algorithm. Finally, the effectiveness of the algorithm is verified by building a simulation environment. The simulation results show that the algorithm can improve the autonomous air combat maneuver decision-making ability of UAVs.

Autonomous Air Combat Maneuvering Decision Method of UCAV Based on LSHADE-TSO-MPC under Enemy Trajectory Prediction

In this paper, an autonomous UCAV air combat maneuvering decision method based on LSHADE-TSO optimization in a model predictive control framework is proposed, along with enemy trajectory prediction. First, a sliding window recursive prediction method for multi-step enemy trajectory prediction using a Bi-LSTM network is proposed. Second, Model Predictive Control (MPC) theory is introduced, and when combined with enemy trajectory prediction, a UCAV maneuver decision model based on the MPC framework is proposed. The LSHADE-TSO algorithm is proposed by combining the LSHADE and TSO algorithms, which overcomes the problem of traditional sequential quadratic programming falling into local optimum when solving complex nonlinear models. The LSHADE-TSO-MPC air combat maneuver decision method is then proposed, which combines the LSHADE-TSO algorithm with the MPC framework and employs the LSHADE-TSO algorithm as the optimal control sequence solver. To validate the effectiveness of the maneuvering decision method proposed in this paper, it is tested against the test maneuver and the LSHADE-TSO decision algorithm, respectively, and the experimental results show that the maneuvering decision method proposed in this paper can beat the opponent and win the air combat using the same weapons and flight platform. Finally, to demonstrate that LSHADE-TSO can better exploit the decision-making ability of the MPC model, LSHADE-TSO is compared to various optimization algorithms based on the MPC model, and the results show that LSHADE-TSO-MPC can not only help obtain air combat victory faster but also demonstrates better decision-making ability.

Maneuver Decision-Making for Autonomous Air Combat Based on FRE-PPO

Maneuver decision-making is the core of autonomous air combat, and reinforcement learning is a potential and ideal approach for addressing decision-making problems. However, when reinforcement learning is used for maneuver decision-making for autonomous air combat, it often suffers from awful training efficiency and poor performance of maneuver decision-making. In this paper, an air combat maneuver decision-making method based on final reward estimation and proximal policy optimization is proposed to solve the above problems. First, an air combat environment based on aircraft and missile models is constructed, and an intermediate reward and final reward are designed. Second, the final reward estimation is proposed to replace the original advantage estimation function of the surrogate objective of proximal policy optimization to improve the training performance of reinforcement learning. Third, sampling according to the final reward estimation is proposed to improve the training efficiency. Finally, the proposed method is used in a self-play framework to train agents for maneuver decision-making. Simulations show that final reward estimation and sampling according to final reward estimation are effective and efficient.

Interdependent Autonomous Human-Machine Systems: The Complementarity of Fitness, Vulnerability and Evolution.

For the science of autonomous human–machine systems, traditional causal-time interpretations of reality in known contexts are sufficient for rational decisions and actions to be taken, but not for uncertain or dynamic contexts, nor for building the best teams. First, unlike game theory where the contexts are constructed for players, or machine learning where contexts must be stable, when facing uncertainty or conflict, a rational process is insufficient for decisions or actions to be taken; second, as supported by the literature, rational explanations cannot disaggregate human–machine teams. In the first case, interdependent humans facing uncertainty spontaneously engage in debate over complementary tradeoffs in a search for the best path forward, characterized by maximum entropy production (MEP); however, in the second case, signified by a reduction in structural entropy production (SEP), interdependent team structures make it rationally impossible to discern what creates better teams. In our review of evidence for SEP–MEP complementarity for teams, we found that structural redundancy for top global oil producers, replicated for top global militaries, impedes interdependence and promotes corruption. Next, using UN data for Middle Eastern North African nations plus Israel, we found that a nation’s structure of education is significantly associated with MEP by the number of patents it produces; this conflicts with our earlier finding that a U.S. Air Force education in air combat maneuvering was not associated with the best performance in air combat, but air combat flight training was. These last two results exemplify that SEP–MEP interactions by the team’s best members are made by orthogonal contributions. We extend our theory to find that competition between teams hinges on vulnerability, a complementary excess of SEP and reduced MEP, which generalizes to autonomous human–machine systems.

Air Combat Maneuver Strategy Algorithm Based on Two-Layer Game Decision-Making and Distributed Double Game Trees MCTS under Uncertain Information

In this paper, a model for maneuver decisions in air combat is established based on position situation information, the performance of the fighter, the threat of combat intention, and the multi-fighter collaboration effect. Additionally, a two-layer game decision algorithm based on the double game tree distributed Monte Carlo search strategy is proposed, and the operational rules of interval numbers and possibility degree comparison rules are adopted to solve the designed method. The experiment results show that the model and algorithm are effective in their intended purpose. The two-layer game decision-making and distributed double game tree MCTS can precut the huge game tree strategy space and quickly identify the optimal air combat game decision scheme, which improves the efficiency of strategy searches. By compared experiments, it was found that the proposed algorithm can improve the performance of air combat.

UCAV Air Combat Maneuver Decisions Based on a Proximal Policy Optimization Algorithm with Situation Reward Shaping

Autonomous maneuver decision by an unmanned combat air vehicle (UCAV) is a critical part of air combat that requires both flight safety and tactical maneuvering. In this paper, an unmanned combat air vehicle air combat maneuver decision method based on a proximal policy optimization algorithm (PPO) is proposed. Firstly, a motion model of the unmanned combat air vehicle and a situation assessment model of air combat was established to describe the motion situation of the unmanned combat air vehicle. An enemy maneuver policy based on a situation assessment with a greedy algorithm was also proposed for air combat confrontation, which aimed to verify the performance of the proximal policy optimization algorithm. Then, an action space based on a basic maneuver library and a state observation space of the proximal policy optimization algorithm were constructed, and a reward function with situation reward shaping was designed for accelerating the convergence rate. Finally, a simulation of air combat confrontation was carried out, which showed that the agent using the proximal policy optimization algorithm learned to combine a series of basic maneuvers, such as diving, climb and circling, into tactical maneuvers and eventually defeated the enemy. The winning rate of the proximal policy optimization algorithm reached 62%, and the corresponding losing rate was only 11%.

The effect of helmet mass and aircraft acceleration on cervical spine loads during typical fast jet aircraft pilot head motions

ObjectivesGravitational Force (Gz), head motion, and helmet mass are associated with neck pain in high performance aircraft pilots. Few studies have quantified neck kinetics (intersegmental neck moments) during aerial combat manoeuvres. DesignCross-sectional. MethodsWe quantified net joint moments between the skull and C1, and C6–7 during typical flight related headchecks using the Musculoskeletal Model for the Analysis of Spinal Injuries (MASI). We measured the influence of pilot-specific helmets and Gz on joint moments. Nineteen fighter pilots performed four head checks (check6 left, check6 right, extension hold and extension scan) under two helmet conditions. Motion data were transferred to OpenSim where joint moments were calculated at 1G to 9G. Net joint moments were compared across helmet conditions, Gz and headchecks. ResultsThe Joint Helmet Mounted Cueing System (JHMCS) resulted in higher moments at each segment- by a factor of 1.25 per unit of Gz, at C1, and by a factor of 1.08 per unit of Gz for C7. ExtensionScan and Check6Left were associated with the highest peak (96.13 Nm and 92.56 Nm). ExtensionScan and ExtensionHold accrued the highest mean cumulative loads at C7 at 9Gz (607.35 Nm.sec/motion, 362.99 Nm.sec/motion respectively). Asymmetries were observed between the Left and Right Check6 motions. High variability was evident between and within pilots. ConclusionsThe MASI model has been successfully applied to quantify intersegmental neck joint moments for typical headchecks that are performed during combat flight manoeuvres. In future, data derived from this model may inform conditioning, rehabilitative and preventative interventions to reduce neck pain in fast jet pilots.

UAV Cooperative Air Combat Maneuvering Confrontation Based on Multi-agent Reinforcement Learning

Focusing on the problem of multi-UAV cooperative air combat decision-making, a multi-UAV cooperative maneuvering decision-making approach is proposed based on multi-agent deep reinforcement learning (MARL) theory. First, the multi-UAV cooperative short-range air combat environment is established. Then, by combining the value-decomposition networks (VDNs) deep reinforcement learning theory with the embedded expert collaborative air combat experience reward function, an air combat cooperative strategy framework is proposed based on the networked decentralized partially observable Markov decision process (NDec-POMDP). The air combat maneuvering strategy is then optimized to improve the cooperative degree between UAVs in cooperative combat scenarios. Finally, multi-UAV cooperative air combat simulations are carried out and the results show the feasibility and effectiveness of the proposed cooperative air combat decision-making framework and method.