H3E: Learning air combat with a three-level hierarchical framework embedding expert knowledge

Abstract

Learning to perform air combat autonomously has been a long-standing challenge. The design of intelligent game strategies was difficult due to complex dynamic constraints and long decision-making process. Most existing approaches depend heavily on simplified aircraft models or complex hand-crafted rules. Different from previous works, this paper presents H3E, a novel and efficient Three-level Hierarchical decision framework embedding Expert knowledge, which gives birth to various strategies for high-fidelity one-on-one beyond-visual-range (BVR) air combat game. Inspired by the way pilots make decisions, we build a hierarchical framework to divide the air combat into several sub systems in which each level can perform its own task with much smaller exploration space. In addition, to make full use of the expert knowledge while minimizing its limitation, a novel “Rule-Imitation-Reinforcement” (RIR) training paradigm with an adjustable expert-guidance (AEG) loss function is established, which can increase the exploration efficiency as well as the game win rate. Finally, this work is evaluated in the Intelligent Air Game Simulator (IAGSim), a high-fidelity air combat simulation platform, through a series of games against the state-of-the-art (SOTA) methods. The learning process and game results verify the superior performance of our framework in terms of the exploration efficiency (higher rewards with the same training samples) and win rate (at least 72.9%) compared with the existing SOTA approaches.