Control-oriented imitation learning for atmospheric flight control of launch vehicles under wind disturbance

Abstract

The atmospheric flight control of launch vehicle is required to track the nominal trajectory while obeying path constraints on the aerodynamic load peak. Due to the wind disturbance, wind-induced angle of attack may increase the aerodynamic load peak, and reducing of it will lead to larger trajectory tracking error. Because of the cumbersome tuning of gain schedule and the dependency on accurate wind prediction, it is difficult for existing methods to handle the trade-off problem between reduction of the aerodynamic load peak and trajectory tracking error under uncertain wind disturbance. The presented research is intended to remedy this difficulty through learning based methods. The ascent flight phase is modelled as a multiple objective Markov decision process, where the trade-off under wind disturbance is precisely modelled as the optimization of a multi-objective vector under the stochastic state transition. A neural network flight control policy is then trained by a novel control-oriented imitation learning (CoIL) scheme. The CoIL scheme incorporates two control-oriented design within the standard reinforcement learning scheme: the inverse reinforcement learning that infers a proper reward from demonstration trajectories to represent the multi-objective vector of atmospheric flight control, as well as the policy hierarchy that simplifies interaction with the high order vehicle dynamics. Compared with previous learning-based attempts in the literature, this scheme uses orders of magnitude fewer demonstration trajectories and avoids hand-programmed reward design. Evaluations are performed under an ascent flight mission, indicating that the obtained policy outperforms the prevalent baseline controllers with reduction of the normalized maximum aerodynamic load by 28.3%, altitude error by 38.6%, velocity error by 17.1% and path angle error by 42.7% under unpredicted wind disturbance.