Station-keeping for high-altitude balloon with reinforcement learning

Abstract

The operating altitudes of the stratospheric balloons can reach near-space altitudes of more than 20 km, where the appearance of the quasi-zero wind layer follows the seasonal rule. As unmanned aerial vehicles with application potential, the effective flight control of balloon position is crucial. This research develops a station-keeping control approach based on reinforcement learning, and the control strategy also considers the characteristics of the local wind field. Firstly, an atmospheric environment model with an uncertain wind field is established according to the analysis of the historical wind data. The model serves as a training environment for the balloon station-keeping strategy training. Secondly, the thermal model, dynamic model, and altitude control model are introduced, and an environment based on historical real wind data is developed. Thirdly, the dueling double Q-learning deep network with prioritized experience replay method is applied to the station keeping of high-altitude balloons. The Priority Experience Replay based on High-Value Samples (HVS-PER) is developed to improve the stability of strategy training. Finally, the performance of the optimal network is evaluated by the total reward, horizontal displacement, and effective working time under the uncertain wind environment. The strategy analysis also has reference to exploiting appropriate initial positions and capturing the opportunity of releasing a balloon. This work confirms that the control strategy is viable in complex and variable wind field environments and is capable of long-duration flights.