A health-aware energy management strategy for fuel cell hybrid electric UAVs based on safe reinforcement learning

Abstract

Energy management strategies (EMSs) are crucial for the hydrogen economy and energy component lifetimes of fuel cell hybrid electric unmanned aerial vehicles (UAVs). Reinforcement learning (RL)-based schemes have been a hotspot for EMSs, but most of RL-based EMSs focus on the energy-saving performance and rarely consider energy component durability and safe exploration. This paper proposes a health-aware energy management strategy based on a safe RL framework to minimize the overall flight cost and achieve safe operation of UAVs. In this framework, a universal three-dimensional environment that integrates the UAV kinematics and dynamics model is developed. In addition, wind disturbances and random loading of the mission payload during flight are considered for robust training. The energy management problem is formulated as a constrained Markov decision process, where both hydrogen consumption and energy component degradation are incorporated in the multi-objective reward function. A safety optimizer is then designed to satisfy operation constraints by correcting the action through analytical optimization. The results indicate that the safety of the explored action is guaranteed, maintaining zero constraint violations in both training and real-time control scenarios. Compared with other RL-based methods, the proposed method had better convergence capability and reduced the training time. Furthermore, the simulation showed that the proposed method can reduce the total flight cost and fuel cell degradation by 14.6% and 15.3%, respectively, compared with the online benchmark method.