Multi-agent target search strategy optimization: Hierarchical reinforcement learning with multi-criteria negative feedback

Abstract

Recently, using unmanned platforms to perform target search tasks has received extensive attention and research. The complexity of the search scenario and the unpredictability of the target movement pose significant challenges for unmanned platforms to perform the search task. Developing efficient search strategies is crucial for their success. In this study, we model the problem as a Partially Observable Markov Decision Process (POMDP) and propose a Hierarchical Deep Q Network with Multi-criteria Negative Feedback method named MNF-HDQN to solve the problem efficiently. The MNF-HDQN incorporates point and area evaluations instead of the original sparsity calculation to enhance the cooperation competence of unmanned platforms in searching for a target in various search tasks. Additionally, the integration of convex polygon theory into reward shaping and the design of a new two-stage search strategy encouragement function further improve the performance of the proposed method. We conduct detailed experiments to verify the superiority of the MNF-HDQN. And experimental results show that our method provides a high successful search rate in a shorter timestep compared with state-of-the-art baselines. This advantage is more evident when the search scenario is more complex or the target movement is more unpredictable.