Skill-Based Hierarchical Reinforcement Learning for Target Visual Navigation

Abstract

Target visual navigation aims at controlling the agent to find a target object based on a monocular visual RGB image in each step. It is crucial for the agent to adapt to new environments. As target visual navigation is a complex task, understanding the behavior of the agent is beneficial for analyzing the reasons for failure. This work focuses on improving the readability and success rate of navigation policies. In this paper, we propose a framework named Skill-based Hierarchical Reinforcement Learning (SHRL) for target visual navigation. SHRL contains a high-level policy and three low-level skills. The high-level policy accomplishes the task by utilizing or stopping low-level skills at each step. Low-level skills are designed to separately solve three sub-tasks, i.e., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Search, Adjustment</i> , and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Exploration</i> . In addition, we propose an Abstract Representation and two penalty items to feed robust features to the high-level policy. Abstract Representation is designed to focus on selecting low-level skills rather than the details of navigation. Experimental results in the artificial environment AI2-Thor indicate that the proposed method outperforms state-of-the-art by a large margin in unseen indoor environments. Moreover, we also provide case studies to illustrate the advantages of SHRL.