A graph-theoretic approach toward autonomous skill acquisition in reinforcement learning

Abstract

Hierarchical reinforcement learning facilitates learning in large and complex domains by exploiting subtasks and creating hierarchical structures using these subtasks. Subtasks are usually defined through finding subgoals of the problem. Providing mechanisms for autonomous subgoal discovery and skill acquisition is a challenging issue in reinforcement learning. Among the proposed algorithms, a few of them are successful both in performance and also efficiency in terms of the running time of the algorithm. In this paper, we study four methods for subgoal discovery which are based on graph partitioning. The idea behind the methods proposed in this paper is that if we partition the transition graph, then the edges connecting two partitions and the end points of these edges are good candidates for subgoals. Hence, we proposed some methods for partitioning the transition graphs and evaluate them on some benchmark problems. The first method uses a genetic algorithm to partition the transition graph. The second method uses graph partitioning learning automata. We provide optimizations to the previous method in the third proposed method. Having observed the features of these methods and their capabilities and drawbacks in subgoal discovery, we propose a novel method, which is based on strongly connected components, to find subgoal states and create skills in early episodes of learning. The proposed method brings together the advantages of the graph partitioning method as well as the experience of graph traversals in earlier episodes. The performance of the proposed method is evaluated by conducting several experiments on famous problems. The results show high accuracy in identifying subgoals that help the learning agent reach its goals expeditiously.