Dynamic heuristic acceleration of linearly approximated SARSA(lambda ): using ant colony optimization to learn heuristics dynamically

Abstract

Heuristically accelerated reinforcement learning (HARL) is a new family of algorithms that combines the advantages of reinforcement learning (RL) with the advantages of heuristic algorithms. To achieve this, the action selection strategy of the standard RL algorithm is modified to take into account a heuristic running in parallel with the RL process. This paper presents two approximated HARL algorithms that make use of pheromone trails to improve the behaviour of linearly approximated SARSA(lambda ) by dynamically learning a heuristic function through the pheromone trails. The proposed dynamic algorithms are evaluated in comparison to linearly approximated SARSA(lambda ), and heuristically accelerated SARSA(lambda ) using a static heuristic in three benchmark scenarios: the mountain car, the mountain car 3D and the maze scenarios.