Experiment of reinforcement learning with extremum seeking

Abstract

Recent studies pay attention to the similarity between black-box optimization problem and reinforcement learning (RL) problem. They expect that black-box optimization algorithm can solve RL problems. Among the black-box optimization algorithms, extremum seeking (ES) is a notable one. Two past studies implicitly solved RL problem using ES, but their problem settings were limited to linear deterministic system. In this study, we propose the novel algorithm to solve a more general RL problem using ES. Specifically, we employ a new objective function and online optimization technique to solve a problem with stochastic non-linear state transition environment. As experiment, the proposed method solves two tasks. First, it solves a robot arm task in order to compare with the previous method PoWER. Second, it solves the one-dimensional reaching task to know the effects of exploration manner. As a result, we found that our algorithm showed better performance in adaptability than PoWER, and clarified that the initial exploration noise strongly affects the direction of search.