Optimal Agent Search Using Surrogate-Assisted Genetic Algorithms

Abstract

An intelligent agent is a program that can make decisions or perform a service based on its environment, user input, and experiences. Due to the complexity of its state and action spaces, agents are approximated by deep neural networks (DNNs), and it can be optimized using methods such as deep reinforcement learning and evolution strategies. However, these methods include simulation-based evaluations in the optimization process, and they are inefficient if the simulation cost is high. In this study, we propose surrogate-assisted genetic algorithms (SGAs), whose surrogate models are used in the fitness evaluation of genetic algorithms, and the surrogates also predict cumulative rewards for an agent’s DNN parameters. To improve the SGAs, we applied stepwise improvements that included multiple surrogates, data standardization, and sampling with dimensional reduction. We conducted experiments using the proposed SGAs in benchmark environments such as cart-pole balancing and lunar lander, and successfully found optimal solutions and significantly reduced computing time. The computing time was reduced by 38% and 95%, in the cart-pole balancing and lunar lander problems, respectively. For the lunar lander problem, an agent with approximately 4% better quality than that found by a gradient-based method was even found.