An Actor-Critic Method for Simulation-Based Optimization

Abstract

In this work, we study simulation-based optimization, where the agent aims to select the best configuration from the design space with as few as possible iterations. Inspired by the success of deep reinforcement learning (DRL), we formulate the sampling process as policy searching and give a solving method from the perspective of policy iteration. Concretely, a surrogate model for predicting the performance of each configuration and a parameterized sampling policy are applied, which correspond to the critic and actor in actor-critic (AC) method, respectively. We further derive the updating rule and propose two algorithms for configuration selection in continuous and discrete design spaces, respectively. Finally, the algorithms are validated experimentally on 1) two toy examples to intuitively explain the principle and 2) two high-dimensional tasks to reveal the effectiveness in large-scale problems. The results show that the proposed algorithms can efficiently deal with large-scale problems and effectively eliminate sub-optimal configurations.