An evolutionary variant of Q-learning applied to derivative-free optimization

Abstract

The design of complex aeronautical systems requires the solution of multidisciplinary design optimization (MDO) problems. Despite the considerable technological progress of the last decades, MDO is still computationally expensive and involves a large number of high�fidelity simulations to evaluate the objective functions. This paper describes a derivative-free optimization method, based on reinforcement learning, for global unconstrained optimization problems. Specifically, an evolutionary variant of the well-known Q-learning (QL) algorithm, namely the Evolutionary Q-learning (EVQL), is developed to reduce the computational cost needed to find the optimal solution. Both single- and multi-agent formulations’ performance are assessed on six analytical benchmark problems, showing how EVQL outperforms the QL in terms of learning process acceleration and accuracy. Furthermore, EVQL is also compared with a deterministic particle swarm optimization, providing comparable results. Finally, the EVQL algorithm is used for the solution of an aeroacoustic problem, pertaining to the identification of optimal engine installation to maximize noise shielding. Specifically, the multi-agent EVQL is used to find the minimum of the insertion loss (IL), conditional to the noise source location and the Mach number. The IL is given by a surrogate model, based on stochastic radial basis functions, and trained by a boundary element method solver. The EVQL has a double use for the present application: a) it is used to find the minimum of the objective function, b) it defines where to add new training points based on the surrogate uncertainty, which is used to define algorithm trust regions. to find the optimal solution of the surrogate function and to identify new samples to update the metamodel.