Practical initialization of the Nelder–Mead method for computationally expensive optimization problems

Abstract

Black-box optimization (BBO) algorithms are widely employed by practitioners to address computationally expensive real-world problems such as automatic tuning of machine learning models and evacuation route planning. The Nelder–Mead (NM) method is a well-known local search heuristic for BBO that has been applied to solve many real-world problems from way back because of its promising performance. However, this method has a strong dependence on initialization due to its local search tendency. Nevertheless, a discussion on the proper initialization of the NM method is limited to the recent study by Wessing (Optim Lett 13(4):847–856, 2019), which is solely based on an analysis using the simple sphere function. In this study, we take a further step to improve Wessing’s result by massively investigating how the initialization affects the search performance in views of the initial simplex size and shape and a constraint handling method that is employed on 24 BBO benchmarking problems. Based on the numerical results, we present the empirical best practice for the initialization of the NM method for cases involving a limited evaluation budget.