Proper initialization is crucial for the Nelder–Mead simplex search

Abstract

The Nelder–Mead simplex search as one of the oldest direct search methods is still in use today. Despite its flaws, such as possible convergence to non-stationary points, in practice the algorithm offers a competitive performance compared to other direct search algorithms. However, we show that some of the most common initialization procedures can increase the runtime of the algorithm by orders of magnitude, even on a trivial spherical test function. These results also hold for the Nelder–Mead variant with sufficient decrease condition and oriented restarts by Kelley and the variant with adaptive parameters by Gao and Han. As a remedy, we recommend to normalize the search space and to choose the initial simplex independent of the starting point.