A probabilistic metric for comparing metaheuristic optimization algorithms

Abstract

The evolution of metaheuristic optimization algorithms towards identification of a global minimum is based on random numbers, making each run unique. Comparing the performance of different algorithms hence requires several runs, and some statistical metric of the results. Mean, standard deviation, best and worst values metrics have been used with this purpose. In this paper, a single probabilistic metric is proposed for comparing metaheuristic optimization algorithms. It is based on the idea of population interference, and yields the probability that a given algorithm produces a smaller (global?) minimum than an alternative algorithm, in a single run. Three benchmark example problems and four optimization algorithms are employed to demonstrate that the proposed metric is better than usual statistics such as mean, standard deviation, best and worst values obtained over several runs. The proposed metric actually quantifies how much better a given algorithm is, in comparison to an alternative algorithm. Statements about the superiority of an algorithm can also be made in consideration of the number of algorithm runs and the number of objective function evaluations allowed in each run.