Paradox-Free Data Analysis in Numerical Comparisons of Optimization Algorithms

Abstract

Numerical comparison is essential for evaluating an optimization algorithm. Unfortunately, recent research has shown that two paradoxes may occur, namely the cycle ranking paradox and survival of the nonfittest paradox. Further exploitation reveals that these paradoxes stem from the method of data analysis, especially its comparison strategy. Therefore, the design and widespread use of paradox-free data analysis methods has become urgent. This paper is dedicated to reviewing the recent progress in paradox in numerical comparisons of optimization algorithms, especially the reasons for paradoxes and the ways to eliminate them. Specifically, significant progress has been made in eliminating paradoxes from two popular data analysis methods, including hypothesis testing and methods based on the cumulative distribution function. Furthermore, in this paper the authors provide case studies, aiming to show that paradoxes are common and how to eliminate them with paradox-free data analysis methods.