Comparative optimizer rank and score: A modern approach for performance analysis of optimization techniques

Abstract

The performance analysis of optimization techniques is very important to understand the strengths and weaknesses of each technique. It is not very common to find an optimization technique that performs equally on all optimization problems, and the numbers offered by the most common performance measures, the achieved function value (fitness) and the number of function evaluations, are not representative by their own. For instance, reporting that an optimization technique O on a benchmark function B achieved a fitness F after a number of evaluations E is not semantically meaningful. Some of the logical questions that would arise for such report are: (a) how other techniques performed on the same benchmark, and (b) what are the characteristics of this benchmark (for example, modality and separability). The comparative optimizer rank and score (CORS) proposes an easy to apply and interpret method for the investigation of the problem solving abilities of optimization techniques. CORS offers eight new performance measures that are built on the basic performance measures (that is, achieved fitness, number of function evaluations, and time consumed). The CORS performance measures represent the performance of an optimization technique in comparison to other techniques that were tested under the same benchmarks, making the results more meaningful. Besides, these performance measures are all normalized in a range from 1 to 100, which helps the results to keep well-interpretable by their own. Furthermore, all the CORS performance measures are aggregatable, in which the results are easily accumulated and represented by the common characteristics defining optimization problems (such as dimensionality, modality, and separability), instead of a per benchmark function basis (such as F1, F2, and F3). In order to demonstrate and validate the CORS method, it was applied to the performance data of eight novel optimization techniques of the recent contributions to metaheuristics, namely, the bat algorithm (BA), cuckoo search (CS), differential search (DS), firefly algorithm (FA), gravitational search algorithm (GSA), one rank cuckoo search (ORCS), separable natural evolution strategy (SNES), and exponential natural evolution strategy (xNES). These performance data were generated by 96 tests of 16 benchmark functions and 6 dimensionalities. Along with the basic and CORS performance data, the aggregated CORS results were found to offer a very helpful knowledge regarding the performance of the examined techniques.