Review of the Research Landscape of Multi-Criteria Evaluation and Benchmarking Processes for Many-Objective Optimization Methods: Coherent Taxonomy, Challenges and Recommended Solution

Abstract

Evaluation and benchmarking of many-objective optimization (MaOO) methods are complicated. The rapid development of new optimization algorithms for solving problems with many objectives has increased the necessity of developing performance indicators or metrics for evaluating the performance quality and comparing the competing optimization algorithms fairly. Further investigations are required to highlight the limitations of how criteria/metrics are determined and the consistency of the procedures with the evaluation and benchmarking processes of MaOO. A review is conducted in this study to map the research landscape of multi-criteria evaluation and benchmarking processes for MaOO into a coherent taxonomy. Then contentious and challenging issues related to evaluation are highlighted, and the performance of optimization algorithms for MaOO is benchmarked. The methodological aspects of the evaluation and selection of MaOO algorithms are presented as the recommended solution on the basis of four distinct and successive phases. First, in the determination phase, the evaluation criteria of MaOO are collected, classified and grouped for testing experts’ consensus on the most suitable criteria. Second, the identification phase involves the process of establishing a decision matrix via a crossover of the ‘evaluation criteria’ and MaOO’, and the level of importance of each selective criteria and sub-criteria from phase one is computed to identify its weight value by using the best–worst method (BWM). Third, the development phase involves the creation of a decision matrix for MaOO selection on the basis of the integrated BWM and VIKOR method. Last, the validation phase involves the validation of the proposed solution.