A simulation study on the probabilities of rank reversal, tie making, and tie breaking for multiple criteria decision making methods

Abstract

Multiple criteria decision making (MCDM) methods can be affected by preference reversal, meaning that the order of two alternatives is reversed after adding or deleting another alternative. Here, we focus on methods that produce rankings with ties (i.e., weak orders). In this context, one usually talks about rank reversal. Existing rank reversal probability simulation experiments are subject to improvement on the following points: (1) the small number of MCDM methods included, (2) the unclear relation between the rank reversal probability and the rank of the deleted alternative, and (3) the lack of consideration of ties. In this paper, considering both strict preferences and ties, we distinguish two new phenomena: tie breaking, i.e., the shift from a tie to a strict preference, and tie making, i.e., the shift from a strict preference to a tie. To investigate the probabilities of rank reversal, tie making, and tie breaking, a simulation study involving thirty versions of twelve MCDM methods and six simulation factors is set up. Results demonstrate that for MCDM methods using pairwise comparison data, deleting an alternative ranked first or last leads to smaller probabilities than deleting an alternative in the middle, while the opposite holds for the methods using evaluation data under criteria. Four findings and three suggestions are given to help decision makers select MCDM methods to use.