Measuring robustness in rank aggregation based on the error-effectiveness curve

Abstract

Rank aggregation is an obligatory operation for many tasks of democratic elections, product recommendation, and gene identification. While the awareness of imperfect information in input rankings would lead to unreliable aggregation results has promoted the robust rank aggregation to become an increasingly active research topic recently. In this study, we focus on the problem of robustness measuring of rank aggregation methods. We first provide an analytical framework of rank aggregation robustness, in which we generate rankings with adjustable imperfect information and observe the response of rank aggregation methods. In particular, to quantify the robustness of rank aggregation methods, we introduce the concept of error-effectiveness curve, which presents the aggregation effectiveness under different imperfect information scenarios, and describes the ability of each method in giving stable aggregation results. By doing so, a robustness measure of rank aggregation with random error is developed. Comprehensive experimental evaluations were conducted considering synthetic datasets with various levels of random error to demonstrate the validity of the proposed measure. Exact robustness was quantified for each evaluated rank aggregation methods, and significant robustness distinctions were achieved among them.