Properties of Group Fairness Measures for Rankings

Abstract

In recent years, several measures have been developed for evaluating group fairness of rankings. Given that these measures were developed with different application contexts and ranking algorithms in mind, it is not straightforward which measure to choose for a given scenario. Previous work has already laid out some categorizations of measures and explored relationships between these, however, there has not yet been a thorough mathematical analysis of practically grounded properties of individual measures. In this paper, we therefore apply an axiomatic approach to perform a comprehensive analysis of existing group fairness measures that have been developed in the context of fair ranking. To this end, we propose a set of fourteen properties for group fairness measures that consider different ranking settings. These properties specifically provide information about how fairness scores of ranked outputs can be interpreted and contextualized. For a given use case, one can then identify which properties are of interest, and select a measure based on whether it satisfies these properties. We further apply our properties on twelve existing group fairness measures, and through both empirical and theoretical results demonstrate that most of these measures only satisfy a small subset of the proposed properties. These findings highlight limitations of existing measures, and provide insights into how to evaluate and interpret different fairness measures in practical deployment. Overall, our work can assist practitioners in selecting appropriate group fairness measures for a specific application, and also aid researchers in designing and evaluating such measures.