Enhancing Peer Fairness via Data-Driven Analysis for Outlier Detection

Abstract

Fairness in peer review is of vital importance in academic activities. Current peer review systems focus on matching suitable experts with proposals but often ignore the existence of outliers. Previous research has shown that outlier scores in reviews could decrease the fairness of these systems. Therefore, outlier detection in peer review systems is essential for maintaining fairness. In this paper, we introduce a novel method that employs data-crossing analysis to detect outlier scores, aiming to improve the reliability of peer review processes. We utilize a confidential dataset from a review organization. Due to the inability to access ground truth scores, we systematically devise data-driven deviations from an estimated ground truth through data-crossing analysis. These deviations reveal inconsistencies and abnormal scoring behaviors of different reviewers. Subsequently, the review process is strengthened by providing a structured mechanism to identify and mitigate biases. Extensive experiments demonstrate its effectiveness in improving the accuracy and fairness of academic assessments, contributing to the broader application of AI-driven methodologies to achieve more reliable and equitable outcomes.