A Systematic Review of Fairness, Accountability, Transparency and Ethics in Information Retrieval

Abstract

We live in an information society that strongly relies on information retrieval systems, such as search engines and conversational assistants. Consequently, the trustworthiness of these systems is of critical importance, and has attracted a significant research attention in recent years. In this work, we perform a systematic literature review of the field of fairness, accountability, transparency, and ethics in information retrieval. In particular, we investigate the definitions, approaches, and evaluation methodologies proposed to build trustworthy information retrieval systems. This review reveals the lack of standard definitions, arguably due to the multi-dimensional nature of the different notions. In terms of approaches, most of the work focuses on building either a fair or a transparent information retrieval system. As for evaluation, fairness is often assessed by means of automatic evaluation, while accountability and transparency are most commonly evaluated using audits and user studies. Based on the surveyed literature, we develop taxonomies of requirements for the different notions, and further use these taxonomies to propose practical definitions to quantify the degree to which an information retrieval system satisfies a given notion. Finally, we discuss challenges that have yet to be solved for information retrieval systems to be trustworthy.