Reviewable Automated Decision-Making

Abstract

In this paper we introduce the concept of ‘review-ability' as an alternative approach to improving the accountability of automated decision-making that involves machine learning systems. In doing so, we draw on an understanding of automated decision-making as a socio-technical process, involving both human (organisational) and technical components, beginning before a decision is made and extending beyond the decision itself. Although explanations for automated decisions may be useful in some contexts, they focus more narrowly on the model and therefore do not provide the information about that process as a whole that is necessary for many aspects of accountability, regulatory oversight, and assessments for legal compliance. Drawing on previous work on the application of administrative law and judicial review mechanisms to automated decision-making in the public sector, we argue that breaking down the automated decision-making process into its technical and organisational components allows us to consider how appropriate record-keeping and logging mechanisms implemented at each stage of that process would allow for the process as a whole to be reviewed. Although significant research is needed to explore how it can be implemented, we argue that a review-ability framework potentially offers for a more useful and more holistic form of accountability for automated decision-making than approaches focused more narrowly on explanations.