The Right to Be an Exception to a Data-Driven Rule

Abstract

Data-driven tools are increasingly used to make consequential decisions. In recent years, they have begun to advise employers on which job applicants to interview, judges on which defendants to grant bail, lenders on which homeowners to give loans, and more. In such settings, different data-driven rules result in different decisions. The problem is, to every data-driven rule, there are exceptions. While a data-driven rule may be appropriate for some, it may not be appropriate for all. In this case study, we argue that individuals have the right to be an exception to a data-driven rule. That is, they should not be held responsible when they are, through no fault of their own, the data-driven exceptions. We motivate the right and explain why it is not addressed by existing frameworks. The right also places a duty on decision makers—that the presumption should not always be that a data-driven rule is appropriate for every decision-subject, but rather that the data-driven rule must be justified for use on a given decision-subject, providing a level of consideration fit to the risk of harm. We provide a framework for justifying and contesting on the basis of the right to be an exception. The framework requires that a data-driven rule be assessed in terms of three components: individualization, uncertainty, and harm. We emphasize the importance of uncertainty—that the decision maker utilize a data-driven recommendation only if the levels of individualization and certainty are high enough to justify the level of harm that would result from that recommendation.