People underestimate the errors made by algorithms for credit scoring and recidivism prediction but accept even fewer errors

Felix G Rebitschek,Gert G Wagner,Gerd Gigerenzer

doi:10.1038/s41598-021-99802-y

Felix G Rebitschek, Gert G Wagner + Show 1 more

Open Access

https://doi.org/10.1038/s41598-021-99802-y

Copy DOI

Abstract

This study provides the first representative analysis of error estimations and willingness to accept errors in a Western country (Germany) with regards to algorithmic decision-making systems (ADM). We examine people’s expectations about the accuracy of algorithms that predict credit default, recidivism of an offender, suitability of a job applicant, and health behavior. Also, we ask whether expectations about algorithm errors vary between these domains and how they differ from expectations about errors made by human experts. In a nationwide representative study (N = 3086) we find that most respondents underestimated the actual errors made by algorithms and are willing to accept even fewer errors than estimated. Error estimates and error acceptance did not differ consistently for predictions made by algorithms or human experts, but people’s living conditions (e.g. unemployment, household income) affected domain-specific acceptance (job suitability, credit defaulting) of misses and false alarms. We conclude that people have unwarranted expectations about the performance of ADM systems and evaluate errors in terms of potential personal consequences. Given the general public’s low willingness to accept errors, we further conclude that acceptance of ADM appears to be conditional to strict accuracy requirements.

Highlights

This study provides the first representative analysis of error estimations and willingness to accept errors in a Western country (Germany) with regards to algorithmic decision-making systems (ADM)
We examine how accurately algorithms are expected to perform in predicting credit defaults, recidivism of an offender, suitability of a job applicant, and health behavior
Layperson’s knowledge about ADM systems is usually limited[12], if the algorithms themselves are not even secret. This is where layperson’s theories about people’s ADM systems—their theory of machine—become crucial[18]. What do they think about input, processing, and output, and, about quality and fairness of algorithmic compared to expert judgments? Given the limited possibilities to observe ADM errors, expected instead of observed performance deficits may underpin critical attitudes of the public toward ADM

Summary

Introduction

This study provides the first representative analysis of error estimations and willingness to accept errors in a Western country (Germany) with regards to algorithmic decision-making systems (ADM). We examine people’s expectations about the accuracy of algorithms that predict credit default, recidivism of an offender, suitability of a job applicant, and health behavior. Error estimates and error acceptance did not differ consistently for predictions made by algorithms or human experts, but people’s living conditions (e.g. unemployment, household income) affected domain-specific acceptance (job suitability, credit defaulting) of misses and false alarms. This study provides the first representative analysis of error estimations and willingness to accept errors in a Western population (in Germany) with regards to specific algorithmic decision-making (ADM) systems. We examine how accurately algorithms are expected to perform in predicting credit defaults, recidivism of an offender, suitability of a job applicant, and health behavior. To the best of our knowledge (after a literature search in Web of Science, PsycNet, and Google Scholar, which revealed eleven survey studies on algorithm p erception9,10,12–17,25–27), this is the first representative study comparing error estimates and the willingness to accept errors for ADM

Objectives

Methods

Results

Conclusion