Algorithmic bias in social research: A meta-analysis.

Alrik Thiem,Tim Haesebrouck,Tim Haesebrouck,David Sanchez,Lusine Mkrtchyan,Sandro Meloni

doi:10.1371/journal.pone.0233625

Alrik Thiem, Tim Haesebrouck + Show 4 more

Open Access

https://doi.org/10.1371/journal.pone.0233625

Copy DOI

Abstract

Both the natural and the social sciences are currently facing a deep "reproducibility crisis". Two important factors in this crisis have been the selective reporting of results and methodological problems. In this article, we examine a fusion of these two factors. More specifically, we demonstrate that the uncritical import of Boolean optimization algorithms from electrical engineering into some areas of the social sciences in the late 1980s has induced algorithmic bias on a considerable scale over the last quarter century. Potentially affected are all studies that have used a method nowadays known as Qualitative Comparative Analysis (QCA). Drawing on replication material for 215 peer-reviewed QCA articles from across 109 high-profile management, political science and sociology journals, we estimate the extent this problem has assumed in empirical work. Our results suggest that one in three studies is affected, one in ten severely so. More generally, our article cautions scientists against letting methods and algorithms travel too easily across disparate disciplines without sufficient prior evaluation of their suitability for the context in hand.

Highlights

Both the natural and the social sciences are currently facing a deep “reproducibility crisis” [1,2,3,4]
Even when everyone involved in the production and release of scientific work—authors, reviewers, and journal editors—is assumed to have acted as ethically and objectively as they possibly could, and even when the analyzed data are assumed not to be beset by any other problems impacting negatively on the quality of the reported results, our analysis suggests that every third Qualitative Comparative Analysis (QCA) solution presented in a management, political science or sociology journal has oversold its solution because the algorithm in the chosen QCA software operated under an objective function that is suitable for one specific purpose in electrical engineering applications, but not for causal inference
For almost all QCA solutions that are affected by algorithmic bias, at least half of all viable candidate models have never been brought to the attention of researchers or the readers of these researchers’ published article that presented this solution

Summary

Introduction

Both the natural and the social sciences are currently facing a deep “reproducibility crisis” [1,2,3,4]. The uncritical import of Boolean optimization algorithms from electrical engineering into causal data analysis with QCA has generated such bias through a transfer of context.

Results

Conclusion