Managing unintended consequences of algorithmic decision-making: The case of Robodebt

Abstract

In 2016, the Australian welfare agency Centrelink implemented an information system to automate the identification and recollection of welfare overpayments. Such algorithmic decision-making systems are increasingly leveraged to improve the efficiency of public administration. However, Centrelink’s scheme went horribly wrong: the system, branded as “Robodebt” by the popular media, generated debt notices that were inaccurate and based on insufficient evidence. Numerous vulnerable citizens who received the debt notices suffered a great deal of distress. While public controversy ensued, the ruling government continued to defend the flawed system until a court decision ruled it unlawful in 2019. This teaching case challenges one to analyze what went wrong in the implementation and management of the Robodebt system through the lens of sociotechnical systems.