There Is No Ethical Automation: Stanislav Petrov’s Ordeal by Protocol

Abstract

ABSTRACT While the story of Stanislav Petrov – the Soviet Lieutenant Colonel who likely saved the world from nuclear holocaust in 1983 – is often trotted out to advocate for the view that human beings ought to be kept “in the loop” of automated weapons’ responses, I argue that the episode in fact belies this reading. By attending more closely to the features of this event – to Petrov’s professional background, to his familiarity with the warning system, and to his decisions to defy his protocol – it becomes clear that Petrov was not seamlessly working “in his loop,” but reassessing it entirely. I claim that this exhibits a paradox intrinsic to all automated loops: namely, that their optimum function in fact rests on unforeseen human interventions that cannot be reliably codified ex ante, that is, that good judgment cannot simply be “programmed into” their protocols. This dependence, moreover, reveals automation’s ineluctable need for virtue ethics – not in the usual sense (whereby ethics deliberates about, say, possible damages and loss of life), but in the sense that ethical judgment, rightly understood, entails the reassessment of all of the conditions surrounding a decision, including whether a protocol should be followed at all.