Autonomous systems, superior orders and manifest unlawfulness: is there a duty to disobey?

Abstract

The recent development of the next generation of ‘drones’ featuring some form of artificial intelligence or machine learning – the autonomous military system or AMS – has fuelled speculation about who the law ought to hold responsible for any negligence or malfeasance in their use. In a scenario recently posted by the Australian War College, an Air Force pilot relies on sensor information and artificial intelligence assessments provided by AMS which led to an (ultimately unlawful) order to attack a target. If that story played out in real life, what disciplinary liability should that ADF member face for their actions: do AMS fundamentally challenge the circumstances in which Defence Force members should disobey unlawful orders? Would disciplinary liability be assigned differently if the pilot was from another host jurisdiction? This article comparatively examines the superior orders defence in the domestic law, with a view to identifying the application of the duty to disobey in relation to orders involving AMS.