Explainable Artificial Intelligence (XAI): A reason to believe?

Abstract

Artificial intelligence is an alluring technology which companies and governments hope to benefit from. In many circumstances a condition of its use is that humans can understand an explanation of why the action of an AI system took place. This has encouraged the development of a field of “explainable artificial intelligence”, or XAI. Much of the work in this field has been encouraged by the US Defense Advanced Research Projects Agency (DARPA), through its XAI program initiated in 2016. This paper argues that an underacknowledged challenge of XAI is that unlike most traditional technology, many AI systems contain inherent uncertainty. These systems are widely described as “black boxes”, and can be described only through their behavior, a technique described in the literature as post-hoc, rather than through an understanding of their functioning. Explaining such systems is akin to explaining the functioning of the natural world, rather than explaining the functioning of a known technology. While extensive work has been undertaken to explain the behavior of black box AI systems, there are limitations to the certainty that a post-hoc method can bring. Recognizing this is an important part of understanding the limitations of post-hoc reasoning in the use of advanced AI systems. Far simpler technologies have been seen to cause significant social damage: the UK Post Office Horizon system, and the Australian federal government Robodebt program. Coming to advanced AI system examples, two recent prestigious reports on AI systems and law display an unreasoned enthusiasm for AI explainability. AI researchers should be acknowledging that many advanced AI systems remain black boxes, that post-hoc explanations of these are inferences describing how the AI system may function, not how it does function, and the application of these technologies should be managed accordingly. Otherwise, the search for explanations may simply become a reason to believe.