Unintended Behavior in Learning-Enabled Systems: Detecting the Unknown Unknowns

Abstract

One of the important certification objectives for airborne software is demonstrating the absence of unintended behavior. In current software development processes, unintended behavior is associated with some identifiable structural feature, such as specific lines of code or a model element. However, in learning-enabled systems (like neural networks or other machine learning approaches), unintended behavior emerges from the data used to train the system. New inputs not encountered during training may result in novel activations in a neural network, leading to unexpected (and potentially dangerous) outputs. In this paper we will first review the rationale and methods for detecting unintended behavior in current airborne software systems, including the use of model based development techniques and formal methods for software verification. Then we will consider the challenges posed by learning-enabled components (LECs) and examine new techniques that are being developed to address these challenges, as well as how these techniques may shape new certification guidance.