Formal Validation of Emergent Behavior in a Machine Learning Based Collision Avoidance System

Abstract

Collision Avoidance Systems play a major role in the development and integration of Unmanned Aerial Vehicles into the airspace. There has been extensive research on various collision avoidance algorithms and techniques. Typically, these algorithms and techniques involve sensing and detection, and approaches to avoidance in 2D or 3D scenarios. Recently, there has been an exponential increase on the adoption of various neural network based machine learning models for Collision Avoidance Systems. With this trend, the systems are becoming increasingly complex, and achieving confidence in these systems has become increasingly difficult. There is a need to ensure that emergent behavior associated with such complex systems are well analyzed and understood. A complex system is characterized by emergence of global properties which are very difficult, if not impossible, to anticipate just from complete knowledge of component behaviors. Emergence, hierarchical organization and numerosity are some of the characteristics of complex systems. Traditional verification and validation approaches are often inadequate to bring in the nuances of potential emergent behavior, which may be positive or negative. This paper describes a novel approach towards application of formal methods for analyzing and evaluating emergent behavior of Collision Avoidance Systems that are governed by machine learning models. The proposed approach involves developing a machine learning classifier model that learns on potential negative and positive emergent behaviors, and leveraging the classifier in a formal verification model checking environment to assert negative emergent behavior.