An Approach for Formal Verification of Machine Learning based Complex Systems

Abstract

AbstractA complex system is characterized by emergence of global properties which are very difficult, if not impossible, to anticipate just from complete knowledge of component behaviors. Emergence, hierarchical organization and numerosity are some of the characteristics of complex systems. With increasing system complexity, achieving confidence in systems becomes increasingly difficult. With the recent trend towards significant footprint of complex system's functionality being governed by machine learning based models and algorithms, there is a need to ensure that emergent behavior associated with such systems are well analyzed and understood. Traditional verification and validation approaches are often inadequate to bring in the nuances of potential emergent behavior, which may be positive or negative. This paper describes a novel approach towards application of formal methods for analyzing and evaluating emergent behavior of complex systems that are governed by machine learning models. The proposed approach involves developing a machine learning classifier model that learns on potential negative and positive emergent behaviors, and leveraging the classifier in a formal verification model checking environment to assert negative emergent behavior. The approach is illustrated through an example of a pitch control system of an aircraft. The effectiveness and performance of the approach are quantified.