Adaptive Stress Testing of Safety-Critical Systems

Abstract

Stress testing in simulation plays a critical role in the validation of safety-critical systems, including aircraft, cars, medical devices, and spacecraft. The analysis of failure events is important in understanding the causes and conditions of failure, informing improvements to the system, and the estimation and categorization of risk. However, stress testing of safety-critical systems can be very challenging. Finding failure events can be difficult due to the size and complexity of the system, interactions with an environment over many time steps, and rarity of failure events. While Monte Carlo sampling is frequently used in practice, it can be very inefficient when the algorithm is undirected. We present adaptive stress testing (AST), an accelerated stress testing method for finding the most likely path to a failure event. Adaptive stress testing formulates stress testing as a sequential decision process and then uses reinforcement learning to optimize it. By using learning during search, the algorithm can automatically discover important parts of the state space and adaptively focus the search. We apply adaptive stress testing to stress test a prototype of next-generation aircraft collision avoidance system in simulated encounters, where we find and analyze the most likely paths to near mid-air collision.