Identification of Failure Modes in the Collision Avoidance System of an Autonomous Ferry using Adaptive Stress Testing

Abstract

As complex autonomous systems emerge in the maritime sector, measures must be taken in order to ensure thorough safety assessment. Real-world testing can be costly and potentially dangerous, and therefore there is a need for suitable simulation-based methods. This paper presents an implementation of the Adaptive Stress Testing (AST) method applied to the collision avoidance (COLAV) system of a small passenger ferry. AST is a simulation-based technique which has shown promising results in safety assessment of aviation and automobile systems. Given a simulator of a system, AST uses reinforcement learning to optimize toward system failure, and returns the most likely failure scenarios. AST is here shown to successfully identify scenarios where the criteria for failure are met, which is when the ferry collides with an adversary vessel controlled by AST. However, most of the initial results exhibit failures where the COLAV system of the ferry is not responsible for the failure, making the results less valuable to system developers. To improve the relevance, augmentations are made to the optimization problem. The augmentations result in four distinct problem formulations presented in the paper. Finally, the results are clustered using an unsupervised machine learning method called Soft Dynamic Time Warping k-means clustering in order to present a general summary of the identified failure scenarios. Our results demonstrate the relevance and potential of AST for the maritime sector and illustrate how common drawbacks of AST can be circumvented by method adjustment.