Mutation testing of unsupervised learning systems

Abstract

Unsupervised learning (UL) is one of the most important areas in artificial intelligence. UL systems are capable of learning patterns from unlabeled data and playing an increasingly critical role in many fields. Therefore, more and more attention has been paid to the security and stability of UL systems. Testing has achieved great success in ensuring the safety of traditional software systems and been gradually applied to supervised learning. However, UL is not in the consideration of most current testing methods. To fill this gap, we propose a novel mutation testing technique specific to UL systems. We design a series of mutation operators to simulate the unstable situations and possible errors that UL systems may encounter, and define corresponding mutation scores. Further, we combine the proposed technique with autoencoder for generating adversarial samples. In the evaluation phase, we demonstrate the practicability of the proposed technique based on three datasets. We have also developed an open-source tool MTGAN, which incorporates the functionality of mutation testing for GANs, to evaluate their stability and assess their capacity to address given issues.