DCT: Differential Combination Testing of Deep Learning Systems

Abstract

Deep learning (DL) systems are increasingly used in security-related fields, where the accuracy and predictability of DL systems are critical. However the DL models are difficult to test and existing DL testing relies heavily on manually labeled data and often fails to expose erroneous behavior for corner inputs. In this paper, we propose Differential Combination Testing (DCT), an automated DL testing tool for systematically detecting the erroneous behavior of more corner cases without relying on manually labeled input data or manually checking the correctness of the output behavior. Our tool aims at automatically generating test cases, that is, applying image combination transformations to seed images to systematically generate synthetic images that can achieve high neuron coverage and trigger inconsistencies between multiple similar DL models. In addition, DCT utilizes multiple DL models with similar functions as cross-references, so that input data no longer must be manually marked and the correctness of output behavior can be automatically checked. The results show that DCT can find thousands of erroneous corner behaviors in the most commonly used DL models effectively and quickly, which can better detect the reliability and robustness of DL systems.