Test case prioritization with neuron valuation based pattern

Abstract

• Filtering error-triggering data in testing DNNs is cost-effective. • The set of test cases may contain various types of data. • Patterns for DNN-based classifiers can help in prioritizing the abnormal inputs. • Our pattern consists of neurons and the computed thresholds for pattern matching. • Evaluation results show our advantage in prioritizing error-triggering samples. Deep neural networks (DNNs) have been increasingly adopted in various applications. Systematic verification and validation are essential techniques to guarantee the quality of such systems. Testing is one of the feasible solutions for system validation. However, testing DNNs usually requires a large number of test cases and leads to the high labeling cost. Meanwhile, the test cases may contain various noisy data, which also increases the burden of testing. In this paper, we put forwards a test case prioritization method for DNN-based classifiers, which assigns high priorities to those cases that can lead to problematic classifications. The test cases are prioritized w.r.t. the pattern extracted from the neuron outputs. The difference between the pattern acquired from the training set and the neurons matched from a given input determines the priority of the input. For a trained model, the method consists of two steps. First, we collect neuron output values over the training set and construct a neuron-based pattern for every class of training samples. Second, the metrics are computed according to the comparison between neuron output values of the DNN model from an input and the selected pattern. We carry out the experimentation over three popular datasets with various neural network structures. The extensive experimental results demonstrate that the efficiency and the generalizability of the prioritization method outperform most of the existing techniques.