Test Deep Neural Networks Research Articles

Input Distribution Coverage: Measuring Feature Interaction Adequacy in Neural Network Testing

Testing deep neural networks (DNNs) has garnered great interest in the recent years due to their use in many applications. Black-box test adequacy measures are useful for guiding the testing process in covering the input domain. However, the absence of input specifications makes it challenging to apply black-box test adequacy measures in DNN testing. The Input Distribution Coverage (IDC) framework addresses this challenge by using a variational autoencoder to learn a low dimensional latent representation of the input distribution, and then using that latent space as a coverage domain for testing. IDC applies combinatorial interaction testing on a partitioning of the latent space to measure test adequacy. Empirical evaluation demonstrates that IDC is cost-effective, capable of detecting feature diversity in test inputs, and more sensitive than prior work to test inputs generated using different DNN test generation methods. The findings demonstrate that IDC overcomes several limitations of white-box DNN coverage approaches by discounting coverage from unrealistic inputs and enabling the calculation of test adequacy metrics that capture the feature diversity present in the input space of DNNs.

Information Entropy Measures for Evaluation of Reliability of Deep Neural Network Results.

Deep neural networks (DNN) try to analyze given data, to come up with decisions regarding the inputs. The decision-making process of the DNN model is not entirely transparent. The confidence of the model predictions on new data fed into the network can vary. We address the question of certainty of decision making and adequacy of information capturing by DNN models during this process of decision-making. We introduce a measure called certainty index, which is based on the outputs in the most penultimate layer of DNN. In this approach, we employed iEEG (intracranial electroencephalogram) data to train and test DNN. When arriving at model predictions, the contribution of the entire information content of the input may be important. We explored the relationship between the certainty of DNN predictions and information content of the signal by estimating the sample entropy and using a heatmap of the signal. While it can be assumed that the entire sample must be utilized for arriving at the most appropriate decisions, an evaluation of DNNs from this standpoint has not been reported. We demonstrate that the robustness of the relationship between certainty index with the sample entropy, demonstrated through sample entropy-heatmap correlation, is higher than that with the original signal, indicating that the DNN focuses on information rich regions of the signal to arrive at decisions. Therefore, it can be concluded that the certainty of a decision is related to the DNN's ability to capture the information in the original signal. Our results indicate that, within its limitations, the certainty index can be used as useful tool in estimating the confidence of predictions. The certainty index appears to be related to how effectively DNN heatmaps captured the information content in the signal.

Test case prioritization with neuron valuation based pattern

• Filtering error-triggering data in testing DNNs is cost-effective. • The set of test cases may contain various types of data. • Patterns for DNN-based classifiers can help in prioritizing the abnormal inputs. • Our pattern consists of neurons and the computed thresholds for pattern matching. • Evaluation results show our advantage in prioritizing error-triggering samples. Deep neural networks (DNNs) have been increasingly adopted in various applications. Systematic verification and validation are essential techniques to guarantee the quality of such systems. Testing is one of the feasible solutions for system validation. However, testing DNNs usually requires a large number of test cases and leads to the high labeling cost. Meanwhile, the test cases may contain various noisy data, which also increases the burden of testing. In this paper, we put forwards a test case prioritization method for DNN-based classifiers, which assigns high priorities to those cases that can lead to problematic classifications. The test cases are prioritized w.r.t. the pattern extracted from the neuron outputs. The difference between the pattern acquired from the training set and the neurons matched from a given input determines the priority of the input. For a trained model, the method consists of two steps. First, we collect neuron output values over the training set and construct a neuron-based pattern for every class of training samples. Second, the metrics are computed according to the comparison between neuron output values of the DNN model from an input and the selected pattern. We carry out the experimentation over three popular datasets with various neural network structures. The extensive experimental results demonstrate that the efficiency and the generalizability of the prioritization method outperform most of the existing techniques.

심층신경망 기반의 무릎 생체역학 변인 추정기

Knee contact forces and knee stiffness are biomechanical factors worth considering for walking in knee osteoarthritis patients. However, it is challenging to acquire these factors in real time; thus, making it difficult to use them in robotic rehabilitation and assistive systems. This study investigated whether trained deep neural networks (DNNs) can capture the biomechanical factors only using kinematics during gait, which is possible to measure via sensors in real time. A public dataset of walking on the ground was analyzed through biomechanical analysis to train and test DNNs. Using the training dataset, several DNN topologies were explored via Bayesian optimization to tune the hyperparameters. After optimization, DNNs were trained to estimate the biomechanical factors in a supervised manner. The trained DNNs were then evaluated using two new datasets, which were not used in the training process. The trained DNNs estimated the biomechanical factors with a high level of accuracy in both types of test datasets. Results confirmed that DNNs can estimate the biomechanical factors based on only kinematics during gait.