Deep Learning Test Optimization Method Using Multi-objective Optimization

Abstract

PDF HTML XML Export Cite reminder Deep Learning Test Optimization Method Using Multi-objective Optimization DOI: 10.21655/ijsi.1673-7288.00282 Author: Affiliation: Clc Number: Fund Project: Article | Figures | Metrics | Reference | Related | Cited by | Materials | Comments Abstract:With the rapid development of deep learning technology, research on its quality assurance is raising more attention. Meanwhile, it is no longer difficult to collect test data owing to the mature sensor technology, but it costs a lot to label the collected data. To reduce the cost of labeling, the existing studies attempt to select a test subset from the original test set. The test subset, however, only ensures that the overall accuracy (the accuracy of the target deep learning model on all test inputs of the test set) of the test subset is similar to that of the original test set; it cannot maintain other test properties similar to those of the original test set. For example, it cannot fully cover all kinds of test input in the original test set. This study proposes a method based on multi-objective optimization called Deep Multi-Objective Selection (DMOS). It firstly analyzes the data distribution of the original test set by Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN). Then, it designs multiple optimization objectives given the characteristics of the clustering results and then carries out multi-objective optimization to find out the appropriate selection solution. Massive experiments are carried out on eight pairs of classic deep learning test sets and models. The results reveal that the best test subset selected by the DMOS method (the test subset corresponding to the Pareto optimal solution with the best performance) can not only cover more test input categories in the original test set but also estimate the accuracy of each test input category extremely close to that of the original test set. Meanwhile, it can also ensure that the overall accuracy and test adequacy are close to those of the original test set: the average error of the overall accuracy estimation is only 1.081%, which is 0.845% lower than that of Practical ACcuracy Estimation (PACE), an improvement of 43.87%. The average error of the accuracy estimation of each test input category is only 5.547%, which is 2.926% less than that of PACE, an improvement of 34.53%. The average estimation error of the five test adequacy measures is only 8.739%, which is 7.328% lower than that of PACE, an improvement of 45.61%. Reference Related Cited by