Performance Of Defect Prediction Models Research Articles

A Method-Level Defect Prediction Approach Based on Structural Features of Method-Calling Network

Software defect prediction models help testers find program modules that have a high probability of having defects. A method-calling network can express the dependencies between methods in a program. Existing approaches do not sufficiently utilize method-calling network to characterize the structural features between methods. To address this problem, in this study, it is proposed for the first time that the characteristics of methods in a program are obtained by analyzing the method-calling network, and a new approach is proposed for defect prediction at the method-level. Specifically in this study, the method-calling network of the program was first constructed and the network metrics of the method-calling network were obtained. Next, the new network embedding technique (node2vec) was used to automatically encode the method-calling network structure into a low-dimensional vector to obtain the network embedding metrics. Finally, they were combined with code metrics to construct defect prediction models. We evaluated our approaches on 13 open-source software systems. The experimental results show that the proposed method improved the values of area under the receiver operating characteristic curve by 2.5% to 6.7% and Matthews correlation coefficient by 13% to 178.4% compared to the baselines. Therefore the method-calling network contains rich structural features between methods, and the structural features extend the features used for method-level defect prediction and further improve the performance of defect prediction models.

Exploring Design smells for smell-based defect prediction

Defect prediction is commonly used to reduce the effort from the testing phase of software development. A promising strategy is to use machine learning techniques to predict which software components may be defective. Features are key factors to the prediction’s success, and thus extracting significant features can improve the model’s accuracy. In particular, code smells are a category of those features that have been shown to improve the prediction performance significantly. However, Design code smells, a state-of-the-art collection of code smells based on the violations of the object-oriented programming principles, have not been studied in the context of defect prediction. In this paper, we study the performance of defect prediction models by training multiple classifiers for 97 real projects. We compare using Design code smells as features and using other Traditional smells from the literature and both. Moreover, we cluster and analyze the models’ performance based on the categories of Design code smells. We conclude that the models trained with both the Design code smells and the smells from the literature performed the best, with an improvement of 4.1% for the AUC score, compared to models trained with only Traditional smells. Consequently, Design smells are a good addition to the smells commonly studied in the literature for defect prediction.

The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models

Defect prediction models that are trained on class imbalanced datasets (i.e., the proportion of defective and clean modules is not equally represented) are highly susceptible to produce inaccurate prediction models. Prior research compares the impact of class rebalancing techniques on the performance of defect prediction models. Prior research efforts arrive at contradictory conclusions due to the use of different choice of datasets, classification techniques, and performance measures. Such contradictory conclusions make it hard to derive practical guidelines for whether class rebalancing techniques should be applied in the context of defect prediction models. In this paper, we investigate the impact of 4 popularly-used class rebalancing techniques on 10 commonly-used performance measures and the interpretation of defect prediction models. We also construct statistical models to better understand in which experimental design settings that class rebalancing techniques are beneficial for defect prediction models. Through a case study of 101 datasets that span across proprietary and open-source systems, we recommend that class rebalancing techniques are necessary when quality assurance teams wish to increase the completeness of identifying software defects (i.e., Recall). However, class rebalancing techniques should be avoided when interpreting defect prediction models. We also find that class rebalancing techniques do not impact the AUC measure. Hence, AUC should be used as a standard measure when comparing defect prediction models.

Improving defect prediction with deep forest

Abstract Context Software defect prediction is important to ensure the quality of software. Nowadays, many supervised learning techniques have been applied to identify defective instances (e.g., methods, classes, and modules). Objective However, the performance of these supervised learning techniques are still far from satisfactory, and it will be important to design more advanced techniques to improve the performance of defect prediction models. Method We propose a new deep forest model to build the defect prediction model (DPDF). This model can identify more important defect features by using a new cascade strategy, which transforms random forest classifiers into a layer-by-layer structure. This design takes full advantage of ensemble learning and deep learning. Results We evaluate our approach on 25 open source projects from four public datasets (i.e., NASA, PROMISE, AEEEM and Relink). Experimental results show that our approach increases AUC value by 5% compared with the best traditional machine learning algorithms. Conclusion The deep strategy in DPDF is effective for software defect prediction.

Estimating the threshold of software metrics for web applications

Estimating thresholds for software metrics is a key step towards assigning a quality index. In defect prediction, two approaches are widely used those based on statistics and, that which uses rigorous mathematical models. Although significant insights have been surmised, a general consensus on their results is still far from generalizations. In these perspectives, we attempt to check whether there exists any relationship between the two approaches. An empirical investigation is carried out in this work to study the relationship between estimated threshold values calculated at various risk levels using Bender’s approach and measures of central tendency using the Apache Click web application. The effect of these different threshold estimates on the performance of the developed defect prediction models is also studied and validated using different releases of the dataset. We find that the threshold indicator obtained from the representational models such as that due to Bender has an intricate relationship with the median value of the dataset. The close association between the model and statistical parameters mainly stems from the underlying characteristics of the data set itself. Descriptive statistical analysis of all Apache Click metrics dataset is found to be positively skewed, and hence median render the most relevant central measure for threshold estimation. Additionally, we also find that with increasing risk level, the threshold value subsequently shifts from median to mean value of the underlying metric data. Our preposition that the performance of the defect prediction model is best when threshold estimates are closer to the median is also verified with inter-version project comparison.

The Use of Summation to Aggregate Software Metrics Hinders the Performance of Defect Prediction Models

Defect prediction models help software organizations to anticipate where defects will appear in the future. When training a defect prediction model, historical defect data is often mined from a Version Control System (VCS, e.g., Subversion), which records software changes at the file-level. Software metrics, on the other hand, are often calculated at the class- or method-level (e.g., McCabe's Cyclomatic Complexity). To address the disagreement in granularity, the class- and method-level software metrics are aggregated to file-level, often using summation (i.e., McCabe of a file is the sum of the McCabe of all methods within the file). A recent study shows that summation significantly inflates the correlation between lines of code (Sloc) and cyclomatic complexity (Cc) in Java projects. While there are many other aggregation schemes (e.g., central tendency, dispersion), they have remained unexplored in the scope of defect prediction. In this study, we set out to investigate how different aggregation schemes impact defect prediction models. Through an analysis of 11 aggregation schemes using data collected from 255 open source projects, we find that: (1) aggregation schemes can significantly alter correlations among metrics, as well as the correlations between metrics and the defect count; (2) when constructing models to predict defect proneness, applying only the summation scheme (i.e., the most commonly used aggregation scheme in the literature) only achieves the best performance (the best among the 12 studied configurations) in 11 percent of the studied projects, while applying all of the studied aggregation schemes achieves the best performance in 40 percent of the studied projects; (3) when constructing models to predict defect rank or count, either applying only the summation or applying all of the studied aggregation schemes achieves similar performance, with both achieving the closest to the best performance more often than the other studied aggregation schemes; and (4) when constructing models for effort-aware defect prediction, the mean or median aggregation schemes yield performance values that are significantly closer to the best performance than any of the other studied aggregation schemes. Broadly speaking, the performance of defect prediction models are often underestimated due to our community's tendency to only use the summation aggregation scheme. Given the potential benefit of applying additional aggregation schemes, we advise that future defect prediction models should explore a variety of aggregation schemes.

An empirical framework for defect prediction using machine learning techniques with Android software

ContextSoftware defect prediction is important for identification of defect-prone parts of a software. Defect prediction models can be developed using software metrics in combination with defect data for predicting defective classes. Various studies have been conducted to find the relationship between software metrics and defect proneness, but there are few studies that statistically determine the effectiveness of the results. ObjectiveThe main objectives of the study are (i) comparison of the machine-learning techniques using data sets obtained from popular open source software (ii) use of appropriate performance measures for measuring the performance of defect prediction models (iii) use of statistical tests for effective comparison of machine-learning techniques and (iv) validation of models over different releases of data sets. MethodIn this study we use object-oriented metrics for predicting defective classes using 18 machine-learning techniques. The proposed framework has been applied to seven application packages of well known, widely used Android operating system viz. Contact, MMS, Bluetooth, Email, Calendar, Gallery2 and Telephony. The results are validated using 10-fold and inter-release validation methods. The reliability and significance of the results are evaluated using statistical test and post-hoc analysis. ResultsThe results show that the area under the curve measure for Naïve Bayes, LogitBoost and Multilayer Perceptron is above 0.7 in most of the cases. The results also depict that the difference between the ML techniques is statistically significant. However, it is also proved that the Support Vector Machines based techniques such as Support Vector Machines and voted perceptron do not possess the predictive capability for predicting defects. ConclusionThe results confirm the predictive capability of various ML techniques for developing defect prediction models. The results also confirm the superiority of one ML technique over the other ML techniques. Thus, the software engineers can use the results obtained from this study in the early phases of the software development for identifying defect-prone classes of given software.