Build Defect Prediction Models Research Articles

Towards an understanding of intra-defect associations: Implications for defect prediction

In previous studies, when collecting defect data, if the fix of a defect spans multiple modules, each involved module is labeled as defective. In this context, the defect prediction models are built based on the features of each individual module, ignoring the potential associations between the modules involved in the same defect(referred to as “intra-defect associations”). Considering the possibility of numerous cross-module defects in practice, we hypothesize that these intra-defect associations could play a crucial role in enhancing defect prediction performance. Unfortunately, there is no empirical evidence to know that. To this end, we are motivated to conduct a comprehensive study to explore the implications of intra-defect associations for defect prediction. We first examine the proportion of cross-module defects and the relationships between the involved modules. The results reveal that, at function level, the majority of defects occur across functions, with most of the cross-module defects exhibiting implicit dependencies. Inspired by these findings, we propose a novel data processing approach for building defect prediction models. This approach leverages the intra-defect associations by merging the involved modules into new instances with mean or median variables to augment the training data. The experimental results indicate that considering intra-defect associations can significantly improve the defect prediction performance in both the ranking and classification scenarios. This study provides valuable insights into the implications of intra-defect associations for defect prediction.

Tsbagging: A Novel Cross-Project Software Defect Prediction Algorithm Based on Semisupervised Clustering

Software defect prediction (SDP) is an important technology which is widely applied to improve software quality and reduce development costs. It is difficult to train the SDP model when software to be test only has limited historical data. Cross-project defect prediction (CPDP) has been proposed to solve this problem by using source project data to train the defect prediction model. Most of CPDP methods build defect prediction models based on the similarity of feature space or data distance between different projects. However, when the target project has a small amount of label data, these methods usually do not consider this part of data information. Therefore, when the distribution between source project and target project is quite different, these methods are difficult to achieve good prediction performance. To solve this problem, this paper proposes a CPDP method based on a semisupervised clustering (namely, Tsbagging). Tsbagging has two stages; in the first stage, we cluster to the source project data based on the limited labeled data in the target project and assign different weights to these source project data according to the clustering results. In the second stage, we use bagging method to train the prediction model based on the weight assigned in the first stage. The experimental results show that the performance achieved by Tsbagging is better than other existing SDP methods.

WIFLF: An approach independent of the target project for cross‐project defect prediction

AbstractCross‐project defect prediction (CPDP) is used to build defect prediction models when data from the target project are not enough. There has been several approaches to improve the performance of CPDP, such as feature transformation and instance selection methods. However, existing techniques are strongly dependent on the target data to reduce the distribution discrepancy between source and target projects. That is, the performance of these methods is determined by the effectiveness of feature transformation or the similarity between two projects. Additionally, when there is a large amount of source data that needs to be matched with target data, it will take much time and reduce the efficiency of model construction. Therefore, it is vital to explore a target project‐agnostic approach to build CPDP models. This paper presents a Weighted Isolation Forest with class Label information Filter (WIFLF) to relieve the issues above. Four groups of datasets from AEEEM, Relink and PROMISE Data Repository are used to conduct CPDP models. Besides, WIFLF is compared with 12 approaches. The experimental results indicate that WIFLF significantly outperforms all the baselines. Specifically, WIFLF with random forest significantly improves the performance over the baselines on average by at least 14.64% and 4.90% with respect to Skewed F‐Measure and G‐Measure, respectively.

Predictive maintenance of infrastructure code using “fluid” datasets: An exploratory study on Ansible defect proneness

AbstractThis work consolidates and compounds previous investigations in recognizing defects for infrastructure‐as‐code (IaC) scripts using general software development quality metrics with a focus on defect severity but adding to previous work an explorative look at creating datasets, which may boost the predictive power of provided models—we call this notion a fluid dataset. More specifically, we experiment with 50 different metrics harnessing a multiple dataset creation process whereby different versions of the same datasets are rigged with auto‐training facilities for model retraining and redeployment in a DataOps fashion. At this point, with a focus on the Ansible infrastructure code language—as a de facto standard for industrial‐strength infrastructure code—we build defect prediction models and manage to improve on the state of the art by finding an F1 score of 0.52 and a recall of 0.57 using a Naive–Bayes classifier. On the one hand, by improving state‐of‐the‐art defect prediction models using metrics generalizable for different IaC languages, we provide interesting leads for the future of infrastructure‐as‐code. On the other hand, we have barely scratched the surface on the novel approach of fluid‐datasets creation and automated retraining of Machine Learning (ML) defect prediction models, warranting for more research on the same direction in the future.

Software Defect Prediction Based on Gated Hierarchical LSTMs

Software defect prediction, aimed at assisting software practitioners in allocating test resources more efficiently, predicts the potential defective modules in software products. With the development of defect prediction technology, the inability of traditional software features to capture semantic information is exposed, hence related researchers have turned to semantic features to build defect prediction models. However, sometimes traditional features such as lines of code (LOC) also play an important role in defect prediction. Most of the existing researches only focus on using a single type of feature as the input of the model. In this article, a defect prediction method based on gated hierarchical long short-term memory networks (GH-LSTMs) is proposed, which uses hierarchical LSTM networks to extract both semantic features from word embeddings of abstract syntax trees (ASTs) of source code files, and traditional features provided by the PROMISE repository. More importantly, we adopt a gated fusion strategy to combine the outputs of the hierarchical networks properly. Experimental results show that GH-LSTMs outperforms existing methods under both noneffort-aware and effort-aware scenarios.

Predicting Software Defects for Object-Oriented Software Using Search-based Techniques

Development without any defect is unsubstantial. Timely detection of software defects favors the proper resource utilization saving time, effort and money. With the increasing size and complexity of software, demand for accurate and efficient prediction models is increasing. Recently, search-based techniques (SBTs) have fascinated many researchers for Software Defect Prediction (SDP). The goal of this study is to conduct an empirical evaluation to assess the applicability of SBTs for predicting software defects in object-oriented (OO) softwares. In this study, 16 SBTs are exploited to build defect prediction models for 13 OO software projects. Stable performance measures — GMean, Balance and Receiver Operating Characteristic-Area Under Curve (ROC-AUC) are employed to probe into the predictive capability of developed models, taking into consideration the imbalanced nature of software datasets. Proper measures are taken to handle the stochastic behavior of SBTs. The significance of results is statistically validated using the Friedman test complied with Wilcoxon post hoc analysis. The results confirm that software defects can be detected in the early phases of software development with help of SBTs. This paper identifies the effective subset of SBTs that will aid software practitioners to timely detect the probable software defects, therefore, saving resources and bringing up good quality softwares. Eight SBTs — sUpervised Classification System (UCS), Bioinformatics-oriented hierarchical evolutionary learning (BIOHEL), CHC, Genetic Algorithm-based Classifier System with Adaptive Discretization Intervals (GA_ADI), Genetic Algorithm-based Classifier System with Intervalar Rule (GA_INT), Memetic Pittsburgh Learning Classifier System (MPLCS), Population-Based Incremental Learning (PBIL) and Steady-State Genetic Algorithm for Instance Selection (SGA) are found to be statistically good defect predictors.

How Well Do Change Sequences Predict Defects? Sequence Learning from Software Changes

Software defect prediction, which aims to identify defective modules, can assist developers in finding bugs and prioritizing limited quality assurance resources. Various features to build defect prediction models have been proposed and evaluated. Among them, process metrics are one important category. Yet, existing process metrics are mainly encoded manually from change histories and ignore the sequential information arising from the changes during software evolution. Are the change sequences derived from such information useful to characterize buggy program modules? How can we leverage such sequences to build good defect prediction models? Unlike traditional process metrics used for existing defect prediction models, change sequences are mostly vectors of variable length. This makes it difficult to apply such sequences directly in prediction models that are driven by conventional classifiers. To resolve this challenge, we utilize Recurrent Neural Network (RNN), which is a deep learning technique, to encode features from sequence data automatically. In this paper, we propose a novel approach called Fences , which extracts six types of change sequences covering different aspects of software changes via fine-grained change analysis. It approaches defects prediction by mapping it to a sequence labeling problem solvable by RNN. Our evaluations on 10 open source projects show that Fences can predict defects with high performance. In particular, our approach achieves an average F-measure of 0.657, which improves the prediction models built on traditional metrics significantly. The improvements vary from 31.6 to 46.8 percent on average. In terms of AUC, Fences achieves an average value of 0.892, and the improvements over baselines vary from 4.2 to 16.1 percent. Fences also outperforms the state-of-the-art technique which learns semantic features automatically from static code via deep learning.

Automatically identifying code features for software defect prediction: Using AST N-grams

Context: Identifying defects in code early is important. A wide range of static code metrics have been evaluated as potential defect indicators. Most of these metrics offer only high level insights and focus on particular pre-selected features of the code. None of the currently used metrics clearly performs best in defect prediction.Objective: We use Abstract Syntax Tree (AST) n-grams to identify features of defective Java code that improve defect prediction performance.Method: Our approach is bottom-up and does not rely on pre-selecting any specific features of code. We use non-parametric testing to determine relationships between AST n-grams and faults in both open source and commercial systems. We build defect prediction models using three machine learning techniques.Results: We show that AST n-grams are very significantly related to faults in some systems, with very large effect sizes. The occurrence of some frequently occurring AST n-grams in a method can mean that the method is up to three times more likely to contain a fault. AST n-grams can have a large effect on the performance of defect prediction models.Conclusions: We suggest that AST n-grams offer developers a promising approach to identifying potentially defective code.

Predicting defective modules in different test phases

Defect prediction is a well-established research area in software engineering . Prediction models in the literature do not predict defect-prone modules in different test phases. We investigate the relationships between defects and test phases in order to build defect prediction models for different test phases. We mined the version history of a large-scale enterprise software product to extract churn and static code metrics. We used three testing phases that have been employed by our industry partner, namely function, system and field, to build a learning-based model for each testing phase. We examined the relation of different defect symptoms with the testing phases. We compared the performance of our proposed model with a benchmark model that has been constructed for the entire test phase (benchmark model). Our results show that building a model to predict defect-prone modules for each test phase significantly improves defect prediction performance and shortens defect detection time. The benefit analysis shows that using the proposed model, the defects are detected on the average 7 months earlier than the actual. The outcome of prediction models should lead to an action in a software development organization. Our proposed model gives a more granular outcome in terms of predicting defect-prone modules in each testing phase so that managers may better organize the testing teams and effort.

A STUDY OF SOFTWARE METRIC SELECTION TECHNIQUES: STABILITY ANALYSIS AND DEFECT PREDICTION MODEL PERFORMANCE

Software metrics (features or attributes) are collected during the software development cycle. Metric selection is one of the most important preprocessing steps in the process of building defect prediction models and may improve the final prediction result. However, the addition or removal of program modules (instances or samples) can alter the subsets chosen by a feature selection technique, rendering the previously-selected feature sets invalid. Very limited research have been done considering both stability (or robustness) and defect prediction model performance together in the software engineering domain, despite the importance of both aspects when choosing a feature selection technique. In this paper, we test the stability and classification model performance of eighteen feature selection techniques as the magnitude of change to the datasets and the size of the selected feature subsets are varied. All experiments were conducted on sixteen datasets from three real-world software projects. The experimental results demonstrate that Gain Ratio shows the least stability while two different versions of ReliefF show the most stability, followed by the PRC- and AUC-based threshold-based feature selection techniques. Results also show that the signal-to-noise ranker performed moderately in terms of robustness and was the best ranker in terms of model performance. Finally, we conclude that while for some rankers, stability and classification performance are correlated, this is not true for other rankers, and therefore performance according to one scheme (stability or model performance) cannot be used to predict performance according to the other.

Empirical evaluation of the effects of mixed project data on learning defect predictors

ContextDefect prediction research mostly focus on optimizing the performance of models that are constructed for isolated projects (i.e. within project (WP)) through retrospective analyses. On the other hand, recent studies try to utilize data across projects (i.e. cross project (CP)) for building defect prediction models for new projects. There are no cases where the combination of within and cross (i.e. mixed) project data are used together. ObjectiveOur goal is to investigate the merits of using mixed project data for binary defect prediction. Specifically, we want to check whether it is feasible, in terms of defect detection performance, to use data from other projects for the cases (i) when there is an existing within project history and (ii) when there are limited within project data. MethodWe use data from 73 versions of 41 projects that are publicly available. We simulate the two above-mentioned cases, and compare the performances of naive Bayes classifiers by using within project data vs. mixed project data. ResultsFor the first case, we find that the performance of mixed project predictors significantly improves over full within project predictors (p-value<0.001), however the effect size is small (Hedges′g=0.25). For the second case, we found that mixed project predictors are comparable to full within project predictors, using only 10% of available within project data (p-value=0.002, g=0.17). ConclusionWe conclude that the extra effort associated with collecting data from other projects is not feasible in terms of practical performance improvement when there is already an established within project defect predictor using full project history. However, when there is limited project history, e.g. early phases of development, mixed project predictions are justifiable as they perform as good as full within project models.

Practical considerations in deploying statistical methods for defect prediction: A case study within the Turkish telecommunications industry

Context Building defect prediction models in large organizations has many challenges due to limited resources and tight schedules in the software development lifecycle. It is not easy to collect data, utilize any type of algorithm and build a permanent model at once. We have conducted a study in a large telecommunications company in Turkey to employ a software measurement program and to predict pre-release defects. Based on our prior publication, we have shared our experience in terms of the project steps (i.e. challenges and opportunities). We have further introduced new techniques that improve our earlier results. Objective In our previous work, we have built similar predictors using data representative for US software development. Our task here was to check if those predictors were specific solely to US organizations or to a broader class of software. Method We have presented our approach and results in the form of an experience report. Specifically, we have made use of different techniques for improving the information content of the software data and the performance of a Naïve Bayes classifier in the prediction model that is locally tuned for the company. We have increased the information content of the software data by using module dependency data and improved the performance by adjusting the hyper-parameter (decision threshold) of the Naïve Bayes classifier. We have reported and discussed our results in terms of defect detection rates and false alarms. We also carried out a cost–benefit analysis to show that our approach can be efficiently put into practice. Results Our general result is that general defect predictors, which exist across a wide range of software (in both US and Turkish organizations), are present. Our specific results indicate that concerning the organization subject to this study, the use of version history information along with code metrics decreased false alarms by 22%, the use of dependencies between modules further reduced false alarms by 8%, and the decision threshold optimization for the Naïve Bayes classifier using code metrics and version history information further improved false alarms by 30% in comparison to a prediction using only code metrics and a default decision threshold. Conclusion Implementing statistical techniques and machine learning on a real life scenario is a difficult yet possible task. Using simple statistical and algorithmic techniques produces an average detection rate of 88%. Although using dependency data improves our results, it is difficult to collect and analyze such data in general. Therefore, we would recommend optimizing the hyper-parameter of the proposed technique, Naïve Bayes, to calibrate the defect prediction model rather than employing more complex classifiers. We also recommend that researchers who explore statistical and algorithmic methods for defect prediction should spend less time on their algorithms and more time on studying the pragmatic considerations of large organizations.

Building Defect Prediction Models in Practice

Defective software modules cause software failures, increase development and maintenance costs, and decrease customer satisfaction. Effective defect prediction models can help developers focus quality assurance activities on defect-prone modules and thus improve software quality by using resources more efficiently. These models often use static measures obtained from source code, mainly size, coupling, cohesion, inheritance, and complexity measures, which have been associated with risk factors, such as defects and changes.