Defect Prediction Models Research Articles

An effective software cross-project fault prediction model for quality improvement

An effective software cross-project fault prediction model for quality improvement

The impact of feature selection techniques on effort‐aware defect prediction: An empirical study

AbstractEffort‐Aware Defect Prediction (EADP) methods sort software modules based on the defect density and guide the testing team to inspect the modules with high defect density first. Previous studies indicated that some feature selection methods could improve the performance of Classification‐Based Defect Prediction (CBDP) models, and the Correlation‐based feature subset selection method with the Best First strategy (CorBF) performed the best. However, the practical benefits of feature selection methods on EADP performance are still unknown, and blindly employing the best‐performing CorBF method in CBDP to pre‐process the defect datasets may not improve the performance of EADP models but possibly result in performance degradation. To assess the impact of the feature selection techniques on EADP, a total of 24 feature selection methods with 10 classifiers embedded in a state‐of‐the‐art EADP model (CBS+) on the 41 PROMISE defect datasets were examined. We employ six evaluation metrics to assess the performance of EADP models comprehensively. The results show that (1) The impact of the feature selection methods varies in classifiers and datasets. (2) The four wrapper‐based feature subset selection methods with forwards search, that is, AdaBoost with Forwards Search, Deep Forest with Forwards Search, Random Forest with Forwards Search, and XGBoost with Forwards Search (XGBF) are better than other methods across the studied classifiers and the used datasets. And XGBF with XGBoost as the embedded classifier in CBS+ performs the best on the datasets. (3) The best‐performing CorBF method in CBDP does not perform well on the EADP task. (4) The selected features vary with different feature selection methods and different datasets, and the features noc (number of children), ic (inheritance coupling), cbo (coupling between object classes), and cbm (coupling between methods) are frequently selected by the four wrapper‐based feature subset selection methods with forwards search. (5) Using AdaBoost, deep forest, random forest, and XGBoost as the base classifiers embedded in CBS+ can achieve the best performance. In summary, we recommend the software testing team should employ XGBF with XGBoost as the embedded classifier in CBS+ to enhance the EADP performance.

A Comparative Study of Data Collection Periods for Just-In-Time Defect Prediction Using the Automatic Machine Learning Method

This paper focuses on the “data collection period” for training a better Just-In-Time (JIT) defect prediction model — the early commit data vs. the recent one —, and conducts a large-scale comparative study to explore an appropriate data collection period. Since there are many possible machine learning algorithms for training defect prediction models, the selection of machine learning algorithms can become a threat to validity. Hence, this study adopts the automatic machine learning method to mitigate the selection bias in the comparative study. The empirical results using 122 open-source software projects prove the trend that the dataset composed of the recent commits would become a better training set for JIT defect prediction models.

Inconsistent Defect Labels: Essence, Causes, and Influence

The label quality of defect data sets has a direct influence on the reliability of defect prediction models. In this paper, we conduct a systematic study of inconsistent defect labels in multi-version-project defect data sets, i.e., many instances having the same source code but different labels over multiple versions of a software project. First, we report the phenomena of inconsistent labels by real examples and analyze their essence in the context of defect prediction. Then, we uncover the causes that lead to the occurrence of inconsistent labels for the representative label collection approaches. Finally, we investigate the actual influence of inconsistent labels on defect prediction models. We find that inconsistent labels in general exist in six multi-version-project defect data sets (either widely used or the most up-to-date in the literature) collected by diverse label collection approaches. In particular, inconsistent labels in a training data set significantly reduce the prediction performance of a model, while inconsistent labels in a test data set can lead to a considerable evaluation bias on the real performance. Therefore, we recommend that: on the one hand, researchers leverage our findings to make targeted methodological improvements on existing defect label collection approaches to reduce the generation of inconsistent labels; on the other hand, practitioners detect and exclude inconsistent labels in defect data sets to avoid their potential negative influence on defect prediction.

A Procedure to Continuously Evaluate Predictive Performance of Just-In-Time Software Defect Prediction Models During Software Development

Just-In-Time Software Defect Prediction (JIT-SDP) uses machine learning to predict whether software changes are defect-inducing or clean. When adopting JIT-SDP, changes in the underlying defect generating process may significantly affect the predictive performance of JIT-SDP models over time. Therefore, being able to continuously track the predictive performance of JIT-SDP models during the software development process is of utmost importance for software companies to decide whether or not to trust the predictions provided by such models over time. However, there has been little discussion on how to continuously evaluate predictive performance in practice, and such evaluation is not straightforward. In particular, labeled software changes that can be used for evaluation arrive over time with a delay, which in part corresponds to the time we have to wait to label software changes as ‘clean’ (waiting time). A clean label assigned based on a given waiting time may not correspond to the true label of the software changes. This can potentially hinder the validity of any continuous predictive performance evaluation procedure for JIT-SDP models. This paper provides the first discussion of how to continuously evaluate predictive performance of JIT-SDP models over time during the software development process, and the first investigation of whether and to what extent waiting time affects the validity of such continuous performance evaluation procedure in JIT-SDP. Based on 13 GitHub projects, we found that waiting time had a significant impact on the validity. Though typically small, the differences in estimated predicted performance were sometimes large, and thus inappropriate choices of waiting time can lead to misleading estimations of predictive performance over time. Such impact did not normally change the ranking between JIT-SDP models, and thus conclusions in terms of which JIT-SDP model performs better are likely reliable independent of the choice of waiting time, especially when considered across projects.

Candidate project selection in cross project defect prediction using hybrid method

Cross Project Defect Prediction (CPDP) is a process that develops a defect prediction model on source projects and then applies the same model to the target project. Day by day, new software projects are being developed, so selecting an appropriate training project from existing projects and from new upcoming projects, to train a predictor model is a challenging task in CPDP. In the present study, we have proposed a hybrid selection method to select a candidate project from existing projects and a probabilistic method to select a candidate project from new projects. The proposed hybrid method is a weighted combination of the Collaborative filtering (CF) method and the Content Based (CB) method. The probabilistic method is based on a Naïve Bayes classifier and is used to predict the relation between the target project and the new target project. In the CF method, a usability score is generated for each project by making use of classification techniques, and the CB method calculates the matching score of candidate projects by using the K-dimensional tree. Finally, both the methods are combined by parallelized hybridization design, and weights for the proposed method are estimated with an empirical bootstrapping method. The score generated by the proposed hybrid technique is then used to identify the most suitable candidate project for the new project. The experimental results show that the suggestion list of the best three candidate projects is consistent when employing different classifiers. The recommendation performance is evaluated in terms of F-score and Mean Average Precision (MAP), and the proposed method has shown improved performance as compared to the existing methods in both terms.

Fault Prediction of On-Board Train Control Equipment Using a CGAN-Enhanced XGBoost Method with Unbalanced Samples

On-board train control equipment is an important component of the Train Control System (TCS) of railway trains. In order to guarantee the safe and efficient operation of the railway system, Predictive Maintenance (PdM) is significantly required. The operation data of the on-board equipment allow us to build fault prediction models using a data-driven approach. However, the problem of unbalanced fault samples makes it difficult to achieve the expected modeling performance. In this paper, a Conditional Generative Adversarial Network (CGAN) is adopted to solve the unbalancing problem by generating synthetic samples corresponding to specific fault labels that belong to the minority classes. With this basis, a CGAN-enhanced eXtreme Gradient Boosting (XGBoost) solution is presented for training the fault prediction models. From the pre-processing to the field data, artificial fault samples are generated and integrated into the training sample sets, and the XGBoost models can be derived with multiple decision trees. Both the feature importance sequence list and the knowledge graph are derived to describe the characteristics obtained by the models. Filed data sets from practical operation are utilized to validate the proposed solution. By comparison with conventional machine learning algorithms, it can be found that higher accuracy, precision, recall, and F1 scores, which are up to 99.76%, can be achieved by the proposed solution. By involving the CGAN strategy, the maximum enhancement to the F1 score with the XGBoost approach reaches 6.13%. The advantages of the proposed solution show great potential in implementing equipment health management and intelligent condition-based maintenance.

An effective feature selection based cross-project defect prediction model for software quality improvement

An effective feature selection based cross-project defect prediction model for software quality improvement

Application of state parameter learning for fault diagnosis on the large reciprocating compressor

According to the statistical results of the reciprocating compressor maintenance in chemical enterprises, the probability of faults caused by wear or damage of vulnerable parts inside the cylinder is close to 80%. Now, fault diagnosis of vulnerable parts inside the cylinder is relied on vibration signal, acoustic emission signal, or thermal parameters. However, it is difficult to extract eigenvalues from vibration signals and acoustic emission signals, which can be disturbed by noise easily. Thermal parameters are relatively stable and less affected by noise. The measurement of thermal parameters requires drilling testing holes through the cylinder wall, which will decrease the strength of the cylinder. According to the working principle of the compressor, fault of the vulnerable parts inside the cylinder would change the pressure and temperature distribution at the suction and the discharge port of the cylinder, while these state parameters are monitored by the parameter monitoring system on most of the compressors. For these reasons, a fault prediction model based on state parameters learning is proposed in this paper aiming to fault diagnosis of vulnerable parts inside the cylinder, which could make fault prediction without adding new hardware cost, such as sensors. Optimized back propagation neural network method by genetic algorithm (GA-BP) is applied to establish the fault prediction model to describe normal working process of the compressor. When fault of vulnerable parts inside the cylinder occurs, monitored pressure and temperature would deviate from the predicted value by the fault prediction. The degree of the deviation is adopted to the fault diagnosis. Then, experiments simulating the fault condition were carried out, and it shows that the accuracy of fault diagnosis using this method could exceed 95%. Verification test shows that the proposed method could successfully predict the fault before the unplanned shutdown.

Addressing Noise and Class Imbalance Problems in Heterogeneous Cross-Project Defect Prediction

When a software project either lacks adequate historical data to build a defect prediction (DP) model or is in the initial phases of development, the DP model based on related source project's defect data might be used. This kind of SDP is categorized as heterogeneous cross-project defect prediction (HCPDP). According to a comprehensive literature review, no research has been done in the field of CPDP to deal with noise and class imbalance problem (CIP) at the same time. In this paper, the impact of noise and imbalanced data on the efficiency of the HCPDP and with-in project defect prediction (WPDP) model is examined empirically and conceptually using four different classification algorithms. In addition, CIP is handled using a novel technique known as chunk balancing algorithm (CBA). Ten prediction combinations from three open-source projects are used in the experimental investigation. The findings show that noise in an imbalanced dataset has a significant impact on defect prediction accuracy.

A Method-Level Defect Prediction Approach Based on Structural Features of Method-Calling Network

Software defect prediction models help testers find program modules that have a high probability of having defects. A method-calling network can express the dependencies between methods in a program. Existing approaches do not sufficiently utilize method-calling network to characterize the structural features between methods. To address this problem, in this study, it is proposed for the first time that the characteristics of methods in a program are obtained by analyzing the method-calling network, and a new approach is proposed for defect prediction at the method-level. Specifically in this study, the method-calling network of the program was first constructed and the network metrics of the method-calling network were obtained. Next, the new network embedding technique (node2vec) was used to automatically encode the method-calling network structure into a low-dimensional vector to obtain the network embedding metrics. Finally, they were combined with code metrics to construct defect prediction models. We evaluated our approaches on 13 open-source software systems. The experimental results show that the proposed method improved the values of area under the receiver operating characteristic curve by 2.5% to 6.7% and Matthews correlation coefficient by 13% to 178.4% compared to the baselines. Therefore the method-calling network contains rich structural features between methods, and the structural features extend the features used for method-level defect prediction and further improve the performance of defect prediction models.

A Review on Software Fault Detection Mechanisms and Fault Prevention Mechanisms in Networks

It is possible to improve software quality by anticipating fault location through the utilization of software metrics within fault prediction models in network. This article provides a comprehensive literature review on the topic of software fault forecasting. The paper also seeks to identify software metrics and evaluate how applicable those metrics are to the process of software fault prediction. It is recommended that additional research be conducted on large industrial software systems to identify metrics that are more pertinent for the industry and to find an answer to the question of which metrics should be employed in a particular setting.

Heterogeneous Defect Prediction Algorithm Combined With Federated Sparse Compression

Heterogeneous defect prediction (HDP) constructs a defect prediction model through the source project to realize the defect tendency prediction of the target project. HDP based on federated learning can combine multi-party defect data to improve the defect prediction effect under the condition of protecting privacy. However, with the increase of task complexity and model performance requirements, the number of neural network layers gradually deepens, and the model parameters also increase. increase accordingly. In a federated learning scenario with multiple model parameters, limited communication bandwidth, and multiple clients, the amount of data received by the server is very large, causing huge communication pressure and seriously affecting the overall training efficiency. In order to reduce the communication cost, this paper proposes a federated sparse compression (FSC) algorithm. First, in order to improve the generalization performance of the model, each participant performs local training based on CapsNet, and achieves better prediction performance by calculating the relative position information of feature combinations. Then the gradient parameters are encrypted based on differential privacy to ensure data security. In order to reduce the amount of communication bits, the encrypted model parameters are subjected to sparse binary compression, the model training is converted from dense calculation to sparse calculation, and Golomb encoding is selected to be sent to the server for aggregation. Finally, the server side decodes the received data, performs sparse binary compression and encoding and sends it back to each participant. Experiments on nine projects in three public databases (Relink, NASA, and AEEEM) show that FSC can effectively reduce the amount of communication bits.

Exploring the challenges in software testing of the 5G system at Nokia: A survey

The ever-growing size and complexity of industrial software products pose significant quality assurance challenges to engineering researchers and practitioners, despite the constant effort to increase knowledge and improve the processes. 5G technology developed by Nokia is one example of such a grand and highly complex system with improvement potential. The following paper provides an overview of the current quality assurance processes used by Nokia to develop the 5G technology and provides insight into the most prominent challenges by an evaluation of perceived importance, urgency, and difficulty to understand the future opportunities. Nokia mode of operation, briefly introduced in this paper, has been subjected to extensive analysis by a selected group of experienced test-oriented professionals to define the most critical areas of concern. Secondly, the identified problems were evaluated by Nokia gNB system-level test professionals in a dedicated survey. The questionnaire was completed by 312 out of 2935 (10.63%) possible respondents. The challenges are seen as the most important and urgent: customer scenario testing, performance testing, and competence ramp-up. Challenges seen as the most difficult to solve are low occurrence failures, hidden feature dependencies, and hardware configuration-specific problems. Our research identified several improvement areas in the quality assurance processes used to develop the 5G technology by determining the most important and urgent problems that at the same time have a low perceived difficulty. Such initiatives are attractive from a business perspective. On the other hand, challenges seen as the most impactful yet difficult may be of interest to the academic research community. • Our survey, designed to elicit understanding of the most significant challenges in testing large and complex 5G gNB software system, received 312 out of 2935 (10.63%) system-level test practitioners in Nokia. • Challenges seen as having the highest importance are: customer scenario testing, performance testing, and competence ramp-up. • Challenges seen as having the highest urgency are: customer scenario testing, performance testing, and competence ramp-up. • Challenges seen as being the most difficult are: low occurrence failures, hidden feature dependencies, and hardware configuration-specific problems. • Obtained results imply which future improvement opportunities can have the most significant impact on the quality and cost of the software testing processes in the company. • Interestingly, respondents still seem to be not well familiar with software defect prediction models.

Analysis of Feature Selection Methods in Software Defect Prediction Models

Analysis of Feature Selection Methods in Software Defect Prediction Models

Fault prediction of industrial machinery and equipment

In this paper, we conduct a study on failure prediction of industrial machinery and equipment in order to improve the competitiveness of enterprises. We use Bootstrap to resample the dataset before the experiment. Then, we used four algorithms to build the prediction model, and used GridSearchCV to automatically adjust the parameters and train the equipment records extracted from the dataset. We selected the accuracy, F1 score, and ROC curve as the evaluation indexes of the model, and the evaluation results were compared, so we finally chose the fault prediction model built by LightGBMClassifier algorithm, and then used the confusion matrix to evaluate the model performance.

Data-level Hybrid Strategy Selection for Disk Fault Prediction Model Based on Multivariate GAN

Data-level Hybrid Strategy Selection for Disk Fault Prediction Model Based on Multivariate GAN

Software Defect Prediction Framework Using Hybrid Software Metric

Software fault prediction is widely used in the software development industry. Moreover, software development has accelerated significantly during this epidemic. However, the main problem is that most fault prediction models disregard object-oriented metrics, and even academician researcher concentrate on predicting software problems early in the development process. This research highlights a procedure that includes an object-oriented metric to predict the software fault at the class level and feature selection techniques to assess the effectiveness of the machine learning algorithm to predict the software fault. This research aims to assess the effectiveness of software fault prediction using feature selection techniques. In the present work, software metric has been used in defect prediction. Feature selection techniques were included for selecting the best feature from the dataset. The results show that process metric had slightly better accuracy than the code metric.

GIS Fault Prediction Approach Based on IPSO-LSSVM Algorithm

With the improvement of industrialization, the importance of equipment failure prediction is increasing day by day. Accurate failure prediction of gas-insulated switchgear (GIS) in advance can reduce the economic loss caused by the failure of the power system to operate normally. Therefore, a GIS fault prediction approach based on Improved Particle Swarm Optimization Algorithm (IPSO)-least squares support vector machine (LSSVM) is proposed in this paper. Firstly, the future gas conditions of the GIS to determine the characteristic data of SF6 decomposition gas are analyzed; Secondly, a GIS fault prediction model based on LSSVM is established, and the IPSO algorithm is used to normalize the parameters LSSVM. The parameters of c and radial basis kernel function σ2 are optimized, which can meet the needs of later search accuracy while ensuring the global search capability in the early stage. Finally, the effectiveness of the proposed method is verified by the fault data of gas-insulated switch. Simulation results shows that, compared with the prediction methods based on IGA-LSSVM and PSO-LSSVM, the prediction accuracy rate of the proposed method reached 92.1%, which has the smallest prediction absolute error, higher accuracy and stronger prediction ability.

Software defect prediction with semantic and structural information of codes based on Graph Neural Networks

Most defect prediction methods consider a series of traditional manually designed static code metrics. However, only using these hand-crafted features is impractical. Some researchers use the Convolutional Neural Network (CNN) to capture the potential semantic information based on the program’s Syntax Trees (ASTs). In recent years, leveraging the dependency relationships between software modules to construct a software network and using network embedding models to capture the structural information have been helpful in defect prediction. This paper simultaneously takes the semantic and structural information into account and proposes a method called CGCN. This study aims to validate the feasibility and performance of the proposed method in software defect prediction. Abstract Syntax Trees and a Class Dependency Network (CDN) are first generated based on the source code. For ASTs, symbolic tokens are extracted and encoded into vectors. The numerical vectors are then used as input to the CNN to capture the semantic information. For CDN, a Graph Convolutional Network (GCN) is used to learn the structural information of the network automatically. Afterward, the learned semantic and structural information are combined with different weights. Finally, we concatenate the learned features with traditional hand-crafted features to train a classifier for more accurate defect prediction. The proposed method outperforms the state-of-the-art defect prediction models for both within-project prediction (including within-version and cross-version) and cross-project prediction on 21 open-source projects. In general, within-version prediction achieves better performance in the three prediction tasks. The proposed method of combining semantic and structural information can improve the performance of software defect prediction. • A novel defect prediction model (CGCN), which combines CNN and GCN. The proposed model can extract the semantic information from the ASTs of the source codes, as well as the structural information between modules from the software network. Therefore, the defect prediction performance is improved. • Different weight parameters are used for semantic and structural features, in order to make them play different contributions in different tasks and improve the model generalization ability. • The efficiency of CGCN is evaluated on seven Java open-source projects (each project selects three versions, with a total of 21 datasets) and three tasks (withinversion defect prediction, cross-version defect prediction and cross-project defect prediction). The results demonstrate that the CGCN variant using weight parameters outperforms the state-of-the-art methods, while the CGCN variant without using weight parameters outperforms the benchmarks only in the withinversion defect prediction. • Different from “ GCN2defect: Graph Convolutional Networks for SMOTETomekbased Software Defect Prediction ” on 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE), in addition to the features of software network, this paper also combines the semantic features of source codes, and extends to cross-version and cross-project predictions. Experimental results show that the proposed method in this paper has better robustness.