Software Defect Datasets Research Articles

Bug numbers matter: An empirical study of effort‐aware defect prediction using class labels versus bug numbers

AbstractPrevious research have utilized public software defect datasets such as NASA, RELINK, and SOFTLAB, which only contain class label information. Most effort‐aware defect prediction (EADP) studies are carried out around these datasets. However, EADP studies typically relying on predicted bug number (i.e., considering modules as effort) or density (i.e., considering lines of code as effort) for ranking software modules. To explore the impact of bug number information in constructing EADP models, we access the performance degradation of the best‐performing learning‐to‐rank methods when using class labels instead of bug numbers for training. The experimental results show that using class labels instead of bug numbers in building EADP models results in an decrease in the detected bugs when module is considering as effort. When effort is LOC, using class labels to construct EADP models can lead to a significant increase in the initial false alarms and a significant increase in the modules that need to be inspected. Therefore, we recommend not only the class labels but also the bug number information should be disclosed when publishing software defect datasets, in order to construct more accurate EADP models.

Hybrid feature selection method for predicting software defect

To address the challenges associated with the abundance of features in software datasets, this study proposes a novel hybrid feature selection method that combines quantum particle swarm optimization (QPSO) and principal component analysis (PCA). The objective is to identify a subset of relevant features that can effectively contribute to the accuracy of a predictive model based on an artificial neural network (ANN). The quantum particle swarm optimization algorithm is employed to optimize the selection of features by simulating the behavior of quantum particles in a search space. This approach enhances the exploration and exploitation capabilities, allowing for a more effective identification of relevant features. Furthermore, principal component analysis is integrated into the hybrid method to reduce dimensionality and remove multicollinearity among features, thereby improving the efficiency of the feature selection process. The proposed hybrid method is applied to software defect datasets, where the selected subset of features is fed into an artificial neural network for defect prediction. The performance of the hybrid model is compared with traditional feature selection methods, standalone QPSO, and PCA. Experimental results demonstrate the effectiveness of the hybrid approach in achieving superior predictive accuracy while reducing the dimensionality of the dataset. The proposed approach not only enhances prediction accuracy but also provides a more interpretable and efficient subset of features for building robust defect prediction models.

Understanding the Impact of Changes in Application Characteristics on SRGM

Due to the increasing reliance on technology in nearly every industry over the past three decades, it has become necessary to evaluate the performance of a software product prior to its formal release in the market. The properties of a software application, such as its complexity and lines of code, are subject to change over time as a result of factors namely the testing environment, allocation of resources, testing efficiency, and testing team’s expertise. The assumption of constant Fault Detection Rate (FDR) may not accurately anticipate the potential number of bugs correctly. Keeping all these considerations in mind, a framework is developed to incorporate change point in the development of a testing effort-based Software Reliability Growth Model (SRGM) that takes into account the effect of application characteristics under both perfect and imperfect debugging settings. In addition, these outcomes are compared to the model without a change point. The proposed model is validated on two real-life software fault datasets. The results demonstrate that the proposed model performs better than the model without a change point.

A hybrid approach for optimizing software defect prediction using a grey wolf optimization and multilayer perceptron

PurposeSoftware defect prediction (SDP) is a critical aspect of software quality assurance, aiming to identify and manage potential defects in software systems. In this paper, we have proposed a novel hybrid approach that combines Grey Wolf Optimization with Feature Selection (GWOFS) and multilayer perceptron (MLP) for SDP. The GWOFS-MLP hybrid model is designed to optimize feature selection, ultimately enhancing the accuracy and efficiency of SDP. Grey Wolf Optimization, inspired by the social hierarchy and hunting behavior of grey wolves, is employed to select a subset of relevant features from an extensive pool of potential predictors. This study investigates the key challenges that traditional SDP approaches encounter and proposes promising solutions to overcome time complexity and the curse of the dimensionality reduction problem.Design/methodology/approachThe integration of GWOFS and MLP results in a robust hybrid model that can adapt to diverse software datasets. This feature selection process harnesses the cooperative hunting behavior of wolves, allowing for the exploration of critical feature combinations. The selected features are then fed into an MLP, a powerful artificial neural network (ANN) known for its capability to learn intricate patterns within software metrics. MLP serves as the predictive engine, utilizing the curated feature set to model and classify software defects accurately.FindingsThe performance evaluation of the GWOFS-MLP hybrid model on a real-world software defect dataset demonstrates its effectiveness. The model achieves a remarkable training accuracy of 97.69% and a testing accuracy of 97.99%. Additionally, the receiver operating characteristic area under the curve (ROC-AUC) score of 0.89 highlights the model’s ability to discriminate between defective and defect-free software components.Originality/valueExperimental implementations using machine learning-based techniques with feature reduction are conducted to validate the proposed solutions. The goal is to enhance SDP’s accuracy, relevance and efficiency, ultimately improving software quality assurance processes. The confusion matrix further illustrates the model’s performance, with only a small number of false positives and false negatives.

Software Fault Prediction Using Optimal Classifier Selection: An Ensemble Approach

Fault prediction is the process of using data analysis and machine learning models to anticipate potential defects or faults in the software system. Using only the base machine learning models for software fault prediction leads to limited performance, difficulty in handling non-linear relationships and imbalanced data, inadequate feature representation, and limited complexity handling. Hence, in order to overcome these challenges, this paper proposes a new technique for the selection of classifiers that forms a heterogeneous ensemble. The main goal is to remove or trim out the classifiers that show poor performance compared to the other base classifiers, which can result into a more effective ensemble and can produce better results. The algorithm proposed in this paper finds a set of classifiers that can perform better than using all the classifiers. The challenge that was faced was how to identify the poor-performing classifiers. This challenge is dealt with by performing an experiment using different threshold values to choose the trimmed set of classifiers. For evaluation of the proposed model, 8 different benchmark software fault datasets were used, which are taken from PROMISE and the Apache repository, and AUC is used as the performance measure. The results obtained after the experimental analysis demonstrate the effectiveness of our algorithm compared to the traditional approaches, which used all the base classifiers. There is a significant increase in the AUC values for 6 datasets out of 8, while using the average of probabilities and majority voting, it was seen that there is improvement in 7 out of 8 datasets used. The best-performing dataset by using the average of probabilities is ARC, where the AUC values increase from 0.6505 to 0.694, and while using majority voting, the best-performing dataset is XALAN, where the AUC values increase from 0.5455 to 0.679. From this, it can be seen that the proposed ensemble approach achieved higher AUC values for the tested datasets when compared to the base machine learning classifiers.

Leveraging Ensemble Learning with Generative Adversarial Networks for Imbalanced Software Defects Prediction

Software defect prediction is an active research area. Researchers have proposed many approaches to overcome the imbalanced defect problem and build highly effective machine learning models that are not biased towards the majority class. Generative adversarial networks (GAN) are one of the state-of-the-art techniques that can be used to generate synthetic samples of the minority class and produce a balanced dataset. However, it was not investigated thoroughly in the area of imbalanced defect prediction. In this paper, we proposed to combine GAN-based methods with boosting ensembles to yield robust defect prediction models. GAN-based methods were used to balance the defect datasets, and the AdaBoost ensemble was employed to classify the modules into defective and non-defective modules. Our proposed approach was investigated within the context of 10 software defect datasets with different imbalance ratios. Wilcoxon effect size and Scott–Knott effect size difference tests were used as statistical tests to quantify the model’s performance differences statistically. Empirical results indicated that GAN-based methods need hyperparameter optimization when used for imbalanced software defect prediction. In comparison to the traditional sampling techniques, GAN methods outperformed all traditional techniques when used for imbalanced defect prediction. Lastly, results demonstrated that GAN-based methods should not be combined with undersampling to handle imbalance problems.

A software defect prediction method based on learnable three-line hybrid feature fusion

Software defect prediction (SDP) plays a crucial role in ensuring the security and quality of software systems. However, it faces challenges posed by high-dimensional features present in software defect datasets and the limited effectiveness of traditional nonlinear dimensionality reduction methods in extracting essential feature information. To address these issues, we propose a novel approach called learnable three-line hybrid feature fusion (LTHFFA), which incorporates the principle of three-line hybrid breeding into feature fusion for the first time. In this method, three distinct dimensionality reduction techniques are initially employed to obtain three separate sets of features. Subsequently, a learnable weight factor feature fusion method is proposed to facilitate automatically learn and dynamically update of feature weights. By integrating the three feature sets based on the principle of three-line hybrid breeding, we derive learnable three-line hybrid fusion features. These features are then utilized in the context of software defect prediction. Experimental results demonstrate the superior performance of LTHFFA compared to nine other dimensionality reduction methods across seventeen publicly available software defect datasets. LTHFFA exhibits the ability to effectively integrate multiple feature sets, reduce feature redundancy, and enhance predictive accuracy. Moreover, statistical analysis using Friedman ranking and Holm's post-hoc test confirms the significant advantage of LTHFFA over alternative dimensionality reduction methods.

Privacy Protection Optimization for Federated Software Defect Prediction via Benchmark Analysis

<p>Federated learning is a privacy-preserving machine learning technique that coordinates multi-participant co-modeling. It can alleviate the privacy issues of software defect prediction, which is an important technical way to ensure software quality. In this work, we implement Federated Software Defect Prediction (FedSDP) and optimize its privacy issues while guaranteeing performance. We first construct a new benchmark to study the performance and privacy of Federated Software defect prediction. The benchmark consists of (1) 12 NASA software defect datasets, which are all real software defect datasets from different projects in different domains, (2) Horizontal federated learning scenarios, and (3) the Federated Software Defect Prediction algorithm (FedSDP). Benchmark analysis shows that FedSDP provides additional privacy protection and security with guaranteed model performance compared to local training. It also reveals that FedSDP introduces a large amount of model parameter computation and exchange during the training process. There are model user threats and attack challenges from unreliable participants. To provide more reliable privacy protection without losing prediction performance we proposed optimization methods that use homomorphic encryption model parameters to resist honest but curious participants. Experimental results show that our approach achieves more reliable privacy protection with excellent performance on all datasets.</p> <p> </p>

Improving Software Defect Prediction in Noisy Imbalanced Datasets

Software defect prediction is a popular method for optimizing software testing and improving software quality and reliability. However, software defect datasets usually have quality problems, such as class imbalance and data noise. Oversampling by generating the minority class samples is one of the most well-known methods to improving the quality of datasets; however, it often introduces overfitting noise to datasets. To better improve the quality of these datasets, this paper proposes a method called US-PONR, which uses undersampling to remove duplicate samples from version iterations and then uses oversampling through propensity score matching to reduce class imbalance and noise samples in datasets. The effectiveness of this method was validated in a software prediction experiment that involved 24 versions of software data in 11 projects from PROMISE in noisy environments that varied from 0% to 30% noise level. The experiments showed a significant improvement in the quality of datasets pre-processed by US-PONR in noisy imbalanced datasets, especially the noisiest ones, compared with 12 other advanced dataset processing methods. The experiments also demonstrated that the US-PONR method can effectively identify the label noise samples and remove them.

A Hybrid Software Defects Prediction Model for Imbalance Datasets Using Machine Learning Techniques: (S-SVM Model)

Software defect prediction (SDP) is an essential task for developing quality software, and various models have been developed for this purpose. However, the imbalanced nature of software defect datasets has challenged these models, resulting in decreased performance. To address this challenge, the author has proposed a hybrid machine learning model that combines Synthetic Minority Oversampling Technique (SMOTE) with Support Vector Machine (SVM)—SMOTE-SVM (S-SVM) model. The author has empirically examined SDP using multiple datasets (CM1, PC1, JM1, PC3, KC1, EQ and JDT) from the PROMISE and AEEEM repositories. The experimental study indicates that the S-SVM model involved training and compared with previously developed balanced and imbalanced test datasets using four evaluation metrics: Precision, Recall, F1 score, and Accuracy. For the balanced dataset, the S-SVM model achieved precision values ranging from 70 to 96, recall values ranging from 52 to 94, F1-score values ranging from 67 to 90, and accuracy values ranging from 69 to 98. For the imbalanced dataset, the S-SVM model achieved precision values ranging from 60 to 93, recall values ranging from 64 to 97, F1-score values ranging from 69 to 91, and accuracy values ranging from 67 to 87. The proposed S-SVM model outperforms other models’ ability to classify and predict software defects. Therefore, the hybridisation of SMOTE and SVM improved the model’s ability to categories and predict balanced and imbalanced datasets when sufficient defective and non-defective data is provided.

Class Imbalance Reduction and Centroid based Relevant Project Selection for Cross Project Defect Prediction

Cross-Project Defect Prediction (CPDP) is the process of predicting defects in a target project using information from other projects. This can assist developers in prioritizing their testing efforts and finding flaws. Transfer Learning (TL) has been frequently used at CPDP to improve prediction performance by reducing the disparity in data distribution between the source and target projects. Software Defect Prediction (SDP) is a common study topic in software engineering that plays a critical role in software quality assurance. To address the cross-project class imbalance problem, Centroid-based PF-SMOTE for Imbalanced data is used. In this paper, we used a Centroid-based PF-SMOTE to balance the datasets and Centroid based relevant data selection for Cross Project Defect Prediction. These methods use the mean of all attributes in a dataset and calculating the difference between mean of all datasets. For experimentation, the open source software defect datasets namely, AEEM, Re-Link, and NASA, are considered.

Concept Drift in Software Defect Prediction: A Method for Detecting and Handling the Drift

Software Defect Prediction (SDP) is crucial towards software quality assurance in software engineering. SDP analyzes the software metrics data for timely prediction of defect prone software modules. Prediction process is automated by constructing defect prediction classification models using machine learning techniques. These models are trained using metrics data from historical projects of similar types. Based on the learned experience, models are used to predict defect prone modules in currently tested software. These models perform well if the concept is stationary in a dynamic software development environment. But their performance degrades unexpectedly in the presence of change in concept (Concept Drift). Therefore, concept drift (CD) detection is an important activity for improving the overall accuracy of the prediction model. Previous studies on SDP have shown that CD may occur in software defect data and the used defect prediction model may require to be updated to deal with CD. This phenomenon of handling the CD is known as CD adaptation. It is observed that still efforts need to be done in this direction in the SDP domain. In this article, we have proposed a pair of paired learners (PoPL) approach for handling CD in SDP. We combined the drift detection capabilities of two independent paired learners and used the paired learner (PL) with the best performance in recent time for next prediction. We experimented on various publicly available software defect datasets garnered from public data repositories. Experimentation results showed that our proposed approach performed better than the existing similar works and the base PL model based on various performance measures.

Efficient Random Forest Algorithm for Multi-objective Optimization in Software Defect Prediction

The Software Defect Prediction (SDP) process provides reliable software by identifying defect-prone modules before the testing stage. It efficiently and effectively utilizes quality assurance resources. Most predictive models are trained on historical data which belong to the same project or comparable project. These models show satisfactory performance as they utilize similar settings to the considered projects. But the limitation of these models is that they are effective only if there are adequate historical data to train a predictive model. In reality, however, such historical data are minimal for some projects and absent for new projects. The defect prediction in such projects which lack historical data can be accomplished by training prediction models on different project data. This process is known as Cross-Project Defect Prediction (CPDP). Software defect datasets also suffered from class imbalance issues which further degrades the model’s performance. In this research work, the authors have proposed a Multi-Objective Random Forest (MO-RF) algorithm with a data resampling technique to minimize the probability of false alarms, to maximize the probability of detection and to overcome the class imbalance problem. The study also evaluates the performance of other prediction models. The proposed method has shown percentage improvement (in terms of AUC) of 2.78 and 3.46 over MONB and MONBNN, respectively.

A GABP based method to improved software defect prediction

Considering the uneven distribution of software defect datasets, the performance of traditional SDP (software defect prediction) model is relatively low, which cannot predict the defects well and fix them in time, resulting in the low reliability of the system. This paper proposed an improved SDP method based on GABP, which helps to allocate test resources reasonably, ensures the quality of software and improves the reliability of the system. Firstly, combining with the traditional BP neural network, SMOTE and TOMEK are applied to select the better data preprocessing method and obtain the initial network architecture; Secondly, GA is employed to optimize the network architecture obtained previously to build the final predictor. Experiments on five real datasets demonstrate that the proposed method can improve the prediction performance and show its effectiveness.

Improving Cross-Project Software Defect Prediction Method Through Transformation and Feature Selection Approach

In the traditional software defect prediction methodology, the historical record (dataset) of the same project is partitioned into training and testing data. In a practical situation where the project to be predicted is new, traditional software defect prediction cannot be employed. An alternative method is cross-project defect prediction, where the historical record of one project (source) is used to predict the defect status of another project (target). The cross-project defect prediction method solves the limitations of the historical records in the traditional software defect prediction method. However, the performance of cross-project defect prediction is relatively low because of the distribution differences between the source and target projects. Furthermore, the software defect dataset used for cross-project defect prediction is characterized by high-dimensional features, some of which are irrelevant and contribute to low performance. To resolve these two issues, this study proposes a transformation and feature selection approach to reduce the distribution difference and high-dimensional features in cross-project defect prediction. A comparative experiment was conducted on publicly available datasets from the AEEEM. Analysis of the results obtained shows that the proposed approach in conjugation with random forest as the classification model outperformed the other four state-of-the-art cross-project defect prediction methods based on the commonly used performance evaluation metric F1_score.

Data and Ensemble Machine Learning Fusion Based Intelligent Software Defect Prediction System

The software engineering field has long focused on creating high-quality software despite limited resources. Detecting defects before the testing stage of software development can enable quality assurance engineers to concentrate on problematic modules rather than all the modules. This approach can enhance the quality of the final product while lowering development costs. Identifying defective modules early on can allow for early corrections and ensure the timely delivery of a high-quality product that satisfies customers and instills greater confidence in the development team. This process is known as software defect prediction, and it can improve end-product quality while reducing the cost of testing and maintenance. This study proposes a software defect prediction system that utilizes data fusion, feature selection, and ensemble machine learning fusion techniques. A novel filter-based metric selection technique is proposed in the framework to select the optimum features. A three-step nested approach is presented for predicting defective modules to achieve high accuracy. In the first step, three supervised machine learning techniques, including Decision Tree, Support Vector Machines, and Naïve Bayes, are used to detect faulty modules. The second step involves integrating the predictive accuracy of these classification techniques through three ensemble machine-learning methods: Bagging, Voting, and Stacking. Finally, in the third step, a fuzzy logic technique is employed to integrate the predictive accuracy of the ensemble machine learning techniques. The experiments are performed on a fused software defect dataset to ensure that the developed fused ensemble model can perform effectively on diverse datasets. Five NASA datasets are integrated to create the fused dataset: MW1, PC1, PC3, PC4, and CM1. According to the results, the proposed system exhibited superior performance to other advanced techniques for predicting software defects, achieving a remarkable accuracy rate of 92.08%.

Cloud-based bug tracking software defects analysis using deep learning

Cloud technology is not immune to bugs and issue tracking. A dedicated system is required that will extremely error prone and less cumbersome and must command a high degree of collaboration, flexibility of operations and smart decision making. One of the primary goals of software engineering is to provide high-quality software within a specified budget and period for cloud-based technology. However, defects found in Cloud-Based Bug Tracking software’s can result in quality reduction as well as delay in the delivery process. Therefore, software testing plays a vital role in ensuring the quality of software in the cloud, but software testing requires higher time and cost with the increase of complexity of user requirements. This issue is even cumbersome in the embedded software design. Early detection of defect-prone components in general and embedded software helps to recognize which components require higher attention during testing and thereby allocate the available resources effectively and efficiently. This research was motivated by the demand of minimizing the time and cost required for Cloud-Based Bug Tracking Software testing for both embedded and general-purpose software while ensuring the delivery of high-quality software products without any delays emanating from the cloud. Not withstanding that several machine learning techniques have been widely applied for building software defect prediction models in general, achieving higher prediction accuracy is still a challenging task. Thus, the primary aim of this research is to investigate how deep learning methods can be used for Cloud-Based Bug Tracking Software defect detection with a higher accuracy. The research conducted an experiment with four different configurations of Multi-Layer Perceptron neural network using five publicly available software defect datasets. Results of the experiments show that the best possible network configuration for software defect detection model using Multi-Layer Perceptron can be the prediction model with two hidden layers having 25 neurons in the first hidden layer and 5 neurons in the second hidden layer.

A Survey of Different Approaches for the Class Imbalance Problem in Software Defect Prediction

The imbalanced nature of the software datasets leads to the biased learning of prediction model toward the observations of the majority class (non-defective class). The prediction model can produce poor results for the minority class observations. Such misappropriations can prove costly especially in software development where minority class (defective) is the one that has the highest interest from the learning point of view. Various approaches have been used for dealing with class imbalance problem of software defect prediction but no one dominates and hence developing a generalized software defect prediction model for imbalanced datasets remains problematic. This paper surveys existing approaches for handling class imbalance problem of software defect datasets. In this survey, most relevant software defect prediction studies and identified the two main approaches that have been used for handling imbalance issue of software defect datasets. Furthermore, we also provide some comparison of findings in state-of-the-art literature and the guidelines for carrying out future research.

Generative Oversampling Methods for Handling Imbalanced Data in Software Fault Prediction

Imbalanced software fault datasets, having fewer faulty modules than the nonfaulty modules, make accurate fault prediction difficult. It is challenging for software practitioners to handle imbalanced fault data during software fault prediction (SFP). Earlier, several researchers have applied oversampling techniques such as synthetic minority oversampling techniques and others for imbalanced learning in SFP. However, most of these techniques resulted in overfitted prediction models. This article presents generative oversampling methods to handle imbalanced data problems in the SFP. Using the generative adversarial network (GAN) based approach, the presented methods generate synthetic samples of the faulty modules to balance the proportion of faulty and nonfaulty modules in the fault datasets. Further, SFP models are built on the processed fault datasets using different machine learning techniques. Experimental validation of the presented oversampling methods is done on 18 fault datasets gathered from PROMISE, JIRA, Eclipse data repositories, and precision, recall, f1-score, and AUC are used as evaluation measures. We extensively compared presented oversampling methods with various state-of-the-art class imbalance techniques and baseline models. The experimental results evidenced that the presented methods improved fault prediction performance and yielded better performance than the state-of-the-art class imbalance techniques.

Software defect prediction based on nested-stacking and heterogeneous feature selection

Software testing guarantees the delivery of high-quality software products, and software defect prediction (SDP) has become an important part of software testing. Software defect prediction is divided into traditional software defect prediction and just-in-time software defect prediction (JIT-SDP). However, most of the existing software defect prediction frameworks are relatively simplified, which makes it extremely difficult to provide developers with more detailed reference information. To improve the effectiveness of software defect prediction and realize effective software testing resource allocation, this paper proposes a software defect prediction framework based on Nested-Stacking and heterogeneous feature selection. The framework includes three stages: data set preprocessing and feature selection, Nested-Stacking classifier, and model classification performance evaluation. The novel heterogeneous feature selection and nested custom classifiers in the framework can effectively improve the accuracy of software defect prediction. This paper conducts experiments on two software defect data sets (Kamei, PROMISE), and demonstrates the classification performance of the model through two comprehensive evaluation indicators, AUC, and F1-score. The experiment carried out large-scale within-project defect prediction (WPDP) and cross-project defect prediction (CPDP). The results show that the framework proposed in this paper has an excellent classification performance on the two types of software defect data sets, and has been greatly improved compared with the baseline models.