Cross-project Prediction Research Articles

KCO: Balancing class distribution in just-in-time software defect prediction using kernel crossover oversampling.

The performance of the defect prediction model by using balanced and imbalanced datasets makes a big impact on the discovery of future defects. Current resampling techniques only address the imbalanced datasets without taking into consideration redundancy and noise inherent to the imbalanced datasets. To address the imbalance issue, we propose Kernel Crossover Oversampling (KCO), an oversampling technique based on kernel analysis and crossover interpolation. Specifically, the proposed technique aims to generate balanced datasets by increasing data diversity in order to reduce redundancy and noise. KCO first represents multidimensional features into two-dimensional features by employing Kernel Principal Component Analysis (KPCA). KCO then divides the plotted data distribution by deploying spectral clustering to select the best region for interpolation. Lastly, KCO generates the new defect data by interpolating different data templates within the selected data clusters. According to the prediction evaluation conducted, KCO consistently produced F-scores ranging from 21% to 63% across six datasets, on average. According to the experimental results presented in this study, KCO provides more effective prediction performance than other baseline techniques. The experimental results show that KCO within project and cross project predictions especially consistently achieve higher performance of F-score results.

Automatic prediction of developers’ resolutions for software merge conflicts

In collaborative software development, developers simultaneously work in parallel on different branches that they merge periodically. When edits from different branches textually overlap, conflicts may occur. Manually resolving conflicts can be tedious and error-prone. Researchers proposed tool support for conflict resolution, but these tools barely consider developers’ preferences. Conflicts can be resolved by: keeping the local version only KL, keeping the remote version only (KR), or manually editing them (ME). Recent studies show that developers resolved the majority of textual conflicts by KL or KR. Thus, we created a machine learning-based approach RPredictor to predict developers’ resolution strategy (KL, KR, or ME) given a merge conflict. We did large-scale experiments on the historical resolution of 74,861 conflicts. Our experiments show that RPredictor achieved 63% F-score for within-project prediction and 46% F-score for cross-project prediction. Compared with other classifiers, RPredictor provides the highest effectiveness when using a random forest (RF) classifier. Finally, we proposed a variant technique RPredictorv, which enables developers to customize its prediction conservativeness. For a highly conservative setting, RPredictorv achieved 34% effort saving while minimizing the risk of producing incorrect prediction labels.

SMOTE-Based Homogeneous Prediction for Aging-Related Bugs in Cloud-Oriented Software

Software aging is the process caused by Aging-Related Bugs (ARBs) which leads to the depletion of resources and degradation of performance in the long run. ARBs are difficult to find and replicate in future studies as they are less in number, thus prediction of ARB is necessary to save cost and time in the testing phase. ARBs are present in low proportion as compared to non-ARBs known as the class Imbalance problem resulting in insufficient training dataset for prediction models. In this study, Synthetic Minority Oversampling Technique (SMOTE) is applied along with homogeneous cross-project ARB prediction to reduce the effect of imbalance problem in software. SMOTE is oversampling of the minority instances synthetically to balance the dataset and improve the capability of defect prediction models. Homogeneous cross-project prediction is implemented where the datasets are different but the distribution of metric sets of both training and testing datasets is similar. The experiment is conducted on five cloud-oriented software such as Cassandra, Hive, Storm, Hadoop HDFS and Hadoop Mapreduce. The novelty of this study is the combination of SMOTE and homogeneous cross-project defect prediction for ARBs in cloud-oriented software. The comparative analysis is also conducted to understand the difference between SMOTE and non-SMOTE results with the help of machine learning classifiers. The result conveys that SMOTE is an efficient method to address class imbalance problem in ARB prediction.

Cross-project prediction for rock mass using shuffled TBM big dataset and knowledge-based machine learning methods

Cross-project prediction for rock mass using shuffled TBM big dataset and knowledge-based machine learning methods

Software defect prediction with semantic and structural information of codes based on Graph Neural Networks

Most defect prediction methods consider a series of traditional manually designed static code metrics. However, only using these hand-crafted features is impractical. Some researchers use the Convolutional Neural Network (CNN) to capture the potential semantic information based on the program’s Syntax Trees (ASTs). In recent years, leveraging the dependency relationships between software modules to construct a software network and using network embedding models to capture the structural information have been helpful in defect prediction. This paper simultaneously takes the semantic and structural information into account and proposes a method called CGCN. This study aims to validate the feasibility and performance of the proposed method in software defect prediction. Abstract Syntax Trees and a Class Dependency Network (CDN) are first generated based on the source code. For ASTs, symbolic tokens are extracted and encoded into vectors. The numerical vectors are then used as input to the CNN to capture the semantic information. For CDN, a Graph Convolutional Network (GCN) is used to learn the structural information of the network automatically. Afterward, the learned semantic and structural information are combined with different weights. Finally, we concatenate the learned features with traditional hand-crafted features to train a classifier for more accurate defect prediction. The proposed method outperforms the state-of-the-art defect prediction models for both within-project prediction (including within-version and cross-version) and cross-project prediction on 21 open-source projects. In general, within-version prediction achieves better performance in the three prediction tasks. The proposed method of combining semantic and structural information can improve the performance of software defect prediction. • A novel defect prediction model (CGCN), which combines CNN and GCN. The proposed model can extract the semantic information from the ASTs of the source codes, as well as the structural information between modules from the software network. Therefore, the defect prediction performance is improved. • Different weight parameters are used for semantic and structural features, in order to make them play different contributions in different tasks and improve the model generalization ability. • The efficiency of CGCN is evaluated on seven Java open-source projects (each project selects three versions, with a total of 21 datasets) and three tasks (withinversion defect prediction, cross-version defect prediction and cross-project defect prediction). The results demonstrate that the CGCN variant using weight parameters outperforms the state-of-the-art methods, while the CGCN variant without using weight parameters outperforms the benchmarks only in the withinversion defect prediction. • Different from “ GCN2defect: Graph Convolutional Networks for SMOTETomekbased Software Defect Prediction ” on 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE), in addition to the features of software network, this paper also combines the semantic features of source codes, and extends to cross-version and cross-project predictions. Experimental results show that the proposed method in this paper has better robustness.

Towards building a pragmatic cross-project defect prediction model combining non-effort based and effort-based performance measures for a balanced evaluation

• We propose a two-phase transfer boosting-based cross-project prediction model. • The proposed model is assessed using non-effort based and effort-based measures. • Training data weights are based on feature importance and inter-project similarity. • We validate the effectiveness of the proposed model on a large corpus of data. Recent years have witnessed the growing trend in cross-project defect prediction (CPDP), where the training and the testing data come from different projects having different data distributions. Several CPDP methods have been presented in the literature to overcome differences in their distributions, but the majority of the existing approaches have been evaluated considering the availability of unlimited inspection effort, which is practically impossible, thus leading to fallacious conclusions. Further, they focused more on improving Recall over Precision leading to a high probability of false alarm (PF), causing significant wastage of developer's efforts and time. Addressing these issues, we propose a Two-Phase Transfer Boosting (TPTB) model, which aims at improving the performance not only in terms of non-effort based measures (NEBMs) (making a balance between Recall and PF) but also in terms of effort based measures (EBMs), considering the availability of limited inspection effort. To mitigate the distribution differences, the first phase assigns initial weights to the training modules based on the feature distribution and feature importance. The second phase applies the Dynamic Transfer AdaBoost algorithm to build an ensemble classifier to lessen the impact of contradictory training modules. In addition, a sorting strategy is designed to prioritize the modules for further inspection. Statistical results on 62 datasets revealed a better-balanced performance of our TPTB model holistically over NN-filter, ManualDown, EASC, and Cruz model with performance comparable to WPDP (Within-project defect prediction) considering NEBMs. Besides, when considering EBMs together, TPTB showed statistically and practically more balanced performance as compared to ManualUP and Cruz with overall performance comparable to EASC. Our results demonstrate the efficacy of the TPTB model in a practical setting empowering the quality assurance team to predict and prioritize the defective modules allocating limited inspection effort by optimally focusing on highly defective modules.

Are Source Code Metrics “Good Enough” in Predicting Security Vulnerabilities?

Modern systems produce and handle a large volume of sensitive enterprise data. Therefore, security vulnerabilities in the software systems must be identified and resolved early to prevent security breaches and failures. Predicting security vulnerabilities is an alternative to identifying them as developers write code. In this study, we studied the ability of several machine learning algorithms to predict security vulnerabilities. We created two datasets containing security vulnerability information from two open-source systems: (1) Apache Tomcat (versions 4.x and five 2.5.x minor versions). We also computed source code metrics for these versions of both systems. We examined four classifiers, including Naive Bayes, Decision Tree, XGBoost Classifier, and Logistic Regression, to show their ability to predict security vulnerabilities. Moreover, an ensemble learner was introduced using a stacking classifier to see whether the prediction performance could be improved. We performed cross-version and cross-project predictions to assess the effectiveness of the best-performing model. Our results showed that the XGBoost classifier performed best compared to other learners, i.e., with an average accuracy of 97% in both datasets. The stacking classifier performed with an average accuracy of 92% in Struts and 71% in Tomcat. Our best-performing model—XGBoost—could predict with an average accuracy of 87% in Tomcat and 99% in Struts in a cross-version setup.

Security versus Compliance: An Empirical Study of the Impact of Industry Standards Compliance on Application Security

The integration of security aspects into software development is an open topic, especially in highly regulated industries where standards are accompanied by a high degree of complexity. The research question of this paper relates to the misconception of industry standards compliance and security in the field of software development. Cyber attackers are constantly inventing new tools to penetrate systems and exploit even the most minor flaws, and adherence to an industry standard is not a solution. In this study, an empirical investigation is conducted over a six-month period to observe various customer relationship management (CRM) systems. To analyze and anticipate the vulnerabilities of various CRMs, penetration testing methodologies and cross-project prediction approaches are employed. Classification using multiple machine learning approaches is utilized in the study to increase the discovery of vulnerable components in each CRM. The Student [Formula: see text]-test is also used to assess if the mean values of the two CRM datasets are substantially different from each other in order to evaluate the efficacy of overall security and its features. The results show that security best practices during application development have a significant influence on applications created in regulated environments. The action research approach used to validate this study provided positive results and its feasibility in practice to optimize security throughout the application development. This study adds to the literature on information security management systems (ISMS) and best practices in application development in terms of creating and implementing opportunities based on broader information security management measures.

Local modeling approach for cross-project defect prediction

Prediction approaches used for cross-project defect prediction (CPDP) are usually impractical because of high false alarms, or low detection rate. Instance based data filter techniques that improve the CPDP performance are time-consuming and each time a new test set arrives for prediction the entire filter procedure is repeated. We propose to use local modeling approach for the utilization of ever-increasing cross-project data for CPDP. We cluster the cross-project data, train per cluster prediction models and predict the target test instances using corresponding cluster models. Over 7 NASA Data sets performance comparison using statistical methods between within-project, cross-project, and our local modeling approach were performed. Compared to within-project prediction the cross-project prediction increased the probability of detection (PD) associated with an increase in the probability of false alarm (PF) and decreased overall performance Balance. The application of local modeling decreased the (PF) associated with a decrease in (PD) and an overall performance improvement in terms of Balance. Moreover, compared to one state of the art filter technique – Burak filter, our approach is simple, fast, performance comparable, and opens a new perspective for the utilization of ever-increasing cross-project data for defect prediction. Therefore, when insufficient within-project data is available we recommend training local cluster models than training a single global model on cross-project datasets.

Deep Cross-Project Software Reliability Growth Model Using Project Similarity-Based Clustering

Software reliability is an essential characteristic for ensuring the qualities of software products. Predicting the potential number of bugs from the beginning of a development project allows practitioners to make the appropriate decisions regarding testing activities. In the initial development phases, applying traditional software reliability growth models (SRGMs) with limited past data does not always provide reliable prediction result for decision making. To overcome this, herein, we propose a new software reliability modeling method called a deep cross-project software reliability growth model (DC-SRGM). DC-SRGM is a cross-project prediction method that uses features of previous projects’ data through project similarity. Specifically, the proposed method applies cluster-based project selection for the training data source and modeling by a deep learning method. Experiments involving 15 real datasets from a company and 11 open source software datasets show that DC-SRGM can more precisely describe the reliability of ongoing development projects than existing traditional SRGMs and the LSTM model.

Why do builds fail?—A conceptual replication study

Previous studies have investigated a wide range of factors potentially explaining software build breakages, focusing primarily on build-triggering code changes or previous CI outcomes. However, code quality factors such as the presence of code/test smells have not been yet evaluated in the context of CI, even though such factors have been linked to problems of comprehension and technical debt, and hence might introduce bugs and build breakages. This paper performs a conceptual replication study on 27,675 Travis CI builds of 15 GitHub projects, considering the features reported by Rausch et al. and Zolfagharinia et al., as well as those related to code/test smells. Using a multivariate model constructed from nine dimensions of features, results indicate a precision (recall) ranging between 58.3% and 79.0% (52.4% and 69.6%) in balanced project datasets, and between 2.5% and 37.5% (2.5% and 12.4%) in imbalanced project datasets. Models trained on our balanced project datasets were later used to perform cross-project prediction on the imbalanced projects, achieving an average improvement of 9.3% (16.2%) in precision (recall). Statistically, the results confirm that features from the build history, author, code complexity, and code/test smell dimensions are the most important predictors of build failures.

Within-project and cross-project software defect prediction based on improved transfer Naive Bayes algorithm

With the continuous expansion of software scale, software update and maintenance have become more and more important. However, frequent software code updates will make the software more likely to introduce new defects. So how to predict the defects quickly and accurately on the software change has become an important problem for software developers. Current defect prediction methods often cannot reflect the feature information of the defect comprehensively, and the detection effect is not ideal enough. Therefore, we propose a novel defect prediction model named ITNB (Improved Transfer Naive Bayes) based on improved transfer Naive Bayesian algorithm in this paper, which mainly considers the following two aspects: (1) Considering that the edge data of the test set may affect the similarity calculation and final prediction result, we remove the edge data of the test set when calculating the data similarity between the training set and the test set; (2) Considering that each feature dimension has different effects on defect prediction, we construct the calculation formula of training data weight based on feature dimension weight and data gravity, and then calculate the prior probability and the conditional probability of training data from the weight information, so as to construct the weighted bayesian classifier for software defect prediction. To evaluate the performance of the ITNB model, we use six datasets from large open source projects, namely Bugzilla, Columba, Mozilla, JDT, Platform and PostgreSQL. We compare the ITNB model with the transfer Naive Bayesian (TNB) model. The experimental results show that our ITNB model can achieve better results than the TNB model in terms of accurary, precision and pd for within-project and cross-project defect prediction.

A systematic review of unsupervised learning techniques for software defect prediction

Abstract Background Unsupervised machine learners have been increasingly applied to software defect prediction. It is an approach that may be valuable for software practitioners because it reduces the need for labeled training data. Objective Investigate the use and performance of unsupervised learning techniques in software defect prediction. Method We conducted a systematic literature review that identified 49 studies containing 2456 individual experimental results, which satisfied our inclusion criteria published between January 2000 and March 2018. In order to compare prediction performance across these studies in a consistent way, we (re-)computed the confusion matrices and employed the Matthews Correlation Coefficient (MCC) as our main performance measure. Results Our meta-analysis shows that unsupervised models are comparable with supervised models for both within-project and cross-project prediction. Among the 14 families of unsupervised model, Fuzzy CMeans (FCM) and Fuzzy SOMs (FSOMs) perform best. In addition, where we were able to check, we found that almost 11% (262/2456) of published results (contained in 16 papers) were internally inconsistent and a further 33% (823/2456) provided insufficient details for us to check. Conclusion Although many factors impact the performance of a classifier, e.g., dataset characteristics, broadly speaking, unsupervised classifiers do not seem to perform worse than the supervised classifiers in our review. However, we note a worrying prevalence of (i) demonstrably erroneous experimental results, (ii) undemanding benchmarks and (iii) incomplete reporting. We therefore encourage researchers to be comprehensive in their reporting.

An empirical study of factors affecting cross-project aging-related bug prediction with TLAP

Software aging is a phenomenon in which long-running software systems show an increasing failure rate and/or progressive performance degradation. Due to their nature, Aging-Related Bugs (ARBs) are hard to discover during software testing and are also challenging to reproduce. Therefore, automatically predicting ARBs before software release can help developers reduce ARB impact or avoid ARBs. Many bug prediction approaches have been proposed, and most of them show effectiveness in within-project prediction settings. However, due to the low presence and reproducing difficulty of ARBs, it is usually hard to collect sufficient training data to build an accurate prediction model. A recent work proposed a method named Transfer Learning based Aging-related bug Prediction (TLAP) for performing cross-project ARB prediction. Although this method considerably improves cross-project ARB prediction performance, it has been observed that its prediction result is affected by several key factors, such as the normalization methods, kernel functions, and machine learning classifiers. Therefore, this paper presents the first empirical study to examine the impact of these factors on the effectiveness of cross-project ARB prediction in terms of single-factor pattern, bigram pattern, and triplet pattern and validates the results with the Scott-Knott test technique. We find that kernel functions and classifiers are key factors affecting the effectiveness of cross-project ARB prediction, while normalization methods do not show statistical influence. In addition, the order of values in three single-factor patterns is maintained in three bigram patterns and one triplet pattern to a large extent. Similarly, the order of values in the three bigram patterns is also maintained in the triplet pattern.

Substantiation of multinomial classification using ensemble learning approach

The software defect prediction is the most operative research domain in software engineering as it enhances its reliability. The availability of defect data related to different projects leads to cross project defect prediction an open issue. This paper instanced on multiclass/multinomial classification of defect prediction on different categories of cross projects. The ensemble learning statistical models – Random forest and Gradient Boosting are used for classification. An empirical study is carried out to determine the predictive performance of the within project and cross project prediction models. Depending on the number of defects, class level information is classified into one of three defined multiclass. The homogeneous set of object oriented metrics is used for training the model. Furthermore, k-fold cross validation is done to evaluate the training accuracy of the statistical models. Major outcome of the paper concludes that multinomial/multiclass classification is applicable on cross project data and has comparable results to within project defect data with statistical significance.

Cross-project bug type prediction based on transfer learning

The prediction of bug types provides useful insights into the software maintenance process. It can improve the efficiency of software testing and help developers adopt corresponding strategies to fix bugs before releasing software projects. Typically, the prediction tasks are performed through machine learning classifiers, which rely heavily on labeled data. However, for a software project that has insufficient labeled data, it is difficult to train the classification model for predicting bug types. Although labeled data of other projects can be used as training data, the results of the cross-project prediction are often poor. To solve this problem, this paper proposes a cross-project bug type prediction framework based on transfer learning. Transfer learning breaks the assumption of traditional machine learning methods that the training set and the test set should follow the same distribution. Our experiments show that the results of cross-project bug type prediction have significant improvement by adopting transfer learning. In addition, we have studied the factors that influence the prediction results, including different pairs of source and target projects, and the number of bug reports in the source project.

Neural Network-based Detection of Self-Admitted Technical Debt

Technical debt is a metaphor to reflect the tradeoff software engineers make between short-term benefits and long-term stability. Self-admitted technical debt (SATD), a variant of technical debt, has been proposed to identify debt that is intentionally introduced during software development, e.g., temporary fixes and workarounds. Previous studies have leveraged human-summarized patterns (which represent n-gram phrases that can be used to identify SATD) or text-mining techniques to detect SATD in source code comments. However, several characteristics of SATD features in code comments, such as vocabulary diversity, project uniqueness, length, and semantic variations, pose a big challenge to the accuracy of pattern or traditional text-mining-based SATD detection, especially for cross-project deployment. Furthermore, although traditional text-mining-based method outperforms pattern-based method in prediction accuracy, the text features it uses are less intuitive than human-summarized patterns, which makes the prediction results hard to explain. To improve the accuracy of SATD prediction, especially for cross-project prediction, we propose a Convolutional Neural Network-- (CNN) based approach for classifying code comments as SATD or non-SATD. To improve the explainability of our model’s prediction results, we exploit the computational structure of CNNs to identify key phrases and patterns in code comments that are most relevant to SATD. We have conducted an extensive set of experiments with 62,566 code comments from 10 open-source projects and a user study with 150 comments of another three projects. Our evaluation confirms the effectiveness of different aspects of our approach and its superior performance, generalizability, adaptability, and explainability over current state-of-the-art traditional text-mining-based methods for SATD classification.

Collective transfer learning for defect prediction

Most software defect prediction approaches require extensive data from the project under test for training. However, for a new project, enough training data is often not available. It is therefore necessary to build a predictive model using the data from other relevant projects and then use the model to predict defects in the target project. Such direct cross-project prediction performance can still be improved, mainly due to the distribution differences between the source and target projects, as well as the uncertainty in determining which source project should be selected to train the model. In this work, we propose a collective training mechanism for defect prediction (CTDP), which includes two phases: source data expansion phase and adaptive weighting phase. CTDP makes the feature distributions of source and target projects similar to each other by transfer learning, and uses the particle swarm optimization algorithm to comprehensively consider the multiple source projects to predict the target project. The experiments on a total of 28 projects in 5 groups show that our method can improve the performance of cross-project defect prediction, achieving the sate-of-the-art results.

Whom are you going to call? determinants of @-mentions in Github discussions

Open Source Software (OSS) project success relies on crowd contributions. When an issue arises in pull-request based systems, @-mentions are used to call on people to task; previous studies have shown that @-mentions in discussions are associated with faster issue resolution. In most projects there may be many developers who could technically handle a variety of tasks. But OSS supports dynamic teams distributed across a wide variety of social and geographic backgrounds, as well as levels of involvement. It is, then, important to know whom to call on, i.e., who can be relied or trusted with important task-related duties, and why. In this paper, we sought to understand which observable socio-technical attributes of developers can be used to build good models of them being future @-mentioned in GitHub issues and pull request discussions. We built overall and project-specific predictive models of future @-mentions, in order to capture the determinants of @-mentions in each of two hundred GitHub projects, and to understand if and how those determinants differ between projects. We found that visibility, expertise, and productivity are associated with an increase in @-mentions, while responsiveness is not, in the presence of a number of control variables. Also, we find that though project-specific differences exist, the overall model can be used for cross-project prediction, indicating its GitHub-wide utility.

Too trivial to test? An inverse view on defect prediction to identify methods with low fault risk.

BackgroundTest resources are usually limited and therefore it is often not possible to completely test an application before a release. To cope with the problem of scarce resources, development teams can apply defect prediction to identify fault-prone code regions. However, defect prediction tends to low precision in cross-project prediction scenarios.AimsWe take an inverse view on defect prediction and aim to identify methods that can be deferred when testing because they contain hardly any faults due to their code being “trivial”. We expect that characteristics of such methods might be project-independent, so that our approach could improve cross-project predictions.MethodWe compute code metrics and apply association rule mining to create rules for identifying methods with low fault risk (LFR). We conduct an empirical study to assess our approach with six Java open-source projects containing precise fault data at the method level.ResultsOur results show that inverse defect prediction can identify approx. 32–44% of the methods of a project to have a LFR; on average, they are about six times less likely to contain a fault than other methods. In cross-project predictions with larger, more diversified training sets, identified methods are even 11 times less likely to contain a fault.ConclusionsInverse defect prediction supports the efficient allocation of test resources by identifying methods that can be treated with less priority in testing activities and is well applicable in cross-project prediction scenarios.