Software Fault Research Articles

A neighborhood rough sets-based ensemble method, with application to software fault prediction

Software fault prediction (SFP) aims to detect fault-prone software modules, which is beneficial for allocating software testing resources and improving software quality. Recently, ensemble learning(EL)-based SFP methods have attracted much attention. Although many EL algorithms have been applied to SFP, they are still insufficient to generate multiple accurate and diverse base learners. Therefore, this paper presents a multi-modal EL algorithm (called NRSEL) based on neighborhood rough sets. In NRSEL, the technique of neighborhood approximate reduct (NAR) is used to implement the perturbation of attribute space and the bootstrap sampling technique is used to implement the perturbation of sample space. As a novel technique for the perturbation of attribute space, NAR stems from the concept of approximate reduct in rough sets. We also consider the application of NRSEL to SFP, and employ a hybrid scheme (called SMOTE-NRSEL) to handle the problem of imbalanced data in SFP. We compare SMOTE-NRSEL with existing EL algorithms using 20 public datasets. Experimental results indicate that SMOTE-NRSEL is effective for SFP. Compared with the baseline algorithms, on average, SMOTE-NRSEL improves the AUC, F1-score, and MCC by 3.09%, 3.18%, and 7.5%, respectively. Moreover, the results of three statistical tests (including the paired t-test, Friedman test, and Nemenyi test) indicate that SMOTE-NRSEL is significantly better than the baseline algorithms in most cases. This paper shows that NAR is a good choice for the perturbation of attribute space. With the help of NAR and the multi-modal perturbation strategy based on it, SMOTE-NRSEL can generate accurate and diverse base learners. The code is available at https://github.com/jiangfeng0278/NRSEL.

BLESS Behavior Correctness Proof as ConvincingVerification Artifact

Safety-critical cyber-physical systems require evidence they are indeed safe. In practice, such evidence is results of system tests. Unfortunately, tests can only demonstrate the presence of software errors, not their absence, and can practically cover a tiny fraction of system state space. Valiant efforts to formally verify program correctness have either been excruciatingly difficult (theorem proving) or incomplete (static analysis, model checking, SAT or SMT solving). The BLESS Methodology was created specifically for engineers in industry to formally verify software controlling machines. The BLESS Methodology transforms programs that control machines, annotated with assertions to form proof outlines, into deductive proofs that every possible program execution will conform to its specification. To the extent that cyber-physical system specifications have been validated to express system safety and performance, a deductive proof can be a convincing argument to a person of program correctness. This paper uses a simple safety-critical system to argue that behavior correctness proof under the BLESS Methodology is a convincing verification artifact in addition to customary testing.

Quantification of effects of degraded hardware on the execution of self-healing software using Continuous Time Markov Chain and Reliability

Embedded systems and Systems on Chips (SoC) are widely used in many applications. The tasks are to be executed in a time-bound manner for critical applications. The latency and worst-case delays shall not exceed the acceptable limits. These systems are designed using software/firmware and hardware components that are tightly coupled. The state of hardware and software/firmware plays a major role in determining the performance and dependability of the system. This paper studies the effects of the degraded state of the hardware on the execution of self-healing software and the system. While hardware and software design teams aim to ensure predictability, it is impossible to guarantee it completely due to the randomness of hardware failure events and resident errors in software. Usage of the system for safety critical systems requires these uncertainties to be quantified for design assurance and certification. Therefore, we propose a Continuous Time Markov Chain (CTMC) based approach to characterize the effects of degraded hardware on software execution. Reliability, which is one of the dependability metrics, is also quantified. The proposed approach is validated using data collected from one of the software-intensive complex hardware equipment of a fighter aircraft.

User Interface Bug Classification Model Using ML and NLP Techniques: A Comparative Performance Analysis of ML Models

Analyzing user interface (UI) bugs is an important step taken by testers and developers to assess the usability of the software product. UI bug classification helps in understanding the nature and cause of software failures. Manually classifying thousands of bugs is an inefficient and tedious job for both testers and developers. Objective of this research is to develop a classification model for the User Interface (UI) related bugs using supervised Machine Learning (ML) algorithms and Natural Language Processing (NLP) techniques. Also, to assess the effect of different sampling and feature vectorization techniques on the performance of ML algorithms. Classification is based upon ‘Summary’ feature of the bug report and utilizes six classifiers i.e., Gaussian Naïve Bayes (GNB), Multinomial Naïve Bayes (MNB), Logistic Regression (LR), Support Vector Machines (SVM), Random Forest (RF) and Gradient Boosting (GB). Dataset obtained is vectored using two vectorization techniques of NLP i.e., Bag of Words (BoW) and Term Frequency-Inverse Document Frequency (TF-IDF). ML models are trained after vectorization and data balancing. The models ' hyperparameter tuning (HT) has also been done using the grid search approach to improve their efficacy. This work provides a comparative performance analysis of ML techniques using Accuracy, Precision, Recall and F1 Score. Performance results showed that a UI bug classification model can be built by training a tuned SVM classifier using TF-IDF and SMOTE (Synthetic Minority Oversampling Techniques). SVM classifier provided the highest performance measure with Accuracy: 0.88, Precision: 0.86, Recall: 0.85 and F1: 0.85. Result also inferred that the performance of ML algorithms with TF-IDF is better than BoW in most cases. This work provides classification of bugs that are related to only the user interface. Also, the effect of two different feature extraction techniques and sampling techniques on algorithms were analyzed, adding novelty to the research work.

Understanding the impact of data gaps on long-term offshore wind resource estimates

Abstract. In the context of a wind farm project, the wind resource is assessed to predict the power output and the optimal positioning of wind turbines. This requires taking wind measurements on the site of interest and extrapolating these to the long term using so-called “measure, correlate, and predict” (MCP) methods. Sensor, power supply, and software failures are common phenomena. These disruptions cause gaps in the measured data, which can especially be long in offshore measurement campaigns due to harsh weather conditions causing system failures and preventing servicing and redeployment. The present study investigates the effect of measurement data gaps on long-term offshore wind estimates by analyzing the bias they introduce in the parameters commonly used for wind resource assessment. Furthermore, it aims to show how filling the gaps can mitigate their effect. To achieve this, we perform investigations for three offshore sites in Europe with 2 years of concurrent measurements. We use reanalysis data and various MCP methods to fill gaps in the measured data and extrapolate these data to the long term. Current standards demand high data availability (80 % or 90 %) for wind measurement campaigns, so we expect that the effect of missing data on the uncertainty in long-term extrapolations is of the same order of magnitude as other uncertainty components such as the measurement uncertainty or the inter-annual variability. Nevertheless, our results show that the effects of gaps are considerably smaller than the other uncertainty components. For instance, gaps of 180 d cause an average deviation of the long-term mean wind speed of less than 0.04 m s−1 and a 95th percentile deviation of less than 0.075 m s−1 for all tested sites. Due to the low impact of gaps, gap filling does not have the potential to significantly reduce the uncertainty in the long-term extrapolation.

Prioritization of Software Bugs Using Entropy‐Based Measures

ABSTRACTOpen‐source software is evolved through the active participation of users. In general, a user request for bug fixing, the addition of new features, and feature enhancements. Due to this, the software repositories are increasing day by day at an enormous rate. Additionally, user distinct requests add uncertainty and irregularity to the reported bug data. The performance of machine learning algorithms drastically gets influenced by the inappropriate handling of uncertainty and irregularity in the bug data. Researchers have used machine learning techniques for assigning priority to the bug without considering the uncertainty and irregularity in reported bug data. In order to capture the uncertainty and irregularity in the reported bug data, the summary entropy–based measure in combination with the severity and summary weight is considered in this study to predict the priority of bugs in the open‐source projects. Accordingly, the classifiers are build using these measures for different machine learning techniques, namely, k‐nearest neighbor (KNN), naïve Bayes (NB), J48, random forest (RF), condensed nearest neighbor (CNN), multinomial logistic regression (MLR), decision tree (DT), deep learning (DL), and neural network (NNet) for bug priority prediction This research aims to systematically analyze the summary entropy–based machine learning classifiers from three aspects: type of machine learning technique considered, estimation of various performance measures: Accuracy, Precision, Recall, and F‐measure and through existing model comparison. The experimental analysis is carried out using three open‐source projects, namely, Eclipse, Mozilla, and OpenOffice. Out of 145 cases (29 products X 5 priority levels), the J48, RF, DT, CNN, NNet, DL, MLR, and KNN techniques give the maximum F‐measure for 46, 35, 28, 11, 15, 4, 3, and 1 cases, respectively. The result shows that the proposed summary entropy–based approach using different machine learning techniques performs better than without entropy‐based approach and also entropy‐based approach improves the Accuracy and F‐measure as compared with the existing approaches. It can be concluded that the classifier build using summary entropy measure significantly improves the machine learning algorithms' performance with appropriate handling of uncertainty and irregularity. Moreover, the proposed summary entropy–based classifiers outperform the existing models available in the literature for predicting bug priority.

Accounting for Natural Disasters Using Risk of Information Security

Objective: The harm that can be done to an information technology system is referred to as "information security risk" (IS). Data breaches, administrative actions, financial expenses, and reputational harm are just a few of the many possible dangers that fall under the broad category of information systems risks. IS hazards include hostile assaults, spam, viruses, human error, hardware and software malfunctions, and natural disasters like floods, hurricanes, and fires. Methods: Key security controls for applications are identified, evaluated, and implemented through security risk assessments. Another priority is to avoid security gaps and defects in software. This article analyzes documented historical research on natural disaster accounting. Based on the results of 35 publications on general topics, economic history, and accounting, 11 papers were selected and reviewed. By analyzing the scattered literature, emerging themes, investigated disasters, investigated time periods, and main contributions of published research were identified. Result: To compare the predictive capabilities of the machine learning models and validate the landslide, flood, and wildfire hazard maps, the area under the curve (AUC) was calculated. For the landslide, flood, and wildfire hazard maps, the RF model provided the highest AUC values (0.81, 0.85, and 0.94, respectively).

Software defect density prediction using grey system theory and fuzzy logic

Software defect density prediction using grey system theory and fuzzy logic

A Brief Analysis of the Progress and Trends in Software Defect Prediction Methods

Software defect detection is particularly important for modern society, as it is a crucial step in ensuring the quality and reliability of software systems. With the emergence of artificial intelligence (AI), research in software defect detection has evolved from traditional methods to more complex approaches that utilize deep learning and large language models (LLMs). The advent of LLMs has fundamentally changed the paradigm of software development and defect detection, bringing new challenges and confusion to the field of software defect prediction research. To address these issues, we compare software defect detection methods based on traditional techniques, deep learning approaches, and LLMs through a literature review. We analyze the changes brought about by the introduction of LLMs to software development and propose new insights. Additionally, we examine the progress and trends in software defect prediction to provide inspiration for subsequent research.

An adaptive synthetic sampling and batch generation-oriented hybrid approach for addressing class imbalance problem in software defect prediction

An adaptive synthetic sampling and batch generation-oriented hybrid approach for addressing class imbalance problem in software defect prediction

Multi objective binary Rao feature optimization for software defect prediction using machine learning models

Multi objective binary Rao feature optimization for software defect prediction using machine learning models

Reinforcing defect prediction: a reinforcement learning approach to mitigate class imbalance in software defect prediction

Reinforcing defect prediction: a reinforcement learning approach to mitigate class imbalance in software defect prediction

Enhancing software defect prediction models using metaheuristics with a learning to rank approach

In today’s world, software systems are extensively used across industries like medicine, commerce, education, entertainment, and industry. The development process includes design, development, and testing, where software testing is vital for quality assurance. Manual testing is laborious and resource-heavy, especially in large projects. Researchers are investigating methods to predict software defects by analyzing source code metrics, with Machine Learning techniques showing promise. The Machine learning models typically employs three approaches to predict defects: presence, number, and density. One cutting-edge approach in such models is LTR (Learning To Rank), which considers the order and values of predictions, aiding enterprises in efficiently allocating resources to more defective components. This article aims to enhance the performance of 5 machine learning models by leveraging 5 metaheuristics to optimize their hyperparameters. The study was conducted using 28 common datasets from the literature, with the FPA metric as the evaluation criteria. The results demonstrate that the proposed method achieved a 13% increase compared to state-of-the-art models with Random Forest, achieving an average FPA of 0.84 across the datasets.

Generation of algebraic data type values using evolutionary algorithms

Automatic data generation is a key component of automated software testing. Random generation of test input data can uncover some bugs in software, but its effectiveness decreases when those inputs must satisfy complex properties in order to be meaningful. In this work, we study an evolutionary approach to generate values that can be encoded as algebraic data types plus additional properties. First, the approach is illustrated with the generation of sorted lists. Then, we generalize the technique to arbitrary algebraic data type definitions. Finally, we consider the problem of constrained data types where the data must satisfy some nontrivial property, using the well-known example of red-black trees for our experiments. This example will allow us to introduce the main principles of evolutionary algorithms and how these principles can be applied to obtain valid, nontrivial samples of a given data structure. Our experiments have revealed that this evolutionary approach is able to improve diversity, and increase the size of valid generated values with respect to simple random sampling techniques.

A Systematic Exploration of Mutation‐Based Fault Localization Formulae

ABSTRACTFault localization (FL) aims to automatically find the location of bugs in a software program. In the family of FL approaches, spectrum‐based fault localization (SBFL) is the most widely used and has been extensively studied, which computes the suspicious scores of a code element to be buggy via test coverage information. Mutation‐based fault localization (MBFL) further analyses the relationship between mutants and test results. Although MBFL involves more information, it also faces the following limitations. (1) To precisely evaluate suspicious sources, SBFL approaches proposed dozens of formulae based on coverage, while only a few of them have been adopted by MBFL. (2) The current MBFL approaches are based on the assumption that the capability in localizing bugs of each mutant is the same, so they assign the same weight to the mutants that change the test outputs and consider the mutant from the same code element equivalent. In this paper, we intend to enrich the MBFL family by the following two approaches. First, we collect 25 typical SBFL formulae and transform them into MBFL versions. Second, we propose new assumptions that the mutants should have different weights in computing suspicious sources and propose BLMu, a novel MBFL approach that treats mutants differently. We also propose novel metrics for MBFL by considering the high cost of mutation analysis. We perform large‐scale experiments by evaluating all the MBFL approaches against 395 real‐world Java bugs of the Defects4J benchmark. Our evaluation results reveal that BLMu demonstrates a substantial improvement over both MUSE and Metallaxis, the most popular MBFL approaches, at both the statement level and the method level. Specifically, in terms of Top‐1, BLMu improves by 106% at the statement level and 69% at the method level. When compared with other types of FL approaches, MBFL outperforms typical SBFL approaches while still far behind the state‐of‐the‐art learning‐based FL approaches.

An ensemble model for addressing class imbalance and class overlap in software defect prediction

An ensemble model for addressing class imbalance and class overlap in software defect prediction

Visual software defect prediction method based on improved recurrent criss-cross residual network

PurposeThis study aims to solve the problems of large training sample size, low data sample quality, low efficiency of the currently used classical model, high computational complexity of the existing concern mechanism, and high graphics processing unit (GPU) occupancy in the current visualization software defect prediction, proposing a method for software defect prediction termed recurrent criss-cross attention for weighted activation functions of recurrent SE-ResNet (RCCA-WRSR). First, following code visualization, the activation functions of the SE-ResNet model are replaced with a weighted combination of Relu and Elu to enhance model convergence. Additionally, an SE module is added before it to filter feature information, eliminating low-weight features to generate an improved residual network model, WRSR. To focus more on contextual information and establish connections between a pixel and those not in the same cross-path, the visualized red as integer, green as integer, blue as integer images are inputted into a model incorporating a fused RCCA module for defect prediction.Design/methodology/approachSoftware defect prediction based on code visualization is a new software defect prediction technology, which mainly realizes the defect prediction of code by visualizing code as image, and then applying attention mechanism to extract the features of image. However, the challenges of current visualization software defect prediction mainly include the large training sample size and low sample quality of the data, and the classical models used today are not efficient, and the existing attention mechanisms have high computational complexity and high GPU occupancy.FindingsExperimental evaluation using ten open-source Java data sets from PROMISE and five existing methods demonstrates that the proposed approach achieves an F-measure value of 0.637 in predicting 16 cross-version projects, representing a 6.1% improvement.Originality/valueRCCA-WRSR is a new visual software defect prediction based on recurrent criss-cross attention and improved residual network. This method effectively enhances the performance of software defect prediction.

FunFuzz : Greybox Fuzzing with Function Significance

Greybox fuzzing is dedicated to revealing software bugs by maximizing code coverage. Concentrating on code coverage, greybox fuzzing effectively exposes bugs in real-world programs by continuously executing the program under test (PUT) with the test inputs generated from initial seeds, making it a popular software testing technique. Although powerful, the effectiveness of greybox fuzzing can be restricted in some cases. Ignoring the significant degrees of executed functions, traditional greybox fuzzing usually fails to identify significant seeds that execute more significant functions, and thus may assign similar energy to significant and trivial seeds when conducting power scheduling. As a result, the effectiveness of greybox fuzzing can be degraded due to wasting too much energy on trivial seeds. In this paper, we introduce function significance (FS) to measure the significant degrees of functions. Our key insight is that the influential functions that connect to many other functions are significant to greybox fuzzing as they provide more probabilities to reach previously unexplored code regions. To quantify FS, we conduct influence analysis upon the call graphs extracted from the PUTs to obtain the centrality values of function nodes. With FS as the significance measurement, we further propose FunFuzz , an FS-aware greybox fuzzing technique, to optimize significant seeds and tackle the aforementioned restriction. To this end, FunFuzz dynamically tracks the functions executed by a seed during fuzzing, and computes the significance score for the seed by accumulating the FS values of the functions executed by it. Based on the computed FS values, FunFuzz then takes an estimation-based power scheduling to assign more (or less) energy to seeds that achieve over-estimated (or under-estimated) significance scores. Specifically, the seed energy is adjusted by multiplying with a scale factor computed regarding the ratio of the actual significance score achieved by executing the seed and the estimated significance score predicted by a linear model constructed on-the-fly. To evaluate FunFuzz , we prototype it on top of AFL++ and conduct experiments with 15 programs, of which 10 are from common real-world projects and five are from Magma, and compare it to seven popular fuzzers. The experimental results obtained through fuzzing exceeding 40,800 CPU hours show that: (1) In terms of covering code, FunFuzz outperforms AFL++ by achieving 0.1% \(\sim\) 18.4% more region coverage on 13 out of 15 targets. (2) In terms of finding bugs, FunFuzz unveils 114 unique crashes and 25 Magma bugs (which are derived from CVEs) in 20 trials of 24-hour fuzzing, which are the most compared to the competitor fuzzers and include 32 crashes and 1 Magma bug that the other fuzzers fail to discover. Besides the experiments focusing on code coverage and bug finding, we evaluate the key components of FunFuzz , namely the FS-centered estimation-based power scheduling and the lazy FS computation mechanism. The extensive evaluation not only suggests FunFuzz ’s superiority in code coverage and bug finding, but also demonstrates the effectiveness of the two components.

HetFL: Heterogeneous Graph-Based Software Fault Localization

HetFL: Heterogeneous Graph-Based Software Fault Localization

The effect of data complexity on classifier performance

The research area of Software Defect Prediction (SDP) is both extensive and popular, and is often treated as a classification problem. Improvements in classification, pre-processing and tuning techniques, (together with many factors which can influence model performance) have encouraged this trend. However, no matter the effort in these areas, it seems that there is a ceiling in the performance of the classification models used in SDP. In this paper, the issue of classifier performance is analysed from the perspective of data complexity. Specifically, data complexity metrics are calculated using the Unified Bug Dataset, a collection of well-known SDP datasets, and then checked for correlation with the defect prediction performance of machine learning classifiers (in particular, the classifiers C5.0, Naive Bayes, Artificial Neural Networks, Random Forests, and Support Vector Machines). In this work, different domains of competence and incompetence are identified for the classifiers. Similarities and differences between the classifiers and the performance metrics are found and the Unified Bug Dataset is analysed from the perspective of data complexity. We found that certain classifiers work best in certain situations and that all data complexity metrics can be problematic, although certain classifiers did excel in some situations.