Software Failure Data Research Articles

Staffing Level and Cost Analyses for Software Debugging Activities Through Rate-Based Simulation Approaches

Research in the field of software reliability, dedicated to the analysis of software failure processes, is quite diverse. In recent years, several attractive rate-based simulation approaches have been proposed. Thus far, it appears that most existing simulation approaches do not take into account the number of available debuggers (or developers). In practice, the number of debuggers will be carefully controlled. If all debuggers are busy, they may not address newly detected faults for some time. Furthermore, practical experience shows that fault-removal time is not negligible, and the number of removed faults generally lags behind the total number of detected faults, because fault detection activities continue as faults are being removed. Given these facts, we apply the queueing theory to describe and explain possible debugging behavior during software development. Two simulation procedures are developed based on G/G/infin, and G/G/m queueing models, respectively. The proposed methods will be illustrated using real software failure data. The analysis conducted through the proposed framework can help project managers assess the appropriate staffing level for the debugging team from the standpoint of performance, and cost-effectiveness.

Fitting software reliability growth curves using nonparametric regression methods

A simple and effective method of assessing the reliability of a piece of software is to plot the cumulative number of failures observed during testing, N ( x ) , against time x . Since no software is ever completely free of errors, be it careless minor oversights or the results of serious design problems, it is therefore expected that with prolonged and systematic testing, N ( x ) will increase with x . Since the 1970s, there have been many models, aptly named Software Reliability Growth Models (SRGMs) which have been proposed to fit software failure data to the curve m ( x ) = E ( N ( x ) ) . Unfortunately, due to the complexity of the software development processes, which include the possibility of imperfect debugging and introduction of new faults into the system, many of these SRGMs are very complex and standard estimation procedures such as Maximum Likelihood estimation (MLE) fails to estimate correctly, if at all, the parameters of these models. In this paper, we investigate the potential benefits of using Nonparametric Regression (NPR) methods to fit SRGMs. In addition, we will also develop methods based on Stein two-stage and modified two-stage sequential procedures to find a fixed-width confidence interval for the estimator of m ( x ) . Finally, numerical examples based on real software failure data will be presented to illustrate the techniques developed and compare the results with some parametric SRGMs.

Common Trends in Software Fault and Failure Data

The benefits of the analysis of software faults and failures have been widely recognized. However, detailed studies based on empirical data are rare. In this paper, we analyze the fault and failure data from two large, real-world case studies. Specifically, we explore: 1) the localization of faults that lead to individual software failures and 2) the distribution of different types of software faults. Our results show that individual failures are often caused by multiple faults spread throughout the system. This observation is important since it does not support several heuristics and assumptions used in the past. In addition, it clearly indicates that finding and fixing faults that lead to such software failures in large, complex systems are often difficult and challenging tasks despite the advances in software development. Our results also show that requirement faults, coding faults, and data problems are the three most common types of software faults. Furthermore, these results show that contrary to the popular belief, a significant percentage of failures are linked to late life cycle activities. Another important aspect of our work is that we conduct intra- and interproject comparisons, as well as comparisons with the findings from related studies. The consistency of several main trends across software systems in this paper and several related research efforts suggests that these trends are likely to be intrinsic characteristics of software faults and failures rather than project specific.

Unified framework for developing testing effort dependent software reliability growth models

Several software reliability growth models (SRGMs) have been presented in the literature in the last three decades. These SRGMs take into account different testing environment depending on size and efficiency of testing team, type of components and faults, design of test cases, software architecture etc. The plethora of models makes the model selection an uphill task. Recently, some authors have tried to develop a unifying approach so as to capture different growth curves, thus easing the model selection process. The work in this area done so far relates the fault removal process to the testing/execution time and does not consider the consumption pattern of testing resources such as CPU time, manpower and number of executed test cases. More realistic modeling techniques can result if the reliability growth process is studied with respect to the amount of expended testing efforts. In this paper, we propose a unified framework for testing effort dependent software reliability growth models incorporating imperfect debugging and error generation. The proposed framework represents the realistic case of time delays between the different stages of fault removal process i.e Failure Observation/Fault Detection and Fault Removal/Correction processes. The Convolution of probability distribution functions have been used to characterize time differentiation between these two processes. Several existing and new effort dependent models have been derived by using different types of distribution functions. We have also provided data analysis based on the actual software failure data sets for some of the models discussed and proposed in the paper.

Quasi-Renewal Time-Delay Fault-Removal Consideration in Software Reliability Modeling

Software reliability growth models based on a nonhomogeneous Poisson process (NHPP) have been considered as one of the most effective among various models since they integrate the information regarding testing and debugging activities observed in the testing phase into the software reliability model. Although most of the existing NHPP models have progressed successfully in their estimation/prediction accuracies by modifying the assumptions with regard to the testing process, these models were developed based on the instantaneous fault-removal assumption. In this paper, we develop a generalized NHPP software reliability model considering quasi-renewal time-delay fault removal. The quasi-renewal process is employed to estimate the time delay due to identifying and prioritizing the detected faults before actual code change in the software reliability assessment. Model formulation based on the quasi-renewal time-delay assumption is provided, and the generalized mean value function (MVF) for the proposed model is derived by using the method of steps. The general solution of the MVFs for the proposed model is also obtained for some specific existing models. The numerical examples, based on a software failure data set, show that the consideration of quasi-renewal time-delay fault-removal assumption improves the descriptive properties of the model, which means that the length of time delay is getting decreased since testers and programmers adapt themselves to the working environment as testing and debugging activities are in progress.

Empirical Bayesian Software Reliability Model Using Rayleigh Distribution

An Empirical Bayesian software reliability model is considered in this paper. It is assumed that the times between failures follow Rayleigh distribution with the parameter in the failure rate function with stochastically decreasing order on successive failure time intervals. The reasoning for the assumption on the parameter is that the intention of the software tester to improve the software quality by the correction of each failure. With the Bayesian approach, the predictive distribution has been arrived at by combining Rayleigh time between failures and gamma prior distribution for the parameter. The expected time between failure measures has been obtained. The posterior distribution of the parameter and its mean has been deduced. For the parameter estimation, Maximum likelihood estimation (MLE) method has been adopted. The proposed model has been applied to two sets of actual software failure data and it has been observed that the predicted failure times as per the proposed model are closer to the actual failure times. The predicted failure times based on Littlewood-Verall (LV) model is also computed. Sum of Square Errors (SSE) criteria has been used for comparing the actual time between failures and predicted time between failures based on proposed model and LV model.

Gompertz software reliability model: Estimation algorithm and empirical validation

Gompertz curve has been used to estimate the number of residual faults in testing phases of software development, especially by Japanese software development companies. Since the Gompertz curve is a deterministic function, the curve cannot be applied to estimating software reliability which is the probability that software system does not fail in a prefixed time period. In this article, we propose a stochastic model called the Gompertz software reliability model based on non-homogeneous Poisson processes. The proposed model can be derived from the statistical theory of extreme-value, and has a similar asymptotic property to the deterministic Gompertz curve. Also, we develop an EM algorithm to determine the model parameters effectively. In numerical examples with software failure data observed in real software development projects, we evaluate performance of the Gompertz software reliability model in terms of reliability assessment and failure prediction.

IMPROVING SOFTWARE RELIABILITY MODELING USING MACHINE LEARNING TECHNIQUES

It is well-known that software failure data often contain noise, making the reliability estimation problematic. In particular, the kind of data noise inherent in software failure data is biased. There is no upper bound for the value of a noisy data point, but there is a lower bound of zero. This may lead to over-optimistic estimation of the reliability when using maximum likelihood or least square methods based on standard software reliability models. We attempt to address this problem by modeling software failure data using machine learning techniques such as support vector machine regression and generalized additive models, which have mechanisms that are capable of dealing with data noise. We then analyse the results from machine learning modeling, and compare them to that of some generalized linear modeling techniques that are equivalent to standard software reliability models. The validity of the machine learning modeling of noisy software failure data is evaluated through this comparison.

On the Trend of Remaining Software Defect Estimation

Software defects play a key role in software reliability, and the number of remaining defects is one of most important software reliability indexes. Observing the trend of the number of remaining defects during the testing process can provide very useful information on the software reliability. However, the number of remaining defects is not known and has to be estimated. Therefore, it is important to study the trend of the remaining software defect estimation (RSDE). In this paper, the concept of RSDE curves is proposed. An RSDE curve describes the dynamic behavior of RSDE as software testing proceeds. Generally, RSDE changes over time and displays two typical patterns: 1) single mode and 2) multiple modes. This behavior is due to the different characteristics of the testing process, i.e., testing under a single testing profile or multiple testing profiles with various change points. By studying the trend of the estimated number of remaining software defects, RSDE curves can provide further insights into the software testing process. In particular, in this paper, the Goel-Okumoto model is used to estimate this number on actual software failure data, and some properties of RSDE are derived. In addition, we discuss some theoretical and application issues of the RSDE curves. The concept of the proposed RSDE curves is independent of the selected model. The methods and development discussed in this paper can be applied to any valid estimation model to develop and study its corresponding RSDE curve. Finally, we discuss several possible areas for future research.

A study of software reliability growth from the perspective of learning effects

For the last three decades, reliability growth has been studied to predict software reliability in the testing/debugging phase. Most of the models developed were based on the non-homogeneous Poisson process (NHPP), and S-shaped type or exponential-shaped type of behavior is usually assumed. Unfortunately, such models may be suitable only for particular software failure data, thus narrowing the scope of applications. Therefore, from the perspective of learning effects that can influence the process of software reliability growth, we considered that efficiency in testing/debugging concerned not only the ability of the testing staff but also the learning effect that comes from inspecting the testing/debugging codes. The proposed approach can reasonably describe the S-shaped and exponential-shaped types of behaviors simultaneously, and the results in the experiment show good fit. A comparative analysis to evaluate the effectiveness for the proposed model and other software failure models was also performed. Finally, an optimal software release policy is suggested.

FEEDBACK CONTROL SYSTEM FOR SOFTWARE

Feedback control systems are used in many walks of life, including automobiles, airplanes, and nuclear reactors. These are all physical systems, albeit with a considerable does of software. It occurred to us that there is no reason that feedback control systems could not be applied to the software process, specifically dealing with reliability analysis, test, and prediction. Thus, we constructed a model of such a system and analyzed whether feedback control, in the form of error signals representing deviations from desired behavior, could bring observed behavior in conformance with specifications. To conduct the experiment, we used NASA Space Shuttle software failure data and analyzed the feedback when no faults were removed versus removing faults. In making this evaluation two software reliability models were used: the Musa Logarithmic Model and the Schneidewind Model. In general, feedback based on fault removal allowed the software reliability process to provide more accurate predictions and, hence, finer control over the process.

An Assessment of Testing-Effort Dependent Software Reliability Growth Models

Over the last several decades, many Software Reliability Growth Models (SRGM) have been developed to greatly facilitate engineers and managers in tracking and measuring the growth of reliability as software is being improved. However, some research work indicates that the delayed S-shaped model may not fit the software failure data well when the testing-effort spent on fault detection is not a constant. Thus, in this paper, we first review the logistic testing-effort function that can be used to describe the amount of testing-effort spent on software testing. We describe how to incorporate the logistic testing-effort function into both exponential-type, and S-shaped software reliability models. The proposed models are also discussed under both ideal, and imperfect debugging conditions. Results from applying the proposed models to two real data sets are discussed, and compared with other traditional SRGM to show that the proposed models can give better predictions, and that the logistic testing-effort function is suitable for incorporating directly into both exponential-type, and S-shaped software reliability models

RELIABILITY ANALYSIS OF PROJECT AND PRODUCT TYPE SOFTWARE IN OPERATIONAL PHASE INCORPORATING THE EFFECT OF FAULT REMOVAL EFFICIENCY

Since the early 1970's numerous Software Reliability Growth Models (SRGM) have been proposed in the literature to estimate the software reliability measures such as the remaining number of faults, failure rate and reliability growth during the testing phase. These models are applied to the software testing data collected during the testing phase and then are often used to predict the software failures in operational phase. In practice simulating mirror image of the diverse testing environment representative of the operational environment is difficult in practice and hence the simulated testing environment during the testing phase may not be similar to the conditions that exist in the operational phase. During testing phase testing is performed under a controlled environment whereas during the operational phase failure phenomenon depends on the operational environment and usage of software. Therefore an SRGM developed for the testing phase is not suitable for estimating the reliability growth during the operational phase. In this paper, we propose a generalized Software Reliability Growth Model, which can be used to estimate number of faults during the testing phase and can be easily extended to the operation phase. In the testing phase, it is appropriate to estimate the reliability growth with respect to the amount of testing resources spent on testing whereas in the operational phase the amount of effort to be spent on removing a fault reported by a user is fixed by the developer. The number of failures detected and hence the reliability growth during the user phase depends on the usage of software. The proposed model appropriately incorporates these changes. Further we categorize the software into two-categories- (a) project and (b) product type software. Appropriate usage functions are linked to both project and product type software. To describe the fault removal phenomenon, imperfect debugging environment is incorporated into the model building. The paper highlights an interdisciplinary mathematical modeling approach in Software Reliability Engineering and Marketing. The proposed model is validated for both phases using the software failure data sets obtained from different sources. Model describes the failure phenomenon for these data sets fairly.

Flexible software reliability growth model with testing effort dependent learning process

A lot of importance has been attached to the testing phase of the Software Development Life Cycle (SDLC). It is during this phase it is checked whether the software product meets user requirements or not. Any discrepancies that are identified are removed. But testing needs to be monitored to increase its effectiveness. Software Reliability Growth Models (SRGMs) that specify mathematical relationships between the failure phenomenon and time have proved useful. SRGMs that include factors that affect failure process are more realistic and useful. Software fault detection and removal during the testing phase of SDLC depend on how testing resources (test cases, manpower and time) are used and also on previously identified faults. With this motivation a Non-Homogeneous Poisson Process (NHPP) based SRGM is proposed in this paper which is flexible enough to describe various software failure/reliability curves. Both testing efforts and time dependent fault detection rate (FDR) are considered for software reliability modeling. The time lag between fault identification and removal has also been depicted. The applicability of our model is shown by validating it on software failure data sets obtained from different real software development projects. The comparisons with established models in terms of goodness of fit, the Akaike Information Criterion (AIC), Mean of Squared Errors (MSE), etc. have been presented.

POSSIBILITIES AND LIMITATIONS OF APPLYING SOFTWARE RELIABILITY GROWTH MODELS TO SAFETY-CRITICAL SOFTWARE

It is generally known that software reliability growth models such as the Jelinski-Moranda model and the Goel-Okumoto's non-homogeneous Poisson process (NHPP) model cannot be applied to safety-critical software due to a lack of software failure data. In this paper, by applying two of the most widely known software reliability growth models to sample software failure data, we demonstrate the possibility of using the software reliability growth models to prove the high reliability of safety-critical software. The high sensitivity of a piece of software's reliability to software failure data, as well as a lack of sufficient software failure data, is also identified as a possible limitation when applying the software reliability growth models to safety-critical software.

The Astringency of the GP Algorithm for Forecasting Software Failure Data Series

The forecasting of software failure data series by Genetic Programming (GP) can be realized without any assumptions before modeling. This discovery has transformed traditional statistical modeling methods as well as improved consistency for model applicability. The individuals' different characteristics during the evolution of generations, which are randomly changeable, are treated as Markov random processes. This paper also proposes that a GP algorithm with optimal individuals reserved strategy is the best solution to this problem, and therefore the adaptive individuals finally will be evolved. This will allow practical applications in software reliability modeling analysis and forecasting for failure behaviors. Moreover it can verify the feasibility and availability of the GP algorithm, which is applied to software failure data series forecasting on a theoretical basis. The results show that the GP algorithm is the best solution for software failure behaviors in a variety of disciplines.

Non-parametric Estimation for NHPP Software Reliability Models

The non-homogeneous Poisson process (NHPP) model is a very important class of software reliability models and is widely used in software reliability engineering. NHPPs are characterized by their intensity functions. In the literature it is usually assumed that the functional forms of the intensity functions are known and only some parameters in intensity functions are unknown. The parametric statistical methods can then be applied to estimate or to test the unknown reliability models. However, in realistic situations it is often the case that the functional form of the failure intensity is not very well known or is completely unknown. In this case we have to use functional (non-parametric) estimation methods. The non-parametric techniques do not require any preliminary assumption on the software models and then can reduce the parameter modeling bias. The existing non-parametric methods in the statistical methods are usually not applicable to software reliability data. In this paper we construct some non-parametric methods to estimate the failure intensity function of the NHPP model, taking the particularities of the software failure data into consideration.

A General Software Reliability Growth Model with Testing Effort Dependent Learning Process

Large number of software reliability growth models (SRGMs) exists in the literature, still more are being proposed as no single SRGM can be deemed to be the best. Better models are those which can capture the variability in the software failure curves obtained during the testing phase. These models are called flexible models and in this paper an attempt has been made to propose a flexible model with more realistic assumptions. The model takes into account the testing effort and knowledge gained by the testing team during testing. The applicability of the proposed model is shown by validating it on software failure data sets obtained from different real software development projects. Comparisons with established models in terms of goodness of fit, the Akaike Information Criterion, mean squared errors have been presented.

A New Methodology for Predicting Software Reliability in the Random Field Environments

This paper presents a new methodology for predicting software reliability in the field environment. Our work differs from some existing models that assume a constant failure detection rate for software testing and field operation environments, as this new methodology considers the random environmental effects on software reliability. Assuming that all the random effects of the field environments can be captured by a unit-free environmental factor, eta, which is modeled as a random-distributed variable, we establish a generalized random field environment (RFE) software reliability model that covers both the testing phase and the operating phase in the software development cycle. Based on the generalized RFE model, two specific random field environmental reliability models are proposed for predicting software reliability in the field environment: the gamma-RFE model, and the beta-RFE model. A set of software failure data from a telecommunication software application is used to illustrate the proposed models, both of which provide very good fittings to the software failures in both testing and operation environments. This new methodology provides a viable way to model the user environments, and further makes adjustments to the reliability prediction for similar software products. Based on the generalized software reliability model, further work may include the development of software cost models and the optimum software release policies under random field environments

Neural-network-based approaches for software reliability estimation using dynamic weighted combinational models

Software reliability is the probability of failure-free software operation for a specified period of time in a specified environment. During the last three decades, many software reliability growth models (SRGMs) have been proposed and analyzed for measuring software reliability growth. SRGMs are mathematical models that represent software failures as a random process and can be used to evaluate development status during testing. However, most of SRGMs depend on some assumptions or distributions. In this paper, we propose an artificial neural-network-based approach for software reliability estimation and modeling. We first explain the neural networks from the mathematical viewpoints of software reliability modeling. We will show how to apply neural network to predict software reliability by designing different elements of neural networks. Furthermore, we will use the neural network approach to build a dynamic weighted combinational model (DWCM). The applicability of proposed model is demonstrated through real software failure data sets. The results obtained from the experiments show that the proposed model has a fairly accurate prediction capability.