Software Reliability Testing Research Articles

Software Networks Nodes Impact Analysis of Complex Software Systems

目前一些研究利用复杂网络理论揭示了软件网络的特性，为人们从系统的角度了解软件的结构提供了方法.但这些研究的一些结论却与软件的实际表现有着较大的差异.分析了软件网络结点的特性，揭示了产生上述差距的部分原因；并提出一种加权软件网络模型，以更准确地描述软件网络结点间的依赖关系；在这个加权软件网络模型的基础上，分析了软件的实际依赖关系及几个统计特性；分析了各统计特性与软件网络结点影响的关系；进一步提出了软件网络关键结点的概念；同时，在分析各种结点影响的基础上，提出了4 个合理的假设；最后，通过对两款软件的实验来验证这4 个假设的有效性.;The complex network theory has been used to reveal some typical features of software networks. It provides a new way for us to understand the software structure from the system view. However, there exist some gaps between the theoretical results and the practical performance of software systems. This paper aims to reveal some essential causes for the above difference by analyzing the characteristics of software network nodes. This paper proposes a novel weighted network model to much more accurately describe the dependencies among the software network nodes. Based on this model, this paper analyzes the actual dependencies of the software network nodes and several statistical characteristics. Also, this paper further analyzes the relationships between these statistical characteristics and the nodes impact. Furthermore, this paper introduces the concept of the key node and four fundamental hypotheses. Finally, this paper verifies the effectiveness of the above four hypotheses by designing the experiments on two software systems. This study provides a guide to research in defects propagation, software reliability and software integration testing.

The Technology of Software Reliability Virtual Test Based on Software Conventional Failure Data

The technology of software reliability quantitative assessment (SRQA) is based on failure data collected in software reliability test or actual use. However, software reliability testing is a long test cycle and difficult to collect enough failure data, which limits SRQA in the actual project. A large number of software failure found from the software growth test cant be used because the process has nothing to do with the actual use or no record of failure time. In this paper, software reliability virtual testing technology based on software conventional failure data is presented. According to the internal data association between input space of software reliability test and failure data found in conventional software testing, a data matching algorithm is proposed to obtain possible failure time in software reliability testing by matching conventional failure data and the input space. Finally, the imitate engine control software is used as the experimental subject to verify the feasibility and effectiveness of the method.

Combined Reliability Testing Profile Model and Application for Embedded System

Embedded system is applied more and more extensively in the fields of national defense weapon equipment. With the increasing complexity of embedded systems, the combination of the hardware with software is more and closer. Because of the overall characteristic of hardware and software combination in embedded system, traditional reliability test can hardly accurately identify the reliability of embedded systems. The proposed combined reliability testing method, which combines software reliability test with system integrated environmental stress test to simulate the state of system running, provides the basis for the accurate validation and evaluation of reliability of embedded system.

Critical Operations Selecting Method

t is very important to select critical operations in software reliability testing and software safety testing. In this paper, a framework for selecting critical operations based on Analytic Hierarchy Process (AHP) is proposed. The hierarchies of goals, criteria and alternative programs are introduced separately. A preliminary application is practiced to show the methods application process and demonstrate the feasibility of this method, which can be taken as a reference for critical operations identification in engineering application.

Determining the Minimal Software Reliability Test Effort by Stratified Sampling

Software reliability testing provided the necessary guarantee for improving software reliability level and estimation. The traditional method for generating software reliability test cases could be seen as a simple random sampling, which was randomly choosen according to the operational profile defined by Musa. The stratified sampling was well known as a complex but more accurate sampling approach which could make the sampling results more accurate and less variance by dividing the population into several subsets and randomly sampling from each subset respectively. First, this paper introduced the traditional approach to determine the number of test cases in the discrete software reliability demonstration testing. Second, the background of the stratified sampling and necessary condition for the minimal test effort based on the stratified sampling were put forward. Third, the new method’s principle and details were analyzed and a sample was given to show the method’s effectiveness.Finally, the future work was discussed. DOI: http://dx.doi.org/10.11591/telkomnika.v11i8.3041

Software Reliability Test Methods Based on Component Composition and Markov Process

With the continuous improvement of the software quality requirements, more and more attentions are paid to the reliability of software systems. Most of the existing reliability models rely on the cumulative test time to characterize the reliability improvement, while ignoring the saturation effects and software module failure characteristic differences exist, making the test results to low confidence. In this paper, Markov description methods are utilized in reliability calculating process to improve the fictitious-excellence problem in current testing models. Finally, the method is applied to a flight control system software reliability testing. The results demonstrate the effectiveness of the method.

The Preprocessing Method for Software Reliability Failure Data

The software reliability failure data is the foundation of the software reliability’s quantitative evaluation based on the failure data, and it has an important influence on the accuracy of reliability evaluation. But there are always noises in the original software reliability failure data and make the reliability evaluation accuracy affected. This paper put forward the collecting method of reliability failure data and data preprocessing method including data cleaning and data analysis method, which based on the analysis of the importance and the source of failure data in the software reliability testing and the classification of software failure data. Finally through an example, it displayed the reduction of data noises and the promotion of data quality which produced by the preprocessing methods.

Minimum Test Case for Discrete-Type Software Reliability Testing by Stratified Sampling

The traditional approach for generating software reliability test cases according to operational profile (OP) can be regarded as a simple random sampling. The stratified sampling is a complex but more accurate sampling approach which can make the sampling results more accurate and less variance by dividing the population into several subsets and randomly sampling from each subset respectively. Firstly, this paper introduced the traditional method which can be used to determine the number of test cases in the discrete-type software reliability demonstration testing. Secondly, the necessary condition for the minimum test case based on the stratified sampling was studied. Then a new approach was proposed and its principle was analyzed for determining the minimum test case of the discrete-type software by using the stratified sampling. Finally, the future work was discussed.

Reliability Measurement of Digital Forensic Open Source Tools Using Fuzzy Logic

In this paper, a fuzzy logic based optimization technique has been adopted to accurately measure software reliability. In the literature, methods like, multiple linear regression, multivariate adaptive regression splines, back propagation trained neural network, dynamic evolving neuro-fuzzy inference system and TreeNet are available. Even a number of models are predicted for software reliability by ensembling the one or more of the above mentioned methods. It has been seen that software reliability cannot be uniformly treated by any one of the above mentioned method as the parameters of software reliability differs in weights depending upon the type of the application. Any application would need a defined recipe of ingredients like- interoperability, scalability, evolvability, pluggability, dynamicity, accuracy, security, cost optimality etc. One or more elements may prevail upon other and be considered as control variables in the optimization techniques applied. The concept calls for an optimization technique when weight can be changed dynamically in parameters depending on the conditionality of responses of the system or the user. This paper introduces the concept of shifting reliability based on dynamic decision making over the various control variables by changing their weight within time-in period in a real time system. Measurement of software reliability, in particular shifting reliability will make the software reliability prediction much more pragmatic in real time system. Digital forensic is in need for a suitable shifting reliability measurement technique. software reliability in case of DF tools governs the legal use. A number of open source tools are waiting for their professional run for want of reliability testing. This paper suggests software reliability testing, even for testing for shifting reliability. A method which has been adapted from simplex method for fuzzy variable linear programming problem has been applied to the particular case study. Typical formulation of fuzzy optimization depends on piece wise linear membership function. Linear member ship function introduces regions of no differentiability. In fact no deterministic method can be applied for fuzzy optimization, utilization of continuous differentiable membership function permits the use of gradient base methods. An expression for the gradient base decision degree function is available in literature. This formulation has been applied in a case study having a number of control variables using a suitable tool e.g. ProjectSixPap.

Software Reliability Test Based on Markov Usage Model

The reliability test for embedded software system is very difficult due to its complex structure and large scale. Markov usage model which describes the software usage with Markov process is widely used for statistical test. Software statistical test based on Markov usage model is an effective approach to the generation of test cases with high efficiency and the evaluation of software reliability in a quantitative way. This paper mainly focuses on the generation of Markov usage model of software system and the method of software reliability test based on it. Firstly, a method to build Markov usage model based on improved state transition matrix (STM), which is a table-based modeling language, is proposed. Then a software reliability test method including test case generation and test adequacy determination based on Markov usage model is presented. An improved Kullback discriminant is selected as the judgment criteria of convergence from the test chain to the usage chain in order to measure whether the testing process is sufficient. Finally, a Markov test cases generation tool MTCG is developed which implements the methods put forward in this paper. Experimental verification of test adequacy and efficiency is made through a comparison between the Markov usage model-based method and a completely random test method, the results of which show that software reliability test based on Markov usage model is approving, high-efficient and promising.

Scenario-Based Software Reliability Testing Profile for Autonomous Control System

Operational profile is often used in software reliability testing, but it is limited to non-obvious-operation software such as Autonomous Control System. After analyzing the autonomous control system and scenario technology, a scenario-based profile constructing method for software reliability testing is presented. Two levels of scenario-based profile in the paper are introduced: system level and software level, and the scenario-based profile could be obtained through mapping them. With the method, the testing data for software reliability testing could be generated.

Software Reliability Testing Data Generation Approach Based on a Mixture Model

Software Reliability Testing Data Generation Approach Based on a Mixture Model

Mathematical modeling of software reliability testing with imperfect debugging

Software reliability testing is concerned with the quantitative relationship between software testing and software reliability. Our previous work develops a mathematically rigorous modeling framework for software reliability testing. However the modeling framework is confined to the case of perfect debugging, where detected defects are removed without introducing new defects. In this paper the modeling framework is extended to the case of imperfect debugging and two models are proposed. In the first model it is assumed that debugging is imperfect and may make the number of remaining defects reduce by one, remain intact, or increase by one. In the second model it is assumed that when the number of remaining defects reaches the upper bound, the probability that the number of remaining defects is increased by one by debugging is zero. The expected behaviors of the cumulative number of observed failures and the number of remaining defects in the first model show that the software testing process may induce a linear or nonlinear dynamic system, depending on the relationship between the probability of debugging introducing a new defect and that of debugging removing a detected defect. The second-order behaviors of the first model also show that in the case of imperfect debugging, although there may be unbiased estimator for the initial number of defects remaining in the software under test, the cumulative number of observed failures and the current number of remaining defects are not sufficient for precisely estimating the initial number of remaining defects. This is because the variance of the unbiased estimator approaches a non-zero constant as the software testing process proceeds. This may be treated as an intrinsic principle of uncertainty for software testing. The expected behaviors of the cumulative number of observed failures and the number of remaining defects in the second model show that the software testing process may induce a nonlinear dynamic system. However theoretical analysis and simulation results show that, if defects are more often removed from than introduced into the software under test, the expected behaviors of the two models tend to coincide with each other as the upper bound of the number of remaining defects approaches infinity.

Research of combined forecasting model for software reliability

Based on grey model, harmonic analysis and time series analysis theory, the fault data sets obtained from software reliability testing phase were analyzed, and then a combined forecasting model for software reliability was proposed. Referring to practical data, the realization method was presented. The experimental result indicates that the combined model has higher precision of prediction and better adaptability, compared with other single time series prediction models.

A mathematical modeling framework for software reliability testing†

Software reliability testing refers to various software testing activities that are driven to achieve a quantitative reliability goal given a priori or lead to a quantitative reliability assessment for the software under test. In this paper we develop a modeling framework for the software reliability testing process, comprising a simplifying model and a generalized model. In both models the software testing action selection process and the defect removal mechanism are explicitly described. Both the discrete-time domain and the continuous-time domain are involved. The generalized model is more accurate or realistic than the simplifying model since the former avoids the assumption that defects are equally detectable and the assumption that defects are removed upon being detected. However simulation examples show that the simplifying model really captures some of essential features of the software testing process after a short initial testing stage. The modeling framework is practically realistic, mathematically rigorous, and quantitatively precise. It demonstrates that the relationship between software testing and delivered software reliability, which was poor understood, can well be formulated and quantified. Rigorous examinations show that several common assumptions adopted in software reliability modeling, including the independence assumption, the exponentiality assumption, and the NHPP assumption, are theoretically false in general. This paper sets a good starting point to further formalize and quantify the software testing process and its relation to delivered software reliability.

Practical stopping criteria for validating safety-critical software by estimating impartial reliability

In this research, we investigate stopping rules for software testing and propose two stopping rules from the aspect of software reliability testing based on the impartial reliability model. The impartial reliability difference (IRD-MP) rule considers the difference between the impartial transition-probability reliabilities estimated for both software developer and consumers at their predetermined prior information levels. The empirical–impartial reliability difference (EIRD-MP) rule suggests stopping a software test when the computed empirical transition reliability is tending to its estimated impartial transition reliability. To insure the high-standard requirement for safety-critical software, both rules take the maximum probability (MP) of untested paths into account. The main benefit of using the IRD-MP stopping rule is that it may incorporate extra confidence information based on the previous acquisition processes between a software developer and its user. In addition, a demonstrated application example of field-use safety-critical software has shown that the EIRD-MP stopping rule provides a good criterion when either the tested software is a new product or no other prior judgment is available from the customer and developer.

Random observations of marked Cox processes. Time insensitive functionals

This article has as an objective to analyze the behavior of multivariate, delayed stationary marked Cox processes with mutually dependent components about some critical levels. The original problems arise in biology, computer engineering, computer networks, software reliability testing, and stock market. The process under investigation can describe the evolution of stocks, indexes, cancer cells, proliferation of bacteria, inventories, military conflicts, in which the process is being observed only restrictively, i.e., at some specified random epochs. Given this (sometimes limited) information, it is possible to “predict” the “first passage time” when the process crosses the critical level (or levels) and see the main probability characteristics (such as distribution) of the components of the process upon the first passage time that occurs at one of the observation times. Among various questions to arise, one is how to choose the frequency of observations to provide more accurate information but not to “exceed the budget” (a quint essence of reliability analysis). On the other hand, there are ways to scrutinize the available information, as to making it analytically more “time sensitive,” without any additional efforts, which is one of the primary goals of this investigation. We formalize and provide preliminary results for the work to be continued in [J. Math. Anal. Appl. 293 (2004) 14–27] (about time sensitive functionals) and give closed-form expressions. Many examples from science and technology are presented.

DETERMINING THE OPTIMAL RELEASE TIME FOR SOFTWARE IN THE GEOMETRIC POISSON RELIABILITY MODEL

The geometric Poisson model for software reliability and debugging was introduced by P. B. Moranda to deal with the situation when one is collecting grouped data on faults (say for example the number of faults detected weekly or monthly) in a new piece of software. In this model one assumes that the number of faults or bugs detected in subsequent intervals of equal length of time in testing a piece of software are independent Poisson random variables with respective Poisson parameters which form a decreasing geometric sequence. An important problem in software reliability testing is to decide when to stop testing and release software for commercial use. In this paper we determine the optimal time T* to release software under the geometric Poisson model with a cost structure for repairing or fixing faults by minimizing the expected cost. We consider both the situation where one is interested in the software performing well until some fixed software life-cycle time t, and for the case where we want the software to function well for an additional τ units of (mission) time after release. Solutions are obtained both when parameter estimates are available as well as in the Bayesian framework when prior information on these parameters is at hand.

An alternative reliability evaluation of non‐homogeneous Poisson process models for software reliability

Non‐homogeneous Poisson process (NHPP) models, as the stochastic models frequently employed in reliability engineering, have been successfully used in the reliability study of software systems. The software reliability based on NHPP models was proposed by Goel and Okumoto. In general, the software reliability will increase along with the correction of the software errors. This idea gives rise to a hypothesis: extending the time on the software reliability test could result in obtaining a higher reliability. Nevertheless, the scheme was found to be discrepant for some NHPP models, which are utilized in the analysis. As this often occurs in practice, the reliability outcome could differ from its appropriate estimation when the test procedures are intentionally terminated prior to the end of the required testing time span. An “above average software reliability” is proposed in this paper to accomplish the inconsistency. Under the investigation of “average software reliability”, a higher software reliability can be achieved as long as the testing time increases. Also, in this paper, an optimal software release policy is proposed to explore the issue of compromising the expenses on software development and software reliability improvement.

Trends in reliability and test strategies

Software fails because it contains faults. If we could be certain that our attempts to fix a fault had been effective and that we had introduced no new faults, we would know that we had improved the software's reliability. Unfortunately, no single testing method can be trusted to give accurate results in every circumstance. There are at least two areas of uncertainty in testing. First, we cannot predict when a failure will occur as a result of choosing an input that cannot currently be processed correctly. Second, we do not know what effect fixing a fault will have on the software's reliability. Most of today's reliability models consider the software system to be a black box, using indirect measures of failure by making strong assumptions about the test cases. White-box testing measures, which are direct measures, are generally not used in software-reliability models. The assumption is that the more the program is covered, the more likely that the software is reliable. Hence we must develop new testing strategies. For example, we do not understand very well the notion of stress as it applies to software; what effect the dependence between versions of the same application has on reliability; the relationship between software reliability and white-box testing measures; and how to achieve the ultra-high reliability levels obtained through formal verification and fault-tolerance methods. These are just some of the areas for future research into improved testing strategies.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>