Statistical Usage Testing Research Articles

Statistical Usage Testing at Different Levels of Testing

Statistical Usage Testing at Different Levels of Testing

Using Statistical Usage Testing in Conjunction with Other Black Box Testing Techniques

Cleanroom methodology is a scrupulous incremental software development approach16 for the development of zero defect and high-reliability software using box structure specification, statistical quality control and certification.2 Statistical Usage Testing (SUT) technique is the testing technique defined it Cleanroom software engineering. It based on developing the usage models and then performing statistical tests on the usage models.15 The paper shows the usage of SUT in conjunction with other black box testing techniques. Other types of testing can be performed along with SUT depending on the requirement. Use of other testing techniques can be essential to show specific scenarios of use or to attain full usage model coverage with reduced test cases.17 The paper also presents the effectiveness of applying SUT in conjunction with other black box testing techniques by using various test cases. Other black box testing techniques used include Equivalence Class Partitioning (ECP), Boundary Value Analysis (BVA), Cause Effect Graphing (CEG), Use Case Testing (UCT) and Orthogonal Array Testing (OATS).

Testing Apache OpenOffice Writer using statistical usage testing technique

Statistical usage testing (SUT) technique is the testing technique defined it cleanroom software engineering. It based on developing the usage models and then performing statistical tests on the usage models (Runeson and Wohlin, IEEE Trans Softw Eng 20:494–499, 1994). The failure data from testing is then used in a software reliability model to certify the reliability of the software. The main purpose of SUT is to test a software product from a user’s point of view. Hence, usage models are designed and then test cases are developed from the models (Prowell et al., Cleanroom software engineering technology and process, 1999). The paper tests Apache OpenOffice Writer using SUT Technique. The paper makes use of two software RAM Commander and ITEM toolkit for testing. In this paper, Markov chains are used to represent the usage models. The development of Usage Markov Chain, the Testing Markov Chain and incorporation of failure data is done using the same software. The paper also discusses the analytical results of applying SUT and its impact on software reliability.

An Enhancement to Cleanroom Software Engineering using csUnit Tool

Clean-room software engineering is a technical and managerial process for the development of high-quality software for zero defects with certified reliability, which combines the correctness, verifications and statistical usage testing to certify the software fitness. This paper describes the clean-room software engineering along with its technology, processes, tools support and its comparison with other traditional approaches, advantages and disadvantages. In this paper is presenting the detail description of Clean-room Software Engineering and Object Oriented Technology and also the experimental study of collaborative engineering and mutation testing performed through csUnit tool integrated with Visual studio 2008, which have been introduced to enhanced or improved Clean-room Software Engineering..

Generic method for statistical testing of parallel programs based on task trees

This paper deals with a particular class of parallel programs, which are based on task trees. The main objective of this paper was to adapt the generic method for statistical testing of sequential programs (GMST-SP) for this class of parallel programs, such that adapted method (GMST) can treat a family of task trees rather than just a single task tree, and that it can respect various evolutions of individual task trees. In this paper, we compare GMST with the adapted exhaustive testing method (ET) and with the previously adapted statistical usage testing method (SUT), based on experimentally measured testing effort and path coverage. GMST and SUT both have better deep path coverage than ET. SUT requires less testing effort than GMST and ET, but its deep path coverage decreases with the number of tasks. Finally, GMST has advantage over SUT because it provides constant mean level of deep path coverage, which can be regulated by the required testing quality. Key words: Multicores, parallel programming, parallel programs, structural testing, statistical testing.

A Polyhedron Approach to Calculate Probability Distributions for Markov Chain Usage Models

Statistical usage testing of hardware/software systems is based in the main on a Markov chain usage model. This kind of model represents the expected use of the system by a usage profile, i.e. appropriate probability values that are attached to the state transitions. In this paper we present a constraint-based polyhedron approach to calculate the probability distribution for the MCUM from a given set of usage constraints. Comparing the computed probability distributions of our polyhedron approach with the maximum entropy technique shows that our result is much closer to the intented constraint semantics. Using the polyhedron method, customer profiles can be calculated so that they reflect the intended system usage of different customers or customer types much better. In order to demonstrate the applicability of our approach, workflow testing of a complex RIS/PACS system in the medical domain was carried through and yielded very promising results.

Test case generation for the task tree type of architecture

Context Emerging multicores and clusters of multicores that may operate in parallel have set a new challenge – development of massively parallel software composed of thousands of loosely coupled or even completely independent threads/processes, such as MapReduce and Java 3.0 workers, or Erlang processes, respectively. Testing and verification is a critical phase in the development of such software products. Objective Generating test cases based on operational profiles and certifying declared operational reliability figure of the given software product is a well-established process for the sequential type of software. This paper proposes an adaptation of that process for a class of massively parallel software – large-scale task trees. Method The proposed method uses statistical usage testing and operational reliability estimation based on operational profiles and novel test suite quality indicators, namely the percentage of different task trees and the percentage of different paths. Results As an example, the proposed method is applied to operational reliability certification of a parallel software infrastructure named the TaskTreeExecutor. The paper proposes an algorithm for generating random task trees to enable that application. Test runs in the experiments involved hundreds and thousands of Win32/Linux threads thus demonstrating scalability of the proposed approach. For practitioners, the most useful result presented is the method for determining the number of task trees and the number of paths, which are needed to certify the given operational reliability of a software product. The practitioners may also use the proposed coverage metrics to measure the quality of automatically generated test suite. Conclusion This paper provides a useful solution for the test case generation that enables the operational reliability certification process for a class of massively parallel software called the large-scale task trees. The usefulness of this solution was demonstrated by a case study – operational reliability certification of the real parallel software product.

Impartial evaluation in software reliability practice

In the past decade, the size scale of software systems and their technical complexity has become much more complicated. Accordingly, quality assessment of software applications has been intensively investigated lately. Among popular software quality metrics, software reliability has been proven to be one of the most useful indices in evaluating software applications. In the literature, statistical usage testing has been widely shown to be effective in estimating software reliability. Essentially, it first transfers the practical operations of a software system into a usage model, which then forms a basis for performing statistical testing and analyzing software reliability. This research investigates the extension of statistical usage testing by considering any prior information or prejudgment on software quality before performing a validation test, and propose the derivation of impartial reliability evaluation that is fair for both the software producer and consumer (end user). A numerical demonstration of validating the correctness of hyper links on a web site via the proposed computation is illustrated and discussed. The suggested mechanism with some prior information will converge much more quickly than other similar reliability models. In addition, the proposed framework also provides the flexibility of taking the practical prejudgment into account.

Assessing the quality of Web-based applications via navigational structures

We study the link validity of a Web site's navigational structure to enhance Web quality. Our approach employs the principle of statistical usage testing to develop an efficient and effective testing mechanism. Some advantages of our approach include generating test scripts systematically, providing coverage metrics, and executing hyperlinks only once.

Towards integration of use case modelling and usage-based testing

This paper focuses on usage modelling as a basis for both requirements engineering (RE) and testing, and investigates the possibility of integrating the two disciplines of use case modelling (UCM) and statistical usage testing (SUT). The paper investigates the conceptual framework for each discipline, and discusses how they can be integrated to form a seamless transition from requirements models to test models for reliability certification. Two approaches for such an integration are identified: integration by model transformation and integration by model extension. The integration approaches are illustrated through an example, and advantages as well as disadvantages of each approach are discussed. Based on the fact that the two disciplines have models with common information and similar structure, it is argued that an integration may result in coordination benefits and reduced costs. Several areas of further research are identified.

SOFTWARE RELIABILITY ESTIMATIONS THROUGH USAGE ANALYSIS OF SPECIFICATIONS AND DESIGNS

This paper presents a method proposal for estimation of software reliability before the implementation phase. The method is based upon that a formal specification technique is used and that it is possible to develop a tool performing dynamic analysis, i.e., locating semantic faults in the design. The analysis is performed with both applying a usage profile as input as well as doing a full analysis, i.e., locate all faults that the tool can find. The tool must provide failure data in terms of time since the last failure was detected. The mapping of the dynamic failures to the failures encountered during statistical usage testing and operation is discussed. The method can be applied either on the software specification or as a step in the development process by applying it on the software design. The proposed method allows for software reliability estimations that can be used both as a quality indicator, and for planning and controlling resources, development times, etc. at an early stage in the development of software systems.

Adopting Cleanroom software engineering with a phased approach

Cleanroom software engineering is a theory-based, team-oriented engineering process for developing very high quality software under statistical quality control. The Cleanroom process combines formal methods of object-based box structure specification and design, function-theoretic correctness verification, and statistical usage testing for reliability certification to produce software approaching zero defects. Management of the Cleanroom process is based on a life cycle of development and certification of a pipeline of user-function increments that accumulate into the final product. Teams in IBM and other organizations that use the process are achieving remarkable quality results with high productivity. A phased implementation of the Cleanroom process enables quality and productivity improvements with an increased control of change. An introductory implementation involves the application of Cleanroom principles without the full formality of the process; full implementation involves the comprehensive use of formal Cleanroom methods; and advanced implementation optimizes the process through additional formal methods, reuse, and continual improvement. The AOEXPERT/MVS™ project, the largest IBM Cleanroom effort to date, successfully applied an introductory level of implementation. This paper presents both the implementation strategy and the project results.