Concurrent streams in Markov chain usage models for statistical testing of complex systems

CenUMs – concurrency enhanced usage models for statistical testing of complex systems with concurrent streams of use

In this paper a general framework is presented for testing time-critical systems and software. The main focus is to derive a state-oriented statistical usage model from a set of usage scenarios in order to automatically generate test cases. We describe a methodology that was developed within the European IST project MaTeLo to ease testing by combining the advantages of formal description techniques, namely MSC, UML and TTCN-3. In the first step of our approach, a MCUM (Markov Chain Usage Model) is constructed. This model represents the formal basis for deriving TTCN-3 test case descriptions to perform executable specification-based tests for the system under test (SUT). In order to be independent of the chosen specification technique, i.e. MSC or UML sequence diagrams, we have defined an XML-based representation format for the MCUM, called MCML (Markov Chain Markup Language). This format represents a common interface between various tools of the MaTeLo approach. All steps in our methodology do also support the testing of QoS (Quality of Service) requirements that are annoted in a UML profile standard notation.KeywordsSoftware TestingAutomatic Test GenerationMarkov Chain Usage ModelUML Sequence DiagramMSCTTCN-3QoS

The TestUS framework offers a range of techniques to obtain a TTCN-3 test suite starting from UML 2.0 requirement definitions. Use case diagrams that contain functional and non-functional requirements are first transformed to a Markov Chain usage model (MCUM). Probability annotations of MCUM state transitions enable the generation of TTCN-3 test cases that reflect the expected usage patterns of system users. Because compiling the associated TTCN-3 test suite can take quite a long time for a realistic SUT (System under Test) we decided to map the MCUM directly into the executable test suite without generating test cases in advance. Test cases and the evaluation of test verdicts are interpreted on-the-fly during executing the test suite. We proved the concept by testing an existing DECT communication system. The compilation time for deriving an executable TTCN-3 test suite was reduced to only 15 minutes and one can interpret as many test cases as one likes on-the-fly.

Markov chain usage models support test planning, test automation, and analysis of test results. In practice, transition probabilities for Markov chain usage models are often specified using a cycle of assigning, verifying, and revising specific values for individual transition probabilities. For large systems, such an approach can be difficult for a variety of reasons. We describe an improved approach that represents transition probabilities by explicitly preserving the information concerning test objectives and the relationships between transition probabilities in a format that is easy to maintain and easy to analyze. Using mathematical programming, transition probabilities are automatically generated to satisfy test management objectives and constraints. A more mathematical treatment of this approach is given in References [1] (Poore JH, Walton GH, Whittaker JA. A constraint-based approach to the representation of software usage models. Information and SoftwareTechnology 2000; at press) and [2] (Walton GH. Generating transition probabilities for Markov chain usage models. PhD Thesis, University of Tennessee, Knoxville, TN, May 1995.). In contrast, this paper is targeted at the software engineering practitioner, software development manager, and test manager. This paper also adds to the published literature on Markov chain usage modeling and model-based testing by describing and illustrating an iterative process for usage model development and optimization and by providing some recommendations for embedding model-based testing activities within an incremental development process. Copyright © 2000 John Wiley & Sons, Ltd.

It is time consuming for lecturers to manually compile, run and verify students' programming assignments. Moreover, they also need to define test cases for different programming exercises in order to assess students' code. This paper presents a swarm inspired automatic test case generation for Online Programming Assessment or Judge (UOJ) system. The UOJ is built with the purpose to assist lecturers in evaluating codes submitted from students and generate test cases for every programming problem automatically. It also helps students to practice and enhance their programming skills. In this study, the mechanism of automatic test case generation was explored using Particle Swarm Optimization (PSO) algorithm and new fitness function is proposed to determine the optimal solution for each problem. Experiments were set up to benchmark between manual and automatic PSO test case generation results to evaluate the efficiency of the proposed method. The preliminary results are satisfactory for automatic test case generation using swarm inspired PSO algorithm in C++ programming assignments.

Model-driven testing based on Markov chain usage models (MCUM) is an established method to address testing issues. It is not possible, however, to describe by means of MCUMs the timing of stimuli and time dependencies between inputs and outputs of a system. Additional concepts and information is needed. In this paper it is presented how Timed Usage Models (TUM) can be used to solve this issue. Concepts for stimuli and responses are introduced, that comprise that something must happen before a time interval, after a time interval or within a time interval. It is presented how this can be described by TUMs. TUMs are enhanced MCUMs that allow the usage of distributions of time. The computations for TUMs are based on semi-Markov processes and are therefore not restricted to discrete steps in time. Therefore, indicators and metrics for the test planning and management can be derived that take into account time. Test cases can be derived that reflect variability in inputs and, additionally, variability in timing of inputs. Complex real time systems require a test method that provides the possibility to handle the effect of timing and variability in timing of inputs to the system.

When a population is too large for study, as is the case for all possible uses of a software system, a statistically correct sample must be drawn as a basis for inferences about the population. In statistical testing of software based on a Markov chain usage model, the rich body of analytical results available for Markov chains provides numerous insights that can be used in test planning. Further, the connection between Markov chains and operations research techniques permits a Markov usage model to be expressed as a system of constraints, with mathematical programming used to generate the optimal model for a particular objective function. Since a software usage model is based on the specification, all analyses may be performed early in the development cycle and used as a quantitative basis for management decisions. These techniques have been reduced to engineering practice and used in large projects by IBM, Ericsson, all branches of the US military, and others. In this paper, statistical experiments, Markov models, and optimization techniques are shown to provide a sound theoretical and practical basis far quantifying the reliability of software.

We compare manual testing without any automation performed by a tester at a software company with model-based testing (MBT) performed by a tester at a research center. The system under test (SUT), of which two different versions were tested by each of the two testers, is a professionally developed web-based data collection system that now is in use. The two testers tested the same versions, had identical testing goals (to detect defects), had access to the same resources, but used different processes (i.e. manual without any automation vs. model-based with automatic test case generation and automatic test case execution). The testers did not interact with each other. We compare the effectiveness (issues found) and efficiency (effort spent) of the two approaches. The results show, for example, that manual testing required less preparation time and that its test coverage was somewhat uneven. In contrast, MBT required more preparation time, was more systematic, and detected more issues. While the manual approach detected more inconsistencies between specified and actual text labels, MBT detected more functional issues. This is reflected in the severity score summary which was about 60% higher for MBT than Manual.

In software reliability testing based on Markov chain usage model, how to assign the migrating probability between states of the Markov chain usage model is still a problem yet to be solved. In order to deal with the problems such as the sample data for estimating the migrating probability being small and uncertain, a migrating probability assignment method based on a three-parameter interval number is proposed to build the Markov chain usage model. The method for generating test cases is also given. An example shows in this paper, the software usage model based on the three-parameter interval number is closer to the actual situation, and the software reliability testing efficiency is improved.

Timed usage models (TUMs) represent a model-based statistical approach for system testing of real-time embedded systems. They enable an automatic test case generation and the calculation of parameters that aid the test process. However, a classical TUM only supports sequential uses of the system under test (SUT). It is not capable of dealing with concurrency, which is required for state of the art real-time embedded systems. Therefore, we introduce TUMs with parallel regions. They also allow automatic test case generation, which is carried out similarly to classical TUMs. But, the semi-Markov process (SMP) that is usually used for analysis is not suitable here. We apply Markov renewal theory and define an SMP with parallel regions, which is used to calculate parameters. We validated our analytical approach by simulations.

Automatic test case generation is a potentially effective technique for program testing, but it still suffers from the lack of appropriate tool support. Our research presented in this paper mainly focuses on the developing of a tool for automatic test case generation based on formal specifications. We take advantage of the Liu’s decompositional test case generation method and put forward a set of algorithms for automatically generating test cases based on various data types. A supporting tool on the application of the approaches is presented. The tool can generate test cases according to the users’ given test conditions, and the result shows that our tool can produce test cases that satisfy most kinds of test conditions.Keywordsautomatic test case generationspecificationSOFLdecompositional methodfunctional scenario

Automatic test case generation has been a hotspot and a difficult problem in the software testing, Accurately and efficiently generate test cases can improve the efficiency of software testing. Java programs have many characteristics such as encapsulation, inheritance, polymorphism and so on, it is convenient for software design and development, but to bring automated testing some difficulties. This article on the Java program of automatic test case generation method is studied and presents a framework for automatic generation of test cases. With this framework, test case suite will be generated quickly and accurately. Experimental results show that automatic Java test case generation framework can quickly and accurately generate test cases , reduce labor costs and improve efficiency.

Software Testing plays a non-negligible role in minimizing the Software development cost. One of the most important trends for this objective is the use of Model Based Testing (MBT). MBT consists in automatically generating validation tests from a customized model describing some functional aspects of the System Under Test (SUT). However, a dedicated test model should be created. This process is time and man power consuming and hence, it is not widely adopted in the industry. In our work, we present a methodology to facilitate the use of MBT in companies by using project artefacts to automatically create a test model based on the usage of the SUT (what we call a usage model). In this paper, we focus on transforming UML Sequence Diagrams into a usage model. This transformation will be done based on the notion of context, which is the history of previous events on the SUT to be able to factorize and get a valid usage model. This notion is very important when two events are managed differently by the SUT depending on the set of previous inputs. A use case of an ATM (Automated Teller Machine) is presented to describe the need and the associated issues. The work presented in this article is supported by the French project Clarity.

Testing mobile application refers to the process of ensuring the quality and functionality of the software applications implemented for mobile phones. It is a fundamental step to test mobile applications specially before being released to the market. Load testing is a kind of software testing which is carried out to test the performance of mobile applications under different workloads. However, most of mobile apps are implemented within a bounded resource, budget, and time due to the market pressure. This extremely influences the quality of the mobile applications development industry. The automatic test case generation is the key factor for efficient and cost-effective software verification process. Model-based load testing represents both performance aspects and functional aspects of mobile applications by using modeling languages like unified modeling language (UML). Then, adopting the resulting models to automatically design test cases that will be used for load testing. This paper proposes an automatic model-based test case generation approach for mobile applications load testing. The proposed approach lessens testing time and satisfies fast time-to market constraint of mobile apps. Additionally, the proposed approach validates the mobile applications functionalities as well as the performance requirements.

Statistical testing based on a Markov chain usage model, as a rigorous testing method, has been around for more than two decades. Through the comprehensive application of statistical science to the testing of software, it provides audit trails of evidence to support correctness arguments for a software-intensive system as well as a decision that the system is of requisite quality for its intended use. This paper reports a real-world case study in which we applied standard statistical testing practices to the phone flag copy testing problem at the site of our industrial collaborator, and presents our solution from problem formalization, usage modeling and model analysis to test case generation and analysis. Our results helped evaluate the coverage of a heuristically generated test suite, and shed light on what other test cases to craft/generate to improve test coverage and adequacy, as well as reliability estimates both at the arc (usage event) level and at the path (system) level.