Test Suite Execution Research Articles

An algorithm for capturing variables dependences in test suites

The use of dynamic dependence analysis spans several areas of software research including software testing, debugging, fault localization, and security. Many of the techniques devised in these areas require the execution of large test suites in order to generate profiles that capture the dependences that occurred between given types of program elements. When the aim is to capture direct and indirect dependences between finely granular elements, such as statements and variables, this process becomes highly costly due to: (1) the large number of elements, and (2) the transitive nature of the indirect dependence relationship. The focus of this paper is on computing dynamic dependences between variables, i.e., dynamic information flow analysis or DIFA. First, because the problem of tracking dependences between statements, i.e., dynamic slicing, has already been addressed by numerous researchers. Second, because DIFA is a more difficult problem given that the number of variables in a program is unbounded. We present an algorithm that, in the context of test suite execution, leverages the already computed dependences to efficiently compute subsequent dependences within the same or later test runs. To evaluate our proposed algorithm, we conducted an empirical comparative study that contrasted it, with respect to efficiency, to three other algorithms: (1) a naïve basic algorithm, (2) a memoization based algorithm that does not leverage computed dependences from previous test runs, and (3) an algorithm that uses reduced ordered binary decision diagrams ( roBDDs) to maintain and manage dependences. The results indicated that our new DIFA algorithm performed considerably better in terms of both runtime and memory consumption.

Specification of Conditions for Error Diagnostics

This paper describes the basic concepts of error diagnostics and an associated rule system whose application helps to identify potential hardware/software locations of errors which caused a failure observed while executing tests of embedded systems. Further on, an overview over the “Avionics Smoke Detection System” is given, where the main algorithms have been applied. The methods, techniques and tools described here rely on the preceding investigations performed with respect to signal flow and the observation of causal chains in distributed test benches during test suite executions. These results have been elaborated in the KATO project within the German aerospace research programme LUFO III.

Simulated time for host‐based testing with TTCN‐3

AbstractPrior to testing embedded software in a target environment, it is usually tested in a host environment used for developing the software. When a system is tested in a host environment, its real‐time behaviour is affected by the use of simulators, emulation and monitoring. In this paper, the authors provide a semantics for host‐based testing with simulated time and propose a simulated‐time solution for distributed testing with TTCN‐3, which is a standardized language for specifying and executing test suites. The paper also presents the application of testing with simulated time to two real‐life systems. Copyright © 2007 John Wiley & Sons, Ltd.

World-class product certification using Erlang

It is now ten years ago since the decision was made to apply the functional programming language Erlang to real production projects at Ericsson. In late 1995, development on the Open Telecom Platform (OTP) started, and in mid 1996 the AXD 301 project became the first user of OTP. The AXD 301 Multi-service Switch was released in October 1998, and later became "the heart of ENGINE", Ericsson's leading Voice over Packet solution.In those early days of Erlang programming, high-level tools for development and testing were not really available, and programmers used mainly the Emacs editor and the Erlang shell. Still, anecdotal evidence suggested a 4-10x productivity increase compared to mainstream programming techniques.Through the years, significant progress has been made, especially in the area of automated testing of Erlang programs. The OTP team designed an Erlang-based test suite execution environment, were developers can easily write their own automated test suites, and now performs nightly builds where more than one thousand test cases are executed on ten different platforms. OTP designers can view the outcome in web-based test reports as they come to work the next day. Each corrected bug results in a new test case that is incorporated into the ever-growing test suite. Thus, this world-class middleware is certified to telecom-class quality without a dedicated test team!The AXD 301 project uses OTP's test environment, and executes more than 10,000 automated test cases before each major release. Designers and testers compose their own test suites, and the designers carry out function tests with little or no help from the Integration and Certification team. Each test case can be run both in a simulated environment on the designer's workstation and in the test lab on real hardware. In order to provide stimuli to the system, the testers often design their own traffic generators in Erlang.To analyze the faults that occur, Erlang offers an increasing wealth of debugging options. Beyond the symbolic error messages, which are often sufficient to locate the fault, Erlang developers are able to dynamically turn on tracing on message passing, scheduling events, garbage collections, selected function calls, etc.This paper demonstrates how Erlang's declarative syntax and pattern matching provide an outstanding environment for test suite development.

Interoperability test suite derivation for communication protocols

This paper addresses the problem of interoperability testing for communication protocols. We develop a coherent framework of interoperability testing in which the notions of interoperability, interoperability testing, interoperability test case and interoperability test architecture are interrelated and a systematic interoperability test suite derivation method based on the framework. The approach to interoperability testing is illustrated with the example of the ATM Signaling Protocol. To demonstrate practicality of the approach, we implemented executable test suites derived by the method on a commercial ATM test platform, applied them for interoperability testing of various ATM equipment and analyzed the testing results.

Finite state machine based formal methods in protocol conformance testing: from theory to implementation

Abstract Recent evolution of communication networks has led toward increasingly complex communication protocols to interconnect heterogeneous systems. In order for such systems to function properly, communication protocols require formal methodologies for verification, implementation and testing. In this paper, the use of formal methods for protocol conformance testing is presented. Four major formal techniques for conformance test generation reported in the literature are compared: transition tours, distinguishing sequences, characterizing sequences and unique input/output sequences. The implementation of each approach as a test generation methodology is illustrated. The result is the capability to generate abstract test suites that require a minimum number of steps to execute while maintaining complete coverage of the state transitions (or test purposes) of the protocol. The impact of the formal methodologies on the practical aspects of conformance testing, including the test system implementation and automatic test script generation, is discussed. The experience based on testing various protocol implementations from multiple manufacturers at AT&T Bell Laboratories is described. The protocols tested include X.25 and ISDN Q.921 and Q.931 for basic and primary rate interfaces. In order to standardize various efforts on conformance testing, the ISO and the CCITT have developed principles to specify abstract and executable test suites, and methodologies to design testbed architectures. The relationship between such standards and the formal methods for test generation is explored.

An improved algorithm for multibridge network reduction : N. S. Fard and Xitong Zheng. IEEE Trans. Reliab.R-36, 54 (1987)

In software, there are the error cases that are anticipated at specification and design time, those encountered at development and testing time, and those that were never anticipated before happening in production. Is it possible to learn from the anticipated errors during design to analyze and improve the resilience against the unanticipated ones in production?In this paper, we aim at analyzing and improving how software handles unanticipated exceptions. The first objective is to set up contracts about exception handling and a way to assess them automatically. The second one is to improve the resilience capabilities of software by transforming the source code.We devise an algorithm, called short-circuit testing, which injects exceptions during test suite execution so as to simulate unanticipated errors. It is a kind of fault-injection techniques dedicated to exception-handling. This algorithm collects data that is used for verifying two formal contracts that capture two resilience properties w.r.t. exceptions: the source-independence and pure-resilience contracts. Then we propose a code modification technique, called “catch-stretching” which allows error-recovery code (of the form of catch blocks) to be more resilient.Our evaluation is performed on 9 open-source software applications and consists in analyzing 241 catch blocks executed during test suite execution. Our results show that 101/214 of them (47%) expose resilience properties as defined by our exception contracts and that 84/214 of them (39%) can be transformed to be more resilient.Our work shows that it is possible to reason on software resilience by injecting exceptions during test suite execution. The collected information allows us to apply one source code transformation that improves the resilience against unanticipated exceptions. This works best if the test suite exercises the exceptional programming language constructs in many different scenarios.