Specification-based testing using cause-effect graphs

Abstract

Software testing is a very important phase in the software development life cycle. There are two approaches to software testing. Specification based testing treats a program as a black box and disregards its internal structure. Program based testing, on the other hand, depends on the implementation code. Even though studies indicate that specification based testing is comparable, or even superior, to program based testing, most of specification based testing approaches are informal. This thesis presents a specification based testing strategy, called CEG-BOR, which combines the use of cause-effect graphs (CEGs) as a mechanism for representing specifications and the use of the Boolean operator (BOR) strategy for generating tests for a Boolean expression. If all causes of a CEG are independent from each other, a test set for the CEG can be constructed such that all boolean operator faults in the CEG can be detected and the size of this test set grows linearly with the number of nodes in the CEG. Four case studies are conducted to provide empirical data on the performance of CEG-BOR. Empirical results indicate that CEGs can be used to model a large class of software specifications and that CEG-BOR is very effective in detecting a broad spectrum of faults. Also, a BOR test set based on a CEG specification provides better coverage of the implementation code than test sets based on random testing, functional testing, and state-based testing. For a CEG that does not have mutually independent causes, the BOR strategy does not perform well. To remedy this problem, a new test generation strategy is presented, which combines the BOR strategy with the Meaningful Impact (MI) strategy, a recently developed test generation strategy for Boolean expressions. This new strategy, called BOR+MI, decomposes a Boolean expression into mutually independent components, applies the BOR or MI strategy to each component for test generation, and then applies the BOR strategy to combine the test sets for all components. The size and fault detection capability of a BOR+MI test set are investigated. Both analytical and empirical results show that the BOR+MI strategy generates a smaller test set than the MI strategy and provides comparable fault detection ability as the MI strategy. The BOR strategy is also refined to improve the detection of several types of parenthesis faults in a Boolean expression. The BOR+MI approach is further extended to CEGs with causes being relational expressions. This extension, called BRO+MI, detects incorrect relational operators in relational expressions and also accounts for user-defined or implicit restrictions on the causes of a CEG. Use of constraint logic programming techniques to automatically generate test data to satisfy BRO+MI test cases is proposed. Heuristics for incremental constraint propagation and deferred constraint satisfaction to make test data generation more efficient are presented. Empirical results indicate that the application of BOR based test generation strategies to specifications represented as predicates is a practical, scalable and cost-effective approach for development of effective software test suites across a wide set of application areas.