A replicated empirical study to evaluate software testing methods

Abstract

Many empirical studies have been performed to evaluate software testing methods in past decades. However, we are still not able to generalize the results as most studies are not complete and differ significantly from one another. To contribute to the existing knowledge base of software testing methods, we performed an empirical study to evaluate 3 testing methods: (1) code reading by stepwise abstraction, (2) functional testing using equivalence partitioning and boundary value analysis, and (3) structural testing using 100% branch, multiple-condition, loop, and relational-operator coverage using a well-defined and standard schema. A controlled experiment is performed with 18 subjects who applied the 3 defect detection techniques to 3 C programs in a fractional factorial experimental design to observe failures and isolate faults. The experimental results show that (1) the techniques are equally effective in terms of observing failures and finding faults, (2) the effectiveness of the techniques depends on the nature of the program, (3) all the testing techniques are equally efficient in case of failure observation, (4) the techniques are different in their efficiency in terms of fault isolation where code reading performed better than that of structural and functional testing, and (5) with respect to the fault types, all the techniques were equally effective in observing failures and isolating faults except in case of cosmetic faults where functional testing performed better than the other 2 techniques The effectiveness and efficiency of testing techniques were significantly influenced by the type of program. The results presented in this paper contribute to an empirical knowledge base of testing methods and may be helpful for the software engineers to decide the appropriate techniques in improving the software testing process.