An empirical study on software error detection: voting, instrumentation and Fagan inspection

Abstract

The paper presents the results of an experiment that compared error detection capability of voting, instrumentation, and Fagan inspection methods. Several experiments have measured effectiveness of various error detection methods. However, most experiments have used different programs; consequently, the results are generally incompatible and do not allow one to make objective comparison on the cost-effectiveness of various approaches. Software cannot be developed using an unlimited amount of resources, and practitioners need empirical and objective data on the cost-effectiveness of various error detection methods to decide which methods to use during software development. Results of this experiment is significant because these methods have been applied to the same program. Furthermore, the participant's educational and industrial experience are comparable to that of the previous experiments. We confirmed the previous finding that detecting errors in reliable programs is difficult; none of the three methods detected more than half of all the known errors in the programs. Of the three methods employed, participants detected more errors by using Fagan inspection method than they did by voting or instrumentation. When the average number of hours needed to detect an error was compared, Fagan inspection method was shown to be more cost-effective than instrumentation method.