A Replicated Study on Random Test Case Generation and Manual Unit Testing: How Many Bugs Do Professional Developers Find?

Abstract

This paper describes the replication of an empirical study comparing tool-supported test case generation and manual development of unit tests. As variation to the original study, which was based on test results from students performing manual unit testing for 60 minutes, the replication involves professional software developers with several years of industry experience and extends the initial time restriction. As part of the replication the paper explores the differences in unit testing by students and professionals and investigates the impact of the extended time limit. The main findings are: There are no significant differences in the results produced by students and by professional developers when performing manual unit testing for 60 minutes. Furthermore, there is a non-linear increase in the number of defects found when the time limit is extended from one to two hours, which indicates the transition from the initial ramp-up phase to productive testing. The replication also confirms the conclusions of the original study: Automated test case generation can be equally efficient as manual unit testing under severe time restrictions and it may be used to complement a manual testing approach.