Assessing model-based testing: an empirical study conducted in industry

Abstract

We compare manual testing without any automation performed by a tester at a software company with model-based testing (MBT) performed by a tester at a research center. The system under test (SUT), of which two different versions were tested by each of the two testers, is a professionally developed web-based data collection system that now is in use. The two testers tested the same versions, had identical testing goals (to detect defects), had access to the same resources, but used different processes (i.e. manual without any automation vs. model-based with automatic test case generation and automatic test case execution). The testers did not interact with each other. We compare the effectiveness (issues found) and efficiency (effort spent) of the two approaches. The results show, for example, that manual testing required less preparation time and that its test coverage was somewhat uneven. In contrast, MBT required more preparation time, was more systematic, and detected more issues. While the manual approach detected more inconsistencies between specified and actual text labels, MBT detected more functional issues. This is reflected in the severity score summary which was about 60% higher for MBT than Manual.