Evaluating the Impact of Different Testers on Model-based Testing

Abstract

Context: Model-Based Testing (MBT) is an approach that allows testers to represent the behavior of the system under test as models, specifying inputs and their expected outputs. From such models, existing tools might be employed to generate test cases automatically. While MBT represents a promising step towards the automation of test case generation, the quality of the model designed by the tester may impact, either positively or negatively, on its ability to reveal faults (i.e., the test effectiveness). Objective: In this context, we conducted a preliminary experiment to evaluate the impact caused by different testers when designing a test model for the same functionality. Method: In the experiment, the participants used Event Sequence Graphs and its supporting tool FourMA to create test models for two mobile apps: arXiv-mobile and WhoHasMyStuff. From the test models, test cases were generated using FourMA and concretized by means of the Robotium framework. In order to measure the impact of different testers, we employed code coverage (namely, instruction and branch coverage) as an estimation of test effectiveness. Results: Based on the results obtained, we observe high variation of code coverage among the testers. No tester was capable of producing a test model that subsumes all other testers' models with respect to code coverage. Moreover, factor learning seems not to reduce the code coverage variation. The relation between model size, modeling time, and code coverage were inconclusive. Conclusion: We conclude that further research effort on the MBT's modeling step is required to not only reduce the variation between testers, but also improving its effectiveness.