An empirical comparison of model-based and capture and replay approaches for performance testing

Abstract

A variety of testing tools has been developed to support and automate the software testing activity. Some of them may use different techniques such as Model-based Testing (MBT) or Capture and Replay (CR). Model-based Testing is a technique for automatic generation of testing artifacts based on software models. One of the main benefits of using MBT is related to the easiness of maintaining models over code; hence, it is likely that using models as a source for automatic generation of scripts requires less effort and reduces the number of faults. Otherwise, CR-based tools basically record the user interaction with the System Under Test (SUT) and then playback the recorded test. This paper presents our effort on setting up and running an experimental study performed in order to evaluate the effort to use MBT and CR-based tools to generate performance scripts. Thus, we apply an MBT and a CR approaches for the purpose of evaluation with respect to the effort to generate scripts and scenarios from the perspective of the performance testers and the performance test engineers in the context of undergraduates, M.Sc. and Ph.D. students, performance testers and performance test engineers for the generation of performance test scripts and scenarios. Our results indicate that, for simple testing tasks, the effort of using a CR-based tool was lower than using an MBT tool, but as the complexity or size of the activities of the testing tasks increases, the advantage of using MBT increased significantly.