Skyfire: Model-Based Testing with Cucumber

Abstract

In the software industry, a Behavior-Driven Development (BDD) tool, Cucumber, has been widely used by practitioners. Usually product analysts, developers, and testers manually write BDD test scenarios that describe system behaviors. Testers write implementation for the BDD scenarios by hand and execute the Cucumber tests. Cucumber provides transparency about what test scenarios are covered and how the test scenarios are mapped to executable tests. One drawback of the Cucumber BDD approach is that test scenarios are generated manually. Thus, the test scenarios are usually weak. More importantly, practitioners do not have a metric to measure test coverage. In this paper, we present a Model-Based Testing (MBT) tool, skyfire. Skyfire can automatically generate effective Cucumber test scenarios to replace manually generated test scenarios. Skyfire reads a behavioral UML diagram (e.g., a state machine diagram), identifies all necessary elements (e.g., transitions) of the diagram, generates effective tests to satisfy various graph coverage criteria, and converts the tests into Cucumber scenarios. Then testers write Cucumber mappings for the generated scenarios. Skyfire does not only generate effective tests but is also completely compatible with the existing agile development and continuous integration (CI) rhythm. We present the design architecture and implementation of skyfire, as well as an industrial case study to show how skyfire is used in practice.