RESTest: Black-Box Constraint-Based Testing of RESTful Web APIs

Abstract

Automated testing approaches for RESTful web APIs typically follow a black-box strategy, where test cases are derived from the API specification. These techniques show promising results, but they neglect constraints among input parameters (so-called inter-parameter dependencies), as these cannot be formally described in current API specification languages. As a result, black-box tools rely on brute force to generate valid test cases, i.e., those satisfying all the input constraints. This is not only extremely inefficient, but it is also unlikely to work for most real-world services, where inter-parameter dependencies are complex and pervasive. In this paper, we present RESTest, a framework for automated black-box testing of RESTful APIs. Among its key features, RESTest supports the specification and automated analysis of inter-parameter dependencies, enabling the use of constraint solvers for the automated generation of valid test cases. This allows to detect more faults, and faster, through a deeper evaluation of valid and invalid input parameters’ combinations and the use of novel test oracles. Evaluation results on 6 commercial APIs show that RESTest can efficiently generate up to 99% more valid test cases than random testing techniques, 60% on average. More importantly, RESTest revealed 2K failures undetected by random testing, uncovering bugs in all the services under test.