Two case studies in grammar-based test generation

Abstract

Grammar-based test generation (GBTG) has seen extensive study and practical use since the 1970s. GBTG was introduced to generate source code for testing compilers from context-free grammars specifying language syntax. More recently, GBTG has been applied to many other testing problems, including the generation of eXtensible Markup Language (XML) documents and the generation of packets for testing communications protocols. Recent research has shown how to integrate covering-array techniques such as pairwise testing into GBTG tools. While the integration offers considerable power to the tester, there are few practical demonstrations in the literature. We present two case studies showing how to use grammars and covering arrays for automated software testing. The first case study exposes HTML injection vulnerabilities in an RSS feed parser. The second case study determines the effectiveness of network firewalls when faced with TCP flag attacks. The case studies illustrate the use of covering arrays in a GBTG context, the use of visualization to understand large test logs, and the issues and tradeoffs in the design of fully automated GBTG test suites.