Automated testing of DSL implementations—experiences from building mbeddr

Abstract

Domain-specific languages promise to improve productivity and quality of software development by providing problem-adequate abstractions to developers. Projectional language workbenches, in turn, allow the definition of modular and extensible domain specific languages, generators, and development environments. While recent advances in language engineering have enabled the definition of DSLs and tooling in a modular and cost-effective way, the quality assurance of their implementation is still challenging. In this paper, we discuss our work on testing different aspects of the implementation of domain specific languages and associated tools, and present several approaches to increase the automation of language testing. We illustrate these approaches with the Jetbrains MPS language workbench and our experience with testing mbeddr, a set of domain specific languages and tools on top of C tailored to embedded software development. Based on the experience gained from the mbeddr project, we extract generic lessons for practitioners as well as challenges which need more research.