Dynamut: A Mutation Testing Tool for Industry-Level Embedded System Applications

Darin Weffenstette,Kristen R Walcott

doi:10.5121/ijesa.2018.8302

Abstract

Test suite evaluation is important when developing quality software. Mutation testing, in particular, can be helpful in determining the ability of a test suite to find defects in code. Because of challenges incurred developing on complex embedded systems, test suite evaluation on these systems is very difficult and costly. We developed and implemented a tool called DynaMut to insert conditional mutations into the software under test for embedded applications. We then demonstrate how the tool can be used to automate the collection of data using an existing proprietary embedded test suite in a runtime testing environment. Conditional mutation is used to reduce the time and effort needed to perform test quality evaluation in 48% to 67% less time than it would take to perform the testing with a more traditional mutate-compile-test methodology. We also analyze if testing time can be further reduced while maintaining quality by sampling the mutations tested.

Highlights

When engineering a software solution, testing is essential
Explore ways of reducing cost of testing through sampling of mutations. Discuss these results and how they can be applied to reduce the cost of mutation testing in an embedded system environment
We examine the time overhead of conditional mutation testing in an embedded environment versus the traditional mutation-compile-deploy approach and evaluate how sampling techniques can be applied to reduce the number of test runs without reducing effectiveness

Summary

Introduction

To ensure a test suite is effective at finding defects, it is important to evaluate the test suite with regard to quality. While code coverage metrics, such as statement or branch coverage, are useful in determining how to improve a test suite, mutation testing has been shown to be a better indicator of the ability of a test suite to find faults in code [17, 21]. Many tools have been created to automate test suite evaluation for unit tests On embedded systems in industry, functional testing of the whole system is much more common than unit testing [11]. Combined with uncommon build and runtime environments, the time overhead inherent to the embedded platform, and a lack of applicable tools, makes automated test suite evaluation challenging on embedded systems

Results

Discussion

Conclusion