Mutation testing meets approximate computing

Abstract

One of the most widely studied techniques in software testing research is mutation testing - a technique for evaluating the quality of test suites. Despite over four decades of academic advances in this technique, mutation testing has not found its way to mainstream development. The key issue with mutation testing is its high computational cost: it requires running the test suite against not just the program under test but against typically thousands of mutants, i.e., syntactic variants, of the program. Our key insight is that exciting advances in the upcoming, yet unrelated, area of approximate computing allow us to define a principled approach that provides the benefits of traditional mutation testing at a fraction of its usually large cost.This paper introduces the idea of a novel approach, named ApproxiMut, that blends the power of mutation testing with the practicality of approximate computing. To demonstrate the potential of our approach, we present a concrete instantiation: rather than executing tests against each mutant on the exact program version, ApproxiMut obtains an approximate test/program version by applying approximate transformations and runs tests against each mutant on the approximated version. Our initial goal is to (1) measure the correlation between mutation scores on the exact and approximate program versions, (2) evaluate the relation among mutation operators and approximate transformations, (3) discover the best way to approximate a test and a program, and (4) evaluate the benefits of ApproxiMut. Our preliminary results show similar mutation scores on the exact and approximate program versions and uncovered a case when an approximated test was, to our surprise, better than the exact test.