The Science of Killing Bugs in a Black Box

Abstract

AbstractIn this talk I will discuss the combination of model-based testing and mutation testing. Model-based testing is a black-box testing technique that avoids the labour of manually writing hundreds of test cases, but instead advocates the capturing of the expected behaviour in a model of the system under test. The test cases are automatically generated from this model. The technique is receiving growing interest in the embedded-systems domain, where models are the rule rather than the exception.Mutation testing is a technique for assessing and improving a test suite. A number of faulty versions of a program under test are produced by injecting bugs into its source code. These faulty programs are called mutants. A tester analyses if his test suite can ”kill” all mutants. We say that a test kills a mutant if it is able to distinguish it from the original. The tester improves his test suite until all faulty mutants get killed.In model-based mutation testing, we combine the central ideas of model-based testing and mutation testing: we inject bugs in a model and generate a test suite that will kill these bugs. In this talk, I will discuss its scientific foundations, tools, and results. The foundations include semantics and conformance relations; the supporting tools involve model checkers, constraint solvers and SMT solvers; our experimental results are taken from two European projects on embedded-systems. I will conclude with a proposal how model-based mutation testing can be integrated into an agile, iterative development process.KeywordsTest SuiteMutation TestingSystem Under TestGenerate Test CaseTest Oracle