Reasonability of MC/DC for safety-relevant software implemented in programming languages with short-circuit evaluation

Abstract

MC/DC (modified condition/decision coverage) is a structural code coverage metric, originally defined in the standard DO-178B [1], intended to be an efficient coverage metric for the evaluation of the testing process of software incorporating decisions with complex Boolean expressions. The upcoming standard ISO 26262 [2] for safety-relevant automotive systems prescribes MC/DC for ASIL D as a highly recommended coverage metric. One assumed benefit of MC/DC is that it requires a much smaller number of test cases in comparison to MCC (multiple condition coverage), while sustaining a quite high error detection probability [3]. Programming languages like C, commonly used for implementing software for the automotive domain, are using short-circuit evaluation. For short-circuit evaluation the number of test cases for MCC is much smaller than in a non-short-circuit environment because many redundant test cases occur. We evaluated the trade-off between the number of test cases for MCC and MC/DC for a case study from the automotive domain and observed an overhead of only approximately 5% for the number of test cases necessary for MCC compared to MC/DC. This motivated an analysis of programs containing decisions where the number and structure of the referring Boolean expressions vary. Our results show that the overhead for a test suite for MCC is on the average only about 35% compared MC/DC (for decisions with up to 5 conditions). We conclude with the strong recommendation to use MCC as a coverage metric for testing safety-relevant software implemented in programming languages with short-circuit evaluation.