Evaluating specification-level MC/DC criterion in model-based testing of safety critical systems

Abstract

Safety-critical software systems in the aviation domain, e.g., a UAV autopilot software, needs to go through a formal process of certification (e.g., DO-178C standard). One of the main requirements for this certification is having a set of explicit test cases for each software requirement. To achieve this, the DO-178C standard recommends using a model-driven approach. For instance, model-based testing (MBT) is recommended in its DO-331 supplement to automatically generate system-level test cases for the requirements provided as the specification models. In addition, the DO-178C standard also requires high level of source code coverage, which typically is achieved by a separate set of structural testing. However, the standard allows targeting high code with MBT, only if the applicants justify their plan on how to achieve high code through model-level testing. In this study, we propose using the Modified Condition and Decision (MC/DC) on the specification-level constraints rather than the standard-recommended all coverage criterion, to achieve higher code through MBT. We evaluate our approach in the context of a case study at MicroPilot Inc., our industry collaborator, which is a UAV producer company. We implemented our idea as an MC/DC on guards in a UML state-machine-based testing tool that was developed in-house. The results show that applying model-level MC/DC outperforms the typical transition-coverage (DO-178C's required MBT criterion), with respect to source code-level all condition-decision criterion by 33%. In addition, our MC/DC test suite detected three new faults and two instances of legacy specification in the code that are no longer in use, compared to the all transition test suite.