RISK-DRIVEN SOFTWARE TESTING AND RELIABILITY

Abstract

We develop a risk-driven reliability model and testing process that borrows concepts from classical sequential testing methodology that is used for hardware, with adaptation to software. Both consumer and producer risk are considered, reflecting the fact the consumer (e.g., customer) and producer (e.g., contractor) have different perspectives concerning what they consider to be tolerable risks of software failure. Similarly, there is also a differentiation based on what the consumer and producer consider to be acceptable reliability. Using the consumer-producer framework, we develop a model and process for executing sequential tests, based on software risk and reliability and model risk and reliability prediction accuracy. Test rules are specified for determining at each decision point in testing whether the software and the model prediction accuracy are acceptable. In addition, the test rules serve as stopping criteria for testing. Both empirical and predicted quantities are assessed. The test rules are integrated with several levels of criticality of software (i.e., the higher the criticality, the more stringent the tests). Based on our experience, we provide lessons learned with the objective of improving the model and process for future applications. We start the analysis by developing a model and process template based on the Poisson distribution of failures. This hypothetical example allows us to debug the model and process before it is applied to a real application involving the NASA Space Shuttle flight software.