Trends in reliability and test strategies

Abstract

Software fails because it contains faults. If we could be certain that our attempts to fix a fault had been effective and that we had introduced no new faults, we would know that we had improved the software's reliability. Unfortunately, no single testing method can be trusted to give accurate results in every circumstance. There are at least two areas of uncertainty in testing. First, we cannot predict when a failure will occur as a result of choosing an input that cannot currently be processed correctly. Second, we do not know what effect fixing a fault will have on the software's reliability. Most of today's reliability models consider the software system to be a black box, using indirect measures of failure by making strong assumptions about the test cases. White-box testing measures, which are direct measures, are generally not used in software-reliability models. The assumption is that the more the program is covered, the more likely that the software is reliable. Hence we must develop new testing strategies. For example, we do not understand very well the notion of stress as it applies to software; what effect the dependence between versions of the same application has on reliability; the relationship between software reliability and white-box testing measures; and how to achieve the ultra-high reliability levels obtained through formal verification and fault-tolerance methods. These are just some of the areas for future research into improved testing strategies.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>