Software testing--how much is enough?

Abstract

aspect of software verification and validation as defined by the Food and Drug Administration (FDA). Along with requirement specification and configuration management, it is one of the aspects of software development and design control that receive the most scrutiny by FDA reviewers as well as inspectors. The most difficult aspect of testing (and the hardest to teach) is determining “how much is enough.” One can always do more, use a more rigorous method, add other tools, provide more specificity and detail in specifications and test documents, or perform more extensive reviews of tests and their results. For software related to medical devices, it is now quite common to assert that the rigor and detail required is relative to the safety risks involved. The FDA says this explicitly in their guidance documents; their General Principles of Software Validation (final version, January 2002) and the associated Federal Register notice provide explanation and commentary. But what exactly does this mean? In a particular situation, how much is enough? Performing adequate testing would be simple if software were simple—one could identify and execute a small number of tests that would thoroughly evaluate all quality-related attributes of the software under all conditions (eg, stress, fault, etc.). However, because even “small” software programs can have a large number of permutations and combinations of inputs, logic, and execution paths, it is generally impossible to “fully” test software. Once this is understood, it becomes clear that test planning is a sampling-based activity best done with a risk-based perspective. Random sampling has not been demonstrated to be an adequate approach because the locations of software bugs do not follow a normal distribution, and some bugs are more important to find than others on the basis of safety and effectiveness concerns. This is important for developers, testers, and regulators to recognize. A failure to do so can result in either unrealistic requirements for testing or a dangerous false sense of security that if a person runs all defined tests then he or she is fully testing the software. This is especially dangerous if one assumes that just by repeating all original tests as part of regression testing that the software must therefore be safe and effective. Testing has many approaches, tools, and methodologies, and in a regulated environment it has additional explicit and implicit requirements. Although not trivial, it is relatively easy to develop an understanding of specific approaches, methods, tools, and test documentation requirements. However, there is more to consider. A key factor in planning adequate testing, as well as in determining and demonstrating when one has “done enough,” is deciding if test coverage is adequate. This is a different perspective than simply determining if test documentation is adequately detailed or if test plans and test cases are properly approved and reviewed. Instead, QUALITY MATTERS