An empirical bayesian stopping rule in testing and verification of behavioral models

Abstract

Software stopping rules are tools to effectively minimize the time and cost involved in software testing. The algorithms serve to guide the testing process such that if a certain level of branch or fault (or failure) coverage is obtained without the expectation of further significant coverage, then the testing strategy can be stopped or changed to accommodate further, more advanced testing strategies. By combining cost analysis with a variety of stopping-rule algorithms, a comparison can be made to determine an optimally cost-effective stopping point. A novel cost-effective stopping rule using empirical Bayesian principles for a nonhomogeneous Poisson counting process compounded with logarithmic-series distribution (LSD) is derived and satisfactorily applied to digital software testing and verification. It is assumed that the software failures or branches covered, whichever the case may be, clustered at the application of a given test-case are positively correlated, i.e., contagious, implying that the occurrence of one software failure (or coverage of a branch) positively influences the occurrence of the next. This phenomenon of clustering of software failures or branch coverage is often observed in software testing practice. The r.v. w/sub i/ of the failure-clump size of the interval is assumed to have LSD(/spl theta/) and justified on the data sets by employing a chi-square goodness of fit testing while the distribution of the number of test cases is Poisson(/spl lambda/). Then, the distribution of the total number of observed failures, or similarly covered branches, X is a compound Poisson /sup /spl and// LSD, i.e., negative binomial distribution, given that a certain mathematical identity holds. For each checkpoint in time, either the software satisfies a desired reliability attached to an economic criterion, or else the software testing is allowed to continue. By using a one-step-look-ahead formula derived for the model, the proposed stopping rule is applied to five test case-based data sets acquired by testing embedded chips through the complex VHDL models. Further, multistrategy testing is conducted to show its superiority to single-stage testing. Results are satisfactorily interpreted from a practitioner's viewpoint as an innovative alternative to the ubiquitous test-it-to-death approach, which is known to waste billions of test cases in a tedious process of finding more bugs. Moreover, the proposed dynamic stopping-rule algorithm can validly be employed as an alternative paradigm to the existing on-line statistical process control methods static in nature for the manufacturing industry, provided that underlying statistical assumptions hold. A detailed comparative literature survey of stopping-rule methods is also included in terms of pros and cons, and cost effectiveness.