Single and Multiple Period Decision Models for Analysis of Quality and Quantity of Validation*

Abstract

ABSTRACTThis paper investigates a single period model for the analysis of the impact of the quality of the validation effort. The single period model uses a Bayesian approach to find that validation is a critical point process. That model is then extended to allow for the uncertainty of the validation process to determine the quality of the underlying model. Some monotonicity results are developed for the model and investigated in light of the process being a critical point process. The model indicates that, consistent with comments from real world settings, the impact of the quality of the validation effort can be substantial.The paper also presents two multiperiod models of the impact of the quantity of the validation effort. In practice, the development of an expert system may follow a recurring multiperiod life cycle, where a prototype is built, the system is validated to determine how well it performs, and based on that performance, is either funded or not funded. The first multiple period model assumes that validation and funding occurs at each point in the PVF budget cycle. The model employs Bayesian revision of probabilities to update the prior probability of obtaining a model with an appropriate level of success. It is found that the critical point for multiperiod problems is different than that for single period problems. This model forms the basis of the second model. The second multiple period model extends the first by assuming that the quantity of validation can be varied. The more validation, the more likely that flaws in the model will be found. Thus, the more validation, the better the understanding of the level of performance of the model.