Some Metrics and a Bayesian Procedure for Validating Predictive Models in Engineering Design

Abstract

Even though model-based simulations are widely used in engineering design, it remains a challenge to validate models and assess the risks and uncertainties associated with the use of predictive models for design decision making. In most of the existing work, model validation is viewed as verifying the model accuracy, measured by the agreement between computational and experimental results. However, from the design perspective, a good model is considered as the one that can provide the discrimination (good resolution) between design candidates. In this work, a Bayesian approach is presented to assess the uncertainty in model prediction by combining data from both physical experiments and the computer model. Based on the uncertainty quantification of model prediction, some design-oriented model validation metrics are further developed to guide designers for achieving high confidence of using predictive models in making a specific design decision. We demonstrate that the Bayesian approach provides a flexible framework for drawing inferences for predictions in the intended but may be untested design domain, where design settings of physical experiments and the computer model may or may not overlap. The implications of the proposed validation metrics are studied, and their potential roles in a model validation procedure are highlighted.