Mapping model validation metrics to subject matter expert scores for model adequacy assessment

Abstract

This paper develops a novel approach to incorporate the contributions of both quantitative validation metrics and qualitative subject matter expert (SME) evaluation criteria in model validation assessment. The relationship between validation metrics (input) and SME scores (output) is formulated as a classification problem, and a probabilistic neural network (PNN) is constructed to execute this mapping. Establishing PNN classifiers for a wide variety of combinations of validation metrics allows for a quantitative comparison of validation metric performance in representing SME judgment. An advantage to this approach is that it semi-automates the model validation process and subsequently is capable of incorporating the contributions of large data sets of disparate response quantities of interest in model validation assessment. The effectiveness of this approach is demonstrated on a complex real-world problem involving the shock qualification testing of a floating shock platform. A data set of experimental and simulated pairs of time history comparisons along with associated SME scores and computed validation metrics is obtained and utilized to construct the PNN classifiers through K-fold cross validation. A wide range of validation metrics for time history comparisons is considered including feature-specific metrics (phase and magnitude error), a frequency metric (shock response spectra), a time-frequency metric (wavelet decomposition), and a global metric (index of agreement). The PNN classifiers constructed using a Parzen kernel for the class conditional probability density function whose smoothing parameter is optimized using a genetic algorithm performs well in representing SME judgment.