Modifying Goodness-of-Fit Indicators to Incorporate Both Measurement and Model Uncertainty in Model Calibration and Validation

Abstract

Because of numerous practical implications of uncertainty in measured data and model predictions, improved techniques are needed to analyze and understand uncertainty and incorporate it into hydrologic and water quality evaluations. In the present study, a correction factor was developed to incorporate measurement uncertainty and model uncertainty in evaluations of model goodness-of-fit (predictive ability). The correction factor, which was developed for pairwise comparisons of measured and predicted values, modifies the typical error term calculation to consider both sources of uncertainty. The correction factor was applied with common distributions and levels of uncertainty (represented by coefficients of variation ranging from 0.026 to 0.256) for each measured value and each predicted value from five example data sets. The modifications resulted in inconsequential changes in goodness-of-fit conclusions for example data sets with very good and poor model simulations, which is both logical and appropriate because very good model performance should not improve greatly and poor model performance should not become satisfactory when uncertainty is considered. In contrast, incorporating uncertainty in example data sets with initially moderate goodness-of-fit resulted in important improvements in indicator values and in model performance ratings. A model evaluation matrix was developed to present appropriate model performance conclusions, considering both model accuracy and precision, based on various levels of measurement and model uncertainty. In cases with highly uncertain calibration/validation data, definitive good fit conclusions are cautioned against even with good indicator values because of the uncertain standard of comparison; however, in these cases, poor model accuracy can be confidently concluded from unsatisfactory indicator values. In contrast, model accuracy can be confidently concluded from goodness-of-fit indicator values in cases with low measurement uncertainty. It is hoped that the modified goodness-of-fit indicators and the model evaluation matrix contribute to improved goodness-of-fit conclusions and to more complete assessments of model performance.