1 Assessing the predictive adequacy of simple and complex mathematical models

Abstract

Abstract The establishment of credibility for a mathematical model’s (MM) predictive ability is an essential component for improving the MM because it stimulates the evolutionary thinking (i.e., the next generation of the model) of mental conceptualizations, assumptions, and boundaries of the MM. Its predictive adequacy is commonly assessed through its ability to precisely or accurately predict observed (real) values. The precision component measures how closely the model predicted values are of each other or whether a defined pattern of predictions exists. The accuracy component, on the other hand, measures how closely the average of the model predicted values are to the actual (true) average. Many statistics exist to determine precision and accuracy of MM such as mean bias, resistant coefficient of determination, coefficient of determination, modeling efficiency, concordance correlation coefficient (CCC), the mean square error of prediction, Kleijnen’s statistic (regression of the difference between predicted and observed on their sum), and Altman and Bland’s limits of agreement statistics among many more. However, for complex models that use skewed data or repeated data in which the data is not independent (e.g., multiple measurements on the same subject), simple statistics may not suffice. For instance, four methods to compute CCC exist (moment, variance components, U-statistics, and generalized estimating equations—GEE), but only the last two methods are resilient to lightly skewed data. Another type of complexity arises when meta-analytical approaches are used at the model development phase or the model evaluation phase. In general, meta-analytical approaches remove errors (i.e., variation) associated with random variables that are believed to be known. Under these circumstances, MM tends to overperform (i.e., they have greater predictive adequacy) and their future performance may be deceitful when trying to forecast at scenarios in which the random variable(s) is(are) indeterminable or unquantifiable.