Model evaluation, discrepancy function estimation, and social choice theory

Abstract

A discrepancy function provides for an evaluation of a candidate model by quantifying the disparity between the candidate model and the true model that generated the observed data. The favored model from a candidate class is the one judged to have minimum discrepancy with the true model. The observed data can be regarded as a manifestation of the underlying true model. However, since the data provides only partial information as to the nature of the true model, the selection of a model is a decision that is made in the presence of uncertainty. To characterize this uncertainty, we consider employing resampling to generate multiple manifestations of the true model. Each of the candidate models can then be judged against each of the simulated versions of the true model, resulting in multiple panels of discrepancies. Model evaluation is subsequently achieved by providing an overall judgment on each candidate model. This overall assessment is based on combining the information in the individual discrepancy panels. As social choice theory, or voting theory, addresses the problem of turning individual preferences into a group preference, we see that social choice theory can be used in developing a novel approach to model evaluation.