Consistency among alternative performance measures in an applied judgement setting

Abstract

The extent to which different performance measures can lead to different conclusions about human performance is investigated in a task involving the prediction of experts' ratings of the quality of corporate bonds. Results produced by the traditional correlational performance measure - with respect to both the relative performance of ‘man’ versus model and the effects of feedback on performance - are compared with results produced by five alternative measures including mean absolute error, mean squared error, and various types of ‘hit rates’. The correlational measure produces results that are moderately consistent with those produced by the other five measures in this applied prediction setting. Potential implications of using alternative performance measures in applied settings, especially the need to identify appropriate loss functions, are emphasized.