Multiple Experts vs. Multiple Methods: Combining Correlation Assessments

Abstract

Averaging forecasts from several experts has been shown to lead to improved forecasting accuracy and reduced risk of bad forecasts. Similarly, it is accepted knowledge in decision analysis that an expert can benefit from using more than one assessment method to look at a situation from different viewpoints. In this paper, we investigate gains in accuracy in assessing correlations by averaging different assessments from a single expert and/or from multiple experts. Adding experts and adding methods can both improve accuracy, with diminishing returns to extra experts or methods. The gains are generally much greater from adding experts than from adding methods, and restricting the set of experts to those who do particularly well individually leads to the greatest improvements in the averaged assessments. The variability of assessment accuracy decreases considerably as the number of experts or methods increases, implying a large risk reduction. We discuss conditions under which the general pattern of results obtained here might be expected to be similar or different in other situations with multiple experts and/or multiple methods.