Trimmed Opinion Pools and the Crowd's Calibration Problem

Abstract

We introduce an alternative to the popular linear opinion pool for combining individual probability forecasts. One of the well-known problems with the linear opinion pool is that it can be poorly calibrated. It tends toward underconfidence as the crowd’s diversity increases, i.e., as the variance in the individuals’ means increases. To address this calibration problem, we propose the exterior-trimmed opinion pool. To form this pool, forecasts with low and high means, or cumulative distribution function (cdf) values, are trimmed away from a linear opinion pool. Exterior-trimming decreases the pool’s variance and improves its calibration. A linear opinion pool, however, will remain overconfident when individuals are overconfident and not very diverse. For these situations, we suggest trimming away forecasts with moderate means, or cdf values. This interior-trimming increases variance and reduces overconfidence. Using probability forecast data from US and European Surveys of Professional Forecasters, we present empirical evidence that trimmed opinion pools can outperform the linear opinion pool.