Evaluating and Improving Probability Assessment in an Ambiguous, Sequential Environment

Abstract

The current study explores the validity of verbal probability assessments in a sequential and highly ambiguous task, that is, one in which it is virtually impossible to know or learn about the true probabilities of possible outcomes. Participants observed the pre-defined motion of an unmanned aerial vehicle (UAV), such that the participant’s success depended on the UAV reaching a target sector without being spotted by an opponent UAV. At several points in each trajectory, participants’ task was to evaluate the likelihood of reaching the target successfully. The study utilized a 2 × 2 independent-groups factorial design to examine the effect of probability incentivization (Brier vs none), in which participants receive payment based on the nearness of their predictions to actual outcomes, and informational reviews (present vs absent), in which participants engage in detailed discussion with the experimenter, regarding their assessments in seven previous trials before continuing, on probability assessment. A statistically significant main effect of Brier scoring was found, such that Brier based incentivization improved assessment accuracy. The effect of informational review and the interaction effect were not significant. All groups performed significantly better than random and uninformed performance. Outcomes from this study improve our understanding of the validity of online judgments made by operators of unmanned vehicles in strategic settings. It is concluded that non-expert probability assessments carry important information value even in ambiguous settings and even without incentives, and importantly, are further amenable to incentives and training.