Comparative Testing of Experts

Abstract

We show that a simple \reputation-style test can always identify which of two experts is informed about the true distribution. The test presumes no prior knowledge of the true distribution, achieves any desired degree of precision in some xed nite time, and does not use \counterfactual predictions. Our analysis capitalizes on a result due to Fudenberg and Levine (1992) on the rate of convergence of supermartingales. We use our setup to shed some light on the apparent paradox that a strategically motivated expert can ignorantly pass any test. We point out that this paradox arises because in the single-expert setting, any mixed strategy for Nature over distributions is reducible to a pure strategy. This eliminates any meaningful sense in which Nature can randomize. Comparative testing reverses the impossibility result because the presence of an expert who knows the realized distribution eliminates the reducibility of Nature’s compound lotteries.