Abstract
There is evidence that prediction markets are useful tools to aggregate information on researchers' beliefs about scientific results including the outcome of replications. In this study, we use prediction markets to forecast the results of novel experimental designs that test established theories. We set up prediction markets for hypotheses tested in the Defense Advanced Research Projects Agency's (DARPA) Next Generation Social Science (NGS2) programme. Researchers were invited to bet on whether 22 hypotheses would be supported or not. We define support as a test result in the same direction as hypothesized, with a Bayes factor of at least 10 (i.e. a likelihood of the observed data being consistent with the tested hypothesis that is at least 10 times greater compared with the null hypothesis). In addition to betting on this binary outcome, we asked participants to bet on the expected effect size (in Cohen's d) for each hypothesis. Our goal was to recruit at least 50 participants that signed up to participate in these markets. While this was the case, only 39 participants ended up actually trading. Participants also completed a survey on both the binary result and the effect size. We find that neither prediction markets nor surveys performed well in predicting outcomes for NGS2.
Highlights
Prediction markets can be used to aggregate private information [1]
The final market forecasts ranged from −0.18 to 0.8, with a mean of 0.17, and the survey-based forecasts ranged from −0.01 to 0.44, with a mean of 0.16
When using absolute errors to compare the accuracy of market-based and survey-based forecasts, we find no evidence for a statistically significant difference for binary forecasts and for effect size forecasts
Summary
Prediction markets can be used to aggregate private information [1]. By trading contracts with payoffs that depend on clearly defined outcomes, market participants can generate forecasts about future events. The price of a contract with a binary event can, with some caveats [2] be interpreted as the probability the market assigns to this event. Accumulating evidence suggests that prediction markets are effective for accurately identifying which results will replicate in replication studies [9,10,11,12]. As reported in Gordon et al [13], pooling the results from four prediction market studies on binary outcomes (whether the study replicated or not; table 1) gives a 73% (76/104) correct prediction rate if we interpret prices above 50 as market prediction for successful replication. The peer beliefs about replicability estimated in prediction markets can be viewed as a reproducibility indicator, and allows us to assess additional information such as prior and posterior probabilities of the hypotheses being tested
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
