Abstract
We provide a novel articulation of the epistemic peril of p-hacking using three resources from philosophy: predictivism, Bayesian confirmation theory, and model selection theory. We defend a nuanced position on p-hacking: p-hacking is sometimes, but not always, epistemically pernicious. Our argument requires a novel understanding of Bayesianism, since a standard criticism of Bayesian confirmation theory is that it cannot represent the influence of biased methods. We then turn to pre-analysis plans, a methodological device used to mitigate p-hacking. Some say that pre-analysis plans are epistemically meritorious while others deny this, and in practice pre-analysis plans are often violated. We resolve this debate with a modest defence of pre-analysis plans. Further, we argue that pre-analysis plans can be epistemically relevant even if the plan is not strictly followed—and suggest that allowing for flexible pre-analysis plans may be the best available policy option.
Highlights
We provide a novel articulation of the epistemic peril of p-hacking using three resources from philosophy: predictivism, Bayesian confirmation theory, and model selection theory
We made this argument by drawing on the fruits of the prediction-accommodation debate, and by appealing to formal tools from Bayesian confirmation theory and model selection theory
In accordance with our position on p-hacking, we articulated a modest defence of pre-registration
Summary
‘P-hacking,’ a term used widely in contemporary scientific discourse, refers to a variety of practices. The second contribution is an articulation of the epistemic benefits of pre-analysis plans, in which scientists note in advance what analyses they will perform as a means to mitigate the perils of p-hacking (§3). An intuition that many hold about the epistemic peril of p-hacking is that the evidence in Scenario 1 provides less confirmation to the hypothesis than does the evidence in Scenario 2. In some cases prediction of evidence does provide more confirmation to a hypothesis than mere accommodation of the same evidence, and it is precisely this fact, one might say, that explains why some forms of p-hacking are at least sometimes epistemically pernicious. To see that prediction of evidence can provide more confirmation to a hypothesis than mere accommodation of the same evidence, consider the following scenarios. In scenarios like 5 and 6, p-hack holds, while in scenarios like 3 and 4, p-hack does not hold
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
