Abstract
Current attempts at methodological reform in sciences come in response to an overall lack of rigor in methodological and scientific practices in experimental sciences. However, most methodological reform attempts suffer from similar mistakes and over-generalizations to the ones they aim to address. We argue that this can be attributed in part to lack of formalism and first principles. Considering the costs of allowing false claims to become canonized, we argue for formal statistical rigor and scientific nuance in methodological reform. To attain this rigor and nuance, we propose a five-step formal approach for solving methodological problems. To illustrate the use and benefits of such formalism, we present a formal statistical analysis of three popular claims in the metascientific literature: (i) that reproducibility is the cornerstone of science; (ii) that data must not be used twice in any analysis; and (iii) that exploratory projects imply poor statistical practice. We show how our formal approach can inform and shape debates about such methodological claims.
Highlights
Widespread concerns about unsound research practices, lack of transparency in science and low reproducibility of empirical claims have led to calls for methodological reform across scientific disciplines [1,2,3,4]
To show why formalism is essential in establishing the validity of methodological proposals and how informal approaches making the jump from step 0 to 5 might misinform scientific practice, we evaluate three specific examples of methodological claims from the reform literature:
— reproducibility is the cornerstone of, or a demarcation criterion for, science; — using data more than once invalidates statistical inference; and — exploratory research uses ‘wonky’ statistics. We focus on these claims as case studies to illustrate our approach because all three are methodological claims with statistical implications that have been impactful1 in the metascience literature as well as on post-replication crisis practices of empirical scientists while receiving considerable but informal criticism
Summary
Widespread concerns about unsound research practices, lack of transparency in science and low reproducibility of empirical claims have led to calls for methodological reform across scientific disciplines [1,2,3,4]. To show why formalism is essential in establishing the validity of methodological proposals and how informal approaches making the jump from step 0 to 5 might misinform scientific practice, we evaluate three specific examples of methodological claims from the reform literature:. — reproducibility is the cornerstone of, or a demarcation criterion for, science; — using data more than once invalidates statistical inference; and — exploratory research uses ‘wonky’ statistics We focus on these claims as case studies to illustrate our approach because all three are methodological claims with statistical implications that have been impactful in the metascience literature as well as on post-replication crisis practices of empirical scientists while receiving considerable but informal criticism. We conclude that methodological reform first needs a mature theory of reproducibility to be able to identify whether sufficient conditions exist that may justify labelling reproducibility as a measure of true regularities
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
