Lessons from the first two years of operating a study registry.

Abstract

The need for improved methodology for psychological research has recently received much attention. The primary recommendation has been increased emphasis on confirmatory or replication research that is carefully planned with adequate sample size and is pre-registered (Wagenmakers et al., 2012; Nosek and Lakens, 2014; Simons et al., 2014). Study registration options are currently being developed and implemented. Based on our experience operating a study registry, we offer practical recommendations and observations that may be useful when implementing study registration more widely. In the fall of 2012, we opened a study registry at the University of Edinburgh's Koestler Parapsychology Unit (KPU) (KPU Registry, 2012). Consistent with the standards for registering clinical trials (International Committee of Medical Journal Editors, 2005), the registry focuses on public, prospective registration with specified registration information, and is not affiliated with a specific journal. The present discussion addresses methodology, not the findings of the registered studies. Parapsychological researchers have strived to utilize the established research methods of experimental psychology. This aspiration has resulted in increasing publications in high profile psychology journals (Bosch et al., 2006; Storm et al., 2010; Bem, 2011), but has not provided noticeable progress in resolving the debates about parapsychology. This situation was a significant factor in the recognition by psychologists that improved research methodology was needed (Pashler and Wagenmakers, 2012; Wagenmakers et al., 2012). Based on experience working in regulated medical research, the second author has long advocated that the standard research methods for academic psychology were not adequate for controversial research like parapsychology and that formal, pre-registered, well-powered confirmatory research was needed (Kennedy, 2004). The first author also pointed out the value of pre-registered confirmatory research (Watt, 2005). However, these proposals received little interest at that time. The limitations of the common psychological research methods became increasingly apparent over the years and we began developing the KPU Registry (2012). As we were starting to send notices that the registry was open, a group of articles was published (Pashler and Wagenmakers, 2012) that significantly increased awareness of the need for these practices. Discussions of study registration now usually focus on how registration should be done rather than whether registration is beneficial. In the present paper we make several recommendations for avoiding pitfalls and obtaining the full benefits of study registration.