P-Curve Analysis of the Köhler Motivation Gain Effect in Exercise Settings: A Demonstration of a Novel Technique to Estimate Evidential Value Across Multiple Studies.

Abstract

Practitioners and researchers may not always be able to adequately evaluate the evidential value of findings from a series of independent studies. This is partially due to the possibility of inflated effect size estimates for these findings as a result of researcher manipulation or selective reporting of analyses (i.e., p-hacking). In light of the possible overestimation of effect sizes in the literature, the p-curve analysis has been proposed as a worthwhile tool that may help identify bias across a series of studies focused on a single effect. The p-curve analysis provides a measure of the evidential value in the published literature and might highlight p-hacking practices. Therefore, the purpose of this paper is to introduce the mechanics of the p-curve analysis to individuals researching phenomena in the psychosocial aspects of behavior and provide a substantive example of a p-curve analysis using findings from a series of studies examining a group dynamic motivation gain paradigm. We performed a p-curve analysis on a sample of 13 studies that examined the Köhler motivation gain effect in exercise settings as a means to instruct readers how to conduct such an analysis on their own. The p-curve for studies examining the Köhler effect demonstrated evidential value and that this motivation effect is likely not a byproduct of p-hacking. The p-curve analysis is explained, as well as potential limitations of the analysis, interpretation of the results, and other uses where a p-curve analysis could be implemented.