A systematic and critical response to Pendrous et al. (2020) replication study

Abstract

Conducting direct replication studies is crucial for the progress of science because they increase our confidence in the effect of the independent variables under the same or mostly the same experimental conditions. Pendrous et al. (2020) recently published an “extended direct replication” with negative results concerning the study by Sierra et al. (2016) and suggested the disparity in results was due to the supposed more stringent conditions of their study. However, a detailed comparison of the studies reveals that (a) they differed in many relevant aspects (e.g., participants' characteristics, experimental task, procedure, and experimental protocols) that preclude considering Pendrous et al.'s study as a “direct replication,” (b) the replication study did not specify some methodological strengths of the original study, and (c) the replication study had unnoticed methodological problems. In the replication study: (a) there was an overrepresentation of females, (b) there were notable differences across experimental conditions in the naïve status of the participants in terms of previous ACT/RFT knowledge and experience with the cold pressor task, (c) 21.4% of the participants were not native English speakers, (d) compensation was not the same for all participants, and (e) there were differences in the pauses prompting for relational elaboration across the experimental conditions. These methodological problems might limit the conclusions reached in the replication study. We call for greater precision in reporting and discussing replication studies by highlighting the commonalities and differences between the original and replication studies.