Replication Crisis In Psychology Research Articles

Replication studies in engineering design – a feasibility study

AbstractThis paper examines the replicability of studies in design research triggered by the replication crisis in psychology. It highlights the importance of replicating studies to ensure the robustness of research results and examines whether the description in a publication is sufficient to replicate. Therefore, the publication of a reference study was analysed and a replication study was conducted. The design of the replication study appears similar to the reference study, but the results differ. Possible reasons for the differences and implications for replication studies are discussed.

Expanding the Interpretive Potential of Eye Tracking for the Rorschach: Replication and Extension of the Findings of Ales et al. (2020)

The present study was undertaken to determine if the findings of Ales et al. (2020) could be replicated and extended, especially in light of the replication crisis in psychology and the social sciences. Ales et al. (2020) found that measures from the engagement and cognitive processing domain of the Rorschach performance assessment system (R-PAS) were associated with Eye Tracking variables that reflect cognitive engagement and effort. Notably, Complexity was associated with the number of fixations participants made while scanning the blots and Vg% was inversely associated with a participant’s average fixation durations. The present study utilized a non-clinical sample of 60 adult participants. The basic findings of Ales et al. (2020) were replicated. In addition, we found that Complexity and Vg% are associated with additional Eye Tracking variables not utilized in the original study. The current findings bolster and extend the interpretation of Ales et al. (2020), indicating that higher levels of Complexity are also associated with scanning more regions of the blot overall, albeit at a slower rate. Similar effect size values were observed in the two different cultural contexts. Higher levels of Vg% are associated with measures indicating shallower and more superficial search strategies, consistent with the interpretation of Vg% as indicative of a vague, impressionistic, and unsophisticated cognitive style.

Dynamic Structural Equation Models: Promising Yet Concerning

Dynamic Structural Equation Model (DSEM) is a powerful statistical modeling approach that has recently gained popularity among researchers studying intensive longitudinal data. Despite its exciting potential, the stability and replicability of DSEM is yet to be closely examined. This study empirically investigates DSEM using recently published data to explore its strengths and potential limitations. The results show that while some of its parameter estimates are stable, others are characterized by substantial variation as a function of seemingly innocuous initial model estimation conditions. Indeed, some parameters fluctuate between significance and non-significance for the same model estimated using the same data. The instability of DSEM estimates poses a serious threat to the internal and external validity of conclusions drawn from its analyses, challenging the reproducibility of findings from applied research. Given the recent focus on the replication crisis in psychology, it is critical to address these issues as the popularity of DSEM in psychological research continues to rise. Several potential solutions are investigated to address this problem and recommendations of best practice are offered to applied researchers who plan to use DSEM in intensive longitudinal data analysis. KEYWORDS: Dynamic Structural Equation Model; Bayesian; Robust Estimation; Intensive Longitudinal Data

On the association between personality, fear of missing out (FoMO) and problematic social media use tendencies in European and Arabian samples

Recent meta-analytical evidence indicates a mild association between higher neuroticism and lower conscientiousness scores and a tendency towards problematic social media use (PSMU). However, fear of missing out (FoMO) has emerged as a critical variable in understanding the positive link between neuroticism and PSMU. Given the replication crisis in psychology, this study aimed to reinvestigate personality-PSMU associations and, crucially, the less-studied FoMO mediation effect. To ensure generalizability of the findings, we recruited two diverse samples with European and Arabian backgrounds. The results revealed a significant total effect of neuroticism on PSMU for both cultural groups, with the European sample demonstrating a fully mediated effect via FoMO, whereas the Arab sample showed a partially mediated effect via FoMO, along with a significant direct effect. This study contributes to the existing literature by highlighting the importance of FoMO as a mediator between neuroticism and PSMU and some minor potential cultural differences in this association.

Extraordinary Claims in the Literature on High-Intensity Interval Training (HIIT): I. Bonafide Scientific Revolution or a Looming Crisis of Replication and Credibility?

The literature on high-intensity interval training (HIIT) contains claims that, if true, could revolutionize the science and practice of exercise. This critical analysis examines two varieties of claims: (i) HIIT is effective in improving various indices of fitness and health, and (ii) HIIT is as effective as more time-consuming moderate-intensity continuous exercise. Using data from two recent systematic reviews as working examples, we show that studies in both categories exhibit considerable weaknesses when judged through the prism of fundamental statistical principles. Predominantly, small-to-medium effects are investigated in severely underpowered studies, thus greatly increasing the risk of both type I and type II errors of statistical inference. Studies in the first category combine the volatility of estimates associated with small samples with numerous dependent variables analyzed without consideration of the inflation of the type I error rate. Studies in the second category inappropriately use the p > 0.05 criterion from small studies to support claims of 'similar' or 'comparable' effects. It is concluded that the situation in the HIIT literature is reminiscent of the research climate that led to the replication crisis in psychology. As in psychology, this could be an opportunity to reform statistical practices in exercise science.

Direct Replication in Psychological Assessment Research

Direct Replication in Psychological Assessment Research

Remarkably reproducible psychological (memory) phenomena in the classroom: some evidence for generality from small-N research

BackgroundMainstream psychology is experiencing a crisis of confidence. Many of the methodological solutions offered in response have focused largely on statistical alternatives to null hypothesis statistical testing, ignoring nonstatistical remedies that are readily available within psychology; namely, use of small-N designs. In fact, many classic memory studies that have passed the test of replicability used them. That methodological legacy warranted a retrospective look at nonexperimental data to explore the generality of the reported effects.MethodVarious classroom demonstrations were conducted over multiple semesters in introductory psychology courses with typical, mostly freshman students from a predominantly white private Catholic university in the US Midwest based on classic memory experiments on immediate memory span, chunking, and depth of processing.ResultsStudents tended to remember 7 ± 2 digits, remembered more digits of π following an attached meaningful story, and remembered more words after elaborative rehearsal than after maintenance rehearsal. These results amount to replications under uncontrolled classroom environments of the classic experiments originally conducted largely outside of null hypothesis statistical testing frameworks.ConclusionsIn light of the ongoing replication crisis in psychology, the results are remarkable and noteworthy, validating these historically important psychological findings. They are testament to the reliability of reproducible effects as the hallmark of empirical findings in science and suggest an alternative approach to commonly proffered solutions to the replication crisis.

Power or Alpha? The Better Way of Decreasing the False Discovery Rate

The replication crisis in psychology has led to an increased concern regarding the false discovery rate (FDR) – the proportion of false positive findings among all significant findings. In this article, we compare two previously proposed solutions for decreasing the FDR: increasing statistical power and decreasing significance level α. First, we provide an intuitive explanation for α, power, and FDR to improve the understanding of these concepts. Second, we investigate the relationship between α and power. We show that for decreasing FDR, reducing α is more efficient than increasing power. We suggest that researchers interested in reducing the FDR should decrease α rather than increase power. By investigating the relative importance of both α level and power, we connect the literature on these topics and our results have implications for increasing the reproducibility of psychological science.

Evidence for a single rather than a triple dissociation in the medial temporal lobe: An fMRI recognition memory replication study

Common research practices in neuroimaging studies using functional magnetic resonance imaging may produce outcomes that are difficult to replicate. Results that cannot be replicated have contributed to a replication crisis in psychology, neuroscience, and other disciplines over the years. Here we replicate two previous papers in which the authors present two analysis paths for a dataset in which participants underwent fMRI while performing a recognition memory test for old and new words. Both studies found activation in the medial temporal lobe including the hippocampus, with the first demonstrating a distinction in activation corresponding to true and perceived oldness of stimuli and the second demonstrating that activation reflects the subjective experience of the participant. We replicated the behavioral and MRI acquisition parameters reported in the two target articles (Daselaar et al., 2006; Daselaar et al., 2006) with N = 53 participants. We focused fMRI analyses on regions of interest reported in the target articles examining fMRI activation for differences corresponding with true and perceived oldness and those associated with the subjective memory experiences of recollection, familiarity, and novelty. Comparisons between true and perceived oldness revealed main effects not only for true, but also perceived oldness along with a significant interaction. We replicate the findings of recollection and familiarity signals in the hippocampus and medial temporal lobe cortex, respectively, but failed to replicate a novelty signal in the anterior medial temporal lobe. These results remained when we analyzed only correct trials, indicating that the effects were not due to selectively averaging correct and incorrect trials. Taken together, our findings demonstrate that activation in the hippocampus corresponds to the subjective experience associated with correct recognition memory retrieval.

Replicable quantitative psychological and educational research: Possibility or pipe dream?

ABSTRACT Since the advent of the twenty-first century, science has experienced a crisis pertaining to the replicability of quantitative research findings, which has become known as the ‘replication crisis’. The replication crisis has particularly afflicted research in the behavioural sciences, and psychology in particular. Given the relevance of psychology to education, it is unsurprising that the replication crisis also presents an issue for quantitative educational research, thus potentially compromising its practical usefulness. This article outlines the replication crisis in psychology, and highlights its significance for quantitative educational research. Greater methodological rigour, aimed at addressing methodological deficiencies in the conduct of scientific research, has been suggested as a response to the replication crisis. Following a review of various calls for greater methodological rigour, the current article argues that, whilst methodological deficiencies may be a contributory factor, there is potentially a more fundamental reason for the replication crisis. The case is made that the non-separability of a measured attribute from the measurement instrument, and the irreducible uncertainty in unmeasured attributes, may be the principal reasons for the replication crisis in psychology and education. Implications of such an explanation for the crisis are articulated in relation to the evidence-based policy agenda in education.

How Do Researchers in Psychology Perceive the Field? A Qualitative Exploration of Critiques and Defenses

As awareness of the replication crisis in psychology has become increasingly widespread, several meta-scientific investigations have focused on the research practices and attitudes of researchers in psychology. Here, we aimed to add to this body of work by exploring academic psychologists’ perceptions of the state of the field using both quantitative and qualitative approaches. As part of a larger project, psychological researchers (N = 548) used 3-point scales to rate their perceptions of: 1) the rate of false positive findings in psychology and 2) the quality of research practices in psychology. They then wrote about the reasons for their ratings. Using a qualitative approach, we assessed the prevalence of criticisms and defenses of the field, as well as subtypes of each. Overall, these data shed light on the extent, and nature, of concerns about false positives and research practices within the psychological community.

Is forensic science in crisis?

The results of forensic science are believed to be reliable, and are widely used in support of verdicts around the world. However, due to the lack of suitable empirical studies, we actually know very little about the reliability of such results. In this paper, I argue that phenomena analogous to the main culprits for the replication crisis in psychology (questionable research practices, publication bias, or funding bias) are also present in forensic science. Therefore forensic results are significantly less reliable than is commonly believed. I conclude that in order to obtain reliable estimates for the reliability of forensic results, we need to conduct studies analogous to the large-scale replication projects in psychology. Additionally, I point to some ways for improving the reliability of forensic science, inspired by the reforms proposed in response to the replicability crisis.

Metascience Is Not Enough – A Plea for Psychological Humanities in the Wake of the Replication Crisis

The replication crisis led to the rise of metascience as a possible solution. In this article, we examine central metascientific premises and argue that attempts to solve the replication crisis in psychology will benefit from a tighter integration of approaches from the psychological humanities. The first part of our article identifies central epistemic merits that metascientific endeavors can contribute to psychology. However, we argue secondly against the widespread claim that metascience is the only way to deal with the replication crisis in psychology and point to major epistemic problems: the one-sided notion of a singular scientific method, the homogenizing view of psychology, and the exclusion of practices of theorizing. As a possible compensation for such shortcomings, we introduce, third, the reflective and pluralistic approach of psychological humanities. In so doing, we show how psychological humanities can serve as an important complement to the objective of improving psychological research. Psychological humanities contribute to a more precise determination of validity, to ethical considerations, and a better understanding of psychology’s objects in regard to replication. Accordingly, we argue for the integration of psychological humanities into both metascience and psychology to provide a better basis for addressing epistemic and ethical questions.

Introduction: Replication of Crises - Interdisciplinary Reflections on the Phenomenon of the Replication Crisis in Psychology

The replication crisis has preoccupied psychology for over a decade and has led to many reform proposals. In this Special Issue, we argue that a reflexive discussion of both the replication crisis and possible reforms is crucial. With the plural ‘replication of crises’ in the title, we want to make clear that the current crisis is more than one. What is perceived as a crisis varies depending on the scientific field, theoretical background, and epistemological perspective. As a consequence, this Special Issue aims to promote both an intra-disciplinary dialogue between scientific and theoretical psychology, and an inter-disciplinary dialogue between psychology, the humanities and the social sciences. The individual contributions focus on three central questions: (1) What is specific about the replication crisis in psychology? (2) What are the connections between the replication crisis in psychology and that in other scientific fields? (3) What are possible underlying causes of the replication crisis in psychology, and what are the opportunities for improvement? Although each of the articles offers a unique and sometimes challenging perspective to understanding the replication crisis, they all share the assumption that we need to reflect in order to learn and improve.

Replication and Reproduction: Crises in Psychology and Academic Labour

Discussions of the replication crisis in psychology require more substantive analysis of the crisis of academic labour and of social reproduction in the university. Both the replication crisis and the crisis of social reproduction in the university describe a failure in processes of reproducing something. The financial crisis of 2007–8 shortly preceded the emergence of the replication crisis, as well as exacerbated ongoing tendencies in the organisation and practices of university research (particularly the use of precarious contracts and the adjunctification of research). These provide two reasons to address these two named crises together. But many analyses of and responses to the replication crisis turn to research culture, often at the expense of adequate investigations of research labour. Today’s psychological sciences are made through multiple forms of labour: these include researchers, who range from senior principal investigators to sub-contracted, and exploited, research assistants; research participants/subjects, who include those providing labour for experiments via exploitative platforms including Amazon Mechanical Turk; and workers providing heterogeneous technical and administrative labour. Through understanding what is at stake for these multiple forms of labour, psychology might better analyse problems besetting psychology today, as well as develop different imaginaries and practices for how to address them.

Ad hominem rhetoric in scientific psychology.

Ad hominem discourse is largely prohibited in scientific journals. Historically, this prohibition restricted the dissemination of ad hominem discussion, but during the last decade, blogs and social media platforms became popular among researchers. With the use of social media now entrenched among researchers, there are important questions about the role of ad hominems. Ad hominems and other forms of strong criticism became particularly evident in online discussions associated with the recent replication crisis in psychology. Here, these discussions, and a few incidences of ad hominems in journal articles, are situated in the broader history of science. It is argued that explicit codes of conduct should be considered to curb certain kinds of ad hominem comments in certain fora, but that some ad hominem discussions have an important role to play in a healthier science.

Psychology Exceptionalism and the Multiple Discovery of the Replication Crisis

This article outlines what we call the “narrative of psychology exceptionalism” in commentaries on the replication crisis: many thoughtful commentaries link the current crisis to the specificity of psychology’s history, methods, and subject matter, but explorations of the similarities between psychology and other fields are comparatively thin. Historical analyses of the replication crisis in psychology further contribute to this exceptionalism by creating a genealogy of events and personalities that shares little in common with other fields. We aim to rebalance this narrative by examining the emergence and evolution of replication discussions in psychology alongside their emergence and evolution in biomedicine. Through a mixed-methods analysis of commentaries on replication in psychology and the biomedical sciences, we find that these conversations have, from the early years of the crisis, shared a common core that centers on concerns about the effectiveness of traditional peer review, the need for greater transparency in methods and data, and the perverse incentive structure of academia. Drawing on Robert Merton’s framework for analyzing multiple discovery in science, we argue that the nearly simultaneous emergence of this narrative across fields suggests that there are shared historical, cultural, or institutional factors driving disillusionment with established scientific practices.

The Tone Debate: Knowledge, Self, and Social Order

In the replication crisis in psychology, a “tone debate” has developed. It concerns the question of how to conduct scientific debate effectively and ethically. How should scientists give critique without unnecessarily damaging relations? The increasing use of Facebook and Twitter by researchers has made this issue especially pressing, as these social technologies have greatly expanded the possibilities for conversation between academics, but there is little formal control over the debate. In this article, we show that psychologists have tried to solve this issue with various codes of conduct, with an appeal to virtues such as humility, and with practices of self-transformation. We also show that the polemical style of debate, popular in many scientific communities, is itself being questioned by psychologists. Following Shapin and Schaffer’s analysis of the ethics of Robert Boyle’s experimental philosophy in the 17th century, we trace the connections between knowledge, social order, and subjectivity as they are debated and revised by present-day psychologists.

“The replication crisis in psychology: An overview for theoretical and philosophical psychology”: Correction to Wiggins and Christopherson (2019).

“The replication crisis in psychology: An overview for theoretical and philosophical psychology”: Correction to Wiggins and Christopherson (2019).

Morpheme Position Coding in Reading Development as Explored With a Letter Search Task.

Suffixes have been shown to be recognized as units of processing in visual word recognition and their identification has been argued to be position-specific in skilled adult readers: in lexical decision tasks suffixes are automatically identified at word endings, but not at word beginnings. The present study set out to investigate whether position-specific coding can be detected with a letter search task and whether children already code suffixes as position-specific units. A preregistered experiment was conducted in Italian in which 3rd-graders, 5th-graders, and adults had to detect a target letter that was either contained in the suffix of a pseudoword (e.g., S in flagish) or in a non-suffix control (e.g., S in flagosh). To investigate sensitivity to position, letters also had to be detected in suffixes and non-suffixes placed in reversed position, that is in the beginning of pseudowords (e.g., S in ishflag vs. oshflag). Results suggested position-specific processing differences between suffixes and non-suffixes that develop throughout reading development. However, some effects were weak and only partially compatible with the hypotheses. Therefore, a second experiment was conducted. The effects of position-specific suffix identification could not be replicated. A combined analysis additionally using a Bayesian approach indicated no processing differences between suffixes and non-suffixes in our task. We discuss potential interpretations and the possibility of letter search being unsuited to investigate morpheme processing. We connect our example of failed self-replication to the current discussion about the replication crisis in psychology and the lesson psycholinguistics can learn.