Replication Crisis Research Articles

Conflicting Results and Statistical Malleability: Embracing Pluralism of Empirical Results

Abstract Conflicting results undermine making inferences from the empirical literature. So far, the replication crisis is mainly seen as resulting from honest errors and questionable research practices such as p-hacking or the base-rate fallacy. I discuss the malleability (researcher degrees of freedom) of quantitative research and argue that conflicting results can emerge from two studies using different but plausible designs (e.g., eligibility criteria, operationalization of concepts, outcome measures) and statistical methods. I also explore how the choices regarding study design and statistical techniques bias results in a way that makes them more or less relevant for a given policy or clinical question.

From code sharing to sharing of implementations: Advancing reproducible AI development for medical imaging through federated testing

BackgroundThe reproducibility crisis in AI research remains a significant concern. While code sharing has been acknowledged as a step toward addressing this issue, our focus extends beyond this paradigm. In this work, we explore “federated testing” as an avenue for advancing reproducible AI research and development especially in medical imaging. Unlike federated learning, where a model is developed and refined on data from different centers, federated testing involves models developed by one team being deployed and evaluated by others, addressing reproducibility across various implementations. MethodsOur study follows an exploratory design aimed at systematically evaluating the sources of discrepancies in shared model execution for medical imaging and outputs on the same input data, independent of generalizability analysis. We distributed the same model code to multiple independent centers, monitoring execution in different runtime environments while considering various real-world scenarios for pre- and post-processing steps. We analyzed deployment infrastructure by comparing the impact of different computational resources (GPU vs. CPU) on model performance. To assess federated testing in AI models for medical imaging, we performed a comparative evaluation across different centers, each with distinct pre- and post-processing steps and deployment environments, specifically targeting AI-driven positron emission tomography (PET) imaging segmentation. More specifically, we studied federated testing for an AI-based model for surrogate total metabolic tumor volume (sTMTV) segmentation in PET imaging: the AI algorithm, trained on maximum intensity projection (MIP) data, segments lymphoma regions and estimates sTMTV. ResultsOur study reveals that relying solely on open-source code sharing does not guarantee reproducible results due to variations in code execution, runtime environments, and incomplete input specifications. Deploying the segmentation model on local and virtual GPUs compared to using Docker containers showed no effect on reproducibility. However, significant sources of variability were found in data preparation and pre-/post- processing techniques for PET imaging. These findings underscore the limitations of code sharing alone in achieving consistent and accurate results in federated testing. ConclusionAchieving consistently precise results in federated testing requires more than just sharing models through open-source code. Comprehensive pipeline sharing, including pre- and post-processing steps, is essential. Cloud-based platforms that automate these processes can streamline AI model testing across diverse locations. Standardizing protocols and sharing complete pipelines can significantly enhance the robustness and reproducibility of AI models.

Type I Error Rates are Not Usually Inflated

The inflation of Type I error rates is thought to be one of the causes of the replication crisis. Questionable research practices such as p-hacking are thought to inflate Type I error rates above their nominal level, leading to unexpectedly high levels of false positives in the literature and, consequently, unexpectedly low replication rates. In this article, I offer an alternative view. I argue that questionable and other research practices do not usually inflate relevant Type I error rates. I begin by introducing the concept of Type I error rates and distinguishing between statistical errors and theoretical errors. I then illustrate my argument with respect to model misspecification, multiple testing, selective inference, forking paths, exploratory analyses, p-hacking, optional stopping, double dipping, and HARKing. In each case, I demonstrate that relevant Type I error rates are not usually inflated above their nominal level, and in the rare cases that they are, the inflation is easily identified and resolved. I conclude that the replication crisis may be explained, at least in part, by researchers’ misinterpretation of statistical errors and their underestimation of theoretical errors.

Biomedical researchers' perspectives on the reproducibility of research.

We conducted an international cross-sectional survey of biomedical researchers' perspectives on the reproducibility of research. This study builds on a widely cited 2016 survey on reproducibility and provides a biomedical-specific and contemporary perspective on reproducibility. To sample the community, we randomly selected 400 journals indexed in MEDLINE, from which we extracted the author names and emails from all articles published between October 1, 2020 and October 1, 2021. We invited participants to complete an anonymous online survey which collected basic demographic information, perceptions about a reproducibility crisis, perceived causes of irreproducibility of research results, experience conducting reproducibility studies, and knowledge of funding and training for research on reproducibility. A total of 1,924 participants accessed our survey, of which 1,630 provided useable responses (response rate 7% of 23,234). Key findings include that 72% of participants agreed there was a reproducibility crisis in biomedicine, with 27% of participants indicating the crisis was "significant." The leading perceived cause of irreproducibility was a "pressure to publish" with 62% of participants indicating it "always" or "very often" contributes. About half of the participants (54%) had run a replication of their own previously published study while slightly more (57%) had run a replication of another researcher's study. Just 16% of participants indicated their institution had established procedures to enhance the reproducibility of biomedical research and 67% felt their institution valued new research over replication studies. Participants also reported few opportunities to obtain funding to attempt to reproduce a study and 83% perceived it would be harder to do so than to get funding to do a novel study. Our results may be used to guide training and interventions to improve research reproducibility and to monitor rates of reproducibility over time. The findings are also relevant to policy makers and academic leadership looking to create incentives and research cultures that support reproducibility and value research quality.

Balance of power: The choice between trial and participant numbers to optimise the detection of phase-dependent effects

Abstract The fields of neuroscience and psychology are currently in the midst of a so-called reproducibility crisis, with growing concerns regarding a history of weak effect sizes and low statistical power in much of the research published in these fields over the last few decades. Whilst the traditional approach for addressing this criticism has been to increase participant sample sizes, there are many research contexts in which the number of trials per participant may be of equal importance. The present study aimed to compare the relative importance of participants and trials in the detection of phase-dependent phenomena, which are measured across a range of neuroscientific contexts (e.g., neural oscillations, non-invasive brain stimulation). This was achievable within a simulated environment where one can manipulate the strength of this phase dependency in two types of outcome variables: one with normally distributed residuals (idealistic) and one comparable with motor-evoked potentials (an MEP-like variable). We compared the statistical power across thousands of experiments with the same number of sessions per experiment but with different proportions of participants and number of sessions per participant (30 participants × 1 session, 15 participants × 2 sessions, and 10 participants × 3 sessions), with the trials being pooled across sessions for each participant. These simulations were performed for both outcome variables (idealistic and MEP-like) and four different effect sizes (0.075—“weak,” 0.1—“moderate,” 0.125—“strong,” 0.15—“very strong”), as well as separate control scenarios with no true effect. Across all scenarios with (true) discoverable effects, and for both outcome types, there was a statistical benefit for experiments maximising the number of trials rather than the number of participants (i.e., it was always beneficial to recruit fewer participants but have them complete more trials). These findings emphasise the importance of obtaining sufficient individual-level data rather than simply increasing number of participants.

Trends in null hypothesis significance testing: Still going strong

Null hypothesis significance testing (NHST) is among the most prominent and widely used methods for analyzing data. At the same time, NHST has been criticized since many years because of misuses and misconceptions that can be found extensively in the scholarly literature. Furthermore, in recent years, NHST has been identified as one reason for the replication crisis because many studies place too much emphasis on statistical significance for drawing conclusions. As a response to those problems, calls for actions have been raised, among others by the American Statistical Association (ASA), to rectify these issues, for instance, by modifying or even abandoning NHST. In this paper, we study the reaction of the community on these discussions. Specifically, we conduct a scientometric analysis of bibliographic records to investigate the publication behavior about the usage of NHST. We conduct a trend analysis for the general community, for specific subject areas and for individual journals. Furthermore, we conduct a change-point analysis to investigate if there are continued movements or actual changes. As a result, we find that for the general community NHST is more popular than ever, however, for particular subject-areas and journals there is a clear heterogeneity and no uniform publication behavior is observable.

A libraries reproducibility hackathon: connecting students to University research and testing the longevity of published code

Reproducibility is a basis of scientific integrity, yet it remains a significant challenge across disciplines in computational science. This reproducibility crisis is now being met with an Open Science movement, which has risen to prominence within the scientific community and academic libraries especially. To address the need for reproducible computational research and promote Open Science within the community, members of the Open Science and Data Collaborations Program at Carnegie Mellon University Libraries organized a single-day hackathon centered around reproducibility. Partnering with a faculty researcher in English and Digital Humanities, this event allowed several students an opportunity to interact with real research outputs, test the reproducibility of data analyses with code, and offer feedback for improvements. With Python code and data shared by the researcher in an open repository, we revealed that students could successfully reproduce most of the data visualizations, but they required completing some manual setup and modifications to address depreciated libraries to successfully rerun the code. During the event, we also investigated the option of using ChatGPT to debug and troubleshoot rerunning this code. By interacting with a ChatGPT API in the code, we found and addressed the same roadblocks and successfully reproduced the same figures as the participating students. Assessing a second option, we also collaborated with the researcher to publish a compute capsule in Code Ocean. This option presented an alternative to manual setup and modifications, an accessible option for more limited devices like tablets, and a simple solution for outside researchers to modify or build on existing research code.

Visualizemi: Visualization, Effect Size, and Replication of Measurement Invariance for Registered Reports.

Latent variable modeling as a lens for psychometric theory is a popular tool for social scientists to examine measurement of constructs. Journals, such as Assessment regularly publish articles supporting measures of latent constructs wherein a measurement model is established. Confirmatory factor analysis can be used to investigate the replicability and generalizability of the measurement model in new samples, while multigroup confirmatory factor analysis is used to examine the measurement model across groups within samples. With the rise of the replication crisis and "psychology's renaissance," interest in divergence in measurement has increased, often focused on small parameter differences within the latent model. This article presents visualizemi, an R package that provides functionality to calculate multigroup models, partial invariance, visualizations for (non)-invariance, effect sizes for models and parameters, and potential replication rates compared with random models. Readers will learn how to interpret the impact and size of the proposed non-invariance in models with a focus on potential replication and how to plan for registered reports.

Animal researchers' views on the publication of negative results and subsequent policy adoptions.

For over a decade, the non-publication of negative results from preclinical studies has been identified as a significant concern in biomedical research. Such underreporting is considered a contributor to the reproducibility crisis in the field and has been recognized by significant journals such as Science and Nature. In response to the consistently high non-publication rates of preclinical animal research in Europe, a survey was conducted among the biomedical research community to gather their views on publishing negative results. Using the EUSurvey platform, over 200 researchers directly working with animals were surveyed. The study aimed to understand the frequency of negative results, the reasons behind their non-publication, and the perceived pros and cons of making such results public. Insights from the survey could guide steps toward promoting transparency in science, refining research methodologies, reducing animal usage in experiments and minimizing research waste.

Introduction: Replicating John Hedley Brooke’s Work on the History of Science and Religion

In this introduction, we first briefly describe the replication crisis as it occurred primarily in the biomedical and social sciences. We then argue for the possibility and desirability of replication in the humanities, particularly historiography. After that, we clarify why we opted for the replication of John Hedley Brooke’s 1991 book Science and Religion: Some Historical Perspectives, specifically its third chapter. We elucidate why we chose to do both a direct and a conceptual replication. Finally, we lay out the procedures we used to conduct the two replication studies and look ahead to what is to come in this thematic section.

Asymmetry in Galaxy Spin Directions: A Fully Reproducible Experiment Using HSC Data

The asymmetry in the large-scale distribution of the directions in which spiral galaxies rotate has been observed by multiple telescopes, all showing a consistent asymmetry in the distribution of galaxy spin directions as observed from Earth. Here, galaxies with a redshift from HSC DR3 are annotated by their direction of rotation, and their distribution is analyzed. The results show that galaxies that rotate in the opposite direction relative to the Milky Way as observed from Earth are significantly more prevalent compared to galaxies that rotate in the same direction relative to the Milky Way. The asymmetry also forms a dipole axis that becomes stronger when the redshift gets higher. These results are aligned with observations from virtually all premier digital sky surveys, as well as space telescopes such as the HST and the JWST. This shows that the distribution of galaxy spin directions as observed from Earth is not symmetrical, and has a possible link to the rotational velocity of the Milky Way. This experiment provides data, code, and a full protocol that allows the results to be easily reproduced in a transparent manner. This practice is used to overcome the “reproducibility crisis” in science.

Changing Human Behavior to Conserve Biodiversity

Conservation of biodiversity is above all else an exercise in human persuasion. Human behavior drives all substantive threats to biodiversity; therefore, influencing it is the only path to mitigating the current extinction crisis. We review the literature across three different axes to highlight current evidence on influencing human behavior for conservation. First, we look at behavioral interventions to mitigate different threats, from pollution and climate change to invasive species and human disturbance. Next, we examine interventions focused on different stakeholders, from voters, investors, and environmental managers to consumers, producers, and extractors. Finally, we review delivery channels, ranging from mass and social media to interventions involving changes to the physical environment or carried out in person. We highlight key gaps, including the lack of scale and robust impact evaluation of most interventions, and the need to prioritize behaviors, overcome the reproducibility crisis, and deal with inequality when designing and implementing behavior change interventions.

The prevalence of direct replication articles in top-ranking psychology journals.

Despite lip service about replication being a cornerstone of science, replications have historically received little real estate in the published literature. Following psychology's recent replication crisis, we assessed the prevalence of one type of replication contribution: direct replication articles-articles where a direct or close replication of a previously published study is one of the main contributions of the article. This prevalence provides one indicator of how much the field values and incentivizes this type of self-correction. We used a keyword search combined with manual checking to identify direct replication articles that were published from 2010 to 2021 in the 100 highest impact psychology journals. In total, only 0.2% of articles (169 articles out of 84,834) were direct replication articles. There was a small suggestive increase in the prevalence of direct replication articles over time. Additionally, journals with a stated policy of considering replication submissions (31% of journals) were 7.85 times more likely to publish direct replication articles than those without such a policy. Fifty-four out of 88 journals did not publish any direct replication articles in the 11 years surveyed. Our estimate is not the same as the prevalence of direct replication studies overall (direct replication results can be shared in many ways other than as direct replication articles in top journals). Ultimately, direct replication articles are still rare, with a few journals doing most of the heavy lifting. Based on these findings, we argue it would be premature to declare that psychology's replication crisis is over. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Housing and Husbandry Factors Affecting Zebrafish (Danio rerio) Novel Tank Test Responses: A Global Multi-Laboratory Study.

The reproducibility crisis in bioscience, characterized by inconsistent study results, impedes our understanding of biological processes and global collaborative studies offer a unique solution. This study is the first global collaboration using the zebrafish (Danio rerio) novel tank test, a behavioral assay for anxiety-like responses. We analyzed data from 20 laboratories worldwide, focusing on housing conditions and experimental setups. Our study included 488 adult zebrafish, tested for 5 min, focusing on a variety of variables. Key findings show females exhibit more anxiety-like behavior than males, underscoring sex as a critical variable. Housing conditions, including higher stocking densities and specific feed types, influenced anxiety levels. Optimal conditions (5 fish/L) and nutritionally rich feeds (e.g., rotifers), mitigated anxiety-like behaviors. Environmental stressors, like noise and transportation, significantly impacted behavior. We recommend standardizing protocols to account for sex differences, optimal stocking densities, nutritionally rich feeds, and minimizing stressors to improve zebrafish behavioral study reliability.

Giving Away the Science of School Psychology

Within the field of school psychology, we often put a premium on evidence-based practices, but what is often missing from these conversations is a consideration of replication and open science principles in the evaluation of evidence. Amid growing concerns regarding the replicability of psychological science, calls for greater research transparency, and more careful consideration of the conditions under which a finding might replicate, research suggest that school psychologists’ uptake of open science practices may lag behind that of other social science disciplines. The present paper synthesizes the literature on the replication crisis and corresponding credibility revolution in psychology, describes various open science practices, and considers barriers and adaptations needed to implement these practices in school-based research. The paper ends with recommendations on how to better give away the science of school psychology via transparent research practices so that research consumers can better reap the benefits of evidence-based practice. Impact Statement This paper summarizes the literature on open science, including key concepts relevant to the replication crisis and corresponding credibility revolution. These concepts are then made applicable to the field of school psychology, including our roles as researchers, practitioners, research consumers, and publication editors. By engaging in more transparent research practices, school psychologists can promote a more replicable, reputable, and reliable science, which yields greater benefits and buy-in for the communities we serve.

Quantifying convergence and consistency.

The reproducibility crisis highlights several unresolved issues in science, including the need to develop measures that gauge both the consistency and convergence of data sets. While existing meta-analytic methods quantify the consistency of evidence, they do not quantify its convergence: the extent to which different types of empirical methods have provided evidence to support a hypothesis. To address this gap in meta-analysis, we and colleagues developed a summary metric-the cumulative evidence index (CEI)-which uses Bayesian statistics to quantify the degree of both consistency and convergence of evidence regarding causal hypotheses between two phenomena. Here, we outline the CEI's underlying model, which quantifies the extent to which studies of four types-positive intervention, negative intervention, positive non-intervention and negative non-intervention-lend credence to any of three types of causal relations: excitatory, inhibitory or no-connection. Along with p-values and other measures, the CEI can provide a more holistic perspective on a set of evidence by quantitatively expressing epistemic principles that scientists regularly employ qualitatively. The CEI can thus address the reproducibility crisis by formally demonstrating how convergent evidence across multiple study types can yield progress toward scientific consensus, even when an individual type of study fails to yield reproducible results.

Reproducibility in the Classroom

Difficulties in reproducing results from scientific studies have lately been referred to as a reproducibility crisis. Scientific practice depends heavily on scientific training. What gets taught in the classroom is often practiced in labs, fields, and data analysis. The importance of reproducibility in the classroom has gained momentum in statistics education in recent years. In this article, we review the existing literature on reproducibility education. We delve into the relationship between computing tools and reproducibility through visiting historical developments in this area. We share examples for teaching reproducibility and reproducible teaching while discussing the pedagogical opportunities created by these examples as well as challenges that the instructors should be aware of. We detail the use of teaching reproducibility and reproducible teaching practices in an introductory data science course. Lastly, we provide recommendations on reproducibility education for instructors, administrators, and other members of the scientific community.

Statistics in Service of Metascience: Measuring Replication Distance with Reproducibility Rate.

Motivated by the recent putative reproducibility crisis, we discuss the relationship between the replicability of scientific studies, the reproducibility of results obtained in these replications, and the philosophy of statistics. Our approach focuses on challenges in specifying scientific studies for scientific inference via statistical inference and is complementary to classical discussions in the philosophy of statistics. We particularly consider the challenges in replicating studies exactly, using the notion of the idealized experiment. We argue against treating reproducibility as an inherently desirable property of scientific results, and in favor of viewing it as a tool to measure the distance between an original study and its replications. To sensibly study the implications of replicability and results reproducibility on inference, such a measure of replication distance is needed. We present an effort to delineate such a framework here, addressing some challenges in capturing the components of scientific studies while identifying others as ongoing issues. We illustrate our measure of replication distance by simulations using a toy example. Rather than replications, we present purposefully planned modifications as an appropriate tool to inform scientific inquiry. Our ability to measure replication distance serves scientists in their search for replication-ready studies. We believe that likelihood-based and evidential approaches may play a critical role towards building statistics that effectively serve the practical needs of science.

The utility of high-dosage experiments in everyday life to test theories in clinical science.

This viewpoint article discusses the utility of high-dosage experiments (HDEs) in everyday life to test theories in clinical science. HDEs involve experimental manipulations and assessments that occur over much longer periods of time than traditional experiments-generally days or even weeks. By nature, they also occur outside the lab, in the everyday environments of participants. Additionally, as with other experiments, the purpose of the study is concealed from participants. Experimental design is one of the most distinguishable characteristics of psychology that separates it from other behavioral sciences. Studies that rely on experiments are essential for theory testing and establishing the potential causal role of mechanisms that underlie psychopathology. Yet despite the value of experimental research, experimental studies are not currently given special prominence in clinical psychological science. For example, in the Journal of Psychopathology and Clinical Science, of all the empirical studies in the most recent year (2023), only three of 77 incorporated an experimental manipulation. Experimental research appears to be less popular in clinical psychology than in other fields, such as social psychology. What might account for this discrepancy? First, clinical samples are more difficult to recruit. This is important because experimental manipulations may produce small effects that require large samples for detection. Additionally, mechanisms hypothesized to underlie psychopathology are often chronic and intransigent. For example, cognitive factors (e.g., perfectionistic beliefs) could require an especially strong manipulation to modify in isolation. Researchers have argued that psychology has been experiencing a crisis in theory development. Eronen and Bringmann (2021) stated that one major reason for this crisis is the difficulty in establishing causal relationships between psychological constructs. The replication crisis has garnered even more attention (Open Science Collaboration, 2015). HDEs would help address these two crises and provide stronger and more replicable tests of theory. This could allow us to more precisely identify important mechanisms underlying psychopathology, potentially enhancing treatment efficacy, and enabling us to move the field forward. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Reproducible brain PET data analysis: easier said than done.

While a great deal of recent effort has focused on addressing a perceived reproducibility crisis within brain structural magnetic resonance imaging (MRI) and functional MRI research communities, this article argues that brain positron emission tomography (PET) research stands on even more fragile ground, lagging behind efforts to address MRI reproducibility. We begin by examining the current landscape of factors that contribute to reproducible neuroimaging data analysis, including scientific standards, analytic plan pre-registration, data and code sharing, containerized workflows, and standardized processing pipelines. We then focus on disparities in the current status of these factors between brain MRI and brain PET. To demonstrate the positive impact that further developing such reproducibility factors would have on brain PET research, we present a case study that illustrates the many challenges faced by one laboratory that attempted to reproduce a community-standard brain PET processing pipeline. We identified key areas in which the brain PET community could enhance reproducibility, including stricter reporting policies among PET dedicated journals, data repositories, containerized analysis tools, and standardized processing pipelines. Other solutions such as mandatory pre-registration, data sharing, code availability as a condition of grant funding, and online forums and standardized reporting templates, are also discussed. Bolstering these reproducibility factors within the brain PET research community has the potential to unlock the full potential of brain PET research, propelling it toward a higher-impact future.