Replication Crisis Research Articles

Bayesian Methods: A Means of Improving Statistical Power in Preclinical Neurotrauma?

The field of neurotrauma is grappling with the effects of the recently identified replication crisis. As such, care must be taken to identify and perform the most appropriate statistical analyses. This will prevent misuse of research resources and ensure that conclusions are reasonable and within the scope of the data. We anticipate that Bayesian statistical methods will see increasing use in the coming years. Bayesian methods integrate prior beliefs (or prior data) into a statistical model to merge historical information and current experimental data. These methods may improve the ability to detect differences between experimental groups (i.e., statistical power) when used appropriately. However, researchers need to be aware of the strengths and limitations of such approaches if they are to implement or evaluate these analyses. Ultimately, an approach using Bayesian methodologies may have substantial benefits to statistical power, but caution needs to be taken when identifying and defining prior beliefs.

Avoiding a reproducibility crisis in regulatory toxicology—on the fundamental role of ring trials

The ongoing transition from chemical hazard and risk assessment based on animal studies to assessment relying mostly on non-animal data, requires a multitude of novel experimental methods, and this means that guidance on the validation and standardisation of test methods intended for international applicability and acceptance, needs to be updated. These so-called new approach methodologies (NAMs) must be applicable to the chemical regulatory domain and provide reliable data which are relevant to hazard and risk assessment. Confidence in and use of NAMs will depend on their reliability and relevance, and both are thoroughly assessed by validation. Validation is, however, a time- and resource-demanding process. As updates on validation guidance are conducted, the valuable components must be kept: Reliable data are and will remain fundamental. In 2016, the scientific community was made aware of the general crisis in scientific reproducibility—validated methods must not fall into this. In this commentary, we emphasize the central importance of ring trials in the validation of experimental methods. Ring trials are sometimes considered to be a major hold-up with little value added to the validation. Here, we clarify that ring trials are indispensable to demonstrate the robustness and reproducibility of a new method. Further, that methods do fail in method transfer and ring trials due to different stumbling blocks, but these provide learnings to ensure the robustness of new methods. At the same time, we identify what it would take to perform ring trials more efficiently, and how ring trials fit into the much-needed update to the guidance on the validation of NAMs.

Experiments as Code and its application to VR studies in human-building interaction

Experiments as Code (ExaC) is a concept for reproducible, auditable, debuggable, reusable, & scalable experiments. Experiments are a crucial tool to understand Human-Building Interactions (HBI) and build a coherent theory around it. However, a common concern for experiments is their auditability and reproducibility. Experiments are usually designed, provisioned, managed, and analyzed by diverse teams of specialists (e.g., researchers, technicians, engineers) and may require many resources (e.g., cloud infrastructure, specialized equipment). Although researchers strive to document experiments accurately, this process is often lacking. Consequently, it is difficult to reproduce these experiments. Moreover, when it is necessary to create a similar experiment, the “wheel is very often reinvented”. It appears easier to start from scratch than trying to reuse existing work. Thus valuable embedded best practices and previous experiences are lost. In behavioral studies, such as in HBI, this has contributed to the reproducibility crisis. To tackle these challenges, we propose the ExaC paradigm, which not only documents the whole experiment, but additionally provides the automation code to provision, deploy, manage, and analyze the experiment. To this end, we define the ExaC concept, provide a taxonomy for the components of a practical implementation, and provide a proof of concept with an HBI desktop VR experiment that demonstrates the benefits of its “as code” representation, that is, reproducibility, auditability, debuggability, reusability, & scalability.

Post-selection Inference in Multiverse Analysis (PIMA): An Inferential Framework Based on the Sign Flipping Score Test.

When analyzing data, researchers make some choices that are either arbitrary, based on subjective beliefs about the data-generating process, or for which equally justifiable alternative choices could have been made. This wide range of data-analytic choices can be abused and has been one of the underlying causes of the replication crisis in several fields. Recently, the introduction of multiverse analysis provides researchers with a method to evaluate the stability of the results across reasonable choices that could be made when analyzing data. Multiverse analysis is confined to a descriptive role, lacking a proper and comprehensive inferential procedure. Recently, specification curve analysis adds an inferential procedure to multiverse analysis, but this approach is limited to simple cases related to the linear model, and only allows researchers to infer whether at least one specification rejects the null hypothesis, but not which specifications should be selected. In this paper, we present a Post-selection Inference approach to Multiverse Analysis (PIMA) which is a flexible and general inferential approach that considers for all possible models, i.e., the multiverse of reasonable analyses. The approach allows for a wide range of data specifications (i.e., preprocessing) and any generalized linear model; it allows testing the null hypothesis that a given predictor is not associated with the outcome, by combining information from all reasonable models of multiverse analysis, and provides strong control of the family-wise error rate allowing researchers to claim that the null hypothesis can be rejected for any specification that shows a significant effect. The inferential proposal is based on a conditional resampling procedure. We formally prove that the Type I error rate is controlled, and compute the statistical power of the test through a simulation study. Finally, we apply the PIMA procedure to the analysis of a real dataset on the self-reported hesitancy for the COronaVIrus Disease 2019 (COVID-19) vaccine before and after the 2020 lockdown in Italy. We conclude with practical recommendations to be considered when implementing the proposed procedure.

Perspectives of FTIR as Promising Tool for Pathogen Diagnosis, Sanitary and Welfare Monitoring in Animal Experimentation Models: A Review Based on Pertinent Literature.

Currently, there is a wide application in the literature of the use of the Fourier Transform Infrared Spectroscopy (FTIR) technique. This basic tool has also proven to be efficient for detecting molecules associated with hosts and pathogens in infections, as well as other molecules present in humans and animals' biological samples. However, there is a crisis in science data reproducibility. This crisis can also be observed in data from experimental animal models (EAMs). When it comes to rodents, a major challenge is to carry out sanitary monitoring, which is currently expensive and requires a large volume of biological samples, generating ethical, legal, and psychological conflicts for professionals and researchers. We carried out a survey of data from the relevant literature on the use of this technique in different diagnostic protocols and combined the data with the aim of presenting the technique as a promising tool for use in EAM. Since FTIR can detect molecules associated with different diseases and has advantages such as the low volume of samples required, low cost, sustainability, and provides diagnostic tests with high specificity and sensitivity, we believe that the technique is highly promising for the sanitary and stress and the detection of molecules of interest of infectious or non-infectious origin.

Knowing What We're Talking About

A theory crisis and measurement crisis have been argued to be root causes of psychology's replication crisis. In both, the lack of conceptual clarification and the jingle-jangle jungle at the construct definition level as well the measurement level play a central role. We introduce a conceptual tool that can address these issues: Decentralized Construct Taxonomy specifications (DCTs). These consist of comprehensive specifications of construct definitions, corresponding instructions for quantitative and qualitative research, and unique identifiers. We discuss how researchers can develop DCT specifications as well as how DCT specifications can be used in research, practice, and theory development. Finally, we discuss the implications and potential for future developments to answer the call for conceptual clarification and epistemic iteration. This contributes to the move towards a psychological science that progresses in a cumulative fashion through discussion and comparison.

Experimenter bias: exploring the interaction between participant’s and investigator’s gender/sex in VR

This study explores the effect of the experimenter’s gender/sex and its interaction with the participant’s gender/sex as potential contributors to the replicability crisis, particularly in the man-gendered domain of VR. 75 young men and women from Western France were randomly evaluated by either a man or a woman during a 13-min immersion in a first-person shooter game. Self-administered questionnaires were used to measure variables commonly assessed during VR experiments (sense of presence, cybersickness, video game experience, flow). MANOVAs, ANOVAs and post-hoc comparisons were used. Results indicate that men and women differ in their reports of cybersickness and video game experience when rated by men, whereas they report similar measures when rated by women. These findings are interpreted as consequences of the psychosocial stress triggered by the interaction between the two genders/sexes, as well as the gender conformity effect induced, particularly in women, by the presence of a man in a masculine domain. Corroborating this interpretation, the subjective measure of flow, which is not linked to video games and/or computers, does not seem to be affected by this experimental effect. Methodological precautions are highlighted, notably the brief systematic description of the experimenter, and future exploratory and confirmatory studies are outlined.Graphical abstract

Forcing a Deterministic Frame on Probabilistic Phenomena: A Communication Blind Spot in Media Coverage of the “Replication Crisis”

The current controversy surrounding research replication in biomedical and psychosocial sciences often overlooks the uncertainties surrounding both the original and replication studies. Overemphasizing single attempts as definitive replication successes or failures, as exemplified by media coverage of the landmark Reproducibility Project: Psychology, fosters misleading dichotomies and erodes public trust. To avoid such unintended consequences, science communicators should more clearly articulate statistical variation and other uncertainty sources in replication, while emphasizing the cumulative nature of science in general and replication in particular.

Hyper-ambition and the Replication Crisis: Why Measures to Promote Research Integrity can Falter

AbstractThis paper introduces the concept of ‘hyper-ambition’ in academia as a contributing factor to what has been termed a ‘replication crisis’ across some sciences. The replication crisis is an umbrella term that covers a range of ‘questionable research practices’, from sloppy reporting to fraud. There are already many proposals to address questionable research practices, some of which focus on the values, norms, and motivations of researchers and institutes, and suggest measures to promote research integrity. Yet it is not easy to promote integrity in hyper-competitive academic environments that value high levels of ambition. I argue that in such contexts, it is as likely that a kind of hyper-ambition is fostered that (inadvertently or otherwise) prioritises individual success above all, including to the detriment of scientific quality. In addition, efforts to promote values like integrity falter because they rely on sufficient uniformity in motivations or tendencies. Codes and guidance promoting integrity are, however, likely to influence those for whom such values are not optional, while others simply find ways around them. To demonstrate this I offer a thought experiment in which we consider the imaginary working situations of two ordinary academics. I conclude that tackling questionable research practices in the light of the replication crisis requires robust ‘top down’ measures that expect and accommodate a broader range of academic values, motivations, and tendencies, while challenging those that help to promote hyper-ambition.

Responsible Research is also concerned with generalizability: Recognizing efforts to reflect upon and increase generalizability in hiring and promotion decisions in psychology

We concur with the authors of the two target articles that Open Science practices can help combat the ongoing reproducibility and replicability crisis in psychological science and should hence be acknowledged as responsible research practices in hiring and promotion decisions. However, we emphasize that another crisis is equally threatening the credibility of psychological science in Germany: The sampling or generalizability crisis. We suggest that scientists’ efforts to contextualize their research, reflect upon, and increase its generalizability should be incentivized as responsible research practices in hiring and promotion decisions. To that end, we present concrete suggestions for how efforts to combat the additional generalizability crisis could be operationalized within Gärtner et al. (2022) evaluation scheme. Tackling the replicability and the generalizability crises in tandem will advance the credibility and quality of psychological science and teaching in Germany.

Remembering Reasons for Reform: A More Replicable and Reproducible Communication Literature Without the Rancor

Increasing awareness of the “replication crisis” has prompted discussion about replicability and reproducibility in social and behavioral science research, including in communication. As with other fields, communication has seen discussion about concerns with the interpretation of existing research. One response has been the piecemeal adoption of “open science” practices in communication to reduce selectivity in analysis, reporting, and publication of research. Calls for further adoption of such practices have, in turn, been met with criticisms and concerns about the negative consequences of their adoption. Amidst disparate perspectives regarding solutions to replicability and reproducibility issues in communication science, difficulties building consensus and caution about negative outcomes are understandable, but they also present the risk of a status quo bias that could stall the improvement of the replicability and reproducibility of communication research. The urgency of the replication crisis for communication and the cost of inaction are presented here along three exemplifying dimensions perhaps of particular importance in communication research: (a) responsibility to the public, (b) stewardship of resources, and (c) membership in a community of scholars. While debate over solutions will continue, we would do well to keep in mind that problems with replicability and reproducibility in communication research are indeed a crisis needing immediate attention.

The replication crisis and its relevance to Earth Science studies: Case studies and recommendations

Numerous scientific fields are facing a replication crisis, where the results of a study often cannot be replicated when a new study uses independent data. This issue has been particularly emphasized in psychology, health, and medicine, as incorrect results in these fields could have serious consequences, where lives might be at stake. While other fields have also highlighted significant replication problems, the Earth Sciences seem to be an exception. The paucity of Earth Science research aimed at understanding the replication crisis prompted this study. Specifically, this work aims to fill that gap by seeking to replicate geological results involving various types of time-series. We identify and discuss 11 key variables for replicating U-Pb age distributions: independent data, global sampling, proxy data, data quality, disproportionate non-random sampling, stratigraphic bias, potential filtering bias, accuracy and precision, correlating time-series segments, testing assumptions and divergent analytical methods, and analytical transparency. Even while this work primarily focuses on U-Pb age distributions, most of these factors (or variations of them) also apply to other geoscience disciplines. Thus, some of the discussions involve time-series consisting of εHf, δ18O-zircon, 14C, 10Be, marine δ13C, and marine δ18O. We then provide specific recommendations for minimizing adverse effects related to these factors, and in the process enhancing prospects for replicating geological results.

Virtues and vocation: An historical perspective on scientific integrity in the twenty-first century

According to the Dutch chemist Gerrit Jan Mulder (1802–1880), the principal aim of university education was character building and moral edification. Professional training was of secondary importance. Mulder’s ideas about the vocation and moral mission of the university professor can serve as a historical counterpart to later Weberian, Mertonian, and contemporary ideas on the ethos of science. I argue that a revaluation of the moral precepts that Mulder saw as defining the life of an academic is helpful in dealing with the problems of late modern science, such as the replication crisis and research misconduct. Addressing such problems must start in the university classrooms. To empower students to internalize the principles of responsible conduct of research, we need an updated version of Mulder’s idea of the university professor as a moral agent.

Abstract A023: Assessing the reproducibility crisis in vaginal microbiome studies for clinical applications in endometrial cancer

Abstract Recent associations between vaginal microbiota and gynecological cancer have led to high-throughput sequencing datasets identifying diagnostic biomarkers for Endometrial Cancer (EC). However, lack of bioinformatics standards results in inconsistent independent studies with poor reproducibility, making them clinically inapplicable. This study leverages publicly available amplicon sequence data to address the reproducibility of EC microbiome studies. We implemented bioinformatics pipelines from five studies to reproduce and measure the replicability of published results across cohorts using metrics such as alpha diversity, beta diversity and differentially expressed taxa. We further evaluated separability between benign and cancer patients at different taxonomic levels (phylum, class, order, family, and genus) within and across datasets using boosted tree classifiers and Area Under the Curve (AUC) as the performance metric. While we reproduced the general trend of observing higher microbial diversity in EC compared to healthy controls, we found irregularities in two cohorts. Using beta diversity distance metrics, we identified that histology alone explains less than 3% of the variance in all cohorts. Three microbiome differential abundance methods were used in the five studies. While they all agree on a decrease in the Lactobacillus genus in EC patients, there is no consensus on other taxa associated with EC. We also found that separability between benign and cancerous conditions is highest at the class level, having an AUC score of 0.86. In subsequent steps, we will perform an integrative analysis to identify an EC vaginal microbiome predictive signature that is preserved across all five cohorts, benefiting screening programs. Citation Format: Dollina D. Dodani, Aline Talhouk. Assessing the reproducibility crisis in vaginal microbiome studies for clinical applications in endometrial cancer [abstract]. In: Proceedings of the AACR Special Conference on Endometrial Cancer: Transforming Care through Science; 2023 Nov 16-18; Boston, Massachusetts. Philadelphia (PA): AACR; Clin Cancer Res 2024;30(5_Suppl):Abstract nr A023.

Aligning the principles and practice of research integrity and research fairness in global health: a mixed-methods study

IntroductionIn the past decade, global health research has seen a growing emphasis on research integrity and fairness. The concept of research integrity emerged in response to the reproducibility crisis in...

Driving effect of P16 methylation on telomerase reverse transcriptase-mediated immortalization and transformation of normal human fibroblasts.

P16 inactivation is frequently accompanied by telomerase reverse transcriptase (TERT) amplification in human cancer genomes. P16 inactivation by DNA methylation often occurs automatically during immortalization of normal cells by TERT. However, direct evidence remains to be obtained to support the causal effect of epigenetic changes, such as P16 methylation, on cancer development. This study aimed to provide experimental evidence that P16 methylation directly drives cancer development. A zinc finger protein-based P16-specific DNA methyltransferase (P16-Dnmt) vector containing a "Tet-On" switch was used to induce extensive methylation of P16 CpG islands in normal human fibroblast CCD-18Co cells. Battery assays were used to evaluate cell immortalization and transformation throughout their lifespan. Cell subcloning and DNA barcoding were used to track the diversity of cell evolution. Leaking P16-Dnmt expression (without doxycycline-induction) could specifically inactivate P16 expression by DNA methylation. P16 methylation only promoted proliferation and prolonged lifespan but did not induce immortalization of CCD-18Co cells. Notably, cell immortalization, loss of contact inhibition, and anchorage-independent growth were always prevalent in P16-Dnmt&TERT cells, indicating cell transformation. In contrast, almost all TERT cells died in the replicative crisis. Only a few TERT cells recovered from the crisis, in which spontaneous P16 inactivation by DNA methylation occurred. Furthermore, the subclone formation capacity of P16-Dnmt&TERT cells was two-fold that of TERT cells. DNA barcoding analysis showed that the diversity of the P16-Dnmt&TERT cell population was much greater than that of the TERT cell population. P16 methylation drives TERT-mediated immortalization and transformation of normal human cells that may contribute to cancer development.

All that Glitters Is not Gold: Type-I Error Controlled Variable Selection from Clinical Trial Data.

Clinical trials are primarily conducted to estimate causal effects, but the data collected can also be invaluable for additional research, such as identifying prognostic measures of disease or biomarkers that predict treatment efficacy. However, these exploratory settings are prone to false discoveries (type-I errors) due to the multiple comparisons they entail. Unfortunately, many methods fail to address this issue, in part because the algorithms used are generally designed to optimize predictions and often only provide the measures used for variable selection, such as machine learning model importance scores, as a byproduct. To address the resulting unclear uncertainty in the selection sets, the knockoff framework offers a model-agnostic, robust approach to variable selection with guaranteed type-I error control. Here, we review the knockoff framework in the setting of clinical data, highlighting main considerations using simulation studies. We also extend the framework by introducing a novel knockoff generation method that addresses two main limitations of previously suggested methods relevant for clinical development settings. With this new method, we empirically obtain tighter bounds on type-I error control and gain an order of magnitude in computational efficiency in mixed data settings. We demonstrate comparable selections to those of the competing method for identifying prognostic biomarkers for C-reactive protein levels in patients with psoriatic arthritis in four clinical trials. Our work increases access to the knockoff framework for variable selection from clinical trial data. Hereby, this paper helps to address the current replicability crisis which can result in unnecessary research efforts, increased patient burden, and avoidable costs.

Everything you ever wanted to know about the Think/No-Think task, but forgot to ask

The Think/No-Think (TNT) task has just celebrated 20 years since its inception, and its use has been growing as a tool to investigate the mechanisms underlying memory control and its neural underpinnings. Here, we present a theoretical and practical guide for designing, implementing, and running TNT studies. For this purpose, we provide a step-by-step description of the structure of the TNT task, methodological choices that can be made, parameters that can be chosen, instruments available, aspects to be aware of, systematic information about how to run a study and analyze the data. Importantly, we provide a TNT training package (as Supplementary Material), that is, a series of multimedia materials (e.g., tutorial videos, informative HTML pages, MATLAB code to run experiments, questionnaires, scoring sheets, etc.) to complement this method paper and facilitate a deeper understanding of the TNT task, its rationale, and how to set it up in practice. Given the recent discussion about the replication crisis in the behavioral sciences, we hope that this contribution will increase standardization, reliability, and replicability across laboratories.

The role of theory in mitigating the ‘reproducibility crisis’

The lack of reproducibility of scientific results is jeopardizing the trust in science. An effort to inform the dynamic, but non-arbitrary, nature of scientific evidence is required along with strengthening the reliability of published results. Concerns and actions aimed at testing and increasing the reproducibility of scientific conclusions are usually directed at increasing data validation through careful management of the data-curation process, paying attention to the empirical aspects of hypotheses testing. But important as they are, these aspects may be vulnerable to the perils of radical empiricism and therefore should be combined with more theoretical and conceptual tools. Pattern consistency and an understanding of the physiological, behavioral, or ecological mechanisms that cause patterns make scientific assertions more robust. Testing a priori hypotheses (e.g., about functional biological traits) with fresh (and redundant) evidence offers theoretical as well as empirical support for ecological research and may help strengthen the reliability of published results.

Power to Detect What? Considerations for Planning and Evaluating Sample Size

Academic In the wake of the replication crisis, social and personality psychologists have increased attention to power analysis and the adequacy of sample sizes. In this article, we analyze current controversies in this area, including choosing effect sizes, why and whether power analyses should be conducted on already-collected data, how to mitigate the negative effects of sample size criteria on specific kinds of research, and which power criterion to use. For novel research questions, we advocate that researchers base sample sizes on effects that are likely to be cost-effective for other people to implement (in applied settings) or to study (in basic research settings), given the limitations of interest-based minimums or field-wide effect sizes. We discuss two alternatives to power analysis, precision analysis and sequential analysis, and end with recommendations for improving the practices of researchers, reviewers, and journal editors in social-personality psychology. Public Recently, social-personality psychology has been criticized for basing some of its conclusions on studies with low numbers of participants. As a result, power analysis, a mathematical way to ensure that a study has enough participants to reliably “detect” a given size of psychological effect, has become popular. This article describes power analysis and discusses some controversies about it, including how researchers should derive assumptions about effect size, and how the requirements of power analysis can be applied without harming research on hard-to-reach and marginalized communities. For novel research questions, we advocate that researchers base sample sizes on effects that are likely to be cost-effective for other people to implement (in applied settings) or to study (in basic research settings). We discuss two alternatives to power analysis, precision analysis and sequential analysis, and end with recommendations for improving the practices of researchers, reviewers, and journal editors in social-personality psychology.