Scientific Self-correction Research Articles

Replication marketplaces would help science to become more self-correcting.

Independent replications are very rare in the behavioural and social sciences. This is problematic because they can help to detect 'false positives' in published research and, in turn, contribute to scientific self-correction. The lack of replication studies is, among other factors, due to a rather passive editorial approach concerning replications by many journals, which does not encourage and may sometimes even actively discourage submission of replications. In this Perspective article, we advocate for a more proactive editorial approach concerning replications and suggest introducing journal-based replication marketplaces as a new publication track. We argue that such replication marketplaces could solve the long-standing problem of lacking independent replications. To establish these marketplaces, a designated part of a journal's editorial board identifies the most relevant new findings reported within the journal's pages and publicly offers them for replication. This public offering could be combined with small grants for authors to support these replications. Authors then compete for the first accepted registered report to conduct the related replications and can thus be sure that their replication will be published independent of the later findings. Replication marketplaces would not only increase the prevalence of independent replications but also help science to become more self-correcting.

Psychology’s reform movement needs a reconceptualization of scientific expertise

Science is supposed to be a self-correcting endeavor, but who is “the scientific expert” that corrects faulty science? We grouped traditional conceptualizations of expertise in psychology under three classes (substantialist, implicitist, and social conventionalist), and then examined how these approaches affect scientific self-correction in reference to various components of the credibility crisis such as fraud/QRPs, the inadequate number of replication studies, challenges facing big team science, and perverse incentives. Our investigation pointed out several problems with the traditional views. First, traditional views conceptualize expertise as something possessed, not performed, ignoring the epistemic responsibility of experts. Second, expertise is conceived as an exclusively individual quality, which contradicts the socially distributed nature of scientific inquiry. Third, some aspects of expertise are taken to be implicit or relative to the established research practices in a field, which leads to disputes over replicability and makes it difficult to criticize mindless scientific rituals. Lastly, a conflation of expertise with eminence in practice creates an incentive structure that undermines the goal of self-correction in science. We suggest, instead, that we conceive an expert as a reliable informant. Following the extended virtue account of expertise, we propose a non-individualist and a performance-based model, and discuss why it does not suffer from the same problems as traditional approaches, and why it is more compatible with the reform movement's goal of creating a credible psychological science through self-correction.

Retracted articles use less free and open-source software and cite it worse

Abstract As an essential mechanism of scientific self-correction, articles are retracted for many reasons, including errors in processing data and computation of results. In today’s data-driven science, the validity of research data and results significantly depends on the software employed. We investigate the relationship between software usage and research validity, eventually leading to article retraction, by analyzing software mentioned across 1,924 retraction notices and 3,271 retracted articles. We systematically compare software mentions and related information with control articles sampled by coarsened exact matching by recognizing publication year, scientific domain, and journal rank. We identify article retractions caused by software errors or misuse and find that retracted articles use less free and open-source software, hampering reproducible research and quality control. Moreover, such differences are also present concerning software citation, where retracted articles less frequently follow software citation guidelines regarding free and open-source software.

Where Are the Self-Correcting Mechanisms in Science?

It is often said that science is self-correcting, but the replication crisis suggests that self-correction mechanisms have fallen short. How can we know whether a particular scientific field has effective self-correction mechanisms, that is, whether its findings are credible? The usual processes that supposedly provide mechanisms for scientific self-correction, such as journal-based peer review and institutional committees, have been inadequate. We describe more verifiable indicators of a field’s commitment to self-correction. These fall under the broad headings of 1) transparency, which is already the subject of many reform efforts and 2) critical appraisal, which has received less attention and which we focus on here. Only by obtaining Observable Self-Correction Indicators (OSCIs) can we begin to evaluate the claim that “science is self-correcting.” We expect that the veracity of this claim varies across fields and subfields, and suggest that some fields, such as psychology and biomedicine, fall far short of an appropriate level of transparency and, especially, critical appraisal. Fields without robust, verifiable mechanisms for transparency and critical appraisal cannot reasonably be said to be self-correcting, and thus do not warrant the credibility often imputed to science as a whole.

Citation Patterns Following a Strongly Contradictory Replication Result: Four Case Studies From Psychology

Replication studies that contradict prior findings may facilitate scientific self-correction by triggering a reappraisal of the original studies; however, the research community’s response to replication results has not been studied systematically. One approach for gauging responses to replication results is to examine how they affect citations to original studies. In this study, we explored postreplication citation patterns in the context of four prominent multilaboratory replication attempts published in the field of psychology that strongly contradicted and outweighed prior findings. Generally, we observed a small postreplication decline in the number of favorable citations and a small increase in unfavorable citations. This indicates only modest corrective effects and implies considerable perpetuation of belief in the original findings. Replication results that strongly contradict an original finding do not necessarily nullify its credibility; however, one might at least expect the replication results to be acknowledged and explicitly debated in subsequent literature. By contrast, we found substantial citation bias: The majority of articles citing the original studies neglected to cite relevant replication results. Of those articles that did cite the replication but continued to cite the original study favorably, approximately half offered an explicit defense of the original study. Our findings suggest that even replication results that strongly contradict original findings do not necessarily prompt a corrective response from the research community.

Putting the Self in Self-Correction: Findings From the Loss-of-Confidence Project

Science is often perceived to be a self-correcting enterprise. In principle, the assessment of scientific claims is supposed to proceed in a cumulative fashion, with the reigning theories of the day progressively approximating truth more accurately over time. In practice, however, cumulative self-correction tends to proceed less efficiently than one might naively suppose. Far from evaluating new evidence dispassionately and infallibly, individual scientists often cling stubbornly to prior findings. Here we explore the dynamics of scientific self-correction at an individual rather than collective level. In 13 written statements, researchers from diverse branches of psychology share why and how they have lost confidence in one of their own published findings. We qualitatively characterize these disclosures and explore their implications. A cross-disciplinary survey suggests that such loss-of-confidence sentiments are surprisingly common among members of the broader scientific population yet rarely become part of the public record. We argue that removing barriers to self-correction at the individual level is imperative if the scientific community as a whole is to achieve the ideal of efficient self-correction.

Scientific self-correction: the Bayesian way

The enduring replication crisis in many scientific disciplines casts doubt on the ability of science to estimate effect sizes accurately, and in a wider sense, to self-correct its findings and to produce reliable knowledge. We investigate the merits of a particular countermeasure—replacing null hypothesis significance testing (NHST) with Bayesian inference—in the context of the meta-analytic aggregation of effect sizes. In particular, we elaborate on the advantages of this Bayesian reform proposal under conditions of publication bias and other methodological imperfections that are typical of experimental research in the behavioral sciences. Moving to Bayesian statistics would not solve the replication crisis single-handedly. However, the move would eliminate important sources of effect size overestimation for the conditions we study.

On the Suppression of Medical Evidence

Financial conflicts of interest in medical research foster deviations from research standards and evidentially lead to the suppression of research findings that are at odds with commercial interests of pharmaceutical companies. Questionable research practices prevent data from being created, made available, or given suitable recognition. They run counter to codified principles of responsible conduct of research, such as honesty, openness or respect for the law. Resulting in ignorance, misrepresentation and suspension of scientific self-correction, suppression of medical evidence in its various forms is both a threat to the epistemic and the moral integrity of medical science. This paper discusses an example of the suppression of medical evidence in recent influenza research and develops a conceptual framework for the description and assessment of questionable research practices applied in research and publication processes to suppress evidence.

Reproducibility2020: Progress and priorities.

The preclinical research process is a cycle of idea generation, experimentation, and reporting of results. The biomedical research community relies on the reproducibility of published discoveries to create new lines of research and to translate research findings into therapeutic applications. Since 2012, when scientists from Amgen reported that they were able to reproduce only 6 of 53 "landmark" preclinical studies, the biomedical research community began discussing the scale of the reproducibility problem and developing initiatives to address critical challenges. Global Biological Standards Institute (GBSI) released the "Case for Standards" in 2013, one of the first comprehensive reports to address the rising concern of irreproducible biomedical research. Further attention was drawn to issues that limit scientific self-correction, including reporting and publication bias, underpowered studies, lack of open access to methods and data, and lack of clearly defined standards and guidelines in areas such as reagent validation. To evaluate the progress made towards reproducibility since 2013, GBSI identified and examined initiatives designed to advance quality and reproducibility. Through this process, we identified key roles for funders, journals, researchers and other stakeholders and recommended actions for future progress. This paper describes our findings and conclusions.

Accuracy, bias, self-fulfilling prophecies, and scientific self-correction.

In my Précis of Social Perception and Social Reality (Jussim 2012, henceforth abbreviated as SPSR), I argued that the social science scholarship on social perception and interpersonal expectancies was characterized by a tripartite pattern: (1) Errors, biases, and self-fulfilling prophecies in person perception were generally weak, fragile, and fleeting; (2) Social perceptions were often quite accurate; and (3) Conclusions appearing throughout the social psychology scientific literature routinely overstated the power and pervasiveness of expectancy effects, and ignored evidence of accuracy. Most commentators concurred with the validity of these conclusions. Two, however, strongly disagreed with the conclusion that the evidence consistently has shown that stereotypes are moderately to highly accurate. Several others, while agreeing with most of the specifics, also suggested that those arguments did not necessarily apply to contexts outside of those covered in SPSR. In this response, I consider all these aspects: the limitations to the tripartite pattern, the role of politics and confirmation biases in distorting scientific conclusions, common obstructions to effective scientific self-correction, and how to limit them.

Can the behavioral sciences self-correct? A social epistemic study

Advocates of the self-corrective thesis argue that scientific method will refute false theories and find closer approximations to the truth in the long run. I discuss a contemporary interpretation of this thesis in terms of frequentist statistics in the context of the behavioral sciences. First, I identify experimental replications and systematic aggregation of evidence (meta-analysis) as the self-corrective mechanism. Then, I present a computer simulation study of scientific communities that implement this mechanism to argue that frequentist statistics may converge upon a correct estimate or not depending on the social structure of the community that uses it. Based on this study, I argue that methodological explanations of the “replicability crisis” in psychology are limited and propose an alternative explanation in terms of biases. Finally, I conclude suggesting that scientific self-correction should be understood as an interaction effect between inference methods and social structures.

Open Data and the Future of Science

>> See video of presentation (54 min.) Open Science is not new, it was the bedrock on which the scientific revolutions of the 18th and 19th centuries were built. Its open publication of scientific concepts and the evidence (the data) on which they were based allowed scrutiny of the logic of an argument and replication of observations or experiments or their refutation. It has been the basis of so-called “scientific self-correction”. But the technological revolution of recent decades has produced an unprecedented explosion in the human capacity to acquire, store and manipulate data and information and to instantaneously communicate them globally, irrespective of location. It has produced fundamental changes in human, social and economic behaviour and has implications for research and learning that are far more profound and pervasive than those that followed Gutenberg’s invention of the printing press, including the way that science engages with the wider community. But the explosion of data is also undermining self-correction through the difficulty of making large volumes of supporting data concurrently available in a scrutinisable form, risking both the method of science and its credibility. This poses a major challenge for modern science to stimulate open release and sharing of data in ways that also facilitate new modes of collaboration and that increase creativity through interaction of many brains and many communities unbounded by institutional walls. We need to challenge and re-define many of the habits and norms of researchers and their institutions if the research community if it is to exploit technological opportunities, maintain self-correction and maximize the contribution of research to human understanding and welfare. Fortunately the response to these imperatives is growing, through the enunciation of principles for open science, the development of procedures and tools and the engagement of researchers, universities, funders, publishers, learned societies and governments.

Race Science, Academic Freedom, and the Limits of Scientific Self-Correction

Race Science, Academic Freedom, and the Limits of Scientific Self-Correction