Creating falseness-How to establish statistical evidence of the untrue.

Per Lytsy

doi:10.1111/jep.12823

Abstract

Null hypothesis significance testing is the typical statistical approach in search of the truthfulness of hypotheses. This method does not formally consider the prior credence in the hypothesis, which affects the chances of reaching correct conclusions. When scientifically implausible or empirically weakly supported hypotheses are tested, there is an increased risk that a positive finding in a test in fact is false positive. This article argues that when scientifically weakly supported hypotheses are tested repeatedly—such as when studying the clinical effects of homeopathy—the accumulation of false positive study findings will risk providing false evidence also in systematic reviews and meta‐analyses. False positive findings are detrimental to science and society, as once published, they accumulate persistent untrue evidence, which risks giving rise to nonpurposive research programmes, policy changes, and promotion of ineffective treatments. The problems with false positive findings are discussed, and advice is given on how to minimize the problem. The standard of evidence of a hypothesis should depend not only on the results of statistical analyses but also on its a priori support. Positive findings from studies investigating hypotheses with poor theoretical and empirical foundations should be viewed as tentative until the results are replicated and/or the hypothesis gains more empirical evidence supporting it as likely to be true.

Highlights

The ultimate goal of science is to build knowledge about the world, where knowledge intrinsically is understood to be true
Ever since the null hypothesis significance test (NHST) was formed as a hybrid of Fisher and Neyman/Pearson different frequentist approaches,[1] there has been an ongoing debate about the method's shortcomings
But in medical studies rarely acknowledged, weakness of the NHST method is that it disregards biological understanding and previous research results of the hypothesis it is testing.[18]

Summary

Introduction

The ultimate goal of science is to build knowledge about the world, where knowledge intrinsically is understood to be true. But in medical studies rarely acknowledged, weakness of the NHST method is that it disregards biological understanding and previous research results of the hypothesis it is testing.[18] While this to some may sound like a good thing—a test that is not prejudicious about the analysed data—it has profound consequences on the chances of making correct inferences about how the world works.

Results

Conclusion