Probability, proof, and clinical significance

Abstract

Statistical tests help distinguish true differences (associations) from chance and result in a P value which is an estimation of probability that the results are due to chance. An arbitrary test threshold value (eg, usually alpha = .05) is used to distinguish results that are assumed to be due to chance from the results that are due to other factors. The bottom line: If the probability that the results are due to chance is less than the threshold value (P < .05), it is assumed the differences are due to these other factors (eg, true differences in treatment effects). However, we may be wrong 5% of the time. Significance testing in itself does not take into account factors which may bias study results. The possible exception to this is multivariate analysis, which is a subject for future articles. Sample size and random variation play an important role in whether a result is statistically significant or not and, together with expected effect size, whether the study is “powered” to detect statistical differences. A statistically significant result does not “prove” anything and does not establish a causal relationship between the exposure and outcome. The finding of an “association” does not mean that the association is causal in most instances. This is a topic for future articles. Although a result may be statistically significant, the effect size (ie, magnitude of the effect) may not be biologically or clinically important. In critiquing, designing, and reporting studies, the minimal clinically important difference is important to consider. While statistical testing is an important part of research analysis, its limitations, uses, and misuses need to be considered in order to put results of a study in the proper context. In the next issue, we tackle the topic of confounding, an important source of potential bias which needs to be considered in all studies regardless of design.