Common sense, nonsense and statistics.

Abstract

The media has recently been echoing a claim that medical statistics too frequently inspire cries of 'breakthrough!' later exposed in the cold light of experience, as nothing but mirages. The arbitrary parameters and potential subjectivity of conventional medical statistics have been debated, without consensus, in most of the major journals. The question stands: do we allow conventional medical statistics to bias our interpretation of the true significance of medical research? Undoubtedly, some form of statistics is needed to help interpret the results of clinical trials. The problem is the cut-off point: what qualifies as an important finding? For years, conference halls have rung with sceptical voices questioning the clinical relevance of statistically significant data. Neither P values nor confidence intervals seem consistently reliable as a guide to clinical significance. Moreover, the assumptions underlying parametric significance tests (such as equality of variance in group comparisons) may be untrue of some studies. A systematic approach often put forward to correct for subjectivity bias (prior probability) among clinical investigators is Bayesian analysis. We are told that elimination of subjectivity by use of Bayesian inference paves the way to truly objective, evidence-based medical practice. Yet who but a statistically minded minority can begin to interpret Bayesian analysis? Reading various exchanges of letters in the medical journals on statistical models, I am struck by the fact that almost all are written by biostatisticians. The opinions of practising physicians are deafeningly silent. In our mounting enthusiasm for evidence-based medicine, we may be giving too much weight to clumsy statistical formulations that should not hinge our judgment of clinical relevance. In this article I suggest, firstly, that conventional medical statistics are better than more sophisticated alternatives since they are better understood by practising physicians; and, secondly, that the problems we have with statistics derive less from our methodology than from our stiff conventions of interpretation.