Make Reporting of Confidence Intervals de Rigueur

Abstract

Editors’ Note: In this extended edition of Making Sense of Methods and Measurement, our guest columnist, Greg Gilbert, presents the convincing argument for adopting confidence intervals (CIs) for reporting research results. Traditionally, when researchers report whether outcomes from two interventions were significantly different, they use P values. However, we support the use of CIs to communicate not only whether differences were significant but also what the magnitude of the difference was. If you want to compare whether using simulation first to teach concepts versus using simulation to reinforce concepts is more effective (i.e., produces better learning outcomes), it is helpful to know the magnitude of the effect. A P value can only tell you if there is a significant difference between modalities. A CI allows you to state that if an infinite number of samples were taken, 95% of the time the population mean difference would be between the lower confidence bound and the upper confidence bound. That is a pretty powerful statement. Quantification of the magnitude of the effect lies at the heart of the P valueeCI controversy. For over 30 years, statisticians have been advocating the use of CIs when reporting results; it is time, practitioners heeded this call. This column will discuss a straightforward, simplistic argument for including CIs in your articles. Why notP values?AP value is the probability that random sampling leads to a difference between two statistics, such as samplemeans, that is as extreme ormore extreme than the one observed. Another way of looking atP values is that if another sample of data were taken from the same population, what is the probability of the new test statistic being as extreme or more extreme than the one calculated in the first experiment? P values only provide information regarding statistical