The Problem of Too Many Statistical Tests: Subgroup Analyses in a Study Comparing the Effectiveness of Online and Live Lectures

Abstract

The more statistical analyses performed in the analysis of research data, the more likely it is that one or more of the conclusions will be in error. Multiple statistical analyses can occur when the sample contains several subgroups and the researchers perform separate analyses for each subgroup. For example, separate analyses may be done for different ethnic groups, different levels of education, and/or for both genders. Media reports of research frequently omit information on the number of subgroup analyses performed thus leaving the reader with insufficient information to assess the validity of the conclusions. This article discusses the problems with a media report on research that was analyzed by conducting many subgroup analyses. The article concludes that the quantitatively literate reader should be skeptical of articles that report subgroup analyses without reporting the number of analyses that were done.