The case against retrospective statistical power analyses with an introduction to power analysis

Abstract

Statistical power analysis is an important tool for planning an experiment because this type of analysis allows researchers to identify an appropriate sample size for a particular experimental design. In recent years, it seems many biology journals (see Table 1 in Hoenig and Heisey 2001) have been encouraging researchers to calculate statistical power after their experiments when they have obtained non-significant results (hereafter, termed “retrospective power calculation or analysis” as opposed to “prospective power analysis”, which is conducted preexperimentally). For example, a leading journal in the field of animal behaviour, Animal Behaviour, asks for retrospective power calculations as a matter of editorial policy, stating “where a significance test based on a small sample size yields a nonsignificant result, explicit consideration should be given to the power of the data for accepting the null hypothesis” (issue xiii, revised November 2004). However, retrospective power analysis is controversial (Goodman and Berklin 1994; Thomas 1997; Gerard et al. 1998). Furthermore, Hoenig and Heisey (2001; see also Colegrave and Ruxton 2003) have clearly demonstrated the logical flaw of retrospective power analysis when used for data-analytic purposes. We believe that many researchers have both not recognised the serious logical flaw in retrospective power analysis when it is used for interpreting nonsignificant results and that an accurate understanding of power analysis has yet to be established among some researchers, especially among students in the study of animal behaviour. The purpose of this article is—using the independent t-test as an example—to: (1) outline statistical power analysis and its components; (2) describe three common ways used to make retrospective power calculations and their logical flaws and shortcomings; (3) discuss solutions to the current situation and make recommendations.