Power to the researchers: Calculating power after estimation

Abstract

AbstractThis study demonstrates a simple and reliable method for calculating ex post power. We first conduct a series of Monte Carlo experiments to assess its performance. The experiments are designed to produce artificial datasets that resemble actual data from 23 studies funded by the International Initiative for Impact Evaluation (3ie). After determining that the method performs adequately, we then apply it to the 23 studies and compare their ex post power with the ex‐ante power claimed on their funding applications. We find the average ex post power of the 3ie studies is close to 80%. However, there are more estimates of low power than would be expected if all studies had 80% true power. Most of the differences between ex post and ex ante power can be explained by differences between planned and actual total observations, number of clusters, and the degree of intracluster correlation. This demonstrates how ex post power can be used by funders to evaluate previously funded research and identify areas for improved power estimation in future research. We further show how ex post power can aid in the interpretation of both insignificant and significant estimates.