Effect Size Magnification and Epidemiologic Design Calculations: their use in EPA's Office of Pesticide Programs in evaluating study size in epidemiology studies

Abstract

While most researchers recognize issues associated with small/low powered studies vis-a-vis their lessened ability to detect true effects, fewer recognize issues associated with small/low powered studies and their tendency to produce inflated estimates if those estimated effects are required to pass a statistical threshold to be judged important, relevant, or “discovered”. Effect size magnification (ESM) is a term used to refer to this phenomenon. Specifically: low-powered studies that find evidence of an effect often provide inflated estimates of the size of that effect. This talk will discuss some of the regulatory implications of ESM and our efforts in EPA’s Office of Pesticide Programs to understand, reproduce, and finally apply this knowledge to better evaluate the reliability of reported (statistically significant) effect sizes in epidemiology studies and put these into a fuller context. Routinely performing such ESM calculations (aka “post-hoc design calculations”) in epidemiology in a regulatory context can assist in determining the extent to which ESM may be an issue or should be accounted for in interpretation of epidemiological results. In particular, the talk will relate these design calculations to both judging adequacy of power and sample size issues vis-a-vis the observed effect size and in interpreting their potential implications for study conclusions. While such design calculations do not change a statistically significant result to a non-significant result, they do allow regulatory staff to consider that a reportedly large effect may in fact be much lower, to a degree that the effect may have less influence on regulatory decisions.