When power analyses based on pilot data are biased: Inaccurate effect size estimators and follow-up bias

Casper Albers,Daniël Lakens

doi:10.1016/j.jesp.2017.09.004

Casper Albers, Daniël Lakens

Open Access

https://doi.org/10.1016/j.jesp.2017.09.004

Copy DOI

Abstract

When designing a study, the planned sample size is often based on power analyses. One way to choose an effect size for power analyses is by relying on pilot data. A-priori power analyses are only accurate when the effect size estimate is accurate. In this paper we highlight two sources of bias when performing a-priori power analyses for between-subject designs based on pilot data. First, we examine how the choice of the effect size index (η2, ω2 and ε2) affects the sample size and power of the main study. Based on our observations, we recommend against the use of η2 in a-priori power analyses. Second, we examine how the maximum sample size researchers are willing to collect in a main study (e.g. due to time or financial constraints) leads to overestimated effect size estimates in the studies that are performed. Determining the required sample size exclusively based on the effect size estimates from pilot data, and following up on pilot studies only when the sample size estimate for the main study is considered feasible, creates what we term follow-up bias. We explain how follow-up bias leads to underpowered main studies.Our simulations show that designing main studies based on effect sizes estimated from small pilot studies does not yield desired levels of power due to accuracy bias and follow-up bias, even when publication bias is not an issue. We urge researchers to consider alternative approaches to determining the sample size of their studies, and discuss several options.

Highlights

It is common practice in psychological and behavioral research to express the results of a quantitative study in at least two numbers: One expressing the probability or likelihood of data under specified statistical models, usually through a p-value or Bayes factor, and one expressing the magnitude of the effect, often through a effect size (ES)
We focus on two other sources of bias in power analyses that play an important role in power analysis even when publication bias and researchers' degrees of freedom do not influence effect size estimates
We have looked at simulations where the population effect size is large

Summary

Introduction

It is common practice in psychological and behavioral research to express the results of a quantitative study in at least two numbers: One expressing the probability or likelihood of data under specified statistical models, usually through a p-value or Bayes factor, and one expressing the magnitude of the effect, often through a (standardized) effect size (ES). It has been pointed out that effect sizes reported in the literature are known to be inflated due to publication bias, and this widespread bias in reported effect sizes is a challenge when performing a-priori power analyses based on published research. We focus on two other sources of bias in power analyses that play an important role in power analysis even when publication bias and researchers' degrees of freedom do not influence effect size estimates (e.g., when researchers perform their own pilot study) These sources of bias point out clear limitations of the common practice to use the effect size from a pilot study to determine the sample size of a follow-up study through an a-priori power analysis. Researchers are more likely to follow-up on initial studies that yielded higher effect size estimates than initial studies that

Objectives

Results

Discussion

Conclusion