Controlling test size while gaining the benefits of an internal pilot design.

Abstract

To compensate for a power analysis based on a poor estimate of variance, internal pilot designs use some fraction of the planned observations to reestimate error variance and modify the final sample size. Ignoring the randomness of the final sample size may bias the final variance estimate and inflate test size. We propose and evaluate three different tests that control test size for an internal pilot in a general linear univariate model with fixed predictors and Gaussian errors. Test 1 uses the first sample plus those observations guaranteed to be collected in the second sample for the final variance estimate. Test 2 depends mostly on the second sample for the final variance estimate. Test 3 uses the unadjusted variance estimate and modifies the critical value to bound test size. We also examine three sample-size modification rules. Only test 2 can control conditional test size, align with a modification rule, and provide simple power calculations. We recommend it if the minimum second (incremental) sample is at least moderate (perhaps 20). Otherwise, the bounding test appears to have the highest power in small samples. Reanalyzing published data highlights some advantages and disadvantages of the various tests.