Distribution of a P-Value When the Alternative Hypothesis is True for Binary Outcomes

Abstract

P-values are widely used to assess scientific evidence in randomized clinical trials despite the controversy surrounding its appropriate use and interpretation. A p-value is uniformly distributed under the null hypothesis while a distribution of a p-value under the alternative hypothesis is a function of sample size and posited alternative value in cases where the normal approximation can be assumed. This gives a sense of how sample sizes and posited alternative values affect the distribution of the p-value under the alternative hypothesis and it helps to plan a more plausible study based on its aims during the design stage. In this article, we investigate the distribution of a p-value under the alternative hypothesis when the outcome is binary, focusing on the exact approach where the normal approximation is not applicable due to a small sample size. The characteristics of a p-value distribution under the alternative hypothesis were illustrated in various scenarios. Our investigation of the distribution showed a p-value at an alternative value can be useful, especially when there is not much information during the planning stage due to a lack of preliminary findings.