Type I and II error rates of Bayesian two-sample tests under preliminary assessment of normality in balanced and unbalanced designs and its influence on the reproducibility of medical research

Abstract

Student's two-sample t-test is often used in medical research like randomized controlled trials. To control type I errors, normality of the observed data needs to be assessed. In practice, a two-stage procedure is acknowledged: First, a preliminary test for normality is conducted. If the test is not significant, the two-sample t-test is applied, and else a nonparametric test like Mann–Whitney's U is conducted. It is unknown how Bayesian tests behave under this procedure. A simulation study was conducted to study the error rates of these Bayesian alternatives under preliminary assessment of normality in balanced and unbalanced designs. Results show that Bayesian counterparts yield 50–60% fewer type I errors at the cost of slightly increased type II error rates, and that the two-stage procedure is not recommended in unbalanced Bayesian designs. This makes them an attractive alternative for biomedical research, as decreased power can be overcome by increasing sample size.