Comparison of tests for association of 2 × 2 tables under multiple testing setting

Abstract

Fisher’s exact test and Pearson’s chi-squared test are frequently used for testing associations of two binary variables in 2-by-2 contingency tables. In the single test setting, many studies have shown that the asymptotic Pearson’s chi-squared test cannot preserve the test size for small samples and Fisher Exact test tends to be overly conservative. Multiple unconditional exact tests were proposed for small samples as they perform better than the commonly used chi-square and Fisher’s exact test. No comparison of these approaches have been done in the multiple testing setting.This study examines the performances of two unconditional tests (Boschloo and Z-pooled test statistics are used) with Fisher’s exact test as well as asymptotic Pearson’s chi-squared test in a small sample multiple testing scenario via a simulation study. When testing simultaneously many null hypotheses, Benjamini-Hochberg (BH) procedure is typically applied to control the false discovery rate (FDR). The results show that in terms of sensitivity rate, the performances of Z-pooled and Boschloo Statistic are close to each other; Asymptotic chi-squared test is slightly better than the unconditional exact tests; Fisher’s exact test is the least powerful in all different settings. Boschloo’s test is more computation intensive. Z-pooled test is preferred if running time is a concern.