Exact inference around ordinal measures of association is often not exact

Abstract

In this paper, we build upon the work of DiCiccio and Romano (2017) by extending their permutation test approach, based on the Pearson correlation coefficient in the continuous case, to ordinal measures of association. We investigate commonly used ordinal measures such as the Spearman correlation, Kendall’s tau-b, and gamma, which are widely implemented in commercial and open-source software packages for exact testing routines based on generalized hypergeometric probabilities. Similar to DiCiccio and Romano’s method, we apply studentization to correct the test statistic, which yields asymptotically valid inference for testing no ordinal association. We present a comprehensive theoretical framework for our approach, followed by a simulation study. Furthermore, we use toy examples to highlight the differences between the exact tests and the asymptotically valid tests. Our findings align with those of DiCiccio and Romano, indicating that exact permutation tests based on ordinal measures of association are often not exact, whereas the asymptotically correct tests perform well for moderate to large sample sizes.