Valid and powerful second-level group statistics for decoding accuracy: Information prevalence inference using the i-th order statistic (i-test)

Abstract

In functional magnetic resonance imaging (fMRI) decoding studies using pattern classification, a second-level group statistical test is typically performed after first-level decoding analyses for individual participants. In the second-level test, the mean decoding accuracy across participants is often tested against the chance-level accuracy (for example, one-sample Student t-test) to check whether information about the label, such as, experimental condition or cognitive content, is included in brain activation. Meanwhile, Allefeld et al., (2016) highlighted that significant results for such tests only indicate that “there are some people in the population whose fMRI data carry information about the experimental condition.” Therefore, such tests failed to conclude whether the effect is typical in the population. Based on this argument, they proposed an alternative method implementing the prevalence inference. In the present study, that method is extended to propose a novel statistical test called as the “information prevalence inference using the i-th order statistic” (i-test). The i-test has a high statistical power compared with the method proposed in Allefeld et al., (2016) and provides an inference regarding the typical effect in the population. In the i-test, the i-th lowest sample decoding accuracy (the i-th order statistic) is compared to the null distribution to verify whether the proportion of higher-than-chance decoding accuracy in the population (information prevalence) is higher than the threshold. Hence, a significant result in the i-test is interpreted as a majority of the population has information about the label in the brain. Theoretical details of the i-test are provided, its high statistical power is identified by numerical calculation, and the application of this method in an fMRI decoding is demonstrated.