A non-parametric statistical inference framework for Deep Learning in current neuroimaging

Abstract

Deep Learning (DL) predictions are uncertain; but how uncertain? Statistical inference estimates the probabilities of uncertainty from a sample drawn from a population. Assessing the statistical significance of accuracies reported by DL remains largely unexplored. A framework to do so would usefully support a range of applications, and in particular group classifications from neuroimages where, for operational reasons, sample sizes are necessary limited and thus often do not generalise well. We applied a random-effects inference based on a label permutation test to calculate the statistical significance of K-fold cross-validation (CV) from statistical power and Type-I error rates. Our hypothesis is that in low sample size scenarios, the use of resubstitution with upper bound correction (RUB) as a validation would mitigate the debate on the generalisation ability of DL models. The derived framework enables testing such generalisation ability of DL models as feature extraction methods. A combination of autoencoders and support vector machines as feature extraction and classification models is evaluated in a case-control analysis of Alzheimer’s disease with well-established outcomes. We found that RUB slightly outperforms K-fold CV as a validation method, especially estimating statistical power in the most heterogeneous samples. Therefore, we suggest RUB as potent and valid method for DL with neuroimages in terms of bias, variance and computational demand.