Experiments with Random Projections Ensembles: Linear Versus Quadratic Discriminants

Abstract

Gaussian classifiers make the modelling assumption that each class follows a multivariate Gaussian distribution, and include the Linear Discriminant Analysis (LDA) and Quadratic Discriminant Analysis (QDA). In high dimensional low sample size settings, a general drawback of these methods is that the sample covariance matrix may be singular, so their inverse is not available. Among many regularisation methods that attempt to get around this problem, the random projection ensemble approach for LDA has been both practically successful and theoretically well justified. The aim of this work is to investigate random projection ensembles for QDA, and to experimentally determine the relative strengths of LDA vs QDA in this context. We identify two different ways to construct averaging ensembles for QDA. The first is to aggregate low dimensional matrices to construct a smoothed estimate of the inverse covariance, which is then plugged into QDA. The second approach is to aggregate the predictions of low dimensional QDAs instead. By implementing experiments in eight different kinds of data sets, we conduct a thorough evaluation. From it we conclude that both QDA ensembles are able to improve on vanilla QDA, however the LDA ensemble tends to be more stable with respect to the choice of target dimension of the random projections.