A Random Matrix-Theoretic Approach to Handling Singular Covariance Estimates

Abstract

In many practical situations we would like to estimate the covariance matrix of a set of variables from an insufficient amount of data. More specifically, if we have a set of <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i> independent, identically distributed measurements of an <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</i> dimensional random vector the maximum likelihood estimate is the sample covariance matrix. Here we consider the case where <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i> <; <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</i> such that this estimate is singular (noninvertible) and therefore fundamentally bad. We present a radically new approach to deal with this situation based on the idea of dimensionality reduction through an ensemble of isotropically random unitary matrices. We obtain two estimates cov and invcov which are estimates for the covariance matrix and the inverse covariance matrix respectively. Both estimates retain the original eigenvectors while altering the eigenvalues. We have a closed form analytical expression for cov and invcov in terms of the eigenvector/eigenvalue decomposition of the sample covariance. We motivate the use of invcov through applications to linear estimation, supervised learning, and high-resolution spectral estimation. We also compare the performance of these estimators with other more conventional methods.