Covariance estimation for classifying high dimensional data

Abstract

When classifying data with the Gaussian maximum likelihood classifier, the mean vector and covariance matrix of each class usually are not known and must be estimated from training samples. For p-dimensional data, the sample covariance estimate is singular, and therefore unusable, if fewer than p+1 training samples from each class are available, and it is a poor estimate of the true covariance unless many more than p+1 samples are available. Since inaccurate estimates of the covariance matrix lead to lowered classification accuracy and labeling training samples can be difficult and expensive in remote sensing applications, having too few training samples is a major impediment in using the Gaussian maximum likelihood classifier with high dimensional remote sensing data. In the paper, a new covariance estimator is presented that selects an appropriate mixture of the sample covariance and the common covariance estimates. The mixture deemed appropriate is the one that provides the best fit to the training samples in the sense that it maximizes the average likelihood of training samples not used in the estimates. When the number of training samples is limited or when the covariance matrices of the classes are similar, this estimator tends to select an estimate close to the common covariance, otherwise it favors the sample covariance estimate. Since it is non-singular whenever the common covariance estimate is non-singular, the new estimator can be used even when some of the sample covariance matrices are singular.