Robust covariance matrix estimation and identification of unusual data points: New tools

Abstract

Most consistent estimators are prone to total breakdown in the presence of a handful of unusual data points (UDPs). This compromises inference. Robust estimation is a (seldom-used) solution; but methods commonly-used in applied research have severe drawbacks. In this paper, building upon methods that are relatively unknown outside of the robust statistics literature, we provide an enhanced tool for robust estimates of mean and covariance, useful both for robust estimation and for detection of unusual data points. It is relatively fast and useful for large data sets. We also provide a new robust cluster method, an input to our broader method, but also useful for standalone UDP detection or cluster analysis. We provide a comparative study of numerous methods that is not available in the current literature. Testing indicates that our method performs at par with, and often better than, two of the currently best available methods. We also demonstrate that the issues we discuss are not merely hypothetical, by applying our tools to real world data, and to re-examine two prominent economic studies. Our methods reveal that their central results are driven by a set of unusual points.