Robust Covariance Estimation Research Articles

Algorithmic approaches to aid species' delimitation in multidimensional morphospace

BackgroundThe species is a fundamental unit of biological pattern and process, but its delimitation has proven a ready source of argument and disagreement. Here, we discuss four key steps that utilize statistical thresholds to describe the morphological variability within a sample and hence assess whether there is evidence for one or multiple species. Once the initial set of biologically relevant traits on comparable individuals has been identified, there is no need for the investigator to hypothesise how specimens might be divided among groups, nor the traits on which groups might be separated.ResultsPrincipal components are obtained using robust covariance estimates and retained only if they exceed threshold amounts of explanatory power, before model-based clustering is performed on the dimension-reduced space. We apply these steps in an attempt to resolve ongoing debates among taxonomists working on the extinct Eocene planktonic foraminifera Turborotalia, providing statistical evidence for two species shortly before the lineage's extinction near the Eocene/Oligocene boundary.ConclusionBy estimating variance robustly (samples containing incipient species are unlikely to be scaled optimally by means and standard deviations) and identifying thresholds relevant to a particular system rather than universal standards, the steps of the framework aim to optimize the chances of delineation without imposing pre-conceived patterns onto estimates of species limits.

A New Method of Robust Linear Regression Analysis: Some Monte Carlo Experiments

This paper has elaborated upon the deleterious effects of outliers and corruption of dataset on estimation of linear regression coefficients by the Ordinary Least Squares method. Motivated to ameliorate the estimation procedure, we have introduced the robust regression estimators based on Campbell's robust covariance estimation method. We have investigated into two possibilities: first, when the weights are obtained strictly as suggested by Campbell and secondly, when weights are assigned in view of the Hampel's median absolute deviation measure of dispersion. Both types of weights are obtained iteratively. Using these two types of weights, two different types of weighted least squares procedures have been proposed. These procedures are applied to detect outliers in and estimate regression coefficients from some widely used datasets such as stackloss, water salinity, Hawkins-Bradu-Kass, Hertzsprung-Russell Star and pilot-point datasets. It has been observed that Campbell-II in particular detects the outlier data points quite well. Subsequently, some Monte Carlo experiments have been carried out to assess the properties of these estimators. Findings of these experiments indicate that for larger number and size of outliers, the Campbell-II procedure outperforms the Campbell-I procedure. Unless perturbations introduced to the dataset are numerous and very large in magnitude, the estimated coefficients are also nearly unbiased.

Modeling Adolescent Drug-Use Patterns in Cluster-Unit Trials With Multiple Sources of Correlation Using Robust Latent Class Regressions

The purpose of the study is to examine variation in adolescent drug-use patterns by using latent class regression analysis and evaluate the properties of an estimating-equations approach under different cluster-unit trial designs. A set of second-order estimating equations for latent class models under the cluster-unit trial design are proposed. This approach models the correlation within subclusters (drug-use behaviors), but ignores the correlation within clusters (communities). A robust covariance estimator is proposed that accounts for within-cluster correlation. Performance of this approach is addressed through a Monte Carlo simulation study, and practical implications are illustrated by using data from the National Evaluation of the Enforcing Underage Drinking Laws Randomized Community Trial. The example shows that the proposed method provides useful information about the heterogeneous nature of drug use by identifying two subtypes of adolescent problem drinkers. A Monte Carlo simulation study supports the proposed estimation method by suggesting that the latent class model parameters were unbiased for 30 or more clusters. Consistent with other studies of generalized estimating equation (GEE) estimators, the robust covariance estimator tended to underestimate the true variance of regression parameters, but the degree of inflation in the test size was relatively small for 70 clusters and only slightly inflated for 30 clusters. The proposed model for studying adolescent drug use provides an alternative to standard diagnostic criteria, focusing on the nature of the drug-use profile, rather than relying on univariate symptom counts. The second-order GEE-type estimation procedure provided a computationally feasible approach that performed well for a moderate number of clusters and was consistent with prior studies of GEE under the generalized linear model framework.

Fast cross-validation of high-breakdown resampling methods for PCA

Cross-validation (CV) is a very popular technique for model selection and model validation. The general procedure of leave-one-out CV (LOO-CV) is to exclude one observation from the data set, to construct the fit of the remaining observations and to evaluate that fit on the item that was left out. In classical procedures such as least-squares regression or kernel density estimation, easy formulas can be derived to compute this CV fit or the residuals of the removed observations. However, when high-breakdown resampling algorithms are used, it is no longer possible to derive such closed-form expressions. High-breakdown methods are developed to obtain estimates that can withstand the effects of outlying observations. Fast algorithms are presented for LOO-CV when using a high-breakdown method based on resampling, in the context of robust covariance estimation by means of the MCD estimator and robust principal component analysis. A robust PRESS curve is introduced as an exploratory tool to select the number of principal components. Simulation results and applications on real data show the accuracy and the gain in computation time of these fast CV algorithms.

Robustness properties of a robust partial least squares regression method

The presence of multicollinearity in regression data is no exception in real life examples. Instead of applying ordinary regression methods, biased regression techniques such as principal component regression and ridge regression have been developed to cope with such datasets. In this paper, we consider partial least squares (PLS) regression by means of the SIMPLS algorithm. Because the SIMPLS algorithm is based on the empirical variance–covariance matrix of the data and on least squares regression, outliers have a damaging effect on the estimates. To reduce this pernicious effect of outliers, we propose to replace the empirical variance–covariance matrix in SIMPLS by a robust covariance estimator. We derive the influence function of the resulting PLS weight vectors and the regression estimates, and conclude that they will be bounded if the robust covariance estimator has a bounded influence function. Also the breakdown value is inherited from the robust estimator. We illustrate the results using the MCD estimator and the reweighted MCD estimator (RMCD) for low-dimensional datasets. Also some empirical properties are provided for a high-dimensional dataset.

Robust Modeling of Multivariate Financial Data

The bottom line in many statistical analysis in finance is the basic issue of modeling a set of multivariate data. Financial data are characterized by their fat tails containing some proportion of extreme observations. We propose a simple model able to capture these main characteristics, and to provide a good fit for the bulk of the data as well as for the atypical observations. Basically, we use a robust covariance estimator to define the center and orientations of the data, and the classical sample covariance to estimate how inflated could this distribution be by the effect of extreme observations. Estimation of the model is done either empirically or by maximum likelihood based on elliptical distributions. Simulation experiments verified the adequacy of the model to real data. We provide illustrations of the usefulness of the proposed procedure, in particular when constructing efficient frontiers. We show that robust portfolios may yield higher cumulative returns and have more stable weights.

Robust factor analysis

Our aim is to construct a factor analysis method that can resist the effect of outliers. For this we start with a highly robust initial covariance estimator, after which the factors can be obtained from maximum likelihood or from principal factor analysis (PFA). We find that PFA based on the minimum covariance determinant scatter matrix works well. We also derive the influence function of the PFA method based on either the classical scatter matrix or a robust matrix. These results are applied to the construction of a new type of empirical influence function (EIF), which is very effective for detecting influential data. To facilitate the interpretation, we compute a cutoff value for this EIF. Our findings are illustrated with several real data examples.

Nearest-Neighbor Variance Estimation (NNVE)

Robust covariance estimation commonly proceeds by downweighting outliers. In this article we measure the “outlyingness” of a data point by the standardized distance between the point and its Kth nearest neighbor. The appropriate weights for robust estimation are found by a model-based mixture modeling approach that follows from considering the data cloud as a realization of a high-dimensional point process. To correct a potential bias when there are no outliers, we introduce a boundary correction procedure that artificially adds in extra outlying points; the resulting methodology is called nearest-neighbor variance estimation (NNVE). The strength of NNVE is its robustness against a large proportion of noise points and against deviation from normality of the signal. A consistency result for the method is established. Under some reasonable assumptions, it is shown that the covariance estimate is bounded and that each point has only bounded influence on the final estimates. NNVE outperformed the popular minimum volume ellipsoid (MVE) estimator in simulation studies, with a big improvement when the proportion of outliers was very large (> 50%). In our simulation study, when the proportion of outliers was ≥ 50%, the mean squared error of the NNVE estimator of variance was at least 100 times smaller than that of the MVE estimator. The proposed estimator also outperformed MVE in cases where the underlying data distribution was not normal. Good performance of NNVE in several real examples is demonstrated. A potential drawback of NNVE is that data points condensed in moderate-sized clusters would be classified as signal. Even though we do not support approaches discarding moderate-sized clusters as outliers without checking, this feature of NNVE could be problematic, particularly when only the main data cloud is of interest. A simple diagnostic tool built on existing model-based clustering procedure is proposed. This procedure allows us to check whether there is more than one separate data cloud in the data after cleaning. It also supplies the central locations of the separated moderate-sized clusters, which allows further investigation. Finally, because NNVE reduces the problem of finding the robustness weights to a one-dimensional problem, it may be useful in high-dimensional problems, such as those encountered in data mining.

Nearest-neighbor variance estimation (NNVE): Robust covariance estimation via nearest-neighbor cleaning - Comment

Nearest-neighbor variance estimation (NNVE): Robust covariance estimation via nearest-neighbor cleaning - Comment

Functional methods for logistic regression on random-effect-coefficients for longitudinal measurements

We study logistic regression analysis when covariate variables are the underlying regression coefficients of another random effects model. For each subject, the covariate variables to the primary regression model are not observed, but can be estimated from observed longitudinal measurements. Wang et al. (Biometrics 56 (2000) 487–495) investigated estimation methods based on the regression calibration approximation and the expected estimating equations conditional on observed data. In this paper, we extend the sufficiency score and conditional score estimators of Stefanski and Carroll (Biometrics 74 (1987) 703–716) to this problem. These methods do not need the assumption of the underlying distribution of the random effects for each subject. We apply a robust sandwich covariance estimation procedure for both methods. Simulation results are provided for various random effects distributions.

A covariance estimator for GEE with improved small-sample properties.

In this paper, we propose an alternative covariance estimator to the robust covariance estimator of generalized estimating equations (GEE). Hypothesis tests using the robust covariance estimator can have inflated size when the number of independent clusters is small. Resampling methods, such as the jackknife and bootstrap, have been suggested for covariance estimation when the number of clusters is small. A drawback of the resampling methods when the response is binary is that the methods can break down when the number of subjects is small due to zero or near-zero cell counts caused by resampling. We propose a bias-corrected covariance estimator that avoids this problem. In a small simulation study, we compare the bias-corrected covariance estimator to the robust and jackknife covariance estimators for binary responses for situations involving 10-40 subjects with equal and unequal cluster sizes of 16-64 observations. The bias-corrected covariance estimator gave tests with sizes close to the nominal level even when the number of subjects was 10 and cluster sizes were unequal, whereas the robust and jackknife covariance estimators gave tests with sizes that could be 2-3 times the nominal level. The methods are illustrated using data from a randomized clinical trial on treatment for bone loss in subjects with periodontal disease.

A diagnostic for Cox regression with discrete failure-time models.

Changes in maximum likelihood parameter estimates due to deletion of individual observations are useful statistics, both for regression diagnostics and for computing robust estimates of covariance. For many likelihoods, including those in the exponential family, these delete-one statistics can be approximated analytically from a one-step Newton-Raphson iteration on the full maximum likelihood solution. But for general conditional likelihoods and the related Cox partial likelihood, the one-step method does not reduce to an analytic solution. For these likelihoods, an alternative analytic approximation that relies on an appropriately augmented design matrix has been proposed. In this paper, we extend the augmentation approach to explicitly deal with discrete failure-time models. In these models, an individual subject may contribute information at several time points, thereby appearing in multiple risk sets before eventually experiencing a failure or being censored. Our extension also allows the covariates to be time dependent. The new augmentation requires no additional computational resources while improving results.

Robust covariance estimates based on resampling

Many asymptotic covariance estimates are generated using uncontaminated model distributions and thus are often based in part on the information matrix. Such covariance estimators have a low breakdown point (Donoho and Huber, 1983; Huber, 1981; Hampel et al., 1986, p. 98; Lopuhaa and Rousseeuw, 1991), even if the estimate itself has a high breakdown point. These covariance estimates will not be reliable when there are outliers present. As alternative estimates of variability for robust estimators, we consider using the bootstrapped or jackknifed sample covariance matrix. As will be shown in this paper, the bootstrapped sample covariance matrix can have a breakdown point of 1 n regardless of the breakdown point of the estimator; thus both bootstrapped and asymptotic covariance estimates may be heavily influenced by outliers even if the original estimate is not. This is not the case for the jackknifed sample covariance matrix. If an estimate is not heavily influenced by outliers, its jackknifed sample covariance estimate is not likely to be heavily influenced by outliers. On the other hand, the jackknife, but not the bootstrap, may have a low breakdown point because the covariance estimate can often be shifted to zero by shifting a few points.

Robust Singular Value Decompositions: A New Approach to Projection Pursuit

Abstract Robust location and covariance estimators are developed via general M estimation for covariance matrix eigenvectors and eigenvalues. The solution to this GM estimation problem is obtained by transforming it into a series of robust regression problems based on a new algorithm for the singular value decomposition. It is shown here that the singular value decomposition can be represented as an iteration of two steps: a least squares regression fit of the data matrix followed by a rotation to the regression hyperplanes. An algorithm to obtain the solution to this GM estimation problem is presented, along with results of a Monte Carlo study and examples of its application. In addition, it is shown how the output of this algorithm can be used to numerically search for multivariate outliers, which is especially useful in exploratory data analysis with high-dimensional data and large sample sizes, where standard graphical techniques are difficult to implement. Because the algorithm computes robust estima...

Asymptotically Minimax Adaptive Estimation. I: Upper Bounds. Optimally Adaptive Estimates

Asymptotically Minimax Adaptive Estimation. I: Upper Bounds. Optimally Adaptive Estimates

Unmasking Multivariate Outliers and Leverage Points

Abstract Detecting outliers in a multivariate point cloud is not trivial, especially when there are several outliers. The classical identification method does not always find them, because it is based on the sample mean and covariance matrix, which are themselves affected by the outliers. That is how the outliers get masked. To avoid the masking effect, we propose to compute distances based on very robust estimates of location and covariance. These robust distances are better suited to expose the outliers. In the case of regression data, the classical least squares approach masks outliers in a similar way. Also here, the outliers may be unmasked by using a highly robust regression method. Finally, a new display is proposed in which the robust regression residuals are plotted versus the robust distances. This plot classifies the data into regular observations, vertical outliers, good leverage points, and bad leverage points. Several examples are discussed.

Unmasking Multivariate Outliers and Leverage Points: Comment

Detecting outliers in a multivariate point cloud is not trivial, especially when there are several outliers. The classical identification method does not always find them, because it is based on the sample mean and covariance matrix, which are themselves affected by the outliers. To avoid this masking effect, we propose to compute distances based on very robust estimates of location and covariance. In the case of regression data, the outliers also may be unmasked by using a highly robust regression method. A new display is proposed in which the robust regression residuals are plotted versus the robust distances

Unmasking Multivariate Outliers and Leverage Points

Abstract Detecting outliers in a multivariate point cloud is not trivial, especially when there are several outliers. The classical identification method does not always find them, because it is based on the sample mean and covariance matrix, which are themselves affected by the outliers. That is how the outliers get masked. To avoid the masking effect, we propose to compute distances based on very robust estimates of location and covariance. These robust distances are better suited to expose the outliers. In the case of regression data, the classical least squares approach masks outliers in a similar way. Also here, the outliers may be unmasked by using a highly robust regression method. Finally, a new display is proposed in which the robust regression residuals are plotted versus the robust distances. This plot classifies the data into regular observations, vertical outliers, good leverage points, and bad leverage points. Several examples are discussed.

Eigenstructure variability of the multiple-source multiple-sensor covariance matrix with contaminated Gaussian data

Several methods of current interest for counting and locating signal sources using data from a passive array depend on the accuracy of estimating the eigenstructure of the covariance matrix of the array's data vectors. When errors in the measured data vectors are Gaussian conventional covariance estimation is optimal, but robust procedure are required for data with nonGaussian additive contamination. Two different robust covariance estimators are compared by simulation with the conventional one for different degrees of contamination. Even in relatively good signal-to-noise ratios, however, closeness of signal sources in the temporal and spatial frequency domains can cause inaccurate signal-related eigenvalue and eigenvector estimates. The degree of adversity for these problems is also shown by simulation. >

Robust adapative Kalman filtering for systems with unknown step inputs and non-Gaussian measurement errors

Target tracking with Kalman filters is hampered by target maneuvering and unknown process and measurement noises. We show that moving data windows may be used to analyze state and measurement error sequences, determining robust estimates of bias and covariance. For steps in the system forcing functions and non-Gaussian measurement errors, the robust estimators yield improvements over linear bias and covariance estimators. Extensive simulations compare conventional, linear adaptive, and robust adaptive average step responses of a first-order system filter. Quantities examined are state estimate, state error, process and measurement covariance estimates, Kalman gain, and input step estimate.