Huber M-estimator Research Articles

Fuzzy c-ordered-means clustering

Fuzzy clustering helps to find natural vague boundaries in data. The fuzzy c-means method is one of the most popular clustering methods based on minimization of a criterion function. However, one of the greatest disadvantages of this method is its sensitivity to the presence of noise and outliers in data. This paper introduces a new robust fuzzy clustering method named Fuzzy C-Ordered-Means (FCOM) clustering. This method uses both the Huber's M-estimators and the Yager's OWA operators to obtain its robustness. The proposed method is compared to many other ones, e.g.: the Fuzzy C-Means (FCM), the Possibilistic Clustering (PC), the fuzzy Noise Clustering Method (NCM), the Lp norm clustering (Lp FCM) (0<p<1), the L1 norm clustering (L1 FCM), the Fuzzy Clustering with Polynomial Fuzzifier (PFCM) and the ε-insensitive Fuzzy C-Means (βFCM). To this end experiments on synthetic data with outliers have been performed as well as on data with heavy-tailed and overlapping groups of points in background noise.

On robust fuzzy c-regression models

One of the most popular clustering methods based on minimization of a criterion function is the fuzzy c-means one. Its generalization by application of hyperplane shaped prototypes of the clusters is known as the Fuzzy C-Regression Models (FCRM) method. Although with this generalization many new applications of clustering emerged, it appeared to be rather sensitive to poor initialization and to the presence of noise and outliers in data. In this paper we introduce a new objective function, using the Huber's M-estimators and the Yager's OWA operators to overcome the disadvantages of the approach considered. We derive and describe an algorithm for minimization of the objective function defined. We have called it the Fuzzy C-Ordered-Regression Models (FCORM) clustering algorithm. The algorithm is compared to a few other important reference ones. To this end experiments on synthetic data with various types of noise and different numbers of outliers are carried out. We investigate the methods performance in the conditions that can be encountered in signal analysis. Large-scale simulations demonstrate the competitiveness and usefulness of the method proposed.

Comparison of Statistical Methods for Pretest–Posttest Designs in Terms of Type I Error Probability and Statistical Power

The pretest–posttest design is widely used to investigate the effect of an experimental treatment in biomedical research. The treatment effect may be assessed using analysis of variance (ANOVA) or analysis of covariance (ANCOVA). The normality assumption for parametric ANOVA and ANCOVA may be violated due to outliers and skewness of data. Nonparametric methods, robust statistics, and data transformation may be used to address the nonnormality issue. However, there is no simultaneous comparison for the four statistical approaches in terms of empirical type I error probability and statistical power. We studied 13 ANOVA and ANCOVA models based on parametric approach, rank and normal score-based nonparametric approach, Huber M-estimation, and Box–Cox transformation using normal data with and without outliers and lognormal data. We found that ANCOVA models preserve the nominal significance level better and are more powerful than their ANOVA counterparts when the dependent variable and covariate are correlated. Huber M-estimation is the most liberal method. Nonparametric ANCOVA, especially ANCOVA based on normal score transformation, preserves the nominal significance level, has good statistical power, and is robust for data distribution.

Robust Recovery of Corrupted Low-RankMatrix by Implicit Regularizers.

Low-rank matrix recovery algorithms aim to recover a corrupted low-rank matrix with sparse errors. However, corrupted errors may not be sparse in real-world problems and the relationship between ℓ1 regularizer on noise and robust M-estimators is still unknown. This paper proposes a general robust framework for low-rank matrix recovery via implicit regularizers of robust M-estimators, which are derived from convex conjugacy and can be used to model arbitrarily corrupted errors. Based on the additive form of half-quadratic optimization, proximity operators of implicit regularizers are developed such that both low-rank structure and corrupted errors can be alternately recovered. In particular, the dual relationship between the absolute function in ℓ1 regularizer and Huber M-estimator is studied, which establishes a connection between robust low-rank matrix recovery methods and M-estimators based robust principal component analysis methods. Extensive experiments on synthetic and real-world data sets corroborate our claims and verify the robustness of the proposed framework.

Half-Quadratic-Based Iterative Minimization for Robust Sparse Representation

Robust sparse representation has shown significant potential in solving challenging problems in computer vision such as biometrics and visual surveillance. Although several robust sparse models have been proposed and promising results have been obtained, they are either for error correction or for error detection, and learning a general framework that systematically unifies these two aspects and explores their relation is still an open problem. In this paper, we develop a half-quadratic (HQ) framework to solve the robust sparse representation problem. By defining different kinds of half-quadratic functions, the proposed HQ framework is applicable to performing both error correction and error detection. More specifically, by using the additive form of HQ, we propose an ℓ1-regularized error correction method by iteratively recovering corrupted data from errors incurred by noises and outliers; by using the multiplicative form of HQ, we propose an ℓ1-regularized error detection method by learning from uncorrupted data iteratively. We also show that the ℓ1-regularization solved by soft-thresholding function has a dual relationship to Huber M-estimator, which theoretically guarantees the performance of robust sparse representation in terms of M-estimation. Experiments on robust face recognition under severe occlusion and corruption validate our framework and findings.

Convex Optimization inR

Convex optimization now plays an essential role in many facets of statistics. We briefly survey some recent developments and describe some implementations of these methods in R . Applications of linear and quadratic programming are introduced including quantile regression, the Huber M-estimator and various penalized regression methods. Applications to additively separable convex problems subject to linear equality and inequality constraints such as nonparametric density estimation and maximum likelihood estimation of general nonparametric mixture models are described, as are several cone programming problems. We focus throughout primarily on implementations in the R environment that rely on solution methods linked to R, like MOSEK by the package Rmosek. Code is provided in R to illustrate several of these problems. Other applications are available in the R package REBayes, dealing with empirical Bayes estimation of nonparametric mixture models.

Using the R-MAPE index as a resistant measure of forecast accuracy.

The mean absolute percentage error (MAPE) is probably the most widely used goodness-of-fit measure. However, it does not meet the validity criterion due to the fact that the distribution of the absolute percentage errors is usually skewed to the right, with the presence of outlier values. In these cases, MAPE overstates the corresponding population parameter. In this study, we propose an alternative index, called Resistant MAPE or R-MAPE based on the calculation of the Huber M-estimator, which allows overcoming the aforementioned limitation. The results derived from the application of Artificial Neural Network (ANN) and Autoregressive Integrated Moving Average (ARIMA) models are used to forecast a time series. The arithmetic mean, MAPE, overstates the corresponding population parameter, unlike R-MAPE, on a set of error distributions with a statistically significant right skew, as well as outlier values. Our results suggest that R-MAPE represents a suitable alternative measure of forecast accuracy, due to the fact that it provides a valid assessment of forecast accuracy compared to MAPE.

Robust derivative-free Kalman filter based on Huber's M-estimation methodology

In this study, a discrete-time robust nonlinear filtering algorithm is proposed to deal with the contaminated Gaussian noise in the measurement, which is based on a robust modification of the derivative-free Kalman filter. By interpreting the Kalman type filter (KTF) as the recursive Bayesian approximation, the innovation is reformulated capitalizing on the Huber's M-estimation methodology. The proposed algorithm achieves not only the robustness of the M-estimation but also the accuracy and flexibility of the derivative-free Kalman filter for the nonlinear problems. The reliability and accuracy of the proposed algorithm are tested in the Univariate Nonstationary Growth Model.

Robust tests for one or more allometric lines

In allometry, the study of how size variables scale against each other, it is often of interest to fit lines to bivariate data and test hypotheses about slope and elevation about one or several lines. The nature of the problem suggests that bivariate techniques related to principal component analysis are more appropriate than linear regression. Inference methods have been developed for this problem and are in widespread use, however, we demonstrate that such methods are not robust to bivariate contamination, and propose alternative approaches which are. The new approaches use Huber's M-estimator via a plug-in approach, where robust test procedures have the same form as classical ones, but where we plug in robust estimators of parameters and standard errors in place of classical estimators. Simulations demonstrate that these new procedures are robust against bivariate contamination, and can make accurate inferences even from small samples.

A New Outlier Detection Method Considering Outliers As Model Errors

Outlier detection is an important task for fitting a model to a set of data. Two different outlier detection approaches are given as tests for outliers and robust methods. For these approaches, usually outliers are considered as additive bias terms neglected in the original adjustment model. However, there is another approach that outlier is considered as a model error in the Gauss–Markov model. This model error is represented as an unknown parameter. As it cannot be known before which observation includes outlier; this method is applied on the data for each observation separately and tested with t-test or F-test. It is successful if the sample includes only one outlier. To detect multiple outliers more successfully, in this article, a new outlier detection method is introduced. In this method, all the possible combinations of multiple outliers are considered as model errors and it is accepted that the smallest variance of them gives the solution for a certain number of outliers, then the estimated model errors are detected by comparing with a critical value. The critical value is chosen as 3σo. To compare the results of the new method, with those of the Least Median of Squares (LMS) and Huber M-estimators, Monte Carlo simulation technique is used for linear regression. The Mean Success Rate is proposed to measure the reliabilities of the methods. We showed that the new method is robust and includes the property of high breakdown point as LMS; and more efficient than LMS.

Robust estimation and inference for bivariate line-fitting in allometry

In allometry, bivariate techniques related to principal component analysis are often used in place of linear regression, and primary interest is in making inferences about the slope. We demonstrate that the current inferential methods are not robust to bivariate contamination, and consider four robust alternatives to the current methods -- a novel sandwich estimator approach, using robust covariance matrices derived via an influence function approach, Huber's M-estimator and the fast-and-robust bootstrap. Simulations demonstrate that Huber's M-estimators are highly efficient and robust against bivariate contamination, and when combined with the fast-and-robust bootstrap, we can make accurate inferences even from small samples.

Multiple Comparisons Based on Studentized M-Statistics in a Randomized Block Design

In a randomized block design, robust versions based on scale-invariant M-statistics are proposed for single-step multiple comparisons procedures discussed by Tukey (1953) and Dunnett (1955). Although the symmetry of the underlying distribution is needed in the asymptotic theory of Huber's M-estimators, the proposed procedures do not demand the symmetry. It is found that the M-procedures are superior to the classical normal theory procedures except the case that an underlying distribution is normal.

MMLEs are as good as M-estimators or better

Tiku–Suresh modified maximum likelihood estimators necessitate the assumption of a particular distribution. New forms of the estimators which, like Huber M-estimators, only assume that the distribution is long-tailed symmetric (unspecified) are given. They have high breakdown and, through simulations, are shown to be overall more efficient than M-estimators.

Evaluation of Different Outlier Detection Methods for GPS Networks.

GPS (Global Positioning System) devices can be used in many applications which require accurate point positioning in geosciences. Accuracy of GPS decreases due to outliers resulted from the errors inherent in GPS observations. Several approaches have been developed to detect outliers in geodetic observations. It is important to determine which method is most effective at distinguishing outliers from normal observations. This paper investigates the behavior of conventional statistical test methods (Data Snooping (DS), Tau and t tests), some robust methods (Andrews's M-Estimation, Huber's M-Estimation, Tukey's M-Estimation, Danish Method, Yang-I M-Estimation, Yang-II M-Estimation, and fuzzy logic method in detection of outliers for three GPS networks having different characteristics. Test results are evaluated and the performances of different methods are presented quantitatively.

Robust Ultra-Wideband Signal Acquisition

Ultra-wideband (UWB) communication is envisaged to be deployed in indoor environments, where the noise distribution is decidedly non-Gaussian. A critical challenge for impulse radio UWB is the synchronization of nanosecond-long pulses. In this paper, we propose two acquisition schemes which are robust to uncertainties in the noise distribution and which operate directly on samples taken at symbol-rate. The only channel state information that the algorithms need can be obtained by estimation of an aggregate channel gain, avoiding channel-related complications. Following Huber's M-estimates for the Gaussian mixture noise model, the proposed robust acquisition systems outperform their traditional counterparts designed according to the Gaussian noise assumption. Necessary modifications for operation under multiple access interference are also introduced. Theoretical and simulation-based performance evaluations that reflect the asymptotic variance, normalized mean-square error and the bit error rates demonstrate the gains offered by the robust procedures.

Empirical Choice of the Shape Parameter for Robust Support Vector Machines

Inspired by using a robust loss function in the support vector machine regression to control training error and the idea of robust template matching with M-estimator, Chen (2004) applies M-estimator techniques to gaussian radial basis functions and form a new class of robust kernels for the support vector machines. We are specially interested in the shape of the Huber's M-estimator in this context and propose a way to find the shape parameter of the Huber's M-estimating function. For simplicity, only the two-class classification problem is considered.

Symmetrised M-estimators of multivariate scatter

In this paper we introduce a family of symmetrised M-estimators of multivariate scatter. These are defined to be M-estimators only computed on pairwise differences of the observed multivariate data. Symmetrised Huber's M-estimator and Dümbgen's estimator serve as our examples. The influence functions of the symmetrised M-functionals are derived and the limiting distributions of the estimators are discussed in the multivariate elliptical case to consider the robustness and efficiency properties of estimators. The symmetrised M-estimators have the important independence property; they can therefore be used to find the independent components in the independent component analysis (ICA).

Simple, Robust, and Memory-Efficient FastICA Algorithms Using the Huber M-Estimator Cost Function

The goal of blind source separation is to separate multiple signals from linear mixtures without extensive knowledge about the statistical properties of the unknown signals. The design of separation criteria that achieve accurate and robust source estimates within a simple adaptive algorithm is an important part of this task. The purpose of this paper is threefold: (1) We introduce the Huber M-estimator cost function as a contrast function for use within prewhitened blind source separation algorithms such as the well-known and popular FastICA algorithm of Hyvarinen and Oja. The resulting algorithm obtained from this cost is particularly simple to implement. We establish key properties regarding the local stability of the algorithm for general non-Gaussian source distributions, and its separating capabilities are shown through analysis to be largely insensitive to the cost function's single threshold parameter. (2) We illustrate the use of the Huber M-estimator cost as a criterion within the winning algorithm entry for the blind source separation portion of the first Machine Learning for Signal Processing Workshop Data Analysis Competition, describing the key features of the algorithm design for successful separation of large-scale and ill-conditioned signal mixtures with reduced data set requirements. (3) We show how the FastICA algorithm can be implemented without significant additional memory resources by careful use of sequential processing strategies.

Sensitivity Analysis for m‐Estimates, Tests, and Confidence Intervals in Matched Observational Studies

Huber's m-estimates use an estimating equation in which observations are permitted a controlled level of influence. The family of m-estimates includes least squares and maximum likelihood, but typical applications give extreme observations limited weight. Maritz proposed methods of exact and approximate permutation inference for m-tests, confidence intervals, and estimators, which can be derived from random assignment of paired subjects to treatment or control. In contrast, in observational studies, where treatments are not randomly assigned, subjects matched for observed covariates may differ in terms of unobserved covariates, so differing outcomes may not be treatment effects. In observational studies, a method of sensitivity analysis is developed for m-tests, m-intervals, and m-estimates: it shows the extent to which inferences would be altered by biases of various magnitudes due to nonrandom treatment assignment. The method is developed for both matched pairs, with one treated subject matched to one control, and for matched sets, with one treated subject matched to one or more controls. The method is illustrated using two studies: (i) a paired study of damage to DNA from exposure to chromium and nickel and (ii) a study with one or two matched controls comparing side effects of two drug regimes to treat tuberculosis. The approach yields sensitivity analyses for: (i) m-tests with Huber's weight function and other robust weight functions, (ii) the permutational t-test which uses the observations directly, and (iii) various other procedures such as the sign test, Noether's test, and the permutation distribution of the efficient score test for a location family of distributions. Permutation inference with covariance adjustment is briefly discussed.

Robust regression techniques: A useful alternative for the detection of outlier data in chemical analysis

The validation of an analytical procedure means the evaluation of some performance criteria such as accuracy, sensitivity, linear range, capability of detection, selectivity, calibration curve, etc. This implies the use of different statistical methodologies, some of them related with statistical regression techniques, which may be robust or not. The presence of outlier data has a significant effect on the determination of sensitivity, linear range or capability of detection amongst others, when these figures of merit are evaluated with non-robust methodologies. In this paper some of the robust methods used for calibration in analytical chemistry are reviewed: the Huber M-estimator; the Andrews, Tukey and Welsh GM-estimators; the fuzzy estimators; the constrained M-estimators, CM; the least trimmed squares, LTS. The paper also shows that the mathematical properties of the least median squares (LMS) regression can be of great interest in the detection of outlier data in chemical analysis. A comparative analysis is made of the results obtained by applying these regression methods to synthetic and real data. There is also a review of some applications where this robust regression works in a suitable and simple way that proves very useful to secure an objective detection of outliers. The use of a robust regression is recommended in ISO 5725-5.