MSE Analysis of the Iteratively Reweighted Least Squares Algorithm when Applied to M Estimators

Abstract

M estimators have been widely used for parameter estimation in the presence of outliers or impulsive noise. A number of papers use the iteratively reweighted least squares (IRLS) algorithm for M estimation. The IRLS method tries to iteratively converge to the non-linear M estimate using a weighted least squares algorithm. While the performance of the IRLS algorithm has been demonstrated through simulation, to our knowledge, the MSE of the IRLS based M estimation approach has not been theoretically derived in signal processing literature. In this paper, we derive the theoretical MSE of three M estimators, namely, the Huber's M (HM) estimator, the extreme value theory (EVT) based estimator and the Hampel's 3-part (HP) estimator when they are implemented using the IRLS algorithm. This theoretical MSE is a function of the M estimator cost function, the noise distribution, and the iteration number of the IRLS algorithm. Based on the theoretical analysis in this paper, we show that for both Cauchy and Gaussian impulsive noise, the MSE of the IRLS based M estimator converges to the MSE of the desired M estimator within 3 to 5 iterations.