Abstract The performance degradation problem of the angle of arrival (AOA)-based acoustic localization methods in the presence of unreliable bearing measurements (outliers) is addressed in this paper. Two typical M-estimators based on Tukey and Huber functions are applied to tackle the problem. Both functions are solved by the iterative reweighted nonlinear least squares (IRNLS) method. Considering the Huber function is convex in nature, it is specifically utilized to mitigate the influence of large residuals on pseudolinear estimator (PLE) by convex optimization. To make the IRNLS method more feasible to use, an approximate relationship between the outlier probability and the bound parameter is provided. The robustness and effectiveness of the proposed methods are clearly demonstrated through a series of simulation results in the presence of various unreliable measurements.