Innovative Nonparametric Method for Data Outlier Filtering

Abstract

Outlier filtering of empirical travel time data is essential for traffic analyses. Most of the widely applied outlier filtering algorithms are parametric in nature and based on assumed data distributions. The assumption, however, might not hold under unstable traffic conditions. This paper proposes a nonparametric outlier filtering method based on a robust locally weighted regression scatterplot smoothing model. The proposed method identifies outliers based on a data point’s standard residual in the robust local regression model. This approach fits a regression surface with no constraint on parametric distributions and limited influence from outliers. The proposed outlier filtering algorithm can be applied to various data collection technologies and for real-time applications. The performance of the new outlier filtering algorithm is compared with the moving standard deviation method and other traditional filtering algorithms. The test sites include GPS data of an Interstate highway in Indiana and Bluetooth data of an urban arterial roadway in Texas. It is shown that the proposed filtering algorithm has several advantages over the traditional filtering algorithms.