Robust smoothing of one‐dimensional data with missing and/or outlier values

Abstract

Penalized least squares (PLS) is a popular data smoothing technique. However, existing PLS smoothing algorithms behave as low-pass filters (LPF), and, hence, they may introduce distortions to bandpass signals. These algorithms also have difficulties associated with the selection of the smoothing parameter and the order of the difference matrix they utilize. A new PLS smoothing algorithm is developed to overcome these limitations. In the proposed algorithm, two difference matrices and two regularisation parameters are utilized. As a result, the proposed algorithm can act as either an LPF or a bandpass filter. The particular shape of the filter is determined by the values of the regularisation parameters and the orders of the two difference matrices. The spectral characteristics of the smooth signal are utilized to derive closed-form expressions for the optimum values of these four parameters. The derived algorithm is further developed to handle the case of having missing and/or outlier samples in the measured data. The proposed algorithms are completely data driven, and they do not require any special optimisation tools. In addition, the proposed algorithms are also shown to produce significantly improved results compared to some existing smoothing techniques.