An adaptive data cleaning framework: a case study of the water quality monitoring system in China

Abstract

Robust detection of patterns at low signal-to-noise ratios (SNR) is a fundamental challenge of analyzing high-frequency data, particularly in water quality monitoring. When conducting water quality data analysis, an indispensable step is to clean the data noise. In this paper, a new method named ADAPTIVE-EWT-MFE based on empirical wavelet transform (EWT) and multi-scale fuzzy entropy (MFE) is proposed to implement time series data cleaning. EWT-MFE can decompose the spectrum into different inherent mode functions (IMFs). According to different characteristics of the IMFs, an adaptive and adjustable parameter based on MFE, which reflects the intrinsic characteristics is introduced into the threshold function to improve the performance of noise-cleaning. Finally, this hybrid data cleaning method is designed to filter the high-frequency noise on the IMFs. Results show that our proposed method not only has great potential to improve the better noise-cleaning performance but also does not distort noise-free data.