An Adaptive Outlier Detection and Processing Approach Towards Time Series Sensor Data

Minghu Zhang,Lili Wang,Xin Li

doi:10.1109/access.2019.2957602

Abstract

The intelligent environment monitoring network, as the foundation of ecosystem research, has rapidly developed with the ever-growing Internet of Things (IoT). IoT-networked sensors deployed to monitor ecosystems generate copious sensor data characterized by nonstationarity and nonlinearity such that outlier detection remains a source of concern. Most outlier detection models involve hypothesis tests based on setting outlier threshold values. However, signal decomposition describes stationary and nonstationary relationships sensor data. Therefore, this paper proposes a three-level hybrid model based on the median filter (MF), empirical mode decomposition (EMD), classification and regression tree (CART), autoregression (AR) and exponential weighted moving average (EWMA) methods called MF-EMD-CART-AR-EWMA to detect outliers in sensor data. The first-level performance is compared to that of the Butterworth filter, FIR filter, moving average filter, wavelet filter and Wiener filter. The second-level prediction performance is compared to support vector regression (SVR), K-nearest neighbor (KNN), CART, complementary ensemble EEMD with CART and AR (EEMD-CART-AR) and ensemble CEEMD with CART and AR (CEEMD-CART-AR) methods. Finally, EWMA is compared to Cumulative Sum Control Chart (CUSUM) and Shewhart control charts. The proposed hybrid model was evaluated with a real dataset from the hydrometeorological observation network in the Heihe River Basin, yielding experimental results with better generalization ability and higher accuracy than the compared models, and providing extremely effective detection of minor outliers in predicted values. This paper provides valuable insight and a promising reference for outlier detection involving sensor data and presents a new perspective for detecting outliers.

Highlights

The intelligent environment monitoring network consists of numerous sensor devices that form a ubiquitous, reliable and distributed internet of things (IoT) network for sensing and communicating which is gradually driving the evolution of ecosystem research, and massive amounts of time series sensor data have been collected [1]–[3]
Y ArouH (t) is decomposed into 10 components, including 9 intrinsic mode functions (IMFs), namely, IM Fi(i = 1, 2, · · ·, 9), and 1 trend term rArouH (t), which are given in Fig.9 To obtain relatively stable original data and the partial stationary trend, the IMFs in IMFi are reconstructed
In this paper, the proposed three-level hybrid model, which integrates preprocessing, prediction and outlier detection tasks, achieves excellent performance in outlier detection for non-stationary and nonlinear data collected by environment monitoring network networked sensors

Summary

Introduction

The intelligent environment monitoring network consists of numerous sensor devices that form a ubiquitous, reliable and distributed internet of things (IoT) network for sensing and communicating which is gradually driving the evolution of ecosystem research, and massive amounts of time series sensor data have been collected [1]–[3]. According to a published report, the total amount of global Earth monitoring. Devices deployed in high altitude and harsh regions often generate spatiotemporal variations in networked sensor data [5], [6]. The sensed data are largely affected by the environment of the underlying surface of the atmosphere in cold and arid regions, where high uncertainty can be caused by local climate change with non-stationary and nonlinear characteristics [7].

Objectives

Methods

Results

Discussion

Conclusion