Recovering Missing Data via Top-k Repeated Patterns for Fuzzy-Based Abnormal Node Detection in Sensor Networks

Abstract

The stream data acquired by heterogeneous Internet of Things (IoT) sensors are seldom perfect. Most of the collected data streams include either missing or abnormal values caused by various factors such as failure, malfunction, or integrity attacks. Such unreliable data affect the real-time monitoring and compromise the quality of data analysis. By simply analyzing the sensor data via anomaly detection, applications may still be unreliable over the incomplete sensor data streams. Therefore, a reliable method for recovering the missing data and detecting the abnormal ones is indispensable in the IoT environment. This paper presents FuzHD++, a new method to recover missing sensor data and detect abnormal nodes jointly rather than independently. Both elements, data recovery and abnormal node detection, rely on the observed temporal and spatial correlation of sensor data to effectively achieve reliable recovery estimation and detection performance. In the data recovery process, the system adopts a matrix profile to extract the top-<i>k</i> repeated patterns from different sensor nodes. Furthermore, it utilizes the <i>k</i>-nearest neighbor estimator to recover the missing data based on the extracted pattern information of multiple neighbor nodes. During the abnormal node detection process, the system adopts a refined fuzzy rule-based detection method. The refined fuzzy rule-based inference system integrates the expert rules and the rules obtained from sensor data analysis to treat the ambiguity in the decision-making process. We validated the performance of FuzHD++ by comparing it with existing methods using two real-world datasets. Our results showed that the proposed missing sensor data recovery method achieves more than 20% improved root mean square error results than most existing methods. Furthermore, FuzHD++ achieved an average accuracy of 92% for analyzing the sensor readings and detecting the abnormal ones. According to the results, the proposed mechanisms based on the observed temporal and spatial correlation analysis improve the robustness of IoT against data loss and integrity attacks.