Anomaly detection with a container-based stream processing framework for Industrial Internet of Things

Abstract

Online anomaly detection is a key challenge for industrial internet of things (IIoT) applications, as anomalies may occur in data streams from sensors and cause losses or damages. However, most existing methods for online anomaly detection have limitations in efficiency, effectiveness and timeliness, especially with the massive and distributed data streams from IIoT devices. Therefore, developing a data stream processing framework to discover anomalies in time and ensure the proper operation of the system is an urgent issue for IIoT. In this paper, we propose a flexible stream processing framework that enables online anomaly detection for IIoT applications. The framework exploits a distributed computing architecture based on docker containers to improve flexibility, migration capability and customization. The framework also uses a central mediator to coordinate data stream processing tasks running on different docker nodes. Moreover, we develop a prediction-based online anomaly detection model that consists of batch model training and data stream anomaly detection processes. The model uses long short-term memory (LSTM) neural networks to predict data stream values and a dynamic sliding window method to model prediction errors and detect anomalies. We implement a case study to detect abnormal heating temperatures from an industrial heating plant and evaluate the performance of the proposed framework and anomaly detection model. The results show that our framework and model can achieve high accuracy and low latency in detecting anomalies, and they outperform existing methods in terms of scalability, efficiency and adaptability for IIoT applications.