Yet Another Method for Heterogeneous Data Fusion and Preprocessing in Proactive Decision Support Systems: Distributed Architecture Approach

Abstract

In the multi-sensors environment, the crucial issue is collecting data from different sources. The paper considers the problem of data gathering from different data sources in the framework of proactive intelligent decision support systems design. The aim is to provide the invariate access to heterogeneous data stored in a data warehouse for further processing. There are two types of data sources are considered in the paper: machine or sensor data and video streams. We propose an ‘on-fly’ method of heterogeneous data fusion and preprocessing toward to minimisation of execution time of queries. A proposed method is implemented in the five-layer distributed architecture of the system based on Apache Kafka and Spark Streaming technology. The main conclusion is that in case of heterogeneous data (like video and loged data) and functional requirements for query execution over these data, the distributed data preprocessing might be efficient in comparison with batch processing.