An Efficient Hadoop-Based Framework for Data Storage and Fault Recovering in Large-Scale Multimedia Sensor Networks

Abstract

In today's time, we live in the big data era where every event and thing about us is monitored and registered for a later analysis. In addition, such amount of big data is collected with a large speed (velocity) and does not fit a fixed structure (unstructured type). One of the most contributors of big data in this era is wireless multimedia sensor network (WMSN). Typically, WMSN consists of a set of sensors that collect three types of data about a zone of interest: numerical, images and videos. Indeed, the big data collected in WMSN along with the density deployment of network, especially in large-scale zones, provide real challenges for the end users in terms of data storage and processing. In this paper, we propose an efficient and robust Hadoop-based framework for big data collection, processing and storage in WMSN. The proposed framework relies on Hadoop ecosystem tools and introduces two fault detection algorithms (moving average and exponential smoothing) in order to preprocess data before storage. Through real sensor data with various types, we show the effectiveness of our framework in terms of processing storage speed and regenerating of missing data.