Design of Distributed Sensor Stream Data Input System

Abstract

Sensor networks are widely used in various environment monitoring systems. For example, in storehouse, it is used for monitoring the temperature, humidity, and carbon dioxide level inside the storehouse. Those sensing values are packaged in relatively small data, only around 100 bytes. However, we want to get those sensing values periodically every a certain cycle; every several seconds or minutes depending on the system requirement. And since in a storehouse sensor network there might be hundreds of storehouse and thousands of sensor nodes, resulting in millions of sensor data over only one day (depending on the cycle length), we are facing tremendous number of data access into the file system; using the naive onedata one-file storing mechanism. In this paper, we are trying to design a data input system which collect and combine those individuals of sensor data into a big chunk of file and store it in HDFS, while using MapReduce framework to access and process those chunks inside HDFS.