Efficient Resource Management System Based on 4Vs of Big Data Streams

Abstract

Big data streams are generated continuously at unprecedented speed by thousands of data sources. The analysis of such streams need cloud resources. Due to growth of big data over cloud, allocating appropriate cloud resources has emerged as a major research problem. The current methodologies allocate cloud resources based upon data characteristics. But due to random nature of data generation, the characteristics of data in big data streams are unknown. This poses difficulty in selecting and allocating appropriate resources to big data stream. Solving this problem, an efficient resource management system is proposed in this paper. The proposed system initially estimates the data characteristics of big data stream in terms of volume, velocity, variety and variability. The estimated values are expressed in terms of a vector called Characteristics of Data (CoD). On the other hand, clusters of cloud resources are created dynamically with the help of Self-Organizing Maps (SOM). SOM uses CoD to create and allocate cluster to big data stream. Moreover, the topological ordering of clusters formed by SOM is used to reduce waiting time. The proposed system is tested experimentally. The experimental results show that the proposed system not only efficiently predicts data characteristics but also effectively enhanced the performance of cloud resources.