Dynamic resource allocation for big data streams based on data characteristics (5Vs)

Abstract

SummaryVarious Internet‐based applications such as social media, business transactions, mobile applications, cyber‐physical systems, and Internet of Things have led to the generation of big data streams in every field. The growing need to extract knowledge from big data streams has pioneered the challenge of selecting appropriate cloud resources. The current techniques allocate resources based on data characteristics. But because of the stochastic nature of data generation, the characteristics of data in big data streams are unknown. This poses difficulty in selecting and allocating appropriate resources to big data stream. Working towards this direction, this paper proposes a system that predicts the data characteristics in terms of volume, velocity, variety, variability, and veracity. The predicted values are expressed in a quadruple called Characteristics of Big data (CoBa). Thereafter, the proposed system uses self‐organizing maps to dynamically create clusters of cloud resources. One of these clusters is allocated to the big data stream based on its CoBa. The proposed system is dynamic in the sense that it changes the cloud cluster allocated to big data stream if its CoBa changes. Experimental results show that the proposed system has a performance edge over other streaming data processing tools such as Storm, Flume, and S4.