QoS-aware Resource Provisioning for Big Data Processing in Cloud Computing Environment

Abstract

Nowadays, the quantity of collected data from many different sources is increasing dramatically. As the traditional on-hand computing resources are not sufficient enough to handle Big data, deploying the processing services into clouds is becoming an inevitable trend. For QoS (quality of service)-aware Big data processing, a specially designed cloud resource allocation approach is required. Presently, it is challenging to incorporate the comprehensive QoS demand of Big data with cloud while minimizing the total cost. In order to solve this problem, a general problem formulation is established in this paper. By giving certain assumptions, we prove that the reduction of resource waste has a direct relation with cost minimization. Based on that, we develop efficient heuristic algorithms with tuning parameters to find cost minimized dynamic resource allocation solutions for the above-mentioned problem. In paper, we study and test the workload of Big data by running a group of typical Big data jobs, i.e., video surveillance services, on Amazon Cloud EC2. Then we create a large simulation scenario and compare our proposed method with other approaches.