The bandwidth-aware backup task scheduling strategy using SDN in Hadoop

Abstract

In the era of big data, the traditional capacity of computing and storage has been unable to meet the growing demand. In this case, Cloud Computing technology is emerging. Researching on task scheduling is a way from the perspective of resource allocation and management to improve performance of Hadoop system. In this paper, a speculative task scheduling strategy that based on SDN technology is improved. For LATE mechanism, some slow tasks are slower than speculative tasks. This is not only unable to reduce task turnaround time and a waste of system resources. In this paper, we join the slow task compared with the speculative task for the speculative task scheduling strategy of LATE. Wherein, the run time of speculative tasks contains the input data transfer time, real-time bandwidth corresponding to a bandwidth of the link. Based on this model, we propose a bandwidth-aware speculative task run time estimation model (BWRE) based on SDN, using this model to accurately speculative the backup task run time. And we use SDN to provide bandwidth guarantees for the speculative task. Finally, BWRE is verified by simulation experiments. Evaluation results show that BWRE outperforms the shortening job turnaround time by an average of 9.85%.