Increased Task Execution with a Bandwidth-Aware Hadoop Scheduling Approach

Abstract

This research presents a novel bandwidth-aware Hadoop scheduling method that addresses the challenge of task scheduling in Hadoop clusters while considering the real-time network conditions. The proposed method involves the establishment of a job time completion model and a mathematical model for a Hadoop scheduling system. Furthermore, it transforms the Hadoop task scheduling problem into an optimization problem to find the task scheduling method that minimizes job completion time. By leveraging Software-Defined Networking (SDN) capabilities, a time slot-based network bandwidth allocation mechanism is introduced to allocate bandwidth fairly across network links. The proposed method also takes into account task locality and network bandwidth availability when allocating computational nodes for individual tasks. Through this approach, the limitations of existing methods, which fail to simultaneously consider global task scheduling and actual network bandwidth availability, are overcome. Experimental evaluations demonstrate the effectiveness of the proposed method in enhancing the performance of Hadoop task scheduling.