An Adaptive Feedback Load Balancing Algorithm in HDFS

Abstract

As one of the most popular cloud storage distributed file systems, Hadoop Distributed File System (HDFS) has the characteristics of open source, low-cost, high fault tolerance and high scalability. Thus, HDFS plays a critical role in cloud storage. However, there are some important issues in HDFS which should be resolved urgently, such as the insufficient load balancing. In order to improve the shortages of insufficient load balancing in storage server, we propose a new adaptive feedback load balancing algorithm in HDFS (AFLBA) in this paper, which is mainly achieved by the establishment of the disk utilization rate model and the service blocking rate model. The disk utilization rate model describes the disk utilization of each Data node precisely, and the service blocking rate model quantifies the service busy degree to qualitatively describe the service performance of each Data node using an integrated parameter. Analysis and simulation demonstrate the AFLBA has improved the load balancing greatly in HDFS compared with the random static load balancing method used in exiting HDFS system.