New efficient Hadoop scheduler: Generalized particle swarm optimization and simulated annealing‐dominant resource fairness

Abstract

SummaryDigital growth during the Corona pandemic has generated massive data. The Hadoop in big data has to be more efficient in resource handling and job scheduling. This article proposes the improved job scheduler which is more efficient in fair job scheduling even with the heterogeneous resources. The faster job execution depends upon the localization too. The nearer the slots are, the faster is the execution. So, this article proposes a hybrid metaheuristic algorithm with fair scheduling and data locality as the two objectives in job scheduling. The dominant resource fairness policy in Hadoop YARN is updated by hybrid generalized particle swarm optimization and simulated annealing for minimum locality and maximum fairness in scheduling. The algorithm is tested on various workloads for heterogeneous resources.