Performance optimization of computing task scheduling based on the Hadoop big data platform

Abstract

AbstractHadoop, a distributed computing framework that can efficiently process large-scale datasets, has been used by an increasing number of organizations as the basic computing framework to build cloud computing platforms. Improving its execution efficiency is a hot research direction in the industry, and the scheduling problem is a key factor affecting the execution efficiency of Hadoop. It is very important to identify its shortcomings and improve them. This paper examines and analyses the optimization of computing task scheduling performance based on the Hadoop big data platform. This paper first analyses Hadoop big data processing. Hadoop has high scalability. Computing nodes can be added at any time, and they can participate in cluster work through simple configuration. The paper discusses the improvement in the Hadoop resource scheduling algorithm. The task scheduling algorithm in the Hadoop-based data task localization proposed in this paper is compared with the default algorithm used in the Hadoop task scheduling algorithm. The former shows better local data in all four jobs, there are more data localization tasks, and the expected goal is achieved. The effectiveness of the algorithm is verified, and the performance is improved by 30%.