Load Balancing in Big Data Processing Systems

Abstract

The article examines the load balancing methods used in the construction of big data processing systems. In the light of the widespread transition to Industry 4.0, the authors analyze the major trends in the development of science and technology, describe the existing balancing methods and suggest their recommendations to improve their effective use. The analysis has revealed the key features of load balancing algorithms. The study examines load balancers in terms of the four layers of the TCP/IP network model – data link, network, transport and application. Furthermore, the authors refer to the features of load balancing algorithms and formalize the relevant requirements. The study primarily focuses on the load balancing at the Internet layer. The authors suggest formulas for calculating the response time for various load balancing algorithms. A new method of load balancing is proposed for big data processing systems. The article presents a typical network topology. The solution involves the integration of large data methods into a load balancing system. To implement the load distribution in the server cluster, the authors use a processing cluster analyzing the server machines and managing the distribution of the load in the network, based on the received data. In the end, the authors suggest the load balancing algorithm.