Dynamic Load Balancing Method for Urban Surveillance Video Big Data Storage Based on HDFS

Abstract

HDFS has been widely used by many video service websites, but its load balancing tool does not consider the bandwidth consumption characteristics of video file online playback and the heterogeneous performance difference of NameNode in metadata allocation problem. The dynamic load imbalance of cluster makes the utilization of bandwidth resources low. In this paper, a HDFS NameNode dynamic load balancing tool (NDLBT) for city monitoring video in urban surveillance video big data storage in cloud storage environment is proposed. method. Firstly, it analyzes the relationship between the bandwidth consumption and the bit rate, data block size and access heat of the video file when the video file is played online, and a new load evaluation model is established. On this basis, it adds consideration to the bandwidth consumption factor in the load scheme generation and load scheduling, and through the dynamic adaptive backup of multi-replica heterogeneous nodes of metadata. The dynamic distribution of metadata is realized under the consideration of node performance and load, and the performance of metadata server cluster is guaranteed. Finally, combined with cache strategy and automatic recovery mechanism, the reading and writing of metadata is improved. The simulation results show that compared with the proposed method, we can effectively avoid the aggregation of high bandwidth consumption data blocks. In the experimental scenario where high bandwidth consumption video files are used as service access hotspots, the proposed method is superior to the original load balancing method in 90% scenarios, and can reduce the bandwidth peak value of bottleneck nodes in data node clusters by 20%.