Spark Load Balancing Strategy Optimization Based on Internet of Things

Abstract

The data collected by the Internet of Things (IOT) technology is becoming larger and larger, and the traditional data processing methods have encountered tremendous challenges. Spark, as a memory-based distributed computing framework, provides support for the data processing of the IOT. Load balancing is an important indicator to measure the performance of Spark computing. The load balancing strategy of Spark cluster only takes into account the locality of data, and neglects the computing capability and resource utilization of each node, which is prone to load unbalance and affecting the IOT data processing efficiency. Aiming at this issue, this paper optimizes and improves the current load balancing strategy of Spark based on the computing performance of each node in the Spark cluster, and proposes a task execution node assignment algorithm based on genetic algorithm and particle swarm optimization (TENAA). Experiments show that, compared with the Spark load balancing strategy, the load balancing strategy proposed in this paper has a significant increase both in load deviation and task completion time.