EdgeMesh: A hybrid distributed training mechanism for heterogeneous edge devices

Abstract

AbstractThe proliferation of large‐scale distributed Internet of Things (IoT) applications has resulted in a surge in demand for network models such as deep neural networks (DNNs) to be trained and inferred at the edge. Due to the central data transmission mechanism, heterogeneity of edge devices, and resource constraints, the existing single data‐parallel, and model‐parallel distributed training mechanisms frequently fail to fully utilize the computing power of edge devices, network topology and bandwidth resources. In light of the shortcomings mentioned earlier, this article proposes EdgeMesh, a hybrid parallel training mode based on the Mesh‐Tensorflow framework, consisting of an adaptive meshing strategy and a dynamic model convolutional partitioning strategy. The computing power of IoT edge devices significantly speeds up the DNN training process. In a resource‐constrained environment, each node only supports a subset of the model's parallel computing tasks, reducing communication and memory overhead while retaining high scalability. Experiments show that when compared to single‐machine training and data parallel mode, EdgeMesh distributed training mechanism can reduce the average delay by 3.2 times and average memory overhead by 43% while maintaining model accuracy. The computing power of IoT edge devices effectively accelerates the DNN training process.