Joint Dynamic Grouping and Gradient Coding for Time-Critical Distributed Machine Learning in Heterogeneous Edge Networks

Abstract

In edge networks, distributed computing resources have been widely utilized to collaboratively perform a machine learning task by multiple nodes. However, the model training time in heterogeneous edge networks is becoming longer because of excessive computation and delay caused by slow nodes, namely, stragglers. The parameter server even abandons stragglers which fail to return the outcome within a reasonable deadline, called straggler dropout, decreasing the model accuracy. To optimize the computation cost and maintain the model accuracy, we focus on mitigating the heavy computation of stragglers and preventing straggler dropout. Therefore, we propose a novel scheme named dynamic grouping and heterogeneity-aware gradient coding (DGH-GC) to tolerate stragglers by employing dynamic grouping and gradient coding. DGH-GC evenly distributes stragglers in each group and encodes gradients based on their computation capacity to prevent them drop out. However, DGH-GC exacerbates the communication burden by making data duplication to tolerate stragglers. Relying on the scheme, we further propose an algorithm called DGH-(GC)2 to compress transferred gradients in both upstream communication and downstream communication. Experimental evaluations prove that DGH-(GC) outperforms all state-of-the-art methods and DGH-(GC)2 further speeds up the convergence time of the trained model and saves about 26% average iteration time compared to the DGH-(GC).