Decentralized Parallel SGD Based on Weight-Balancing for Intelligent IoV

Abstract

Training machine learning models in a decentralized way has attracted tremendous attention on intelligent Internet of Vehicles (IIoV). However, it is highly dynamic and asymmetric for the connections between vehicles in IIoV due to the mobility of vehicles and the complex communication environment, which poses great challenges on designing efficient distributed learning algorithms. To address this problem, we focus on the basic stochastic gradient descent (SGD) algorithm and propose a decentralized parallel SGD algorithm (DPSGD-WB) for the complex IIoV. The algorithm is based on weight-balancing to overcome the difficulty caused by the dynamic and asymmetric connectivity in IIoV. With rigorous analysis, we show that DPSGD-WB converges on the optimal rate of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$O(1/\sqrt{Kn})$</tex-math> </inline-formula> , where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n$</tex-math> </inline-formula> is the number of vehicle terminals and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$K$</tex-math> </inline-formula> is the number of iterations. To the best of our knowledge, our proposed algorithm is the first known decentralized parallel SGD algorithm that can be implemented in asymmetric and dynamic intelligent IoV systems. Finally, extensive experiments demonstrate the efficacy of our algorithm.