Distributed machine learning based link allocation strategy

Abstract

In the field of machine learning, a machine learning system with multiple nodes is usually used, and each node is used to perform a machine learning distributed training process for a part of the data that is allocated to it and provide a server by performing the machine learning distributed training process. The obtained training result, its machine learning data needs to be transmitted through the network. This paper proposes a link allocation method for distributed machine learning. For machine learning computing nodes distributed across domains, due to inconsistencies in link distance, node performance, and link load, the traffic distribution between computing nodes is unbalanced. Aiming at the complex computing requirements of distributed machine learning, a link pre-allocation method is proposed, which establishes a central server-link-node topology map, integrates link resources, and determines the logical distance of nodes. For the synchronously distributed machine learning training set, preallocate transmission link resources and initiate transmission according to the remaining storage capacity of nodes. In order to improve the network utilization efficiency in the process of machine learning, it can break through the influence of large network transmission delay on the efficiency of distributed machine learning.