Abstract
Distributed processing using high-performance computing resources is essential for developers to train large-scale deep neural networks (DNNs). The major impediment to distributed DNN training is the communication bottleneck during the parameter exchange among the distributed DNN training workers. The communication bottleneck increases training time and decreases the utilization of the computational resources. Our previous study, SoftMemoryBox (SMB1) presented considerably superior performance compared to message passing interface (MPI) in the parameter communication of distributed DNN training. However, SMB1 had disadvantages such as the limited scalability of the distributed DNN training due to the restricted communication bandwidth from a single memory server, inability to provide a synchronization function for the shared memory buffer, and low portability/usability as a consequence of the kernel-level implementation. This paper proposes a scalable, shared memory buffer framework, called SoftMemoryBox II (SMB2), which overcomes the shortcomings of SMB1. With SMB2, distributed training processes can easily share virtually unified shared memory buffers composed of memory segments provided from remote memory servers and can exchange DNN parameters at high speed through the shared memory buffer. The scalable communication bandwidth of the SMB2 framework facilitates the reduction of DNN distributed training times compared to SMB1. According to intensive evaluation results, the communication bandwidth of the proposed SMB2 is 6.3 times greater than that of SMB1 when the SMB2 framework is scaled out to use eight memory servers. Moreover, the training time of SMB2-based asynchronous distributed training of five DNN models is up to 2.4 times faster than SMB1-based training.
Highlights
Deep learning is currently implemented in numerous application domains including face recognition, image classification, object detection, visual relationship detection, speech recognition, and security [1]–[6]
We proved the advantage of SMB1 by comparing the computation and communication time of different deep neural networks (DNNs) by emulating distributed deep learning parameter communication using SMB1 and message passing interface (MPI)
In this paper, we proposed a scalable, shared memory buffer framework called SMB2, which can be used as a replacement for the parameter server for asynchronous distributed DNN training
Summary
Deep learning is currently implemented in numerous application domains including face recognition, image classification, object detection, visual relationship detection, speech recognition, and security [1]–[6]. When using a traditional message communication protocol (e.g., message passing interface (MPI)) based on TCP/IP, the parameter communication bottleneck becomes severe, and the communication bottleneck increases the idle time of the computational resources such as GPUs and reduces the resource utilization To address this communication bottleneck issue in distributed DNN training, we proposed SoftMemoryBox (SMB1), a virtual shared memory framework in previous work, where we presented the concept, architecture, components, and basic application programming interface (API) of SMB1 [16]. Instead of a well-known synchronization function, SMB1 supports a restricted method for parameter update, cumulative API, which provides an exclusive accumulation function between two shared memory buffers with the assistance of the SMB server This requires allocating extra memory buffers to be used to stage the temporal data (e.g., weight difference to be updated).
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have