A Hybrid Parallelization Approach for Distributed and Scalable Deep Learning

Samson Akintoye,Haoming Chen,Prof Daoqiang Zhang,Prof Liangxiu Han,Dr Xin Zhang

doi:10.2139/ssrn.4043672

Samson Akintoye, Haoming Chen + Show 3 more

Open Access

https://doi.org/10.2139/ssrn.4043672

Copy DOI

Abstract

Recently, Deep Neural Networks (DNNs) have recorded great success in handling medical and other complex classification tasks. However, as the sizes of a DNN model and the available dataset increase, the training process becomes more complex and computationally intensive, which usually takes a longer time to complete. In this work, we have proposed a generic full end-to-end hybrid parallelization approach combining both model and data parallelism for efficiently distributed and scalable training of DNN models. We have also proposed a Genetic Algorithm based heuristic resources allocation mechanism (GABRA) for optimal distribution of partitions on the available GPUs for computing performance optimization. We have applied our proposed approach to a real use case based on 3D Residual Attention Deep Neural Network (3D-ResAttNet) for efficient Alzheimer Disease (AD) diagnosis on multiple GPUs. The experimental evaluation shows that the proposed approach is efficient and scalable, which achieves almost linear speedup with little or no differences in accuracy performance when compared with the existing non-parallel DNN models.

Highlights

In recent time, Deep Neural Networks (DNNs) have gained popularity as an important tool for solving complex tasks ranging from image classification [1], speech recognition [2], medical diagnosis [3, 4], to the recommendation systems [5] and complex games [7, 6]
The above-aforementioned approaches adopted data, model and pipeline parallelization separately or the combination of the methods to improve the performance of DNN models training
We conducted the experiments on 3D-ResAttNet model for two classification tasks: Stable mild cognitive impairment (MCI) (sMCI) vs. pMCI and Alzheimer’s Disease (AD) vs. Normal cohort (NC)

Summary

Introduction

Deep Neural Networks (DNNs) have gained popularity as an important tool for solving complex tasks ranging from image classification [1], speech recognition [2], medical diagnosis [3, 4], to the recommendation systems [5] and complex games [7, 6]. Training a DNN model requires a large volume of data, which is both data and computational intensive, leading to increased training time. To overcome this challenge, various parallel and distributed computing methods [8] have been proposed to scale up the DNN models to provide timely and efficient learning solutions. Various parallel and distributed computing methods [8] have been proposed to scale up the DNN models to provide timely and efficient learning solutions It can be divided into data parallelism, model parallelism, pipeline parallelism and hybrid parallelism (a combination of data and model parallelism).

Objectives

Methods

Results

Conclusion