Abstract
Recently, Deep Neural Networks (DNNs) have recorded great success in handling medical and other complex classification tasks. However, as the sizes of a DNN model and the available dataset increase, the training process becomes more complex and computationally intensive, which usually takes a longer time to complete. In this work, we have proposed a generic full end-to-end hybrid parallelization approach combining both model and data parallelism for efficiently distributed and scalable training of DNN models. We have also proposed a Genetic Algorithm based heuristic resources allocation mechanism (GABRA) for optimal distribution of partitions on the available GPUs for computing performance optimization. We have applied our proposed approach to a real use case based on 3D Residual Attention Deep Neural Network (3D-ResAttNet) for efficient Alzheimer Disease (AD) diagnosis on multiple GPUs. The experimental evaluation shows that the proposed approach is efficient and scalable, which achieves almost linear speedup with little or no differences in accuracy performance when compared with the existing non-parallel DNN models.
Highlights
In recent time, Deep Neural Networks (DNNs) have gained popularity as an important tool for solving complex tasks ranging from image classification [1], speech recognition [2], medical diagnosis [3, 4], to the recommendation systems [5] and complex games [7, 6]
The above-aforementioned approaches adopted data, model and pipeline parallelization separately or the combination of the methods to improve the performance of DNN models training
We conducted the experiments on 3D-ResAttNet model for two classification tasks: Stable mild cognitive impairment (MCI) (sMCI) vs. pMCI and Alzheimer’s Disease (AD) vs. Normal cohort (NC)
Summary
Deep Neural Networks (DNNs) have gained popularity as an important tool for solving complex tasks ranging from image classification [1], speech recognition [2], medical diagnosis [3, 4], to the recommendation systems [5] and complex games [7, 6]. Training a DNN model requires a large volume of data, which is both data and computational intensive, leading to increased training time. To overcome this challenge, various parallel and distributed computing methods [8] have been proposed to scale up the DNN models to provide timely and efficient learning solutions. Various parallel and distributed computing methods [8] have been proposed to scale up the DNN models to provide timely and efficient learning solutions It can be divided into data parallelism, model parallelism, pipeline parallelism and hybrid parallelism (a combination of data and model parallelism).
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
