Automating CNN Parallelism with Components

Abstract

Training Convolutional Neural Networks (CNNs) is a computationally intensive and time-consuming task. For this reason, distributing the training process of CNNs has become a crucial approach to decrease the training duration and effectively train CNN models in a reasonable time. Nevertheless, introducing parallelism to CNNs is a challenging task in practice. It is a tedious, repetitive and error-prone process. In this paper, we present Auto-CNNp, a novel framework to automate CNNs training parallelization task. To achieve this goal, the Auto-CNNp introduces a key component which is called CNN-Parallelism-Generator. The latter component encapsulates and hides typical CNNs parallelization routine tasks while being extensible for user-specific customization. Our proposed reference implementation provides a high level of abstraction over MPI-based CNNs parallelization process, despite the CNN-based imaging task and its related architecture and training dataset. The quantitative and qualitative assessment of our proposal on two medical imagining segmentation case studies show its (1) significant impact in facilitating the distributing of the CNNs training task and (2) its generalization for a wider scope of use cases.