Distributed Training on a Highly Heterogeneous HPC System

Abstract

During the recent years HPC systems are being targeted as suitable systems to run DeepLearning workloads. In that respect, a number of machine learning libraries exist targeting different HPC computing platforms. In the context of the European DeepHealth project, the European Distributed Deep Learning library (EDDL) and the European Computing Vision library (ECVL) have been developed. These libraries target heterogeneous HPC systems including multi/many-core processors (CPUs), GPUs and FPGAs. In this paper we describe the approach followed within the project to exploit HPC resources in an efficient and transparent manner with special focus on FPGAs. The complete process is hidden from the end user perspective, allowing a simplification on the complexity to run DeepLearning workloads on heterogeneous systems.