Application of iLauncher interfaces to the abstraction of complex deep learning workflows across a diversity of computing resources

Abstract

We present some of the latest advancements in the development of the Interface Launcher (iLauncher), along with the application of this technology to the development of plugins that support distributed PyTorch deep learning workflows across a diversity of computing resources including Amazon Web Services (AWS) GovCloud, distributed clusters of heterogeneous nodes with multiple graphics processing units (GPUs) per node running the Slurm batch queuing software, and Department of Defense (DoD) high performance computing (HPC) supercomputers running the Portable Batch Scheduling (PBS) software. The iLauncher technology automates the submission of HPC jobs and provides a mechanism for rapidly prototyping web interfaces from the user’s desktop to powerful capabilities running on the HPC nodes. We describe the extension of previous work to show the development of the client-side plugin JavaScript Object Notation (JSON) description, the underlying server-side scripts for running distributed PyTorch deep learning models on various platforms with different queuing systems, and the recipes for the software along with all dependencies in an all-inclusive software packaging technology called a container. Finally, we show a representative use case running distributed PyTorch in a Jupyter Notebook through iLauncher on the various backend platforms along with some guidance on when each one may be beneficial for a range of scenarios based on models and data.