Interactive visual analytics of parallel training strategies for DNN models

Abstract

Understanding and optimizing the parallel training strategies in the training of large-scale Deep Neural Network (DNN) models is crucial to enhance training efficiency. Existing works tried to demonstrate the layer-level information of the computational graph to support parallel training strategy selection. Whereas, the overall parallel execution logic is rarely considered by previous methods. In this paper, we proposed a novel visual analytics approach for parallel training strategies, demonstrating the execution logic of the distributed computing from up to bottom via explaining communication operators. Specifically, a computation-communication bipartite construction algorithm is designed for the computational graph visualization. Furthermore, a system is developed to help users easily access the proposed approach and explore the parallel training strategies interactively. With empirical evaluation through a quantitative user study and a qualitative expert interview, the practicality and superiority of the proposed approach is verified.