PHDFS: Optimizing I/O performance of HDFS in deep learning cloud computing platform

Abstract

Abstract For deep learning cloud computing platforms, file system is a fundamental and critical component. Hadoop distributed file system (HDFS) is widely used in large scale clusters due to its high performance and high availability. However, in deep learning datasets, the number of files is huge but the file size is small, making HDFS suffer a severe performance penalty. Although there have been many optimizing methods for addressing the small file problem, none of them take the file correlation in deep learning datasets into consideration. To address such problem, this paper proposes a Pile-HDFS (PHDFS) based on a new file aggregation approach. Pile is designed as the I/O unit merging a group of small files according to their correlation. In order to effectively access small files, we design a two-layer manager and add the inner organization information to data blocks. Experimental results demonstrate that, compared with the original HDFS, PHDFS can dramatically decrease the latency when accessing small files and improve the FPS (Frames Per Second) of typical deep learning models by 40%.