Performance Evaluation of Merging Techniques for Handling Small Size Files in HDFS

Abstract

AbstractWhen dealing with the storage of large files, HDFS is one of the good choices as a distributed storage. Processing a large number of small files results in the performance bottleneck of HDFS. A massive number of small files will produce excessive metadata that leads to inefficient utilization of the Name Node memory, and frequent function calls will consume all over more time to process; therefore, it can be concluded that HDFS degrades when handling with small files. A detailed performance evaluation is being conducted to understand the impact of increasing small files in Hadoop for processing. This paper mainly evaluates sequential files, CombineFileInputFormat, HAR and Hadoop streaming techniques to deal with small file problem in HDFS. Empirical evaluation conducted in this paper shows that HAR and CombineFileInputFormat perform better and have consistent and stable results when increasing number of files for processing. KeywordsHadoopMapReduceHARHadoop streamingSequential fileCombineFileInputFormatSmall filesHDFS