Small files storing and computing optimization in Hadoop parallel rendering

Abstract

The Hadoop framework has been widely used in the animation industry to build a large scale, high performance parallel render system. However, Hadoop Distributed File System (HDFS) and MapReduce programming model are designed to manage large files and suffer performance penalty while rendering and storing small RIB files in rendering system. Therefore, method that merging small RIB files based on two intelligent algorithms is proposed to solve the problem. The method uses Particle Swarm Optimization (PSO) and Support Vector Machine (SVM) to choose the optimal merge value for any scene file, by mainly considering the rendering time, memory limitation and other indicators. Then, the method takes advantage of frame-to-frame coherence to merge RIB files at an interval way with the optimal merge value. Finally, the proposed method is compared with the naive method under three different render scenes. Experimental results show that the proposed method significantly reduces the number of RIB files and render tasks, and improves the storage efficiency and computing efficiency of RIB Files.