A Novel Approach for Improving Security and Storage Efficiency on HDFS

Abstract

Abstract Distributed file system for the storage of massive files have obvious advantages compared with the conventional file system. For instance, Hadoop Distributed File System (HDFS) implemented with commodity hardware has the advantages of low cost, high fault tolerance, scalability, etc. However, HDFS has the potential safety hazard due to the unencrypted data stored in Datanode, which may cause data leakage during the manipulation. In this paper, we purpose a new architecture based on HDFS, combined with network coding and multi-node reading, to improve the security and storage efficiency of the distributed file system. Experiments have shown that the method proposed in this paper greatly improves the safety of files transfer and storage, while the speed of files reading has increased threefold compared to the original HDFS model.