Object Storage System Using Replication and Erasure Codes (OSSREC)

Abstract

Cloud Storage has been designed for storing large file that could occupy several HDDs (Hard Disk Drive). Most Cloud Storage Systems use Block Storage, which is efficient for large file, however, wasteful for small and medium files that are generally less than the size of a typical block (64/128 Mb). Research shows that most of the files created for personal usage is most of the time less than block size. As such for a Network Attached Storage (NAS), it is imperative to adopt another approach than Block Storage. Object Storage is the technique for storing of a group of files as one object and associated metadata stored independently. In this work, an Object Based NAS system of 12 TB is being implemented, and it has been developed using Hadoop 3.0. Given that Redundancy is important to guard against failure, the novelty of OSSREC is that is uses the default 3x Replication for individual files and coupled with Erasure Coding for Objects. A web interface is implemented to interact with Hadoop through the use of WebHDFS API and added to that, the functionality of Erasure Coding is also developed. The system is then evaluated in terms of Storage Capacity, Availability and Write/Read Performances. And finally, the Object Storage solution of OSSREC is benchmarked against the default Block Storage of Hadoop.