Elastic HDFS: interconnected distributed architecture for availability–scalability enhancement of large-scale cloud storages

Abstract

This paper presents an interconnected distributed architecture for storing data and metadata in large-scale cloud storage systems. The primary goal of the proposed architecture is to enhance the scalability of namespace directory in large-scale file systems. Structural shift from distinguished distributed model to interconnected distributed model and conducting effective coordination among file servers for namespace management are two key solutions considered in the context of proposed architecture. To this intent, a coordination protocol is designed for communication among file servers, and maintaining user transparency in the presence of different file system actions/reactions. The experimental results, obtained via emulations under different network conditions and cloud storage sizes, show up to 43.9% availability and 37.8% connection throughput improvements with negligible storage overhead compared to the latest released version of Hadoop distributed file system.