EDRFS: An Effective Distributed Replication File System for Small-File and Data-Intensive Application

Abstract

With the system scale keeping grown, the key challenge is to mask the failures that arise among the system components and to improve the performance of data-intensive applications. This paper designs and implements a cluster-based distributed replication file system EDRFS to meet these critical demands. EDRFS works with a single metadata server and multiple storage nodes, deploys whole-file replication scheme at the file level, and tracks what storage node a file is replicated on. We use a linear hash algorithm to evenly distribute data and load across multiple storage nodes so as to achieve balancing workload and incremental scalability of throughput and storage capacity as the system scale grows. In addition, we employ metadata caches and file data caches in clients to enhance system performance. Furthermore, we deploy a concurrency lock scheme to avoid namespace operation bottleneck and a replicas consistency method to keep a consistent mutation order among replicas of a file. We provide the initial experimental evaluations of our prototypical system on a small-file and data-intensive workload.