AR-dedupe: An efficient deduplication approach for cluster deduplication system

Abstract

As data are growing rapidly in data centers, inline cluster deduplication technique has been widely used to improve storage efficiency and data reliability. However, there are some challenges faced by the cluster deduplication system: the decreasing data deduplication rate with the increasing deduplication server nodes, high communication overhead for data routing, and load balance to improve the throughput of the system. In this paper, we propose a well-performed cluster deduplication system called AR-Dedupe. The experimental results of two real datasets demonstrate that AR-Dedupe can achieve a high data deduplication rate with a low communication overhead and keep the system load balancing well at the same time through a new data routing algorithm. In addition, we utilize application-aware mechanism to speed up the index of handprints in the routing server which has a 30% performance improvement.