SS-dedup: A high throughput stateful data routing algorithm for cluster deduplication system

Abstract

As data grows exponentially within data centers, cluster deduplication storage systems face challenges in providing high throughput, high deduplication ratio and load balance. As the key technique, data routing algorithm has a strong impact on the deduplication ratio, throughput and load balance in cluster deduplication storage systems. In this paper, we propose SS-Dedup, a novel stateful data routing algorithm for cluster deduplication storage system which can achieve higher system throughput and good load balance at the cost of deduplication ratio loss and memory space in client servers. SS-Dedup takes advantage of data similarity to increases system throughput with little deduplication ratio loss. Specifically, to decrease network traffic and response time, SS-Dedup maintains LRU caches in client servers to store fingerprints of historical routed chunks for each data server. Our experiment results show that while maintaining good load balance and high deduplication ratio, SS-Dedup takes up much lower network bandwidth and provides higher system throughput.