Space-efficient and high-performance inline deduplication for emerging hybrid storage system with Libra＋

Abstract

In the face of the ever-increasing data volume of data-intensive applications, deduplication, as a promising solution to reduce space consumption, has attracted increasing attention in modern storage systems. Specifically, in modern SSD caching-based hybrid storage systems, inline deduplication achieves data reduction on the fly before storing data in the SSDs, thereby enhancing performance and extending their lifespan. However, with the emergence of new storage media featuring fast access speed and high endurance, hybrid storage systems have evolved by incorporating these new storage devices in place of SSDs at the front end of the primary storage. In this work, we revisit the concept of inline deduplication in the context of emerging hybrid storage and propose a new inline deduplication system designed for emerging hybrid storage, named Libra＋. Libra＋ contains three components: First, we propose two cooperative index structures for memory and cache. Second, we design a selective deduplication approach to perform targeted deduplication. Third, we introduce a lightweight dirty list that replaces the conventional dirty list in inline deduplication, reducing time overhead and space costs. Experimental results show that Libra＋ can achieve simultaneous performance improvement and space cost reduction.