Inline Deduplication Research Articles

FSDedup: Feature-Aware and Selective Deduplication for Improving Performance of Encrypted Non-Volatile Main Memory

Enhancing the endurance, performance, and energy efficiency of encrypted Non-Volatile Main Memory (NVMM) can be achieved by minimizing written data through inline deduplication. However, existing approaches applying inline deduplication to encrypted NVMM suffer from substantial performance degradation due to high computing, memory footprint, and index-lookup overhead to generate, store, and query the cryptographic hash (fingerprint). In the preliminary ESD [ 14 ], we proposed the Error Correcting Code (ECC) assisted selective deduplication scheme, utilizing the ECC information as a fingerprint to identify similar data effectively and then leveraging the selective deduplication technique to eliminate a large amount of redundant data with high reference counts. In this article, we proposed FSDedup. Compared with ESD, FSDedup could leverage the prefetch cache to reduce the read overhead during similarity comparison and utilize the cache refresh mechanism to identify further and eliminate more redundant data. Extensive experimental evaluations demonstrate that FSDedup can enhance the performance of the NVMM system further than the ESD. Experimental results show that FSDedup can improve both write and read speed by up to 1.8×, enhance Instructions Per Cycle by up to 1.5×, and reduce energy consumption by up to 2.0×, compared to ESD.

Design and Implementation of Deduplication on F2FS

Data deduplication technology has gained popularity in modern file systems due to its ability to eliminate redundant writes and improve storage space efficiency. In recent years, the flash-friendly file system (F2FS) has been widely adopted in flash memory-based storage devices, including smartphones, fast-speed servers, and Internet of Things. In this article, we propose F2DFS (deduplication-based F2FS), which introduces three main design contributions. First, F2DFS integrates inline and offline hybrid deduplication. Inline deduplication eliminates redundant writes and enhances flash device endurance, while offline deduplication mitigates the negative I/O performance impact and saves more storage space. Second, F2DFS follows the file system coupling design principle, effectively leveraging the potentials and benefits of both deduplication and native F2FS. Also, with the aid of this principle, F2DFS achieves high-performance and space-efficient incremental deduplication. Third, F2DFS adopts virtual indexing to mitigate deduplication-induced many-to-one mapping updates during the segment cleaning. We conducted comprehensive experimental comparisons between F2DFS, native F2FS, and other state-of-the-art deduplication schemes, using both synthetic and real-world workloads. For inline deduplication, F2DFS outperforms SmartDedup, Dmdedup, and ZFS, in terms of both I/O bandwidth performance and deduplication rates. And for offline deduplication, compared to SmartDedup, XFS, and BtrFS, F2DFS shows higher execution efficiency, lower resource usage, and greater storage space savings. Moreover, F2DFS demonstrates more efficient segment cleanings than native F2FS.

From SSDs Back to HDDs: Optimizing VDO to Support Inline Deduplication and Compression for HDDs as Primary Storage Media

Deduplication and compression are powerful techniques to reduce the ratio between the quantity of logical data stored and the physical amount of consumed storage. Deduplication can impose significant performance overheads, as duplicate detection for large systems induces random accesses to the backend storage. These random accesses have led to the concern that deduplication for primary storage and HDDs are not compatible. Most inline data reduction solutions are therefore optimized for SSDs and discourage their use for HDDs, even for sequential workloads. In this work, we show that these concerns are valid if and only if the lessons learned from deduplication research are not applied. We have therefore investigated data reduction solutions for primary storage based on the RedHat Virtual Disk Optimizer (VDO) and show that directly applying them can decrease sequential write performance for HDDs by 36×. We then show that slight modifications to VDO plus the integration of a very small SSD area significantly improve performance even beyond the performance without data reduction enabled, making HDDs more cost-efficient for a wide range of mostly sequential cloud workloads than SSDs. Additionally, these VDO optimizations do not require to maintain different code bases for HDDs and SSDs, and we therefore provide the first data reduction solution applicable to both storage media.

Space-efficient and high-performance inline deduplication for emerging hybrid storage system with Libra＋

In the face of the ever-increasing data volume of data-intensive applications, deduplication, as a promising solution to reduce space consumption, has attracted increasing attention in modern storage systems. Specifically, in modern SSD caching-based hybrid storage systems, inline deduplication achieves data reduction on the fly before storing data in the SSDs, thereby enhancing performance and extending their lifespan. However, with the emergence of new storage media featuring fast access speed and high endurance, hybrid storage systems have evolved by incorporating these new storage devices in place of SSDs at the front end of the primary storage. In this work, we revisit the concept of inline deduplication in the context of emerging hybrid storage and propose a new inline deduplication system designed for emerging hybrid storage, named Libra＋. Libra＋ contains three components: First, we propose two cooperative index structures for memory and cache. Second, we design a selective deduplication approach to perform targeted deduplication. Third, we introduce a lightweight dirty list that replaces the conventional dirty list in inline deduplication, reducing time overhead and space costs. Experimental results show that Libra＋ can achieve simultaneous performance improvement and space cost reduction.

ObjDedup: High-Throughput Object Storage Layer for Backup Systems With Block-Level Deduplication

The immense popularity of object storage is also affecting the market of backup. Not only have novel backup solutions emerged that utilize cloud-based object storage as backends, but also support for object storage interfaces is increasingly expected from traditional dedicated backup appliances. This latter trend especially concerns systems with data deduplication, as they can offer compelling gains in storage capacity and throughput. However, such systems have been designed for interfaces and workloads that are markedly different from those encountered in object storage. Notably, they expect data to be written in portions that are orders of magnitude longer than those in the novel object-storage-oriented backup applications. In this light, we contribute twofold. First, contrasting the properties of object storage interfaces with usage patterns from 686 commercial deployments of backup appliances, we identify specific issues an implementation of such an interface has to address to offer adequate performance in a backup system with block-level deduplication. In particular, we show that a major challenge is efficient metadata management. Second, we present distributed data structures and algorithms to handle object metadata in backup systems with block-level deduplication. Subsequently, we implement them as an object storage layer for our HYDRAstor backup system. In comparison to object storage without in-line deduplication, our solution achieves 1.8–3.93x higher write throughput. Compared to object storage on top of a state-of-the-art file-based backup system, it processes 5.26–11.34x more object put operations per time unit.

InDe: An Inline Data Deduplication Approach via Adaptive Detection of Valid Container Utilization

Inline deduplication removes redundant data in real-time as data is being sent to the storage system. However, it causes data fragmentation: logically consecutive chunks are physically scattered across various containers after data deduplication. Many rewrite algorithms aim to alleviate the performance degradation due to fragmentation by rewriting fragmented duplicate chunks as unique chunks into new containers. Unfortunately, these algorithms determine whether a chunk is fragmented based on a simple pre-set fixed value, ignoring the variance of data characteristics between data segments. Accordingly, when backups are restored, they often fail to select an appropriate set of old containers for rewrite, generating a substantial number of invalid chunks in retrieved containers. To address this issue, we propose an inline deduplication approach for storage systems, called InDe , which uses a greedy algorithm to detect valid container utilization and dynamically adjusts the number of old container references in each segment. InDe fully leverages the distribution of duplicated chunks to improve the restore performance while maintaining high backup performance. We define an effectiveness metric, valid container referenced counts (VCRC) , to identify appropriate containers for the rewrite. We design a rewrite algorithm F-greedy that detects valid container utilization to rewrite low-VCRC containers. According to the VCRC distribution of containers, F-greedy dynamically adjusts the number of old container references to only share duplicate chunks with high-utilization containers for each segment, thereby improving the restore speed. To take full advantage of the above features, we further propose another rewrite algorithm called F-greedy+ based on adaptive interval detection of valid container utilization. F-greedy+ makes a more accurate estimation of the valid utilization of old containers by detecting trends of VCRC’s change in two directions and selecting referenced containers in the global scope. We quantitatively evaluate InDe using three real-world backup workloads. The experimental results show that compared with two state-of-the-art algorithms (Capping and SMR), our scheme improves the restore speed by 1.3×–2.4× while achieving almost the same backup performance.

Lightweight fog‐centric auditing scheme to verify integrity of IoT healthcare data in the cloud environment

SummaryIoT healthcare applications have become very popular among medical societies. At regular intervals, the healthcare IoT devices collect the patient's health information and transfer it to cloud storage. Storing the health records in the cloud environment increases the vulnerability. Integrity verification performed at regular intervals ensures the safety and security of the healthcare data stored in the cloud. Existing schemes use RSA and BLS signatures to verify the authenticity. However, the size of the signature is too large in RSA and BLS schemes. Using large signatures for proving the authenticity increases the computation overhead. IoT devices have minimal resource capacity, hence performing heavily weighted integrity schemes will create an unbearable workload. Likewise, private auditing increases the communication overhead between IoT devices and cloud storage. To overcome these issues, a fog‐centric auditing scheme is proposed based on the Cramer–Shoup cryptosystem. The proposed scheme allows the fog nodes to perform integrity verification without compromising security. It reduces the communication overhead by 55%–60%. The size of the CS signature used in the scheme is far lesser than the existing signatures. Moreover, performing the inline deduplication on the fog layer improves the efficiency of cloud storage by 30%.

Inline Deduplication Checking in cloud environment

Inline deduplication is a technique that can reduce the amount of storage space used in a cloud environment. It can also improve restore and backup times. However, it is not always effective due to the various factors that affect the effectiveness of this process. For instance, the type of data that is stored and the method that is used to perform deduplication are all important factors that affect the effectiveness of this solution. The goal of this paper is to provide an overview of the various aspects of inline deduplication and its effectiveness in improving the performance and reducing the cost of storage in a cloud-based environment. We then conduct a case study to demonstrate the efficiency of this process in an Amazon Web Services instance. Experiments were able to demonstrate that in cloud environments, the level of deduplication that is achieved can improve performance and reduce the amount of storage space that is used. In case study, the use of inline deduplication on Amazon Web Services led to a 60% reduction in the storage space usage. After conducting a comprehensive analysis of the various aspects of inline deduplication, we concluded that it is a standard feature that can be used by cloud storage providers to reduce their costs and improve the performance of their customers.

Low‐overhead inline deduplication for persistent memory

AbstractPersistent memory (PM) can play an important role in storing the edge data of IoT devices, but their capacities are restrained by the embedded system, which only provides the limited host space and energy resource. Therefore, a deduplication system is required to improve the PM utilization and energy consumption. According to our observations, the current inline deduplication algorithms incur nonnegligible latency when applied to the PM file systems (PMFSs), mainly because (i) the low tolerance to the extra deduplication I/O path, (ii) the long fingerprint calculation, and (iii) the unnecessary duplication detection when writing content with low duplication ratio. To improve this issue, we propose LO‐Dedup, a novel and low‐overhead deduplication system for PM. LO‐Dedup adopts PM‐friendly design, both in fine‐grained data load/store and fast hash scheme to reduce overhead. Moreover, LO‐Dedup can adaptively sample duplication detection according to the recent contents. We implement a prototype of LO‐Dedup in PMFS, which is a well‐known PMFS. The experimental results show that the write performance only has a slight drop when writing data and saves up to 45% space in PMFS.

Crocus: Enabling Computing Resource Orchestration for Inline Cluster-Wide Deduplication on Scalable Storage Systems

Inline deduplication dramatically improves storage space utilization. However, it degrades I/O throughput due to compute-intensive deduplication operations such as chunking, fingerprinting or hashing of chunk content, and redundant lookup I/Os over the network in the I/O path. In particular, the fingerprint or hash generation of content contributes largely to the degraded I/O throughput and is computationally expensive. In this article, we propose Crocus , a framework that enables compute resource orchestration to enhance cluster-wide deduplication performance. In particular, Crocus takes into account all compute resources such as local and remote {CPU, GPU} by managing decentralized compute pools. An opportunistic Load-Aware Fingerprint Scheduler (LAFS), distributes and offloads compute-intensive deduplication operations in a load-aware fashion to compute pools. Crocus is highly generic and can be adopted in both inline and offline deduplication with different storage tier configurations. We implemented Crocus in Ceph scale-out storage system. Our extensive evaluation shows that Crocus reduces the fingerprinting overhead by 86 percent with 4KB chunk size compared to Ceph with baseline deduplication while maintaining high disk-space savings. Our proposed LAFS scheduler, when tested in different internal and external contention scenarios also showed 54 percent improvement over a fixed or static scheduling approach.

A Content Fingerprint-Based Cluster-Wide Inline Deduplication for Shared-Nothing Storage Systems

Deduplication has been principally employed in distributed storage systems to improve storage space efficiency. Traditional deduplication research ignores the design specifications of shared-nothing distributed storage systems such as no central metadata bottleneck, scalability, and storage rebalancing. Likewise, inline deduplication integration poses serious threats to storage system read/write performance, consistency, and scalability. Mainly, this is due to ineffective and error-prone deduplication metadata, duplicate lookup I/O redirection, and placement of content fingerprints and data chunks. Further, transaction failures after deduplication integration often render inconsistencies in data chunks, deduplication metadata, and garbage data chunks. results in rendering inconsistencies in data chunks, deduplication metadata, and garbage data chunks. In this paper, we propose G rate , a high-performance inline cluster-wide data deduplication, complying with the design constraints of shared-nothing storage systems. In particular, G rate eliminates duplicate copies across the cluster for high storage space efficiency without jeopardizing performance. We employ a distributed deduplication metadata shard , which promises high-performance deduplication metadata and duplicate fingerprint lookup I/Os without introducing a single point of failure. The placement of data and deduplication metadata is made cluster-wide based on the content fingerprint of chunks. We decouple the deduplication metadata shard from read I/O path and replace it with a read manifestation object to further speedup read performance. To guarantee deduplication-enabled transaction consistency and efficient garbage identification, we design a flag-based asynchronous consistency scheme , capable of repairing the missing data chunks on duplicate arrival. We design and implement G rate in Ceph. The evaluation shows an average of 18% performance bandwidth improvement over the content addressable deduplication approach at smaller chunk sizes, i.e., less than 128KB while maintaining high storage space savings.

I can't believe it's not (only) software!

There is a steady increase in the size of data stored and processed as part of data science applications, leading to bottlenecks and inefficiencies at various layers of the stack. One way of reducing such bottlenecks and increasing energy efficiency is by tailoring the underlying distributed storage solution to the application domain, using resources more efficiently. We explore this idea in the context of a popular column-oriented storage format used in big data workloads, namely Apache Parquet. Our prototype uses an FPGA-based storage node that offers high bandwidth data deduplication and a companion software library that exposes an API for Parquet file access. This way the storage node remains general purpose and could be shared by applications from different domains, while, at the same time, benefiting from deduplication well suited to Apache Parquet files and from selective reads of columns in the file. In this demonstration we show, on the one hand, that by relying on the FPGA's dataflow processing model, it is possible to implement in-line deduplication without increasing latencies significantly or reducing throughput. On the other hand, we highlight the benefits of implementing the application-specific aspects in a software library instead of FPGA circuits and how this enables, for instance, regular data science frameworks running in Python to access the data on the storage node and to offload filtering operations.

NV-Dedup: High-Performance Inline Deduplication for Non-Volatile Memory

The byte-addressable non-volatile memory (NVM) is a promising medium for data storage. NVM-oriented file systems have been designed to explore NVM's performance potential. Meanwhile, applications may write considerable duplicate data. For NVM, a removal of duplicate data can promote space efficiency, improve write endurance, and potentially improve the performance by avoidance of repeatedly writing the same data. However, we have observed severe performance degradations when implementing a state-of-the-art inline deduplication algorithm in an NVM-oriented file system. A quantitative analysis reveals that, with NVM, 1) the conventional way to manage deduplication metadata for block devices, particularly in light of consistency, is inefficient, and, 2) the performance with deduplication becomes more subject to fingerprint calculations. We hence propose a deduplication algorithm called NV-Dedup . NV-Dedup manages deduplication metadata in a fine-grained, CPU and NVM-favored way, and preserves the metadata consistency with a lightweight transactional scheme. It also does workload-adaptive fingerprinting based on an analytical model and a transition scheme among fingerprinting methods to reduce calculation penalties. We have built a prototype of NV-Dedup in the Persistent Memory File System (PMFS). Experiments show that, NV-Dedup not only substantially saves NVM space, but also boosts the performance of PMFS by up to 2.1 $\times$ .

Concurrency control algorithms for deduplicated cloud storage

Deduplication of data is essential to effectively use cloud storage. As the metadata in deduplicated cloud storage are shared across multiple users, concurrent updates may result in inconsistencies. A coarse-grained locking strategy that has been proposed earlier to overcome this difficulty is not suited for inline deduplication owing to poor performance. In the present work, a fine-grained locking strategy that overcomes this shortcoming is proposed. A metadata structure along with a set of concurrent control mechanisms to accomplish this is presented. This strategy is shown to improve the throughput by as much as 60% with only marginal lock overhead.

A Scalable HW-Based Inline Deduplication for SSD Arrays

SSD arrays are becoming popular in modern storage servers as a primary storage, and they aim to reduce the high cost of the devices by performing inline deduplications. Unfortunately, existing software-based inline deduplications cannot achieve the devices’ maximum throughput due to their high CPU utilization and power overhead. A recently proposed approach to perform device-wide deduplications inside each SSD can distribute the CPU overhead among multiple SSDs, but it also suffers from severely decreasing deduplication opportunities with the increasing number of SSDs deployed per node. Therefore, we propose a node-wide deduplication engine that relies on specialized hardware to perform two key steps of deduplication; data signature generation and table management. Our FPGA-based prototype detects all duplicates, and compared to software-based inline deduplication, it reduces the overall CPU utilization and power consumption by 93.6 and $\sim$ 20 percent respectively for a slow baseline and more for faster baselines.

Fragmentation Study for Deduplication in cache Backup Storage

In backup environments field deduplication yields major advantages. Deduplication is process of automatic elimination of duplicate data in storage system and it is most effective technique to reduce storage costs. De duplication effects predictably in data fragmentation, because logically continuous data is spread across many disk locations. Fragmentation mainly caused by duplicates from previous backups of the same back upset, since such duplicates are frequent due to repeated full backups containing a lot of data which is not changed. Systems with in-line deduplicate intends to detects duplicates during writing and avoids storing them, such fragmentation causes data from the latest backup being scattered across older backups. This survey focused on various techniques to detect inline deduplication. As per literature, need to develop a focused on deduplication reduce the time and storage space. Proposed novel method to avoid the reduction in restores performance without reducing write performance and without affecting deduplication effectiveness.

Reducing Fragmentation for In-line Deduplication Backup Storage via Exploiting Backup History and Cache Knowledge

In backup systems, the chunks of each backup are physically scattered after deduplication, which causes a challenging fragmentation problem. We observe that the fragmentation comes into sparse and out-of-order containers. The sparse container decreases restore performance and garbage collection efficiency, while the out-of-order container decreases restore performance if the restore cache is small. In order to reduce the fragmentation, we propose History-Aware Rewriting algorithm (HAR) and Cache-Aware Filter (CAF). HAR exploits historical information in backup systems to accurately identify and reduce sparse containers, and CAF exploits restore cache knowledge to identify the out-of-order containers that hurt restore performance. CAF efficiently complements HAR in datasets where out-of-order containers are dominant. To reduce the metadata overhead of the garbage collection, we further propose a Container-Marker Algorithm (CMA) to identify valid containers instead of valid chunks. Our extensive experimental results from real-world datasets show HAR significantly improves the restore performance by 2.84-175.36 × at a cost of only rewriting 0.5-2.03 percent data.

Efficient Hybrid Inline and Out-of-Line Deduplication for Backup Storage

Backup storage systems often remove redundancy across backups via inline deduplication, which works by referring duplicate chunks of the latest backup to those of existing backups. However, inline deduplication degrades restore performance of the latest backup due to fragmentation, and complicates deletion of expired backups due to the sharing of data chunks. While out-of-line deduplication addresses the problems by forward-pointing existing duplicate chunks to those of the latest backup, it introduces additional I/Os of writing and removing duplicate chunks. We design and implement RevDedup , an efficient hybrid inline and out-of-line deduplication system for backup storage. It applies coarse-grained inline deduplication to remove duplicates of the latest backup, and then fine-grained out-of-line reverse deduplication to remove duplicates from older backups. Our reverse deduplication design limits the I/O overhead and prepares for efficient deletion of expired backups. Through extensive testbed experiments using synthetic and real-world datasets, we show that RevDedup can bring high performance to the backup, restore, and deletion operations, while maintaining high storage efficiency comparable to conventional inline deduplication.

Efficient Inline Deduplication on VM Images in Desktop Virtualization Environment

Enterprise service is transforming from traditional physical computing nodes to virtual machines which provides better isolation and more effective use of computing ability of the hardware. However, the widely deployment of virtual machines also increase the pressure of storage significantly with the fact that each must has at list one multi-gigabytes image file to store. To address the high pressure of storage from virtual machine images, we developed a user level inline deduplication file system with Content Addressable Storage (CAS). We use the open-source framework FUSE to encapsulate the deduplication process so as to achieve portability and flexibility. Compared to an ordinary file system without deduplication, we show that our file system can save at least 30% of space of single VM image and even more of multiple VM images while achieving considerable run time performance.

Scalable high performance de-duplication backup via hash join

Apart from high space efficiency, other demanding requirements for enterprise de-duplication backup are high performance, high scalability, and availability for large-scale distributed environments. The main challenge is reducing the significant disk input/output (I/O) overhead as a result of constantly accessing the disk to identify duplicate chunks. Existing inline de-duplication approaches mainly rely on duplicate locality to avoid disk bottleneck, thus suffering from degradation under poor duplicate locality workload. This paper presents Chunkfarm, a post-processing de-duplication backup system designed to improve capacity, throughput, and scalability for de-duplication. Chunkfarm performs de-duplication backup using the hash join algorithm, which turns the notoriously random and small disk I/Os of fingerprint lookups and updates into large sequential disk I/Os, hence achieving high write throughput not influenced by workload locality. More importantly, by decentralizing fingerprint lookup and update, Chunkfarm supports a cluster of servers to perform de-duplication backup in parallel; it hence is conducive to distributed implementation and thus applicable to large-scale and distributed storage systems.