Deduplication Scheme Research Articles

Coupling Secret Sharing with Decentralized Server-Aided Encryption in Encrypted Deduplication

Outsourcing storage to the cloud can save storage costs and is commonly used in businesses. It should fulfill two major goals: storage efficiency and data confidentiality. Encrypted deduplication can achieve both goals via performing deduplication to eliminate the duplicate data within encrypted data. Traditional encrypted deduplication generates the encryption key on the client side, which poses a risk of offline brute-force cracking of the outsourced data. Server-aided encryption schemes have been proposed to strengthen the confidentiality of encrypted deduplication by distributing the encryption process to dedicated servers. Existing schemes rely on expensive cryptographic primitives to provide a decentralized setting on the dedicated servers for scalability. However, this incurs substantial performance slowdown and can not be applied in practical encrypted deduplication storage systems. In this paper, we propose a new decentralized server-aided encrypted deduplication approach for outsourced storage, called ECDedup, which leverages secret sharing to achieve secure and efficient key management. We are the first to use the coding matrix as the encryption key to couple the encryption and encoding processes in encrypted deduplication. We also propose a acceleration scheme to speed up the encryption process of our ECDedup. We prototype ECDedup in cloud environments, and our experimental results based on the real-world backup datasets show that ECDedup can improve the client throughput by up to 51.9% compared to the state-of-the-art encrypted deduplication schemes.

A Systematic Method to Detect Next-Generation Sequencing-Based Microsatellite Instability in Plasma Cell-Free DNA: plasmaMSI.

Microsatellite instability (MSI) detection using tumor tissue is a well-established prognostic and predictive biomarker for certain types of cancers. However, tumor tissue samples are less convenient to obtain than blood plasma samples. The main challenge facing next-generation sequencing-based MSI detection in blood plasma samples is the ultralow signal/noise ratio in plasma cell-free DNA (cfDNA). To address the challenge, plasmaMSI, a highly accurate cfDNA MSI detection method, is introduced with three novel performance-improving features: i) a set of stringent locus selection criteria to select loci with high robustness and compatibility across sequencing platforms; ii) a new deduplication strategy that greatly improves the signal/noise ratio for MSI detection; and iii) an MSI calling algorithm that customizes the baseline for each test sample based on its duplication rate. Through analytical validation in diluted cell line samples, the limit of detection of plasmaMSI was determined to be 0.15%. Furthermore, in analyzing 95 evaluable cfDNA samples from patients with gastrointestinal cancers, plasmaMSI exhibited a positive percentage agreement of 92.9% (39/42) and a negative percentage agreement of 100% (53/53) with tissue MSI-PCR. plasmaMSI provides novel solutions to key challenges in cfDNA MSI detection that have not been addressed by existing methods. It has also been systematically validated and is already used in clinical testing for patients with cancer.

A Verifiable and Secure Industrial IoT Data Deduplication Scheme with Real-time Data Integrity Checking in Fog-Assisted Cloud Environments

A Verifiable and Secure Industrial IoT Data Deduplication Scheme with Real-time Data Integrity Checking in Fog-Assisted Cloud Environments

Data Deduplication Based on Content Locality of Transactions to Enhance Blockchain Scalability

Blockchain is a promising infrastructure for the internet and digital economy, but it has serious scalability problems, that is, long block synchronization time and high storage cost. Conventional coarse-grained data deduplication schemes (block or file level) are proved to be ineffective on improving the scalability of blockchains. Based on comprehensive analysis on typical blockchain workloads, we propose two new locality concepts (economic and argument locality) and a novel fine-grained data deduplication scheme (transaction level) named Alias-Chain. Specifically, Alias-Chain replaces frequently used data, for example, smart contract arguments, with much shorter aliases to reduce the block sizes, which results in both shorter synchronization time and lower storage cost. Furthermore, to solve the potential consistency issue in Alias-Chain, we propose two complementary techniques: one is generating aliases from history blocks with high consistency, and the other is speeding up the generation of aliases via a specific algorithm. Our simulation results show: (1) the average transfer and SC-call transaction (a transaction used to call the smart contracts in the blockchain) sizes can be significantly reduced by up to 11.03% and 79.44% in native Ethereum, and up to 39.29% and 81.84% in Ethereum optimized by state-of-the-art techniques; and (2) the two complementary techniques well address the inconsistency risk with very limited impact on the benefit of Alias-Chain. Prototyping-based experiments are further conducted on a testbed consisting of up to 3200 miners. The results demonstrate the effectiveness and efficiency of Alias-Chain on reducing block synchronization time and storage cost under typical real-world workloads.

P²SimiDedup: Privacy-Preserving and Similarity-Based Deduplication Scheme for Fog-Assisted Vehicular Crowdsensing System

P²SimiDedup: Privacy-Preserving and Similarity-Based Deduplication Scheme for Fog-Assisted Vehicular Crowdsensing System

Improving Cloud Database Performance Using Storage Technologies

Cloud databases have become indispensable for modern applications due to their scalability, flexibility, and cost-effectiveness. However, optimizing performance in cloud environments can be challenging due to factors such as network latency, data distribution, and resource contention. This paper explores various storage technologies and techniques that can significantly enhance cloud database performance. We delve into the characteristics of different storage types, including solid-state drives (SSDs), hard disk drives (HDDs), and object storage, and discuss their suitability for various workloads. Additionally, we examine caching mechanisms, data compression, and deduplication strategies that can optimize data access and reduce storage costs. Furthermore, we explore the impact of database architecture choices, such as distributed databases and inmemory databases, on performance. We also discuss the importance of effective query optimization and indexing techniques. By understanding and implementing these storage technologies and techniques, organizations can achieve substantial improvements in cloud database performance, leading to better user experiences and increased application efficiency

A Study the review of Duplicate Data in Cloud Computing

In the realm of cloud computing, the management of data redundancy, often referred to as duplicate data, poses a significant challenge due to its implications on storage efficiency, data integrity, and overall system performance. This study provides a comprehensive review of the current strategies employed to detect, manage, and eliminate duplicate data in cloud environments. Duplicate and nearly duplicate web pages cause major issues for web search engines. Duplicate document identification, or the process of finding document pairs that represent the same entity, is a basic component of data cleansing. In order to address the structural heterogeneity issue, data must first undergo a data preparation step that involves parsing, data transformation, and data standardization. The findings aim to aid cloud service providers and users in understanding the importance of effective data deduplication strategies to optimize storage resources and enhance cloud computing performance.

Design and Implementation of Deduplication on F2FS

Data deduplication technology has gained popularity in modern file systems due to its ability to eliminate redundant writes and improve storage space efficiency. In recent years, the flash-friendly file system (F2FS) has been widely adopted in flash memory-based storage devices, including smartphones, fast-speed servers, and Internet of Things. In this article, we propose F2DFS (deduplication-based F2FS), which introduces three main design contributions. First, F2DFS integrates inline and offline hybrid deduplication. Inline deduplication eliminates redundant writes and enhances flash device endurance, while offline deduplication mitigates the negative I/O performance impact and saves more storage space. Second, F2DFS follows the file system coupling design principle, effectively leveraging the potentials and benefits of both deduplication and native F2FS. Also, with the aid of this principle, F2DFS achieves high-performance and space-efficient incremental deduplication. Third, F2DFS adopts virtual indexing to mitigate deduplication-induced many-to-one mapping updates during the segment cleaning. We conducted comprehensive experimental comparisons between F2DFS, native F2FS, and other state-of-the-art deduplication schemes, using both synthetic and real-world workloads. For inline deduplication, F2DFS outperforms SmartDedup, Dmdedup, and ZFS, in terms of both I/O bandwidth performance and deduplication rates. And for offline deduplication, compared to SmartDedup, XFS, and BtrFS, F2DFS shows higher execution efficiency, lower resource usage, and greater storage space savings. Moreover, F2DFS demonstrates more efficient segment cleanings than native F2FS.

FSDedup: Feature-Aware and Selective Deduplication for Improving Performance of Encrypted Non-Volatile Main Memory

Enhancing the endurance, performance, and energy efficiency of encrypted Non-Volatile Main Memory (NVMM) can be achieved by minimizing written data through inline deduplication. However, existing approaches applying inline deduplication to encrypted NVMM suffer from substantial performance degradation due to high computing, memory footprint, and index-lookup overhead to generate, store, and query the cryptographic hash (fingerprint). In the preliminary ESD [ 14 ], we proposed the Error Correcting Code (ECC) assisted selective deduplication scheme, utilizing the ECC information as a fingerprint to identify similar data effectively and then leveraging the selective deduplication technique to eliminate a large amount of redundant data with high reference counts. In this article, we proposed FSDedup. Compared with ESD, FSDedup could leverage the prefetch cache to reduce the read overhead during similarity comparison and utilize the cache refresh mechanism to identify further and eliminate more redundant data. Extensive experimental evaluations demonstrate that FSDedup can enhance the performance of the NVMM system further than the ESD. Experimental results show that FSDedup can improve both write and read speed by up to 1.8×, enhance Instructions Per Cycle by up to 1.5×, and reduce energy consumption by up to 2.0×, compared to ESD.

Deduplication on Encrypted Data in Cloud Computing

Cloud storage is a crucial component of cloud computing, allowing users to expand their storage without upgrading their equipment and overcome resource constraints. Cloud users' data is always encrypted before being outsourced to ensure their security and privacy. However, encrypted data may result in a significant waste of cloud resources. Storage complicates data sharing among authorized users. We continue to face issues with encrypted data storage and deduplication. Traditional deduplication strategies are designed for certain application settings where data owners or cloud servers have full control over the process. They cannot meet data owners' varying requests based on data sensitivity. This study proposes a flexible data storage management method that combines deduplication and access control across various Clouds. Service Providers (CSP). We assess its performance through security analyses, comparisons, and implementations. The results demonstrate security, effectiveness, and efficiency for actual applications.

An encrypted deduplication scheme based on files diversity

An encrypted deduplication scheme based on files diversity

A blockchain-based auditable deduplication scheme for multi-cloud storage

A blockchain-based auditable deduplication scheme for multi-cloud storage

A secure and lightweight cloud data deduplication scheme with efficient access control and key management

Data deduplication technology is extensively employed to enhance the storage efficiency of cloud servers by eliminating redundant files. Cloud users commonly encrypt their data prior to uploading it to the server. Conventional encryption algorithms, however, lead to the encryption of duplicated data from different users into distinct ciphertexts. Consequently, these ciphertexts must be stored in the cloud since the cloud server cannot identify such duplicated data. In this paper, we introduce a hybrid cloud-based secure deduplication scheme tailored for implementation on large-scale data systems. Specifically, our approach leverages ciphertext-policy attribute-based encryption (CP-ABE), which enables us to establish access control and key management via a private cloud server. Simultaneously, we leverage a public cloud server to cater to enterprises and groups seeking secure data storage. Notably, our approach ensures mutual zero-interaction verification between both public and private cloud servers through ElGamal encryption, thereby guaranteeing data unforgeability. The security assessment illustrates that our proposed approach ensures both data privacy and integrity. We also show that the approach resists brute-force attacks on the dictionary, prevents malicious users from deceiving cloud servers to return incorrect ciphertext, and achieves secure and efficient access control and key management. Furthermore, functional and performance evaluation underscores the superiority of our method over five other classical data deduplication schemes. Under the premise of having more comprehensive security settings, the performance of the scheme still maintains a good level at every stage.

Scalable and Popularity-Based Secure Deduplication Schemes With Fully Random Tags

Scalable and Popularity-Based Secure Deduplication Schemes With Fully Random Tags

The Design of a Lossless Deduplication Scheme to Eliminate Fine-Grained Redundancy for JPEG Image Storage Systems

The Design of a Lossless Deduplication Scheme to Eliminate Fine-Grained Redundancy for JPEG Image Storage Systems

Meningkatkan Deduplikasi Data melalui Kesamaan Teks dalam Pembelajaran Mesin: Pendekatan Komprehensif

The issue of dirty data, particularly duplicate data, is a common problem in data management that can affect data quality, operational efficiency, and decision-making. This study highlights the importance of implementing sustainable deduplication strategies as a key step in managing dirty data. We explore solutions for detecting duplicate data by measuring text similarity indices. In this study, the authors utilize a literature review research method. Through this method, we collected various journals on data deduplication and text similarity techniques, comparing several methods to identify the most effective approach. The data deduplication process in this study consists of two stages: 1) Matching - calculating the similarity value of a record with previous records, and 2) Clustering - grouping all records deemed duplicates of a single entity. Furthermore, this study extends to the development of a Python application capable of identifying and grouping similar customer data based on text similarity values. The Text Similarity measurement method uses the Jaro-Winkler Similarity technique. Experimental results and evaluations show that the Text Similarity approach is effective in identifying duplicate data with a high degree of accuracy. This study emphasizes the importance of sustainable deduplication, where the deduplication process is conducted periodically and continuously to ensure optimal data quality.

A randomized encryption deduplication method against frequency attack

Encryption deduplication technology first encrypts the file or chunks of data, then detects and removes duplicate ciphertexts, storing only one copy on the cloud storage server (CSP). Although traditional encryption deduplication schemes have the function of ciphertext deduplication, they also have a serious problem. Since the key generation algorithm and the encryption algorithm are deterministic, the frequency of plaintext chunking will be leaked. It is possible for the attacker to infer the information of the plaintext. We propose a Randomized Encryption deduplication method against Frequency Attack (REFA) that provides high randomness in ciphertexts. The core idea of REFA is that after obtaining a convergence key generated with the aid of a key server, the user randomizes the convergence key with a random value of the same length. Then, REFA encrypts the plaintext chunk with a randomized key. Our scheme hides the random value in the ciphertext and encrypts the convergence key for user ownership verification. REFA perfectly masks the frequency of plaintext chunks and has high storage efficiency and data security. Furthermore, the CSP performs ciphertext updates with ciphertexts uploaded by subsequent users. REFA can effectively protect against frequency analysis attacks.

Blockchain-Based Secure and Verifiable Deduplication Scheme for Cloud-Assisted Internet of Things

Blockchain-Based Secure and Verifiable Deduplication Scheme for Cloud-Assisted Internet of Things

An End-to-End High-Performance Deduplication Scheme for Docker Registries and Docker Container Storage Systems

The wide adoption of Docker containers for supporting agile and elastic enterprise applications has led to a broad proliferation of container images. The associated storage performance and capacity requirements place a high pressure on the infrastructure of container registries that store and distribute images and container storage systems on the Docker client side that manage image layers and store ephemeral data generated at container runtime. The storage demand is worsened by the large amount of duplicate data in images. Moreover, container storage systems that use Copy-on-Write (CoW) file systems as storage drivers exacerbate the redundancy. Exploiting the high file redundancy in real-world images is a promising approach to drastically reduce the growing storage requirements of container registries and improve the space efficiency of container storage systems. However, existing deduplication techniques significantly degrade the performance of both registries and container storage systems because of data reconstruction overhead as well as the deduplication cost. We propose DupHunter, an end-to-end deduplication scheme that deduplicates layers for both Docker registries and container storage systems while maintaining a high image distribution speed and container I/O performance. DupHunter is divided into 3 tiers: registry tier, middle tier, and client tier. Specifically, we first build a high-performance deduplication engine at the registry tier that not only natively deduplicates layers for space savings but also reduces layer restore overhead. Then, we use deduplication offloading at the middle tier to eliminate the redundant files from the client tier and avoid bringing deduplication overhead to the clients. To further reduce the data duplicates caused by CoWs and improve the container I/O performance, we utilize a container-aware storage system at the client tier that reserves space for each container and arranges the placement of files and their modifications on the disk to preserve locality. Under real workloads, DupHunter reduces storage space by up to 6.9 × and reduces the GET layer latency up to 2.8 × compared to the state-of-the-art. Moreover, DupHunter can improve the container I/O performance by up to 93% for reads and 64% for writes.

Secure Cloud-Based Media Sharing: Scalable Access Control and Privacy-Preserving Deduplication

This research delves into the realm of secure deduplication schemes, strategically designed to enhance storage efficiency within cloud environments. An initial focus is placed on an advanced AES encryption scheme utilizing a message-derived key, resulting in a consistent mapping of identical plaintexts to ciphertexts. The proposed AES framework not only incorporates convergent encryption but also provides meticulous security definitions. While the landscape of cloud computing boasts numerous techniques for data security, existing methodologies fall short in addressing nuanced aspects related to ciphertext. In response to this gap, our study introduces a pioneering information management paradigm. This comprehensive framework encompasses data gathering, sharing, and restrictive distribution, with a particular emphasis on multi-owner privacy preservation within the cloud. Under this innovative framework, data owners gain the capability to securely disseminate private information to predefined groups of clients through the cloud infrastructure. The research presented herein serves as a significant contribution to the ongoing discourse on secure deduplication in cloud storage. By combining advanced encryption techniques with a forward-looking information sharing strategy, our approach seeks to elevate the effectiveness of data security protocols in cloud environments