Efficient Data Deduplication Research Articles

Provisioning an efficient data deduplication model for cloud storage and integrity

Provisioning an efficient data deduplication model for cloud storage and integrity

BTEDD: Block-level tokens for efficient data deduplication in public cloud infrastructures

BTEDD: Block-level tokens for efficient data deduplication in public cloud infrastructures

Secure Deduplication with User-Defined Access Control in Cloud Storage

Abstract: Cloud storage is one of the most important services of cloud computing, offering significant benefits to users by allowing them to outsource their data for storage and sharing with authorized parties. Secure deduplication has been extensively researched in cloud storage, aiming to eliminate redundancy in encrypted data and reduce storage space. Our proposed solution focuses on uploading files to the cloud storage system. When a file is uploaded, it is checked to see if an identical file already exists. If a duplicate is found, the system displays an entry indicating that the file already exists. This secure and efficient data deduplication scheme ensures that users can store their data securely in the cloud, with access limited to authorized parties. By eliminating redundancy, more space is available for uploading different files. To ensure data security, we utilize the MD5 algorithm for hashing purposes and AES for encryption. Our system offers a practical solution for organizations seeking to securely store their data in the cloud, while also preserving user privacy and data ownership

ANALYSIS OF THE EFFICIENCY OF DATA CHUNKING METHODS FOR DATA DEDUBLICATION SYSTEMS

There is a significant increase in the amount of data that needs to be stored worldwide. More and more companies are turning their attention to deduplication systems, which effectively increase data warehouse volume and reduce storage costs. Deduplication not only reduces the overall amount of information in storage but also reduces the load on networks by eliminating the need to retransmit duplicate data. In this work, we considered the stages that any deduplication system includes, namely chunking, hashing and indexing, mapping. The effectiveness of deduplication systems primarily depends on the choice of the method of dividing the data stream at the chunking stage. We considered the classic Two Threshold Two Divisor (TTTD) method, which is widely used in modern deduplication systems. This method uses Rabin’s fingerprint to find the hash of the substring value. The formula for calculating the hash for the first substring and the formula for calculating the rest of the substring are given. Another method we investigated is Content Based Two Threshold Two Divisor (CB-TTTD) – it uses new hash functions to fragment the data stream, and the corresponding formulas for calculating the first and each subsequent substring are given. To test the effectiveness of these two methods, we developed a test deduplication system, implemented these two fragmentation methods, and tested their performance on two sets of text data. We have modified these methods with the addition of a new string-splitting condition based on the content specification of the data we tested. The results of a comparison of the work of classical and modified methods are given. Using metrics to compare the efficiency of data fragmentation methods, we obtained experimental data, based on which we can make conclusions about the feasibility of using CB-TTTD as an alternative to TTTD in new deduplication systems. The obtained data can be used in the development of new highly efficient data deduplication systems and to improve old solutions

A secure and efficient data deduplication framework for the internet of things via edge computing and blockchain

Data deduplication can solve the problem of resource wastage caused by duplicated data. However, due to the limited resources of Internet of Things (IoT) devices, applying data deduplication to IoT scenarios is challenging. Existing data deduplication frameworks for the IoT are prone to inefficiency or trust crises due to the random allocation of edge computing nodes. Furthermore, side-channel attacks remain a risk. In addition, after IoT devices store data in the cloud through data deduplication, they cannot share their data efficiently. In this paper, we propose a secure and efficient data deduplication framework for the IoT based on edge computing and blockchain technologies. In this scheme, we propose a model based on parallel use of three-layer and two-layer architectures and introduce the RAndom REsponse (RARE) scheme to resist side-channel attacks. We also design a label tree to realise one-to-many data-sharing, which improves efficiency and meets the needs of the IoT. In addition, we use blockchain to resist collusion attacks. Experiments were conducted to demonstrate that our framework has advantages over similar schemes in terms of communication cost, security and efficiency.

Blockchain & Multi-Agent System: A New Promising Approach for Cloud Data Integrity Auditing with Deduplication

Recently, data storage represents one of the most important services in Cloud Computing. The cloud provider should ensure two major requirements which are data integrity and storage efficiency. Blockchain data structure and the efficient data deduplication represent possible solutions to address these exigencies. Several approaches have been proposed, some of them implement deduplication in Cloud server side, which involves a lot of computation to eliminate the redundant data and it becomes more and more complex. Therefore, this paper proposed an efficient, reliable and secure approach, in which the authors propose a Multi-Agent System in order to manipulate deduplication technique that permits to reduce data volumes thereby reduce storage overhead. On the other side, the loss of physical control over data introduces security challenges such as data loss, data tampering and data modification. To solve similar problems, the authors also propose Blockchain as a database for storing metadata of client files. This database serves as logging database that ensures data integrity auditing function.

Security-Aware and Efficient Data Deduplication for Edge-Assisted Cloud Storage Systems

Data deduplication at the network edge significantly improves communication efficiency in edge-assisted cloud storage systems. With the increasing concern about data privacy, secure deduplication has been proposed to provide data security while supporting deduplication. Since conventional secure deduplication schemes are mainly based on deterministic encryption, they are vulnerable to frequency analysis attacks. Some recent research has focused on this problem, where several works studied the trade-off between deduplication efficiency and resistance to frequency analysis attacks. However, no existing work can provide different deduplication efficiency and protection for different data chunks. In this paper, we propose a security-aware and efficient data deduplication scheme for edge-assisted cloud storage systems. It not only improves the efficiency of deduplication but also reduces information leakage caused by frequency analysis attacks. In particular, we first define the security level for chunks to measure the security needs of users. Then, we develop an encryption scheme with multiple levels of security for deduplication. It provides higher security protection for chunks with higher security levels, while sacrificing security to achieve higher deduplication efficiency for chunks with lower security levels. We also analyze the security of our proposed scheme. Evaluations on the real-world datasets show the efficiency of our design.

Secure and Efficient Data Deduplication in JointCloud Storage

Data deduplication can efficiently eliminate data redundancies in cloud storage and reduce the bandwidth requirement of users. However, most previous schemes depending on the help of a trusted key server (KS) are vulnerable and limited because they suffer from revealing information, poor resistance to attacks, great computational overhead, etc. In particular, if the trusted KS fails, the whole system stops working, i.e., single-point-of-failure. In this paper, we propose a Secure and Efficient data Deduplication scheme (named SED) in a JointCloud storage system which provides the global services via collaboration with various clouds. SED also supports dynamic data update and sharing without the help of the trusted KS. Moreover, SED can overcome the single-point-of-failure that commonly occurs in the classic cloud storage system. According to the theoretical analyses, our SED ensures the semantic security in the random oracle model and has strong anti-attack ability such as the brute-force attack resistance and the collusion attack resistance. Besides, SED can effectively eliminate data redundancies with low computational complexity and communication and storage overhead. The efficiency and functionality of SED improves the usability in client-side. Finally, the comparing results show that the performance of our scheme is superior to that of the existing schemes.

New proofs of ownership for efficient data deduplication in the adversarial conspiracy model

The primitive of proofs of ownership (PoWs) enables a prover to efficiently convince a verifier that he/she indeed owns a certain message in a knowledge-proof manner. As a result, it can prevent an adversary who only has a short information of the message from accessing the whole one. We argue that the existing PoWs based on Merkle hash tree and specific encodings are not much efficient if the size of message is sufficiently huge. In this paper, we first propose a new PoW protocol based on the chameleon hash function without key exposure. Interestingly, it is equivalent to having the prover compute a new collision of chameleon hashing as the proof in our construction. Therefore, the proposed protocol is much efficient since the computation and storage overhead of proof is independent of the size of the message. Moreover, we utilize the proposed PoWs to design a deduplication scheme over ciphertext.

Providing Task Allocation and Secure Deduplication for Mobile Crowdsensing via Fog Computing

Mobile crowdsensing enables a crowd of individuals to cooperatively collect data for special interest customers using their mobile devices. The success of mobile crowdsensing largely depends on the participating mobile users. The broader participation, the more sensing data are collected; nevertheless, the more replicate data may be generated, thereby bringing unnecessary heavy communication overhead. Hence it is critical to eliminate duplicate data to improve communication efficiency, a.k.a., data deduplication. Unfortunately, sensing data is usually protected, making its deduplication challenging. In this paper, we propose a fog-assisted mobile crowdsensing framework, enabling fog nodes to allocate tasks based on user mobility for improving the accuracy of task assignment. Further, a fog-assisted secure data deduplication scheme (Fo-SDD) is introduced to improve communication efficiency while guaranteeing data confidentiality. Specifically, a BLS-oblivious pseudo-random function is designed to enable fog nodes to detect and remove replicate data in sensing reports without exposing the content of reports. To protect the privacy of mobile users, we further extend the Fo-SDD to hide users’ identities during data collection. In doing so, Chameleon hash function is leveraged to achieve contribution claim and reward retrieval for anonymous mobile users. Finally, we demonstrate that both schemes achieve secure, efficient data deduplication.

Blockchain & Multi-Agent System: A New Promising Approach for Cloud Data Integrity Auditing with Deduplication

Recently, data storage represents one of the most important services in Cloud Computing. The cloud provider should ensure two major requirements which are data integrity and storage efficiency. Blockchain data structure and the efficient data deduplication represent possible solutions to address these exigencies. Several approaches have been proposed, some of them implement deduplication in Cloud server side, which involves a lot of computation to eliminate the redundant data and it becomes more and more complex. Therefore, this paper proposed an efficient, reliable and secure approach, in which the authors propose a Multi-Agent System in order to manipulate deduplication technique that permits to reduce data volumes thereby reduce storage overhead. On the other side, the loss of physical control over data introduces security challenges such as data loss, data tampering and data modification. To solve similar problems, the authors also propose Blockchain as a database for storing metadata of client files. This database serves as logging database that ensures data integrity auditing function.

Secure and Efficient Cloud Data Deduplication With Randomized Tag

Cross-client data deduplication has been widely used to eliminate redundant storage overhead in cloud storage system. Recently, Abadi et al. introduced the primitive of MLE2 with nice security properties for secure and efficient data deduplication. However, besides the computationally expensive non-interactive zero-knowledge proofs, their fully randomized scheme (R-MLE2) requires the inefficient equality-testing algorithm to identify all duplicate ciphertexts. Thus, an interesting challenging problem is how to reduce the overhead of R-MLE2 and propose an efficient construction for R-MLE2. In this paper, we introduce a new primitive called $\mu \text{R}$ -MLE2, which gives a partial positive answer for this challenging problem. We propose two schemes: static scheme and dynamic scheme, where the latter one allows tree adjustment by increasing some computation cost. Our main trick is to use the interactive protocol based on static or dynamic decision trees. The advantage gained from it is, by interacting with clients, the server will reduce the time complexity of deduplication equality test from linear time to efficient logarithmic time over the whole data items in the database. The security analysis and the performance evaluation show that our schemes are Path-PRV-CDA2 secure and achieve several orders of magnitude higher performance for data equality test than R-MLE2 scheme when the number of data items is relatively large.

Differentially private client‐side data deduplication protocol for cloud storage services

AbstractCloud storage service providers apply data client‐side deduplication across multiple users to achieve cost savings of network bandwidth and disk storage. However, deduplication can be used as a side channel by attackers who try to obtain sensitive information of other users' data. We propose a differentially private client‐side deduplication protocol. A storage gateway allows efficient data deduplication while reducing the risk of information leakage. Its security can be strongly guaranteed according to the definition of differential privacy. We evaluate the effectiveness and efficiency of the proposed protocol through experiments. Copyright © 2014 John Wiley & Sons, Ltd.

효율적인 데이터 중복제거를 위한 GPGPU 병렬 라빈 핑거프린팅

초록·키워드 목차 오류제보하기 등록된 정보가 없습니다. #Flexible substrate(유연 기판) #Ni-Co thin film(Ni-Co 박막) #Nano-indenter(나노인덴터) 등록된 정보가 없습니다.

Design and Implementation of Storage System Using Byte-index Chunking Scheme

In this paper, we present an enhanced storage system that supports Byte-index chunking algorithm. The storage system aims to provide efficient data deduplication with high performance and to be performed in rapid time. We describe the overall procedure of Byteindex chunking based storage system including read/write procedure and how the system works. The key idea of Byte-index chunking is to adapt fixed-size block chunk scheme which are distributed to “Index-table” by chunk’s both side boundary values. We have found that Byte-index chunking in storage system provides high performance compared with other chunking schemes. Experiments result shows that the storage system with Byte-index chunking compresses overall data with high deduplication capability and reduce the speed of file processing.