Deduplication System Research Articles

A Hybrid Cloud Approach for Secure Authorized Deduplication System Using Facial Authentication

Over the use of peer-to-peer encryption, businesses are able to establish private communication channels between devices or parts of devices, ensuring that no intermediary devices are able to decipher important data as it travels over the network. To safeguard sensitive information like credit card details, PIN numbers, and passwords, privacy is essential. Rather of being handed down orally, this kind of knowledge may be exchanged digitally using messaging programs such as WhatsApp or Facebook Messenger.Most of these apps use what is known as peer-to-peer encryption, a kind of two-way communication in which no one other than the two parties involved can decipher the communications transmitted and received. It is necessary to remember the master password in order to access the password safe, which stores all the passphrases in one account. Other malevolent bots and hackers are prompted to plot an assault and attempt to breach the system by this strategy.Therefore, we provide an encryption technique in our suggested work that relies on the authentic user's facial landmarks. Since it is very difficult, if not impossible, to replicate all of the face characteristics in order to fool the authentication system, this method is extremely strong. To determine the legitimate user's identity, we use the Localised Binary Pattern method in conjunction with the Gabor Wavelet on the locality preserving projection model. Both the legitimacy on the peer-to-peer network and the encryption of data communicated across the network channel are supported by the face structure that has been extracted. As a result, this forms a very strong component for the authentication and encryption of sent communications. Key Words: peer-to-peer, system, encryption, authentication, breach, transmitted

Convergent encryption enabled secure data deduplication algorithm for cloud environment

SummaryThe exponential growth of data poses a critical challenge for cloud storage systems. Redundant data consumes valuable storage space and increases infrastructure costs. Data deduplication, a technique for eliminating duplicate data copies, offers a promising solution. However, existing deduplication techniques often compromise data security, especially when dealing with encrypted data. This paper proposes a novel approach that merges convergent encryption (CE) with data deduplication. CE leverages user data itself to generate unique encryption keys, enabling secure deduplication on encrypted data. We analyze existing literature on secure data deduplication and categorize various techniques using UML activity diagrams. We then present our proposed CE‐based deduplication system, outlining its functionalities through UML diagrams. This research contributes to the field of secure data storage by proposing a novel and secure deduplication approach. By demonstrating its efficiency and security benefits, this work paves the way for more efficient and secure cloud storage solutions. Finally, we demonstrate the system's effectiveness through a comparative analysis, highlighting its potential to significantly improve storage efficiency while maintaining data security.

Hybrid deduplication system with content-based cache for cloud environment

Primary storage deduplication systems are performance sensitive. Their performance depends upon two factors — metadata access for duplicate detection and strategy for elimination of duplicate data. Various approaches for duplicate detection through suitable caching mechanisms have been proposed in the literature. Most of the approaches assumed that the primary workloads exhibit strong temporal locality. Whereas, this cannot be assumed in the context of Cloud as the workloads locality does not exist with interferences among different workloads on the same system. Duplicate content among the data blocks with different addresses lead to an inefficient utilization of the data cache. In this context, applying deduplication causes sharing of the data blocks among the clients with different access patterns and frequencies. In this situation, LRU cache, which considers only the recency of the references, is not appropriate. In this paper, Hybrid Deduplication System (HDS) containing the content-based cache with a new replacement policy — Modified Adaptive Replacement Cache (ARC), is proposed. The proposed system is simulated in the Linux environment using three different types of FIU traces. Effectiveness of the system is compared with a full deduplication system. Experimental results show that the system has performed consistently better than the full deduplication system in reducing the metadata overhead for all of the three data sets.

Space-efficient and high-performance inline deduplication for emerging hybrid storage system with Libra＋

In the face of the ever-increasing data volume of data-intensive applications, deduplication, as a promising solution to reduce space consumption, has attracted increasing attention in modern storage systems. Specifically, in modern SSD caching-based hybrid storage systems, inline deduplication achieves data reduction on the fly before storing data in the SSDs, thereby enhancing performance and extending their lifespan. However, with the emergence of new storage media featuring fast access speed and high endurance, hybrid storage systems have evolved by incorporating these new storage devices in place of SSDs at the front end of the primary storage. In this work, we revisit the concept of inline deduplication in the context of emerging hybrid storage and propose a new inline deduplication system designed for emerging hybrid storage, named Libra＋. Libra＋ contains three components: First, we propose two cooperative index structures for memory and cache. Second, we design a selective deduplication approach to perform targeted deduplication. Third, we introduce a lightweight dirty list that replaces the conventional dirty list in inline deduplication, reducing time overhead and space costs. Experimental results show that Libra＋ can achieve simultaneous performance improvement and space cost reduction.

DATA FINDING, SHARING AND DUPLICATION REMOVAL IN THE CLOUD

Deduplication involves eliminating duplicate or redundant data to reduce stored data volume, commonly used in data backup, network optimization, and storage management. However, traditional deduplication methods have limitations with encrypted data and security. The primary objective of this project is to develop new distributed deduplication systems that offer increased reliability. In these systems, data chunks are distributed across the Hadoop Distributed File System (HDFS), and a robust key management system is utilized to ensure secure deduplication with slave nodes. Instead of having multiple copies of the same content, deduplication removes redundant data by retaining only one physical copy and referring other instances to that copy. The granularity of deduplication can vary, ranging from an entire file to a data block. The MD5 and 3DES algorithms are used to enhance the deduplication process. The proposed approach in this project is the Proof of Ownership (POF) of the file. With this method, deduplication can effectively address the issues of reliability and label consistency in HDFS storage systems. The proposed system has successfully reduced the cost and time associated with uploading and downloading data, while also optimizing storage space. Key Words: Cloud computing, data storage, file checksum algorithms, computational infrastructure, duplication.

Secure Data Deduplication With Dynamic Access Control for Mobile Cloud Storage

Data deduplication is of vital importance for mobile cloud computing to cope with the explosive growth of outsourced mobile data. In order to ensure the privacy of sensitive mobile data against an untrusted cloud, Message-Locked Encryption (MLE) has been proposed to enable deduplication over ciphertext. However, MLE prohibits data access control since it uses deterministic content-derived encryption keys. Recently, a lightweight rekeying-aware encrypted deduplication system (REED) has been proposed to achieve dynamic access control for secure data deduplication. However, REED is vulnerable to key-retaining attack and stub-retaining attack, which leads to insecure access revocation, and thus cannot support secure dynamic access control. In response, we present AC-Dedup, an encrypted deduplication storage system that supports <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">secure dynamic access control</i> for mobile cloud storage. At the core of AC-Dedup are two novel encryption techniques named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mixed message locked encryption</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">random stub re-encryption</i> to resist the two types of attacks, respectively. To the best of our knowledge, AC-Dedup is the first practical system that achieves secure data deduplication and secure dynamic access control simultaneously. We conduct security analysis and experimental evaluation on mobile device and cloud platform with real-world IoT datasets. The results show that AC-Dedup enables secure and efficient dynamic access control while preserving deduplication effectiveness.

CLOUD BASED DUPLICATION REMOVAL SYSTEM

Deduplication involves eliminating duplicate or redundant data to reduce stored data volume, commonly used in data backup, network optimization, and storage management. However, traditional deduplication methods have limitations with encrypted data and security. The primary objective of this project is to develop new distributed deduplication systems that offer increased reliability. In these systems, data chunks are distributed across the Hadoop Distributed File System (HDFS), and a robust key management system is utilized to ensure secure deduplication with slave nodes. Instead of having multiple copies of the same content, deduplication removes redundant data by retaining only one physical copy and referring other instances to that copy. The granularity of deduplication can vary, ranging from an entire file to a data block. The MD5 and 3DES algorithms are used to enhance the deduplication process. The proposed approach in this project is the Proof of Ownership (POF) of the file. With this method, deduplication can effectively address the issues of reliability and label consistency in HDFS storage systems. The proposed system has successfully reduced the cost and time associated with uploading and downloading data, while also optimizing storage space. Key Words: Cloud computing, data storage, file checksum algorithms, computational infrastructure, duplication.

A survey on deduplication systems

A survey on deduplication systems

Modelling a Request and Response-Based Cryptographic Model For Executing Data Deduplication in the Cloud

Cloud storage is one of the most crucial components of cloud computing because it makes it simpler for users to share and manage their data on the cloud with authorized users. Secure deduplication has attracted much attention in cloud storage because it may remove redundancy from encrypted data to save storage space and communication overhead. Many current safe deduplication systems usually focus on accomplishing the following characteristics regarding security and privacy: Access control, tag consistency, data privacy and defence against various attacks. But as far as we know, none can simultaneously fulfil all four conditions. In this research, we offer a safe deduplication method that is effective and provides user-defined access control to address this flaw. Because it only allows the cloud service provider to grant data access on behalf of data owners, our proposed solution (Request-response-based Elliptic Curve Cryptography) may effectively delete duplicates without compromising the security and privacy of cloud users. A thorough security investigation reveals that our approved safe deduplication solution successfully thwarts brute-force attacks while dependably maintaining tag consistency and data confidentiality. Comprehensive simulations show that our solution surpasses the evaluation in computing, communication, storage overheads, and deduplication efficiency.

SimLESS: A Secure Deduplication System Over Similar Data in Cloud Media Sharing

SimLESS: A Secure Deduplication System Over Similar Data in Cloud Media Sharing

Retracted: Secure Data Deduplication System with Cyber Security Multikey Management in Cloud Storage

Retracted: Secure Data Deduplication System with Cyber Security Multikey Management in Cloud Storage

FCDedup: A Two-Level Deduplication System for Encrypted Data in Fog Computing

FCDedup: A Two-Level Deduplication System for Encrypted Data in Fog Computing

CJEU grapples with the intertwined nature of private copies and retransmission of online television broadcasts

Abstract In Ocilion, the Court of Justice of the European Union ruled on two intricated aspects of the relationship between copyright law and Internet TV services, clarifying that de-duplication systems of copying of TV broadcasts do not fall within the scope of the private copying exception and that facilitating online retransmission does not amount to a communication to the public.

Authorized Redundant Check Support in a Hybrid Cloud Environment

Data deduplication is one of the most significant data contraction ways for removing indistinguishable clones of recreating data, and it's generally used in pall storehouse to minimise storehouse space and save bandwidth. The coincident encryption approach has been developed to cipher the data before outsourcing to insure the confidentiality of sensitive data while easing deduplication. This work is the first attempt to explicitly address the content of authorised data deduplication in order to ameliorate data security. In discrepancy to standard deduplication systems, the discriminational boons of druggies are taken into account in indistinguishable check in addition to the data itself. In addition, we describe numerous innovative deduplication infrastructures that give authorised indistinguishable check in a cold-blooded pallarchitecture. Our approach is secure in terms of the delineations stated in the proposed security model, according to security analysis. We apply a prototype of our proposed authorised indistinguishable check medium as a evidence of conception and take over testbed tests with it. We demonstrate that our proposed authorised indistinguishable check fashion has a low outflow when compared to typical operations

HETERO-FLEX MULTI-CLOUD STORAGE MANAGEMENT SCHEME FOR CLOUD DATA DE-DUPLICATION REDUCTION

The rise of cloud situations has made doable the conveyance of Internet-scale benefits by tending to various difficulties, for example, security support, fault tolerance and nature of administration. Data de-duplication has pulled in numerous cloud specialist organizations (CSPs) as an approach to lessen capacity costs. Despite the fact that the general deduplication approach has been progressively acknowledged, it accompanies numerous security and protection issues due to the outsourced data conveyance models of cloud stockpiling. To manage particular security and protection issues, secure deduplication systems have been proposed for cloud data, prompting an assorted scope of arrangements and exchange offs. Conventional deduplication conspires dependably center around particular application situations, in which the deduplication is totally controlled by either data proprietors or cloud servers. They can't adaptably fulfill different requests of data proprietors as per the level of data affectability. We propose a Hetero-Flex Multi-cloud (HFMC) stockpiling administration plot, which adaptably offers deduplication administration and multi-server get to controls. The outcomes demonstrate its security, adequacy and effectiveness and protection support towards potential pragmatic use are for the most part enhanced than the current work. Key Words: Cloud Computing, Fog Computing, IoT, Big Data, Access Control, Cloud – Based Applications.

THE USE OF ROUGH CLASSIFICATION AND TWO THRESHOLD TWO DIVISORS FOR DEDUPLICATION

The data deduplication technique efficiently reduces and removes redundant data in big data storage systems. The main issue is that the data deduplication requires expensive computational effort to remove duplicate data due to the vast size of big data. The paper attempts to reduce the time and computation required for data deduplication stages. The chunking and hashing stage often requires a lot of calculations and time. This paper initially proposes an efficient new method to exploit the parallel processing of deduplication systems with the best performance. The proposed system is designed to use multicore computing efficiently. First, The proposed method removes redundant data by making a rough classification for the input into several classes using the histogram similarity and k-mean algorithm. Next, a new method for calculating the divisor list for each class was introduced to improve the chunking method and increase the data deduplication ratio. Finally, the performance of the proposed method was evaluated using three datasets as test examples. The proposed method proves that data deduplication based on classes and a multicore processor is much faster than a single-core processor. Moreover, the experimental results showed that the proposed method significantly improved the performance of Two Threshold Two Divisors (TTTD) and Basic Sliding Window BSW algorithms.

Attribute-Based Storage Supporting Secure Deduplication of Encrypted Data in Cloud

Abstract: Attribute-based encryption (ABE) has been widely used in cloud computing where a data provider outsources his/her encrypted data to a cloud service provider, and can share the data with users possessing specific credentials (or attributes). However, the standard ABE system does not support secure deduplication, which is crucial for eliminating duplicate copies of identical data in order to save storage space and network bandwidth. In this paper, we present an attribute-based storage system with secure deduplication in a hybrid cloud setting, where a private cloud is responsible for duplicate detection and a public cloud manages the storage. Compared with the prior data deduplication systems, our system has two advantages. First, it can be used to confidentially share data with users by specifying access policies rather than sharing decryption keys. Second, it achieves the standard notion of semantic security for data confidentiality while existing systems only achieve it by defining a weaker security notion. In addition, we put forth a methodology to modify a ciphertext over one access policy into ciphertexts of the same plaintext but under other access policies without revealing the underlying plaintext.

FuzzyDedup: Secure Fuzzy Deduplication for Cloud Storage

Data deduplication is of critical importance to reduce the storage cost for clients and to relieve the unnecessary storage pressure for cloud servers. While various techniques have been proposed for secure deduplication of identical files/blocks, the effective and secure deduplication solutions on fuzzy similar data (image, video, and others) which occupy a large portion in the real world across wide applications, remain open. In this paper, we propose a novel deduplication system, named Fuzzy Deduplication (FuzzyDedup), to implement the secure deduplication of similar data (i.e., similar files, chunks, or blocks). In particular, we leverage the similarity-preserving hash, a fuzzy extractor based on error-correcting codes, and the encryption with customized design to construct a fuzzy-style deduplication encryption scheme (FuzzyMLE), achieving the ciphertext-based deduplication for similar data. Besides, to defend against data ownership cheating attack and duplicate-faking attack, a fuzzy-style proof of ownership scheme (FuzzyPoW) is designed for the cloud server to securely verify a client in possession of the similar data. To further enhance security and efficiency, we also propose both server-aided and random-tag FuzzyMLE to make FuzzyDedup robust against off-line brute-force attack and to support tag randomization, respectively. Then, we design Hamming distance reduction and tag cutting optimization algorithms to improve the tag query efficiency of FuzzyDedup. In the end, we formally prove the security of our solution and conduct experiments on real-world datasets for performance evaluation. Experimental results exhibit the efficiency of FuzzyDedup in terms of computation cost and communication overhead.

Bulut Bilişimin Gelişmesi ile Büyük Verinin Tekilleştirilmesinde Yapay Zekâ Öneminin İncelenmesi

With the start of Industry 4.0 in 2011, new concepts and technologies have entered the IT literature. Some of these technologies are virtualization, modularity, big data and deduplication. Big data can be defined as data of a magnitude that exceeds the ability of traditional database systems to collect, store, manage and analyze data. Today, data is diverse, large and rapidly changing. This situation cannot be solved with the traditional database structure. With the emergence of big data, it has become difficult to process data with the algorithms used for data processing. Therefore, new algorithms and technologies have been developed. The most important of these technologies is data deduplication. Deduplication backs up data by dividing it into variable or fixed sizes. In this way, it aims to save storage space by storing only one copy of many repeated data. Today, "deduplication and compression" is an indispensable feature for data storage in both server-storge and hyper-converged architecture systems. Recently, artificial intelligence technologies are advancing very rapidly and their application areas are expanding. Therefore, Artificial Intelligence is a technology that will be very important for the industry and our lives in the future. The purpose of this paper is to give an idea about the relationship between deduplication technology and artificial intelligence by examining various deduplication systems and algorithms. Studies in the literature show that deduplication provides significant savings in storage space, the importance of data security, and the use of artificial intelligence and deduplication as a whole.

An Efficient Cloud Storage Model for GOP-Level Video Deduplication using Adaptive GOP Structure

Cloud storage systems may be used by users and businesses to transfer their enormous volumes of data for storage, processing, and analysis. Cloud storage may be used efficiently by employing the deduplication technique to prevent duplicate copies. Users may save any type of material, including audio, video, images, text, and more. The handling of deduplication techniques for these distinct data types may vary depending on their characteristics, sizes, etc. Video deduplication is one of these, and it’s essential for reducing memory waste in the cloud storage system. There are many levels of video deduplication that may be done. In this research work, we proposed a GOP-level deduplication system using an adaptive GOP structure. Since GOP is the level of deduplication, it is inefficient to build GOP with a fixed size. It can fail to discover a pair of identical frames. The adaptive GOP structure will yield GOPs with closer relation among their frames than with a fixed-size GOP structure. The proposed technique is compared with the fixed-size GOP structure with the GOP sizes of 8, 10, 12, and 15 and the proposed technique achieved a 2.18% PSNR gain which is relatively higher than other fixed-size GOP structures.