Block-level Deduplication Research Articles

Enhancing Resource Utilization Efficiency in Serverless Education: A Stateful Approach with Rofuse

Traditional container orchestration platforms often suffer from resource wastage in educational settings, and stateless serverless services face challenges in maintaining container state persistence during the teaching process. To address these issues, we propose a stateful serverless mechanism based on Containerd and Kubernetes, focusing on optimizing the startup process for container groups. We first implement a checkpoint/restore framework for container states, providing fundamental support for managing stateful containers. Building on this foundation, we propose the concept of “container groups” to address the challenges in educational practice scenarios characterized by a large number of similar containers on the same node. We then propose the Rofuse optimization mechanism, which employs delayed loading and block-level deduplication techniques. This enables containers within the same group to reuse locally cached file system data at the block level, thus reducing container restart latency. Experimental results demonstrate that our stateful serverless mechanism can run smoothly in typical educational practice scenarios, and Rofuse reduces the container restart time by approximately 50% compared to existing solutions. This research provides valuable exploration for serverless practices in the education domain, contributing new perspectives and methods to improve resource utilization efficiency and flexibility in teaching environments.

Cloud storage auditing and data sharing with data deduplication and private information protection for cloud-based EMR

In the cloud-based electronic medical records (EMRs) system, the doctors upload EMRs to the cloud for storage and sharing. However, the integrity of EMRs stored in the cloud cannot be guaranteed and sharing EMRs may leak the patients’ private information. Furthermore, shared EMRs contain a lot of duplicate data, leading to data redundancy. To solve the above issues, we propose a cloud storage auditing and data sharing scheme with data deduplication and private information protection for cloud-based EMR. In our proposed scheme, we introduce the sanitizer to sanitize the data blocks containing the private information in EMR, and transform these blocks’ authenticators into the authenticators of sanitized EMR. The authenticators are utilized to check the data integrity. Using the above method, the EMR stored in the cloud can be shared with researchers while ensuring the privacy of private information and the integrity of EMR. In addition, our scheme can achieve block-level deduplication, in which only one copy of the deduplicated blocks in EMR needs to be stored. The conducted security analysis and performance evaluation affirm the security and efficiency of the proposed scheme.

Secure Cloud-Based Electronic Health Records: Cross-Patient Block-Level Deduplication with Blockchain Auditing.

In today's data-driven world, the exponential growth of digital information poses significant challenges in data management. In recent years, the adoption of cloud-based Electronic Health Records (EHR) sharing schemes has yielded numerous advantages like improved accessibility, availability, and enhanced interoperability. However, the centralized nature of cloud storage presents challenges in terms of information storage, privacy protection, and security. Despite several approaches that have been presented to ensure secure deduplication of similar EHRs, the validation of data integrity without a third-party auditor (TPA) remains a persistent task. Because involving a TPA raises concerns about the confidentiality and privacy of crucial healthcare information. To tackle this challenge, a novel cloud storage auditing technique is proposed that incorporates cross-patient block-level deduplication while upholding strong privacy protection, ensuring that EHR is not compromised. Here, we introduced blockchain technology to achieve integrity verification, thus eliminating the need for a TPA by providing a decentralized and transparent mechanism. Additionally, an index for all EHRs has been generated to facilitate block-level duplicate checks and employ a novel strategy to prevent adversaries from acquiring original information saved in the cloud storage. The security of the proposed approach is established against factorization attacks and decrypt exponent attacks. The performance evaluation demonstrates the superior efficiency of the proposed scheme in terms of file authenticator generation, challenge creation, and proof verification to other existing client-side deduplication approaches.

ObjDedup: High-Throughput Object Storage Layer for Backup Systems With Block-Level Deduplication

The immense popularity of object storage is also affecting the market of backup. Not only have novel backup solutions emerged that utilize cloud-based object storage as backends, but also support for object storage interfaces is increasingly expected from traditional dedicated backup appliances. This latter trend especially concerns systems with data deduplication, as they can offer compelling gains in storage capacity and throughput. However, such systems have been designed for interfaces and workloads that are markedly different from those encountered in object storage. Notably, they expect data to be written in portions that are orders of magnitude longer than those in the novel object-storage-oriented backup applications. In this light, we contribute twofold. First, contrasting the properties of object storage interfaces with usage patterns from 686 commercial deployments of backup appliances, we identify specific issues an implementation of such an interface has to address to offer adequate performance in a backup system with block-level deduplication. In particular, we show that a major challenge is efficient metadata management. Second, we present distributed data structures and algorithms to handle object metadata in backup systems with block-level deduplication. Subsequently, we implement them as an object storage layer for our HYDRAstor backup system. In comparison to object storage without in-line deduplication, our solution achieves 1.8–3.93x higher write throughput. Compared to object storage on top of a state-of-the-art file-based backup system, it processes 5.26–11.34x more object put operations per time unit.

Hybrid cloud storage system with enhanced multilayer cryptosystem for secure deduplication in cloud

Data deduplication is a crucial technique in the field of data compression that aims to eliminate redundant copies of recurring data. This technique has gained significant popularity in the realm of cloud storage due to its ability to effectively reduce storage requirements and optimize bandwidth utilization. To ensure the safeguarding of sensitive data while simultaneously facilitating deduplication, researchers have put forth the concept of convergent encryption as a potential solution. This technique involves encrypting the data prior to its outsourcing, thereby enhancing the confidentiality of the information. In this work, an earnest endeavor is undertaken to formally tackle the issue of authorized data deduplication, with the aim of enhancing data security. Our approach combines the Diffie-Hellman algorithm and symmetrical external decision to protect and popularize information, ensuring end-to-end encryption to encourage user adoption of cloud storage. The proposed model employs block-level deduplication and guarantees the randomness of ciphertexts by generating encryption keys using the Diffie-Hellman algorithm. This method effectively counters both internal and external brute-force attacks, enhancing data security while reducing computational costs. An extensive experimentation is carried out to demonstrate that our approach is particularly beneficial in scenarios with multiple privilege sets. Overall, the proposed model offers an elaborate framework that maintains data privacy and strengthens security measures, contributing to a more efficient and secure cloud-based document search.

A blockchain-based compact audit-enabled deduplication in decentralized storage

Secure deduplication and data auditing are significant in improving the resource utilization and data integrity protection of data outsourcing services. However, existing audit-enabled deduplication schemes cannot implement block-level deduplication over audit tags since they contain file information, and have to bear the consequent storage burden. Also, they suffer from the low coupling problem caused by the difference in optimal data chunking between deduplication and auditing. In this paper, we propose a blockchain-based compact audit-enabled deduplication scheme in decentralized storage. Specifically, we introduce the concept of N-ary tag commitment tree (NTCT), which integrates the information of all data blocks into a file authenticator to support integrity verification and enable block-level deduplication over audit tags. On this basis, we propose a blockchain-based compact audit-enabled deduplication protocol, which adopts an aggregatable vector commitment to generate audit tags, thereby overcoming the low coupling problem between deduplication and auditing. Notably, data users can outsource the task of tag generation to storage servers. Meanwhile, the capabilities of duplication detection, proof of ownership (PoW), and integrity verification are integrated into audit tags to reduce tag redundancy. Furthermore, we present an update protocol to allow data users to achieve a lightweight data update without uploading the entire new data. Finally, security analysis and performance evaluation demonstrate that our scheme is practical.

Dynamic Deduplication Algorithm for Cross-User Duplicate Data in Hybrid Cloud Storage

The escalating growth of distributed big data in hybrid cloud storage architecture introduces a new set of challenges. Constantly, content enrichment puts pressure on capacity. Nonetheless, the explosion of user data places a significant strain on broadband and storage capacity. Consequently, many cloud storage providers will implement deduplication to compress data, reduce transfer bandwidth, and reduce cloud storage space. In cloud storage systems, it is a data compression and storage optimization method. By locating and removing redundant data, it can save storage space and bandwidth. An MTHDedup deduplication strategy based on the Merkle hash tree is presented in a hybrid cloud environment to address the issue of convergent encryption algorithms being susceptible to brute-force attacks and ciphertext computation time overhead. Merkle hash trees are constructed using additional encryption algorithms to generate encryption keys during file- and block-level deduplication, ensuring that generated ciphertexts are unpredictable. The method is effective against both internal and external brute-force attacks, thereby increasing data security. Our method reduces the computational burden of ciphertext generation and the key storage space, and the performance advantage increases with the number of privilege sets.

Backup Storage Block Level Deduplication with DDUMBFS and BACULA

Backup Storage Block Level Deduplication with DDUMBFS and BACULA

Backup Storage Block Level Deduplication with DDUMBFS and BACULA

Backup Storage Block Level Deduplication with DDUMBFS and BACULA

Updatable Block-Level Message-Locked Encryption

Deduplication is widely used for reducing the storage requirement for storage service providers. Nevertheless, it is unclear how to support deduplication of encrypted data securely until the study of Bellare et al. on message-locked encryption (MLE, Eurocrypt 2013). While updating (shared) files is natural, existing MLE solutions do not allow efficient update of encrypted files stored remotely. Even modifying a single bit requires the expensive way of downloading and decrypting a large ciphertext (then re-uploading). This paper initiates the study of updatable block-level MLE, a new primitive in incremental cryptography and cloud cryptography. Our proposed provably-secure construction is updatable with computation cost logarithmic in the file size. It naturally supports block-level deduplication. It also supports proof-of-ownership which protects storage providers from being abused as a free content distribution network. Our experiments show its practical performance relative to the original MLE and existing non-updatable block-level MLE.

ESKEA: Enhanced Symmetric Key Encryption Algorithm Based Secure Data Storage in Cloud Networks with Data Deduplication

Data deduplication approach is utilized in cloud storage to decrease the bandwidth of communication and storage space by eliminating the data copies from the cloud service provider (CSP). However, one of the main problems of cloud storage is data deduplication with secure data storage. To overcome this issue, the researchers presented symmetric data storage methods based on an encryption algorithm. Nonetheless, the Enhanced Symmetric Key Encryption Algorithm (ESKEA) based on secure data storage with data deduplication is proposed in this research to further improve data confidentiality. In this approach, the block-level deduplication of data is performed using the Convergent Encryption (CE) algorithm to check the CSP duplicate copies of data. Then, ESKEA algorithm is presented for secure storage of data. In ESKEA, Spider Monkey Optimization Algorithm (SMOA) optimally selects the secret key. The results of this work showed that the execution of the proposed ESKEA is more effective than that of the SKEA in terms of upload time and download time.

Secure Block Level De-Duplication on Big Data using Proof of Ownership

De-duplication technology is commonly used to decrease the space and bandwidth requirements of services by eliminating repeated data and store only a single copy of them. Unfortunately, it raises issues relating to security and ownership such as, unauthorized users may claim as owner of that file and security threat from curious server itself. To overcome these issues, Secure Block level de-duplication with Proof of Ownership Scheme is proposed in this work. Proof-of-Ownership Scheme allows any owner of the same data to prove to the server that he owns the data in a robust way. This scheme uses convergent encryption and it protects the data from attackers and honest but curious server. It also reduces storage space efficiently by checking the uniqueness of data at block level

A Content Level Based Deduplication on Streaming Data using Poisson Process Filter Technique (PPFT)

This paper proposed on Poisson process-based algorithm is to carry out content-level deduplication for the streaming data. Since Poisson processes are meant to do the counting of different events happening over a period of time and space, it becomes appropriate to use it for identifying duplications of data as it gets streamed based on time and space, which can allow the deduplication process to be carried out in tandem. Some of the research on deduplication has been focusing on File-level and Block-level deduplication while the focus can be brought to content-level, as data get streamed lively and becomes more dynamic. With this approach, the content-level deduplication will allow the data to be scanned intelligently and at the same time, it can save the deduplication operation time. Also, streaming data has its randomness which is innately there and by having Poisson process based deduplication it will address the random behaviour of the data transfer and can work efficiently in the dynamically connected environment. The proposed method identifies the unique data to store in the Database. Based on the experimental result, the Poisson Process-based algorithm produce 0.912 Area Under Curve (AUC) accuracy on real-world streaming data, which means that if AUC is greater than 0.8 then the performance of algorithm is pretty good. So, the machine intelligence-based deduplication model produced reliable and robust deduplication on streaming data compared to existing approaches.

Adaptive deduplication of virtual machine images using AKKA stream to accelerate live migration process in cloud environment

Cloud Computing is a paradigm which provides resources to users from its pool based on demand to satisfy their requirements. During this process, many servers are overloaded and underloaded in the cloud environment. Thus, power consumption and load balancing are the major problems and are resolved by live virtual machine (VM) migration. Load balancing is addressed by moving virtual machines from overloaded host to under loaded host and from under loaded host to any other host which is not overloaded called VM migration. If this process is done without power off (Live) the virtual machines then it is called live VM migration. By this process, the issue of power consumption by physical hosts is also resolved. Migrating virtual machines involves virtualized components like storage disks, memory, CPU and networking, the entire state of VM is captured as a collection of data files. These data files are virtual disk files, configuration files, and log files. The virtual disk files take larger memory and take more migration time and hence the downtime. These disk files include many zero pages, similar and redundant pages. Many techniques such as compression, deduplication is used reduce the size of VM disk image file. Compression techniques are not widely used, due to the disadvantage of compression ratio and decompression time. Many researchers hence used deduplication techniques for reducing the VM disk image file in the live migration process. The significance of the research work is to design an adaptive deduplication mechanism for reducing VM disk image file size by performing fixed length and variable length block-level deduplication processes. The Rabin-Karp rolling hash algorithm is used in variable length block-level deduplication. Akka stream is used for streaming the VM disk image files as it is the bulk volume of live data transfer. To reduce the time of the deduplication process, many researchers used multithreading and multi-core technologies. We use multithreading in Akka framework to run the deduplication process concurrently without OutofMemory errors. The experiment results show that we achieved a maximum of 83% overall reduction in image storage space and 89.76% reduction in total migration time are achieved by adaptive deduplication method. 3% improvement in deduplication rate when compared with the existing image management system. The results are significant because when we apply this in the storage of data centres, there are much space savings. The reduction in size is dependent on the dataset was taken and the applications running on the VM.

АРХІТЕКТУРА СИСТЕМИ ДЕДУБЛІКАЦІЇ ТА РОЗПОДІЛУ ДАНИХ У ХМАРНИХ СХОВИЩАХ ПІД ЧАС РЕЗЕРВНОГО КОПІЮВАННЯ

The conceptual model of the system is developed and described in detail. An intelligent system of deduplication and distribution of data in the cloud storage is developed, the description of the software is described, the stages of the user's work are considered. Testing of the projected system was carried out. Several control samples are described and results are analyzed. The purpose of the system is to deduplicate and distribute data in cloud repositories in such a way that the end result of the backup is to eliminate duplicate pieces of data using distributed computing and cloud repositories. By picking the right approach to distribute tasks and data during deduplication, you can harness the full potential of cloud-based distributed systems to increase backup speed and bandwidth. Analyzes (disadvantages and advantages of using different approaches) are analyzed and effective methods of solution are selected: hybrid block-level deduplication, splitting of data flow on the basis of Rabin's digital imprint, distribution of data based on hash values of blocks of deduplication and use of distributed index. Block-level deduplication involves two types of data flow splitting into blocks, a fixed-length, algorithm-based split. Fixed-length partitioning is rather trivial and fast with respect to the complexity of the algorithm, but the downside is that data is displaced at the beginning of the stream, since the blocks that will follow after the changes will be considered new. However, in the case of partitioning of blocks of variable length, the point of proper partitioning is determined by the algorithm. This algorithm should work with infinite data flows using the ring hash function. The algorithm absorbs each input byte of data from the stream, and as soon as the value of the annular hash function corresponds to the previously specified template, it also serves as a point of splitting the stream into blocks. Thus, if the data is changed or displaced by a couple of bytes, only the data block that covers the data will be considered new. However, in order to track changes and correctly set breakpoints, it is necessary to check the input data for a specific preset digital pattern - a hash value. It is a common practice to calculate a hash value every time an input byte is received in a data stream. The point of partition will be the moment when the resulting hash value matches the specified pattern. To do these calculations effectively, an algorithm has been devised for the ring hash. One of the most common ring hash algorithms is a digital Rabin imprint. During the analysis of the solutions, the Rust programming language for client-side writing, the Scala programming language for the server-side, the Akka distributed computing management tool, and Amazon S3 as the cloud repository were selected.

Minimal re-computation for exploratory data analysis in astronomy

Abstract We present a technique to automatically minimise the re-computation when a data analysis program is iteratively changed, or added to, as is often the case in exploratory data analysis in astronomy. A typical example is flagging and calibration of demanding or unusual observations where visual inspection suggests improvement to the processing strategy. The technique is based on memoization and referentially transparent tasks. We describe the implementation of this technique for the CASA radio astronomy data reduction package. We also propose a technique for optimising efficiency of storage of memoized intermediate data products using copy-on-write and block level de-duplication and measure their practical efficiency. We find that the minimal recomputation technique improves the efficiency of data analysis while reducing the possibility for user error and improving the reproducibility of the final result. It also aids exploratory data analysis on batch-schedule cluster computer systems.

Secure Deduplication Based on Rabin Fingerprinting over Wireless Sensing Data in Cloud Computing

The rapid advancements in the Internet of Things (IoT) and cloud computing technologies have significantly promoted the collection and sharing of various data. In order to reduce the communication cost and the storage overhead, it is necessary to exploit data deduplication mechanisms. However, existing data deduplication technologies still suffer security and efficiency drawbacks. In this paper, we propose two secure data deduplication schemes based on Rabin fingerprinting over wireless sensing data in cloud computing. The first scheme is based on deterministic tags and the other one adopts random tags. The proposed schemes realize data deduplication before the data is outsourced to the cloud storage server, and hence both the communication cost and the computation cost are reduced. In particular, variable-size block-level deduplication is enabled based on the technique of Rabin fingerprinting which generates data blocks based on the content of the data. Before outsourcing data to the cloud, users encrypt the data based on convergent encryption technologies, which protects the data from being accessed by unauthorized users. Our security analysis shows that the proposed schemes are secure against offline brute-force dictionary attacks. In addition, the random tag makes the second scheme more reliable. Extensive experimental results indicate that the proposed data deduplication schemes are efficient in terms of the deduplication rate, the system operation time, and the tag generation time.

Secure Deduplication for Cloud Storage Using Interactive Message-Locked Encryption with Convergent Encryption, To Reduce Storage Space

ABSTRACT The digital data stored in the cloud requires much space due to copy of the same data. It can be reduced by dedupilcation, eliminating the copy of the repeated data in the cloud provided services. Identifying common checkoff data both files storing them only once. Deduplication can yield cost savings by increasing the utility of a given amount of storage. Unfortunately, deduplication has many security problems so more than one encryption is required to authenticate data. We have developed a solution that provides both data security and space efficiency in server storage and distributed content checksum storage systems. Here we adopt a method called interactive Message-Locked Encryption with Convergent Encryption (iMLEwCE). In this iMLEwCE the data is encrypted firstly then the cipher text is again encrypted. Block-level deduplication is used to reduce the storage space. Encryption keys are generated in a consistent configuration of data dependency from the chunk data. The identical chunks will always encrypt to the same cipher text. The keys configuration cannot be deduced by the hacker from the encrypted chunk data. So the information is protected from cloud server. This paper focuses on reducing the storage space and providing security in online cloud deduplication.

DIFFERENT SECURE DATA DEDUPLICATION APPROACHES FOR CLOUD STORAGE: A REVIEW

With the fast growth of the current era, world are moving to digital storage for accomplishment purposes, there is a rising demand for systems that can deliver safe data storage in effective manner. As more corporate and private users outsource their data to cloud storage providers, new data breach incidents make end-to-end encryption progressively prominent obligation. With the growing awareness of data privacy, more and more cloud users choose to encrypt their sensitive data before outsourcing them to the cloud storage. Data deduplication is single instance data storage widely used in cloud storage system to reduce space and upload bandwidth. Duplication-less storage system which de-duplicates the data using file level deduplication and block level deduplication. In this paper we gives an overview of the state of data Deduplication approachesand describes the various application to produce specification according need of current generation.

Centralized Duplicate Removal Video Storage System with Privacy Preservation in IoT.

In recent years, the Internet of Things (IoT) has found wide application and attracted much attention. Since most of the end-terminals in IoT have limited capabilities for storage and computing, it has become a trend to outsource the data from local to cloud computing. To further reduce the communication bandwidth and storage space, data deduplication has been widely adopted to eliminate the redundant data. However, since data collected in IoT are sensitive and closely related to users’ personal information, the privacy protection of users’ information becomes a challenge. As the channels, like the wireless channels between the terminals and the cloud servers in IoT, are public and the cloud servers are not fully trusted, data have to be encrypted before being uploaded to the cloud. However, encryption makes the performance of deduplication by the cloud server difficult because the ciphertext will be different even if the underlying plaintext is identical. In this paper, we build a centralized privacy-preserving duplicate removal storage system, which supports both file-level and block-level deduplication. In order to avoid the leakage of statistical information of data, Intel Software Guard Extensions (SGX) technology is utilized to protect the deduplication process on the cloud server. The results of the experimental analysis demonstrate that the new scheme can significantly improve the deduplication efficiency and enhance the security. It is envisioned that the duplicated removal system with privacy preservation will be of great use in the centralized storage environment of IoT.