Erasure Codes Research Articles

New Design and Analysis of Error-Resilient LRCs for DSSs With Silent Disk Errors

Recently, erasure coding techniques are considered as essential schemes for the reliability of the modern distributed storage systems (DSSs) with the frequent node-level failure. Especially, locally repairable codes (LRCs) are widely adopted by the practical advantage of reducing the latency for repair process. However, recent researches show that many cases for system failure are also originated from the silent disk errors. For the conventional LRCs with low error correction capability, repair process from erasure coding can propagate the silent errors and thus, the DSSs become more vulnerable compared to the cases only with node failure. Therefore, we propose a mean time to data loss (MTTDL) from the modified Markov chain model in order to evaluate effects by silent disk errors. Also, new design of binary error-resilient locally repairable codes (ER-LRCs) with high error and erasure correction capabilities are proposed, which have larger values of bit-wise minimum Hamming distance than the existing LRCs. Here, ER-LRCs can be constructed by modifying the parity check matrix from well-known optimal binary and nonbinary LRCs. From the numerical analysis using the proposed Markov model with empirical parameters, it is shown that the proposed ER-LRCs have better MTTDL values when compared to the existing LRCs.

Review on LDPC Codes for Big Data Storage

The need for highly scalable and reliable big data storage systems is due to the fact that there is an explosive growth in data everywhere particularly due to data generated from social networking sites and IOT Technology for various applications. Hence most of Information Technology, Medical organizations and big Industries, Social networking companies, Government organizations (like ISRO) are required to have storage capacities of 100 PB (petabytes) of data. Therefore to secure and store this kind of large data efficiently research on application of Erasure codes for both Cloud storage and Network or Distributed Storage systems is considered recently. The traditional triple-replication method, which stores 3 copies of every file and requires an extra 200% storage overhead is actually highly expensive. Erasure Codes are considered for parallel storage systems as an alternative to traditional storage system. This paper presents the various techniques of applying LDPC codes for big data storage and provides the research gap to investigate application of LDPC codes for big data storage.

Adaptive and Heterogeneity-Aware Coded Cooperative Computation at the Edge

Cooperative computation is a promising approach for localized data processing at the edge, e.g., for Internet of Things(IoT). Cooperative computation advocates that computationally intensive tasks in a device could be divided into sub-tasks, and offloaded to other devices or servers in close proximity. However, exploiting the potential of cooperative computation is challenging mainly due to the heterogeneous and time-varying nature of edge devices. Coded computation, which advocates mixing data in sub-tasks by employing erasure codes and offloading these sub-tasks to other devices for computation, is recently gaining interest, thanks to its higher reliability, smaller delay, and lower communication costs. In this paper, we develop a coded cooperative computation framework, which we name Coded Cooperative Computation Protocol (C3P), by taking into account the heterogeneous and time-varying resources of edge devices. C3P dynamically offloads coded sub-tasks to helpers and is adaptive to time-varying resources. We show that (i) task completion delay of C3P is very close to optimal coded cooperative computation solutions, (ii) the efficiency of C3P in terms of resource utilization is higher than 99%, and (iii) C3P improves task completion delay significantly as compared to baselines via both simulations and in a testbed consisting of real Android-based smartphones.

LHC Data Storage: Preparing for the Challenges of Run-3

The CERN IT Storage Group ensures the symbiotic development and operations of storage and data transfer services for all CERN physics data, in particular the data generated by the four LHC experiments (ALICE, ATLAS, CMS and LHCb). In order to accomplish the objectives of the next run of the LHC (Run-3), the Storage Group has undertaken a thorough analysis of the experiments’ requirements, matching them to the appropriate storage and data transfer solutions, and undergoing a rigorous programme of testing to identify and solve any issues before the start of Run-3. In this paper, we present the main challenges presented by each of the four LHC experiments. We describe their workflows, in particular how they communicate with and use the key components provided by the Storage Group: the EOS disk storage system; its archival back-end, the CERN Tape Archive (CTA); and the File Transfer Service (FTS). We also describe the validation and commissioning tests that have been undertaken and challenges overcome: the ATLAS stress tests to push their DAQ system to its limits; the CMS migration from PhEDEx to Rucio, followed by large-scale tests between EOS and CTA with the new FTS “archive monitoring” feature; the LHCb Tier-0 to Tier-1 staging tests and XRootD Third Party Copy (TPC) validation; and the erasure coding performance in ALICE.

The first disk-based custodial storage for the ALICE experiment

We proposed a disk-based custodial storage as an alternative to tape for the ALICE experiment at CERN to preserve its raw data. The proposed storage system relies on Redundant Array of Independent Nodes (RAIN) layout – the implementation of erasure coding in the EOS storage suite, which is developed by CERN – for data protection and takes full advantage of high-density Just-Bunch-Of-Disks (JBOD) enclosures to maximize storage capacity as well as to achieve cost-effectiveness comparable to tape. The system we present provides 18 PB of total raw capacity from the 18 set of high-density JBOD enclosures attached to 9 EOS front-end servers. In order to balance between usable space and data protection, the system will stripe a file into 16 chunks on the 4-parity enabled RAIN layout configured on top of 18 containerized EOS FSTs. Although the reduction rate of available space increases up to 33:3% with this layout, the estimated annual data loss rate drops down to 8:6 × 10−5%. In this paper, we discuss the system architecture of the disk-based custodial storage, 4-parity RAIN layout, deployment automation, and the integration to the ALICE experiment in detail.

Multi cloud-based secure privacy preservation of hospital data in cloud computing

The growth of cloud computing has led to privacy concerns abundantly. Any organisation/user sends all the information to the cloud service provider and so the organisation's/user's data security is a concern. Data privacy and security issues can be solved by establishing clear policies that enable authorised data access and security. User authentication is the primary basis for access control and so using cryptographic encryption mechanism like key policy attribute-based encryption we can provide strong authentication to ensure that data can be viewed by only who have to access it. Followed by which a never compromised integrity mechanism like SHA-256 hash mechanism is used to ensure that data is not modified in transit. These hashes are concatenated in a way to form top hash by structuring in a Merkle hash tree. They are used by the erasure code to find lost data during any of the crashes. To make it efficient and to find the data loss, third party auditors are installed to check and report any changes in any of the cloud storage. Data recovery is by means of retrieving the data from another cloud that has the replica of these data.

Flexible Data Integrity Checking With Original Data Recovery in IoT-Enabled Maritime Transportation Systems

Internet of things (IoT) has emerged as a promising technology that can be widely used in various industries to realize real-time information collection, so as to improve production efficiency and reduce running costs. By combining the technology of IoT, maritime transportation systems (MTS) can prevent vessels collision, improve the efficiency of maritime transportation and reduce the loss of revenue for ports and shipbuilders. The large amount of real-time data generated in IoT-enabled MTS can be efficiently utilized to predict the future trajectories and hotspots of vessels on the sea combined with historical data. However, the maritime traffic data in MTS cannot be effectively processed in traditional big data analysis methods, and the integrity of it needs to be checked before being used to achieve the prediction of trajectories and high-density areas of vessels. In this paper, we propose a flexible data integrity checking scheme with original data recovery in IoT-enabled MTS. In the proposed scheme, the data blocks of vessels are encoded based on the technology of erasure coding. To ensure the availability of the historical data, the existence and the integrity of the data elements stored in the cloud can be checked. Moreover, the original data blocks can be recovered efficiently if the encoded data elements have been corrupted or deleted. Security analysis demonstrates that the proposed scheme can be proved to be correct and is secure against malicious attacks. Performance analysis shows that our scheme is more efficient than the previous schemes.

An Optimal Tree-Structured Repair Scheme of Multiple Failure Nodes for Distributed Storage Systems

To reduce recovery cost of repairing multiple failed nodes, many repair schemes have been proposed for erasure codes based distributed storage systems. However, most of the existing researches ignore the network topology of storage devices. Motivated by such considerations, we combine delay repair schemes with network topology and propose a tree-structured model based on fountain codes with large value of (n, k, r) to improve the repair efficiency. More precisely, with the consideration of network topology, a new target named data recovery cost is defined to measure the efficiency of coded fragment download and source file reconstruction, and then the optimal recovery threshold is derived to minimize the average data recovery cost of general tree-structured model. Moreover, we analyze and compare the average data recovery cost of general tree-structure with different systematic parameters. To further improve the data transmission efficiency, an optimal tree-structured scheme based on improved tabu search algorithm (ITSAORT) is proposed. Compared with other algorithms, the ITSA-ORT scheme uses Prim algorithm to generate the initial solution and then uses special method to obtain the corresponding neighborhood structure. The experimental results show that the proposed scheme can find a globally optimal solution and obtain lower cost of data recovery. In addition, the ITSA-ORT scheme has lower computational complexity than the optimal tree-structured scheme based on particle swarm optimization algorithm (PSO-ORT) and the optimal tree-structured scheme based on firefly algorithm (FA-ORT).

VidCloud

As video-streaming services have expanded and improved, cloud-based video has evolved into a necessary feature of any successful business for reaching internal and external audiences. In this article, video streaming over distributed storage is considered where the video segments are encoded using an erasure code for better reliability. We consider a representative system architecture for a realistic (typical) content delivery network (CDN). Given multiple parallel streams/link between each server and the edge router, we need to determine, for each client request, the subset of servers to stream the video, as well as one of the parallel streams from each chosen server. To have this scheduling, this article proposes a two-stage probabilistic scheduling. The selection of video quality is also chosen with a certain probability distribution that is optimized in our algorithm. With these parameters, the playback time of video segments is determined by characterizing the download time of each coded chunk for each video segment. Using the playback times, a bound on the moment generating function of the stall duration is used to bound the mean stall duration. Based on this, we formulate an optimization problem to jointly optimize the convex combination of mean stall duration and average video quality for all requests, where the two-stage probabilistic scheduling, video quality selection, bandwidth split among parallel streams, and auxiliary bound parameters can be chosen. This non-convex problem is solved using an efficient iterative algorithm. Based on the offline version of our proposed algorithm, an online policy is developed where servers selection, quality, bandwidth split, and parallel streams are selected in an online manner. Experimental results show significant improvement in QoE metrics for cloud-based video as compared to the considered baselines.

Modified graph-state codes for single-node recovery in quantum distributed storage

Distributed quantum information stored over nodes within a quantum network are prone to node failures, motivating the need for coded quantum networks. We encode $k$ qubits of quantum information over an $n$-node network in a distributed fashion, with one qubit per node, using modified graph-state codes. We devise a procedure to recover the encoded quantum information in any given arbitrary network after a detected node fails. The recovery procedure is locally within four edges of the failed node. For a network with even number of nodes, we provide a class of scalable modified graph-state codes with rate $(n\ensuremath{-}2)/n$, saturating the quantum Singleton bound. This yields higher coding rates than Grassl et al.'s well-known erasure codes and has asymptotic rate 1. For odd node size $n$, we design the network links to obtain an $(n\ensuremath{-}3)/n$ rate-optimal modified graph state code.

RAID Level 6 and Level 6+ Reliability

Storage systems are built of fallible components but have to provide high degrees of reliability. Besides mirroring and triplicating data, redundant storage of information using erasure-correcting codes is the only possibility to have data survive device failure. We provide here exact formula for the data-loss probability of a disk array composed of several RAID Level 6 stripes. This two-failure tolerant is not only used in practice but can also provide a reference point for the assessment of other data organizations

Slicing, Tokenization, and Encryption Based Combinational Approach to Protect Data-at-Rest in Cloud Using TF-Sec Model

Data in the Cloud, which applies to data as a cloud service provider (CSP), transmits stores, or manages it. The company will enforce the same definition of data usage while the data is resident within the enterprise and thus extend the required cryptographic security criteria to data collected, exchanged, or handled by CSP. The CSP Service Level Agreements cannot override the cryptographic access measures. When the data is transferred securely to CSP, it can be securely collected, distributed, and interpreted. Data at the rest position applies to data as it is processed internally in organized and in the unstructured ways like databases and file cabinets. The Data at the Rest example includes the use of cryptography for preserving the integrity of valuable data when processed. For cloud services, computing takes multiple forms from recording units, repositories, and many unstructured items. This paper presents a secure model for Data at rest. The TF-Sec model suggested is planned for use with Slicing, Tokenization, and Encryption. The model encrypts the given cloud data using AES 256 encryption, and then the encrypted block is sliced into the chunks of data fragments using HD-Slicer. Then it applies tokenization algorithm TKNZ to each chunk of data, applies erasure coding technique to tokens, applies the data dispersion technique to scramble encrypted data fragments, and allocates to storage nodes of the multiple CSP. In taking the above steps, this study aims to resolve the cloud security problems found and to guarantee the confidentiality of their data to cloud users due to encryption of data fragments would be of little benefit to a CSP.

Fully Private Coded Matrix Multiplication From Colluding Workers

Recently, coded computation has been used to reduce the completion time in distributed computing by mitigating straggler effects with erasure codes. As a variation of coded computation, fully private coded matrix multiplication (FPCMM) has been proposed to preserve a master’s privacy in a scenario where the master wants to obtain a matrix multiplication result from the libraries which are shared at the workers, while concealing both of the two indices of the desired matrices from each worker. In this letter, we propose a new FPCMM scheme to keep a master’s privacy from up to $T$ colluding workers. Furthermore, we compare the performance of our scheme with those of the existing private coded matrix multiplication schemes for non-colluding workers and for concealing the index of a single desired matrix.

Counterfactual universal quantum computation

Universal quantum computation is usually associated with interaction among two-level quantum subsystems, as this interaction is commonly viewed as a necessity to achieve universal quantum computation. In this work, we show that, contrary to this intuition, universal quantum computation can be achieved without interaction among initially independent two-level quantum subsystems. We call it counterfactual universal quantum computation. As special cases, we show how to achieve counterfactual communication of quantum states, counterfactual quantum swapping, and counterfactual quantum erasure codes. To ease practical implementation, we analyze counterfactual universal quantum computation with realistic devices, including the effects of finite execution time, photon loss, and atom missing. Besides the theoretical interest of illustrating the mysterious and counterintuitive nature of quantum physics, our work has practical applications to color imaging of ancient arts, upon which light is forbidden to shine.

Optimizing HARQ and Relay Strategies in Limited Feedback Communication Systems

One of the key challenges for future communication systems is to deal with fast changing channels due to the mobility of users. Having a robust protocol capable of handling transmission failures in unfavorable channel conditions is crucial, but the feedback capacity may be greatly limited due to strict latency requirements. This paper studies the hybrid automatic repeat request (HARQ) techniques involved in re-transmissions when decoding failures occur at the receiver and proposes a scheme that relies on codeword bundling and adaptive incremental redundancy (IR) to maximize the overall throughput in a limited feedback system. In addition to the traditional codeword extension IR bits, this paper introduces a new type of IR, bundle parity bits, obtained from an erasure code across all the codewords in a bundle. The type and number of IR bits to be sent as a response to a decoding failure is optimized through a Markov Decision Process. In addition to the single link analysis, the paper studies how the same techniques generalize to relay and multi-user broadcast systems. Simulation results show that the proposed schemes can provide a significant increase in throughput over traditional HARQ techniques.

A method of repairing single node failure in the distributed storage system based on the regenerating-code and a hybrid genetic algorithm

To ensure the reliability and security of the data, the large-scale distributed storage system usually adopts the data redundancy mechanism to repair the data on the faulty nodes. Comparing with the replication-based method, the data redundancy mechanism of erasure code can effectively improve the use of storage space, but it may result in the large network overhead when recovering the data. The regenerating code is an improved erasure code, which can reduce the quantity of data transmission compared to that of the erasure code. Adopting the regenerating code to repair the data on a faulty node requires constructing an optimal repair tree to maximum the bandwidth of the bottleneck link, which is an NP-hard problem. To construct the optimal repair tree, a hybrid genetic algorithm is proposed in this paper. In particular, our proposal comprehensively considers the network topology and link bandwidth of storage nodes and designs a problem-specific cross-correlation operator, mutation operator and local search operator. In addition, we provided the mathematical proof of the global convergence with probability one with respect to the proposed hybrid genetic algorithm. Through a series of simulation experiments, the results show that our proposal is able to determine the optimal repair tree, which effectively reduces the delay of faulty node repair in distributed storage systems, and improve the repairing efficiency.

Survey on Data Updating in Erasure-Coded Storage Systems

Survey on Data Updating in Erasure-Coded Storage Systems

CPU: Cross-Rack-Aware Pipelining Update for Erasure-Coded Storage

Erasure coding is widely used in distributed storage systems (DSSs) to efficiently achieve fault tolerance. However, when the original data need to be updated, erasure coding must update every encoded block, resulting in long update time and high bandwidth consumption. Exiting solutions are mainly focused on coding schemes to minimize the size of transmitted update information, while ignoring more efficient utilization of bandwidth among update racks. In this article, we propose a parallel Cross-rack Pipelining Update scheme ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CPU</i> ), which divides the update information into small-size units and transmits these units in parallel along with an update pipeline path among multiple racks. The performance of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CPU</i> is mainly determined by slice size and update path. More slices bring finer-grained parallel transmissions over cross-rack links, but also introduces more overheads. An update path that traverses all racks with large-bandwidth links provide short update time. We formulate the proposed pipelining update scheme as an optimization problem, based on a new theoretical pipelining update model. We prove the optimization problem is NP-hard and develop a heuristic algorithm to solve it based on the features of practical DSSs and our implementations, including <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Big chunk</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Small overhead</i> . Specifically, we determine the best update path first by solving a max-min problem and then decide the slice size. We further simplify the slice size selection by offline learning a range of interesting ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">RoI</i> ), in which all slice sizes provide similar performance. We implement <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CPU</i> and conduct experiments on Amazon EC2 under a variety of scenarios. The results show that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CPU</i> can reduce the average update time by 48.2 percent, compared with the state-of-the-art update schemes.

Reliable verification of distributed encoded data fragments in the cloud

Cloud storage services allow users to remotely store their data in a distributed environment and enjoy the cloud applications ubiquitously. To maximize users’ trust, it also integrates a verification mechanism that guarantees the stored data’s correctness. The storage application fragments the user data and stores them on multiple cloud storage servers. However, it suffers from expensive data aggregation computations while processing verification services, and inevitably poses a data integrity verification challenge. To avoid these expensive computations, we simplify the verification procedure without needing the data aggregation, just by storing the evidence fragments and data fragments across the datacenters. In distributed environments, the storage correctness verification mechanism depends on the availability of storage servers. Therefore, the challenge of proof/evidence availability may arise due to a server failure or data corruption, hence, decreasing the reliability of storage correctness verification. Thus, the problem of proof reliability is introduced over the distributed data. A few techniques proposed in the literature provide the data reliability; however, none of these existing works have considered the proof reliability to the best of our knowledge. To address the new issue of proof reliability, in this paper, we utilize and leverage the Erasure Coding (EC) to propose a reliable storage correctness verification solution that guarantees the retrieval of evidence and minimizes the effect of server failure/unavailability. The experimental results demonstrate that the proposed approach achieves reliability even after the loss of a certain number of fragments, ranging between 2 and 12 depending upon the number of parity fragments used in the EC scheme. Extensive experiments are performed in real-time, and results show that our proposed solution is highly efficient than well-known state-of-the-art verification schemes.

Treeplication: An Erasure Code for Distributed Full Recovery Under the Random Multiset Channel

This paper presents a new erasure code called Treeplication designed for distributed recovery of the full information word, while most prior work in coding for distributed storage only supports distributed repair of individual symbols. A Treeplication code for $k$ information symbols is defined on a binary tree with $2k-1$ vertices, along with a distribution for selecting code symbols from the tree layers. We analyze and optimize the code under a random-multiset model, which captures the system property that the nodes available for recovery are drawn randomly from the nodes storing the code symbols. Treeplication codes are shown to have full-recovery communication-cost comparable to replication, while offering much better recoverability.