Erasure Codes Research Articles

PyRS: Cross-platform Data Fault-tolerant Storage Library Based on RS Erasure Code

<p>Erasure code has been used by more and more researchers to solve the problem of efficient, reliable, and fault-tolerant data storage. However, the existing libraries based on erase code can only run on the Linux platform, and some of them need GPU support. This paper implements a cross-platform data fault-tolerant storage library based on RS erasure code, PyRS, which is running on CPU without GPU support and developed in Python. PyRS uses Vandermonde matrix as the coding matrix and Numba and NumPy libraries to speed up and optimize the program. This paper compares PyRS with Jerasure on the same Linux platform. The results show that the encoding and decoding speed of PyRS is 8 times faster than that of Jerasure. The CPU usage rate of both is about 25% and the memory usage rate of PyRS is about 5% higher than that of Jerasure. The same experiments are carried out on PyRS on the Windows platform. Experimental results show that compared with running on the Linux platform, PyRS running on the Windows platform has almost the same speed of encoding and decoding, and its CPU usage rate increases by about 15%, while its memory usage rate decreases by about 5%.</p> <p> </p>

Fast Proactive Repair in Erasure-Coded Storage: Analysis, Design, and Implementation

Erasure coding offers a storage-efficient redundancy mechanism for maintaining data availability guarantees in large-scale storage clusters, yet it also incurs high performance overhead in failure repair. Recent developments in accurate disk failure prediction allow soon-to-fail (STF) nodes to be repaired in advance, thereby opening new opportunities for accelerating failure repair in erasure-coded storage. To this end, we present a fast proactive repair solution called FastPR, which carefully couples two repair methods, namely migration (i.e., relocating the chunks of an STF node) and reconstruction (i.e., decoding the chunks of an STF node through erasure coding), so as to fully parallelize the repair operation across the storage cluster. FastPR solves a bipartite maximum matching problem and schedules both migration and reconstruction in a parallel fashion. We show that FastPR significantly reduces the repair time over the baseline repair approaches for both Reed-Solomon codes and Azures Local Reconstruction Codes via mathematical analysis, large-scale simulation, and Amazon EC2 experiments.

Stabilizing Error-Correction Codes for Control Over Erasure Channels

In this article, we propose <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$(k,k^{\prime })$</tex-math></inline-formula> stabilizing codes, which are a type of delayless error-correction codes that are useful for control over networks with erasures. For each input symbol, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> output symbols are generated by the stabilizing code. Receiving at least <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k^{\prime }$</tex-math></inline-formula> of these outputs guarantees stability. Thus, both the system to be stabilized and the channel are taken into account in the design of the erasure codes. Receiving more than <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k^{\prime }$</tex-math></inline-formula> outputs further improves the performance of the system. In the case of i.i.d. erasures, we further demonstrate that the erasure code can be constructed such that stability is achieved if on average at least <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k^{\prime }$</tex-math></inline-formula> output symbols are received. Our focus is on linear and time-invariant systems, and we construct codes based on independent encodings and multiple descriptions. Stability is assessed via Markov jump linear system theory. The theoretical efficiency and performance of the codes are assessed, and their practical performances are demonstrated in a simulation study. There is a significant gain over other delayless codes such as repetition codes.

Bandwidth-Aware Scheduling Repair Techniques in Erasure-Coded Clusters: Design and Analysis

Erasure codes offer a storage-efficient redundancy mechanism for maintaining data availability guarantees in storage clusters, yet also incur high network traffic consumption and recovery time in failure repair. Extensive research has been carried out to reduce the recovery time. However, previous works either target specific erasure code constructions which are not commonly used in today’s distributed storage clusters or neglect the heterogeneous bandwidth property in real network environments. Since erasure-coded clusters are typically composed of multi-node with heterogeneous bandwidth and accessed in parallel, the whole recovery time is mainly restricted by the low-bandwidth links. In this article, we propose SMFRepair, a single-node multi-level forwarding repair technique that is designed to improve the performance in heterogeneous networks based on Reed-Solomon codes for general fault tolerance. SMFRepair carefully selects the helper nodes and uses idle nodes to bypass low-bandwidth links. Idle nodes have sufficient and unused network bandwidth. It also pipelines the repair links that are optimized by idle nodes. Furthermore, a multi-node scheduling repair technique, called MSRepair, is proposed. MSRepair carefully schedules the multi-node repair link to saturate the most unoccupied bandwidth and transfers data from as large-bandwidth links as possible, with the primary objective of minimizing the recovery time. Large-scale simulation and Amazon EC2 real experiments show that compared to state-of-the-art repair techniques, SMFRepair can accelerate the single-node recovery by up to 47.69%, and MSRepair can reduce the multi-node recovery time by 33.78% <inline-formula><tex-math notation="LaTeX">$\sim$</tex-math></inline-formula> 67.53%.

A Novel Completely Local Repairable Code Algorithm Based on Erasure Code

A Novel Completely Local Repairable Code Algorithm Based on Erasure Code

SelectiveEC: Towards Balanced Recovery Load on Erasure-Coded Storage Systems

Erasure coding (EC) has been commonly used to offer high data reliability with low storage cost. Upon failures, the lost blocks are recovered in batches. Due to the limited number of stripes, the data layout within a batch is non-uniform. Together with the random selection of source and replacement nodes for recovery tasks, the recovery workload among live nodes is skewed within a batch, which severely slows down failure recovery. To solve this problem, We present SelectiveEC, a new recovery task scheduling module that provides provable network traffic and recovery load balancing for large-scale EC-based storage systems. It relies on bipartite graphs to model the recovery traffic among live nodes. Then, it intelligently selects tasks to form batches and carefully determines where to read source blocks or to store recovered ones, using theories such as a perfect or maximum matching and <inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> -regular spanning subgraph. SelectiveEC supports single-node failure and multi-node failure recovery, and can be deployed in both homogeneous and heterogeneous network environments. We implement SelectiveEC in HDFS, and evaluate its recovery performance in a local cluster of 18 nodes and AWS EC2 of 50 virtual machine instances. SelectiveEC increases the recovery throughput by up to <inline-formula><tex-math notation="LaTeX">$30.68\%$</tex-math></inline-formula> compared with state-of-the-art baselines in homogeneous network environments. It further achieves <inline-formula><tex-math notation="LaTeX">$1.32\times$</tex-math></inline-formula> recovery throughput and <inline-formula><tex-math notation="LaTeX">$1.23\times$</tex-math></inline-formula> benchmark throughput of HDFS on average in heterogeneous network environments, due to the straggler avoidance by the balanced scheduling.

Corrections to “Optimal Streaming Erasure Codes Over the Three-Node Relay Network”

In the above article <xref ref-type="bibr" rid="ref1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">[1]</xref> , an upper bound on the maximum achievable rate is corrected. If we add the restriction that the packets transmitted by the relay must be independent of the erasures introduced by the first-hop channel, then no correction is needed and the converse proof need not be rectified.

Static multi-sourced data retrieval in elastic optical networks

Extreme-scale science applications are highly innovative and constantly evolving. They are expected to generate data in the petabyte and exabyte ranges. Erasure coding has been widely adopted for data storage in data center networks, where the data are encoded and stored in multiple locations. Therefore, an efficient data retrieval service is needed to transfer encoded data from selected multiple stored nodes to a single destination. Elastic optical networks are a promising backbone technology for data center communication due to their capability to efficiently and flexibly allocate the huge optical bandwidth to heterogeneous traffic demands. In this paper, the erasure-coded multi-sourced data retrieval routing and scheduling problem is studied for static traffic in elastic optical networks, and the objective is to minimize the total transmission completion time of all the requests. An integer linear programming formulation and low-complexity heuristic are proposed. Furthermore, analytical lower bounds are derived and a meta-heuristic, Tabu Search, is adopted to solve the problem. Numerical results are presented to show the effectiveness of the proposed methods.

Packet transport for maritime communications: A streaming coded UDP approach

The maritime communication network (MCN) plays an important role in the 6th generation (6G) system development. In MCNs, packet transport over long-distance lossy links will be ubiquitous. Transmission control protocol (TCP), the dominant transport protocol in the past decades, have had performance issues in such links. In this paper, we propose a novel transport approach which uses user datagram protocol (UDP) along with a simple yet effective bandwidth estimator for congestion control, and with a proactive packet-level forward erasure correction (FEC) code called streaming code to provide low-delay loss recovery without data retransmissions at all. We show that the approach can effectively address two issues of the state-of-the-art TCP variants in the long-distance lossy links, namely 1) the low bandwidth utilization caused by the slow increase of the congestion window (CWND) due to long round-trip time (RTT) and the frequent CWND drop due to random and congestion losses, and 2) the high end-to-end in-order delivery delay when re-transmissions are incurred to recover lost packets. In addition, we show that the scheme's goodput has good smoothness and short-term intra-protocol fairness properties, which are beneficial for multimedia streaming and interactive applications that are prominent parts of today's wireless traffic.

Addendum and Erratum to “The MDS Queue: Analysing the Latency Performance of Erasure Codes”

In the above article <xref ref-type="bibr" rid="ref1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">[1]</xref> , we introduced two scheduling policies and analyzed their average job latencies. With an implicit assumption that the scheduling policies provide sample-path bounds by construction, we claimed that their average job latencies serve as upper and lower bounds on that of a centralized MDS queue. In this note, we present recently discovered counterexamples, disproving the assumption. We replace the assumption with a conjecture that the average latency bounds still hold. We also provide an erratum to the original article to correct any confusing or misleading statements.

Hybrid Approach for Improving the Performance of Data Reliability in Cloud Storage Management.

The digital transformation disrupts the various professional domains in different ways, though one aspect is common: the unified platform known as cloud computing. Corporate solutions, IoT systems, analytics, business intelligence, and numerous tools, solutions and systems use cloud computing as a global platform. The migrations to the cloud are increasing, causing it to face new challenges and complexities. One of the essential segments is related to data storage. Data storage on the cloud is neither simplistic nor conventional; rather, it is becoming more and more complex due to the versatility and volume of data. The inspiration of this research is based on the development of a framework that can provide a comprehensive solution for cloud computing storage in terms of replication, and instead of using formal recovery channels, erasure coding has been proposed for this framework, which in the past proved itself as a trustworthy mechanism for the job. The proposed framework provides a hybrid approach to combine the benefits of replication and erasure coding to attain the optimal solution for storage, specifically focused on reliability and recovery. Learning and training mechanisms were developed to provide dynamic structure building in the future and test the data model. RAID architecture is used to formulate different configurations for the experiments. RAID-1 to RAID-6 are divided into two groups, with RAID-1 to 4 in the first group while RAID-5 and 6 are in the second group, further categorized based on FTT, parity, failure range and capacity. Reliability and recovery are evaluated on the rest of the data on the server side, and for the data in transit at the virtual level. The overall results show the significant impact of the proposed hybrid framework on cloud storage performance. RAID-6c at the server side came out as the best configuration for optimal performance. The mirroring for replication using RAID-6 and erasure coding for recovery work in complete coherence provide good results for the current framework while highlighting the interesting and challenging paths for future research

DualDup: A secure and reliable cloud storage framework to deduplicate the encrypted data and key

Cloud Storage Providers generally maintain a single copy of the identical data received from multiple sources to optimize the space. They cannot deduplicate the identical data when the clients upload the data in the encrypted form. To address this problem, recently, Duplicateless Encryption for Simple Storage (DupLESS) scheme is introduced in the literature. Besides, the data stored in the cloud is unreliable due to the possibility of data losses in remote storage environments. The DupLESS scheme, on the other hand, keeps both the key and the data on a single storage server, which is unreliable if that server goes down. In essence, the existing related works aim to handle either secure-deduplication or reliability limited to either key reliability or the data reliability. Hence, there is a need to develop a secure-deduplication mechanism that is not vulnerable to any malicious activity, semantically secures both data and key, and achieves the reliability. To address these problems, this paper proposes the dualDup framework that (a) optimizes the storage by eliminating the duplicate encrypted data from multiple users by extending DupLESS concept, and (b) securely distributes the data and key fragments to achieve the privacy and reliability using Erasure Coding scheme. The proposed approach is implemented in Python on the top of the Dropbox datacenter and corresponding results are reported. Experiments are conducted in a realistic environment. The results demonstrate that the proposed framework achieves reliability with an average storage overhead of 66.66% corresponding to the Reed–Solomon(3,2) codes. We validated through security analysis that the proposed framework is secure from insider and outsider adversaries. Moreover, dualDup framework provides all the aspects of deduplication, attack mitigation, key security and management, reliability, and QoS features as compared to other state-of-the-art deduplication techniques.

Inspecting Cloud Storage System for Secure Data Forwarding Using Advanced Encryption Standard

Abstract: Data storage and sharing services within the cloud, users can easily modify and share data as a bunch. to make sure share data integrity are often verified publicly, users within the group must compute signatures on all the blocks in shared data. Different blocks in shared data are generally signed by different users thanks to data modifications performed by different users. Encryption schemes available for data privacy but it limits the number of functions exhausted storage system. Building a secure storage system that supports multiple functions is hard when the storage system is distributed and has no central authority. a brand-new idea is proposed proxy re-encryption scheme for decentralizes erasure code for defending the distributed system. The easy method, which allows an existing user to download the corresponding a part of shared data, is inefficient because of the big size of shared data within the cloud. We propose a unique public auditing mechanism. Moreover, our mechanism is in a position to support batch auditing by verifying multiple auditing tasks simultaneously. Key Words: Cloud computing, Cloud storage, AES, Cryptography Upload, Auditing

H2RUN: An efficient vendor lock‐in solution for multi‐cloud environment using horse herd Runge Kutta based data placement optimization

AbstractVendor lock‐in is a scenario where the cost of switching to another vendor is mainly high and for this reason, the consumer is mainly struck with a single cloud vendor. The vendor lock‐in mainly implies the situation of an organization where it is locked into a service or product of a single Cloud Service Providers (CSP) to prevent the disruption of the business operation or due to an insufficient workforce. Due to its high availability, portability, and other benefits, various researchers frequently adopt the multi‐cloud strategy to prevent vendor lock‐in. However, the multi‐cloud architecture also comes with major drawbacks such as cost, multiple CSP management, and the tradeoff between cost and availability. The organizations are mainly considerate about achieving the tradeoff between the cost and availability. To overcome this problem, we have proposed an H2RUN (Horse Herd Runge Kutta) optimization algorithm to achieve a tradeoff between the cost and availability using erasure coding. The proposed H2RUN optimization algorithm selects the optimal CSP and erasure coding parameters that enhance the availability of the CSP with minimal cost. The efficiency of the proposed methodology can be evaluated using the experiments conducted on different real‐time CSPs using various metrics. Based on the results achieved, the H2RUN optimizer minimizes the cost up to 48% and availability up to 58%.

A Novel Repairable Fountain Codes Scheme Based on Partial Duplication Technique

Distributed storage system (DSS) is an emerging paradigm which provides reliable storage services for various source data. As the fault-tolerance scheme for DSS, erasure codes are required to provide redundancy service with high fault-tolerance and low cost.However, the existing coding scheme cannot provide these requirements well. Thus, it becomes an important yet challenging issue to find a code for storing various source data with high fault-tolerance and low cost. In this paper, a novel construction of repairable fountain codes with unequal locality is proposed by combining with partial duplication technique, namely the PD-ULRFC scheme. We construct a multi-tier heterogeneous storage network, where data core, processing units and storage nodes collaboratively store and transmit data. Moreover, the proposed PD-ULRFC scheme can reduce the repair and download cost by sacrificing a little extra storage occupation. Furthermore, the expressions of the repair cost and download cost are derived to analyze the performance of PD-ULRFC scheme. The simulation results demonstrate that the PD-ULRFC scheme significantly outperforms other redundant schemes in communication cost saving.

Optimal Caching for Low Latency in Distributed Coded Storage Systems

Erasure codes have been widely considered as a promising solution to enhance data reliability at low storage costs. However, in modern geo-distributed storage systems, erasure codes may incur high data access latency as they require data retrieval from multiple remote storage nodes. This hinders the extensive application of erasure codes to data-intensive applications. This paper proposes novel caching schemes to achieve low latency in distributed coded storage systems. Assuming that future data popularity and network latency information are available, an offline caching scheme is proposed to explore the optimal caching solution for low latency. The proposed scheme categorizes all feasible caching decisions into a set of cache partitions, and then obtains the optimal caching decision through market clearing price for each cache partition. Furthermore, guided by the optimal scheme, an online caching scheme is proposed according to the measured data popularity and network latency information in real time, without the need to completely override the existing caching decisions. Both theoretical analysis and experiment results demonstrate that the online scheme can approximate the offline optimal scheme well with dramatically reduced computation complexity.

LEGOStore

We design and implement LEGOStore, an erasure coding (EC) based linearizable data store over geo-distributed public cloud data centers (DCs). For such a data store, the confluence of the following factors opens up opportunities for EC to be latency-competitive with replication: (a) the necessity of communicating with remote DCs to tolerate entire DC failures and implement linearizability; and (b) the emergence of DCs near most large population centers. LEGOStore employs an optimization framework that, for a given object, carefully chooses among replication and EC, as well as among various DC placements to minimize overall costs. To handle workload dynamism, LEGOStore employs a novel agile reconfiguration protocol. Our evaluation using a LEGOStore prototype spanning 9 Google Cloud Platform DCs demonstrates the efficacy of our ideas. We observe cost savings ranging from moderate (5-20%) to significant (60%) over baselines representing the state of the art while meeting tail latency SLOs. Our reconfiguration protocol is able to transition key placements in 3 to 4 inter-DC RTTs (&lt; 1s in our experiments), allowing for agile adaptation to dynamic conditions.

Efficient Hardware Architectures and Implementations of Packet-Level Erasure Coding Schemes for High Data Rate Reliable Satellite Communications

Forward error correction coding schemes have been used extensively in order to provide continuous and reliable telemetry data transfer from satellites to ground stations and guarantee reliable communications even at low signal-to-noise ratio regimes, where burst errors are dominant. Nevertheless, conventional bit-level channel coding can fail to provide downlink reliability against long error bursts in future high data rate radio frequency and optical (laser) links, especially when automatic repeat queuing strategies are either problematic or impossible due to specific service delay constraints. Packet-level erasure coding has been considered by the Consultative Committee for Space Data Systems (CCSDS) in CCSDS 131.5-O-1 experimental specification for application in high data rate near-earth and deep-space communications, since it can protect against long error bursts as they may come along with the effect of scintillation outages or transmission errors. Implementations of packet-level encoding and decoding so far exist only in software, running on a general-purpose CPU. In this article, we introduce and compare analytical descriptions of packet-level encoding algorithms suitable for hardware implementations. Based on these algorithmic descriptions, we introduce, for the first time, architectures for hardware acceleration of these functions that allow integration on a high-speed on-board data-processing chain with very low footprint. We validate our hardware implementations and demonstrate the efficiency of the proposed architectures on a Xilinx KCU105 development board, which is built around the commercial equivalent of a space-grade field-programmable gate array device. Apart from offloading packet-level encoding from the on-board embedded processor, the proposed accelerators achieve a significant speedup (over 80 times), when compared with on-board software implementations, by porting on some of the most commonly used and state-of-the-art space-qualified embedded processors.

A New Method for Reconstructing Data Considering the Factor of Selected Provider Nodes Set in Distributed Storage System

In the distributed storage system, when data need to be recovered after node failure, the erasure code redundancy method occupies less storage space than the multi-copy method. At present, the repair mechanism using erasure code to reconstruct the failed node only considers the improvement of link bandwidth on the repair rate and does not consider the impact of the selection of data providing node-set on the repair performance. A single node fault data reconstruction method based on the Software Defined Network (SDN) using the erasure code method is designed to solve the above problems. This method collects the network link-state through SDN, establishes a multi-attribute decision-making model of the data providing node-set based on the node performance, and determines the data providing nodes participating in providing data through the ideal point method. Then, the data recovery problem of a single fault node is modeled as the optimization problem of an optimal repair tree, and a hybrid genetic algorithm is designed to solve it. The experimental results show that under the same erasure code scale, after selecting the nodes of the data providing node-set, compared with the traditional tree topology and star topology, the repair delay distribution of the designed single fault node repair method for a distributed storage system is reduced by 15% and 45% respectively, and the repair flow is close to the star topology, which is reduced by 40% compared with the traditional tree repair.

A Load-Aware Multistripe Concurrent Update Scheme in Erasure-Coded Storage System

Erasure coding has been widely deployed in today’s data centers for it can significantly reduce extra storage costs while providing high storage reliability. However, erasure coding introduced more network traffic and computational overhead in the data update process. How to improve the efficiency and mitigate the system imbalance during the update process in erasure coding is still a challenging problem. Recently, most of the existing update schemes of erasure codes only focused on the single stripe update scenario and ignored the heterogeneity of the node and network status which cannot sufficiently deal with the problems of low update efficiency and load imbalance caused by the multistripe concurrent update. To solve this problem, this paper proposes a Load-Aware Multistripe concurrent Update (LAMU) scheme in erasure-coded storage systems. Notably, LAMU introduces the Software-Defined Network (SDN) mechanism to measure the node loads and network status in real time. It selects nonduplicated nodes with better performance such as CPU utilization, remaining memory, and I/O load as the computing nodes for multiple update stripes. Then, a multiattribute decision-making method is used to schedule the network traffic generated in the update process. This mechanism can improve the transmission efficiency of update traffic and make LAMU adapt to the multistripe concurrent update scenarios in heterogeneous network environments. Finally, we designed a prototype system of multistripe concurrent updates. The extensive experimental results show that LAMU could improve the update efficiency and provide better system load-balancing performance.