Regenerating Codes Research Articles

Bandwidth Adaptive & Error Resilient MBR Exact Repair Regenerating Codes

Regenerating codes are efficient methods for distributed storage in storage networks, where node failures are common. They guarantee low cost data reconstruction and repair through accessing only a predefined number of arbitrarily chosen storage nodes in the network. In this paper, we consider two simultaneous extensions to the original regenerating codes framework introduced by Dimakis et al. ; 1) both data reconstruction and repair are resilient to the presence of a certain number of erroneous nodes in the network and 2) the number of helper nodes in every repair is not fixed, but is a flexible parameter that can be selected during the run-time. We study the fundamental limits of required total repair bandwidth and provide an upper bound for the storage capacity of these codes under these assumptions. We then focus on the minimum repair bandwidth (MBR) case and derive the exact storage capacity by presenting explicit coding schemes with exact repair, which achieve the upper bound of the storage capacity in the considered setup. To this end, we first provide a more natural extension of the well-known product matrix (PM) MBR codes, modified to provide flexibility in choosing the number of helpers in each repair, and simultaneously be robust to erroneous nodes in the network. This is achieved by proving the non-singularity for a family of matrices in large enough finite fields. We next provide another extension of the PM codes, based on a novel repair scheme which enables flexibility in the number of helpers and robustness against erroneous nodes without any extra cost in field size compared with the original PM codes.

Improved Upper Bounds on Systematic-Length for Linear Minimum Storage Regenerating Codes

In this paper, we revisit the problem of finding the longest systematic-length k for a linear minimum storage regenerating (MSR) code with optimal repair of only systematic part, for a given per-node storage capacity l and an arbitrary number of parity nodes r. We study the problem by following a geometric analysis of linear subspaces and operators. First, a simple quadratic bound is given, which implies that k = r + 2 is the largest number of systematic nodes in the scalar scenario. Second, an r-based-log bound is derived, which is superior to the upper bound on log-base 2 in the prior work. Finally, an explicit upper bound depending on the value of r <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> /l is introduced, which further extends the corresponding result in the literature.

HashTag Erasure Codes: From Theory to Practice

Minimum-Storage Regenerating (MSR) codes have emerged as a viable alternative to Reed-Solomon (RS) codes as they minimize the repair bandwidth while they are still optimal in terms of reliability and storage overhead. Although several MSR constructions exist, so far they have not been practically implemented mainly due to the big number of I/O operations. In this paper, we analyze high-rate MDS codes that are simultaneously optimized in terms of storage, reliability, I/O operations, and repair-bandwidth for single and multiple failures of the systematic nodes. The codes were recently introduced in [1] without any specific name. Due to the resemblance between the hashtag sign # and the procedure of the code construction, we call them in this paper HashTag Erasure Codes (HTECs). HTECs provide the lowest data-read and data-transfer, and thus the lowest repair time for an arbitrary sub-packetization level α, where α ≤ r <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">⌈k/r⌉</sup> , among all existing MDS codes for distributed storage including MSR codes. The repair process is linear and highly parallel. Additionally, we show that HTECs are the first high-rate MDS codes that reduce the repair bandwidth for more than one failure. Practical implementations of HTECs in Hadoop release 3.0.0-alpha2 demonstrate their great potentials.

CRL: Efficient Concurrent Regeneration Codes with Local Reconstruction in Geo-Distributed Storage Systems

As a typical erasure coding choice, Reed-Solomon (RS) codes have such high repair cost that there is a penalty for high reliability and storage efficiency, thereby they are not suitable in geo-distributed storage systems. We present a novel family of concurrent regeneration codes with local reconstruction (CRL) in this paper. The CRL codes enjoy three benefits. Firstly, they are able to minimize the network bandwidth for node repair. Secondly, they can reduce the number of accessed nodes by calculating parities from a subset of data chunks and using an implied parity chunk. Thirdly, they are faster than existing erasure codes for reconstruction in geo-distributed storage systems. In addition, we demonstrate how the CRL codes overcome the limitations of the Reed-Solomon codes. We also illustrate analytically that they are excellent in the trade-off between chunk locality and minimum distance. Furthermore, we present theoretical analysis including latency analysis and reliability analysis for the CRL codes. By using quantity comparisons, we prove that CRL(6, 2, 2) is only 0.657x of Azure LRC(6, 2, 2), where there are six data chunks, two global parities, and two local parities, and CRL(10, 4, 2) is only 0.656x of HDFS-Xorbas(10, 4, 2), where there are 10 data chunks, four local parities, and two global parities respectively, in terms of data reconstruction times. Our experimental results show the performance of CRL by conducting performance evaluations in both two kinds of environments: 1) it is at least 57.25% and 66.85% more than its competitors in terms of encoding and decoding throughputs in memory, and 2) it has at least 1.46x and 1.21x higher encoding and decoding throughputs than its competitors in JBOD (Just a Bunch Of Disks). We also illustrate that CRL is 28.79% and 30.19% more than LRC on encoding and decoding throughputs in a geo-distributed environment.

Locally Repairable Regenerating Codes: Node Unavailability and the Insufficiency of Stationary Local Repair

Recent works by Ahmad et al. and by Hollmann studied the concept of “locally repairable regenerating codes (LRRCs)” that successfully combines the functional repair and partial information exchange of regenerating codes (RCs) with the much-desired local repairability feature of locally repairable codes (LRCs). One important issue that needs to be addressed by any local repair schemes (including both LRCs and LRRCs) is that sometimes designated helper nodes may be temporarily unavailable, the result of various reasons that include multiple failures, degraded reads, or power-saving strategies to name a few. Under the setting of LRRCs with temporary node unavailability, this paper studies the impact of different helper selection methods. It proves that with node unavailability, all existing methods of helper selection, including those used in RCs and LRCs, can be insufficient in terms of achieving the optimal repair-bandwidth. For some scenarios, it is necessary to combine LRRCs with a new class of helper selection methods, termed dynamic helper selection , to achieve optimal repair-bandwidth. This paper also compares the performance of different classes of helper selection methods and answers the following fundamental question: is one method of helper selection intrinsically better than the other? for various scenarios.

Proxy-Assisted Regenerating Codes With Uncoded Repair for Distributed Storage Systems

Distributed storage systems can store data with erasure coding to maintain data availability with low storage redundancy. One class of erasure coding is based on regenerating codes , which provably minimize the amount of data transferred for failure repair and realize the optimal tradeoff between the storage redundancy and the amount of traffic transferred for repair. Typical regenerating codes often require surviving storage nodes to encode their stored data for repair. In this paper, we study a framework called proxy-assisted regeneration, which offloads the repair process to a centralized proxy. We extend the previous applied work on proxy-assisted regeneration by providing theoretical validation. Specifically, we study a special class of regenerating codes called proxy-assisted minimum storage regenerating (PMSR) codes, which enable uncoded repair without the need of encoding in surviving nodes, while preserving the minimum storage redundancy and minimum amount of traffic transferred for repair. We formally prove the existence of PMSR codes for two configurations: 1) repairing single-node failures under double fault tolerance and 2) repairing double-node failures under triple fault tolerance. We also provide a semi-deterministic PMSR code construction for repairing single-node failures under double fault tolerance.

Product Matrix MSR Codes With Bandwidth Adaptive Exact Repair

In the distributed storage systems (DSSs) with $k$ systematic nodes, robustness against node failure is commonly provided by storing redundancy in a number of other nodes and performing repair mechanism to reproduce the content of the failed nodes. Efficiency is then achieved by minimizing the storage overhead and the amount of data transmission required for data reconstruction and repair, provided by coding solutions, such as regenerating codes. Common explicit regenerating code constructions enable efficient repair through accessing a predefined number, $d$ , of arbitrary chosen available nodes, namely helpers. In practice, however, the state of the system dynamically changes based on the request load, the link traffic, and so on, and the parameters which optimize system’s performance vary accordingly. It is then desirable to have coding schemes which are able to operate optimally under a range of different parameters simultaneously. Specifically, adaptivity in the number of helper nodes for repair is of interest. While robustness requires capability of performing repair with small number of helpers, it is desirable to use as many helpers as available to reduce the transmission delay and total repair traffic. In this paper, we focus on the minimum storage regenerating (MSR) codes, where each of the $n$ nodes in the network is supposed to store $\alpha $ information units, and the source data of size $k\alpha $ could be recovered from any arbitrary set of $k$ nodes. We introduce a class of MSR codes that realize the optimal repair bandwidth simultaneously with a set of different choices for the number of helpers, namely $D=\{d_{1}, \ldots , d_{\delta }\}$ . Our coding scheme follows the product matrix (PM) framework introduced by Rashmi et al. and could be considered as a generalization of the PM MSR code presented by Rashmi et al. , such that any $d_{i} = (i+1)(k-1)$ helpers can perform an optimal repair. As a result, the coding rate in our construction is limited by $({k}/{n})\leq ({1}/{2})$ . However, similar to the original design of PM MSR codes, our solution can realize practical values of the parameter $\alpha $ . Recently, Ye and Barg have presented another explicit MSR coding scheme which is capable of performing optimal repair for various number of helpers. The solution presented by Ye and Barg works for any arbitrary set of parameters $k$ and $D$ and can achieve high-coding rates, but the required $\alpha $ for this code is exponentially large. We show that the required value for $\alpha $ in the coding scheme presented in this paper is exponentially smaller when compared with the work of Ye and Barg for the same set of other parameters. Particularly, for a DSS with $n$ nodes and $k$ systematic nodes, the required value for $\alpha $ is reduced from $s^{n}$ to $sk$ , where $s=\mathrm {l~cm}(d_{1}-k+1,\cdots ,d_{\delta }-k+1)$ . We also show the required field size in the presented coding scheme is equal to $n$ .

When Can Intelligent Helper Node Selection Improve the Performance of Distributed Storage Networks?

The concept of distributed storage networks (DSNs) mostly follows two main modeling assumptions: Any $k$ out of $n$ surviving nodes should be able to reconstruct the protected file; and if one node fails, the replacement node can access $d$ helper nodes to repair its content either functionally or exactly. Two major existing approaches for DSNs are the so-called regenerating codes (RCs) and locally repairable codes (LRCs), which have different design philosophies and focus on distinct applications. Instead of being limited by the framework of either RCs or LRCs, this work answers a fundamental question for general DSNs: For an arbitrarily given $(n,k,d)$ value, whether there exists an intelligent helper node selection design that can strictly improve the storage-bandwidth tradeoff when compared with naive blind helper selection. Surprisingly, the answer is negative for a large set of $(n,k,d)$ values. Namely, for those $(n,k,d)$ values even the best helper selection design offers no gain over a blind solution. We call those $(n,k,d)$ values indifferent-to-helper-selection (ITHS). The main contribution of this work is a necessary and sufficient condition that characterizes whether an $(n,k,d)$ value is ITHS. As a fundamental study, this work assumes functional repair with unlimited computing power for encoding/decoding and focuses on the fundamental performance limits of intelligent helper selection. A new helper selection scheme, termed family helper selection , is proposed and used in the achievability analysis. For some scenarios, the proposed scheme is indeed optimal (as good as any helper selection one can design).

Locally minimum storage regenerating codes in distributed cloud storage systems

In distributed cloud storage systems, inevitably there exist multiple node failures at the same time. The existing methods of regenerating codes, including minimum storage regenerating (MSR) codes and minimum bandwidth regenerating (MBR) codes, are mainly to repair one single or several failed nodes, unable to meet the repair need of distributed cloud storage systems. In this paper, we present locally minimum storage regenerating (LMSR) codes to recover multiple failed nodes at the same time. Specifically, the nodes in distributed cloud storage systems are divided into multiple local groups, and in each local group (4, 2) or (5, 3) MSR codes are constructed. Moreover, the grouping method of storage nodes and the repairing process of failed nodes in local groups are studied. Theoretical analysis shows that LMSR codes can achieve the same storage overhead as MSR codes. Furthermore, we verify by means of simulation that, compared with MSR codes, LMSR codes can reduce the repair bandwidth and disk I/O overhead effectively.

An Outer Bound on the Storage-Bandwidth Tradeoff of Exact-Repair Cooperative Regenerating Codes

Cooperative regenerating codes are a kind of erasure codes, which are optimal in terms of minimizing the repair bandwidth. An $(n,k,d,r)$ -cooperative regenerating code has $n$ storage nodes, where $k$ arbitrary nodes are enough to reconstruct original data, and $r$ failed nodes can be repaired cooperatively with the help of $d$ arbitrary surviving nodes. In the regenerating-code framework, there exists a tradeoff between the storage capacity of each node $\alpha $ and the repair bandwidth $\gamma $ , but the problem of specifying the optimal storage-bandwidth tradeoff of the exact-repair cooperative regenerating codes remains open. A key contribution of this paper is that an outer bound on the storage-bandwidth tradeoff of exact-repair linear cooperative regenerating codes is proposed. This result can be regarded as a generalization of the outer bound proposed by Prakash et al. , which specifies the optimal tradeoff of exact-repair regenerating codes for the case of $d=k=n-1$ . The proposed outer bound suggests the $(\alpha ,\gamma )$ pairs that no exact-repair codes can achieve but only functional-repair codes can. By observing the size of the set of such $(\alpha ,\gamma )$ pairs, the performance of the proposed outer bound is evaluated under various parameter settings.

Minimum Storage Regenerating Codes for All Parameters

Regenerating codes for distributed storage have attracted much research interest in the past decade. Such codes trade the bandwidth needed to repair a failed node with the overall amount of data stored in the network. Minimum storage regenerating (MSR) codes are an important class of optimal regenerating codes that minimize (first) the amount of data stored per node and (then) the repair bandwidth. Specifically, an $[n,k,d]$ - $(\alpha )$ MSR code $ \mathbb {C}$ over $ \smash {\mathbb {F}_{\!q}}$ stores a file $ {\mathcal{ F}}$ consisting of $\alpha k$ symbols over $ \smash {\mathbb {F}_{\!q}}$ among $n$ nodes, each storing $\alpha $ symbols, in such a way that: 1) the file $ {\mathcal{ F}}$ can be recovered by downloading the content of any $k$ of the $n$ nodes and 2) the content of any failed node can be reconstructed by accessing any $d$ of the remaining $n-1$ nodes and downloading $\alpha /(d{-}k{+}1)$ symbols from each of these nodes. In practice, the file $ {\mathcal{ F}}$ is typically available in uncoded form on some $k$ of the $n$ nodes, known as systematic nodes , and the defining node-repair condition above can be relaxed to requiring the optimal repair bandwidth for systematic nodes only . Such codes are called systematic–repair MSR codes . Unfortunately, finite– $\alpha $ constructions of $[n,k,d]$ MSR codes are known only for certain special cases: either low rate, namely $k/n \leqslant 0.5$ , or high repair connectivity, namely $d = n-1$ . Our main result in this paper is a finite– $\alpha $ construction of systematic-repair $[n,k,d]$ MSR codes for all possible values of parameters $n,k,d$ . We also introduce a generalized construction for $[n,k]$ MSR codes to achieve the optimal repair bandwidth for all values of $d$ simultaneously.

A New Construction of Exact-Repair MSR Codes Using Linearly Dependent Vectors

Regenerating codes focus on the efficient repair of node failures. In an [ $n$ , $k$ , $d$ ] regenerating code system, any $k$ nodes can retrieve the original data and any $d$ nodes can repair a failed node by giving out $\beta$ pieces of data per node. For minimum storage regenerating (MSR) codes, ${d \geq 2 k - 3}$ has been proved. However, as far as we know, there is no construction of exact-repair MSR codes with ${d= 2 k - 3}$ and ${\beta = 1}$ at present. In this letter, we give the first construction of [6, 4, 5] MSR codes with ${\beta = 1}$ , which can perform exact repair of all nodes. Employing the technique of linearly dependent vectors, our codes can be constructed over a small finite field ${F_{4}}$ .

Optimal Construction of Regenerating Code Through Rate-Matching in Hostile Networks

Regenerating code is a class of distributed storage codes that can optimally trade the bandwidth required to repair a failed node with the amount of data stored per node. There are two optimal points in the regeneration tradeoff curve: the minimum storage regeneration code and the minimum bandwidth regeneration code. However, in hostile networks where the storage nodes may be compromised, the storage capacity of the network can be significantly affected. In this paper, we propose two optimal regenerating code constructions through rate-matching to combat this kind of adversarial attacks in hostile networks. We first develop a two-layer rate-matched regenerating code construction. By matching the parameters of the full rate code and the partial rate code, we can optimize the overall storage efficiency while maintaining the corrupted node detection probability. Through comprehensive analysis, we show that the two-layer rate-matched regenerating code can achieve 70% higher storage efficiency than the universally resilient regenerating code. We then propose an optimal $m$ -layer regenerating code construction. While the principle remains the same as the two-layer code, it is designed to optimize the total number of detectable corrupted nodes of $m$ layers from which the errors can be corrected under the constraint of any given code efficiency. Compared with the universally resilient regenerating code with the same rate, our $m$ -layer code can detect 50% more corrupted nodes.

High-Performance General Functional Regenerating Codes with Near-Optimal Repair Bandwidth

Erasure codes are widely used in modern distributed storage systems to prevent data loss and server failures. Regenerating codes are a class of erasure codes that trade storage efficiency and computation for repair bandwidth reduction. However, their nonunified coding parameters and huge computational overhead prohibit their applications. Hence, we first propose a family of General Functional Regenerating (GFR) codes with uncoded repair, balancing storage efficiency and repair bandwidth with general parameters. The GFR codes take advantage of a heuristic repair algorithm, which makes efforts to employ as little repair bandwidth as possible to repair a single failure. Second, we also present a scheduled shift multiplication (SSM) algorithm, which accelerates the matrix product over the Galois field by scheduling the order of coding operations, so encoding and repairing of GFR codes can be executed by fast bitwise shifting and exclusive-OR. Compared to the traditional table-lookup multiplication algorithm, our SSM algorithm gains 1.2 to 2 X speedup in our experimental evaluations, with little effect on the repair success rate.

Constructions of High-Rate Minimum Storage Regenerating Codes Over Small Fields

A novel technique for construction of minimum storage regenerating (MSR) codes is presented. Based on this technique, three explicit constructions of MSR codes are given. The first two constructions provide access-optimal MSR codes, with two and three parities, respectively, which attain the sub-packetization bound for access-optimal codes. The third construction provides longer MSR codes with three parities (i.e., codes with larger number of systematic nodes). This improvement is achieved at the expense of the access-optimality and the field size. In addition to a minimum storage in a node, all three constructions allow the entire data to be recovered from a minimal number of storage nodes. That is, given storage $\ell $ in each node, the entire stored data can be recovered from any $2\log _{2} \ell $ for two parity nodes, and either $3\log _{3}\ell $ or $4\log _{3}\ell $ for three parities. Second, in the first two constructions, a helper node accesses the minimum number of its symbols for repair of a failed node (access-optimality). The goal of this paper is to provide a construction of such optimal codes over the smallest possible finite fields. The generator matrix of these codes is based on perfect matchings of complete graphs and hypergraphs, and on a rational canonical form of matrices. For two parities, the field size is reduced by a factor of two for access-optimal codes compared to previous constructions. For three parities, in the first construction a field size of at least $6\log _{3} \ell +1$ (or $3\log _{3} \ell +1$ for fields with characteristic 2) is sufficient, and in the second construction the field size is larger, yet linear in $\log _{3}\ell $ . Both constructions with three parities provide a significant improvement over previous works due to either decreased field size or lower subpacketization.

Generalised regenerating codes for securing distributed storage systems against eavesdropping

Regenerating codes (RCs) are efficient at both storage cost and repair bandwidth, and thus are regarded as preferable candidates for distributed storage systems (DSSs). For DSSs with RCs, a file stored across n distributed nodes can be reconstructed from k (< n) nodes. The collection of the k nodes is called the reconstruction set. A failed node can be regenerated (i.e., repaired) from d (< n) remaining nodes. The collection of the d nodes is called the regeneration set. In traditional RCs, the numbers of reconstruction sets and regeneration sets are fixed to some specific values. In this paper, we introduce the concept of generalised RCs, in which the value ranges of the numbers of both reconstruction sets and regeneration sets are extended. Compared to traditional RCs, the generalised RCs possess more coding schemes and better system security level in terms of the probability of revealing original data file. An explicit construction of generalised RCs is provided, in which the numbers of both reconstruction sets and regeneration sets can be designed flexibly. Furthermore, based on the generalised RCs, an intruder model where an eavesdropper can access to some nodes is considered and a general upper bound on secrecy capacity is derived. The relationship between the obtained upper bound and existing ones achieved by traditional RCs is discussed in detail. The provided explicit construction is the first optimal construction of generalised RCs, which achieves the upper bound on secrecy capacity and has the flexibility in designing the numbers of reconstruction sets and regeneration sets.

Concurrent regenerating codes

To reduce multiple-failure repair traffic in erasure coded storage systems, Patrick Lee et al. introduce concurrent framework-based minimal-storage regenerating codes (RGCs). The approach is simpler and more practical than the cooperative mechanism in non-fully distributed environment. This study unifies such class of codes as concurrent RGC and further studies the characteristics by analysing the cut-based information flow graph. The authors present a general storage–bandwidth tradeoff and give closed-form expressions for the points on the curve including the minimal-bandwidth point. They show that the concurrent RGC can be constructed by reforming the existing single-node RGC or multiple-node cooperative RGC. Moreover, a connection to strong-maximum distance separable is also analysed.

Minimum Storage Regenerating Codes for Scalable Distributed Storage

Regenerating codes (RGCs) have recently been proposed to reduce the repair traffic of ( $n,k$ ) erasure-coded distributed storage systems. Moreover, RGCs can also be used in a scalable distributed storage scenario where $n$ is increased (decreased) to upgrade (degrade) redundancy while maintaining the maximum distance separable property of erasure codes. In this paper, we propose a new application of minimum storage regenerating (MSR) codes in storage scalability. The connection between repairing invalid nodes and adding new nodes suggests that the two processes can be unified in the same framework. We consider both single and multiple node situations, and two methods for constructing multiple nodes are proposed: concurrent and sequential. We focus on proving the achieved capability of concurrent MSR that can consume minimum traffic for generating multiple nodes. Because concurrent MSR is sensitive to both the number of helpers and added nodes, sequential methods make scalable MSR generalizable. The examples show that the scalable MSR codes have the same advantage of saving network traffic as repairing failures.

Optimal cooperative repair strategy for ( n = 2 k + s , k ) Suh–Ramchandran code

An (n = 2k + s,k) Suh–Ramchandran code is shown, where s > 0 is a positive integer, is a minimum storage regeneration code, and can also achieve the optimal repair bandwidth for multiple systematic nodes failure repair.

Security Concerns in Minimum Storage Cooperative Regenerating Codes

Here, we revisit the problem of exploring the secrecy capacity of minimum storage cooperative regenerating (MSCR) codes under the $\{l_1,l_2\}$-eavesdropper model, where the eavesdropper can observe the data stored on $l_1$ nodes and the repair downloads of an additional $l_2$ nodes. Compared to minimum storage regenerating (MSR) codes which support only single node repairs, MSCR codes allow efficient simultaneous repairs of multiple failed nodes, referred to as a \emph{repair group}. However, the repair data sent from a helper node to another failed node may vary with different repair groups or the sets of helper nodes, which would inevitably leak more data information to the eavesdropper and even render the storage system unable to maintain any data secrecy. In this paper, we introduce and study a special category of MSCR codes, termed "\emph{stable}" MSCR codes, where the repair data from any one helper node to any one failed node is required to be independent of the repair group or the set of helper nodes. Our main contributions include: 1. Demonstrating that two existing MSCR codes inherently are not stable and thus have poor secrecy capacity, 2. Converting one existing MSCR code to a stable one, which offers better secrecy capacity when compared to the original one, 3. Employing information theoretic analysis to characterize the secrecy capacity of stable MSCR codes in certain situations.