Minimum Storage Regenerating Codes Research Articles

Generalization of Minimum Storage Regenerating Codes for Heterogeneous Distributed Storage Systems

Generalization of Minimum Storage Regenerating Codes for Heterogeneous Distributed Storage Systems

An Exponential Lower Bound on the Sub-Packetization of Minimum Storage Regenerating Codes

An <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(n,k,\ell)$ </tex-math></inline-formula> -vector MDS code over a field <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathbb {F}$ </tex-math></inline-formula> is a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathbb {F}$ </tex-math></inline-formula> -linear subspace of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(\mathbb {F}^\ell)^{n}$ </tex-math></inline-formula> of dimension <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k\ell $ </tex-math></inline-formula> , such that any <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> (vector) symbols of the codeword suffice to determine the remaining <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$r=n-k$ </tex-math></inline-formula> (vector) symbols. The length <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell $ </tex-math></inline-formula> of each codeword symbol is called the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sub-packetization</i> of the code. Such a code is called minimum storage regenerating (MSR), if any single symbol of a codeword can be recovered by downloading <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell /r$ </tex-math></inline-formula> field elements (which is known to be the minimum possible) from each of the other symbols. MSR codes are attractive for use in distributed storage systems, and by now a variety of ingenious constructions of MSR codes are available. However, they all suffer from exponentially large sub-packetization <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell \gtrsim r^{k/r}$ </tex-math></inline-formula> . Our main result is an almost tight lower bound showing that for an MSR code, one must have <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell \geqslant \exp (\Omega (k/r))$ </tex-math></inline-formula> . Previously, a lower bound of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\approx \exp (\sqrt {k/r})$ </tex-math></inline-formula> , and a tight lower bound for a restricted class of “optimal access” MSR codes, were known.

Multilinear algebra for minimum storage regenerating codes: a generalization of the product-matrix construction

An $(n, k, d, \alpha)$-MSR (minimum storage regeneration) code is a set of $n$ nodes used to store a file. For a file of total size $k\alpha$, each node stores $\alpha$ symbols, any $k$ nodes recover the file, and any $d$ nodes can repair any other node via each sending out $\alpha/(d-k+1)$ symbols. In this work, we explore various ways to re-express the infamous product-matrix construction using skew-symmetric matrices, polynomials, symmetric algebras, and exterior algebras. We then introduce a multilinear algebra foundation to produce $\bigl(n, k, \frac{(k-1)t}{t-1}, \binom{k-1}{t-1}\bigr)$-MSR codes for general $t\geq2$. At the $t=2$ end, they include the product-matrix construction as a special case. At the $t=k$ end, we recover determinant codes of mode $m=k$; further restriction to $n=k+1$ makes it identical to the layered code at the MSR point. Our codes' sub-packetization level---$\alpha$---is independent of $n$ and small. It is less than $L^{2.8(d-k+1)}$, where $L$ is Alrabiah--Guruswami's lower bound on $\alpha$. Furthermore, it is less than other MSR codes' $\alpha$ for a subset of practical parameters. We offer hints on how our code repairs multiple failures at once.

Single node repair algorithm for a multimedia cloud storage system based on network coding

Multimedia is inconvenient to use, difficult to maintain, and redundant in data storage. In order to solve the above problems and apply cloud storage to the integration of university teaching resources, this paper designs a virtualized cloud storage platform for university multimedia classrooms. The platform has many advantages, such as reducing the initial investment in multimedia classrooms, simplifying management tasks, making maximum use of actual resources and easy access to resources. Experiments and analysis show the feasibility and effectiveness of the platform. Aiming at the problems of the single-node repair algorithm of the existing multimedia cloud storage system, the limited domain is large, the codec complexity is high, the disk I/O (Input/Output) cost is high, the storage overhead and the repair bandwidth are unbalanced, and a network coding-based approach is proposed. Multimedia cloud storage. System single node repair algorithm. The algorithm stores the grouped multimedia file data in groups in the system, and performs XOR (exclusive OR) on the data in the group on the GF(2) finite field. When some nodes fail, the new node only needs to be connected. Two to three non-faulty nodes in the same group can accurately repair the data in the failed node. Theoretical analysis and simulation results show that the algorithm can reduce the complexity and repair of the codec, and reduce the disk I/O overhead. In this case, the storage cost of the algorithm is consistent with the storage cost based on the minimum storage regeneration code algorithm, and the repair bandwidth cost is close to the minimum bandwidth regeneration code algorithm.

Single node failure repair method for distributed storage system based on genetic algorithm and MSR code

In view of the minimum storage regeneration (MSR) single fault node repair plan, design a kind of new single repair fault node topology. Consider link bandwidth available and the node processing power of heterogeneity, the repair delay and repair traffic node of combinatorial optimization. Topology can be converted to Steiner tree model with constraints, and design the corresponding hybrid genetic algorithm to solve the approximate global optimal solution. Finally, a compromise between the two optimization objectives is achieved. Simulation results show that under the same MSR code size, the repair delay of this topology is only 60%-80% of that of the traditional tree topology and 30%-40% of that of the traditional star topology. While the repair traffic for this topology is 9% to 65% higher than for the traditional star topology, the repair traffic is only 40% to 55% higher than for the traditional tree topology.

Hybrid Codes

Erasure codes are being extensively deployed in practical storage systems to prevent data loss with low redundancy. However, these codes require excessive disk I/Os and network traffic for recovering unavailable data. Among all erasure codes, Minimum Storage Regenerating (MSR) codes can achieve optimal repair bandwidth under the minimum storage during recovery, but some open issues remain to be addressed before applying them in real systems. Facing with the huge burden during recovery, erasure-coded storage systems need to be developed with high repair efficiency. Aiming at this goal, a new class of coding scheme is introduced—Hybrid Regenerating Codes (Hybrid-RC). The codes utilize the superiority of MSR codes to compute a subset of data blocks while some other parity blocks are used for reliability maintenance. As a result, our design is near-optimal with respect to storage and network traffic and shows great improvements in recovery performance.

Local Codes With Cooperative Repair in Distributed Storage of Cyber-Physical-Social Systems

Integrating cyber, physical, and social spaces together, cyber-physical-social systems (CPSS) bring more conveniences to humans. For practical applications and user convenience, it is essential that the Big Data produced in CPSS be stored in the distributed storage systems of CPSS. In this paper, we study the fault tolerance scheme for distributed storage systems of CPSS, and propose a framework that can recover multiple failed nodes simultaneously. Considering the reliability of storage nodes in distributed storage systems, the research on locally repairable codes has mostly focused on repairing failed nodes within each repair group. However, when entire repair groups have failed, existing locally repairable codes cannot repair more than one failed group. In this paper, local codes with cooperative repair that can recover more than one failed group are proposed. Specifically, the proposed local codes are constructed based on minimum storage regenerating (MSR) codes, and have an interleaving structure among the local codes, so that the parity symbols of any local code can be generated from the MSR codes in its two adjacent local codes. Taking advantage of this property, more than one failed local group can be repaired cooperatively by their adjacent local groups with lower repair locality. Furthermore, the key parameters of local codes with cooperative repair are derived. Theoretical analysis and simulation results show that, compared with previous codes with local regeneration, our codes have higher bandwidth overhead when repairing failed nodes, but advantages in storage overhead and repair locality either for repair of a single failed node or one failed local group. Moreover for a single failed local group, local codes with cooperative repair achieve almost the same tradeoff curve of storage overhead and bandwidth overhead as MSR-local codes and minimum bandwidth regenerating local (MBR-local) codes.

Efficient VLSI Architectures for Coupled-Layered Regenerating Codes

To enable the continued scaling of distributed storage, failure recovery schemes that have low latency and low redundancy are indispensable. Minimum storage regenerating (MSR) codes have been developed to achieve this goal. Compared to other MSR codes, the recent coupled-layered (Clay) codes have many advantages such as low sub-packetization level, small finite field for construction, and uniform repair bandwidth over data and parities. This brief proposes efficient VLSI architectures for Clay encoder and decoder. Through exploiting subfield elements of finite fields, the logic complexity is substantially lowered. Additionally, the buffer size is reduced by optimizing the coupling scheme. For an example Clay code with $(n,k)=(6,4)$ , the proposed designs achieve 12% and 24% logic complexity reduction on the encoder and decoder, respectively, under the same timing constraint.

Improved Upper Bounds on Systematic-Length for Linear Minimum Storage Regenerating Codes

In this paper, we revisit the problem of finding the longest systematic-length k for a linear minimum storage regenerating (MSR) code with optimal repair of only systematic part, for a given per-node storage capacity l and an arbitrary number of parity nodes r. We study the problem by following a geometric analysis of linear subspaces and operators. First, a simple quadratic bound is given, which implies that k = r + 2 is the largest number of systematic nodes in the scalar scenario. Second, an r-based-log bound is derived, which is superior to the upper bound on log-base 2 in the prior work. Finally, an explicit upper bound depending on the value of r <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> /l is introduced, which further extends the corresponding result in the literature.

Optimal Weakly Secure Minimum Storage Regenerating Codes Scheme

In a distributed storage system, regenerating codes can be utilized to ensure data availability. This reduces the repair bandwidth but increases the risk of data eavesdropping on each node. Previous studies in this area generally only provided an approximate analysis of the security of schemes with information-theoretic security or weak security. Some researchers have further divided weak security into block security; however, they only analyzed several the regenerating codes schemes and proposed improved schemes with specific eavesdropping capabilities. In this study, we analyze the block security of a Cauchy-matrix-based product-matrix minimum storage regenerating(MSR) scheme and determine the optimal block security of MSR codes. Lastly, an improved MSR codes scheme for achieving optimal block security is proposed, and the relevant proof is provided.

HashTag Erasure Codes: From Theory to Practice

Minimum-Storage Regenerating (MSR) codes have emerged as a viable alternative to Reed-Solomon (RS) codes as they minimize the repair bandwidth while they are still optimal in terms of reliability and storage overhead. Although several MSR constructions exist, so far they have not been practically implemented mainly due to the big number of I/O operations. In this paper, we analyze high-rate MDS codes that are simultaneously optimized in terms of storage, reliability, I/O operations, and repair-bandwidth for single and multiple failures of the systematic nodes. The codes were recently introduced in [1] without any specific name. Due to the resemblance between the hashtag sign # and the procedure of the code construction, we call them in this paper HashTag Erasure Codes (HTECs). HTECs provide the lowest data-read and data-transfer, and thus the lowest repair time for an arbitrary sub-packetization level α, where α ≤ r <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">⌈k/r⌉</sup> , among all existing MDS codes for distributed storage including MSR codes. The repair process is linear and highly parallel. Additionally, we show that HTECs are the first high-rate MDS codes that reduce the repair bandwidth for more than one failure. Practical implementations of HTECs in Hadoop release 3.0.0-alpha2 demonstrate their great potentials.

Product Matrix MSR Codes With Bandwidth Adaptive Exact Repair

In the distributed storage systems (DSSs) with $k$ systematic nodes, robustness against node failure is commonly provided by storing redundancy in a number of other nodes and performing repair mechanism to reproduce the content of the failed nodes. Efficiency is then achieved by minimizing the storage overhead and the amount of data transmission required for data reconstruction and repair, provided by coding solutions, such as regenerating codes. Common explicit regenerating code constructions enable efficient repair through accessing a predefined number, $d$ , of arbitrary chosen available nodes, namely helpers. In practice, however, the state of the system dynamically changes based on the request load, the link traffic, and so on, and the parameters which optimize system’s performance vary accordingly. It is then desirable to have coding schemes which are able to operate optimally under a range of different parameters simultaneously. Specifically, adaptivity in the number of helper nodes for repair is of interest. While robustness requires capability of performing repair with small number of helpers, it is desirable to use as many helpers as available to reduce the transmission delay and total repair traffic. In this paper, we focus on the minimum storage regenerating (MSR) codes, where each of the $n$ nodes in the network is supposed to store $\alpha $ information units, and the source data of size $k\alpha $ could be recovered from any arbitrary set of $k$ nodes. We introduce a class of MSR codes that realize the optimal repair bandwidth simultaneously with a set of different choices for the number of helpers, namely $D=\{d_{1}, \ldots , d_{\delta }\}$ . Our coding scheme follows the product matrix (PM) framework introduced by Rashmi et al. and could be considered as a generalization of the PM MSR code presented by Rashmi et al. , such that any $d_{i} = (i+1)(k-1)$ helpers can perform an optimal repair. As a result, the coding rate in our construction is limited by $({k}/{n})\leq ({1}/{2})$ . However, similar to the original design of PM MSR codes, our solution can realize practical values of the parameter $\alpha $ . Recently, Ye and Barg have presented another explicit MSR coding scheme which is capable of performing optimal repair for various number of helpers. The solution presented by Ye and Barg works for any arbitrary set of parameters $k$ and $D$ and can achieve high-coding rates, but the required $\alpha $ for this code is exponentially large. We show that the required value for $\alpha $ in the coding scheme presented in this paper is exponentially smaller when compared with the work of Ye and Barg for the same set of other parameters. Particularly, for a DSS with $n$ nodes and $k$ systematic nodes, the required value for $\alpha $ is reduced from $s^{n}$ to $sk$ , where $s=\mathrm {l~cm}(d_{1}-k+1,\cdots ,d_{\delta }-k+1)$ . We also show the required field size in the presented coding scheme is equal to $n$ .

Explicit Constructions of High-Rate MSR Codes with Optimal Access Property over Small Finite Fields

Up to now, many $(k+r,k,N)$ minimum-storage regenerating (MSR) codes with $k$ information nodes, $r$ parity nodes, and node capacity $N$ have been proposed. However, most of them are constructed over a relatively large finite field. In this paper, we propose three high-rate MSR codes over small finite fields. First, the new MSR code $\mathcal {C}_{1}$ with the optimal access property for all nodes is constructed over small finite field $\mathbb {F}_{q}$ , for example $q=3$ for even $r$ or $q\ge r+1$ for odd $r$ , which is much smaller than that of the known one given by Ye and Barg. Further, considering to reduce the node capacity, another new MSR code $\mathcal {C}_{2}$ over $\mathbb {F}_{q}$ with $q\ge r+2$ is generated based on $\mathcal {C}_{1}$ , which can effectively reduce the node capacity of $\mathcal {C}_{1}$ by a factor of $r^{r-1}$ . However, only the first $k$ nodes of $\mathcal {C}_{2}$ have the optimal access property. Therefore, the new MSR code $\mathcal {C}_{3}$ over $\mathbb {F}_{q}$ with $q\ge r+2$ which has the optimal access property for all nodes is proposed by modifying $\mathcal {C}_{2}$ . Notably, in contrast to $\mathcal {C}_{1}$ , the node capacity of $\mathcal {C}_{3}$ is decreased by a factor of $r^{r-2}$ .

Locally minimum storage regenerating codes in distributed cloud storage systems

In distributed cloud storage systems, inevitably there exist multiple node failures at the same time. The existing methods of regenerating codes, including minimum storage regenerating (MSR) codes and minimum bandwidth regenerating (MBR) codes, are mainly to repair one single or several failed nodes, unable to meet the repair need of distributed cloud storage systems. In this paper, we present locally minimum storage regenerating (LMSR) codes to recover multiple failed nodes at the same time. Specifically, the nodes in distributed cloud storage systems are divided into multiple local groups, and in each local group (4, 2) or (5, 3) MSR codes are constructed. Moreover, the grouping method of storage nodes and the repairing process of failed nodes in local groups are studied. Theoretical analysis shows that LMSR codes can achieve the same storage overhead as MSR codes. Furthermore, we verify by means of simulation that, compared with MSR codes, LMSR codes can reduce the repair bandwidth and disk I/O overhead effectively.

Minimum Storage Regenerating Codes for All Parameters

Regenerating codes for distributed storage have attracted much research interest in the past decade. Such codes trade the bandwidth needed to repair a failed node with the overall amount of data stored in the network. Minimum storage regenerating (MSR) codes are an important class of optimal regenerating codes that minimize (first) the amount of data stored per node and (then) the repair bandwidth. Specifically, an $[n,k,d]$ - $(\alpha )$ MSR code $ \mathbb {C}$ over $ \smash {\mathbb {F}_{\!q}}$ stores a file $ {\mathcal{ F}}$ consisting of $\alpha k$ symbols over $ \smash {\mathbb {F}_{\!q}}$ among $n$ nodes, each storing $\alpha $ symbols, in such a way that: 1) the file $ {\mathcal{ F}}$ can be recovered by downloading the content of any $k$ of the $n$ nodes and 2) the content of any failed node can be reconstructed by accessing any $d$ of the remaining $n-1$ nodes and downloading $\alpha /(d{-}k{+}1)$ symbols from each of these nodes. In practice, the file $ {\mathcal{ F}}$ is typically available in uncoded form on some $k$ of the $n$ nodes, known as systematic nodes , and the defining node-repair condition above can be relaxed to requiring the optimal repair bandwidth for systematic nodes only . Such codes are called systematic–repair MSR codes . Unfortunately, finite– $\alpha $ constructions of $[n,k,d]$ MSR codes are known only for certain special cases: either low rate, namely $k/n \leqslant 0.5$ , or high repair connectivity, namely $d = n-1$ . Our main result in this paper is a finite– $\alpha $ construction of systematic-repair $[n,k,d]$ MSR codes for all possible values of parameters $n,k,d$ . We also introduce a generalized construction for $[n,k]$ MSR codes to achieve the optimal repair bandwidth for all values of $d$ simultaneously.

A New Construction of Exact-Repair MSR Codes Using Linearly Dependent Vectors

Regenerating codes focus on the efficient repair of node failures. In an [ $n$ , $k$ , $d$ ] regenerating code system, any $k$ nodes can retrieve the original data and any $d$ nodes can repair a failed node by giving out $\beta$ pieces of data per node. For minimum storage regenerating (MSR) codes, ${d \geq 2 k - 3}$ has been proved. However, as far as we know, there is no construction of exact-repair MSR codes with ${d= 2 k - 3}$ and ${\beta = 1}$ at present. In this letter, we give the first construction of [6, 4, 5] MSR codes with ${\beta = 1}$ , which can perform exact repair of all nodes. Employing the technique of linearly dependent vectors, our codes can be constructed over a small finite field ${F_{4}}$ .

Optimal Construction of Regenerating Code Through Rate-Matching in Hostile Networks

Regenerating code is a class of distributed storage codes that can optimally trade the bandwidth required to repair a failed node with the amount of data stored per node. There are two optimal points in the regeneration tradeoff curve: the minimum storage regeneration code and the minimum bandwidth regeneration code. However, in hostile networks where the storage nodes may be compromised, the storage capacity of the network can be significantly affected. In this paper, we propose two optimal regenerating code constructions through rate-matching to combat this kind of adversarial attacks in hostile networks. We first develop a two-layer rate-matched regenerating code construction. By matching the parameters of the full rate code and the partial rate code, we can optimize the overall storage efficiency while maintaining the corrupted node detection probability. Through comprehensive analysis, we show that the two-layer rate-matched regenerating code can achieve 70% higher storage efficiency than the universally resilient regenerating code. We then propose an optimal $m$ -layer regenerating code construction. While the principle remains the same as the two-layer code, it is designed to optimize the total number of detectable corrupted nodes of $m$ layers from which the errors can be corrected under the constraint of any given code efficiency. Compared with the universally resilient regenerating code with the same rate, our $m$ -layer code can detect 50% more corrupted nodes.

Constructions of High-Rate Minimum Storage Regenerating Codes Over Small Fields

A novel technique for construction of minimum storage regenerating (MSR) codes is presented. Based on this technique, three explicit constructions of MSR codes are given. The first two constructions provide access-optimal MSR codes, with two and three parities, respectively, which attain the sub-packetization bound for access-optimal codes. The third construction provides longer MSR codes with three parities (i.e., codes with larger number of systematic nodes). This improvement is achieved at the expense of the access-optimality and the field size. In addition to a minimum storage in a node, all three constructions allow the entire data to be recovered from a minimal number of storage nodes. That is, given storage $\ell $ in each node, the entire stored data can be recovered from any $2\log _{2} \ell $ for two parity nodes, and either $3\log _{3}\ell $ or $4\log _{3}\ell $ for three parities. Second, in the first two constructions, a helper node accesses the minimum number of its symbols for repair of a failed node (access-optimality). The goal of this paper is to provide a construction of such optimal codes over the smallest possible finite fields. The generator matrix of these codes is based on perfect matchings of complete graphs and hypergraphs, and on a rational canonical form of matrices. For two parities, the field size is reduced by a factor of two for access-optimal codes compared to previous constructions. For three parities, in the first construction a field size of at least $6\log _{3} \ell +1$ (or $3\log _{3} \ell +1$ for fields with characteristic 2) is sufficient, and in the second construction the field size is larger, yet linear in $\log _{3}\ell $ . Both constructions with three parities provide a significant improvement over previous works due to either decreased field size or lower subpacketization.

On Secrecy Capacity of Minimum Storage Regenerating Codes

In this paper, we revisit the problem of characterizing the secrecy capacity of minimum storage regenerating (MSR) codes under the passive $(l_{1},l_{2})$ -eavesdropper model, where the eavesdropper has access to data stored on $l_{1}$ nodes and the repair data for an additional $l_{2}$ nodes. We study it from the information-theoretic perspective. First, some general properties of MSR codes as well as a simple and generally applicable upper bound on secrecy capacity are given. Second, a new concept of stable MSR codes is introduced, where the stable property is shown to be closely linked with secrecy capacity. Finally, a comprehensive and explicit result on secrecy capacity in the linear MSR scenario is present, which generalizes all related works in the literature and also predicts certain results for some unexplored linear MSR codes.

Optimal cooperative repair strategy for ( n = 2 k + s , k ) Suh–Ramchandran code

An (n = 2k + s,k) Suh–Ramchandran code is shown, where s > 0 is a positive integer, is a minimum storage regeneration code, and can also achieve the optimal repair bandwidth for multiple systematic nodes failure repair.