Minimum Storage Regenerating Research Articles

Generalization of Minimum Storage Regenerating Codes for Heterogeneous Distributed Storage Systems

Generalization of Minimum Storage Regenerating Codes for Heterogeneous Distributed Storage Systems

Node Repair on Connected Graphs

We study the problem of erasure correction (node repair) for regenerating codes defined on graphs wherein the cost of transmitting the information to the failed node depends on the graphical distance from this node to the helper vertices of the graph. The information passed to the failed node from the helpers traverses several vertices of the graph, and savings in communication complexity can be attained if the intermediate vertices process the information rather than simply relaying it toward the failed node. We derive simple information-theoretic bounds on the amount of information communicated between the nodes in the course of the repair. Next we show that Minimum Storage Regenerating (MSR) codes can be modified to perform the intermediate processing, thereby attaining the lower bound on the information exchange on the graph. We also consider node repair when the underlying graph is random, deriving conditions on the parameters that support recovery of the failed node with communication complexity smaller than required by the simple relaying.

Exact Construction of BS-Assisted MSCR Codes With Link Constraints

It is clear that 5G network resources would be consumed by heavy data traffic owing to increased mobility, slicing, and layered/distributed storage system architecture. The problem is elevated when multiple node failures are repaired to address service quality requirements. Typical approaches include individual or cooperative data regeneration to efficiently utilize the available bandwidth. It is observed that storage systems of 5G and beyond technologies shall have a multi-layer architecture in which base stations (BS) would be present. Moreover, communication with each layer would be subject to various communication costs and link constraints. Under limited BS assistance and cooperation, the trade-off between storage per node and communication bandwidth has been established. In this trade-off, two operating points, namely minimum storage, and bandwidth regeneration are particularly important. In this study, we first identify the optimal number of BS use at the minimum storage regeneration point. An explicit code construction is provided subsequently for the exact minimum storage regeneration whereby each layer may help the repair process subject to a communication link constraint.

Joint Power and Subchannel Allocation for Distributed Storage in Cellular-D2D Underlays.

Wireless distributed storage is beneficial in the provision of reliable content storage and offloading of cellular traffic. In this paper, we consider a cellular device-to-device (D2D) underlay-based wireless distributed storage system, in which the minimum storage regenerating (MSR) coding combined with the partial downloading scheme is employed. To alleviate burdens on insufficient cellular resources and improve spectral efficiency in densely deployed networks, multiple storage devices can simultaneously use the same uplink cellular subchannel under the non-orthogonal multiple access (NOMA) protocol. Our objective is to minimize the total transmission power for content reconstruction, while guaranteeing the signal-to-interference-plus-noise ratio (SINR) constraints for cellular users by jointly optimizing power and subchannel allocation. To tackle the non-convex combinational program, we decouple the original problem into two subproblems and propose two low-complexity algorithms to efficiently solve them, followed by a joint optimization, implemented by alternately updating the solutions to each subproblem. The numerical results illustrate that our proposed algorithms are capable of performing an exhaustive search with lower computation complexity, and the NOMA-enhanced scheme provides more transmission opportunities for neighbor storage devices, thus significantly reducing the total power consumption.

An Exponential Lower Bound on the Sub-Packetization of Minimum Storage Regenerating Codes

An <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(n,k,\ell)$ </tex-math></inline-formula> -vector MDS code over a field <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathbb {F}$ </tex-math></inline-formula> is a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathbb {F}$ </tex-math></inline-formula> -linear subspace of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(\mathbb {F}^\ell)^{n}$ </tex-math></inline-formula> of dimension <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k\ell $ </tex-math></inline-formula> , such that any <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> (vector) symbols of the codeword suffice to determine the remaining <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$r=n-k$ </tex-math></inline-formula> (vector) symbols. The length <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell $ </tex-math></inline-formula> of each codeword symbol is called the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sub-packetization</i> of the code. Such a code is called minimum storage regenerating (MSR), if any single symbol of a codeword can be recovered by downloading <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell /r$ </tex-math></inline-formula> field elements (which is known to be the minimum possible) from each of the other symbols. MSR codes are attractive for use in distributed storage systems, and by now a variety of ingenious constructions of MSR codes are available. However, they all suffer from exponentially large sub-packetization <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell \gtrsim r^{k/r}$ </tex-math></inline-formula> . Our main result is an almost tight lower bound showing that for an MSR code, one must have <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell \geqslant \exp (\Omega (k/r))$ </tex-math></inline-formula> . Previously, a lower bound of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\approx \exp (\sqrt {k/r})$ </tex-math></inline-formula> , and a tight lower bound for a restricted class of “optimal access” MSR codes, were known.

Multilinear algebra for minimum storage regenerating codes: a generalization of the product-matrix construction

An $(n, k, d, \alpha)$-MSR (minimum storage regeneration) code is a set of $n$ nodes used to store a file. For a file of total size $k\alpha$, each node stores $\alpha$ symbols, any $k$ nodes recover the file, and any $d$ nodes can repair any other node via each sending out $\alpha/(d-k+1)$ symbols. In this work, we explore various ways to re-express the infamous product-matrix construction using skew-symmetric matrices, polynomials, symmetric algebras, and exterior algebras. We then introduce a multilinear algebra foundation to produce $\bigl(n, k, \frac{(k-1)t}{t-1}, \binom{k-1}{t-1}\bigr)$-MSR codes for general $t\geq2$. At the $t=2$ end, they include the product-matrix construction as a special case. At the $t=k$ end, we recover determinant codes of mode $m=k$; further restriction to $n=k+1$ makes it identical to the layered code at the MSR point. Our codes' sub-packetization level---$\alpha$---is independent of $n$ and small. It is less than $L^{2.8(d-k+1)}$, where $L$ is Alrabiah--Guruswami's lower bound on $\alpha$. Furthermore, it is less than other MSR codes' $\alpha$ for a subset of practical parameters. We offer hints on how our code repairs multiple failures at once.

Single node repair algorithm for a multimedia cloud storage system based on network coding

Multimedia is inconvenient to use, difficult to maintain, and redundant in data storage. In order to solve the above problems and apply cloud storage to the integration of university teaching resources, this paper designs a virtualized cloud storage platform for university multimedia classrooms. The platform has many advantages, such as reducing the initial investment in multimedia classrooms, simplifying management tasks, making maximum use of actual resources and easy access to resources. Experiments and analysis show the feasibility and effectiveness of the platform. Aiming at the problems of the single-node repair algorithm of the existing multimedia cloud storage system, the limited domain is large, the codec complexity is high, the disk I/O (Input/Output) cost is high, the storage overhead and the repair bandwidth are unbalanced, and a network coding-based approach is proposed. Multimedia cloud storage. System single node repair algorithm. The algorithm stores the grouped multimedia file data in groups in the system, and performs XOR (exclusive OR) on the data in the group on the GF(2) finite field. When some nodes fail, the new node only needs to be connected. Two to three non-faulty nodes in the same group can accurately repair the data in the failed node. Theoretical analysis and simulation results show that the algorithm can reduce the complexity and repair of the codec, and reduce the disk I/O overhead. In this case, the storage cost of the algorithm is consistent with the storage cost based on the minimum storage regeneration code algorithm, and the repair bandwidth cost is close to the minimum bandwidth regeneration code algorithm.

Single node failure repair method for distributed storage system based on genetic algorithm and MSR code

In view of the minimum storage regeneration (MSR) single fault node repair plan, design a kind of new single repair fault node topology. Consider link bandwidth available and the node processing power of heterogeneity, the repair delay and repair traffic node of combinatorial optimization. Topology can be converted to Steiner tree model with constraints, and design the corresponding hybrid genetic algorithm to solve the approximate global optimal solution. Finally, a compromise between the two optimization objectives is achieved. Simulation results show that under the same MSR code size, the repair delay of this topology is only 60%-80% of that of the traditional tree topology and 30%-40% of that of the traditional star topology. While the repair traffic for this topology is 9% to 65% higher than for the traditional star topology, the repair traffic is only 40% to 55% higher than for the traditional tree topology.

PB: A Product-Bitmatrix Construction to Reduce the Complexity of XOR Operations of PM-MSR and PM-MBR Codes over GF 2 w

Edge computing, as an emerging computing paradigm, aims to reduce network bandwidth transmission overhead while storing and processing data on edge nodes. However, the storage strategies required for edge nodes are different from those for existing data centers. Erasure code (EC) strategies have been applied in some decentralized storage systems to ensure the privacy and security of data storage. Product-matrix (PM) regenerating codes (RGCs) as a state-of-the-art EC family are designed to minimize the repair bandwidth overhead or minimize the storage overhead. Nevertheless, the high complexity of the PM framework contains more finite-domain multiplication operations than classical ECs, which heavily consumes computational resources at the edge nodes. In this paper, a theoretical derivation of each step of the PM minimum storage regeneration (PM-MSR) and PM minimum bandwidth regeneration (PM-MBR) codes is performed and the XOR complexity over finite fields is analyzed. On this basis, a new construct called product bitmatrix (PB) is designed to reduce the complexity of XOR operations in the PM framework, and two heuristics are used to further reduce the XOR numbers of the PB-MSR and PB-MBR codes, respectively. The evaluation results show that the PB construction significantly reduces the XOR number compared to the PM-MSR, PM-MBR, Reed–Solomon (RS), and Cauchy RS codes while retaining optimal performance and reliability.

ϵ-MSR Codes: Contacting Fewer Code Blocks for Exact Repair

$\epsilon $ -Minimum Storage Regenerating ( $\epsilon $ -MSR) codes form a special class of Maximum Distance Separable (MDS) codes, providing mechanisms for exact regeneration of a single code block in their codewords by downloading slightly sub-optimal amount of information from the remaining code blocks. The key advantage of these codes is a significantly lower sub-packetization that grows only logarithmically with the code length, while providing optimality in storage and error-correcting capacity. However, existing constructions of $\epsilon $ -MSR codes require each remaining code block to be available for the repair of any failed code block. In this paper, we construct $\epsilon $ -MSR codes that can repair any failed code block by contacting fewer number of available code blocks. When a code block fails, our repair procedure needs to contact a few compulsory code blocks and is free to choose any subset of available code blocks for the remaining choices. Our construction requires a field size linear in code length and ensures load balancing (in terms of information downloaded) among the contacted code blocks for repairing a failed code block.

Hybrid Codes

Erasure codes are being extensively deployed in practical storage systems to prevent data loss with low redundancy. However, these codes require excessive disk I/Os and network traffic for recovering unavailable data. Among all erasure codes, Minimum Storage Regenerating (MSR) codes can achieve optimal repair bandwidth under the minimum storage during recovery, but some open issues remain to be addressed before applying them in real systems. Facing with the huge burden during recovery, erasure-coded storage systems need to be developed with high repair efficiency. Aiming at this goal, a new class of coding scheme is introduced—Hybrid Regenerating Codes (Hybrid-RC). The codes utilize the superiority of MSR codes to compute a subset of data blocks while some other parity blocks are used for reliability maintenance. As a result, our design is near-optimal with respect to storage and network traffic and shows great improvements in recovery performance.

Beyond the MDS Bound in Distributed Cloud Storage

Regenerating code is a class of distributed storage codes that can optimally trade the bandwidth with the amount of data stored per node to repair a failed node. There are two extreme points in the optimal regenerating trade-off curve, which correspond to minimum-storage regenerating (MSR) and minimum-bandwidth regenerating (MBR). Recently, Reed-Solomon (RS) code based regenerating codes (RS-RC) were constructed under the product-matrix framework. It can also achieve the maximum distance separable (MDS) property in code regeneration and reconstruction. However, in case that the network is hostile and the storage nodes could be compromised or packets be modified, the storage capacity and the bandwidth required to regenerate or reconstruct the original file can be significantly affected. In this paper, we propose Hermitian code based regenerating codes (H-RC) by developing constructions under the product-matrix framework for minimum storage regenerating (H-MSR) and the minimum bandwidth regenerating (H-MBR). We also propose data regeneration and reconstruction algorithms for both H-MSR and H-MBR codes under both error-free and hostile networks. We demonstrate that the proposed algorithms can also successfully determine the erroneous decodings in hostile networks. Theoretical evaluation shows that our proposed H-RC can detect and correct more errors in hostile networks well beyond the RS-RC with the same code rate. Our analysis shows that the proposed H-RC have lower computational complexity than the RS-RC for both code regeneration and code reconstruction.

Intelligent Optimization of Availability and Communication Cost in Satellite-UAV Mobile Edge Caching System With Fault-Tolerant Codes

Mobile computing provides storage and computation resources of proximal devices to satisfy the real-time and low-energy communication demands of the Internet of Things (IoT). However, in the areas without terrestrial base station infrastructures, the IoT sensors have trouble implementing reliable and stable connections, which results in the difficulties of data gathering and data caching. In this paper, we consider a space-air-ground integrated mobile edge caching IoT system composed of satellite and unmanned aerial vehicles (UAVs), where LEO satellite broadcasts data, and UAVs collect the data from decentralized ground sensors. Since the sensors’ low-power property leads data loss, fault-tolerant codes are employed for availability protection. We first derive the exact expressions of system availability and communication cost for data repair and collection. Then, to address the problems of the lower availability, we exploit an intelligent optimization to determine the erasure coding parameters. Lastly, we further optimize the system parameters, i.e., communication ranges and unit power costs of UAV and decentralized sensors, to minimize the total communication cost. Simulation results show that, compared to MDS codes and regenerating codes, adaptive minimum storage regenerating (AMSR) codes with optimized parameters can significantly reduce total communication cost and maintain availability of the system.

Towards Practical Private Information Retrieval From MDS Array Codes

Private information retrieval (PIR) is the problem of privately retrieving one out of $M$ original files from $N$ severs, i.e., each individual server learns nothing about the file that the user is requesting. Usually, the $M$ files are replicated or encoded by a maximum distance separable (MDS) code and then stored across the $N$ servers. Compared to mere replication, MDS coded servers can significantly reduce the storage overhead. Particularly, PIR from minimum storage regenerating (MSR) coded servers can simultaneously reduce the repair bandwidth when repairing failed servers. Existing PIR schemes from MSR coded servers either require large sub-packetization levels or are not capacity-achieving. In this paper, a PIR protocol from MDS array codes is proposed, subsuming PIR from MSR coded servers as a special case. Particularly, the case of non-colluding, honest-but-curious servers is considered. The retrieval rate of the new PIR protocol achieves the capacity of PIR from MDS/MSR coded servers. By choosing different MDS array codes, the new PIR protocol can have some advantages when compared with existing protocols, e.g., 1) small sub-packetization, 2) (near-) optimal repair bandwidth, 3) implementable over the binary field $\mathbf{F}_2$.

Local Codes With Cooperative Repair in Distributed Storage of Cyber-Physical-Social Systems

Integrating cyber, physical, and social spaces together, cyber-physical-social systems (CPSS) bring more conveniences to humans. For practical applications and user convenience, it is essential that the Big Data produced in CPSS be stored in the distributed storage systems of CPSS. In this paper, we study the fault tolerance scheme for distributed storage systems of CPSS, and propose a framework that can recover multiple failed nodes simultaneously. Considering the reliability of storage nodes in distributed storage systems, the research on locally repairable codes has mostly focused on repairing failed nodes within each repair group. However, when entire repair groups have failed, existing locally repairable codes cannot repair more than one failed group. In this paper, local codes with cooperative repair that can recover more than one failed group are proposed. Specifically, the proposed local codes are constructed based on minimum storage regenerating (MSR) codes, and have an interleaving structure among the local codes, so that the parity symbols of any local code can be generated from the MSR codes in its two adjacent local codes. Taking advantage of this property, more than one failed local group can be repaired cooperatively by their adjacent local groups with lower repair locality. Furthermore, the key parameters of local codes with cooperative repair are derived. Theoretical analysis and simulation results show that, compared with previous codes with local regeneration, our codes have higher bandwidth overhead when repairing failed nodes, but advantages in storage overhead and repair locality either for repair of a single failed node or one failed local group. Moreover for a single failed local group, local codes with cooperative repair achieve almost the same tradeoff curve of storage overhead and bandwidth overhead as MSR-local codes and minimum bandwidth regenerating local (MBR-local) codes.

Efficient Storage Scaling for MBR and MSR Codes

Due to the explosive growth of storage demands, distributed storage systems need to support storage scaling efficiently. Recent work optimizes scaling in a decentralized manner for Reed-solomon coded storage systems. In this paper, we focus on storage scaling for storage systems with regenerating codes and design two efficient scaling algorithms for minimum bandwidth regenerating (MBR) and minimum storage regenerating (MSR) codes. We integrate these two scaling algorithms into Hadoop Distributed File System (HDFS), and the experiments on Amazon EC2 show that the scaling bandwidth can be reduced up to 75% and 43.8% over the centralized scaling.

Efficient VLSI Architectures for Coupled-Layered Regenerating Codes

To enable the continued scaling of distributed storage, failure recovery schemes that have low latency and low redundancy are indispensable. Minimum storage regenerating (MSR) codes have been developed to achieve this goal. Compared to other MSR codes, the recent coupled-layered (Clay) codes have many advantages such as low sub-packetization level, small finite field for construction, and uniform repair bandwidth over data and parities. This brief proposes efficient VLSI architectures for Clay encoder and decoder. Through exploiting subfield elements of finite fields, the logic complexity is substantially lowered. Additionally, the buffer size is reduced by optimizing the coupling scheme. For an example Clay code with $(n,k)=(6,4)$ , the proposed designs achieve 12% and 24% logic complexity reduction on the encoder and decoder, respectively, under the same timing constraint.

On the Sub-Packetization Size and the Repair Bandwidth of Reed-Solomon Codes

Reed-Solomon (RS) codes are widely used in distributed storage systems. In this paper, we study the repair bandwidth and sub-packetization size of RS codes. The repair bandwidth is defined as the amount of transmitted information from surviving nodes to a failed node. The RS code can be viewed as a polynomial over a finite field $GF(q^\ell)$ evaluated at a set of points, where $\ell$ is called the sub-packetization size. Smaller bandwidth reduces the network traffic in distributed storage, and smaller $\ell$ facilitates the implementation of RS codes with lower complexity. Recently, Guruswami and Wootters proposed a repair method for RS codes when the evaluation points are the entire finite field. While the sub-packization size can be arbitrarily small, the repair bandwidth is higher than the minimum storage regenerating (MSR) bound. Tamo, Ye and Barg achieved the MSR bound but the sub-packetization size grows faster than the exponential function of the number of the evaluation points. In this work, we present code constructions and repair schemes that extend these results to accommodate different sizes of the evaluation points. In other words, we design schemes that provide points in between. These schemes provide a flexible tradeoff between the sub-packetization size and the repair bandwidth. In addition, we generalize our schemes to manage multiple failures.

Improved Upper Bounds on Systematic-Length for Linear Minimum Storage Regenerating Codes

In this paper, we revisit the problem of finding the longest systematic-length k for a linear minimum storage regenerating (MSR) code with optimal repair of only systematic part, for a given per-node storage capacity l and an arbitrary number of parity nodes r. We study the problem by following a geometric analysis of linear subspaces and operators. First, a simple quadratic bound is given, which implies that k = r + 2 is the largest number of systematic nodes in the scalar scenario. Second, an r-based-log bound is derived, which is superior to the upper bound on log-base 2 in the prior work. Finally, an explicit upper bound depending on the value of r <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> /l is introduced, which further extends the corresponding result in the literature.

Optimal Weakly Secure Minimum Storage Regenerating Codes Scheme

In a distributed storage system, regenerating codes can be utilized to ensure data availability. This reduces the repair bandwidth but increases the risk of data eavesdropping on each node. Previous studies in this area generally only provided an approximate analysis of the security of schemes with information-theoretic security or weak security. Some researchers have further divided weak security into block security; however, they only analyzed several the regenerating codes schemes and proposed improved schemes with specific eavesdropping capabilities. In this study, we analyze the block security of a Cauchy-matrix-based product-matrix minimum storage regenerating(MSR) scheme and determine the optimal block security of MSR codes. Lastly, an improved MSR codes scheme for achieving optimal block security is proposed, and the relevant proof is provided.