Erasure-coded Distributed Storage Systems Research Articles

Tree-structured data placement scheme with cluster-aided top-down transmission in erasure-coded distributed storage systems

In erasure-coded distributed storage systems, the rapid completion of data placement process is very critical to maintain system performance, where the process is defined as to insert coded blocks into a set of redundant storage nodes. However, the existing data placement schemes have some shortcomings, such as load imbalance and long transmission latency. To solve these problems, this paper proposes a tree-structured data placement scheme with cluster-aided top-down transmission, which improves the practicality and the efficiency of the data insertion. Specifically, the storage nodes are clustered by a location-based K-mean method at first, and then selecting a master node as the source node to encode file and distribute coded blocks in each cluster by designing a master node’s selection algorithm based on evolutionary game theory. Since multiple master nodes are used to receive data files from data core, the load imbalance problem caused by single node receiving data files is mitigated. In addition, to improve the data transmission efficiency, we construct a tree-structured network to organize the connections of the storage nodes in cluster, where the master node is the root node. Meanwhile, a data placement scheme with top-down transmission by combining an improved metaheuristic algorithm is proposed to manage the pipelined data transmission along the tree-structured network. Experimental results show that the proposed scheme can effectively improve the survive life of storage network and reduce the data insertion time. Furthermore, the proposed scheme has better performance than the optimal tree-structured scheme based on other optimization algorithms.

Popularity-based full replica caching for erasure-coded distributed storage systems

Popularity-based full replica caching for erasure-coded distributed storage systems

Information-Theoretically Secure Erasure Codes for Distributed Storage

Repair operations in erasure-coded distributed storage systems involve a lot of data movement. This can potentially expose data to malicious acts of passive eavesdroppers or active adversaries, putting security of the system at risk. This paper presents coding schemes and repair algorithms that ensure security of the data in the presence of passive eavesdroppers and active adversaries while maintaining high availability, reliability, and resource efficiency in the system. The proposed codes are optimal in that they meet previously proposed lower bounds on storage and network-bandwidth requirements for a wide range of system parameters. The results thus establish the secure storage capacity of such systems. The proposed codes are based on an optimal class of codes called product-matrix codes. The constructions presented for security from active adversaries provide an additional appealing feature of “on-demand security,” where the desired level of security can be chosen separately for each instance of repair, and the proposed algorithms remain optimal simultaneously for all possible security levels. This paper also provides necessary and sufficient conditions governing the transformation of any (non-secure) code into one providing on-demand security.

TERS: a traffic efficient repair scheme for repairing multiple losses in erasure-coded distributed storage systems

The high repair traffic and the long repair time of erasure coding have posed a new challenge: how to minimise the amount of data transferred among nodes and reduce the repair time when repairing the lost data. Existing schemes are mostly designed for single node failures, which incur high network traffic and result in low efficiency. In this paper, we propose a traffic efficient repair scheme TERS suitable for repairing data losses when multiple nodes fail. To evaluate the repair cost and the repair time, we provide an implementation of integrating TERS into HDFS-RAID. The numerical results confirm that TERS reduces the repair traffic by 44% on average compared with the traditional erasure codes and regenerating codes. Theoretical analysis shows that TERS effectively reduces the repair time.

TERS: a traffic efficient repair scheme for repairing multiple losses in erasure-coded distributed storage systems

The high repair traffic and the long repair time of erasure coding have posed a new challenge: how to minimise the amount of data transferred among nodes and reduce the repair time when repairing the lost data. Existing schemes are mostly designed for single node failures, which incur high network traffic and result in low efficiency. In this paper, we propose a traffic efficient repair scheme TERS suitable for repairing data losses when multiple nodes fail. To evaluate the repair cost and the repair time, we provide an implementation of integrating TERS into HDFS-RAID. The numerical results confirm that TERS reduces the repair traffic by 44% on average compared with the traditional erasure codes and regenerating codes. Theoretical analysis shows that TERS effectively reduces the repair time.

Repair Tree: Fast Repair for Single Failure in Erasure-Coded Distributed Storage Systems

In order to guarantee data reliability, erasure codes have been used in distributed storage systems. Nevertheless, this mechanism suffers from the repair problem that excess data are needed to repair a single failure, causing both high bandwidth consuming for the network and heavy computing load on the replacement node. To reduce repair traffic, researchers pointed out the tradeoff between storage and repair traffic and proposed regenerating codes by combining network coding. However, the combination only focuses on the storage terminal and the construction of the codes is quite complicated. Therefore, this paper further combines network coding with network structure and proposes a repair tree model based on general erasure codes to simplify the repair procedure. By decomposing repair computing and distributing it among the tree nodes, our model can mitigate the computing tension. The performance of repair tree is analyzed and evaluated by preliminary emulation. The result shows it can make about three times faster computing than conventional measure and the repair throughput is doubled if there are network bottlenecks. For proper topology, it can significantly reduce the repair traffic. We present algorithms to generate trees across the network topology. At last, we present the idea of extending repair tree to repair multiple failures.

Minimum Storage Regenerating Codes for Scalable Distributed Storage

Regenerating codes (RGCs) have recently been proposed to reduce the repair traffic of ( $n,k$ ) erasure-coded distributed storage systems. Moreover, RGCs can also be used in a scalable distributed storage scenario where $n$ is increased (decreased) to upgrade (degrade) redundancy while maintaining the maximum distance separable property of erasure codes. In this paper, we propose a new application of minimum storage regenerating (MSR) codes in storage scalability. The connection between repairing invalid nodes and adding new nodes suggests that the two processes can be unified in the same framework. We consider both single and multiple node situations, and two methods for constructing multiple nodes are proposed: concurrent and sequential. We focus on proving the achieved capability of concurrent MSR that can consume minimum traffic for generating multiple nodes. Because concurrent MSR is sensitive to both the number of helpers and added nodes, sequential methods make scalable MSR generalizable. The examples show that the scalable MSR codes have the same advantage of saving network traffic as repairing failures.