TA-Update: An Adaptive Update Scheme with Tree-Structured Transmission in Erasure-Coded Storage Systems

Abstract

Erasure coding has received considerable attentions due to the better tradeoff between the space efficiency and reliability. The frequent update of the stored data in the distributed storage systems has posed a new challenge for erasure codes: how to update the erasure-coded data in a general, efficient and adaptive way. However, existing update schemes of erasure codes are inadequate to meet these requirements, since their code-related update manners lead to a low generality, their star-structured data transmission manners lead to a low update efficiency, and their redo manners when encountering the node failure lead to a low adaptivity. In this paper, we propose an adaptive update scheme with the tree-structured transmission, called TA-Update, which consists of a code-independent update framework and three algorithms: the rack-aware tree construction algorithm, the top-down data processing algorithm and the rollback-based failure processing algorithm. For generality, we propose a code-independent update framework with the tree structure to support the MDS code with any coding parameter. For efficiency, a rack-aware tree construction algorithm is proposed to achieve the high available bandwidth, which organizes the data node and parity nodes as an update tree. Moreover, a top-down data processing algorithm is proposed to achieve the high transmission and computation efficiency, which pipelines the data transmission along the update tree and distributes the encoding computations among all the participating nodes. For adaptivity, we propose a rollback-based failure processing algorithm to achieve high adaptivity, which handles the node failure during update with the existing update tree in a rollback manner. To evaluate the performance of TA-Update, we conduct experiments on HDFS-RAID under various parameter settings on both 30 physical and 200 virtual machines. Extensive experiments confirm that TA-Update could support the various erasure codes with any parameter, improve the update efficiency by 30 percent and the adaptivity by 47 percent on average compared with the state-of-the-art approaches under various parameter settings.