Efficient in-network aggregation mechanism for data block repairing in data centers

Abstract

Many distributed storage systems in data centers, e.g., Google Colossus SF, Facebook HDFS, and Microsoft Azure, adopt erasure codes to improve storage reliability and reduce the storage space overhead. Accordingly, when a block of data fails, multiple involved blocks are sent to a new node to regenerate the failed block. Furthermore, repairing those failed data blocks brings a large amount of bandwidth consumption. Meanwhile, prior repairing methods only focus on how to aggregate data blocks on involved storage nodes, which fails to efficiently save the network bandwidth. By contrast, our observations show that the aggregation is not limited to those storage nodes. Specifically, servers and switches can all be utilized to aggregate those data blocks during the transmission phase. Therefore, in this paper, we propose an efficient in-network aggregation mechanism, called AggreTree, which can effectively leverage the intermediate nodes along the transmission paths to aggregate data blocks, thus repairing failed blocks. Compared with existing methods, the results of experiments show that AggreTree can efficiently improve the repairing speed of failed blocks by up to 4.06 and 2.86 times, and further reduce the transmission cost by 80.86% and 81.68% in Fat-tree and BCube data centers, respectively.