Reducing network cost of data repair in erasure-coded cross-datacenter storage

Abstract

Nowadays, cross-datacenter storage is widely used because of its high disaster-tolerance. Moreover, the erasure code is gradually being applied to cross-datacenter storage since it offers the same reliability as replication with a significant decrease in the amount of storage required. However, the network cost of repairing erasure-coded data in existing cross-datacenter storage (repair cost of cross-datacenter erasure code) is high, which usually results in a long repair time. To reduce the repair time by reducing the repair cost, in this paper, we study both the encoding method and repair method of cross-datacenter erasure code. First, we propose the concept of average weighted locality, which is proportional to the average repair cost of cross-datacenter erasure code. Then, we propose a network environment-adaptive encoding method of cross-datacenter erasure code (NEC). Under different network environments and encoding parameters, NEC can compute the approximate optimal generator matrix and data placement scheme online in a parallel heuristic way to achieve the approximate smallest average weighted locality. After that, we propose a hybrid-structured repair method of cross-datacenter erasure code (HRepair), which can achieve a good tradeoff between repair cost and repair efficiency by constructing a hybrid tree-star transmission topology to organize data transmission in the repair process. Experiments show that: Compared with several state-of-the-art erasure codes, NEC can reduce the repair cost and repair time by 26.8%–40% and 13.9%-37.1%. Besides, HRepair can reduce repair cost and repair time further.