Practical Performance Evaluation of Space Optimal Erasure Codes for High-Speed Data Storage Systems

Abstract

As erasure codes have been widely adopted in most large-scale data storage systems and applications, implementations of high-performance erasure codes have been improved significantly in recent years, especially by employing Intel’s Streaming SIMD Extensions (SSE) instructions. Augmenting the survey work in Plank et al. (Fast, 9:253–65, 2009) conducted almost a decade ago, this paper compares practical performance of three open-source or public domain erasure coding libraries, namely Jerasure and Intel’s ISA-L for RS code, and a STAR code implementation. The goal of this paper is to provide data storage practitioner a guideline when they choose a proper erasure code for storage applications and systems that need high performance in encoding and decoding operations in the order of GBs/s. Additionally, this paper identifies a practical technique that can further improve decoding performance of RS code greatly for both Jerasure and ISA-L for the most frequent disk failure pattern, i.e., one disk failure.