Robot: An efficient model for big data storage systems based on erasure coding

Abstract

It is well-known that with the explosive growth of data, the age of big data has arrived. How to save huge amounts of data is of great importance to both industry and academia. This paper puts forward a solution based on coding technologies in big data system that store a lot of cold data. By studying existing coding technologies and big data systems, we can not only maintain the system's reliability, but also improve the security and the utilization of storage systems. Due to the remarkable reliability and space saving rate of coding technologies, importing coding schema in to big data systems becomes prerequisite. In our presented schema, the storage node is divided into several virtual nodes to keep load balancing. By setting up different virtual node storage groups for different codec server, we can ensure system availability. And by utilizing the parallel decoding computing of the node and the block of data, we can also reduce the system recovery time when data is corrupted. Additionally, different users set different coding parameters can improve the robustness of big data storage systems. We configure various data block m and calibration block k to improve the utilization rate in the quantitative experiments. The results shows that parallel decoding speed can rise up two times than the past serial decoding speed. The encoding efficiency with ICRS coding is 34.2% higher than using CRS and 56.5% more than using RS coding equally. The decoding rate by using ICRS is 18.1% higher than using CRS and 31.1% higher than using RS averagely.