Software and Hardware Integrated Accelerators for Hadoop Appliance

Abstract

With the development of the Internet, the data center for next-generation must be capable of data processing at PB/s level, ensuring a high throughput of the storage and network. This paper displays the design of the hardware accelerator for tuning the throughput of the storage and network. First, we illustrate the importance of a lossless data compression algorithm for data-intense applications based on the analysis of the characteristics of the Hadoop distributed system. Second, we effectively implement Deflate compression algorithm on the FPGA. The experimental results show that the compression speed of the Deflate compression algorithm hardware accelerator can reach 2.44 Gb/S, and the compression ratio is 2.08 on the Calgary standard evaluation set. Finally, we propose a hardware/software integration architecture to combine Deflate compression algorithm hardware accelerator and Hadoop distributed system. This architecture also incorporated the unique design of semaphore and shared-memory mechanism. The overall design can double the performance of a single mapping process and retain good scalability.