A multi‐channel architecture for metadata management in cloud storage systems by binding CPU‐cores to disks

Abstract

SummaryMetadata operations have become dominant file operations in the storage systems. In the scenarios of read‐more and write‐less of massive small files, the current distributed file systems suffer from the unsatisfying performance and scalability of metadata service because of random disk I/O during metadata operations. In this paper, a highly efficient metadata management architecture for cloud storage systems is proposed. The cluster design significantly improves the scalability of the system. A concept of the disk I/O channel is introduced, which is an independent data storage pipe by binding an independent CPU‐core to each physical disk. In addition, a multi‐channel fast key‐value storage engine is proposed to provide the extremely efficient performance for the underlying storage service, which takes full advantages of multi‐core processors and parallel disks I/O. Besides, a new dynamic load‐balancing strategy is proposed to reduce load thrashing and improve the precision of rebalancing among the clusters. Performance measurements under a variety of benchmarks show that the metadata management is capable of handling the massive small files storage and the performance is improved significantly compared to the existing solutions. Copyright © 2016 John Wiley & Sons, Ltd.