WPAR: A Weight-Based Metadata Management Strategy for Petabyte-Scale Object Storage Systems

Abstract

In an object-based network storage system, metadata access is decoupled from the data transferring path to improve system performance, scalability, and management. Designing an efficient metadata management scheme in such a system is critically important to the overall system performance and poses lots of challenges to the system designers. Traditional metadata management schemes, either partitioning the metadata statically, or using pure hashing methods, are not able to adapt to the dynamic metadata workload, and hence suffer from scalability and hotspot problems. In this paper, we propose a new metadata management strategy, referred to as Weighted Partitioning and Adaptive Replication (WPAR) scheme, taking into account the weight (CPUpower, memory capacity, network bandwidth, etc.) of each metadata server. Our objective is to efficiently distribute and dynamically replicate the metadata, based on the weights, to balance the workload and reduce hotspots within the metadata server cluster.