Metadata Workloads Research Articles

Managing Rich Metadata in High-Performance Computing Systems Using a Graph Model

High-performance computing (HPC) systems generate huge amounts of metadata about different entities such as jobs, users, and files. Existing systems can efficiently record and manage part of these metadata, mainly the POSIX metadata of data files (e.g., file size, name, and permissions mode). But another important set of metadata, referred to as “rich” metadata in this study, which record not only wider range of entities (e.g., running processes and jobs) but also more complex relationships between them, are mostly missing in current HPC systems. Yet such rich metadata are critical for supporting many advanced data management functions such as identifying data sources and parameters behind a given result; auditing data usage; or understanding details about how inputs are transformed into outputs. To uniformly and efficiently manage the rich metadata generated in HPC systems, We propose to utilize a graph model in this study. We identify the key challenges of implementing such a graph-based HPC rich metadata management system and present GraphMeta, a graph-based rich metadata management system designed and optimized for HPC platforms, to tackle these challenges. Extensive evaluations on both synthetic and real HPC metadata workloads show its advantages in both performance and scalability compared with existing solutions.

Leveraging OSD+ devices for implementing a high‐throughput parallel file system

SummaryOSD+s are enhanced object‐based storage devices (OSDs) able to deal with both data and metadata operations via data and directory objects, respectively. So far, we have focused on designing and implementing efficient directory objects in OSD+s. This paper, however, presents our work on also supporting data objects and describes how the coexistence of both kinds of objects in each OSD+ is profited to efficiently implement data objects and to speed up some common file operations. We compare our OSD+‐based Fusion Parallel File System (FPFS) with Lustre and OrangeFS through different microbenchmarks and HPCS‐IO scenarios. Results show that FPFS provides a throughput up to 37× better than Lustre and up to 95× better than OrangeFS for metadata workloads. FPFS also provides 34% more bandwidth than OrangeFS for data workloads and competes with Lustre in data writes. Results also show serious scalability problems in Lustre and OrangeFS that limit their performance.

DMFSsim: A Distributed Metadata File System Simulator

Parallel file systems deploy multiple metadata servers to distribute heavy metadata workload from clients. With the increasing number of metadata servers, metadata-intensive operations are facing some problems related with collaboration among them, compromising the performance gain. Consequently, a file system simulator is very helpful to try out some optimization ideas to solve these problems. In this paper, we propose DMFSsim to simulate the metadata-intensive operations on large-scale distributed metadata file systems. DMFSsim can flexibly replay traces of multiple metadata operations, support several commonly used metadata distribution algorithms, simulate file system tree hierarchy and underlying disk blocks management mechanism in real systems. Extensive simulations show that DMFSsim is capable of demonstrating the performance of metadata-intensive operations in distributed metadata file system.

MHS: A distributed metadata management strategy

This paper proposes a novel distributed metadata management strategy to efficiently handle different metadata workloads. It can deliver high performance and scalable metadata service through four techniques, including directory conversion metadata, mimic hierarchical directory structure, flexible partition methods targeted different kinds of metadata of diverse characteristics, and the application of database to metadata backend. Using micro-benchmarks and a prototype system, we firstly demonstrate the performance superiority of our strategy compared to Lazy Hybrid, and then present the detailed performance results and analysis of our strategy on different MDS scales.