Metadata Requests Research Articles

A Risk Management Framework for Large Scale Scrum using Metadata Outer Request Management Methodology

Recently, most software projects became naturally Distributed Agile Development (DAD) projects. The main benefits of DAD projects are cost-saving and being close to markets due to their distributed nature, such as in large-scale Scrum (LeSS). Developing LeSS projects leads to the emergence of challenges in risk management, especially the team collaboration challenges, where there is no standardized process for teams to communicate collaboratively. Team collaboration and the knowledge sharing is a vital resource for a large Scrum team's success. Hence, finding a dynamic technique that facilitates team collaboration in the LeSS environment is necessary. This paper proposes a risk management framework for LeSS using outer metadata requests. The proposed framework manages the outer requests amongst the distributed team. Therefore, it avoids missing team collaboration, risks, and threats to project completion. It also contributes to exchanging team skills and experience. The proposed framework is evaluated by applying it to two different case studies for large-scale Scrum projects. The evaluation results are given. The evaluation proved the effectiveness of the proposed framework.

A GPU-Accelerated In-Memory Metadata Management Scheme for Large-Scale Parallel File Systems

Driven by the increasing requirements of high-performance computing applications, supercomputers are prone to containing more and more computing nodes. Applications running on such a large-scale computing system are likely to spawn millions of parallel processes, which usually generate a burst of I/O requests, introducing a great challenge into the metadata management of underlying parallel file systems. The traditional method used to overcome such a challenge is adopting multiple metadata servers in the scale-out manner, which will inevitably confront with serious network and consistence problems. This work instead pursues to enhance the metadata performance in the scale-up manner. Specifically, we propose to improve the performance of each individual metadata server by employing GPU to handle metadata requests in parallel. Our proposal designs a novel metadata server architecture, which employs CPU to interact with file system clients, while offloading the computing tasks about metadata into GPU. To take full advantages of the parallelism existing in GPU, we redesign the in-memory data structure for the name space of file systems. The new data structure can perfectly fit to the memory architecture of GPU, and thus helps to exploit the large number of parallel threads within GPU to serve the bursty metadata requests concurrently. We implement a prototype based on BeeGFS and conduct extensive experiments to evaluate our proposal, and the experimental results demonstrate that our GPU-based solution outperforms the CPU-based scheme by more than 50% under typical metadata operations. The superiority is strengthened further on high concurrent scenarios, e.g., the high-performance computing systems supporting millions of parallel threads.

Defuse: Decoupling Metadata and Data Processing in FUSE Framework for Performance Improvement

A popular user-space file system framework, FUSE, has been widely used for building various customized file systems (cFS) on top of the underlying kernel file system (kFS). A FUSE-based cFS gains adequate flexibility by developing its specific functions in user space, but brings extra user-kernel mode switches in the request processing flow owing to forwarding all requests from the FUSE kernel driver to the user-space daemon, thus degrading the overall performance. We observe that a file data request does not need to forward to the user-space daemon when its file-to-file mapping between the FUSE-based cFS and the underlying kFS remains unchanged. Based on the insight, we propose a modified FUSE framework - DeFUSE that decouples the processing flow of the metadata and data requests. The metadata requests still follow the original flow to reserve flexibility while the data requests are directly executed in the DeFUSE kernel driver maintaining the file-to-file mappings in the kernel, effectively eliminating the unnecessary mode switches. We have implemented the DeFUSE framework and ported three representative FUSE-based cFSs to DeFUSE. The result shows that for data-centric workloads, the throughput of DeFUSE-based cFSs increases up to 3.5X for write and 3.8X for read respectively, compared to their corresponding FUSE-based implementations. DeFUSE is available on Github.

MetaFlow: A Scalable Metadata Lookup Service for Distributed File Systems in Data Centers

In large-scale distributed file systems, efficient meta- data operations are critical since most file operations have to interact with metadata servers first. In existing distributed hash table (DHT) based metadata management systems, the lookup service could be a performance bottleneck due to its significant CPU overhead. Our investigations showed that the lookup service could reduce system throughput by up to 70%, and increase system latency by a factor of up to 8 compared to ideal scenarios. In this paper, we present MetaFlow, a scalable metadata lookup service utilizing software-defined networking (SDN) techniques to distribute lookup workload over network components. MetaFlow tackles the lookup bottleneck problem by leveraging B-tree, which is constructed over the physical topology, to manage flow tables for SDN-enabled switches. Therefore, metadata requests can be forwarded to appropriate servers using only switches. Extensive performance evaluations in both simulations and testbed showed that MetaFlow increases system throughput by a factor of up to 3.2, and reduce system latency by a factor of up to 5 compared to DHT-based systems. We also deployed MetaFlow in a distributed file system, and demonstrated significant performance improvement.

Optimal metadata replications and request balancing strategy on cloud data centers

In large-scale cloud data centers, metadata accesses will very likely become a severe performance bottleneck as metadata-based transactions account for over 50% of all file system operations. Clusters of Metadata Servers (MDS) that provide metadata searching service can improve the system performance significantly. For a data stored in cloud data centers, there may be several MDS storing the metadata replicas. Therefore, when a data request arrives, it has many potential metadata paths, one of which shall be chosen to obtain the best performance. In this paper, we attempt to determine the number of MDS that each data object in the system shall have and the request rates that each MDS shall serve, in order to achieve the minimum mean response time (MRT) of all the metadata requests. The target optimal constrained function has been formulated and a novel metadata request balancing algorithm based on request arrival rates has been proposed, which can find near-optimal solutions by a theoretical proof. In our experiments, we compare our algorithm with widely used hashing functions that have 0, 1, 2, 3 replicas, respectively. We validate our findings via simulations with respect to several influencing factors and prove that our proposed strategy is scalable, flexible and efficient for the real-life applications. Some interesting perspectives of the work are also presented at the end of this paper.

Log-less metadata management on metadata server for parallel file systems.

This paper presents a novel metadata management mechanism on the metadata server (MDS) for parallel and distributed file systems. In this technique, the client file system backs up the sent metadata requests, which have been handled by the metadata server, so that the MDS does not need to log metadata changes to nonvolatile storage for achieving highly available metadata service, as well as better performance improvement in metadata processing. As the client file system backs up certain sent metadata requests in its memory, the overhead for handling these backup requests is much smaller than that brought by the metadata server, while it adopts logging or journaling to yield highly available metadata service. The experimental results show that this newly proposed mechanism can significantly improve the speed of metadata processing and render a better I/O data throughput, in contrast to conventional metadata management schemes, that is, logging or journaling on MDS. Besides, a complete metadata recovery can be achieved by replaying the backup logs cached by all involved clients, when the metadata server has crashed or gone into nonoperational state exceptionally.

Supporting Scalable and Adaptive Metadata Management in Ultralarge-Scale File Systems

This paper presents a scalable and adaptive decentralized metadata lookup scheme for ultralarge-scale file systems (more than Petabytes or even Exabytes). Our scheme logically organizes metadata servers (MDSs) into a multilayered query hierarchy and exploits grouped Bloom filters to efficiently route metadata requests to desired MDSs through the hierarchy. This metadata lookup scheme can be executed at the network or memory speed, without being bounded by the performance of slow disks. An effective workload balance method is also developed in this paper for server reconfigurations. This scheme is evaluated through extensive trace-driven simulations and a prototype implementation in Linux. Experimental results show that this scheme can significantly improve metadata management scalability and query efficiency in ultralarge-scale storage systems.