Large-scale Distributed File Systems Research Articles

An Adaptive Metadata Management Scheme Based on Deep Reinforcement Learning for Large-Scale Distributed File Systems

A major challenge confronting today’s distributed metadata management schemes is how to meet the dynamic requirements of various applications through effectively mapping and migrating metadata nodes to different metadata servers (MDS’s). Most of the existing works dynamically reallocate nodes to different servers adopting history-based coarse-grained methods, failing to make a timely and efficient update on the distribution of nodes. In this paper, we present the first deep reinforcement learning-leveraged distributed metadata management scheme, AdaM, to address the aforementioned dilemma. AdaM is an adaptive fine-grained metadata management scheme that trains an actor-critic network to migrate “hot” metadata nodes to different MDS’s based on its observations of the current “states” (i.e., access pattern, the structure of namespace tree and current distribution of nodes on MDS’s). Adaptive to varying access patterns, AdaM can automatically migrate hot metadata nodes among servers to keep load balancing while maintaining metadata locality. Besides, we propose a self-adaptive metadata cache policy, which dynamically combines the two strategies of managing caches on the server side and the client side to gain better query performance. Last but not least, we design a distributed metadata processing 2PC Protocol called MST-based 2PC to ensure data consistency. Experiments on a real-world dataset demonstrate the superiority of our proposed method over other schemes.

A Literature Review of Big Data and Metadata Management Systems

Abstract: Metadata is essential for the successful operation of data files, as it acts as the underlying data that helps maintain and manage the namespace, permissions, and address of the file data blocks. The performance of metadata services has a direct impact on the overall performance of the file system, as these operations can account for a significant portion, up to 80%, of file system activity. Large-scale distributed file systems (DFS) have emerged to meet the rising demand for data storing. These systems provide users with access to numerous storage units spread across various locations. The widespread use of large-scale DFSs as the foundation for a range of computing systems and applications, particularly in the era of Big Data, highlights the importance of efficient metadata management. This paper seeks to offer an enhanced knowledge of the current status of metadata systems in large-scale DFSs, this paper presents a comprehensive overview of the latest research in the field. DFSs are defined by three aspects: scalability, performance, and availability. Moreover, the study evaluates various challenges and limitations in the current research, providing valuable insight for future studies. Overall, the study attempts to give a complete knowledge of the present scenario of metadata services across large-scale DFSs and to recommend areas for future research and development.

The State of the Art of Metadata Managements in Large-Scale Distributed File Systems — Scalability, Performance and Availability

File system metadata is the data in charge of maintaining namespace, permission semantics and location of file data blocks. Operations on the metadata can account for up to 80% of total file system operations. As such, the performance of metadata services significantly impacts the overall performance of file systems. A large-scale distributed file system (DFS) is a storage system that is composed of multiple storage devices spreading across different sites to accommodate data files, and in most cases, to provide users with location independent access interfaces. Large-scale DFSs have been widely deployed as a substrate to a plethora of computing systems, and thus their metadata management efficiency is crucial to a massive number of applications, especially with the advent of the Big Data age, which poses tremendous pressure on underlying storage systems. This paper reports the state-of-the-art research on metadata services in large-scale distributed file systems, which is conducted from three indicative perspectives that are always used to characterize DFSs: high-scalability, high-performance, and high-availability, with special focus on their respective major challenges as well as their developed mainstream technologies. Additionally, the paper also identifies and analyzes several existing problems in the research, which could be used as a reference for related studies.

Design and implementation of dynamic I/O control scheme for large scale distributed file systems

In this work, we have analyzed the input/output (I/O) activities of Cori, which is a high-performance computing system at the National Energy Research Scientific Computing Center at Lawrence Berkeley National Laboratory. Our analysis results indicate that most users do not adjust storage configurations but rather use the default settings. In addition, owing to the interference from many applications running simultaneously, the performance varies based on the system status. To configure file systems autonomously in complex environments, we developed DCA-IO, a dynamic distributed file system configuration adjustment algorithm that utilizes the system log information to adjust storage configurations automatically. Our scheme aims to improve the application performance and avoid interference from other applications without user intervention. Moreover, DCA-IO uses the existing system logs and does not require code modifications, an additional library, or user intervention. To demonstrate the effectiveness of DCA-IO, we performed experiments using I/O kernels of real applications in both an isolated small-sized Lustre environment and Cori. Our experimental results shows that our scheme can improve the performance of HPC applications by up to 263% with the default Lustre configuration.

AIOC[formula omitted]: A deep Q-learning approach to autonomic I/O congestion control in Lustre

In high performance computing systems, I/O congestion is a common problem in large-scale distributed file systems. However, the current implementation mainly requires administrator to manually design low-level implementation and optimization, we proposes an adaptive I/O congestion control framework, named AIOC2, which can not only adaptively tune the I/O congestion control parameters, but also exploit the deep Q-learning method to start the training parameters and optimize the tuning for different types of workloads from the server and the client at the same time. AIOC2 combines the feedback-based dynamic I/O congestion control and deep Q-learning parameter tuning technology to achieve autonomic I/O congestion control, improve system I/O throughput, and thus reduce I/O latency without human interference. Experimental results show that AIOC2 can greatly reduce the impact of I/O congestion on I/O throughput and I/O latency performance in Lustre clusters. Compared to existing Lustre cluster systems, AIOC2 can increase write I/O throughput by 34.82% and decrease I/O latency by 26.17% on average.

A Highly Reliable Metadata Service for Large-Scale Distributed File Systems

Many massive data processing applications nowadays often need long, continuous, and uninterrupted data accesses. Distributed file systems are used as the back-end storage to provide the global namespace management and reliability guarantee. Due to increasing hardware failures and software issues with the growing system scale, metadata service reliability has become a critical issue as it has a direct impact on file and directory operations. Existing metadata management mechanisms can provide fault tolerance capability to some level but are inadequate. They often have limitations in system availability, state consistence, and performance overhead and lack an effective mechanism to offer metadata reliability. This paper introduces a novel highly reliable metadata service to address these issues in large-scale file systems. Different from traditional strategies, this proposed reliable metadata service adopts a new active-standby architecture for fault tolerance and uses a holistic approach to improve file system availability. A new shared storage pool (SSP) is designed for transparent metadata synchronization and replication between active and standby servers. Based on the SSP, a new policy called multiple actives multiple standbys (MAMS) is presented to perform metadata service recovery in case of failures. A new global state recovery strategy and a smart client fault tolerance mechanism are achieved to maintain the continuity of metadata service. We have implemented such highly reliable metadata service in a prototype file system CFS (Clover file system) and conducted extensive tests to evaluate it. Experimental results confirm that it can significantly improve file system reliability with fast failover under different failure scenarios while having negligible influence on performance. Compared with typical reliability designs in Hadoop Avatar, Hadoop HA, and Boom-FS file systems, the mean-time-to-recovery (MTTR) with the highly reliable metadata service was reduced by 80.23, 65.46 and 28.13 percent, respectively.

Challenges and Research Disputes and Tools in Big Data Analytics

Big Data is the era of data processing. Big Data is the Collate’s observer data sets that are complicated that traditional data-processing abilities.There are the various challenges include data analysis, capture the data, curation, search, sharing, stowage, transmission, visualization, and privacy violations. A large collections of petabytes of data is engendered day by day from the up-to-date information systems and digital era such as Internet of Things and cloud computing. Big data environs is used to attain, organize and analyse the numerous types of data. A large scale distributed file system which should be a fault tolerant, flexible and scalable. The term big data comes with the new challenges to input, process and output the data The technologies used by big data application to handle the massive data are Hadoop, Map Reduce, Pig, Apache Hive, No SQL and Spark. Initially, we extant the definition of big data and discuss big data challenges. Succeeding, The Propionate Paramour of Big Data Systems Models in the Into Prolonging Seam, Namely data Generation, data Assange, data Storage, and data Analytics. These four modules form a big data value chain. In accumulation, we present the prevalent Hadoop framework for addressing big data.

MetaFlow: A Scalable Metadata Lookup Service for Distributed File Systems in Data Centers

In large-scale distributed file systems, efficient meta- data operations are critical since most file operations have to interact with metadata servers first. In existing distributed hash table (DHT) based metadata management systems, the lookup service could be a performance bottleneck due to its significant CPU overhead. Our investigations showed that the lookup service could reduce system throughput by up to 70%, and increase system latency by a factor of up to 8 compared to ideal scenarios. In this paper, we present MetaFlow, a scalable metadata lookup service utilizing software-defined networking (SDN) techniques to distribute lookup workload over network components. MetaFlow tackles the lookup bottleneck problem by leveraging B-tree, which is constructed over the physical topology, to manage flow tables for SDN-enabled switches. Therefore, metadata requests can be forwarded to appropriate servers using only switches. Extensive performance evaluations in both simulations and testbed showed that MetaFlow increases system throughput by a factor of up to 3.2, and reduce system latency by a factor of up to 5 compared to DHT-based systems. We also deployed MetaFlow in a distributed file system, and demonstrated significant performance improvement.

Parallel network file systems using authenticated key exchange protocols

The keyestablishment for secure many-to-many communications is very important nowadays. The problem is inspired by the proliferation of large-scale distributed file systems supporting parallel access to multiple storage devices. In this, a variety of authenticated key exchange protocols that are designed to address the issues. This shows that these protocols are capable of reducing the workload of the metadata server and concurrently supporting forward secrecy and escrow-freeness. All this requires only a small fraction of increased computation overhead at the client. This proposed three authenticated key exchange protocols for parallel network file system (pNFS). The protocols offer three appealing advantages over the existing Kerberos-based protocol. First, the metadata server executing these protocols has much lower workload than that of the Kerberos-based approach. Second, two of these protocols provide forward secrecy: one is partially forward secure (with respect to multiple sessions within a time period), while the other is fully forward secure (with respect to a session). Third, designed a protocol which not only provides forward secrecy, but is also escrow-free.

AUTHENTICATED KEY EXCHANGE PROTOCOLS FOR PARALLEL NETWORK FILE SYSTEMS

We study the problem of key establishment for secure many-to-many communications. The problem is inspired by the proliferation of large-scale distributed file systems supporting parallel access to multiple storage devices. Our work focuses on the current Internet standard for such file systems, i.e. , parallel Network File System (pNFS), which makes use of Kerberos to establish parallel session keys between clients and storage devices. Our review of the existing Kerberos-based protocol shows that it has a number of limitations: (i) a metadata server facilitating key exchange between the clients and the storage devices has heavy workload that restricts the scalability of the protocol; (ii) the protocol does not provide forward secrecy; (iii) the metadata server generates itself all the session keys that are used between the clients and storage devices, and this inherently leads to key escrow. In this paper, we propose a variety of authenticated key exchange protocols that are designed to address the above issues. We show that our protocols are capable of reducing up to approximately 54% of the workload of the metadata server and concurrently supporting forward secrecy and escrow-freeness. All this requires only a small fraction of increased computation overhead at the client.

Authenticated Key Exchange Protocols for Parallel Network File Systems

We study the problem of key establishment for secure many-to-many communications. The problem is inspired by the proliferation of large-scale distributed file systems supporting parallel access to multiple storage devices. Our work focuses on the current Internet standard for such file systems, i.e., parallel Network File System (pNFS), which makes use of Kerberos to establish parallel session keys between clients and storage devices. Our review of the existing Kerberos-based protocol shows that it has a number of limitations: (i) a metadata server facilitating key exchange between the clients and the storage devices has heavy workload that restricts the scalability of the protocol; (ii) the protocol does not provide forward secrecy; (iii) the metadata server generates itself all the session keys that are used between the clients and storage devices, and this inherently leads to key escrow. In this paper, we propose a variety of authenticated key exchange protocols that are designed to address the above issues. We show that our protocols are capable of reducing up to approximately 54 percent of the workload of the metadata server and concurrently supporting forward secrecy and escrow-freeness. All this requires only a small fraction of increased computation overhead at the client.

Secure Dynamic Fragment and Replica Allocation in Large-Scale Distributed File Systems

The Division and Replication of Data in the Cloud for Optimal Performance and Security (DROPS) that collectively approaches the security and performance issues. Data to a third-party administrative control, as is done in cloud computing, gives rise to security concerns. The data compromise may occur due to attacks by other users and nodes within the cloud. Therefore, high security measures are required to protect data within the cloud. To protected the data in cloud by using fragmentation and replication. In DROPS methodology, divide a file into fragments, and replicate the fragmented data over the cloud nodes. Each of the nodes stores only a single fragment of a particular data file that ensures that even in case of a successful attack, no meaningful information is revealed to the attacker.

Improving reliability and performances in large scale distributed applications with erasure codes and replication

Replication of Data Blocks is one of the main technologies on which Storage Systems in Cloud Computing and Big Data Applications are based. With the heterogeneity of nodes, and an always-changing topology, keeping the reliability of the data contained in the common large-scale distributed file system is an important research challenge. Common approaches are based either on replication of data or erasure codes. The former stores each data block several times in different nodes of the considered infrastructures: the drawback is that this can lead to large overhead and non-optimal resources utilization. Erasure coding instead exploits Maximum Distance Separable codes that minimize the information required to restore blocks in case of node failure: this approach can lead to increased complexity and transfer time due to the fact that several blocks, coming from different sources, are required to reconstruct lost information. In this paper we study, by means of discrete event simulation, the performances that can be obtained by combining both techniques, with the goal of minimizing the overhead and increasing the reliability while keeping the performances. The analysis proves that a careful balance between the application of replication and erasure codes significantly improves reliability and performances avoiding large overheads with respect to the isolated use of replication and redundancy.

Efficient and Scalable Metadata Management in EB-Scale File Systems

Efficient and scalable distributed metadata management is critically important to overall system performance in large-scale distributed file systems, especially in the EB-scale era. Hash-based mapping and subtree partitioning are state-of-the-art distributed metadata management schemes. Hash-based mapping evenly distributes workload among metadata servers, but it eliminates all hierarchical locality of metadata. Subtree partitioning does not uniformly distribute workload among metadata servers, and metadata needs to be migrated to keep the load balanced roughly. Distributed metadata management is relatively difficult since it has to guarantee metadata consistency. Meanwhile, scaling metadata performance is more complicated than scaling raw I/O performance. The complexity further rises with distributed metadata. It results in a primary goal that is to improve metadata management scalability while paying attention to metadata consistency. In this paper, we present a ring-based metadata management mechanism named Dynamic Ring Online Partitioning (DROP). It can preserve metadata locality using locality-preserving hashing, keep metadata consistency, as well as dynamically distribute metadata among metadata server cluster to keep load balancing. By conducting performance evaluation through extensive trace-driven simulations and a prototype implementation, experimental results demonstrate the efficiency and scalability of DROP.

Study on Storage System Performance Evaluation in Complex Application Environment

At present, there is not any technology that can fully reflect the overall performance of various storage systems. In order to accurately evaluate storage performance in complex application environment, in this paper, “Digital City” is adopted as the load prototype. Combined with the existing simulation and Benchmark measuring, a comprehensive evaluation index system is built and comprehensive evaluation platform of storage system performance is designed and implemented. In addition, the aggregate bandwidth performance of large-scale distributed file system using this evaluation platform is introduced. The analysis and experiment show that the proposed comprehensive evaluation system can be universal evaluation platform of storage system performance and give help for storage research and application in the complex application environment.

FARMER: A novel approach to file access correlation mining and evaluation reference model

File semantic has proven effective in optimizing large scale distributed file system. As a consequence of the elaborate and rich I/O interfaces between upper layer applications and file systems, file system can provide useful and insightful information about semantic. Hence, file semantic mining has become an increasingly important practice in both engineering and research community. Unfortunately, it is a challenge to exploit file semantic knowledge because a variety of factors could affect this information exploration process. Even worse, the challenges are exacerbated due to the intricate interdependency between these factors, and make it difficult to fully exploit the potentially important correlation among various semantic knowledges. This article proposes a file access correlation miming and evaluation reference (FARMER) model, where file is treated as a multivariate vector space, and each item within the vector corresponds a separate factor of the given file. The selection of factor depends on the application, examples of factors are file path, creator and executing program. If one particular factor occurs in both files, its value is non-zero. It is clear that the extent of inter-file relationships can be measured based on the likeness of their factor values in the semantic vectors. Benefit from this model, FARMER represents files as structured vectors of identifiers, and basic vector operations can be leveraged to quantify file correlation between two file vectors. FARMER model leverages linear regression model to estimate the strength of the relationship between file correlation and a set of influencing factors so that the “bad knowledge” can be filtered out. To demonstrate the ability of new FARMER model, FARMER is incorporated into a real large-scale object-based storage system as a case study to dynamically infer file correlations. In addition FARMER-enabled optimize service for metadata prefetching algorithm and object data layout algorithm is implemented. Experimental results show that is FARMER-enabled prefetching algorithm is shown to reduce the metadata operations latency by approximately 30%–40% when compared to a state-of-the-art metadata prefetching algorithm and a commonly used replacement policy.

Design and Implementation of a Metadata Management Scheme for Large Distributed File Systems

Most large-scale distributed file systems decouple a metadata operation from read and write operations for a file. In the distributed file systems, a certain server named a metadata server (MDS) is responsible for maintaining the metadata information of the file systems. In this paper, we propose a new metadata management scheme in order to provide the high metadata throughput and scalability for a cluster of MDSs. First, we derive a new metadata distribution technique. Then, we present a load balancing technique based on the distribution technique. Several experiments show that our scheme outperforms existing metadata management scheme in terms of scalability and load balancing.

Guest editorial: Special section on security in distributed computing systems

ESEARCH in the area of security of computing systems has always been of utmost importance, particularly in recent years as orchestrated attacks have sought to cripple critical infrastructures. Security issues in distributed computing systems involve reducing vulnerabilities as well as giving system management the insight and control needed to defend distributed information systems. All aspects of business and government operations and services are dependent upon the security and integrity of our information and communication infrastructures. Critical sectors such as energy, health, transportation, social and emergency services, manufacturing, processing, and logistics and distribution functions rely almost entirely on data processing and interchange. The recent attacks on government and commercial computers indicate that “hacking” has gone beyond the realm of domestic pranks and has entered the domain of state sponsored cyber-terrorism. Clearly, these information security threats to vital national infrastructures present risks of debilitating impact on the defense and economic security of the nation and the state. As such, these issues pose new challenges to researchers. Advances are rapidly being made in the following areas: anonymity and camouflaging, intrusion detection, dynamic coalition, distributed denial of service, security protocols, authentication and verification, and mobile code and agent security. This special section of IEEE Transactions on Parallel and Distributed Systems is targeted at related issues in security of distributed computing systems. Response to the call for papers was overwhelming. From these excellent submissions, seven were accepted for publication in this special section. The first paper by M. Ahamad, S. Lakshmanan, and H. Venkateswaran explores the subject of responsive security for stored data. X. Zhang, L. Xiao, and Z. Xu discuss low-cost and reliable mutual anonymity protocols in peer-to-peer networks. Integrated access control and intrusion detection for Web servers is explained by T. Ryutov, C. Neuman, D. Kim, and L. Zhou. The paper by L. Wang, X. Zhao, D. Pei, R. Bush, D. Massey, and L. Zhang explores ways to protect BGP routes to top-level DNS servers. J. Xu and M. Sung introduce a technique for defending against internet DDoS attacks. The next paper, by H. Wang and K. Shin, describes a built-in protection mechanism to counter DDoS attacks, known as TransportAware IP Routers. The final paper from A. Mei, L. Mancini, and S. Jajodia discusses a secure dynamic fragment and replica allocation in large-scale distributed file systems. Finally, I would like to take this time to thank the contributing authors, reviewers, and the Editor-in-Chief, Dr. Pen-Chung Yew. Additional thanks goes to Mr. Shengquan Wang for Web management and Ms. Larisa Archer for administration of the review process. Without their contributions and support, this special section would not be possible.

Secure dynamic fragment and replica allocation in large-scale distributed file systems

We present a distributed algorithm for file allocation that guarantees high assurance, availability, and scalability in a large distributed file system. The algorithm can use replication and fragmentation schemes to allocate the files over multiple servers. The file confidentiality and integrity are preserved, even in the presence of a successful attack that compromises a subset of the file servers. The algorithm is adaptive in the sense that it changes the file allocation as the read-write patterns and the location of the clients in the network change. We formally prove that, assuming read-write patterns are stable, the algorithm converges toward an optimal file allocation, where optimality is defined as maximizing the file assurance.

HCCM

This paper presents a novel Hierarchical Cache Consistence Mechanism -- HCMM. HCMM is specially designed for the large scale distributed file system in WAN and can well adapt to work in WAN. HCMM uses hierarchical method to manage the consistence of caches, and can minimize the overhead of network communication and cache management. This paper also sets up a theory model to analyze the performance of HCMM. The results of theoretical analysis prove that the performance of HCMM is better than conventional cache consistence mechanism.