Modern Distributed Storage Systems Research Articles

A Learning-based Data Placement Framework for Low Latency in Data Center Networks

Abstract: The challenge of achieving low-latency data services has become increasingly crucial for data center applications. In modern distributed storage systems, efficient data placement plays a vital role in minimizing data movement delays, thereby significantly reducing service latency. While previous solutions for data placement have often relied on pre-assumed data request distributions or traced analysis, the dynamic nature of network conditions and evolving user request patterns poses a complex online decision-making challenge. Traditional static model-based approaches are limited in their ability to cope with such dynamic system dynamics. To address these challenges and consider both data movement and analytical latency, we introduce a novel framework named DataBot+. This framework leverages reinforcement learning techniques to autonomously learn optimal data placement policies. DataBot+ employs neural networks that are trained using a modified version of Q-learning. The input to these networks comprises real-time data flow measurements, while the output is a value function that estimates the imminent future latency. To ensure timely decision-making, DataBot+ is designed with a two-fold asynchronous architecture, consisting of production and training components. This separation guarantees that the training process does not introduce additional overhead to the handling of data flows. Our approach is validated through extensive evaluations using real-world traces, which convincingly demonstrate the efficacy of our proposed design.

A Learning-Based Data Placement Framework for Low Latency in Data Center Networks

Low-latency data service is an increasingly critical challenge for data center applications. In modern distributed storage systems, proper data placement helps reduce the data movement delay, which can contribute to the service latency reduction tremendously. Existing data placement solutions have often assumed the prior distribution of data requests or discovered it via trace analysis. However, data placement is a difficult online decision-making problem faced with dynamic network conditions and time-varying user request patterns. The conventional static model-based solutions are less effective to handle the dynamic system. With an overall consideration of data movement and analytical latency, we develop a reinforcement learning-based framework DataBot+, automatically learning the optimal placement policies. DataBot+ adopts neural networks, trained with a variant of <inline-formula><tex-math notation="LaTeX">$Q$</tex-math></inline-formula> -learning, whose input is the real-time data flow measurements and whose output is a value function estimating the near-future latency. For instantaneous decision making, DataBot+ is decoupled into two asynchronous production and training components, ensuring that the training delay will not introduce extra overheads to handle the data flows. Evaluation results driven by real-world traces demonstrate the effectiveness of our design.

New Constructions of Optimal Locally Repairable Codes With Super-Linear Length

As an important coding scheme in modern distributed storage systems, locally repairable codes (LRCs) have attracted a lot of attentions from perspectives of both practical applications and theoretical research. As a major topic in the research of LRCs, bounds and constructions of the corresponding optimal codes are of particular concerns. In this work, codes with (r,δ)-locality which have optimal minimal distance w.r.t. the bound given by Prakash et al. are considered. Through parity-check matrix approach, constructions of both optimal (r,δ)-LRCs with all symbol locality ( (r,δ) <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a</sub> -LRCs) and optimal (r,δ)-LRCs with information locality ( (r,δ) <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sub> -LRCs) are provided. As a generalization of a work of Xing and Yuan, these constructions are built on a connection between sparse hypergraphs and optimal (r,δ)-LRCs. With the help of constructions of large sparse hypergraphs, the lengths of codes obtained from our construction can be super-linear in the alphabet size. This improves upon previous constructions when the minimal distance of the code is at least 3δ+1. As two applications, optimal H-LRCs with super-linear lengths and GSD codes with unbounded lengths are also constructed.

New Design and Analysis of Error-Resilient LRCs for DSSs With Silent Disk Errors

Recently, erasure coding techniques are considered as essential schemes for the reliability of the modern distributed storage systems (DSSs) with the frequent node-level failure. Especially, locally repairable codes (LRCs) are widely adopted by the practical advantage of reducing the latency for repair process. However, recent researches show that many cases for system failure are also originated from the silent disk errors. For the conventional LRCs with low error correction capability, repair process from erasure coding can propagate the silent errors and thus, the DSSs become more vulnerable compared to the cases only with node failure. Therefore, we propose a mean time to data loss (MTTDL) from the modified Markov chain model in order to evaluate effects by silent disk errors. Also, new design of binary error-resilient locally repairable codes (ER-LRCs) with high error and erasure correction capabilities are proposed, which have larger values of bit-wise minimum Hamming distance than the existing LRCs. Here, ER-LRCs can be constructed by modifying the parity check matrix from well-known optimal binary and nonbinary LRCs. From the numerical analysis using the proposed Markov model with empirical parameters, it is shown that the proposed ER-LRCs have better MTTDL values when compared to the existing LRCs.

LAR: Locality‐Aware Reconstruction for erasure‐coded distributed storage systems

SummaryMany modern distributed storage systems adopt erasure coding to protect data from frequent server failures for cost reason. Reconstructing data in failed servers efficiently is vital to these erasure‐coded storage systems. To this end, tree‐structured reconstruction mechanisms where blocks are transmitted and combined through a reconstruction tree have been proposed. However, existing tree‐structured reconstruction mechanisms build reconstruction trees from the perspective of available network bandwidths between servers, which are fluctuating and difficult to measure. Besides, these reconstruction mechanisms cannot reduce data transmission. In this study, we overcome these limitations by proposing LAR, a locality‐aware tree‐structured reconstruction mechanism. LAR builds reconstruction trees from the perspective of data locality, which is stable and easy to obtain. More importantly, by building reconstruction trees that combine blocks closer to each other first, LAR can reduce the data transmitted through the network core and hence speed up reconstruction. We prove that a minimum spanning tree is an optimal reconstruction tree that minimizes core bandwidth usage. We also design and implement a general reconstruction framework that supports all tree‐structured reconstruction mechanisms and nearly all erasure codes. Large‐scale simulations on commonly deployed network topologies show that LAR consumes 20%–61% less core bandwidth than previous reconstruction mechanisms. Thorough experiments on a testbed consisting of 40 physical servers show that LAR improves proactive recovery throughput by 23% at least and improves degraded read rate by up to 68%.

Byzantine fault-tolerant and semantic-driven consensus protocol

To provide fault tolerance, modern distributed storage systems use specialized network topologies and consensus protocols that create high overheads. The main disadvantage of existing specialized topologies is a difficulty to implement an efficient data placement that takes into account locality of the data. In scientific problems very often it is necessary to provide the semantic proximity of the data at the nodes of the distributed system. Core drawback of modern consensus protocols is that most of them use messaging through broadcasting, which is impossible for large amounts of data and storage nodes. With more stringent requirements for a fault tolerance, e.g. requirement for resistance to Byzantine errors [1], ensuring of data localization becomes nearly impossible. The purpose of this paper is to implement a distributed consensus protocol that preserves data localization and is resistant to Byzantine errors. Theoretical tasks: a) consider how to implement distributed services based on finite state machines; b) consider how to ensure Byzantine fault tolerance; c) consider methods for organizing the semantic proximity of data. Practical tasks: a) develop ways to jointly ensure the Byzantine fault tolerance and locality of data; b) study the shortcomings of the joint provision of Byzantine fault tolerance and locality of data; c) develop a protocol of Byzantine consensus using semantic links between data; d) analyze the performance of the developed protocol within the framework of applied tasks. The object of the study is a distributed data storage system with the requirement for Byzantine fault tolerance. The subject of this study is the provision of Byzantine fault tolerance in conjunction with the use of semantic links between the data. The subject of this study has never been the subject of a special scientific study. Separate scientific foundations of this problematic set in the present study were developed in separate works on Byzantine fault tolerance, local hash methods and dimensional reduction.

Sprout: A Functional Caching Approach to Minimize Service Latency in Erasure-Coded Storage

Modern distributed storage systems often use erasure codes to protect against disk and node failures to increase reliability, while trying to meet the latency requirements of the applications and clients. Storage systems may have caches at the proxy or client ends in order to reduce the latency. In this paper, we consider a novel caching framework with erasure code called functional caching . Functional caching involves using erasure-coded chunks in the cache such that the code formed by the chunks in storage nodes and cache combined are maximal-distance-separable erasure codes. Based on the arrival rates of different files, placement of file chunks on the servers, and the service time distribution of storage servers, an optimal functional caching placement and the access probabilities of the file request from different disks are considered. The proposed algorithm gives significant latency improvement in both simulations and a prototyped solution in an open-source, cloud storage deployment.

Efficiently Coding Replicas to Erasure Coded Blocks in Distributed Storage Systems

Modern distributed storage systems usually store new data in replicas, and later code these data into erasure coded blocks when they get cold. This letter studies optimal bandwidth consumption problem for this replica to erasure coded blocks (R2E) coding process, and proposes schemes of R2E_singleTree and R2E_multiTree based on problem observations. Theoretical analysis and evaluation are conducted for these two schemes.

Redundancy Does Not Imply Fault Tolerance

We analyze how modern distributed storage systems behave in the presence of file-system faults such as data corruption and read and write errors. We characterize eight popular distributed storage systems and uncover numerous problems related to file-system fault tolerance. We find that modern distributed systems do not consistently use redundancy to recover from file-system faults: a single file-system fault can cause catastrophic outcomes such as data loss, corruption, and unavailability. We also find that the above outcomes arise due to fundamental problems in file-system fault handling that are common across many systems. Our results have implications for the design of next-generation fault-tolerant distributed and cloud storage systems.

Modified Generalized Integrated Interleaved Codes for Local Erasure Recovery

Locally recoverable codes allow erasures to be recovered from a portion of codeword symbols and are essential to address the high reliability and performance requirements of modern hyper-scale distributed storage systems. Generalized integrated interleaved (GII) codes create redundancy sharable by the interleaves, and hence reduce the overall redundancy for achieving target reliability. However, all interleaves need to be read in order to make use of the shared redundancy. In this letter, GII codes are modified to improve the locality. Without sacrificing the erasure correction capability, the proposed scheme accesses much fewer interleaves when only a small number of extra erasures need to be recovered using the shared redundancy, which happens in most cases. As a result, the average locality is greatly improved. Compared to existing local erasure recovery schemes, the modified GII codes achieve a good tradeoff on locality and correction capability.

High-Performance General Functional Regenerating Codes with Near-Optimal Repair Bandwidth

Erasure codes are widely used in modern distributed storage systems to prevent data loss and server failures. Regenerating codes are a class of erasure codes that trade storage efficiency and computation for repair bandwidth reduction. However, their nonunified coding parameters and huge computational overhead prohibit their applications. Hence, we first propose a family of General Functional Regenerating (GFR) codes with uncoded repair, balancing storage efficiency and repair bandwidth with general parameters. The GFR codes take advantage of a heuristic repair algorithm, which makes efforts to employ as little repair bandwidth as possible to repair a single failure. Second, we also present a scheduled shift multiplication (SSM) algorithm, which accelerates the matrix product over the Galois field by scheduling the order of coding operations, so encoding and repairing of GFR codes can be executed by fast bitwise shifting and exclusive-OR. Compared to the traditional table-lookup multiplication algorithm, our SSM algorithm gains 1.2 to 2 X speedup in our experimental evaluations, with little effect on the repair success rate.

Joint Latency and Cost Optimization for Erasure-Coded Data Center Storage

Modern distributed storage systems offer large capacity to satisfy the exponentially increasing need of storage space. They often use erasure codes to protect against disk and node failures to increase reliability, while trying to meet the latency requirements of the applications and clients. This paper provides an insightful upper bound on the average service delay of such erasure-coded storage with arbitrary service time distribution and consisting of multiple heterogeneous files. Not only does the result supersede known delay bounds that only work for a single file or homogeneous files, it also enables a novel problem of joint latency and storage cost minimization over three dimensions: selecting the erasure code, placement of encoded chunks, and optimizing scheduling policy. The problem is efficiently solved via the computation of a sequence of convex approximations with provable convergence. We further prototype our solution in an open-source, cloud storage deployment over three geographically distributed data centers. Experimental results validate our theoretical delay analysis and show significant latency reduction, providing valuable insights into the proposed latency-cost tradeoff in erasure-coded storage.

Dynamic erasure coding decision for modern block-oriented distributed storage systems

Modern block-oriented distributed storage systems like Hadoop distributed file system have proliferated in this era of big data and cloud computing. These systems feature block-level replication in which their files are partitioned into equal-sized blocks and multiple copies for each block are then arbitrarily distributed across nodes for fault tolerance and data availability. However, many storage volumes are just wasted only for keeping block copies whose data may not be accessed frequently in the strategy. Therefore, distributed storage systems begin to adopt erasure codes. However, classical parity encoding scheme are hard to be directly applied to the distributed storage systems since block copies are arbitrarily placed across nodes in the systems. We present a novel technique, called DynaEC, to address the issues in modern block-oriented distributed storage systems. DynaEC provides a unique parity encoding algorithm that encodes data blocks arbitrarily distributed across machines to parities and then places the parities guaranteeing fault tolerance. Parity encoding in DynaEC is performed without any change of the original block placement policy in Hadoop distributed file system. This makes DynaEC work seamlessly with Hadoop distributed file system. Finally, during the encoding procedure each data node encodes each own data blocks, not requiring any information about other blocks located in other data nodes. As such, the encoding procedure in DynaEC is fully performed in parallel without any synchronization issue. With extensive experiments, we show that DynaEC saves storage volumes up to the theoretical limit while outperforming previous approaches by multiple orders of magnitude.

Joint latency and cost optimization for erasurecoded data center storage

Modern distributed storage systems offer large capacity to satisfy the exponentially increasing need of storage space. They often use erasure codes to protect against disk and node failures to increase reliability, while trying to meet the latency requirements of the applications and clients. This paper provides an insightful upper bound on the average service delay of such erasure-coded storage with arbitrary service time distribution and consisting of multiple heterogeneous files. Not only does the result supersede known delay bounds that only work for homogeneous files, it also enables a novel problem of joint latency and storage cost minimization over three dimensions: selecting the erasure code, placement of encoded chunks, and optimizing scheduling policy. The problem is efficiently solved via the computation of a sequence of convex approximations with provable convergence. We further prototype our solution in an open-source, cloud storage deployment over three geographically distributed data centers. Experimental results validate our theoretical delay analysis and show significant latency reduction, providing valuable insights into the proposed latency-cost tradeoff in erasure-coded storage.