Large-scale Distributed Storage Systems Research Articles

Construction of Singleton-Type Optimal LRCs from Existing LRCs and Near-MDS Codes

Modern large scale distributed storage systems play a central role in data center and cloud storage, while node failure in data center is common. The lost data in failure node must be recovered efficiently. Locally repairable codes (LRCs) are designed to solve this problem. The locality of an LRC is the number of nodes that participate in recovering the lost data from node failure, which characterizes the repair efficiency. An LRC is called optimal if its minimum distance attains Singleton-type upper bound [1]. In this paper, using basic techniques of linear algebra over finite field, infinite optimal LRCs over extension fields are derived from a given optimal LRC over base field (or small field). Next, this paper investigates the relation between near-MDS codes with some constraints and LRCs, further, proposes an algorithm to determine locality of dual of a given linear code. Finally, based on near-MDS codes and the proposed algorithm, those obtained optimal LRCs are shown.

Boosting Erasure-Coded Multi-Stripe Repair in Rack Architecture and Heterogeneous Clusters: Design and Analysis

Large-scale distributed storage systems have introduced erasure codes to guarantee high data reliability, yet inevitably at the expense of high repair costs. In practice, storage nodes are usually divided into different racks, and data blocks in storage nodes are often organized into multiple stripes independently manipulated by erasure code. Due to the scarcity and heterogeneity of the cross-rack bandwidth, the cross-rack network transmission dominates the entire repair costs. We argue that when erasure code is deployed in a rack architecture, existing repair techniques are limited in different aspects: neglecting the heterogeneous cross-rack bandwidth, less consideration for multi-stripe failure, and no special treatment on repair link scheduling. In this paper, we present CMRepair, a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Cross-rack Multi-stripe Repair</i> technique that aims to reduce the repair time for multi-stripes failure repair in heterogeneous erasure-coded clusters. CMRepair first carefully chooses the nodes for reading/repairing blocks and searches for the multi-stripe repair solution. It adopts different algorithms to adjust per stripe solution, including the Computation Time Priority (CTP) algorithm based on the greedy idea and the Repair Time Priority (RTP) algorithm based on the meta-heuristics idea. Furthermore, CMRepair selectively schedules the execution orders of cross-rack links, with the primary objective of saturating the unused upload/download bandwidth resources and avoiding network congestion. The experiments show that CMRepair with the CTP algorithm can reduce 27.59%-58.12% of the repair time while only introducing negligible computation overhead, and CMRepair with the RTP algorithm can reduce 33.52%-97.75% of the repair time in an acceptable computation time, over existing repair techniques.

Data Distribution for Heterogeneous Storage Systems

The exponential growth of data in many science and engineering domains poses significant challenges to storage systems. Data distribution is a critical component in large-scale distributed storage systems and plays a vital role in placing petabytes of data and beyond, among tens to hundreds of thousands of storage devices. Meantime, heterogeneous storage systems, such as those having devices with hard disk drives (HDDs) and storage class memories (SCMs), have become increasingly popular for massive data storage due to their distinct and complement characteristics. This paper presents a new data distribution algorithm called SUORA (Scalable and Uniform storage via Optimally-adaptive and Random number Addressing) specifically for heterogeneous devices to maximize the benefits of them. SUORA provides a fully symmetric, highly efficient methodology to distribute data across a hybrid and tiered storage cluster. It divides heterogeneous devices into different buckets and segments, and adopts pseudo-random functions to map data onto them with the balanced consideration of capacity, performance and life-time. By analyzing hotness and access patterns, SUORA gradually moves hot data from HDDs to SCMs to optimize the throughput, and moves cold data reversely for load balance. It combines data replication with migration to significantly reduce movement overhead while making data placement more adaptive to different workloads. Extensive evaluations on simulation and Sheepdog storage system show that, with considering distinct characteristics of various devices thoroughly, SUORA improves the overall performance efficiency of heterogeneous storage systems.

Smash: Flexible, Fast, and Resource-efficient Placement and Lookup of Distributed Storage

Large-scale distributed storage systems, such as object stores, usually apply hashing-based placement and lookup methods to achieve scalability and resource efficiency. However, when object locations are determined by hash values, placement becomes inflexible, failing to optimize or satisfy application requirements such as load balance, failure tolerance, parallelism, and network/system performance. This work presents a novel solution to achieve the best of two worlds: flexibility while maintaining cost-effectiveness and scalability. The proposed method Smash is an object placement and lookup method that achieves full placement flexibility, balanced load, low resource cost, and short latency. Smash utilizes a recent space-efficient data structure and applies it to object-location lookups. We implement Smash as a prototype system and evaluate it in a public cloud. The analysis and experimental results show that Smash achieves full placement flexibility, fast storage operations, fast recovery from node dynamics, and lower DRAM cost (&lt;60%) compared to existing hash-based solutions such as Ceph and MapX.

New Optimal Linear Codes With Hierarchical Locality

Locally repairable codes with hierarchical locality (H-LRCs) are designed to correct different numbers of erasures, which play a crucial role in large-scale distributed storage systems. In this paper, we construct three classes of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$q$ </tex-math></inline-formula> -ary optimal H-LRCs by employing matrix product codes, concatenated codes and cyclic codes, respectively. The first two constructions are based on the idea of constructing new codes from old, which produces several new classes of optimal H-LRCs whose lengths can reach up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$q^{2}+q$ </tex-math></inline-formula> or unbounded. The final construction generates a class of new optimal cyclic H-LRCs whose lengths divide <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$q-1$ </tex-math></inline-formula> . Compared with the previously known ones, our constructions are new in the sense that their parameters are not covered by the codes available in the literature.

HV-SNSP: A Low-Overhead Data Recovery Method Based on Cross-Checking

The failure of a single unreliable commodity component is very common in large-scale distributed storage systems. In order to ensure the reliability of data in large-scale distributed storage systems, many studies have emerged one after another. Among them, Erasure Codes are widely used in actual storage systems, such as Hadoop Distributed File System (HDFS), which provides high fault-tolerance with lower storage overhead. However, usually the recovery of erasure-coded storage system when encountering node failure results in severe cross-node and cross-rack bandwidth consumption, which affects the efficiency of failure recovery and wastes additional resources. In this paper, we improve the erasure coding storage strategy in distributed storage systems, and propose a low-overhead data recovery method based on cross-checking, namely HV-SNSP. In HV-SNSP, horizontal and vertical cross parity checking is realized by adding RS parity inside the data node, that is, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{H}^{\mathrm {RS(n, k)}}$ </tex-math></inline-formula> - <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{V}^{\mathrm {RS(n', \textrm {}k')}}$ </tex-math></inline-formula> storage architecture. Based on <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{H}^{\mathrm {RS(n, \textrm {}k)}}$ </tex-math></inline-formula> - <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{V}^{\mathrm {RS(n', k')}}$ </tex-math></inline-formula> , a low-cost supply node selection strategy, namely SNSP, is designed, and nodes with shorter network distance and lower load are selected to participate in recovery. This strategy can effectively reduce the amount of data transmission, shorten the recovery time, and improve the recovery efficiency. The experimental results show that compared with traditional RS, HV-SNSP can reduce the amount of cross-rack data transmission by 62.5% during data recovery, and can shorten the recovery time by up to 42.41%; Compared with D3, HV-SNSP can reduce the occupation of cross-rack bandwidth by 25% and shorten the recovery time by 36.58%.

Data Utilization-Based Adaptive Data Management Method for Distributed Storage System in WAN Environment

Recently, research on a distributed storage system that efficiently manages a large amount of data has been actively conducted following data production and demand increase. Physical expansion limits exist for traditional standalone storage systems, such as I/O and file system capacity. However, the existing distributed storage system does not consider where data is consumed and is more focused on data dissemination and optimizing the lookup cost of data location. And this leads to system performance degradation due to low locality occurring in a Wide Area Network (WAN) environment with high network latency. This problem hinders deploying distributed storage systems to multiple data centers over WAN. It lowers the scalability of distributed storage systems to accommodate data storage needs. This paper proposes a method for distributing data in a WAN environment considering network latency and data locality to solve this problem and increase overall system performance. The proposed distributed storage method monitors data utilization and locality to classify data temperature as hot, warm, and cold. With assigned data temperature, the proposed algorithm adaptively selects the appropriate data center and places data accordingly to overcome the excess latency from the WAN environment, leading to overall system performance degradation. This paper also conducts simulations to evaluate the proposed and existing distributed storage methods. The result shows that our proposed method reduced latency by 38% compared to the existing method. Therefore, the proposed method in this paper can be used in large-scale distributed storage systems over a WAN environment to improve latency and performance compared to existing methods, such as consistent hashing.

A function field approach toward good polynomials for further results on optimal LRC codes

Because of the recent applications to distributed storage systems, researchers have introduced a new class of block codes, i.e., locally recoverable (LRC) codes. LRC codes can recover information from erasure(s) by accessing a small number of erasure-free code symbols and increasing the efficiency of repair processes in large-scale distributed storage systems. In this context, Tamo and Barg first gave a breakthrough by cleverly introducing a good polynomial notion. Constructing good polynomials for locally recoverable codes achieving Singleton-type bound (called optimal codes) is challenging and has attracted significant attention in recent years. This article aims to increase our knowledge of good polynomials for optimal LRC codes. Using tools from algebraic function fields and Galois theory, we continue investigating those polynomials and studying them by developing the Galois theoretical approach initiated by Micheli in 2019. Specifically, we push further the study of a crucial parameter G(f) (of a given polynomial f), which measures how much a polynomial is “good” in the sense of LRC codes. We provide some characterizations of polynomials with minimal Galois groups and prove some properties of finite fields where polynomials exist with a specific size of Galois groups. We also present some explicit shapes of polynomials with small Galois groups. For some particular polynomials f, we give the exact formula of G(f).

A New Piggybacking Design With Low-Repair Bandwidth and Complexity

Most of the piggybacking designs can only effectively repair systematic nodes. In this letter, we propose a new piggybacking method for repairing parity nodes based on the existing piggybacking design. By elaborately designing piggyback functions and making full use of each parity node, the improved piggybacking design reduces the repair bandwidth of parity nodes while retaining the repair bandwidth of systematic nodes. Compared with the existing designs, the proposed method has lower complexity, fewer substripes and better comprehensive repair properties in large-scale distributed storage systems.

Novel Data Placement Algorithm for Distributed Storage System Based on Fault-Tolerant Domain

The 3-replica redundancy strategy is widely used to solve the problem of data reliability in large-scale distributed storage systems. However, its storage capacity utilization is only 33%. In this paper, a data placement algorithm based on fault-tolerant domain (FTD) is proposed. Owing to the fine-grained design of the FTD, the data reliability of systems using two replicas is comparable to that of current mainstream systems using three replicas, and the capacity utilization is increased to 50%. Moreover, the proposed FTD provides a new concept for the design of distributed storage systems. Distributed storage systems can take FTDs as the units for data placement, data migration, data repair and so on. In addition, fault detection can be performed independently and concurrently within the FTDs.

A method of repairing single node failure in the distributed storage system based on the regenerating-code and a hybrid genetic algorithm

To ensure the reliability and security of the data, the large-scale distributed storage system usually adopts the data redundancy mechanism to repair the data on the faulty nodes. Comparing with the replication-based method, the data redundancy mechanism of erasure code can effectively improve the use of storage space, but it may result in the large network overhead when recovering the data. The regenerating code is an improved erasure code, which can reduce the quantity of data transmission compared to that of the erasure code. Adopting the regenerating code to repair the data on a faulty node requires constructing an optimal repair tree to maximum the bandwidth of the bottleneck link, which is an NP-hard problem. To construct the optimal repair tree, a hybrid genetic algorithm is proposed in this paper. In particular, our proposal comprehensively considers the network topology and link bandwidth of storage nodes and designs a problem-specific cross-correlation operator, mutation operator and local search operator. In addition, we provided the mathematical proof of the global convergence with probability one with respect to the proposed hybrid genetic algorithm. Through a series of simulation experiments, the results show that our proposal is able to determine the optimal repair tree, which effectively reduces the delay of faulty node repair in distributed storage systems, and improve the repairing efficiency.

Constructions of Maximally Recoverable Local Reconstruction Codes via Function Fields

Local Reconstruction Codes (LRCs) allow for recovery from a small number of erasures in a local manner based on just a few other codeword symbols. They have emerged as the codes of choice for large scale distributed storage systems due to the very efficient repair of failed storage nodes in the typical scenario of a single or few nodes failing, while also offering fault tolerance against worst-case scenarios with more erasures. A maximally recoverable (MR) LRC offers the best possible blend of such local and global fault tolerance, guaranteeing recovery from all erasure patterns which are information-theoretically correctable given the presence of local recovery groups. MR LRCs have received much attention recently, with many explicit constructions covering different regimes of parameters. Unfortunately, all known constructions require a large field size. In this work, we develop an approach based on function fields to construct MR LRCs. Our method recovers, and in most parameter regimes improves, the field size of previous approaches. The improvements are modest, but more importantly are obtained in a unified manner via a promising new idea.

Deterministic Data Distribution for Efficient Recovery in Erasure-Coded Storage Systems

Due to individual unreliable commodity components, failures are common in large-scale distributed storage systems. Erasure codes are widely deployed in practical storage systems to provide fault tolerance with low storage overhead. However, random data distribution (RDD), commonly used in erasure-coded storage systems, induces heavy cross-rack traffic, load imbalance, and random access, which adversely affects failure recovery. In this article, with orthogonal arrays, we define a Deterministic Data Distribution (D 3 ) to uniformly distribute data/parity blocks among nodes, and propose an efficient failure recovery approach based on D 3 , which minimizes the cross-rack repair traffic against a single node failure. Thanks to the uniformity of D 3 , the proposed recovery approach balances the repair traffic not only among nodes within a rack but also among racks. We implement D 3 over Reed-Solomon codes and Locally Repairable Codes in Hadoop Distributed File System (HDFS) with a cluster of 28 machines. Compared with RDD, our experiments show that D3 significantly speeds up the failure recovery up to 2.49 times for RS codes and 1.38 times for LRCs. Moreover, D 3 supports front-end applications better than RDD in both of normal and recovery states.

Optimization of the Recovery Time of Pyramid Code in Distributed Storage System

In large scale distributed storage systems, erasure code is a basic technology that provides high reliability at low cost. Compared with traditional redundancy technology, erasure coding technology has low redundancy and high flexibility. Therefore, it is a good choice for distributed systems to construct Pyramid codes that are flexible and suitable for a variety of application scenarios, but its disadvantage is that the data recovery time is still long. To address the above problems, in this paper, we propose an Active Fault-Tolerant Pyramid (AFTP) based code, which dynamically adjusts the length of the group in the Pyramid code and the original data block correlation with redundant blocks by using the hard disk fault prediction model based on the decision tree to reduce the length of the group of data blocks in a potentially faulty hard disk, which can be used for multiple hard disk failures. All read and recovery operations are performed within the group, reducing recovery time without adding additional storage overhead. To verify the validity of the AFTP code, we conduct intensive experiments on the distributed storage system based on Ceph. The results show that, compared with Basic-Pyramid (BP), the recovery time of AFTP code is reduced by 8%-64%, and compared with the commonly used classic block codes, the recovery time of AFTP code is reduced by 11%-52%.

An Ant Colony Optimization Based Data Update Scheme for Distributed Erasure-Coded Storage Systems

Owing to the high availability and space-efficiency of erasure codes, they have become the de facto standard to provide data durability in large scale distributed storage systems. The update-intensive workloads of erasure codes lead to a large amount of data transmission and I/O. As a result, it becomes a major challenge to reduce the amount of data transmission and optimize the use of existing network resources so that the update efficiency of the erasure codes could be improved. However, very little research has been done to optimize the update efficiency of the erasure codes under multiple QoS metrics. In this paper, our proposed update scheme, the Ant Colony Optimization based multiple data nodes Update Scheme (ACOUS) employs a two-stage rendezvous data update procedure to optimize the multiple data nodes updates. Specifically, the two-stage rendezvous data update procedure performs the data delta collection and the parity delta distribution based on a multi-objective update tree which is built by the ant colony optimization routing algorithm. Under typical data center network topologies, extensive experimental results show that, compared to the traditional TA-Update scheme, our scheme is able to achieve a 26% to 37% reduction of update delay with convergence guarantee at the cost of negligible computation overhead.

DCache – Efficient Message Encoding For Inter-Service Communication in dCache: Evaluation of Existing Serialization Protocols as a Replacement for Java Object Serialization

As a well established, large-scale distributed storage system, dCache is used to manage and serve huge amounts of data collected by high energy physics, astrophysics and photon science experiments. Based on a microservices-like architecture, dCache is built as a modular distributed system, where each component provides a different core functionality. These services communicate by passing serialized messages to each other, a core behavior whose performance properties can consequently affect the entire system. This paper compares and evaluates different data serialization protocols in computer science with the objective of replacing and improving upon Java Object Serialization (JOS), which has increasingly presented itself as no longer being sufficiently performant for encoding messages. The criteria for choosing a new framework are collected, analyzed and formalized. The primary motivation for replacing Java serialization for encoding dCache messages is increasing the general speed of message-passing and thereby reducing the round-trip time for user requests. Emphasis is also placed on schema evolution capabilities and framework usability. Approaches for generalizing (de)serialization speed and size measurements based on data structure complexity are introduced, criteria for measuring documentation, learning curve, maintainability and introduction effort are defined. Finally, several selected serialization protocols are evaluated and compared accordingly, concluding with a recommendation for a suitable JOS replacement.

An Efficient Ring-Based Metadata Management Policy for Large-Scale Distributed File Systems

The growing size of modern file system is expected to reach EB-scale. Therefore, an efficient and scalable metadata service is critical to system performance. Distributed metadata management schemes, which use multiple metadata servers (MDS's) to store metadata, provide a highly effective approach to alleviate the workload of a single server. However, it is difficult to maintain good metadata locality and load balancing among MDS's at the same time. In this paper, we propose a novel hashing scheme called AngleCut to partition metadata namespace tree and serve large-scale distributed storage systems. AngleCut first uses a locality preserving hashing (LPH) function to project the namespace tree into linear keyspace, i.e., multiple Chord-like rings. Then we design a history-based allocation strategy to adjust the workload of MDS's dynamically. Besides, we propose a two-layer metadata cache mechanism, including server-side cache and client-side cache to provide the two stage access acceleration. Last but not least, we introduce a distributed metadata processing 2PC Protocol Based on Message Queue (2PC-MQ) to ensure data consistency. In general, our scheme preserves good metadata locality as well as maintains a high load balancing between MDS's. The theoretical proof and extensive experiments on Amazon EC2 demonstrate the superiority of AngleCut over previous literature.

Testing of complex, large-scale distributed storage systems: a CERN disk storage case study

Complex, large-scale distributed systems are frequently used to solve extraordinary computing, storage and other problems. However, the development of these systems usually requires working with several software components, maintaining and improving a large codebase and also providing a collaborative environment for many developers working together. The central role that such complex systems play in mission critical tasks and also in the daily activity of the users means that any software bug affecting the availability of the service has far reaching effects. Providing an easily extensible testing framework is a pre-requisite for building both confidence in the system but also among developers who contribute to the code. The testing framework can address concrete bugs found in the odebase thus avoiding any future regressions and also provides a high degree of confidence for the people contributing new code. Easily incorporating other people's work into the project greatly helps scaling out manpower so that having more developers contributing to the project can actually result in more work being done rather then more bugs added. In this paper we go through the case study of EOS, the CERN disk storage system and introduce the methods and mechanisms of how to achieve all-automatic regression and robustness testing along with continuous integration for such a large-scale, complex and critical system using a container-based environment.

Architecture-aware coding for distributed storage: Repairable block failure resilient codes

In large scale distributed storage systems (DSS) deployed in cloud computing, correlated failures resulting in simultaneous failure (or, unavailability) of blocks of nodes are common. In such scenarios, the stored data or a content of a failed node can only be reconstructed from the available live nodes belonging to the available blocks. To analyze the resilience of the system against such block failures, this work introduces the framework of Block Failure Resilient (BFR) codes, wherein the data (e.g., a file in DSS) can be decoded by reading out from a same number of codeword symbols (nodes) from a subset of available blocks of the underlying codeword. Further, repairable BFR codes are introduced, wherein any codeword symbol in a failed block can be repaired by contacting a subset of remaining blocks in the system. File size bounds for repairable BFR codes are derived, and the trade-off between per node storage and repair bandwidth is analyzed, and the corresponding minimum storage regenerating (BFR-MSR) and minimum bandwidth regenerating (BFR-MBR) points are derived. Explicit codes achieving the two operating points for a special case of parameters are constructed, wherein the underlying regenerating codewords are distributed to BFR codeword symbols according to combinatorial designs. Finally, BFR locally repairable codes (BFR-LRC) are introduced, an upper bound on the resilience is derived and optimal code construction are provided by a concatenation of Gabidulin and MDS codes. Repair efficiency of BFR-LRC is further studied via the use of BFR-MSR/MBR codes as local codes. Code constructions achieving optimal resilience for BFR-MSR/MBR-LRCs are provided for certain parameter regimes. Overall, this work introduces the framework of block failures along with optimal code constructions, and the study of architecture-aware coding for distributed storage systems.

Multilevel Diversity Coding With Regeneration

Digital contents in large scale distributed storage systems may have different reliability and access delay requirements, and for this reason, erasure codes with different strengths need to be utilized to achieve the best storage efficiency. At the same time, in such large scale distributed storage systems, nodes fail on a regular basis, and the contents stored on them need to be regenerated and stored on other healthy nodes, the efficiency of which is an important factor affecting the overall quality of service. In this work, we formulate the problem of multilevel diversity coding with regeneration to address these considerations, for which the storage vs. repair-bandwidth tradeoff is investigated. We show that the extreme point on this tradeoff corresponding to the minimum possible storage can be achieved by a simple coding scheme, where contents with different reliability requirements are encoded separately with individual regenerating codes without any mixing. On the other hand, we establish the complete storage-repair-bandwidth tradeoff for the case of four storage nodes, which reveals that codes mixing different contents can strictly improve this tradeoff over the separate coding solution.