Large-scale Distributed Storage Systems Research Articles

I/O-Efficient Scaling Schemes for Distributed Storage Systems with CRS Codes

System scaling becomes essential and indispensable for distributed storage systems due to the explosive growth of data volume. Considering that fault-protection is a necessity in large-scale distributed storage systems, and Cauchy Reed-Solomon (CRS) codes are widely deployed to tolerate multiple simultaneous node failures, this paper studies the scaling problem of distributed storage systems with CRS codes. In particular, we formulate the scaling problem with an optimization model in which both the post-scaling encoding matrix and the data migration policy are assumed to be unknown in advance. To minimize the I/O overhead, we propose a three-phase optimization scaling scheme for CRS codes. Specifically, we first derive the optimal post-scaling encoding matrix under a given data migration policy, then optimize the data migration process using the selected post-scaling encoding matrix, and finally exploit the Maximum Distance Separable (MDS) property to further optimize the designed data migration process. Our scaling scheme requires minimal data movement while achieving uniform data distribution. Moreover, it requires to read fewer data blocks than conventional minimum data migration schemes, but still guarantees the minimum amount of migrated data. To validate the efficiency of our scheme, we implement it atop a networked file system. Extensive experiments show that our scaling scheme uses less scaling time than the basic scheme.

A Delayed Container Organization Approach to Improve Restore Speed for Deduplication Systems

Data deduplication has become necessary to improve the space-efficiency of large-scale distributed storage systems, as the global data have accumulated at an exponential rate and they have significant redundancy. However, the negative impact on restore performance is a main challenge for deduplication systems. One of the key reasons is that when restoring data, the low average useful data ratio (UDR) of containers wastes a considerable part of disk bandwidth to read useless data. This is mainly attributed to the uncontrollable compositions of containers. To solve this problem, we propose a new approach called Delayed Container Organization (DCO) to delay the construction of containers after accumulating some redundant data chunks in fast Non-Volatile Memory (NVM) devices to organize high-UDR containers. For example, data chunks in the intersection of some data segments can be organized together in one container to achieve both high deduplication ratio and high UDRs when restoring these related data segments. DCO is implemented in a prototype deduplication system. The experimental results indicate that compared with Capping, DCO promotes the average UDR of containers by 38.30 percent, improves the restore performance by a factor of 2.2, and achieves better space-efficiency and higher cost performance.

Elastic-RAID: A New Architecture for Improved Availability of Parity-Based RAIDs by Elastic Mirroring

In this paper, we propose Elastic-RAID, a new RAID architecture to achieve high performance and high reliability for large-scale distributed and parallel storage systems. The key idea behind Elastic-RAID is to smartly utilize the free space existing in parity-based disk arrays to store additional mirroring data. This additional mirroring data redundancy, when strategically and judiciously activated and exploited in a RAID system, enables improved system I/O performance, fault tolerance and recovery. Depending on the amount of free space available and whether the emphasis is on performance or reliability, the elasticity in Elastic-RAID is manifested in how each design objective is achieved. For the performance objective, Elastic-RAID improves small-write performance by writing original and mirroring data synchronously and leaving the costly parity update in the background at a later idle/lightly-loaded time. For the reliability objective, at least two concurrent disk failures can be tolerated when Elastic-RAID is employed in a RAID5 system that has 50 percent or more free space. Higher reliability is provided for important data when free space is less than 50 percent. To achieve the design goal of elasticity, we introduce a novel data layout and addressing scheme. Our extensive trace-driven evaluations on an Elastic-RAID prototype in the typical configurations of RAID5 show that Elastic-RAID boosts the small-write performance in the normal operational state by at least 40 percent, improves the user I/O performance in the reconstruction state by at least 30 percent and shortens the recovery time by at least 40 percent.

Building a large-scale object-based active storage platform for data analytics in the internet of things

Due to consistent improvements in memory and processor technology, object storage devices (OSDs) have greater memory space and more powerful processing power, which allow the OSDs to execute user-defined programs. Shifting part of an application's processing to the disk drives drops the amount of data transferred across the network and explores the parallelism of large-scale distributed storage systems, reducing the execution time for many basic data analytics tasks. In this paper, we propose a large-scale object-based active storage platform, named Gem, for data analytics in the internet of things (IoT). All data from the IoT that resides in disk drives form objects with attributes, methods and policies. For some applications such as data analytics, application-specific operations are executed by the drive processors. In this way, only the results are returned to clients, rather than data files being read by the clients. Therefore, the platform Gem is able to greatly reduce the overhead of data analytics applications in the Internet of Things. By conducting performance evaluation, experimental results demonstrate the effectiveness and scalability of Gem.

Take me to your leader!

The configuration of a distributed storage system typically includes, among other parameters, the set of servers and their roles in the replication protocol. Although mechanisms for changing the configuration at runtime exist, it is usually left to system administrators to manually determine the "best" configuration and periodically reconfigure the system, often by trial and error. This paper describes a new workload-driven optimization framework that dynamically determines the optimal configuration at run-time. We focus on optimizing leader and quorum based replication schemes and divide the framework into three optimization tiers, dynamically optimizing different configuration aspects: 1) leader placement, 2) roles of different servers in the replication protocol, and 3) replica locations. We showcase our optimization framework by applying it to a large-scale distributed storage system used internally in Google and demonstrate that most client applications significantly benefit from using our framework, reducing average operation latency by up to 94%.

Heterogeneity-Aware Codes With Uncoded Repair for Distributed Storage Systems

In practical large-scale distributed storage systems, node failures are unavoidable. It is therefore desirable to quickly recreate the failed nodes in order to maintain the system integrity. In this letter, we consider a family of erasure codes that provide uncoded repair, where the failed node is regenerated by transfer of data without extra arithmetic operations. We introduce flexible fractional repetition (FFR) code, of which the coding scheme is a concatenation of an outer MDS code and an inner repetition code. Our proposed codes are applicable to the heterogeneous network environment where node storage capacities and packet repetition degrees vary in a wide range. We present explicit constructions of FFR codes by utilizing combinatorial designs. We further propose a heuristic code construction. Evaluation results show that FFR codes outperform regenerating codes in node repair efficiency.

A Novel Architecture Design of Large-Scale Distributed Object Storage System

A novel architecture design of large-scale distributed object storage system (called DOSS) is proposed. Our design takes several aspects effecting on the overall performance of DOSS into consideration, including an improved model of interaction based on the traditional interactive mode of object-based storage systems, MDS(Metadata Server) management scheme and a load balancing scheme combines the definition of the maximum load for OSD(Object Storage Device), hierarchical model and re-polling in order to solve access of heat objects issues. Our experimental results show that the architecture is an effective way to promote the performance of DOSS.

On the tradeoff of availability and consistency for quorum systems in data center networks

Large-scale distributed storage systems often replicate data across servers and even geographically-distributed data centers for high availability, while existing theories like CAP and PACELC show that there is a tradeoff between availability and consistency. Thus eventual consistency is proposed to provide highly available storage systems. However, current practice is mainly experience-based and lacks quantitative analysis for identifying a good tradeoff between the two. In this work, we are concerned with providing a quantitative analysis on availability for widely-used quorum systems in data center networks. First, a probabilistic model is proposed to quantify availability for typical data center networks: 2-tier basic tree, 3-tier basic tree, fat tree and folded clos, and even geo-distributed data center networks. Second, we analyze replica placements on network topologies to obtain maximal availability. Third, we build the availability-consistency table and propose a set of rules to quantitatively make tradeoff between availability and consistency. Finally, with Monte Carlo based simulations, we validate our presented quantitative results and show that our approach to make tradeoff between availability and consistency is effective.

Recent advances in the Message Passing Interface

ion for MPI programs with pointer-based data structures. The main features of DRASync are: it amortizes communication among MPI processes to allow efficient parallel allocation in a global address space; it takes advantage of bulk deallocation and good locality with pointer-based data structures. Finally, DRASync supports ownership semantics of regions by MPI processes akin to reader–writer locks, which makes for a high-level, intuitive synchronization tool in MPI programs, without sacrificing message-passing performance. In ‘‘An Evaluation of MPI Message Rate on Hybrid-Core Processors’’, Barrett et al. analyze the ability of simple and more complex cores to perform MPI matching operations for various scenarios in order to gain insight into how MPI implementations for future hybrid-core processors should be designed. The authors compare throughput-oriented cores to University Carlos III of Madrid, Spain single-thread optimized cores in terms of the ability to perform MPI match processing. The intent of this study is to gain insight into the ability of throughputoriented cores to adequately perform MPI matching and to better understand how MPI implementations on future hybrid-core processors should allocate computing resources to try to optimize performance-critical MPI operations. Collective MPI communications have to be executed in the same order by all processes in their communicator and the same number of times, otherwise it is not conforming to the standard and a deadlock can occur. As soon as the control-flow involving these collective operations becomes more complex, in particular including conditionals on process ranks, ensuring the correction of such code is error-prone. The paper ‘‘PARCOACH: Combining Static and Dynamic Validation of MPI Collective Communications’’ by Saillard et al. proposes a static analysis to detect when such situation occurs, combined with a code transformation that prevents deadlocking. They show with several benchmarks the small impact on performance and the ease of integration of their techniques in the development process. In ‘‘Extreme-scale Computing Services Over MPI: Experiences, Observations and Features Proposal for Next Generation Message Passing Interface’’ by Zounmevo et al., the authors present their experiences in using MPI as a network transport for a large-scale distributed storage system. The authors discuss the features of MPI that facilitate adoption as well as aspects which require various workarounds. Based on use cases, the authors derive a wish-list for both MPI implementations and the MPI forum to facilitate the adoption of MPI large-scale persistent services. Finally in the paper ‘‘Optimization of MPI collective operations on the IBM Blue Gene/Q Supercomputer’’, Kumar et al. present scalable algorithms to optimize MPI collective operations by taking advantage of the various features of the Blue Gene/Q torus and collective networks. The authors accelerate summing of network packets with local buffers by the use of the Quad Processing SIMD unit in the Blue Gene/Q cores and executing the sums on multiple communication threads supported by the optimized communication libraries.

Extreme-scale computing services over MPI: Experiences, observations and features proposal for next-generation message passing interface

The message passing interface (MPI) is one of the most portable high-performance computing (HPC) programming models, with platform-optimized implementations typically delivered with new HPC systems. Therefore, for distributed services requiring portable, high-performance, user-level network access, MPI promises to be an attractive alternative to custom network portability layers, platform-specific methods, or portable but less performant interfaces such as BSD sockets. In this paper, we present our experiences in using MPI as a network transport for a large-scale distributed storage system. We discuss the features of MPI that facilitate adoption as well as aspects which require various workarounds. Based on use cases, we derive a wish list for both MPI implementations and the MPI forum to facilitate the adoption of MPI by large-scale persistent services. The proposals in the wish list go beyond the sole needs of distributed services; we contend that they will benefit mainstream HPC applications at extreme scales as well.

Effective Metadata Cluster Management for Mass Data Storage

An effective decentralized metadata management schema is critical to a reliable and scalable large-scale distributed storage system. With regard to the defects of huge expansion cost and high sensitivity to cluster alteration borne in both the hash-based and subtree-based metadata management schemas, a metadata server (MDS) clustering schema based on consistent hash structure, named DSVL, is proposed. By introducing virtual MDS into the consistent MDS cluster, it can effectively balance MDS cluster load. Besides, the standby mechanism and the lazy update policy are merged and applied to the MDS cluster to accomplish rapid MDS failover as well as zero data migration in case of cluster alteration. The results of prototype system and simulation test prove that DSVL, with the features of balanced metadata distribution, rapid failover, flexible scalability and zero data migration amount in case of node alteration, can meet the demands for flexible and effective management of large-scale data storage systems with ever increasing data amount.

Efficient Exact Regenerating Codes for Byzantine Fault Tolerance in Distributed Networked Storage

Today's large-scale distributed storage systems are commonly built using commodity software and hardware. As a result, crash-stop and Byzantine failures in such systems become more and more prevalent. In the literature, regenerating codes have been shown to be a more efficient way to disperse information across multiple storage nodes and recover from crash-stop failures. In this paper, we propose a novel decoding design of product-matrix constructed regenerating codes in conjunction with integrity check that allows exact regeneration of failed nodes and data reconstruction in the presence of Byzantine failures. A progressive decoding mechanism is incorporated in both procedures to leverage computation performed thus far. Unlike previous works, our new regenerating code decoding has the advantage that its building blocks, such as Reed-Solomon codes and standard cryptographic hash functions, are relatively well-understood because of their widespread applications. The fault tolerance and security properties of the proposed schemes are also analyzed. In addition, the performance of the proposed schemes, in terms of the average number of access nodes and the reconstruction failure probability versus the node failure probability, are also evaluated by Monte Carlo simulations.

Three-level Mapping Hash Method in Large-scale Distributed Storage System

Three-level Mapping Hash Method in Large-scale Distributed Storage System

Dynamic grid quorum: a reconfigurable grid quorum and its power optimization algorithm

Power-saving has become a central issue for well-configured SOC platforms. In particular, as a high percentage of the total energy is used by the storage systems, the cost effectiveness of data management is equally as important as reliability and availability. To address this issue, we propose the dynamic grid quorum as a method for reducing the power consumption of large-scale distributed storage systems. The basic principle of our approach is to skew the workload toward a small number of quorums. This can be realized using the following three techniques. First, our system allows reconfiguration by exchanging nodes without any data migration, so that high-capacity nodes can be reallocated to busier quorums. Second, for more effective skewing of the workload, we introduce the notion of dual allocation, which makes it possible to consider two distinct allocations in the same grid for write and read quorums. Finally, we present an optimization algorithm to find a pair of a strategy and an allocation of nodes, which minimizes power for a given system setting and its workload. We also demonstrate that the dynamic grid quorum saves, on average, 14---25% energy compared with static configurations, when the intensity of the total workload changes.

A Data Object Placement Algorithm Based on Dynamic Interval Mapping

Efficient and scalable data management becomes increasingly important in large-scale distributed storage systems. A key enabling technique is a flexible, balancing and scalable data object placement and location scheme that automatically adapts to the additions or departures of storage nodes. In this paper, a data object placement algorithm based on dynamic interval mapping is proposed, which is probabilistically optimal in both distributing data evenly and minimizing data movement when storage nodes is changed. Moreover, this algorithm supports weighted allocation of the storage nodes and variable levels of the object replication.