HintStor: A Framework to Study I/O Hints in Heterogeneous Storage

Parallel file systems commonly distribute a file across multiple file servers with a fixed-size strip, so that the data access can be served through multiple devices simultaneously, which is similar to RAID 0. The default data layout works well in traditional homogeneous storage systems, but as the Solid state disks (SSD) are conduct into storage system, the data layout of heterogeneous parallel storage systems need to reconsider in order to get a better performance. In this paper, we propose a strip level data layout strategy for heterogeneous parallel storage systems (SLDP), which find the optimal strip configurations for each solid state disk file server node (SNode)and mechanical hard disk drive file server node (HNode), according to data access pattern, along with the consideration of the limitation of SSD space, use variable strip size to reorganize the data layout of storage systems. the main idea is let the key regions of file get the optimal strip configuration, which can obviously improve the system performance, and rest part of the file are distributed according to the SSD free space threshold, which may reduce the system performance, SLDP can find a data layout that makes the improvement is much bigger than the reduction, so that to get totally better performance of the heterogeneous storage system. Experimental results show that the SLDP is feasible and can improve system performance.

Distributed filesystems are a typical solution in networked environments as clusters and grids. Parallel filesystems are a typical solution in order to reach high performance I/O distributed environment, but those filesystems have some limitations in heterogeneous storage systems. Usually in distributed systems, load balancing is used as a solution to improve the performance, but typically the distribution is made between peer-to-peer computational resources and from the processor point of view. In heterogeneous systems, like heterogeneous clusters of workstations, the existing solutions do not work so well. However, the utilization of those systems is more extended every day, having an extreme example in the grid environment. In this paper we bring attention to those aspects of heterogeneous distributed data systems presenting a parallel file system that take into account heterogeneity of storage nodes, the dynamic addition of new storage nodes, and an algorithm to group requests in heterogeneous systems.

To improve the performance data management efficiency of welded pipe used in the 2nd west to east gas pipeline, a distributed performance data acquiring and management system for welded pipe was developed. The system was established by using two-level three-tier Client/Server model. Performance data for welded pipe from different factories was acquired at clients and input to local database. The data then can be analyzed at local, or be transferred to the server through networks for unified storage and analysis. The System has functions as data management, file transferring, standard technology condition management, statistical analysis, figure displaying and statistical analysis report generation, etc. The system was implemented using VC++. Oracle and Access is adopted as database for server and client respectively, XML is as the data encapsulation for transferring file. The developed system plays a significant role in analyzing and evaluating the whole quality of welded pipe, and is useful for pipeline quality control.

As computers become pervasive and data size increases dramatically, data management systems' security, scalability and availability features turn into major design issues, especially in distributed computingenvironments. This paper proposes a Highly Available, Scalable and Secure distributed data storage system (HASS) for high performance and secure data management. Distributed and parallel data storage or file systems such as Object-based Storage Devices (OSD) and flexible key distribution schemes such as stateless Identity Based Encryption (IBE) are integrated to achieve scalability in terms of performance and key management. OSD provides high performance parallel I/O whereas IBE eliminates pre-shared secrete/symmetric keys and simplifies key distribution. Data at rest (static) and in transit (dynamic) are protected with different encryption strategies for privacy and integrity. With IBE, public keys are not stored whereas private and session keys are generated dynamically for data in transit protection/encryption. SecretSharing is used for data at rest protection. Replication on OSD sites duplicates data shares for high availability. Overall, the proposed HASS system delivers high performance data management with security, scalability and high availability features.

In these times, scientific data intensive applications generate ever-increasing volumes of data that need to be stored, managed, and shared between geographically distributed communities. Data centres are normally able to provide tens of petabytes of storage space through a large variety of heterogeneous storage and file systems. However, storage systems shared by applications need a common data access mechanism, which allocates storage space dynamically, manages stored content, and automatically remove unused data to avoid clogging data stores. To accommodate these needs, the concept of Storage Resource Managers (SRMs) was devised in the context of a project that involved High Energy Physics (HEP) and Nuclear Physics (NP). The Storage Resource Manager (SRM) interface specification was defined and evolved into an international collaboration in the context of the Open Grid Forum (OGF). SRM interface provides the technology needed to share geographically distributed heterogeneous storage resources, with an effective and common interface regardless of the type of the back-end system being used. By implementing the SRM interface, grid storage services provide a consistent homogeneous interface to the Grid to manage storage resource as well as advanced functionality such as dynamic space allocation and file management on shared storage systems. Within Worldwide LHC Grid project exists more than five interoperating implementations of SRM services, and every one shows peculiarity. In this paper, we describe the flexibility of StoRM service, an implementation of the Storage Resource Manager Interface version 2.2. StoRM is designed to foster the adoption of cluster file systems, and thanks to the marked flexibility, StoRM can be used in small data centre with human resource deficiency to administer an other grid service and, at the same time, capable to grow in terms of storage managed and workload. StoRM can be used to manage any storage resources with any kind of POSIX file-system in a transparent way. As demonstration of the StoRM flexibility, the paper describes how applications scheduled via Grid can access files on a file-system directly via POSIX calls, how StoRM can be deployed in a clustered configuration to address scalability needs and finally how StoRM can be used to manage also storage classes based on Storage Cloud, like Amazon Simple Storage Service (S3).

Data replication is a key technique to achieve data availability, reliability, and optimized performance in distributed storage systems and data centers. In recent years, with the emergence of new storage devices, heterogeneous object-based storage system, such as a storage system with the co-existence of hard disk drives and solid state drives, have become increasingly attractive as they combine merits of different storage devices to deliver better promise. However, existing data replication schemes do not place data based on heterogeneous device characteristics as well as considering distinct data access patterns. In this paper, we introduce a novel data replication scheme PRS to achieve efficient data replication for heterogeneous storage systems. Different from traditional schemes, the PRS groups objects according to data access patterns and distributes replicas to heterogeneous devices with their features. It uses a pseudo random algorithm to optimize replica layout by considering storage device performance and capacity. The experimental results confirm that PRS is a highly efficient replication scheme for heterogeneous storage systems.

The data scale in many data centers is growing explosively with emerging applications and usages of big data technologies. Data distribution is a key issue in large-scale distributed storage systems to place petabytes of data or even beyond, among tens or hundreds of thousands of storage devices. In the meantime, heterogeneous storage systems, such as those having devices with hard disk drives (HDDs) and storage class memories (SCMs), have become increasingly popular for massive data storage due to balanced performance, capacity, and cost. Current data distribution algorithms can achieve efficient, scalable, and balanced mapping, but do not distinguish different characteristics of heterogeneous devices well. This paper presents a novel data distribution algorithm called SUORA (Scalable and Uniform storage via Optimally-adaptive and Random number Addressing), to take full advantage of heterogeneous devices. SUORA is a pseudo-random algorithm that uniformly distributes data cross a hybrid and tiered storage cluster. It divides heterogeneous devices, maps them onto different buckets and assigns them to various segments in each bucket. A pseudo-random and deterministic number sequence is generated to map data among segments and devices. Data movement is performed for achieving better read throughput while keeping load balance according to data hotness and bucket threshold. With considering distinct characteristics of heterogeneous storage devices well, the SUORA algorithm achieves a highly efficient adaptive data distribution for data centers and heterogeneous storage systems.

Distributed storage systems play an increasingly critical role in data centers to meet the ever-increasing data growth demand. Heterogeneous storage systems, with the coexistence of hard disk drives (HDDs) and solid state drives (SSDs), can be an attractive distributed store solution due to the balanced performance, large capacity, and economic cost. The consistent hashing distribution algorithm that is widely used in distributed storage systems can achieve scalable and flexible mappings, but do not take full advantages of different characteristics of heterogeneous devices. In this research, we propose a hierarchical consistent hashing (HiCH) algorithm to better manage data distribution in a heterogeneous object-based storage system and better explore the potential of heterogeneous devices. HiCH divides heterogeneous storage devices into different buckets and applies separate consistent hashing rings for each bucket. It places data into various hashing rings according to the hotness, access time, and other data access patterns. The evaluation based on the Sheepdog, a distributed objectbased storage system, confirms that HiCH can improve the performance of storage systems, and also make better utilization of heterogeneous storage devices.

Scientific applications from many problem domains produce and/or access large volumes of data. To support these applications, designers of high-end computing (HEC) systems have greatly increased the capacity of storage systems in recent years. However, because hard disk drives (HDDs) are still the dominant storage device used in HEC storage systems, and because HDD performance has not improved as quickly as the capacity, it can be challenging to deploy a storage system that provides both extreme capacity and extreme performance at a reasonable cost. Solid State Drives (SSDs) are a promising high- bandwidth and low-latency alternative to HDDs for HEC storage systems, but they too have deficiencies: small capacity, limited write cycles, and high cost when compared to HDDs. Because of their complementary characteristics, storage system designers are beginning to consider heterogeneous storage system designs that include both HDDs and SSDs. However, managing the workload so as to take advantage of the strengths of each type of storage device while controlling overhead is a major challenge. In this study, we propose a novel approach for managing a heterogeneous storage system called the Working Set-based Reorganization Scheme (WS-ROS). With WS-ROS, applications write to both HDDs and SSDs using all the available storage system bandwidth. Later, a background process reorganizes the data so as to place the data most likely to be read on SSDs while relegating the data most likely to be written and the data not likely to be accessed onto the slower but higher-capacity HDDs. For our evaluation workloads, the WS-ROS approach provided a 3× to 10× performance improvement compared to a heterogeneous storage system without a working set-based data reorganization scheme, suggesting the value of lazy reorganization of data based on data access working sets.

Solid-state drives (SSDs) have been added into storage systems for improving their performance, which will bring the heterogeneity into the storage medium. The throughput is one of the essential resources in heterogeneous storage systems, and how to allocate the throughput plays a crucial role in user performance. There are many types of research on the throughput allocation of heterogeneous storage systems. However, the throughput allocation of heterogeneous storage is facing new challenges in a dynamic setting, where users are not present in the system simultaneously, and enter the system dynamically. Drawing on economic game-theory, researchers have proposed many methods to tackle dynamic throughput allocation issues for heterogeneous storages, cross out enjoying Sharing Incentive (SI), Envy Freeness (EF), and Pareto Optimality (PO). However, they either relax constraints of fairness property to cause the allocation with weak fairness or interrupt some users present in the system to give up a piece of their allocations for new users entering the system, which will degrade these donors’ performance. Moreover, all of existing methods will cause lower resource utilization due to constraints of users’ dominant share equality. In this article, we propose a dynamic throughout allocation method based on gradual increase (DAGI), which can adapt to various workloads to make a fair allocation with a maximum resource utilization. Without relaxing constraints of fairness properties, when new users enter the system, DAGI can make a dynamic allocation with strong fairness by appropriately postponing the allocation of surplus throughputs, so this can provide an opportunity that DAGI can guarantee the final allocation with strong fairness when allocating remaining throughputs after all users are present in the system. Meanwhile, DAGI can gradually increase user allocation without reduction, which will not interrupt any users present in the system. Furthermore, DAGI can conduct a dynamic throughput allocation based on users’ local bottleneck resources, which can adapt to various workloads of users to improve resource utilization. Extensive experiments are conducted to prove the effectiveness of DAGI. The experimental results show that DAGI can achieve higher resource utilization and performance than existing methods, and can satisfy desirable game-theoretic properties with guaranteeing the strong fairness. In addition, DAGI gradually increases the allocation of each user without interrupting any user to reduce its allocation to degrade its performance.

In distributed storage systems (DSS), the storage costs and download costs with different storage nodes, in general, can be different. In such heterogeneous storage systems, how to establish a fundamental tradeoff between system storage cost and system repair cost is investigated. We formulate the problem of establishing the tradeoff between system storage cost and system repair cost as a bi‐objective linear programming problem subject to the min‐cut constraint of information flow graphs. We give a tight min‐cut bound for heterogeneous DSSs with general setting. Moreover, we show that the tradeoff between system storage cost and system repair cost of some special heterogeneous DSSs can be established in polynomial time. Copyright © 2014 John Wiley & Sons, Ltd.

Distributed storage systems provide reliability by distributing data over multiple storage nodes. Once a node fails, a new node is introduced to the system to maintain the availability of the stored data. The new node downloads information from other surviving nodes called helper nodes to recover the lost data in the failed node. The number of helper nodes is called repair degree. Compared to traditional approaches, e.g., replication and erasure codes, the regenerating codes proposed recently can significantly reduce the repair bandwidth in homogeneous distributed storage systems. Most existing works focus on uniform settings (e.g., in terms of repair degree and repair bandwidth). However, due to network structures or connectivity limitations, for each failed node, the number of required helper nodes may be different for distinct failed nodes. Furthermore, considering the limits of network traffic of bandwidth, the amount of information allowed to be downloaded from each helper node could also vary. Thus we are motivated to investigate heterogeneous distributed storage systems where the repair degree and the amount of information downloaded from each helper node can be different. In order to obtain the minimal bandwidth to recover a failed node, we construct an information flow graph for such heterogeneous systems. By analyzing the cut-set bound of the information flow graph, the optimal tradeoff between storage capacity and repair bandwidth is derived. We then propose asymmetric regenerating codes that can achieve the curve of the optimal tradeoff. A linear construction of asymmetric regenerating codes is presented. Compared with previous regenerating codes, asymmetric regenerating codes are shown to have a lower repair bandwidth under a certain constraint condition, whose reduction can be up to 36.2%.

In article presented the model of decision support system heterogeneous onboard storage system function. The main difference of model is the fact of considering stored data importance. Model can be applied for analyses of different structures of data storage systems and ways of storage for different stages of lifecycle (from constructing to exportation). Modelling of data storage system function based on decomposition information processes of system, detection their interconnections and dependences of quality indicators from system structure and ways of storage.

In article presented the model of Earth remote sensing satellite heterogeneous onboard storage system function. The main difference of model is the fact of considering stored data importance. Model can be applied for analyses of different structures of data storage systems and ways of storage for different stages of lifecycle (from constructing to exportation). Modelling of data storage system function based on decomposition information processes of system, detection their interconnections and dependences of quality indicators from system structure and ways of storage.

Storage management is one of the most important enabling technologies for large-scale scientific investigations. Having to deal with multiple heterogeneous storage and file systems is one of the major bottlenecks in managing, replicating, and accessing files in distributed environments. Storage resource managers (SRMs), named after their Web services control protocol, provide the technology needed to manage the rapidly growing distributed data volumes, as a result of faster and larger computational facilities. SRMs are grid storage services providing interfaces to storage resources, as well as advanced functionality such as dynamic space allocation and file management on shared storage systems. They call on transport services to bring files into their space transparently and provide effective sharing of files. SRMs are based on a common specification that emerged over time and evolved into an international collaboration. This approach of an open specification that can be used by various institutions to adapt to their own storage systems has proven to be a remarkable success - the challenge has been to provide a consistent homogeneous interface to the grid, while allowing sites to have diverse infrastructures. In particular, supporting optional features while preserving interoperability is one of the main challenges we describe in this paper. We also describe using SRM in a large international high energy physics collaboration, called WLCG, to prepare to handle the large volume of data expected when the Large Hadron Collider (LHC) goes online at CERN. This intense collaboration led to refinements and additional functionality in the SRM specification, and the development of multiple interoperating implementations of SRM for various complex multi- component storage systems.