Multi Storage Research Articles

Efficient Scheduling of Data Transfers in Multi-tiered Storage

Multi-tiered persistent storage systems integrate many types of persistent storage devices, such as different types of NVMes, SSDs, and HDDs. This integration provides a multi-level view of persistent storage, where each tier has a different data transmission speed and capacity. Data transfer processes operating on multi-tiered persistent storage allow for the parallelisation of data transfers among the partitions of data at the same or different tiers. This work considers the problem of efficient scheduling of parallel data transfers between the tiers of persistent storage. We consider a data processing model where several data transfer processes move or copy data from one tier to another through the buffers in transient memory. We propose a new model for data processing over multi-tiered persistent storage and new algorithms to minimise both the overall time spent on parallel data transfers and the idle time of data transfer processes. We also describe how the scheduling algorithms dynamically apply different procedures to assign data transfers to the processes. Finally, we present the outcomes from the experiments that confirm the correctness and efficiency of the scheduling algorithms.

Machine Learning Based Power Regulation for Smart Multi Storage Building

Machine Learning Based Power Regulation for Smart Multi Storage Building

Technical, Economic, and Intelligent Optimization for the Optimal Sizing of a Hybrid Renewable Energy System with a Multi Storage System on Remote Island in Tunisia

Due to their small dimension and isolated energy systems, islands face a significant energy supply challenge. In general, they use fossil fuels for electricity generation. Fossil fuels are a major source of CO2 emissions, and they are very costly. The cost of electricity generation on islands is up to 10 times higher than on the mainland. This situation without a doubt represents a financial burden for the islanders. Using renewable sources, especially solar and wind sources, offers great potential for power generation in remote locations, as they are a clean and inexhaustible source of energy. Electrifying these zones with a hybrid system consisting of a photovoltaic (PV) and wind systems associated to a hydraulic and an electrochemical storage system is a promising alternative. The purpose of this study is to optimize the dimension of the components generation of systems, especially for a remote island in Tunisia. The first part of this object outlines the PV-wind-battery-hydraulic generation system architecture and modeling. The optimal sizing of the device additives, satisfying two criteria with the aid of evolutionary algorithms NSGAII, is defined inside the second part of this article. The outcomes are discussed from the point of view of the importance of the system dimension and in terms of compliance with the criteria through the study of the most optimal particular configurations.

FiLiP: A File Lifecycle-based Profiler for hierarchical storage

The increasing gap between computing speed and storage latency leads to possible I/O bottlenecks on massively parallel computers. To mitigate this issue, hierarchical storage provides multi-tiered configurations where each tier has its own physical characteristics and associated performance. Selecting the most appropriate file placement policy on this multi-tiered storage is difficult and there is to our knowledge no tool that systematically provides statistics and metrics for optimal file policy selection. In this paper, we present FiLiP (File Lifecycle Profiler), a software which provides statistics and metrics for a better understanding of file access by applications and the consequences on file movements across hierarchical storage. After the description of FiLiP’s main features and architecture, we highlight the usefulness of our tool using three I/O intensive simulation HPC applications: NEMO, S3DIO and NAMD and a three-tiered burst buffer.

Programming Abstractions for Managing Workflows on Tiered Storage Systems

Scientific workflows in High Performance Computing ( HPC ) environments are processing large amounts of data. The storage hierarchy on HPC systems is getting deeper, driven by new technologies (NVRAMs, SSDs, etc.) There is a need for new programming abstractions that allow users to seamlessly manage data at the workflow level on multi-tiered storage systems, and provide optimal workflow performance and use of storage resources. In previous work, we introduced a software architecture Managing Data on Tiered Storage for Scientific Workflows (MaDaTS ) that used a Virtual Data Space ( VDS ) abstraction to hide the complexities of the underlying storage system while allowing users to control data management strategies. In this article, we detail the data-centric programming abstractions that allow users to manage a workflow around its data on the storage layer. The programming abstractions simplify data management for scientific workflows on multi-tiered storage systems, without affecting workflow performance or storage capacity. We measure the overheads and effectiveness introduced by the programming abstractions of MaDaTS. Our results show that these abstractions can optimally use the storage capacity in lesser capacity storage tiers, and simplify data management without adding any performance overheads.

Predicting file lifetimes for data placement in multi-tiered storage systems for HPC

The emergence of Exascale machines in HPC will have the foreseen consequence of putting more pressure on the storage systems in place, not only in terms of capacity but also bandwidth and latency. With limited budget we cannot imagine using only storage class memory, which leads to the use of a heterogeneous tiered storage hierarchy. In order to make the most efficient use of the high performance tier in this storage hierarchy, we need to be able to place user data on the right tier and at the right time. In this paper, we assume a 2-tier storage hierarchy with a high performance tier and a high capacity archival tier. Files are placed on the high performance tier at creation time and moved to capacity tier once their lifetime expires (that is once they are no more accessed). The main contribution of this paper lies in the design of a file lifetime prediction model solely based on its path based on the use of Convolutional Neural Network. Results show that our solution strikes a good trade-off between accuracy and under-estimation. Compared to previous work, our model made it possible to reach an accuracy close to previous work (around 98.60% compared to 98.84%) while reducing the underestimations by almost 10x to reach 2.21% (compared to 21.86%). The reduction in underestimations is crucial as it avoids misplacing files in the capacity tier while they are still in use.

Dissipativity theory and applications of nonlinear stochastic systems with Markov jump and Lévy noise

This paper focuses on the stochastic dissipativity theory and applications for nonlinear stochastic systems with Markov jump and Lévy noise. First, based on the dissipation inequality, multi supply rates, multi storage functions and stochastic analysis tools, the stochastic dissipation theory is established. And then the results can be utilized to obtain the sufficient conditions for LaSalle-type stability theorem of stochastically dissipative system. In addition, the conclusions on extended stochastic Kalman–Yakubovich–Popov (KYP) conditions and stochastic feedback passivity are also derived. Finally, two examples are given to validate the effectiveness of our results.

Assessment load capacity in floor constructions

The desire to use warehouse space with maximum efficiency has led to the development of the warehouse technology. The emergence of the new types of material handling equipment, an increase in their movement speed, the emergence of new multi-tiered storage systems inevitably affects the carrying capacity of the floor structure. It should also be borne in mind that in the course of operation, the ability of the floor construction to absorb external load decreases. That is why before the start of the operation at the old storage facilities it is advisable to assess their carrying capacity. One of the main types of inspections of building structures is the examination of the concrete floor, which is carried out in order to assess its strength, detect various types of defects, as well as check the operating conditions for compliance with building codes. The most frequent reason for technical inspection of the floor is the reconstruction or overhaul of a construction site. Therefore, the examination of the floor becomes a prerequisite for the object launch in operation. In addition, the reason for the survey may be doubts about the quality of the floors after putting them into operation; increased floor load; detection of cracks or the development of old cracks on the floor covering. In the article the authors consider the examination of the floor construction of warehouse premises belonging to the enterprise of Belgorod.

Cluster mechanism for hot data storage in HBase system

The evolution in different areas like e-business contributed to increase the number of data. So, a scalable database is required to accommodate a large number of data. Moreover, this data could be important and the number of accessing for reading or writing operations can make a pressure in some servers more than others, this pressure called unbalanced accessing. This will be achieved if a system that guarantees the distribution of this important data in useful form is used, because the pressure can causes a delay. Accordingly, when we design a multi storage node to keep this data, the pressure will transfer from a server to another. So, the proposed system searched for a satisfied solution to distribute all the users on the servers in a useful way to detect any mistake or repeating updating operation data by using identification feature. This method is applied on Hbase and called Clusterization.

Lost in the Clouds? Cloud Storage Fits into Media Workflows—Just Not Everywhere

Within the media and entertainment space, the question businesses face with respect to cloud infrastructure is not whether to use it but how best to use it. Public cloud storage provides flexible, on-demand storage capacity and an operating expense financial model. While moving media post-production workflows entirely into “The Cloud” would solve content storage, sharing, and distribution challenges, there are limitations and cost implications. Cloud storage has attributes that make it perfectly suited for some workflow stages and perfectly ill-suited for others. The use of private cloud is adding another dimension to this tiered storage approach, offering many advantages. This paper describes a multitier storage strategy that maximizes the cost, access, and performance benefits of different types of storage with a particular focus on the most appropriate and beneficial uses of private and public cloud. It covers how the right storage and data management technology can aid media companies in making the most of all tiers of storage including cloud infrastructures.

EAD: elasticity aware deduplication manager for datacenters with multi-tier storage systems

The popularity of Big Data applications places pressures on storage systems to efficiently scale to meet the demand. At the same time, new developments like solid-state drives have changed to traditional storage hierarchy. Cloud storage systems are transitioning towards a hybrid architecture consisting of large amounts of memory, solid-state disks (SSDs), and traditional magnetic hard disks (HD). This paper presents elasticity aware deduplication (EAD), a data deduplication framework designed for multi-tier cloud storage architectures consisting of SSD and HD. EAD dynamically adjusts the deduplication parameters at runtime in order to improve performance. Experimental results indicate that EAD is able to detect more than 98% of all duplicate data, but it only consumes less than 5% of expected memory space. Additionally, EAD saves approximately 74% of overall IO access cost compared to the traditional design.

ExaPlan

Multi-tiered storage, where each tier consists of one type of storage device (e.g., SSD, HDD, or disk arrays), is a commonly used approach to achieve both high performance and cost efficiency in large-scale systems that need to store data with vastly different access characteristics. By aligning the access characteristics of the data, either fixed-sized extents or variable-sized files, to the characteristics of the storage devices, a higher performance can be achieved for any given cost. This article presents ExaPlan, a method to determine both the data-to-tier assignment and the number of devices in each tier that minimize the system’s mean response time for a given budget and workload. In contrast to other methods that constrain or minimize the system load, ExaPlan directly minimizes the system’s mean response time estimated by a queueing model. Minimizing the mean response time is typically intractable as the resulting optimization problem is both nonconvex and combinatorial in nature. ExaPlan circumvents this intractability by introducing a parameterized data placement approach that makes it a highly scalable method that can be easily applied to exascale systems. Through experiments that use parameters from real-world storage systems, such as CERN and LOFAR, it is demonstrated that ExaPlan provides solutions that yield lower mean response times than previous works. It supports standalone SSDs and HDDs as well as disk arrays as storage tiers, and although it uses a static workload representation, we provide empirical evidence that underlying dynamic workloads have invariant properties that can be deemed static for the purpose of provisioning a storage system. ExaPlan is also effective as a load-balancing tool used for placing data across devices within a tier, resulting in an up to 3.6-fold reduction of response time compared with a traditional load-balancing algorithm, such as the Longest Processing Time heuristic.

Archiving and Managing Remote Sensing Data using State of the Art Storage Technologies

Abstract. Integrated Multi-mission Ground Segment for Earth Observation Satellites (IMGEOS) was established with an objective to eliminate human interaction to the maximum extent. All emergency data products will be delivered within an hour of acquisition through FTP delivery. All other standard data products will be delivered through FTP within a day. The IMGEOS activity was envisaged to reengineer the entire chain of operations at the ground segment facilities of NRSC at Shadnagar and Balanagar campuses to adopt an integrated multi-mission approach. To achieve this, the Information Technology Infrastructure was consolidated by implementing virtualized tiered storage and network computing infrastructure in a newly built Data Centre at Shadnagar Campus. One important activity that influences all other activities in the integrated multi-mission approach is the design of appropriate storage and network architecture for realizing all the envisaged operations in a highly streamlined, reliable and secure environment. Storage was consolidated based on the major factors like accessibility, long term data protection, availability, manageability and scalability. The broad operational activities are reception of satellite data, quick look, generation of browse, production of standard and valueadded data products, production chain management, data quality evaluation, quality control and product dissemination. For each of these activities, there are numerous other detailed sub-activities and pre-requisite tasks that need to be implemented to support the above operations. The IMGEOS architecture has taken care of choosing the right technology for the given data sizes, their movement and long-term lossless retention policies. Operational costs of the solution are kept to the minimum possible. Scalability of the solution is also ensured. The main function of the storage is to receive and store the acquired satellite data, facilitate high speed availability of the data for further processing at Data Processing servers and help to generate data products at a rate of about 1000 products per day. It also archives all the acquired data on tape storage for long-term retention and utilization. Data sizes per satellite pass range from hundreds of megabytes to tens of gigabytes The images acquired from remote sensing satellites are valuable assets of NRSC and are used as input for further generation of different types of user data products through multiple Data Processing systems. Hence, it is required to collect and store the data within a shared, high speed repository concurrently accessible by multiple systems. After the raw imagery is stored on a high-speed repository, the images must be processed in order for them to be useful for value-added processing or for imagery analysts. The raw image file has to be copied on to data processing servers for further processing. Given the large file sizes, it is impractical to transfer these files to processing servers via a local area network. Even at gigabit Ethernet rates (up to 60 MB/s), a 5 GB file will take at least 83 seconds. For this reason, it is useful to employ a shared file system which allows every processing system to directly access the same pool where raw images were stored. Concurrent access by multiple systems is ensured for processing and generation of data products. With the above reasons, it was chosen to have high speed disk arrays for acquisition and processing purposes and tape based storage systems for long-term huge data (Peta Bytes) archival in a virtualized multitier storage architecture. This paper explains the architecture involved in a virtualized tiered storage environment being used for acquisition, processing and archiving the remote sensing data. It also explains the data management aspects involved in ensuring data availability and archiving Peta bytes sized, remote sensing data acquired over the past 40 years.

Development of Intelligent Network Storage System with Adaptive Decision-Making

An enormous amount of digitization of content has led to extreme demands on storage systems. Multitier storage systems built with parallel distributed file-system enhance their scalability and I/O performance, while achieving cost-effectiveness in handling large data. However, these storage systems are not flexible enough to adapt to the complexity and dynamic environment of data. Designing a storage system to be more intelligent is a proven technique to overcome the challenges of managing complex data in a dynamic environment and hence, this paper proposes an Intelligent Network Storage System (INSS) with adaptive decision-making capability. An intelligent agent is introduced for autotiering and it can dynamically decide which storage tier data can be relocated based on their recent access patterns. The intelligent agent’s design and implementation were inspired by the human brain and consequently, the functions from each of brain’s five core regions were developed in INSS prototype. The intelligent agent utilizes Reinforcement Learning (RL) which guides the system itself to learn the optimal policies in relocating data. The prototype was built on a test bed with Lustre: A SAN file system for Linux with one metadata server and 3 storage nodes. With its adaptive and real-time decisionmaking capability, the policies optimized by RL were evaluated and the analysis has shown to achieve a high degree of accuracy with no performance penalty.

LOFAR Information System

In this paper, we present a newly designed and implemented scientific information system for the LOFAR Long Term Archive. It is a distributed multi-tier storage and data processing system that allows a number of users to process Petabytes of data. The LOFAR Information System is designed on the base of the Astro-WISE Information System. The system allows the use of the computing and storage resources of the BiGGrid combined with the metadata database along with the data access and data processing interfaces of Astro-WISE. The architecture of the system, problems solved during the implementation and scientific use cases for the system are also described.

Turning Back Time--What Impact on Performance?

Consistent with the divide-and-conquer approach to problem solving, a recursive result is presented in the domain of stochastic modelling that derives product-form solutions for the steady state probabilities of certain networks composed from interacting Markov chains. Practical applications include multi-tasking operating systems, communication channels and multi-tiered storage systems. The approach is also applied to the computation of response time quantiles, which are vital in transaction processing, computer communication service level agreements and other operational systems. The joint probability distribution of the sojourn times of a tagged task at each node in a network is determined by noting that this is the same in both the forward and reversed processes. In this way, existing results for response time probability densities in tandem, tree-like, and overtake-free Markovian queueing networks are quickly and systematically obtained. We further show how to apply the method in more general networks.

Soft Error Hardened Latch and Its Estimation Method

We propose soft error robust latches which have multi storage nodes and present their efficiencies. The key technology of the latch is a feedback loop circuit with a data node and four gates. We also discuss a method of soft error estimation in robust circuits in this paper. The soft error immunity of this feedback loop circuit is estimated by circuit simulations with two models. The soft error immunity of the latch is estimated by device simulation more accurately. By these precise simulations, the latch is proven to be highly tolerant to soft errors. In addition, the latch protects from not only retention data upset but also transient noise releasing. The latch provides high immunity against all soft error problems with a simple circuit. It is easy to apply the latch technique to various latches, such as single latches, scan latches, and flip-flops.

A reinforcement learning framework for online data migration in hierarchical storage systems

Multi-tier storage systems are becoming more and more widespread in the industry. They have more tunable parameters and built-in policies than traditional storage systems, and an adequate configura...

LND

The expensive overhead of synchronous I/O has now become the bottleneck of increasing efficiency of data storage in transaction-based systems. This paper proposes a multi-tier storage device, the LND (Local-Network Ram- Disk) device, which uses idle memories and disks in NOW (Network of Workstations) as persistent repositories so that access latency is transformed from magnetic disk seek-and-transfer latency to the sum of network transfer latency, memory write latency, and asynchronous I/O latency. Two fast data consistency protocols based on multicast technique are also introduced to improve performance without sacrificing reliability. Experimental results show that the LND device is one order of magnitude faster than traditional disks in synchronous I/O performance. Moreover, a fault tolerant recovery algorithm is proposed to protect the system from various kinds of failures and to facilitate quick recovery from crashes.