Hierarchical Storage Research Articles

A facile trace potassium assisted catalytic activation strategy regulating pore topology of activated coke for combined removal of toluene/SO2/NO

Coal-fired organic pollutant emission is regarded as an important precursor to regional air pollution. Adsorption using activated coke has the potential to achieve the simultaneous removal of multiple gas pollutants such as SO2, NOx and the volatile organic compounds (VOCs) in organic pollutants, for which developing low-cost activated coke adsorbents with suitable pore configuration is the key. In this work, we propose a green and facile trace potassium assisted catalytic activation strategy to prepared coal-based activated cokes with simultaneously large surface areas, hierarchical pore topology and high production yield. By simply adjusting the type of potassium species and amounts added, potassium induced catalytic activities for the reaction between CO2 and coal framework can be controlled, thus yielding two types of activated cokes, namely microporous and hierarchically porous activated cokes. Evaluated as adsorbents, the optimized hierarchically porous KP_AC_2 shows outstanding toluene capacities in both single adsorption (300.3 mg·g−1) and combined adsorption (262.5 mg·g−1) cases. Moreover, KP_AC_2 also exhibits good SO2 and NO adsorption properties in combined adsorption cases due to the hierarchical storage of various adsorbates. Furthermore, the effects of H2O, O2 and temperature are investigated, providing guidelines for actual flue gas government. This work provides a green, low-cost and scalable method for preparing activated coke adsorbents used in combined adsorption of SO2, NO and VOCs, which hold potentials to replace the traditional physical or chemical activation craft and is promising in the field of coal-fired flue gas purification.

Heat Prediction of High Energy Physical Data Based on LSTM Recurrent Neural Network

High-energy physics computing is a typical data-intensive calculation. Each year, petabytes of data needs to be analyzed, and data access performance is increasingly demanding. The tiered storage system scheme for building a unified namespace has been widely adopted. Generally, data is stored on storage devices with different performances and different prices according to different access frequency. When the heat of the data changes, the data is then migrated to the appropriate storage tier. At present, heuristic algorithms based on artificial experience are widely used in data heat prediction. Due to the differences in computing models of different users, the accuracy of prediction is low. A method for predicting future access popularity based on file access characteristics with the help of LSTM deep learning algorithm is proposed as the basis for data migration in hierarchical storage. This paper uses the real data of high-energy physics experiment LHAASO as an example for comparative testing. The results show that under the same test conditions, the model has higher prediction accuracy and stronger applicability than existing prediction models.

Mass storage interface LTSM for FAIR Phase 0 data acquisition

Since 2018 several FAIR Phase 0 beamtimes have been operated at GSI, Darmstadt. Here the new challenging technologies for the upcoming FAIR facility shall be tested while various physics experiments are performed with the existing GSI accelerators. One of these challenges concerns the performance, reliability, and scalability of the experiment data storage. Raw data as collected by event building software of large scale detector data acquisition has to be safely written to a mass storage system like a magnetic tape library. Besides this long term archive, it is often required to process this data as soon as possible on a high performance compute farm. The C library LTSM (“Lightweight Tivoli Storage Management”) has been developed at the GSI IT department based on the IBM TSM software. It provides a file API that allows for writing raw listmode data files via TCP/IP sockets directly to an IBM TSM storage server. Moreover, the LTSM library offers Lustre HSM (“Hierarchical Storage Management”) capabilities for seamlessly archiving and retrieving data stored on Lustre file system and TSM server. In spring 2019 LTSM has been employed at the FAIR Phase 0 beamtimes at GSI. For the HADES experiment LTSM was implemented into the DABC (“Data Acquisition Backbone Core”) event building software. During the 4 weeks of Ag+Ag@1.58 AGeV beam, the HADES event builders have transferred about 400 TB of data via 8 parallel 10 GbE sockets, both to the TSM archive and to the “GSI green cube” HPC farm. For other FAIR Phase 0 experiments using the vintage MBS (“Multi Branch System”) event builders, an LTSM gateway application has been developed to connect the legacy RFIO (“Remote File I/O”) protocol of these DAQ systems with the new storage interface.

Grouping-Based Consistency Protocol Design for End-Edge-Cloud Hierarchical Storage System

With the increasing of the number of edge-devices and the demand of real-time experience, the end-edge-cloud hierarchical storage system (EECHSS) is emerged recently for reliable caching and fast offloading of massive amounts of data. EECHSS can accommodate various services, save computing power and improve storage capacity, due to transformation from central-cloud to edge-cloud, considerable reducing service delay and communication overhead. One of the main challenges brought by edge-cloud architecture is consistency problem. It is difficult to guarantee that the data from the two distributed clusters is consistent by existing consistency protocols. In addition, as the variety of applications increases, the existing fixed consistency level of classical protocols can no longer satisfy the system dynamic requirements. In this paper, we focus on designing grouping-based consistency protocols with adaptively selecting consistency level in EECHSS. At first, we analyze the internal structure and the workflow of EECHSS, and devise two modified adaptive grouping-based consistency protocols (GM-Paxos and GEPaxos) with efficient grouping algorithms. Then, for the characteristics that data is offloaded frequently, we design two synchronization strategies to ensure the consistency of the data cached in the edge-cloud and the central-cloud, respectively. Experiments show that, our proposed grouping-based consistency protocols of EECHSS can improve the availability as much as possible while ensuring data consistent.

PAM: an efficient power-aware multilevel cache policy to reduce energy consumption of storage systems

Energy consumption is one of the most significant aspects of large-scale storage systems where multilevel caches are widely used. In a typical hierarchical storage structure, upper-level storage serves as a cache for the lower level, forming a distributed multilevel cache system. In the past two decades, several classic LRU-based multilevel cache policies have been proposed to improve the overall I/O performance of storage systems. However, few power-aware multilevel cache policies focus on the storage devices in the bottom level, which consume more than 27% of the energy of the whole system [1]. To address this problem, we propose a novel power-aware multilevel cache (PAM) policy that can reduce the energy consumption of high-performance and I/O bandwidth storage devices. In our PAM policy, an appropriate number of cold dirty blocks in the upper level cache are identified and selected to flush directly to the storage devices, providing high probability extension of the lifetime of disks in standby mode. To demonstrate the effectiveness of our proposed policy, we conduct several simulations with real-world traces. Compared to existing popular cache schemes such as PA-LRU, PB-LRU, and Demote, PAM reduces power consumption by up to 15% under different I/O workloads, and improves energy efficiency by up to 50.5%.

IClaire: A Fast and General Layout Pattern Classification Algorithm With Clip Shifting and Centroid Recreation

Layout pattern classification, which groups similar layout clips into clusters, underlies a variety of design for manufacturability (DFM) applications, such as hotspot library generation, hierarchical data storage, and yield optimization speedup. The key challenges of layout pattern classification are clip representation and clip clustering. In this paper, we present a fast and general layout pattern classification algorithm considering clip shifting and centroid recreation. Our simple but general clip representation captures both topology and density; we can handle not only rigid area match or edge displacement constraints but also variant edge tolerances and don’t care regions. For achieving a small cluster count, our clip clustering is guided by the natural grouping structure of layout clips. The clustering results are further improved by centroid recreation. Our experiments are conducted on 2016 CAD contest at ICCAD benchmark suite. Our results show that our algorithm outperforms the reference solution and all contest winning teams, delivering the smallest cluster count, fastest runtime, and 100% validity. Moreover, our algorithm with clip shifting and centroid recreation further reduces the cluster count effectively and efficiently. In addition to the good solution quality, the interplay between adopted data structures and our algorithm makes it fast and viable to be incorporated into practical DFM flows.

CSED: Client-Side encrypted deduplication scheme based on proofs of ownership for cloud storage

As digital data are explosively generated nowadays, data management becomes a critical problem, which makes cloud storage services important and popular. In reality, the storage overhead can be reduced significantly by performing date deduplication. Among the outsourced data, some of them are very personal and sensitive, and should be prevented for any leakage. Generally, if clients conventionally encrypt the data, deduplication is lost. Message-locked encryption (MLE) is a cryptographic primitive supporting encrypted data deduplication. A secure client-side deduplication scheme can be built upon MLE to reduce both communication and computation overhead for cloud storage systems, where a client interacts with the cloud server to check the duplicate data and only the data which has not been outsourced by other clients before is required to be uploaded. However, existing client-side encrypted data deduplication schemes are confronted with brute-force attacks that can recover files falling into a known set. Furthermore, existing schemes are vulnerable to illegal content distribution attacks, where the adversary can distribute data to other users via the cloud server without detecting. In this paper, we propose a secure and efficient client-side encrypted data deduplication scheme (CSED). In CSED, a dedicated key server is introduced in generating MLE keys to resist brute-force attacks. We propose a Bloom filter-based proofs of ownership (PoW) mechanism and integrate it into CSED to resist illegal content distribution attacks. Moreover, a hierarchical storage architecture is employed to improve the I/O efficiency on the cloud server. Security analysis and performance evaluation demonstrate that CSED is secure and efficient.

HDFS Optimization Strategy Based On Hierarchical Storage of Hot and Cold Data

HDFS has been the core storage service of Hadoop big data platform for many years, with the continuous development of deep learning, artificial intelligence and other technologies, the amount of data continues to grow, the initial batch processing offline scenario did not fit the user’s needs, the data’s access heat appears big difference. Therefore, the HDFS storage system needs to be optimized according to the degree of heat and cold of the data. The heterogeneous storage and erasure-correction code technology of HDFS can improve the storage efficiency to some extent, but users need to specify the behavior of the storage for that particular data. Therefore, it will cause a waste of cluster resources. SSM systems can follow a predetermined set of rules, according to the degree of heat and cold, it uses different storage strategies to optimize the data, so as to improve the efficiency of the whole storage system.

Hierarchical Storage Behavior of Tetrafluoroborate Anion in Graphite Electrodes

Anion-intercalated graphite is one of the feasible solutions for high-energy devices. Various characteristics of graphite may influence its overall performance. In this work, three homologous graphites with different particle sizes are selected as positive electrodes to accommodate tetrafluoroborate anion from propylene carbonate, based on activated carbon/graphite capacitors. In situ X-ray diffraction, Raman spectroscopy and electrochemical dilatometry are combined organically to identify the hierarchical storage behavior of anions in graphite electrodes. The graphite with minimal size exhibits a prior anion accumulation in the shallow region before notable expansion in the crystal structure. This extra charge storage leads to a higher capacity at the normal current density (0.1 A g−1), however, also drags down the anionic de-intercalation process and causes a significant capacity fade at high discharge rate.

Graph database-based network security situation awareness data storage method

With the rapid development of the Internet, network security situation awareness has attracted tremendous attention. In large-scale complex networks, network security situation awareness data presents the characteristics of large-scale, multi-source, and heterogeneous. Recently, much research work have been done on network security situation awareness. However, most of the existing methods store different types of data in different ways, which makes data query and analysis inefficient. To solve this problem, we propose a graph database-based hierarchical multi-domain network security situation awareness data storage method. In our scheme, we build a hierarchical multi-domain network security situation awareness model to divide the network into different domains, which can collect and dispose the awareness data more efficiently. Meanwhile, to unify our storage mode, we also define network security situation awareness data storage rules and methods based on graph database. Finally, extensive experiments on real datasets show that our proposed method is efficient compared to state-of-the-art storage models.

Dynamic sharing of tape drives accessing scientific data

The data management infrastructure operated at CNAF, the central computing and storage facility of INFN (Italian Institute for Nuclear Physics), is based on both disk and tape storage resources. About 40 Petabytes of scientific data produced by LHC (Large Hadron Collider) at CERN and other experiments in which INFN is involved are stored on tape. This is the highest latency storage tier within HSM (Hierarchical Storage Management) environment. Writing and reading requests on tape media are satisfied through a set of Oracle-StorageTek T10000D tape drives, shared among different scientific communities. In the next years, the usage of tape drives will become more intense not only due to the growing amount of scientific data to manage but also due to general trend to use tapes as “slow disk”, announced by the main user communities. In order to reduce hardware purchases, a key point is to minimize the inactivity periods of tape drives. In this paper we present a software solution designed to optimize the efficiency of the shared usage of tape drives in our environment.

ROS

The combination of the explosive growth in digital data and the demand to preserve much of these data in the long term has made it imperative to find a more cost-effective way than HDD arrays and a more easily accessible way than tape libraries to store massive amounts of data. While modern optical discs are capable of guaranteeing more than 50-year data preservation without media replacement, individual optical discs’ lack of the performance and capacity relative to HDDs or tapes has significantly limited their use in datacenters. This article presents a Rack-scale Optical disc library System, or ROS in short, which provides a PB-level total capacity and inline accessibility on thousands of optical discs built within a 42U Rack. A rotatable roller and robotic arm separating and fetching discs are designed to improve disc placement density and simplify the mechanical structure. A hierarchical storage system based on SSDs, hard disks, and optical discs is proposed to effectively hide the delay of mechanical operation. However, an optical library file system (OLFS) based on FUSE is proposed to schedule mechanical operation and organize data on the tiered storage with a POSIX user interface to provide an illusion of inline data accessibility. We further optimize OLFS by reducing unnecessary user/kernel context switches inheriting from legacy FUSE framework. We evaluate ROS on a few key performance metrics, including operation delays of the mechanical structure and software overhead in a prototype PB-level ROS system. The results show that ROS stacked on Samba and FUSE as network-attached storage (NAS) mode almost saturates the throughput provided by underlying samba via 10GbE network for external users, as well as in this scenario provides about 53ms file write and 15ms read latency, exhibiting its inline accessibility. Besides, ROS is able to effectively hide and virtualize internal complex operational behaviors and be easily deployable in datacenters.

Survey of Storage Systems for High-Performance Computing

In current supercomputers, storage is typically provided by parallel distributed file systems for hot data and tape archives for cold data. These file systems are often compatible with local file systems due to their use of the POSIX interface and semantics, which eases development and debugging because applications can easily run both on workstations and supercomputers. There is a wide variety of file systems to choose from, each tuned for different use cases and implementing different optimizations. However, the overall application performance is often held back by I/O bottlenecks due to insufficient performance of file systems or I/O libraries for highly parallel workloads. Performance problems are dealt with using novel storage hardware technologies as well as alternative I/O semantics and interfaces. These approaches have to be integrated into the storage stack seamlessly to make them convenient to use. Upcoming storage systems abandon the traditional POSIX interface and semantics in favor of alternative concepts such as object and key-value storage; moreover, they heavily rely on technologies such as NVM and burst buffers to improve performance. Additional tiers of storage hardware will increase the importance of hierarchical storage management. Many of these changes will be disruptive and require application developers to rethink their approaches to data management and I/O. A thorough understanding of today's storage infrastructures, including their strengths and weaknesses, is crucially important for designing and implementing scalable storage systems suitable for demands of exascale computing.

Hierarchical and hybrid energy storage devices in data centers: Architecture, control and provisioning.

Recently, a new approach has been introduced that leverages and over-provisions energy storage devices (ESDs) in data centers for performing power capping and facilitating capex/opex reductions, without performance overhead. To fully realize the potential benefits of the hierarchical ESD structure, we propose a comprehensive design, control, and provisioning framework including (i) designing power delivery architecture supporting hierarchical ESD structure and hybrid ESDs for some levels, as well as (ii) control and provisioning of the hierarchical ESD structure including run-time ESD charging/discharging control and design-time determination of ESD types, homogeneous/hybrid options, ESD provisioning at each level. Experiments have been conducted using real Google data center workloads based on realistic data center specifications.

Dynamic Outsourced Proofs of Retrievability Enabling Auditing Migration for Remote Storage Security

Remote data auditing service is important for mobile clients to guarantee the intactness of their outsourced data stored at cloud side. To relieve mobile client from the nonnegligible burden incurred by performing the frequent data auditing, more and more literatures propose that the execution of such data auditing should be migrated from mobile client to third‐party auditor (TPA). However, existing public auditing schemes always assume that TPA is reliable, which is the potential risk for outsourced data security. Although Outsourced Proofs of Retrievability (OPOR) have been proposed to further protect against the malicious TPA and collusion among any two entities, the original OPOR scheme applies only to the static data, which is the limitation that should be solved for enabling data dynamics. In this paper, we design a novel authenticated data structure called bv23Tree, which enables client to batch‐verify the indices and values of any number of appointed leaves all at once for efficiency. By utilizing bv23Tree and a hierarchical storage structure, we present the first solution for Dynamic OPOR (DOPOR), which extends the OPOR model to support dynamic updates of the outsourced data. Extensive security and performance analyses show the reliability and effectiveness of our proposed scheme.

A multipurpose computing center with distributed resources

The Computing Center of the Institute of Physics (CC IoP) of the Czech Academy of Sciences serves a broad spectrum of users with various computing needs. It runs WLCG Tier-2 center for the ALICE and the ATLAS experiments; the same group of services is used by astroparticle physics projects the Pierre Auger Observatory (PAO) and the Cherenkov Telescope Array (CTA). OSG stack is installed for the NOvA experiment. Other groups of users use directly local batch system. Storage capacity is distributed to several locations. DPM servers used by the ATLAS and the PAO are all in the same server room, but several xrootd servers for the ALICE experiment are operated in the Nuclear Physics Institute in Řež, about 10 km away. The storage capacity for the ATLAS and the PAO is extended by resources of the CESNET - the Czech National Grid Initiative representative. Those resources are in Plzen and Jihlava, more than 100 km away from the CC IoP. Both distant sites use a hierarchical storage solution based on disks and tapes. They installed one common dCache instance, which is published in the CC IoP BDII. ATLAS users can use these resources using the standard ATLAS tools in the same way as the local storage without noticing this geographical distribution. Computing clusters LUNA and EXMAG dedicated to users mostly from the Solid State Physics departments offer resources for parallel computing. They are part of the Czech NGI infrastructure MetaCentrum with distributed batch system based on torque with a custom scheduler. Clusters are installed remotely by the MetaCentrum team and a local contact helps only when needed. Users from IoP have exclusive access only to a part of these two clusters and take advantage of higher priorities on the rest (1500 cores in total), which can also be used by any user of the MetaCentrum. IoP researchers can also use distant resources located in several towns of the Czech Republic with a capacity of more than 12000 cores in total.

HNType : a diverse trace and migration mechanism in the block based Hierarchical Storage System named HazelNut

HazelNut is a block based Hierarchical Storage System, in which logical data blocks are migrated among storage tiers to achieve better I/O performance. In order to choose migrated blocks, data block I/O process is traced to collect enough information for migration algorithms. There are many ways to trace I/O process and implement block migration. However, how to choose trace metrics and design block migration algorithm is a big problem for system designers. To address this problem, a diverse trace and migration mechanism HNType is proposed. HNType consists two parts, one is a diverse trace mechanism named HNType-t, and the other is a diverse migration mechanism named HNType-s. HNType-t abstracts four base elements and trace operation interfaces based on VFS design concept, which makes it feasible to customize specific trace metrics and trace operations; HNType-s presents three ways of data migration, each way of migration can use customized migration algorithms according to predefined prototypes. Based on HNType, A series of tests are conducted about how block migration is affected by different trace metrics, and three conclusions are drawn according to the experiment results. First, trace metrics of access times and access sectors are not linearly correlated, as a result, these two metrics bring different migration result; Second, I/O completion time used as trace metrics is able to improve sequential I/O by at least 10%; Third, access times used as metrics have a tendency of more migrations upwards.

A Critical Path File Location (CPFL) algorithm for data-aware multiworkflow scheduling on HPC clusters

A representative set of workflows found in bioinformatics pipelines must deal with large data sets. Most scientific workflows are defined as Direct Acyclic Graphs (DAGs). Despite DAGs are useful to understand dependence relationships, they do not provide any information about input, output and temporal data files. This information about the location of files of data intensive applications helps to avoid performance issues.This paper presents a multiworkflow store-aware scheduler in a cluster environment called Critical Path File Location (CPFL) policy where the access time to disk is more relevant than network, as an extension of the classical list scheduling policies. Our purpose is to find the best location of data files in a hierarchical storage system.The resulting algorithm is tested in an HPC cluster and in a simulated cluster scenario with bioinformatics synthetic workflows, and largely used benchmarks like Montage and Epigenomics. The resulting simulator is tuned and validated with the first test results from the real infrastructure. The evaluation of our proposal shows promising results up to 70% on benchmarks in real HPC clusters using 128 cores and up to 69% of makespan improvement on simulated 512 cores clusters with a deviation between 0.9% and 3% regarding the real HPC cluster.

Research of Hierarchical Control Strategy for Photovoltaic & Battery Storage System

According to the complementarity of photovoltaic and battery generation, the structure of photovoltaic and hierarchical storage system and its typical control methods are described in this paper, considering the demand of load side and distribution gird, hierarchical control strategy including primary control and secondary control is discussed, the model of converters and the control loop are presented in detail. The overall structure of hierarchical control composed by P-Q control and voltage-phase droop control is brief and cost-less. A simulation of the basic part in hierarchical control strategy under PSCAD/EMTDC environment is also included in this paper. The evaluation of hierarchical control strategy shows that photovoltaic and hierarchical storage system is practical for the application of distributed generators.

An IFC schema extension and binary serialization format to efficiently integrate point cloud data into building models

In this paper we suggest an extension to the Industry Foundation Classes (IFC) model to integrate point cloud datasets. The proposal includes a schema extension to the core model allowing the storage of points, either as Cartesian coordinates, points in parametric space of associated building element surfaces or as discrete height fields projected as grids onto building elements. To handle the considerable amounts of data generated in the process of scanning building structures, we present intelligent compression approaches combined with the Hierarchical Data Format (HDF) as an efficient serialization and an alternative to clear text encoded ISO 10303 part 21 files. Based on prototypical implementations we show results of various serialization options and their impacts on storage efficiency.In this proposal the deepened semantic relationships have been favoured over compression ratios. Nevertheless, with various near-lossless layers of compression and binary serialization applied, a compression ratio of up to 67.7% is obtained for a building model with integrated point clouds, compared to the raw source data. The binary serialization is able to handle hundreds of millions of points, out of which specific spatial and semantic subsets can rapidly be extracted due to the containerized hierarchical storage model and association to building elements. The authors advocate the use of binary storage for sizeable point cloud scans, but also show how especially the grid discretization can result into usable points cloud segments embedded into text-based IFC models.