Mass Storage System Research Articles

Distributed Metadata Management of Mass Storage System in High Energy Physics

In this contribution, we design and implement a dynamic and scalable distributed metadata management system(StarFS) to High Energy Physics (HEP) mass storage system. Particularly, we discuss the key technologies of the distributed metadata management system. We propose a new algorithm named Adaptive Directory Sub-tree Partition(ADSP) for metadata distribution. ADSP divides the filesystem namespace into sub-trees with directory granularity. Sub-trees will be stored on storage devices in flat structure, whose location and file attributes are recorded as extended attributes. The placement of sub-tree is adjusted adaptively according to the workload of metadata cluster so that the load balance could be improved and metadata cluster could be extended dynamically. Experiments show that ADSP achieves higher metadata performance and scalability compared to Gluster and Lustre. We also propose a new algorithm called Distributed Unified Layout(DULA) to improve dynamic scalability and efficiency of data positioning. A system with DULA could provide uniform data distribution and efficient data positioning. DULA is an improved consistent hashing algorithm which is able to locate data in O(1) without the help of routing information. Experiments prove that the better uniform data distribution and efficient data access can be achieved by DULA. This work is validated in YangBaJing Cosmic Ray(YBJ) experiment.

Tape SCSI monitoring and encryption at CERN

CERN currently manages the largest data archive in the HEP domain; over 180PB of custodial data is archived across 7 enterprise tape libraries containing more than 25,000 tapes and using over 100 tape drives. Archival storage at this scale requires a leading edge monitoring infrastructure that acquires live and lifelong metrics from the hardware in order to assess and proactively identify potential drive and media level issues. In addition, protecting the privacy of sensitive archival data is becoming increasingly important and with it the need for a scalable, compute-efficient and cost-effective solution for data encryption. In this paper, we first describe the implementation of acquiring tape medium and drive related metrics reported by the SCSI interface and its integration with our monitoring system. We then address the incorporation of tape drive real-time encryption with dedicated drive hardware into the CASTOR [1] hierarchical mass storage system.

Simulating European wind power generation applying statistical downscaling to reanalysis data

The growing share of electricity production from solar and mainly wind resources constantly increases the stochastic nature of the power system. Modelling the high share of renewable energy sources – and in particular wind power – crucially depends on the adequate representation of the intermittency and characteristics of the wind resource which is related to the accuracy of the approach in converting wind speed data into power values. One of the main factors contributing to the uncertainty in these conversion methods is the selection of the spatial resolution. Although numerical weather prediction models can simulate wind speeds at higher spatial resolution (up to 1×1km) than a reanalysis (generally, ranging from about 25km to 70km), they require high computational resources and massive storage systems: therefore, the most common alternative is to use the reanalysis data. However, local wind features could not be captured by the use of a reanalysis technique and could be translated into misinterpretations of the wind power peaks, ramping capacities, the behaviour of power prices, as well as bidding strategies for the electricity market. This study contributes to the understanding what is captured by different wind speeds spatial resolution datasets, the importance of using high resolution data for the conversion into power and the implications in power system analyses. It is proposed a methodology to increase the spatial resolution from a reanalysis. This study presents an open access renewable generation time series dataset for the EU-28 and neighbouring countries at hourly intervals and at different geographical aggregation levels (country, bidding zone and administrative territorial unit), for a 30year period taking into account the wind generating fleet at the end of 2015.

HWM: a hybrid workload migration mechanism of metadata server cluster in data center

In data center, applications of big data analytics pose a big challenge to massive storage systems. It is significant to achieve high availability, high performance and high scalability for PB-scale or EB-scale storage systems. Metadata server (MDS) cluster architecture is one of the most effective solutions to meet the requirements of applications in data center. Workload migration can achieve load balance and energy saving of cluster systems. In this paper, a hybrid workload migration mechanism of MDS cluster is proposed and named as HWM. In HWM, workload of MDS is classified into two categories: metadata service and state service, and they can be migrated rapidly from a source MDS to a target MDS in different ways. Firstly, in metadata service migration, all the dirty metadata of one sub file system is flushed to a shared storage pool by the source MDS, and then is loaded by the target MDS. Secondly, in state service migration, all the states of that sub file system are migrated from source MDS to target MDS through network at file granularity, and then all of the related structures of these states are reconstructed in targetMDS. Thirdly, in the process of workload migration, instead of blocking client requests, the source MDS can decide which MDS will respond to each request according to the operation type and the migration stage. The proposed mechanismis implemented in the BlueWhaleMDS cluster. The performance measurements show that the HWM mechanism is efficient to migrate the workload of a MDS cluster system and provides low-latency access to metadata and states.

Modeling, design and analysis of a micro flying slider in near-field optical storage system

A micro flying slider is developed to satisfy the demands for large capacity, high data density, and fast transfer rate of mass storage systems. The air-bearing surface of the micro flying slider is designed based on micro fluid theory of hydrodynamics to obtain positive and negative pressures, thereby enhancing flying height stability. The lubrication model is investigated based on the generalized Reynolds equation. The finite volume method is utilized to solve the revised Reynolds equation that governs the air-bearing pressure and flying system behavior. The dynamic model for the slider–disk interface of the micro flying slider is developed, and the air-bearing system is modeled as a 3-degrees-of-freedom, lumped-parameter model to simulate flying characteristics. The air-bearing pressures with flying attitudes in the loading/unloading processes are characterized and calculated using the proposed model, with flying height and pitch angle as the key variables. The sample data obtained via accurate triggering and superposed averaging are processed to acquire modal frequencies and damping ratios based on the parameter identification method of modal analysis technique. System experiments are presented to illustrate the effectiveness of the proposed micro flying slider and its dynamic model for flying-height control. Theoretical analysis and experimental results show that a stable storage working process can be obtained at 1600 rpm and a pitch angle of 1.5°. Moreover, the micro flying slider exhibits satisfactory flying stability, with a near-field working distance of less than 75 nm. Good correlation between the experimental and simulation results is achieved.

A Mobile Terminal Security Strategy Based On the Cloud Storage

Abstract With the emergence of mass storage systems and the development of the Distributed File System, Cloud storage system has become the focus of the industry. The cloud storage services on mobile terminal have been putted on the agenda based on the rapid development of intelligent mobile terminal. Based on the analysis of the architecture of HDFS and Dynamo, a mobile Terminal Security strategy is presented in this paper. The database technology and t he dynamic consistent hashing algorithm are adopted to deal with different target groups. According to the storage costs of nodes, data would be integrated scheduling by the storage system. Make full use of the advantages of AES(Advanced Encryption Standard) and RSA. A solution that combines AES and RSA encryption algorithm is proposed to implement the mobile terminal cloud storage security. Through the theoretical analysis and the simulation results, the cloud storage strategy proposed in this paper can make the cloud system achieve load balance. Moreover, multi-copy mechanism can improve the overall efficiency of the system.

Data Mining as a Service (DMaaS)

Data Mining as a Service (DMaaS) is a software and computing infrastructure that allows interactive mining of scientific data in the cloud. It allows users to run advanced data analyses by leveraging the widely adopted Jupyter notebook interface. Furthermore, the system makes it easier to share results and scientific code, access scientific software, produce tutorials and demonstrations as well as preserve the analyses of scientists.This paper describes how a first pilot of the DMaaS service is being deployed at CERN, starting from the notebook interface that has been fully integrated with the ROOT analysis framework, in order to provide all the tools for scientists to run their analyses. Additionally, we characterise the service backend, which combines a set of IT services such as user authentication, virtual computing infrastructure, mass storage, file synchronisation, development portals or batch systems. The added value acquired by the combination of the aforementioned categories of services is discussed, focusing on the opportunities offered by the CERNBox synchronisation service and its massive storage backend, EOS.

Automation of Large-scale Computer Cluster Monitoring Information Analysis

High-throughput computing platforms consist of a complex infrastructure and provide a number of services apt to failures. To mitigate the impact of failures on the quality of the provided services, a constant monitoring and in time reaction is required, which is impossible without automation of the system administration processes. This paper introduces a way of automation of the process of monitoring information analysis to provide the long and short term predictions of the service response time (SRT) for a mass storage and batch systems and to identify the status of a service at a given time. The approach for the SRT predictions is based on Adaptive Neuro Fuzzy Inference System (ANFIS). An evaluation of the approaches is performed on real monitoring data from the WLCG Tier 2 center GoeGrid. Ten fold cross validation results demonstrate high efficiency of both approaches in comparison to known methods.

Hybrid Minimum Pearson and Euclidean Distance Detection

The reliability of mass storage systems, such as optical data recording and non-volatile memory (Flash), is seriously hampered by uncertainty of the actual value of the offset (drift) or gain (amplitude) of the retrieved signal. The recently introduced minimum Pearson distance detection is immune to unknown offset or gain, but this virtue comes at the cost of a lessened noise margin at nominal channel conditions. We will present a novel hybrid detection method, where we combine the outputs of the minimum Euclidean distance and Pearson distance detectors so that we may trade detection robustness versus noise margin. We will compute the error performance of hybrid detection in the presence of unknown channel mismatch and additive noise.

The INFN-CNAF Tier-1 GEMSS Mass Storage System and database facility activity

The consolidation of Mass Storage services at the INFN-CNAF Tier1 Storage department that has occurred during the last 5 years, resulted in a reliable, high performance and moderately easy-to-manage facility that provides data access, archive, backup and database services to several different use cases. At present, the GEMSS Mass Storage System, developed and installed at CNAF and based upon an integration between the IBM GPFS parallel filesystem and the Tivoli Storage Manager (TSM) tape management software, is one of the largest hierarchical storage sites in Europe. It provides storage resources for about 12% of LHC data, as well as for data of other non-LHC experiments. Files are accessed using standard SRM Grid services provided by the Storage Resource Manager (StoRM), also developed at CNAF. Data access is also provided by XRootD and HTTP/WebDaV endpoints. Besides these services, an Oracle database facility is in production characterized by an effective level of parallelism, redundancy and availability. This facility is running databases for storing and accessing relational data objects and for providing database services to the currently active use cases. It takes advantage of several Oracle technologies, like Real Application Cluster (RAC), Automatic Storage Manager (ASM) and Enterprise Manager centralized management tools, together with other technologies for performance optimization, ease of management and downtime reduction. The aim of the present paper is to illustrate the state-of-the-art of the INFN-CNAF Tier1 Storage department infrastructures and software services, and to give a brief outlook to forthcoming projects. A description of the administrative, monitoring and problem-tracking tools that play a primary role in managing the whole storage framework is also given.

Performance and Reliability Analysis of Cross-Layer Optimizations of NAND Flash Controllers

NAND flash memories are becoming the predominant technology in the implementation of mass storage systems for both embedded and high-performance applications. However, when considering data and code storage in Non-Volatile Memories (NVMs), such as NAND flash memories, reliability and performance become a serious concern for systems designers. Designing NAND flash-based systems based on worst-case scenarios leads to waste of resources in terms of performance, power consumption, and storage capacity. This is clearly in contrast with the request for runtime reconfigurability, adaptivity, and resource optimization in modern computing systems. There is a clear trend toward supporting differentiated access modes in flash memory controllers, each one setting a differentiated tradeoff point in the performance-reliability optimization space. This is supported by the possibility of tuning the NAND flash memory performance, reliability, and power consumption through several tuning knobs such as the flash programming algorithm and the flash error correcting code. However, to successfully exploit these degrees of freedom, it is mandatory to clearly understand the effect that the combined tuning of these parameters has on the full NVM subsystem. This article performs a comprehensive quantitative analysis of the benefits provided by the runtime reconfigurability of an MLC NAND flash controller through the combined effect of an adaptable memory programming circuitry coupled with runtime adaptation of the ECC correction capability. The full NVM subsystem is taken into account, starting from a characterization of the low-level circuitry to the effect of the adaptation on a wide set of realistic benchmarks in order to provide readers a clear view of the benefit this combined adaptation may provide at the system level.

A Fair-Rank Ant Colony Algorithm in Distributed Mass Storage System

In general, a distributed mass storage system is composed of a large number of computing nodes and storage nodes, allowing users to access resources transparently, without knowing where the resources are physically located. For every storage request, the storage system scheduler chooses several storage nodes from the entire storage system in order to organize a substorage system. This kind of scheduling is an NP-hard problem; to achieve the storage system’s promising potential, this paper proposes a virtual subsystem quality of service scheduling model and a fair-rank ant colony algorithm. The algorithm provides special incentives to the scheduling results that include new nodes, so the new nodes will have more opportunities to participate in the scheduling system. Tests show that this algorithm performs better in the fairness and load balance than the ant colony algorithm.

A secure and scalable storage system for aggregate data in IoT

In recent years, with the impressive rapid development of integrated circuit and networking technologies, computers, devices and networking have become highly pervasive, incurring the introduction, development and deployment of the Internet of Things (IoT). The tiny identifying devices and wearables in IoT have transformed daily life in human society, as they generate, process and store the amount of data increasing at exponential rate all over the world. Due to high demand on data mining and analytics activities in IoT, secure and scalable mass storage systems are highly demanded for aggregate data in efficient processing. In this paper, we propose such a secure and scalable IoT storage system based on revised secret sharing scheme with support of scalability, flexibility and reliability at both data and system levels. Shamir’s secret sharing scheme is applied to achieve data security without complex key management associated with traditional cryptographic algorithms. The original secret sharing scheme is revised to utilize all coefficients in polynomials for larger data capacity at data level. Flexible data insert and delete operations are supported. Moreover, a distributed IoT storage infrastructure is deployed to provide scalability and reliability at system level. Multiple IoT storage servers are aggregated for large storage capacity whereas individual servers can join and leave freely for flexibility at system level. Experimental results have demonstrated the feasibility and benefits of the proposed system as well as tangible performance gains.

Design and Analysis of a Novel Micro Flying Head in Mass Storage System

In order to meet the mass storage system’s need for higher density and integration, a novel structural design for micro flying head is conceived and an air bearing surface is designed based on micro fluid theory of hydrodynamics. Both positive and negative pressures can be generated by the air bearing surface, with the stiffness and motion stability enhanced by negative pressure. A lubrication model has been investigated with 1.5 order slip model of the generalized Reynolds equation, and the finite volume method is used for the simulation of the flying characteristics. The theoretical analysis and experimental results show that the proposed micro flying head exhibits satisfying flying stability, with head-disk spacing less than 75nm, and carrying capacity exceeding 93mN.

Xrootd data access for LHC experiments at the INFN-CNAF Tier-1

The Mass Storage System installed at the INFN-CNAF Tier-1 is one of the biggest hierarchical storage facilities in Europe. It currently provides storage resources for about 12% of all LHC data, as well as for other experiments. The Grid Enabled Mass Storage System (GEMSS) is the current solution implemented at CNAF and it is based on a custom integration between a high performance parallel file system (General Parallel File System, GPFS) and a tape management system for long-term storage on magnetic media (Tivoli Storage Manager, TSM). Data access to Grid users is being granted since several years by the Storage Resource Manager (StoRM), an implementation of the standard SRM interface, widely adopted within the WLCG community.The evolving requirements from the LHC experiments and other users are leading to the adoption of more flexible methods for accessing the storage. These include the implementation of the so-called storage federations, i.e. geographically distributed federations allowing direct file access to the federated storage between sites.A specific integration between GEMSS and Xrootd has been developed at CNAF to match the requirements of the CMS experiment. This was already implemented for the ALICE use case, using ad-hoc Xrootd modifications. The new developments for CMS have been validated and are already available in the official Xrootd builds. This integration is currently in production and appropriate large scale tests have been made.In this paper we present the Xrootd solutions adopted for ALICE, CMS, ATLAS and LHCb to increase the availability and optimize the overall performance.

A FPGA-based Cluster Finder for CMOS Monolithic Active Pixel Sensors of the MIMOSA-26 Family

CMOS Monolithic Active Pixel Sensors (MAPS) demonstrated excellent performances in the field of charged particle tracking. Among their strong points are an single point resolution few μm, a light material budget of 0.05% X0 in combination with a good radiation tolerance and high rate capability. Those features make the sensors a valuable technology for vertex detectors of various experiments in heavy ion and particle physics. To reduce the load on the event builders and future mass storage systems, we have developed algorithms suited for preprocessing and reducing the data streams generated by the MAPS. This real-time processing employs remaining free resources of the FPGAs of the readout controllers of the detector and complements the on-chip data reduction circuits of the MAPS.

The Metadata Dynamic Load-Balancing Strategy of Distributed Filesystem Based on Hash Tags

With the development of information technology, distributed file system is widely used in massive information storage. Usually, distributed file system uses metadata server to achieve quick access to files according to directory, thus the organization and management of metadata are the keys to the file system performance. In general, directory subtree partition method and hash algorithm are used by existing mass storage system to manage metadata. However, to solve the problems, like low access efficiency of metadata, ineffective balance of load and poor extensibility, in existing metadata management strategy of distributed file system, dynamic load balance strategy of metadata based on hash tags is put forward, in which the tags act as partition granularity and hot tags of metadata will be hashed again to achieve the goal of load balance. The experimental results in this paper turn out that modified metadata management strategy based on hash tags has greater system throughput and less average response time than the one based on tags.,

Equipment Servo System Signal Acquisition Module Design Based on Embedded

Contrary to the difficult for complex equipment in the parallel signal acquisition, thereby affecting the equipment condition monitoring results, Design a parallel signal data acquisition device and its collection method based on embedded system. It can achieve real-time multi-point data collection and analyze parallel signal data acquisition system structure and function, then giving the hardware design of embedded systems based on the chip, including the processes of mass storage system and a plurality of communication method. Realize of the equipment condition monitoring and fault diagnosis based on this.

A Metadata Management Method Base on Directory Path Code

Metadata distribution is important in mass storage system. Sub-tree partition and hash are two traditional metadata distribution algorithms used in file system. But they both have a defect in system scalability. This paper presents a new metadata management method, Directory Path Code Hash (DPCH). This method is to store directory and file metadata separately, and effectively solving the unbalanced metadata distribution and access hot point problems in Sub-tree partition and the excessive reading times and large metadata migration amount after directory property modification in hash algorithm. The experiment indicates that this method proposed significantly outweighs other algorithms in terms of throughput rate, metadata distribution, reading times, etc.

The Study of Testing Technologies on Energy Consumption of Mass Storage System

By analyzing the function structure of mass storage system comprehensively, we decompose the whole system into some single equipment by its own function. Then analyze the device working principle to analysis and description the method of equipment energy consumption. And according to the result, test the cluster system in the lab environment.