Hierarchical Storage Management System Research Articles

PGP repository: a plant phenomics and genomics data publication infrastructure.

Plant genomics and phenomics represents the most promising tools for accelerating yield gains and overcoming emerging crop productivity bottlenecks. However, accessing this wealth of plant diversity requires the characterization of this material using state-of-the-art genomic, phenomic and molecular technologies and the release of subsequent research data via a long-term stable, open-access portal. Although several international consortia and public resource centres offer services for plant research data management, valuable digital assets remains unpublished and thus inaccessible to the scientific community. Recently, the Leibniz Institute of Plant Genetics and Crop Plant Research and the German Plant Phenotyping Network have jointly initiated the Plant Genomics and Phenomics Research Data Repository (PGP) as infrastructure to comprehensively publish plant research data. This covers in particular cross-domain datasets that are not being published in central repositories because of its volume or unsupported data scope, like image collections from plant phenotyping and microscopy, unfinished genomes, genotyping data, visualizations of morphological plant models, data from mass spectrometry as well as software and documents.The repository is hosted at Leibniz Institute of Plant Genetics and Crop Plant Research using e!DAL as software infrastructure and a Hierarchical Storage Management System as data archival backend. A novel developed data submission tool was made available for the consortium that features a high level of automation to lower the barriers of data publication. After an internal review process, data are published as citable digital object identifiers and a core set of technical metadata is registered at DataCite. The used e!DAL-embedded Web frontend generates for each dataset a landing page and supports an interactive exploration. PGP is registered as research data repository at BioSharing.org, re3data.org and OpenAIRE as valid EU Horizon 2020 open data archive. Above features, the programmatic interface and the support of standard metadata formats, enable PGP to fulfil the FAIR data principles—findable, accessible, interoperable, reusable.Database URL: http://edal.ipk-gatersleben.de/repos/pgp/

The new CERN tape software - getting ready for total performance

CASTOR (the CERN Advanced STORage system) is used to store the custodial copy of all of the physics data collected from the CERN experiments, both past and present. CASTOR is a hierarchical storage management system that has a disk-based front-end and a tape-based back-end. The software responsible for controlling the tape back-end has been redesigned and redeveloped over the last year and was put in production at the beginning of 2015. This paper summarises the motives behind the redesign, describes in detail the redevelopment work and concludes with the short and long-term benefits.

Data oriented job submission scheme for the PHENIX user analysis in CCJ

The RIKEN Computing Center in Japan (CCJ) has been developed to make it possible analyzing huge amount of data corrected by the PHENIX experiment at RHIC. The corrected raw data or reconstructed data are transferred via SINET3 with 10 Gbps bandwidth from Brookheaven National Laboratory (BNL) by using GridFTP. The transferred data are once stored in the hierarchical storage management system (HPSS) prior to the user analysis. Since the size of data grows steadily year by year, concentrations of the access request to data servers become one of the serious bottlenecks. To eliminate this I/O bound problem, 18 calculating nodes with total 180 TB local disks were introduced to store the data a priori. We added some setup in a batch job scheduler (LSF) so that user can specify the requiring data already distributed to the local disks. The locations of data are automatically obtained from a database, and jobs are dispatched to the appropriate node which has the required data. To avoid the multiple access to a local disk from several jobs in a node, techniques of lock file and access control list are employed. As a result, each job can handle a local disk exclusively. Indeed, the total throughput was improved drastically as compared to the preexisting nodes in CCJ, and users can analyze about 150 TB data within 9 hours. We report this successful job submission scheme and the feature of the PC cluster.

A case study for efficient management of high throughput primary lab data

BackgroundIn modern life science research it is very important to have an efficient management of high throughput primary lab data. To realise such an efficient management, four main aspects have to be handled: (I) long term storage, (II) security, (III) upload and (IV) retrieval.FindingsIn this paper we define central requirements for a primary lab data management and discuss aspects of best practices to realise these requirements. As a proof of concept, we introduce a pipeline that has been implemented in order to manage primary lab data at the Leibniz Institute of Plant Genetics and Crop Plant Research (IPK). It comprises: (I) a data storage implementation including a Hierarchical Storage Management system, a relational Oracle Database Management System and a BFiler package to store primary lab data and their meta information, (II) the Virtual Private Database (VPD) implementation for the realisation of data security and the LIMS Light application to (III) upload and (IV) retrieve stored primary lab data.ConclusionsWith the LIMS Light system we have developed a primary data management system which provides an efficient storage system with a Hierarchical Storage Management System and an Oracle relational database. With our VPD Access Control Method we can guarantee the security of the stored primary data. Furthermore the system provides high performance upload and download and efficient retrieval of data.

Experience with Hierarchical Storage Management based on GPFS and TSM at INFN-CNAF

At the INFN-CNAF Tier-1 computing centre we have evaluated the integration between two IBM products, namely the General Parallel File System (GPFS) and the Tivoli Storage Manager (TSM), with the aim of implementing a high performance Hierarchical Storage Management (HSM) system for managing the data of the experiments operating at the Large Hadron Collider. Starting from version 3.2, GPFS allows for the execution of selective migrations of GPFS disk-resident files to external tape pools by means of appropriate policies defined via the Information Lifecycle Management (ILM) interface and specific user programs. As far as recalls of files from tape to disk are concerned, an optimized tape-ordered access is mandatory, and this has been achieved by exploiting new features of the TSM HSM client, preliminarily made available in version 6.1. In this paper we summarize the work made for implementing the full GPFS-TSM integration, as well as some early but significant results obtained during intensive data access activities.

New developments of the EAST data system

As the discharge time becomes longer, the old EAST data system will not meet the requirements of EAST discharge experiment gradually. The problems mainly focus on continuous data acquisition and real-time data access. To meet the requirements of EAST 1000s discharges, the data acquisition mode, data storage structure and data access interface must be improved or reconstructed. Time slice mechanism is one of the popular solutions for continuous data acquisition and real-time data transfer. Data access of large signal files has been successfully solved via a hierarchical storage management system. Moreover, conventional “remote indexing” is replaced with “local indexing” in data access, which greatly enhanced data access speed. These improvements have been implemented and applied to EAST long-pulse plasma experiments which are described in this paper in detail.

A high performance hierarchical storage management system for the Canadian tier-1 centre at TRIUMF

We describe in this paper the design and implementation of Tapeguy, a high performance non-proprietary Hierarchical Storage Management (HSM) system which is interfaced to dCache for efficient tertiary storage operations. The system has been successfully implemented at the Canadian Tier-1 Centre at TRIUMF. The ATLAS experiment will collect a large amount of data (approximately 3.5 Petabytes each year). An efficient HSM system will play a crucial role in the success of the ATLAS Computing Model which is driven by intensive large-scale data analysis activities that will be performed on the Worldwide LHC Computing Grid infrastructure continuously. Tapeguy is Perl-based. It controls and manages data and tape libraries. Its architecture is scalable and includes Dataset Writing control, a Read-back Queuing mechanism and I/O tape drive load balancing as well as on-demand allocation of resources. A central MySQL database records metadata information for every file and transaction (for audit and performance evaluation), as well as an inventory of library elements. Tapeguy Dataset Writing was implemented to group files which are close in time and of similar type. Optional dataset path control dynamically allocates tape families and assign tapes to it. Tape flushing is based on various strategies: time, threshold or external callbacks mechanisms. Tapeguy Read-back Queuing reorders all read requests by using an elevator algorithm, avoiding unnecessary tape loading and unloading. Implementation of priorities will guarantee file delivery to all clients in a timely manner.

Storage Management and Backup Schemes for Broadcast Video

As videotape gradually disappears from the broadcast plant, storage in a digital file format becomes more important. In most cases, this is a combination of spinning disk and removable media, either data tape or DVD. With that comes the management of material as it moves from video server to nearline to archive, and back again. The class of software products that perform this function are generally called hierarchical storage management or HSM systems. This is actually a misnomer, in that, HSM refers to a specific type of storage management rather than a product category. HSM is often used as a catchall term for two categories of file management: storage management and file backup. Storage management really consists of four broad subcategories of products, each having advantages and disadvantages, as it applies to on-air, production, and news environments. Specifically, the subcategories are shared file systems, HSM systems, disk extenders, and data movers. Related to, but completely different from storage management is file backup, which also often gets included with HSM. In the category of file backup are subcategories of mirroring, simple backup, and disaster recovery.

DIII-D data management

The DIII-D tokamak at the DIII-D National Fusion Facility routinely acquires ∼500MB of raw data per pulse of the experiment through a centralized data management system. It is expected that in FY01 (October 2001 to September 2002), nearly 1TB of data will be acquired. In addition there are several diagnostics, which are not part of the centralized system, which acquire hundreds of MB of raw data per pulse. There is also a growing suite of codes running between pulses that produce analyzed data, which add ∼10MB per pulse with total disk usage of about 100GB. A relational database system has been introduced which further adds to the overall data load. In recent years there has been an order of magnitude increase in magnetic disk space devoted to raw data and a hierarchical storage management system was implemented to allow 24 h/day, 7 days/week unattended access to raw data. The management of all of the data is a significant and growing challenge as the quantities of both raw and analyzed data are expected to continue to increase in the future. This paper will examine the experiences of the approaches that have been taken in management of the data and plans for the continued growth of the data quantity.

StorHouse metanoia - new applications for database, storage & data warehousing

This paper describes the StorHouse/Relational Manager (RM) database system that uses and exploits an active storage hierarchy . By active storage hierarchy, we mean that StorHouse/RM executes SQL queries directly against data stored on all hierarchical storage (i.e. disk, optical, and tape) without post processing a file or a DBA having to manage a data set. We describe and analyze StorHouse/RM features and internals. We also describe how StorHouse/RM differs from traditional HSM (Hierarchical Storage Management) systems. For commercial applications we describe an evolution to the Data Warehouse concept, called Atomic Data Store , whereby atomic data is stored in the database system. Atomic data is defined as storing all the historic data values and executing queries against them. We also describe a Hub-and-Spoke Data Warehouse architecture , which is used to feed or fuel data into Data Marts. Furthermore, we provide analysis how StorHouse/RM can be federated with DB2, Oracle and Microsoft SQL Server 7 (SS7) and thus provide these databases an active storage hierarchy (i.e. tape). We then show two federated data modeling techniques (a) logical horizontal partitioning (LHP) of tuples and (b) logical vertical partitioning (LVP) of columns to demonstrate our database extension capabilities. We conclude with a TPC-like performance analysis of data stored on tape and disk.

MR-AFS: a global hierarchical file-system

The next generation of fusion experiments will use object-oriented technology creating the need for world wide sharing of an underlying hierarchical file-system. The Andrew file system (AFS) is a well known and widely spread global distributed file-system. Multiple-resident-AFS (MR-AFS) combines the features of AFS with hierarchical storage management systems. Files in MR-AFS therefore may be migrated on secondary storage, such as roboted tape libraries. MR-AFS is in use at IPP for the current experiments and data originating from super-computer applications. Experiences and scalability issues are discussed.

The DIII–D computing environment: characteristics and recent changes

The DIII–D tokamak national fusion research facility along with its predecessor Doublet III has been operating for over 21 years. The DIII–D computing environment consists of: real-time systems controlling the tokamak, heating systems, and diagnostics, and systems acquiring experimental data from instrumentation; major data analysis server nodes performing short term and long term data access and data analysis; and systems providing mechanisms for remote collaboration and the dissemination of information over the world wide web. Computer systems for the facility have undergone incredible changes over the course of time as the computer industry has changed dramatically. Yet there are certain valuable characteristics of the DIII–D computing environment that have been developed over time and have been maintained to this day. Some of these characteristics include: continuous computer infrastructure improvements, distributed data and data access, computing platform integration, and remote collaborations. These characteristics are being carried forward as well as new characteristics resulting from recent changes which have included: a dedicated storage system and a hierarchical storage management system for raw shot data, various further infrastructure improvements including deployment of Fast Ethernet, the introduction of MDSplus, Load Sharing Facility (LSF) and common IDL based graphics tools, and improvements to remote collaboration capabilities. This paper will describe this computing environment, important characteristics that over the years have contributed to the success of DIII–D computing systems, and recent changes to computer systems.

Simulation of disaster recovery of a picture archiving and communications system using off-site hierarchal storage management.

The purpose of this communication is to report on the testing of the disaster recovery capability of our hierarchical storage management (HSM) system. Disaster recovery implementation is a requirement of every mission-critical information technology project. Picture archiving and communications systems (PACS) certainly falls into this category, even though the counterpart, conventional film archive, has no protection against fire, for example. We have implemented a method for hierarchical storage with wavelet technology that maximizes on-site case storage (using lossy compression), retains bit-preserved image data for legal purposes, provides an off-site backup (lossless bit-preserving wavelet transform), and provides for disaster recovery. Recovery from a natural (earthquake and subsequent fire) or technical (system crash and data loss) disaster was simulated by attempting to restore from the off-site image and database backup to clean core PACS components. The only existing loaded software was the operating system. The database application was reloaded locally, and then the database contents and image store were loaded from the off-site component of the HSM system. The following measurements were analyzed: (1) the ability to recover all data; (2) the integrity of the recovered database and image data; (3) the time to recover the database relative to the number of studies and age of the archive, as well as bandwidth between the local and remote site; and (4) the time to recover image data relative to compression ratio, number of studies, number of images, and time depth of the archive. This HSM system, which maximizes on-site storage, maintains a legal record, and provides off-site backup, also facilitates disaster recovery for a PACS.

Migration from hierarchal storage management to ASM storage server: a case study.

The Department of Radiology at the University of Utah Hospitals and Clinics had to make a change from its current hierarchical storage management (HSM) system. The HSM software is the heart of any near-line data storage system and any change in this software affects all near-line and on-line data storage. In this case, over a terabyte of data had been migrated in more than 2 million files. The traditional method of reading in the old data and then writing it out to the new system was calculated to take more than 60 years. Here, we will examine the reasons for making such a radical change in the HSM used. We will also discuss why ASM (the new HSM software) was selected, and the performance improvements seen. A second, less difficult transition was made a few months later, of upgrading to a newer faster tape technology. The two types of tapes were incompatible, but the storage software and robotics used allowed for a peaceful coexistence during the transition. The transition from HSM to ASM was not a trivial task. It required a reasonable implementation/migration plan, which involved finding the correct resources and thinking outside the norm for solutions. All sites that have any amount of data stored in near-line devices will face similar conversions. This presentation should help in the event that a data conversion plan is not already in place.

Television home server for integrated services-toward the realization of ISDB "anytime" services

In the age of digital multimedia, demand is bound to rise for a television home server for integrated services which automatically records favorite programs so that users can watch them at any time. We have developed a hierarchical storage management system (HSMS) as the architecture for developing a high-speed home server with very large capacity. This paper first describes the required specifications and memory construction of the home server and then proposes a model of the new HSMS. Also described are its element technologies and an experimental home server system, and the results of experiments conducted to test its performance.

System Design for Television Home Servers

In the age of digital multimedia, demand is bound to rise for television Home Servers that automatically record a viewer's favorite programs so that he/she can watch them at any time. We have developed an experimental Home Server on which a recorded HDTV program can be played back while recording another digital HDTV program. Using a new HSMS (hierarchical storage management system) we proposed, it was able to store HDTV programs for 4 hours and was as small as thickness of flat displays. This paper describes the outline of the experimental system and aims to clarify the recording requirements for hard disks and tapes by examining the read/write performance of hard disks on multi-channels and the nonlinear accessibility to programs recorded on tapes.

Intelligent image management in a distributed PACS and telemedicine environment

Advances in information technology in the past decade have resulted in a proliferation of clinical information systems dedicated to individual user groups and clinical functional areas. This, in turn, has led to the need for hospital-wide management and integration of information, and has triggered major efforts towards the development of integrated hospital information systems. A framework for developing an integrated regional health telematics system is presented, based on the functional and data integration of federated autonomous information systems. The concept of the patient meta-record is introduced as a central element in achieving integration in terms of content, structure, and access to information resources. Intelligent image management at the level of either a single health care provider (hospital) or a regional health telematics network is addressed and is achieved through event and model-driven strategies enabled by a distributed hierarchical storage management system. Intelligence, in this context, refers to methods of efficient and effective real-time resource management in the timely delivery of health care.

Hierarchical storage management in a distributed VOD system

The article describes the storage management algorithms in the Berkeley Video-on-Demand System, a hierarchical storage management system designed to support transparent access to thousands of hours of video material. Storage management algorithms handle continuous media objects stored on video servers within a hierarchical distributed video-on-demand system. Media objects stored permanently on tertiary storage devices are moved to video servers when needed. Algorithms manage the distributed cache in the video servers and select a video server on which to place a requested object. The object-placement algorithm uses server load, network load, and service-wait times to optimize system performance.

The implications of distributed data in a commercial environment on the design of hierarchical storage management

This paper explores the implication of distributed data on the design of data storage management systems. We developed file system measurement programs that collected static file system data from commercial UNIX systems. This sampling of 46 systems, 267 file systems, 151 000 directories and 2 300 000 files is greater than any previously reported study. We developed an analysis technique termed collective cumulative distribution function and applied it to collections of file systems. The results provide insight into file system characteristics and the design of hierarchical storage management systems. Important factors influencing performance are: (1) small files dominate in count but large files dominate secondary storage consumption, (2) file I/O is biased toward reads rather than writes, and (3) directory based storage management is problematic.

Performance analysis of client-server storage systems

The performance analysis of a system in which storage is provided by a file server node on a network of client processors is discussed. The server controls a hierarchy of storage devices, and provides all the necessary file management functions. The clients are large mainframe systems that have their own local storage devices. The movement of files among the various storage levels is managed by a hierarchical storage management system. A closed queuing network model of the client-server storage system is used to analyze the performance of the system for various file movement criteria and to study the tradeoffs between recalling a file and accessing it by a distributed file system. Several network topologies (Ethernet, token ring, channel-to-channel) and their effect on performance are considered.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>