Scientific Data Management System Research Articles

The design of a scientific data management system based on DOMAS at CSNS-II (preliminary stage)

At the second stage of China Spallation Neutron Source (CSNS-II), it is predicted that 2 PB raw experimental data will be produced annually from twenty instruments. Scientific computing puts forward higher requirements for data sharing, utilization, retrieval, analysis efficiency, and security. However, the existing data management system (DMS) based on ICAT has several limitations including poor scalability of metadata database, imperfect data-management lifecycle and inflexible API. To ensure the accuracy, usability, scalability and efficiency of CSNS-II experimental data, a new scientific data management system is therefore designed based on the DOMAS framework developed by the Computing Center of IHEP. The data acquisition, transmission, storage and service systems are re-designed and tailored specifically for CSNS-II. Upon its completion, the new DMS will overcome the existing challenges and offer functions such as online display, search functionality and rapid download capabilities for metadata, raw data and analyzed data; flexible and user-friendly authorization; and data lifecycle management. Ultimately, the implementation of the new Data Management System (DMS) is expected to enhance the efficiency of experimental data analysis, propelling CSNS-II to achieve international advanced standards. Furthermore, it aims to reinforce self-reliance and technological strength in the field of science and technology at a high level in China. The development and deployment of the new DMS begin at the end of 2023.

Scientific and Technical Literature Data Management System Based on Life Cycle Model

Scientific and Technical Literature Data Management System Based on Life Cycle Model

Experience with Rucio in the wider HEP community

Managing the data of scientific projects is an increasingly complicated challenge, which was historically met by developing experiment-specific solutions. However, the ever-growing data rates and requirements of even small experiments make this approach very difficult, if not prohibitive. In recent years, the scientific data management system Rucio has evolved into a successful open-source project that is now being used by many scientific communities and organisations. Rucio is incorporating the contributions and expertise of many scientific projects and is offering common features useful to a diverse research community. This article describes the recent experiences in operating Rucio, as well as contributions to the project, by ATLAS, Belle II, CMS, ESCAPE, IGWN, LDMX, Folding@Home, and the UK’s Science and Technology Facilities Council (STFC).

Automatic test system of full electrical parameters for space traveling-wave tubes

In the development of space traveling wave tubes (TWTs), there have been many problems in testing their performance, such as too many test sections, complex operation processes, long test times and large amounts of test data. In this paper, an integrated test system for space TWTs is studied to achieve an accurate, fast and complete test of its full electrical parameters. The system can automatically switch the test instruments according to the tested contents, which eliminates the manual operation of setting the test bench and reduces personal error. Then, the one-button testing of full electrical parameters is realized by means of automatic test software of space TWTs, which greatly improves the test efficiency. Moreover, a standardized and scientific test data management system is established to ensure the integrity and objectivity of the test data. Finally, the testing accuracy and stability of the system are verified by data comparison.

A Context-aware Approach for Adapting Scientific E-Learning Objects

E-learning has arisen as a promising solution to lifelong learning and job training. It changed the way people learn and gain knowledge. Moreover, it has been embedded in many information systems as user guides to help and support them in their workplaces. Yet, these guides have been designed in a one-size-fits-all fashion that serves generic learners irrespective of their individual contexts. The developers of Scientific Data Management Systems (SDMSs) provide all scientists with a textual guide regardless of their contexts such as learning styles and expertise levels. In this paper, we propose a proactive context-aware approach for recommending personalized scientific data management e-learning objects. The proposed approach improves the learning process of the scientific data management depending on the scientist profile and his or her environmental context. The proposed approach includes two phases. First, the approach semantically detects the scientist’s context. Secondly, a hybrid recommendation technique is introduced to provide the scientist with an adapted list of learning objects. To create a generic approach, the web service-based methodology is exploited to facilitate the integration with any SDMS.

A Study of the Trade-off Between Reducing Precision and Reducing Resolution for Data Analysis and Visualization.

There currently exist two dominant strategies to reduce data sizes in analysis and visualization: reducing the precision of the data, e.g., through quantization, or reducing its resolution, e.g., by subsampling. Both have advantages and disadvantages and both face fundamental limits at which the reduced information ceases to be useful. The paper explores the additional gains that could be achieved by combining both strategies. In particular, we present a common framework that allows us to study the trade-off in reducing precision and/or resolution in a principled manner. We represent data reduction schemes as progressive streams of bits and study how various bit orderings such as by resolution, by precision, etc., impact the resulting approximation error across a variety of data sets as well as analysis tasks. Furthermore, we compute streams that are optimized for different tasks to serve as lower bounds on the achievable error. Scientific data management systems can use the results presented in this paper as guidance on how to store and stream data to make efficient use of the limited storage and bandwidth in practice.

Instrument makers invest in cloud computing

Thermo Fisher Scientific has acquired Core Informatics, a fast-growing, venture-capital-backed provider of cloud-based scientific data management systems with about 100 employees. Thermo Fisher announced the purchase last week at the Pittsburgh Conference on Analytical Chemistry & Applied Spectroscopy (Pittcon) in Chicago. It underscored the growing importance for instrument makers of systems capable of handling, storing, and manipulating the flood of data from today’s scientific instruments. “Their cloud-based offerings are more discovery oriented, while what we already had supported quality control and quality analysis in manufacturing processes,” explained Dan Shine, analytical instruments president at Thermo Fisher. “They will integrate seamlessly into what we already have.” Also at Pittcon, Waters Corp. launched the cloud version of its Empower chromatography data management system. “The cloud has changed the way the world interacts with data,” said Steve Smith, Waters’ informatics vice president, by...

The TileDB array data storage manager

We present a novel storage manager for multi-dimensional arrays that arise in scientific applications, which is part of a larger scientific data management system called TileDB. In contrast to existing solutions, TileDB is optimized for both dense and sparse arrays. Its key idea is to organize array elements into ordered collections called fragments. Each fragment is dense or sparse, and groups contiguous array elements into data tiles of fixed capacity. The organization into fragments turns random writes into sequential writes, and, coupled with a novel read algorithm, leads to very efficient reads. TileDB enables parallelization via multi-threading and multi-processing, offering thread-/process-safety and atomicity via lightweight locking. We show that TileDB delivers comparable performance to the HDF5 dense array storage manager, while providing much faster random writes. We also show that TileDB offers substantially faster reads and writes than the SciDB array database system with both dense and sparse arrays. Finally, we demonstrate that TileDB is considerably faster than adaptations of the Vertica relational column-store for dense array storage management, and at least as fast for the case of sparse arrays.

Abstract 5282: A cloud-enabled open source data management platform supporting a federated research and development organization

Abstract Biopharmaceutical R&D organizations characterize drug candidate target effects and modes of action and create molecular models of target diseases. These data-intensive activities are informed by vast data resources including publicly available data, internally generated data and partnered private data collections. However, rapid evolution in computing, data management tools, analytical and visualization methods, the complexity of data types and the data volumes that must be accommodated present significant technical and logistic hurdles to overcome. It is particularly difficult for a geographically dispersed R&D organization to make data resources easily available to scientists for search, visualization and exploration. Nevertheless, this is required for R&D scientists to gain insight into disease and drug mechanisms and to capture the knowledge needed to sustain the scientific enterprise. Standardized commercial solutions to R&D data challenges are unattractive since they require significant resource investment in platform configuration, user-training and system maintenance. This strategy necessarily creates delay in adopting newly emerging technologies and provides incentive not to adopt alternatives due to investment in existing systems. In contrast, our solution to R&D data demands was to build a cloud-deployed data platform using state of the art tools developed and maintained by the open source software community at the Apache Software Foundation. Partnering with academic data scientists, we selected the best available tools to fit our specific needs. We integrated them into a platform accessible to our federated R&D scientific community while allowing the system to be freely modified and updated on demand to meet evolving user requirements. Priorities for our data platform are to ingest, secure and index R&D source data of all types, make these indexed data assets available to computational scientists for analysis and provide faceted search capability based on a comprehensive metadata model. Three products: LabKey server, Apache OODT and ISATools have all been combined into a scientific data management system to provide a unified data resource enhanced by a search platform powered by Apache Solr. The platform supports both internally generated data and data imported from public, contracted or partnered sources. All data are available for interactive exploration by our R&D community, accessed via integrated search, analysis and visualization tools. Deployment of this system to our R&D organization has been met with enthusiastic adoption. Feedback for improvement or requests for system enhancements and additional capabilities are rapidly addressed in this open source environment, leading to further adoption among the R&D scientists and providing the basis for accessible, stable institutional knowledge collections. Citation Format: Lauren Intagliata, Selina Chu, Garth McGrath, Giuseppe Totaro, Daniel Civello, Nipurn Doshi, Shivika Thapar, Michael Livstone, Chris Mattmann, Paul Ramirez, Maureen Cronin. A cloud-enabled open source data management platform supporting a federated research and development organization. [abstract]. In: Proceedings of the 107th Annual Meeting of the American Association for Cancer Research; 2016 Apr 16-20; New Orleans, LA. Philadelphia (PA): AACR; Cancer Res 2016;76(14 Suppl):Abstract nr 5282.

Effective Integration of Informatics Tools to Enhance the Drug Discovery Process

Effective integration of data and laboratory informatics tools promises the ability for organizations to make better informed decisions about resource allocation during the drug discovery and development process, and for more informed decisions to be made with respect to the market opportunity for compounds. There are technologies on the horizon that could dramatically change how informatics organizations design, develop, deliver, and support applications and data infrastructures to deliver maximum value to drug discovery organizations. There are numerous benefits of integrating laboratory informatics tools like Laboratory Information Management System (LIMS), Chromatographic Data Systems (CDS), Electronic Laboratory Notebooks (ELN), and Analytical Instrumentation and Scientific Data Management Systems (SDMS) for drug discovery processes, but there is no well-established path for effective integration of these tools. We propose in this article four new integration models for effective integration of labor...

Electronic Laboratory Notebooks Progress and Challenges in Implementation

Electronic laboratory notebooks (ELNs) are increasingly replacing paper notebooks in life science laboratories, including those in industry, academic settings, and hospitals. ELNs offer significant advantages over paper notebooks, but adopting them in a predominantly paper-based environment is usually disruptive. The benefits of ELN increase when they are integrated with other laboratory informatics tools such as laboratory information management systems, chromatography data systems, analytical instrumentation, and scientific data management systems, but there is no well-established path for effective integration of these tools. In this article, we review and evaluate some of the approaches that have been taken thus far and also some radical new methods of integration that are emerging.

An Extensible Digital Library Service to Support Network Science

Network science research aims to understand the underlying properties of complex networks. Large-scale modeling and simulation is the core of network science research. Existing systems take a long time to run large network science experiments with high performance computing resources. Scientific data management systems currently lack the performance efficiency needed to support this type of computation and data-intensive research. Memoization provides the ability to index, archive, and reuse frequently requested and expensive to re-compute datasets. In this paper, a domain independent memoization service to increase the computational execution process performance within cyberinfrastructure-based systems is described. The service is formulated as an extensible memoization framework for the computational and simulation network science domains that is built on top of well-defined metadata objects. We present extensible concepts, discuss the proposed algorithm and framework architecture, and examine the flexible nature of the framework. The framework was utilized as a part of the CINET cyberinfrastructure-based digital library (DL). Our experimental results indicate an increase in the efficiency of the system and recommendation of the service's inclusion in scientific DLs.

Scientific Research Data Management for Soil-Vegetation-Atmosphere Data – The TR32DB

The implementation of a scientific research data management system is an important task within long-term, interdisciplinary research projects. Besides sustainable storage of data, including accurate descriptions with metadata, easy and secure exchange and provision of data is necessary, as well as backup and visualisation. The design of such a system poses challenges and problems that need to be solved.This paper describes the practical experiences gained by the implementation of a scientific research data management system, established in a large, interdisciplinary research project with focus on Soil-Vegetation-Atmosphere Data.

Informatics and Computing.

Informatics and Computing.

DEVELOPMENT OF A METADATA MANAGEMENT SYSTEM FOR AN INTERDISCIPLINARY RESEARCH PROJECT

Abstract. In every interdisciplinary, long-term research project it is essential to manage and archive all heterogeneous research data, produced by the project participants during the project funding. This has to include sustainable storage, description with metadata, easy and secure provision, back up, and visualisation of all data. To ensure the accurate description of all project data with corresponding metadata, the design and implementation of a metadata management system is a significant duty. Thus, the sustainable use and search of all research results during and after the end of the project is particularly dependent on the implementation of a metadata management system. Therefore, this paper will describe the practical experiences gained during the development of a scientific research data management system (called the TR32DB) including the corresponding metadata management system for the multidisciplinary research project Transregional Collaborative Research Centre 32 (CRC/TR32) 'Patterns in Soil-Vegetation-Atmosphere Systems'. The entire system was developed according to the requirements of the funding agency, the user and project requirements, as well as according to recent standards and principles. The TR32DB is basically a combination of data storage, database, and web-interface. The metadata management system was designed, realized, and implemented to describe and access all project data via accurate metadata. Since the quantity and sort of descriptive metadata depends on the kind of data, a user-friendly multi-level approach was chosen to cover these requirements. Thus, the self-developed CRC/TR32 metadata framework is designed. It is a combination of general, CRC/TR32 specific, as well as data type specific properties.

Applying Open-Source Software to Laboratory Data Management

The rise of new experimental techniques, such as high-throughput combinatorial methods, and the availability of large data sets by means of the Internet have greatly increased the amount of data that must be managed by relatively small projects. Scientific data management systems developed for large projects are often not available, suitable, nor affordable for projects with lesser resources. Increasing numbers of open-source frameworks have made available numerous options for smaller facilities to build for themselves effective and robust data management solutions. We will present considerations of these options and a case study.

An ontology-centric architecture for extensible scientific data management systems

Data management has become a critical challenge faced by a wide array of scientific disciplines in which the provision of sound data management is pivotal to the achievements and impact of research projects. Massive and rapidly expanding amounts of data combined with data models that evolve over time contribute to making data management an increasingly challenging task that warrants a new approach. In this paper we present an ontology-centric architecture for data management systems that is extensible and domain independent. In this architecture, the behaviors of domain concepts and objects are captured entirely by ontological entities, around which all data management tasks are carried out. The open and semantic nature of ontology languages also makes this architecture amenable to greater data reuse and interoperability. To evaluate the proposed architecture, we have applied it to the challenge of managing phenomics data.

A declarative language and toolkit for scientific workflow implementation and execution

Scientific workflow design is usually complex and demands access to and integration of numerous non-conventional resources. Geographical distribution and semantic heterogeneity of these resources add to this complexity. The cost effectiveness of such workflow design, thus, depends upon the lifespan of the application and its anticipated use. Shorter application lifespan usually entails prohibitive development costs. In this paper, we present an alternative platform for declarative workflow design using BioFlow in such environments. BioFlow is being developed as the query language for a scientific data management system called LifeDB that aims to support on-the-fly data integration and workflow support for life sciences applications. We argue that a declarative and ad hoc workflow design using BioFlow is more efficient and cost effective compared to traditional approaches using systems such as Taverna or Kepler. To demonstrate the advantages of BioFlow, we compare a canonical microarray data analysis workflow application design approach using BioFlow with Taverna and a gene regulation application using BioFlow and Kepler. We show that BioFlow supports ad hoc and modular application design at a throw away cost and produces a superior maintainable application that can adapt to changes in the source without significant effort compared to both Taverna and Kepler.

Tools for Aggregating, Analyzing and Mining Combinatorial Data

AbstractWe demonstrate how data mining techniques can be applied to complex combinatorial data sets and how data from multiple sources can be aggregated via the developed scientific data management system. An example is shown for the case of aggregated combinatorial data for the study of composition, processing, structure, and property relationships of transparent conducting oxides by applying data mining techniques such as principal component analysis. Data mappings of mined results are shown to effectively enable visualization of data trends, identification of anomalies in Fourier transform infrared spectroscopy patterns, and scientifically interesting libraries and spectral regions.

Distributed MDSplus database performance with Linux clusters

The staff at the DIII-D National Fusion Facility, operated for the USDOE by General Atomics, are investigating the use of grid computing and Linux technology to improve performance in our core data management services. We are in the process of converting much of our functionality to cluster-based and grid-enabled software. One of the most important pieces is a new distributed version of the MDSplus scientific data management system that is presently used to support fusion research in over 30 countries worldwide. To improve data handling performance, the staff is investigating the use of Linux clusters for both data clients and servers. The new distributed capability will result in better load balancing between these clients and servers, and more efficient use of network resources resulting in improved support of the data analysis needs of the scientific staff.