Scientific Formats Research Articles

Profiling heliophysics data in the pythonic cloud

Analysis of long timespan heliophysics and space physics data or application of machine learning algorithms can require access to petabyte-scale and larger data sets and sufficient computational capacity to process such “big data”. We provide a summary of Python support and performance statistics for the major scientific data formats under consideration for access to heliophysics data in cloud computing environments. The Heliophysics Data Portal lists 21 different formats used in heliophysics and space physics; our study focuses on Python support for the most-used formats of CDF, FITS, and NetCDF4/HDF. In terms of package support, there is no single Python package that supports all of the common heliophysics file types, while NetCDF/HDF5 is the most supported file type. In terms of technical implementation within a cloud environment, we profile file performance in Amazon Web Services (AWS). Effective use of AWS cloud-based storage requires Python libraries designed to read their S3 storage format. In Python, S3-aware libraries exist for CDF, FITS, and NetCDF4/HDF. The existing libraries use different approaches to handling cloud-based data, each with tradeoffs. With these caveats, Python pairs well with AWS’s cloud storage within the current Python ecosystem for existing heliophysics data, and cloud performance in Python is continually improving. We recommend anyone considering cloud use or optimization of data formats for cloud use specifically profile their given data set, as instrument-specific data characteristics have a strong effect on which approach is best for cloud use.

TSCat: data model and storage engine for AI-based light-curve analysis

ABSTRACT Light-curve data are one of the most important data sources in time domain astronomy research. With the advancement of observation facilities and the continuous accumulation of observation data, and considering the analysis needs of large sample data sets, software or tools based on new technologies, especially artificial intelligence (AI), will be indispensable for light-curve analysis. The light-curve analysis tool designed by an individual will follow its own defined data structure, which will make the tools designed and developed by different individuals incompatible. A unified light-curve data model will be able to solve this compatibility problem, similar to the traditional flexible image transport system (FITS) file format. This paper proposes a light-curve data model named TSCat, designs and implements a data storage engine. The TSCat data model defines the basic metadata and format required for storing optical curve data following the international virtual observatory alliance (IVOA) data format specification system. The TSCat storage engine implements basic operations such as importing, storing, and accessing light-curve data. The function and performance of the TSCat storage engine are evaluated through the actual observation data. The experimental results show that the TSCat data model is complete enough to support the analysis of light curves. TSCat will help support the standardization of the data access level of software and tools in the field of light-curve analysis, and provide a new reference for the specification system of astronomical scientific data formats.

Research data management practice in higher education institutions in Ethiopia

There is increasing pressure from funders, publishers, the public and other research organizations on researchers to improve their research data management. The main objective of the study was to examine the practices of research data management at higher education institutions in Ethiopia. The study was employed multi-stage sampling technique to select the actual sample unit of respondents. Eight first generation Universities and one Science and Technology University were selected based on purposive sampling technique. 390 faculty member and 119 stakeholder respondents were selected based on simple random and purposive sampling techniques. The findings revealed that majority of faculty members were used and produced spreadsheet and structured scientific and statistical research data formats. Majority of faculty members were used their personal device to hold their research data. The findings indicated that majority of faculty members were identified their needs of research data management services such as, data management template, storage facilities, research data management policy, guidelines and providing training. The level of awareness on research data management among faculty members was very low. The study concludes that proper research data management is not being practiced. Further research can be done on research data management framework at higher education institutions.

A review of CAD to CAE integration with a hierarchical data format (HDF)-based solution

Computer-Aided Design (CAD) and Computer-Aided Engineering (CAE) are two of the essential tools to design and develop new products. CAD systems help to design a new product efficiently, whereas CAE systems are used to predict and assess the future functionalities (durability, structural strength, etc.) of a product. Usually, these two systems work independently; however, their mutual dependency is a topic of interest for the researchers. This paper discusses the contemporary research efforts in the domain of CAD to CAE integration and later find the research gap. Based on the research gap this paper develop a methodology to integrate a CAD to CAE systems where scientific data format HDF (Hierarchical Data Format) acts as a bridge to link the gap between a CAD and CAE system. The virtual prototype of a product is getting accurate with the advancement of CAD systems; hence, the transferability of those models to a CAE system has become more sophisticated using traditional (e.g., .step, .iges files) methodologies. Since HDF is often used to handle massive and complex datasets for various research domains; hence, in this research effort HDF acts as a data repository neutral system to interact with the CAD and CAE systems. Based on the comprehensive literature review, this research effort aims to develop an uninterrupted integrated platform to handle complex design and analysis models using a big open-source data handling platform HDF (Hierarchical Data Format). MATLAB is used to bridge from the CAD system to HDF primarily and later from the HDF to CAE. A simple case study of analyzing a classic fluid mixing elbow demonstrates the preliminary implementation effort towards the proposed HDF-based integrated platform.

Evolution of a Long-Track Violent Tornado within a Simulated Supercell

Abstract Tornadoes are among nature’s most destructive forces. The most violent, long-lived tornadoes form within supercell thunderstorms. Tornadoes ranked EF4 and EF5 on the Enhanced Fujita scale that exhibit long paths are the least common but most damaging and deadly type of tornado. In this article we describe an ultra-high-resolution (30-m gridpoint spacing) simulation of a supercell that produces a long-track tornado that exhibits instantaneous near-surface storm-relative winds reaching as high as 143 m s−1. The computational framework that enables this work is described, including the Blue Waters supercomputer, the CM1 cloud model, a data management framework built around the HDF5 scientific data format, and the VisIt and Vapor visualization tools. We find that tornadogenesis occurs in concert with processes not clearly seen in previous supercell simulations, including the consolidation of numerous vortices and vorticity patches along the storm’s forward-flank downdraft boundary and the intensification of a feature we call a streamwise vorticity current (SVC), a current of horizontal vorticity that is tilted upward into the storm’s low-level mesocyclone. The SVC is found throughout the genesis and much of the maintenance phase of the tornado, where it appears to help drive the storm’s vigorous low-level updraft. We compare stages of the storm’s maintenance phase to observations. We find that tornado decay occurs rapidly throughout the depth of the tornado and is associated with a weakening of the SVC and the development of a strong rainy downdraft that encircles the tornado, which has moved rearward into the storm’s cold pool.

Methods for Specifying Scientific Data Standards and Modeling Relationships with Applications to Neuroscience.

Neuroscience continues to experience a tremendous growth in data; in terms of the volume and variety of data, the velocity at which data is acquired, and in turn the veracity of data. These challenges are a serious impediment to sharing of data, analyses, and tools within and across labs. Here, we introduce BRAINformat, a novel data standardization framework for the design and management of scientific data formats. The BRAINformat library defines application-independent design concepts and modules that together create a general framework for standardization of scientific data. We describe the formal specification of scientific data standards, which facilitates sharing and verification of data and formats. We introduce the concept of Managed Objects, enabling semantic components of data formats to be specified as self-contained units, supporting modular and reusable design of data format components and file storage. We also introduce the novel concept of Relationship Attributes for modeling and use of semantic relationships between data objects. Based on these concepts we demonstrate the application of our framework to design and implement a standard format for electrophysiology data and show how data standardization and relationship-modeling facilitate data analysis and sharing. The format uses HDF5, enabling portable, scalable, and self-describing data storage and integration with modern high-performance computing for data-driven discovery. The BRAINformat library is open source, easy-to-use, and provides detailed user and developer documentation and is freely available at: https://bitbucket.org/oruebel/brainformat.

Recognizing the Diversity of Contributions: A Case Study for Framing Attribution and Acknowledgement for Scientific Data

As scientific data volumes, format types, and sources increase rapidly with the invention and improvement of scientific capabilities, the resulting datasets are becoming more complex to manage as well. One of the significant management challenges is pulling apart the individual contributions of specific people and organizations within large, complex projects. This is important for two aspects: 1) assigning responsibility and accountability for scientific work, and 2) giving professional credit to individuals (e.g. hiring, promotion, and tenure) who work within such large projects. This paper aims to review the extant practice of data attribution and how it may be improved. Through a case study of creating a detailed attribution record for a climate model dataset, the paper evaluates the strengths and weaknesses of the current data attribution method and proposes an alternative attribution framework accordingly. The paper concludes by demonstrating that, analogous to acknowledging the different roles and responsibilities shown in movie credits, the methodology developed in the study could be used in general to identify and map out the relationships among the organizations and individuals who had contributed to a dataset. As a result, the framework could be applied to create data attribution for other dataset types beyond climate model datasets. Â

ASDF: A new data format for astronomy

Abstract We present the case for developing a successor format for the immensely successful FITS format. We first review existing alternative formats and discuss why we do not believe they provide an adequate solution. The proposed format is called the Advanced Scientific Data Format (ASDF) and is based on an existing text format, YAML, that we believe removes most of the current problems with the FITS format. An overview of the capabilities of the new format is given along with specific examples. This format has the advantage that it does not limit the size of attribute names (akin to FITS keyword names) nor place restrictions on the size or type of values attributes have. Hierarchical relationships are explicit in the syntax and require no special conventions. Finally, it is capable of storing binary data within the file in its binary form. At its basic level, the format proposed has much greater applicability than for just astronomical data.

The NeXus data format.

NeXus is an effort by an international group of scientists to define a common data exchange and archival format for neutron, X-ray and muon experiments. NeXus is built on top of the scientific data format HDF5 and adds domain-specific rules for organizing data within HDF5 files, in addition to a dictionary of well defined domain-specific field names. The NeXus data format has two purposes. First, it defines a format that can serve as a container for all relevant data associated with a beamline. This is a very important use case. Second, it defines standards in the form of application definitions for the exchange of data between applications. NeXus provides structures for raw experimental data as well as for processed data.

A Web Map Service implementation for the visualization of multidimensional gridded environmental data

We describe ncWMS, an implementation of the Open Geospatial Consortium's Web Map Service (WMS) specification for multidimensional gridded environmental data. ncWMS can read data in a large number of common scientific data formats – notably the NetCDF format with the Climate and Forecast conventions – then efficiently generate map imagery in thousands of different coordinate reference systems. It is designed to require minimal configuration from the system administrator and, when used in conjunction with a suitable client tool, provides end users with an interactive means for visualizing data without the need to download large files or interpret complex metadata. It is also used as a “bridging” tool providing interoperability between the environmental science community and users of geographic information systems. ncWMS implements a number of extensions to the WMS standard in order to fulfil some common scientific requirements, including the ability to generate plots representing timeseries and vertical sections. We discuss these extensions and their impact upon present and future interoperability. We discuss the conceptual mapping between the WMS data model and the data models used by gridded data formats, highlighting areas in which the mapping is incomplete or ambiguous. We discuss the architecture of the system and particular technical innovations of note, including the algorithms used for fast data reading and image generation. ncWMS has been widely adopted within the environmental data community and we discuss some of the ways in which the software is integrated within data infrastructures and portals.

Integrated Modeling of Tokamak Experiments with OMFIT

One Modeling Framework for Integrated Tasks (OMFIT) is a framework that allows data to be easily exchanged among different codes by providing a unifying data structure. The main idea at the base of OMFIT is to treat files, data and scripts as a uniform collection of objects organized into a tree structure, which provides a consistent way to access and manipulate such collection of heterogeneous objects, independent of their origin. Within the OMFIT tree, data can be copied/referred from one node to another and tasks can call each other allowing for complex compound task to be built. A top-level Graphical User Interface (GUI) allowing users to manage tree objects, carry out simulations and analyze the data either interactively or in batch. OMFIT supports many scientific data formats and when a file is loaded into the framework, its data populates the tree structure, automatically endowing it with many potential uses. Furthermore, seamless integration with experimental management systems allows direct manipulation of their data. In OMFIT modeling tasks are organized into modules, which can be easily combined to create arbitrarily-large multi-physics simulations. Modules inter-dependencies are seamlessly defined by variables referencing tree locations among them. Creation of new modules and customization of existing ones is encouraged by graphical tools for their management and an online repository. High level Application Programmer Interfaces (APIs) enable users to execute their codes on remote servers and creation application-specific GUIs. Finally, within OMFIT it is possible to visualize experimental and modeling data for both quick analysis and publication purposes. Examples of application to the DIII-D tokamak are presented.

Assessing Migration Risk for Scientific Data Formats

The majority of information about science, culture, society, economy and the environment is born digital, yet the underlying technology is subject to rapid obsolescence. One solution to this obsolescence, format migration, is widely practiced and supported by many software packages, yet migration has well known risks. For example, newer formats – even where similar in function – do not generally support all of the features of their predecessors, and, where similar features exist, there may be significant differences of interpretation.There appears to be a conflict between the wide use of migration and its known risks. In this paper we explore a simple hypothesis – that, where migration paths exist, the majority of data files can be safely migrated leaving only a few that must be handled more carefully – in the context of several scientific data formats that are or were widely used. Our approach is to gather information about potential migration mismatches and, using custom tools, evaluate a large collection of data files for the incidence of these risks. Our results support our initial hypothesis, though with some caveats. Further, we found that writing a tool to identify “risky” format features is considerably easier than writing a migration tool.

An Interface between 3D Modeling Tool and Scientific Data Format

An Interface between 3D Modeling Tool and Scientific Data Format

An application-level parallel I/O library for Earth system models

We describe the design and implementation of an application-level parallel I/O (PIO) library for the reading and writing of distributed arrays to several common scientific data formats. PIO provides the flexibility to control the number of I/O tasks through data rearrangement to an I/O friendly decomposition. This flexibility enables reductions in per task memory usage and improvements in disk I/O performance versus a serial I/O approach. We illustrate the impact various features within PIO have on memory usage and disk I/O bandwidth on a Cray XT5 system.

Middleware strategies for clouds and grids in e‐Science

Middleware strategies for clouds and grids in e‐Science

Event metadata records as a testbed for scalable data mining

At a data rate of 200 hertz, event metadata records ("TAGs," in ATLAS parlance) provide fertile grounds for development and evaluation of tools for scalable data mining. It is easy, of course, to apply HEP-specific selection or classification rules to event records and to label such an exercise "data mining," but our interest is different. Advanced statistical methods and tools such as classification, association rule mining, and cluster analysis are common outside the high energy physics community. These tools can prove useful, not for discovery physics, but for learning about our data, our detector, and our software. A fixed and relatively simple schema makes TAG export to other storage technologies such as HDF5 straightforward. This simplifies the task of exploiting very-large-scale parallel platforms such as Argonne National Laboratory's BlueGene/P, currently the largest supercomputer in the world for open science, in the development of scalable tools for data mining. Using a domain-neutral scientific data format may also enable us to take advantage of existing data mining components from other communities. There is, further, a substantial literature on the topic of one-pass algorithms and stream mining techniques, and such tools may be inserted naturally at various points in the event data processing and distribution chain. This paper describes early experience with event metadata records from ATLAS simulation and commissioning as a testbed for scalable data mining tool development and evaluation.

High-level buffering for hiding periodic output cost in scientific simulations

Scientific applications often need to write out large arrays and associated metadata periodically for visualization or restart purposes. In this paper, we present active buffering, a high-level transparent buffering scheme for collective I/O, in which processors actively organize their idle memory into a hierarchy of buffers for periodic output data. It utilizes idle memory on the processors, yet makes no assumption regarding runtime memory availability. Active buffering can perform background I/O while the computation is going on, is extensible to remote I/O for more efficient data migration, and can be implemented in a portable style in today's parallel I/O libraries. It can also mask performance problems of scientific data formats used by many scientists. Performance experiments with both synthetic benchmarks and real simulation codes on multiple platforms show that active buffering can greatly reduce the visible I/O cost from the application's point of view.

Alternative Astronomical Software Packages

At a time when the United Nations, in collaboration with Wolfram Research, has selected the Mathematica software package to inform developing countries about available scientific tools, it is important to review other choices of software being used by active scientists around the world. For observational astronomers with large volumes of digital data to be analyzed, the main challenges are data reductions, image handling, model comparisons, interactive fits, data simulations and visualizations, etc. There are several good alternative software packages such as: AIPS, IRAF, MIDAS and IDL The first three packages can be obtained free of charge by contacting the sponsoring institutions. Information can be obtained, via the World Wide Web, from the URLs indicated in the footnotes. IDL is a commercial package that can be used in all kinds of computer platforms and is extensively used in space astronomy (e.g., main language of software reduction packages of missions like IUE, HST, ROSAT, SOHO, etc). All of these packages are able to handle some of the most common commercial and scientific data formats (FITS, CDF and HDF). These software packages provide general tools for image processing and data reduction with emphasis on, but not limited to, astronomical applications. All of these packages have good active customer support strategies, the most useful ones being periodic newsletters, related meetings (e.g., annual meeting on Astronomical Data Analysis Software and Systems), software user groups, bulletin board discussions, FAQs, etc. The purpose of this paper is to present the relative usefulness, available platforms, associated libraries, related resources, of these software packages and the many already existing and potential astronomical applications.