HDF5 Research Articles

Operational surface UV radiation product from GOME-2 and AVHRR/3 data

Abstract. The surface ultraviolet (UV) radiation product, version 1.20, generated operationally in the framework of the Satellite Application Facility on Ozone and Atmospheric Chemistry Monitoring (O3M SAF) of the European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT) is described. The product is based on the total ozone column derived from the measurements of the second Global Ozone Monitoring Experiment (GOME-2) instrument aboard EUMETSAT's polar orbiting meteorological operational (Metop) satellites. Cloud cover is taken into account by retrieving cloud optical depth from the channel 1 reflectance of the third Advanced Very High-Resolution Radiometer (AVHRR/3) instrument aboard both Metop in the morning orbit and Polar Orbiting Environmental Satellites (POES) of the National Oceanic and Atmospheric Administration (NOAA) in the afternoon orbit. In addition, more overpasses are used at high latitudes where the swaths of consecutive orbits overlap. The input satellite data are received from EUMETSAT's Multicast Distribution System (EUMETCast). The surface UV product includes daily maximum dose rates and integrated daily doses with different biological weighting functions, integrated ultraviolet B (UVB) and ultraviolet A (UVA) radiation, solar noon UV index and daily maximum photolysis frequencies of ozone and nitrogen dioxide at the surface level. The quantities are computed in a 0.5°×0.5° regular latitude–longitude grid and stored as daily files in the hierarchical data format (HDF5) within 2 weeks from sensing. The product files are archived in the O3M SAF distributed archive and can be ordered via the EUMETSAT Data Centre.

The SAMI Galaxy Survey: A prototype data archive for Big Science exploration

We describe the data archive and database for the SAMI Galaxy Survey, an ongoing observational program that will cover ≈3400 galaxies with integral-field (spatially-resolved) spectroscopy. Amounting to some three million spectra, this is the largest sample of its kind to date. The data archive and built-in query engine use the versatile Hierarchical Data Format (HDF5), which precludes the need for external metadata tables and hence the setup and maintenance overhead those carry. The code produces simple outputs that can easily be translated to plots and tables, and the combination of these tools makes for a light system that can handle heavy data. This article acts as a contextual companion to the SAMI Survey Database source code repository, samiDB, which is freely available online and written entirely in Python. We also discuss the decisions related to the selection of tools and the creation of data visualisation modules. It is our aim that the work presented in this article–descriptions, rationale, and source code–will be of use to scientists looking to set up a maintenance-light data archive for a Big Science data load.

HDFITS: Porting the FITS data model to HDF5

The FITS (Flexible Image Transport System) data format has been the de facto data format for astronomy-related data products since its inception in the late 1970s. While the FITS file format is widely supported, it lacks many of the features of more modern data serialization, such as the Hierarchical Data Format (HDF5). The HDF5 file format offers considerable advantages over FITS, such as improved I/O speed and compression, but has yet to gain widespread adoption within astronomy. One of the major holdbacks is that HDF5 is not well supported by data reduction software packages and image viewers. Here, we present a comparison of FITS and HDF5 as a format for storage of astronomy datasets. We show that the underlying data model of FITS can be ported to HDF5 in a straightforward manner, and that by doing so the advantages of the HDF5 file format can be leveraged immediately. In addition, we present a software tool, fits2hdf, for converting between FITS and a new `HDFITS' format, where data are stored in HDF5 in a FITS-like manner. We show that HDFITS allows faster reading of data (up to 100x of FITS in some use cases), and improved compression (higher compression ratios and higher throughput). Finally, we show that by only changing the import lines in Python-based FITS utilities, HDFITS formatted data can be presented transparently as an in-memory FITS equivalent.

The Hackbusch conjecture on tensor formats

We prove a conjecture of W. Hackbusch about tensor network states related to a perfect binary tree and train track tree. Tensor network states are used to present seemingly complicated tensors in a relatively simple and efficient manner. Each such presentation is described by a binary tree and a collection of vector spaces, one for each vertex of the tree. A problem suggested by Wolfgang Hackbusch and Joseph Landsberg is to compare the complexities of encodings, if one presents the same tensor with respect to two different trees. We answer this question when the two trees are extremal cases: the most “spread” tree (perfect binary tree), and the “deepest” binary tree (train track tree). The corresponding tensor formats are called hierarchical formats (HF) and tensor train (TT) formats, respectively.

Enabling Scientific Collaboration and Discovery Through the Use of Data Standardization

The Railgun Data Format Standard (RDFS) provides a flexible format for the archival of electromagnetic railgun (EMRG) laboratory test data. This effort utilizes the hierarchical data format version 5 for the data model and file format, as well as extensible markup language for data structure and data type encapsulation. The RDFS was developed through a collaborative effort within the Office of Naval Research (ONR) EMRG program, including members from the Naval Surface Warfare Center Dahlgren, the Naval Research Laboratory, and the Institute for Advanced Technology. The intent behind the RDFS is to facilitate collaborative data sharing among EMRG researchers through the standardization of EMRG test data archives. The structure of the RDFS is discussed, along with details on its implementation and use within the ONR EMRG community.

HDF5 Based Parallel I/O Techniques for Multi-Zone Structured Grids CFD Applications

HDF5 Based Parallel I/O Techniques for Multi-Zone Structured Grids CFD Applications

OBIA System for Identifying Mesoscale Oceanic Structures in SeaWiFS and MODIS-Aqua Images

The ocean covers over 70% of the surface of our planet and plays a key role in the global climate. Most ocean circulation is mesoscale (scales of 50–500 km and 10–100 days), and the energy in mesoscale circulation is at least one order of magnitude greater than general circulation; therefore, the study of mesoscale oceanic structures (MOS) is crucial to ocean dynamics, making it especially useful for analyzing global changes. The detection of MOS, such as upwellings or eddies, from satellites images is significant for marine environmental studies and coastal resource management. In this paper, we present an object-based image analysis (OBIA) system which segments and classifies regions contained in sea-viewing field-of-view sensor (SeaWiFS) and Moderate Resolution Imaging Spectro-radiometer (MODIS)-Aqua sensor satellite images into MOS. After color clustering and hierarchical data format (HDF) file processing, the OBIA system segments images and extracts image descriptors, producing primary regions. Then, it merges regions, recalculating image descriptors for MOS identification and definition. First, regions are labeled by a human-expert, who identifies MOS: upwellings, eddies, cool, and warm eddies. Labeled regions are then classified by learning algorithms (i.e., decision tree, Bayesian network, artificial neural network, genetic algorithm, and near neighbor algorithm ) from selected features. Finally, the OBIA system enables images to be queried from the user interface and retrieved by means of fuzzy descriptors and oceanic structures. We tested our system with images from the Canary Islands and the North West African coast.

Reimplementing the Hierarchical Data System using HDF5

The Starlink Hierarchical Data System has been a very successful niche astronomy file format and library for over 30 years. Development of the library was frozen ten years ago when funding for Starlink was stopped and almost no-one remains who understands the implementation details. To ensure the long-term sustainability of the Starlink application software and to make the extensible N-Dimensional Data Format accessible to a broader range of users, we propose to re-implement the HDS library application interface as a layer on top of the Hierarchical Data Format version 5. We present an overview of the new implementation of version 5 of the HDS file format and describe differences between the expectations of the HDS and HDF5 library interfaces. We finish by comparing the old and new HDS implementations by looking at a comparison of file sizes and by comparing performance benchmarks.

Botanical name changes - nuisance or a quest for precision?

To understand the need for the seemingly regular changes to plant names applied to many tropical forage species, it is necessary to be aware of the rules that govern botanical nomenclature. The binomial naming system, first proposed in 1753, is governed by rules defined in the International Code of Nomenclature for algae, fungi and plants (ICN). These rules have been strengthened as necessary over the years in the interest of providing practitioners with plant names that are unique for each species, and presented in an hierarchical format that shows the evolutionary relationships between plants. This paper includes a table of name changes accepted by the USDA Germplasm Resources Information Network (GRIN) for species used in tropical forage research and development over the last half century. The need to use legitimate plant names is emphasized and suggestions are made on how practitioners might best deal with the changes.Keywords: Taxonomy, nomenclature, tropical forages.DOI: 10.17138/TGFT(3)34-40

Variants of Alternating Least Squares Tensor Completion in the Tensor Train Format

We consider the problem of fitting a low rank tensor $A\in\mathbb{R}^{{\mathcal I}}$, ${\mathcal I} = \{1,\ldots,n\}^{d}$, to a given set of data points $\{M_i\in\mathbb{R} \mid i\in P\}$, $P\subset{\mathcal I}$. The low rank format under consideration is the hierarchical or tensor train or matrix product states format. It is characterized by rank bounds $r$ on certain matricizations of the tensor. The number of degrees of freedom is in ${\cal O}(r^2dn)$. For a fixed rank and mode size $n$ we observe that it is possible to reconstruct random (but rank structured) tensors as well as certain discretized multivariate (but rank structured) functions from a number of samples that is in ${\cal O}(\log N)$ for a tensor having $N=n^d$ entries. We compare an alternating least squares (ALS) fit to an overrelaxation scheme inspired by the LMaFit method for matrix completion. Both approaches aim at finding a tensor $A$ that fulfills the first order optimality conditions by a nonlinear Gauss--Seidel-type solver that consists of an alternating fit cycling through the directions $\mu=1,\ldots,d$. The least squares fit is of complexity ${\cal O}(r^4d\#P)$ per step, whereas each step of ADF (alternating directions fitting) is in ${\cal O}(r^2d\#P)$, albeit with a slightly higher number of necessary steps. In the numerical experiments we observe robustness of the completion algorithm with respect to noise and good reconstruction capability. Our tests provide evidence that the algorithm is suitable in higher dimensions ($>$10) as well as for moderate ranks.

Hierarchical Tensor Approximation of Output Quantities of Parameter-Dependent PDEs

Parametric PDEs appear in a large number of applications such as, e.g., uncertainty quantification and optimization. In many cases, one is interested in scalar output quantities induced by the parameter-dependent solution. The output can be interpreted as a tensor living on a high-dimensional parameter space. Our aim is to adaptively construct an approximation of this tensor in a data-sparse hierarchical tensor format. Once this approximation from an offline computation is available, the evaluation of the output for any parameter value becomes a cheap online task. Moreover, the explicit tensor representation can be used to compute stochastic properties of the output in a straightforward way. The potential of this approach is illustrated by numerical examples.

Applying a sunburst visualization to summarize user navigation sequences.

For many Web-based applications, it's important to be able to analyze the paths users have taken through a site--for example, to understand how they're discovering engaging content. These paths are difficult to summarize visually because of the underlying data's complexity. A Google researcher applied a sunburst visualization to this problem, after simplifying the data into a hierarchical format. The resulting visualization was successful in YouTube and is widely referenced and accessed. The code for the visualization is available as open source.

Automated code generation from LEMS, the general purpose model specification language underpinning NeuroML2

Making models more transparent, reproducible and accessible is a major challenge in computational neuroscience. The wide range of modelling tools and strategies available, while convenient from a pragmatic point of view, often make model reuse harder. Efforts on standardization [1], model sharing [2,3] and practices learned from the open-source community [3-5], on the other hand, are proving increasingly invaluable in improving the transparency and reusability of computational models. NeuroML2, the latest version of a model description language for computational neuroscience, is built on LEMS (Low Entropy Model Specification, [6]) a general purpose model specification language. LEMS has a highly structured, hierarchical format, rooted in physical principles, which allows the definition of physical systems including arbitrary networks of nodes modeled as hybrid dynamical systems. There are currently two LEMS interpreters, jLEMS [5] and PyLEMS [6], implemented in Java and Python respectively. As reference implementations, numerical performance and accuracy are not their primary goals. In order to simulate NeuroML2/LEMS models in an accurate, robust and efficient way it is important to be able to perform the numerical integration using well tested scientific simulators, both neuroscience specific, e.g. NEURON, and general purpose, e.g. XPPAUT, MATLAB. Such simulators provide a robust numerical infrastructure important for modelling in general, such as high-order ODE integration and accurate event detection. We have developed a set of template-based code generators for LEMS whose goal is to transform a LEMS model into a number of representations compatible with different simulators, using back-end specific templates to produce native code. The translation of the models happens through an intermediate format, developed for this purpose, named dLEMS (distilled LEMS). The dLEMS abstraction is ODE centric for convenience of interoperability with the different target languages. It represents a flattened (e.g. hierarchical channel descriptions are reduced to the core ODEs) and dimensionless version of the LEMS model. The advantage of this approach, compared for instance to a compiler based strategy, is that users willing to provide support for a target simulator can more easily work with dLEMS, as it closely reflects the ODE representations used in general purpose simulators. Users will simply have to create new templates for their target simulator or modify existing ones. Our initial results show that the template-based code-generation approach is feasible, producing runnable simulations in a variety of platforms for single cell models – in particular, templates are available for the following platforms: C code generation (using the LLNL Sundials library for numerical integration), XPPAUT, MATLAB and Modelica. All tools we have developed for this are open source and freely available. We are presently working at extending the dLEMS specification of connections between elements, creating a simplified format for template based mapping to neuronal simulators.

Mechanic: The MPI/HDF code framework for dynamical astronomy

We introduce the Mechanic, a new open-source code framework. It is designed to reduce the development effort of scientific applications by providing unified API (Application Programming Interface) for configuration, data storage and task management. The communication layer is based on the well-established Message Passing Interface (MPI) standard, which is widely used on variety of parallel computers and CPU-clusters. The data storage is performed within the Hierarchical Data Format (HDF5). The design of the code follows core–module approach which allows to reduce the user’s codebase and makes it portable for single- and multi-CPU environments. The framework may be used in a local user’s environment, without administrative access to the cluster, under the PBS or Slurm job schedulers. It may become a helper tool for a wide range of astronomical applications, particularly focused on processing large data sets, such as dynamical studies of long-term orbital evolution of planetary systems with Monte Carlo methods, dynamical maps or evolutionary algorithms. It has been already applied in numerical experiments conducted for Kepler-11 (Migaszewski et al., 2012) and νOctantis planetary systems (Goździewski et al., 2013). In this paper we describe the basics of the framework, including code listings for the implementation of a sample user’s module. The code is illustrated on a model Hamiltonian introduced by (Froeschlé et al., 2000) presenting the Arnold diffusion. The Arnold web is shown with the help of the MEGNO (Mean Exponential Growth of Nearby Orbits) fast indicator (Goździewski et al., 2008a) applied onto symplectic SABAn integrators family (Laskar and Robutel, 2001).

In-memory application-level checkpoint-based migration for MPI programs

Process migration provides many benefits for parallel environments including dynamic load balancing, data access locality or fault tolerance. This paper describes an in-memory application-level checkpoint-based migration solution for MPI codes that uses the Hierarchical Data Format 5 (HDF5) to write the checkpoint files. The main features of the proposed solution are transparency for the user, achieved through the use of CPPC (ComPiler for Portable Checkpointing); portability, as the application-level approach makes the solution adequate for any MPI implementation and operating system, and the use of the HDF5 file format enables the restart on different architectures; and high performance, by saving the checkpoint files to memory instead of to disk through the use of the HDF5 in-memory files. Experimental results prove that the in-memory approach reduces significantly the I/O cost of the migration process.

H5MD: A structured, efficient, and portable file format for molecular data

We propose a new file format named “H5MD” for storing molecular simulation data, such as trajectories of particle positions and velocities, along with thermodynamic observables that are monitored during the course of the simulation. H5MD files are HDF5 (Hierarchical Data Format) files with a specific hierarchy and naming scheme. Thus, H5MD inherits many benefits of HDF5, e.g., structured layout of multi-dimensional datasets, data compression, fast and parallel I/O, and portability across many programming languages and hardware platforms. H5MD files are self-contained, and foster the reproducibility of scientific data and the interchange of data between researchers using different simulation programs and analysis software. In addition, the H5MD specification can serve for other kinds of data (e.g. experimental data) and is extensible to supplemental data, or may be part of an enclosing file structure.

Tree Adaptive Approximation in the Hierarchical Tensor Format

The hierarchical tensor format allows for the low-parametric representation of tensors even in high dimensions $d$. The efficiency of this representation strongly relies on an appropriate hierarchical splitting of the different directions $1,\ldots,d$ such that the associated ranks remain sufficiently small. This splitting can be represented by a binary tree which is usually assumed to be given. In this paper, we address the question of finding an appropriate tree from a subset of tensor entries without any a priori knowledge on the tree structure. We propose an agglomerative strategy that can be combined with rank-adaptive cross approximation techniques such that tensors can be approximated in the hierarchical format in an entirely black box way. Numerical examples illustrate the potential and the limitations of our approach.

Comparative evaluation of the content and structure of communication using two handoff tools: Implications for patient safety

PurposeHandoffs vary in their structure and content, raising concerns regarding standardization. We conducted a comparative evaluation of the nature and patterns of communication on 2 functionally similar but conceptually different handoff tools: Subjective, Objective, Assessment and Plan, based on a patient problem-based format, and Handoff Intervention Tool (HAND-IT), based on a body system-based format. MethodA nonrandomized pre-post prospective intervention study supported by audio recordings and observations of 82 resident handoffs was conducted in a medical intensive care unit. Qualitative analysis was complemented with exploratory sequential pattern analysis techniques to capture the characteristics and types of communication events (CEs) and breakdowns. ResultsUse of HAND-IT led to fewer communication breakdowns (F1,80 = 45.66: P < .0001), greater number of CEs (t40 = 4.56; P < .001), with more ideal CEs than Subjective, Objective, Assessment and Plan (t40 = 9.27; P < .001). In addition, the use of HAND-IT was characterized by more request-response CE transitions. ConclusionThe HAND-IT's body system-based structure afforded physicians the ability to better organize and comprehend patient information and led to an interactive and streamlined communication, with limited external input. Our results also emphasize the importance of information organization using a medical knowledge hierarchical format for fostering effective communication.

HDF5 based parallel I/O technology for structure grid CFD applications

Large scale parallel Computational Fluid Dynamics( CFD) simulation has posed high demand on the data I /O capabilities. Hierarchical Data Format version 5( HDF5) can effectively manage large scale scientific data and has a good support to parallel I / O. The HDF5 data storage model for a parallel CFD application was designed. The parallel I / O method for the application's data was implemented based on the HDF5 parallel I / O API. Performance experiments were performed on a parallel computer system. The results show that the data writing speed of this HDF5 based parallel I / O method outperforms the parallel I / O methods( i. e. each process writes an ordinary data file independently) by 6. 9 to 16. 1 times, when 4 to 32processes are used. The data reading speed of this HDF5 based parallel I / O method is slower than the parallel I / O methods,with a speed of 20% to 70% times of the latter. However, the cost of data reading time is much smaller than the cost of data writing time, hence has minor effect on the total performance.

Approximation rates for the hierarchical tensor format in periodic Sobolev spaces

In this note we estimate the asymptotic rates for the L2-error decay and the storage cost when approximating 2π-periodic, d-variate functions from isotropic and mixed Sobolev classes by the recent hierarchical tensor format as introduced by Hackbusch and Kühn. To this end, we survey some results on bilinear approximation due to Temlyakov. The approach taken in this paper improves and generalizes recent results of Griebel and Harbrecht for the bi-variate case.