HDF5 Research Articles

Large-scale parallel lattice Boltzmann–cellular automaton model of two-dimensional dendritic growth

An extremely scalable lattice Boltzmann (LB)–cellular automaton (CA) model for simulations of two-dimensional (2D) dendritic solidification under forced convection is presented. The model incorporates effects of phase change, solute diffusion, melt convection, and heat transport. The LB model represents the diffusion, convection, and heat transfer phenomena. The dendrite growth is driven by a difference between actual and equilibrium liquid composition at the solid–liquid interface. The CA technique is deployed to track the new interface cells. The computer program was parallelized using the Message Passing Interface (MPI) technique. Parallel scaling of the algorithm was studied and major scalability bottlenecks were identified. Efficiency loss attributable to the high memory bandwidth requirement of the algorithm was observed when using multiple cores per processor. Parallel writing of the output variables of interest was implemented in the binary Hierarchical Data Format 5 (HDF5) to improve the output performance, and to simplify visualization. Calculations were carried out in single precision arithmetic without significant loss in accuracy, resulting in 50% reduction of memory and computational time requirements. The presented solidification model shows a very good scalability up to centimeter size domains, including more than ten million of dendrites. Program summaryProgram title: 2DdendCatalogue identifier: AEQZ_v1_0Program summary URL:http://cpc.cs.qub.ac.uk/summaries/AEQZ_v1_0.htmlProgram obtainable from: CPC Program Library, Queen’s University, Belfast, UKLicensing provisions: Standard CPC license, http://cpc.cs.qub.ac.uk/licence/licence.htmlNo. of lines in distributed program, including test data, etc.: 29,767No. of bytes in distributed program, including test data, etc.: 3131,367Distribution format: tar.gzProgramming language: Fortran 90.Computer: Linux PC and clusters.Operating system: Linux.Has the code been vectorized or parallelized?: Yes. Program is parallelized using MPI. Number of processors used: 1–50,000RAM: Memory requirements depend on the grid sizeClassification: 6.5, 7.7.External routines: MPI (http://www.mcs.anl.gov/research/projects/mpi/), HDF5 (http://www.hdfgroup.org/HDF5/)Nature of problem:Dendritic growth in undercooled Al–3 wt% Cu alloy melt under forced convection.Solution method:The lattice Boltzmann model solves the diffusion, convection, and heat transfer phenomena. The cellular automaton technique is deployed to track the solid/liquid interface.Restrictions:Heat transfer is calculated uncoupled from the fluid flow. Thermal diffusivity is constant.Unusual features:Novel technique, utilizing periodic duplication of a pre-grown “incubation” domain, is applied for the scaleup test.Running time:Running time varies from minutes to days depending on the domain size and number of computational cores.

Chunked extendible dense arrays for scientific data storage

Several meetings of the Extremely Large Databases Community for large scale scientific applications advocate the use of multidimensional arrays as the appropriate model for representing scientific databases. Scientific databases gradually grow to massive sizes of the order of terabytes and petabytes. As such, the storage of such databases require efficient dynamic storage schemes where the array is allowed to arbitrarily extend the bounds of the dimensions. Conventional multidimensional array representations cannot extend or shrink their bounds without relocating elements of the data-set. In general, extendibility of the bounds of the dimensions, is limited to only one dimension. This paper presents a technique for storing dense multidimensional arrays by chunks such that the array can be extended along any dimension without compromising the access time for an element. This is done with a computed access mapping function, that maps the k-dimensional index onto a linear index of the storage locations. This concept forms the basis for the implementation of an array file of any number of dimensions, where the bounds of the array dimension can be extended arbitrarily. Such a feature currently exists in the Hierarchical Data Format version 5 (HDF5). However, extending the bound of a dimension in the HDF5 array file can be unusually expensive in time. Such extensions, in our storage scheme for dense array files, can still be performed while still accessing elements of the array at orders of magnitude faster than in HDF5 or conventional arrays-files. We also present theoretical and experimental analysis of our scheme with respect to access time and storage overhead. Such mapping scheme can be readily integrated into existing PGAS models for parallel processing in a cluster networked computing environment.

Considerations for developing a standard for storing electrophysiology data in HDF5

Considerations for developing a standard for storing electrophysiology data in HDF5

EpHDF – a standard for storing electrophysiology data in HDF5

epHDF – a standard for storing electrophysiology data in HDF5

Feasibility Study on a Scheme for Coexistence of DSP-Based PON and 10-Gbps/λ PON Using Hierarchical Star QAM Format

This paper presents a downlink scheme that permits the coexistence, on a passive optical network branch, of future DSP-based optical network units (ONUs) that can receive advanced modulation signals and legacy ONUs that can receive only on-off keying (OOK) signals. In the proposed scheme, the two types of signals, advanced modulation signals and OOK signals, are multiplexed by using a hierarchical star quadrature amplitude modulation (QAM) format and transmitted simultaneously on a single carrier. The design criteria and the feasibility of the proposal are discussed. The coexistence of 10G-class ONUs and DSP-based ONUs using hierarchical star 8-QAM format is numerically analyzed and experimentally evaluated. The results confirm that a 10-Gbps OOK signal and 30-Gbps star 8-QAM signal can be received with 30.5-dB loss budget by an APD and intradyne detector with DSP, respectively. The ability of the proposed approach to expand the coverage area is also introduced.

HAL: a hierarchical format for storing and analyzing multiple genome alignments

Motivation: Large multiple genome alignments and inferred ancestral genomes are ideal resources for comparative studies of molecular evolution, and advances in sequencing and computing technology are making them increasingly obtainable. These structures can provide a rich understanding of the genetic relationships between all subsets of species they contain. Current formats for storing genomic alignments, such as XMFA and MAF, are all indexed or ordered using a single reference genome, however, which limits the information that can be queried with respect to other species and clades. This loss of information grows with the number of species under comparison, as well as their phylogenetic distance.Results: We present HAL, a compressed, graph-based hierarchical alignment format for storing multiple genome alignments and ancestral reconstructions. HAL graphs are indexed on all genomes they contain. Furthermore, they are organized phylogenetically, which allows for modular and parallel access to arbitrary subclades without fragmentation because of rearrangements that have occurred in other lineages. HAL graphs can be created or read with a comprehensive C++ API. A set of tools is also provided to perform basic operations, such as importing and exporting data, identifying mutations and coordinate mapping (liftover).Availability: All documentation and source code for the HAL API and tools are freely available at http://github.com/glennhickey/hal.Contact: hickey@soe.ucsc.edu or haussler@soe.ucsc.eduSupplementary information: Supplementary data are available at Bioinformatics online.

Improving Multifrontal Methods by Means of Block Low-Rank Representations

Matrices coming from elliptic Partial Differential Equations (PDEs) have been shown to have a low-rank property: well defined off-diagonal blocks of their Schur complements can be approximated by low-rank products. Given a suitable ordering of the matrix which gives to the blocks a geometrical meaning, such approximations can be computed using an SVD or a rank-revealing QR factorization. The resulting representation offers a substantial reduction of the memory requirement and gives efficient ways to perform many of the basic dense algebra operations. Several strategies have been proposed to exploit this property. We propose a low-rank format called Block Low-Rank (BLR), and explain how it can be used to reduce the memory footprint and the complexity of direct solvers for sparse matrices based on the multifrontal method. We present experimental results that show how the BLR format delivers gains that are comparable to those obtained with hierarchical formats such as Hierarchical matrices (H matrices) and Hierarchically Semi-Separable (HSS matrices) but provides much greater flexibility and ease of use which are essential in the context of a general purpose, algebraic solver.

Lossless Compression of SKA Data Sets

With the size of astronomical data archives continuing to increase at an enormous rate, the providers and end users of astronomical data sets will benefit from effective data compression techniques. This paper explores different lossless data compression techniques and aims to find an optimal compression algorithm to compress astronomical data obtained by the Square Kilometre Array (SKA), which are new and unique in the field of radio astronomy. It was required that the compressed data sets should be lossless and that they should be compressed while the data are being read. The project was carried out in conjunction with the SKA South Africa office. Data compression reduces the time taken and the bandwidth used when transferring files, and it can also reduce the costs involved with data storage. The SKA uses the Hierarchical Data Format (HDF5) to store the data collected from the radio telescopes, with the data used in this study ranging from 29 MB to 9 GB in size. The compression techniques investigated in this study include SZIP, GZIP, the LZF filter, LZ4 and the Fully Adaptive Prediction Error Coder (FAPEC). The algorithms and methods used to perform the compression tests are discussed and the results from the three phases of testing are presented, followed by a brief discussion on those results.

A Typical Case of Recommending the Use of a Hierarchical Data Format

Storing large amounts of data in an efficient way from the point of view of very fast retrieval is animportant requirement for many industries like KEMA, a company offering consulting, testing andcertification in the energy business. Because their relational database could not cope fast enough with thelarge amounts of data involved, an alternative way for data storage was proposed, called HierarchicalData Format 5 (HDF5). HDF5 is a data model, a library, and file format for storing and managing data.Four hierarchical designs for storing and retrieving the large amounts of data involved were investigated.A benchmark was carried out in order to know which hierarchical structure would perform best.Eventually, a benchmark between HDF5 and MS SQL Server was carried out. It could be shown thatHDF5 performs four (4) times better for inserting and even 200 times better for retrieving data than theMS. SQL Server.

Reconceptualising policy integration in road safety management

Unless sustained, coordinated action is generated in road safety, road traffic deaths are poised to rise from approximately 1.3 to 1.9 million a year by 2020 (Krug, 2012). To generate this harmonised response, road safety management agencies are being urged to adopt multisectoral collaboration (WHO, 2009b), which is achievable through the principle of policy integration. Yet policy integration, in its current hierarchical format, is marred by a lack of universality of its interpretation, a failure to anticipate the complexities of coordinated effort, dearth of information about its design and the absence of a normative perspective to share responsibility. This paper addresses this ill-conception of policy integration by reconceptualising it through a qualitative examination of 16 road safety stakeholders’ written submissions, lodged with the Australian Transport Council in 2011. The resulting, new principle of policy integration, Participatory Deliberative Integration, provides a conceptual framework for the alignment of effort across stakeholders in transport, health, traffic law enforcement, relevant trades and the community. With the adoption of Participatory Deliberative Integration, road safety management agencies should secure the commitment of key stakeholders in the development and implementation of, amongst other policy measures, National Road Safety Strategies and Mix Mode Integrated Timetabling.

Processing of hierarchical data with hash objects Part 1 – Creation of XML documents

AbstractThe introduction of hash objects with SAS® version 9 brought about new opportunities for novel programming techniques. The recent discovery that hash tables can contain even other hash objects, also known as the ‘hash of hashes’, opens the door to their application to hierarchical data structures. This is because hierarchies, like the XML structure, can be considered ‘containers within containers’, and with the hash-of-hashes technique, tables can contain tables, thereby becoming compatible with hierarchical formats. This has the potential of being useful for processing clinical data in XML format. The advent of Clinical Data Interchange Standards Consortium (CDISC) and its increasing reliance on XML technology for handling clinical trial data implies that clinical SAS® programmers now have to get used to hierarchical data structures. Clinical trial data set submissions now have to be accompanied by metadata such as define.xml. The traditional SAS® data sets are tabular and relational, whereas the...

Approximation of the stochastic Galerkin matrix in the low‐rank canonical tensor format

AbstractIn this article we describe an efficient approximation of the stochastic Galerkin matrix which stems from a stationary diffusion equation. The uncertain permeability coefficient is assumed to be a log‐normal random field with given covariance and mean functions. The approximation is done in the canonical tensor format and then compared numerically with the tensor train and hierarchical tensor formats. It will be shown that under additional assumptions the approximation error depends only on smoothness of the covariance function and does not depend either on the number of random variables nor the degree of the multivariate Hermite polynomials. (© 2012 Wiley‐VCH Verlag GmbH & Co. KGaA, Weinheim)

Sports day in Canada: A national celebration of sport

Introduction: Participation in organised sports (OS) has been ecommended as an opportunity to increase young peoples’ hysical activity (PA) levels. Girls, however, spend a substantial roportion of time during OS training and games inactive. Thereore the purpose of this study was to document lesson context and oach behavior during OS practices and games, whichmay assist in dentifying opportunities to increase PA in OS. Methods: A cross-sectional design was used. Participants were 4 girls, mean (±SD) age 13.4±2.2 yr, recruited from 10 teams in hreeOS (netball, basketball, soccer) in theWestern Suburbs of Sydey, Australia. One training session and one game were observed or each team using the SOFIT direct observation system to docment lesson context and coach behavior. Coach behaviour was oded using a hierarchical format to record PA promotion, discourgement, and demonstration. Lesson context was coded into one f six categories: management, knowledge delivery, fitness, skill ractice, game play, and free play. Results:An average of 13.0±11.5 and 15.8±9.6 occurrences/hr ere observed during training and games where coaches prooted PA. Few occurrences/hr were observed during training nd games where coaches discouraged PA (1.9±2.9 vs 1.2±1.5). crossOS, therewere considerablymore occurrences/hr of coaches emonstrating PA during training than during games (7.6±6.1 vs .7±4.4). AcrossOS, a significantly highermeanpercentage of time uring games were spent in game play compared to training sesions (69.4±9.0% vs 22.6±24.9%), whereas a significantly higher ean percentage of time during training sessions were spent in kill practice (34.9±18.2% vs 6.9±4.6%) and fitness (8.5±5.8% s 2.3±2.4%) compared to games. Across OS games and trainng, a considerable percentage of time was spent in management 11.1±3.9% vs 15.0±4.7%) and knowledge delivery (8.8±4.4% vs 8.5±13.7%). Low percentages of time were observed for free play cross OS, for both games and training (1.6±4.6% vs 0.5±0.6%). Discussion: Observations suggest that across OS, coaches prooted PA more than they discouraged PA which may influence irls’ PA levels. Considerable percentages of time, however, were pent in management and knowledge delivery, where it is likely hat girls would be relatively inactive. There may be potential for mprovementwithinOS to optimise girls’ PA levels of by continuing o promote PA and reducing management and knowledge delivery ontent without interfering with fundamental learning opportuniies and skill development that occur in OS. This information on OS anbeused to develop strategies to increase girls’ PA levels through port.

Efficient low-rank approximation of the stochastic Galerkin matrix in tensor formats

In this article, we describe an efficient approximation of the stochastic Galerkin matrix which stems from a stationary diffusion equation. The uncertain permeability coefficient is assumed to be a log-normal random field with given covariance and mean functions. The approximation is done in the canonical tensor format and then compared numerically with the tensor train and hierarchical tensor formats. It will be shown that under additional assumptions the approximation error depends only on the smoothness of the covariance function and does not depend either on the number of random variables nor the degree of the multivariate Hermite polynomials.

Towards data format standardization for X-ray absorption spectroscopy

A working group on data format standardization for X-ray absorption spectroscopy (XAS) has recently formed under the auspices of the International X-ray Absorption Society and the XAFS Commission of the International Union of Crystallography. This group of beamline scientists and XAS practitioners has been tasked to propose data format standards to meet the needs of the world-wide XAS community. In this report, concepts for addressing three XAS data storage needs are presented: a single spectrum interchange format, a hierarchical format for multispectral X-ray experiment, and a relational database format for XAS data libraries.

Using HDF5 to work with large quantities of biological data

Using HDF5 to work with large quantities of biological data

The policy/procedure manual, part III: organizing the manual

PurposeThis article's purpose is to show how to go about developing a scheme for organizing a policy/procedure manual that will also make a good electronic and paper filing system.Design/methodology/approachThe article describes the problem of organizing large bodies of information, argues for a hierarchical format, and then shows how to formulate that kind of organizational scheme.FindingsThe finding is that hierarchical schemes are useful for organizing policy and procedure manuals as well as for filing systems.Practical implicationsHaving policies and procedures well organized saves time each time they are subsequently used.Social implicationsEfficiency and good public relations demand that libraries have well‐organized policy and procedure manuals.Originality/valueThe value of the article is in offering the theoretical organizational framework for the manual.

Reconstructing 3D profiles of flux distribution in array of unshunted Josephson junctions from 2D scanning SQUID microscope images

By using a specially designed algorithm (based on utilizing the so-called Hierarchical Data Format), we report on successful reconstruction of 3D profiles of local flux distribution within artificially prepared arrays of unshunted NbAlOxNb Josephson junctions from 2D surface images obtained via the scanning SQUID microscope. The analysis of the obtained results suggest that for large sweep areas, the local flux distribution significantly deviates from the conventional picture and exhibits a more complicated avalanche-type behavior with a prominent dendritic structure.

Codification and Transaction Costs

Between the Civil War and World War II, every state and the federal government shifted toward codified versions of their statutes. Academia has so far ignored the systemic effects of this dramatic change. For example, the consensus view in the academic literature about rules and standards has been that precise rules present higher enactment costs for legislatures than would general standards, while vague standards present higher information costs for courts and citizens than do rules. Systematic codification – featuring hierarchical format and numbering, topical arrangement, and cross-references – inverts this relationship, lowering transaction costs for legislatures and increasing information costs for courts and citizens, as statutes proliferate. This Article takes a first look at this problem. On the legislative side, codification makes it easier for special interest groups to obtain their desired legislation. It facilitates Coasean bargaining between legislators, and encourages legislative borrowing, which diminishes the “laboratories of democracy” phenomenon. For the courts, codification changes how judges interpret statutes, prompting them to focus more on the meaning of individual words than on the overall policy goals of enactment, and to rely more on external sources, such as legislative history. For both legislators and courts, codification functions as a Hartian rule of recognition, signaling legality for enacted rules. For the citizenry, the reduced legislative costs mean increased legislative output, yielding rapid proliferation of statutes and unmanageable legal information costs. More disturbingly, codification also fosters overcriminalization. While it may not be appropriate to revert to the pre-codified regime now, reexamining the unintended effects of codification can inform present and future choices for our legal system.

What is different about spinal pain?

BackgroundThe mechanisms subserving deep spinal pain have not been studied as well as those related to the skin and to deep pain in peripheral limb structures. The clinical phenomenology of deep spinal pain presents unique features which call for investigations which can explain these at a mechanistic level.MethodsTargeted searches of the literature were conducted and the relevant materials reviewed for applicability to the thesis that deep spinal pain is distinctive from deep pain in the peripheral limb structures. Topics related to the neuroanatomy and neurophysiology of deep spinal pain were organized in a hierarchical format for content review.ResultsSince the 1980’s the innervation characteristics of the spinal joints and deep muscles have been elucidated. Afferent connections subserving pain have been identified in a distinctive somatotopic organization within the spinal cord whereby afferents from deep spinal tissues terminate primarily in the lateral dorsal horn while those from deep peripheral tissues terminate primarily in the medial dorsal horn. Mechanisms underlying the clinical phenomena of referred pain from the spine, poor localization of spinal pain and chronicity of spine pain have emerged from the literature and are reviewed here, especially emphasizing the somatotopic organization and hyperconvergence of dorsal horn “low back (spinal) neurons”. Taken together, these findings provide preliminary support for the hypothesis that deep spine pain is different from deep pain arising from peripheral limb structures.ConclusionsThis thesis addressed the question “what is different about spine pain?” Neuroanatomic and neurophysiologic findings from studies in the last twenty years provide preliminary support for the thesis that deep spine pain is different from deep pain arising from peripheral limb structures.