The Impact of the Data Archiving File Format on Scientific Computing and Performance of Image Processing Algorithms in MATLAB Using Large HDF5 and XML Multimodal and Hyperspectral Data Sets

Abstract

Scientists require the ability to effortlessly share and process data collected and stored on a variety of computer platforms in specialized data storage formats. Experiments often generate large amounts of raw and corrected data and metadata, which describes and characterizes the raw data. Scientific teams and groups develop many formats and tools for internal use for specialized users with particular references and backgrounds. Researchers need a solution for querying, accessing, and analyzing large data sets of heterogeneous data, and demand high interoperability between data and various applications (Shasharina et al., 2007; Shishedjiev et al., 2010). Debate continues regarding which data format provides the greatest transparency and produces the most reliable data exchange. Currently, Extensible Markup Language (XML) and Hierarchical Data Format 5 (HDF5) formats are two solutions for sharing data. XML is a simple, platform-independent, flexible markup meta-language that provides a format for storing structured data, and is a primary format for data exchange across the Internet (McGrath, 2003). XML data files use Document Type Definitions (DTDs) and XML Schemas to define the data structures and definitions, including data formatting, attributes, and descriptive information about the data. A number of applications exist that use XML-based storage implementations for applications, including radiation and spectral measurements, simulation data of magnetic fields in human tissues, and describing and accessing fusion and plasma physics simulations (Shasharina et al., 2007; Shishedjiev et al., 2010). HDF5 is a data model, library, and file format for storing and managing data. HDF5 is portable and extensible, allowing applications to evolve in their use of HDF5 (HDF Group). HDF5 files provide the capability for self-documenting storage of scientific data in that the HDF5 data model provides structures that allow the file format to contain data about the file structure and descriptive information about the data contained in the file (Barkstrom, 2001). Similar to XML, numerous applications using the HDF5 storage format exist, such as fusion