HDF5 for NPP sensor and environmental data records

Abstract

The Joint Polar Satellite System (JPSS) is the next generation of low earth orbiting environmental satellites. The JPSS and the National Polar-orbiting Operational Environmental Satellite System (NPOESS) Preparatory Project (NPP) satellites are sun-synchronous polar orbiters with a period of approximately once every 100 minutes. The NPP satellite launched on October 28, 2011, and the scheduled launch dates for the JPSS-1 and JPSS-2 satellites are 2016 and 2022 respectively. Together with the Interface Data Processing Segment (IDPS), the system provides global measurements of environmental conditions by collecting, disseminating, and processing data about the Earth's weather, atmosphere, oceans, land, and near-space environment with precision and detail never before achieved by operational weather satellites. This volume of data allows scientists and forecasters to monitor and predict weather patterns with greater speed and accuracy. Official NPP data products are delivered as Hierarchical Data Format 5 (HDF5) files. The HDF5 is a general-purpose file format and library designed and developed by the National Center for Supercomputing Applications (NCSA). The HDF5 was developed to permit flexible, portable, and efficient storage and retrieval of scientific datasets. The NPP uses the HDF5 structure to implement a data model specifically for NPP data products without use of any required extensions to the native library. Some advantages of using the HDF5 include efficient storage and input/output (I/O), including parallel I/O, and the fact that free, open source software are available for use with HDF5 data on multiple platforms [1]. This paper will discuss the HDF5 methodology employed by the NPP to address the problem of efficiently storing the vast quantity of data produced by the system.