Tools to export published datasets together with metadata from IDL/Python/MATLAB and πScope

Abstract

This paper presents the efforts at PSFC to comply with a new directive from the White House Office of Science and Technology Policy, which requires that all publications supported by federal funding agencies (e.g. Department of Energy Office of Science, National Science Foundation) include machine-readable datasets for figures and tables (“open data requirement”). The memorandum [1] includes a specific language indicating that the data must be “stored and publicly accessible to search, retrieve and analysis”, which discourages the use of proprietary formats such as an IDL save sets and promotes the use of more openly available data formats. The manual effort required for producing a well documented, and thereby potentially useful dataset is not trivial. At PSFC, we defined a standard schema for data and metadata to comply with the new open data requirement. We chose HDF5 for our dataset file format and developed dataset file generation tools in several commonly used scientific languages (IDL, MATLAB, Python), and πScope [2,3], a python data analysis environment. A new web-based document submission and distribution system was also developed for the PSFC library to manage and distribute the collection of files associated with each manuscript.