NetCDF Files Research Articles

RNetCDF – A Package for Reading and Writing NetCDF Datasets

This paper describes the RNetCDF package (version 1.6), an interface for reading and writing files in Unidata NetCDF format, and gives an introduction to the NetCDF file format. NetCDF is a machine independent binary file format which allows storage of different types of array based data, along with short metadata descriptions. The package presented here allows access to the most important functions of the NetCDF C-interface for reading, writing, and modifying NetCDF datasets. In this paper, we present a short overview on the NetCDF file format and show usage examples of the package.

ParCAT: Parallel Climate Analysis Toolkit

Climate science is employing increasingly complex models and simulations to analyze the past and predict the future of Earth's climate. This growth in complexity is creating a widening gap between the data being produced and the ability to analyze the datasets. Parallel computing tools are necessary to analyze, compare, and interpret the simulation data. The Parallel Climate Analysis Toolkit (ParCAT) provides basic tools to efficiently use parallel computing techniques to make analysis of these datasets manageable. The toolkit provides the ability to compute spatio-temporal means, differences between runs or differences between averages of runs, and histograms of the values in a data set. ParCAT is implemented as a command-line utility written in C. This allows for easy integration in other tools and allows for use in scripts. This also makes it possible to run ParCAT on many platforms – from laptops to supercomputers. ParCAT outputs NetCDF files so it is compatible with existing utilities such as Panoply and UV-CDAT. This paper describes ParCAT and presents results from some example runs on the Titan system at ORNL.

Future Flows Climate: an ensemble of 1-km climate change projections for hydrological application in Great Britain

Abstract. The dataset Future Flows Climate was developed as part of the project ''Future Flows and Groundwater Levels'' to provide a consistent set of climate change projections for the whole of Great Britain at both space and time resolutions appropriate for hydrological applications, and to enable climate change uncertainty and climate variability to be accounted for in the assessment of their possible impacts on the environment. Future Flows Climate is derived from the Hadley Centre's ensemble projection HadRM3-PPE that is part of the basis of UKCP09 and includes projections in available precipitation (water available to hydrological processes after snow and ice storages have been accounted for) and potential evapotranspiration. It corresponds to an 11-member ensemble of transient projections from January 1950 to December 2098, each a single realisation from a different variant of HadRM3. Data are provided on a 1-km grid over the HadRM3 land areas at a daily (available precipitation) and monthly (PE) time step as netCDF files. Because systematic biases in temperature and precipitation were found between HadRM3-PPE and gridded temperature and precipitation observations for the 1962–1991 period, a monthly bias correction procedure was undertaken, based on a linear correction for temperature and a quantile-mapping correction (using the gamma distribution) for precipitation followed by a spatial downscaling. Available precipitation was derived from the bias-corrected precipitation and temperature time series using a simple elevation-dependant snow-melt model. Potential evapotranspiration time series were calculated for each month using the FAO-56 Penman-Monteith equations and bias-corrected temperature, cloud cover, relative humidity and wind speed from HadRM3-PPE along with latitude of the grid and the day of the year. Future Flows Climate is freely available for non-commercial use under certain licensing conditions. It is the dataset used to generate Future Flows Hydrology, an ensemble of transient projections of daily river flow and monthly groundwater time series for representative river basins and boreholes in Great Britain. doi:10.5285/bad1514f-119e-44a4-8e1e-442735bb9797.

Knowledge discovery in large model datasets in the marine environment: the THREDDS Data Server example

In order to monitor, describe and understand the marine environment, many research institutions are involved in the acquisition and distribution of ocean data, both from observations and models. Scientists from these institutions are spending too much time looking for, accessing, and reformatting data: they need better tools and procedures to make the science they do more efficient. The U.S. Integrated Ocean Observing System (US-IOOS) is working on making large amounts of distributed data usable in an easy and efficient way. It is essentially a network of scientists, technicians and technologies designed to acquire, collect and disseminate observational and modelled data resulting from coastal and oceanic marine regions investigations to researchers, stakeholders and policy makers. In order to be successful, this effort requires standard data protocols, web services and standards-based tools. Starting from the US-IOOS approach, which is being adopted throughout much of the oceanographic and meteorological sectors, we describe here the CNR-ISMAR Venice experience in the direction of setting up a national Italian IOOS framework using the THREDDS (THematic Real-time Environmental Distributed Data Services) Data Server (TDS), a middleware designed to fill the gap between data providers and data users. The TDS provides services that allow data users to find the data sets pertaining to their scientific needs, to access, to visualize and to use them in an easy way, without downloading files to the local workspace. In order to achieve this, it is necessary that the data providers make their data available in a standard form that the TDS understands, and with sufficient metadata to allow the data to be read and searched in a standard way. The core idea is then to utilize a Common Data Model (CDM), a unified conceptual model that describes different datatypes within each dataset. More specifically, Unidata (<a href="http://www.unidata.ucar.edu" target="_blank">www.unidata.ucar.edu</a>) has developed CDM specifications for many of the different kinds of data used by the scientific community, such as grids, profiles, time series, swath data. These datatypes are aligned the NetCDF Climate and Forecast (CF) Metadata Conventions and with Climate Science Modelling Language (CSML); CF-compliant NetCDF files and GRIB files can be read directly with no modification, while non compliant files can be modified to meet appropriate metadata requirements. Once standardized in the CDM, the TDS makes datasets available through a series of web services such as OPeNDAP or Open Geospatial Consortium Web Coverage Service (WCS), allowing the data users to easily obtain small subsets from large datasets, and to quickly visualize their content by using tools such as GODIVA2 or Integrated Data Viewer (IDV). In addition, an ISO metadata service is available through the TDS that can be harvested by catalogue broker services (e.g. GI-cat) to enable distributed search across federated data servers. Example of TDS datasets can be accessed at the CNR-ISMAR Venice site <a href="http://tds.ve.ismar.cnr.it:8080/thredds/catalog.html" target="_blank">http://tds.ve.ismar.cnr.it:8080/thredds/catalog.html</a>.

Knowledge discovery in large model datasets in the marine environment: the THREDDS Data Server example

In order to monitor, describe and understand the marine environment, many research institutions are involved in the acquisition and distribution of ocean data, both from observations and models. Scientists from these institutions are spending too much time looking for, accessing, and reformatting data: they need better tools and procedures to make the science they do more efficient. The U.S. Integrated Ocean Observing System (US-IOOS) is working on making large amounts of distributed data usable in an easy and efficient way. It is essentially a network of scientists, technicians and technologies designed to acquire, collect and disseminate observational and modelled data resulting from coastal and oceanic marine regions investigations to researchers, stakeholders and policy makers. In order to be successful, this effort requires standard data protocols, web services and standards-based tools. Starting from the US-IOOS approach, which is being adopted throughout much of the oceanographic and meteorological sectors, we describe here the CNR-ISMAR Venice experience in the direction of setting up a national Italian IOOS framework using the THREDDS (THematic Real-time Environmental Distributed Data Services) Data Server (TDS), a middleware designed to fill the gap between data providers and data users. The TDS provides services that allow data users to find the data sets pertaining to their scientific needs, to access, to visualize and to use them in an easy way, without downloading files to the local workspace. In order to achieve this, it is necessary that the data providers make their data available in a standard form that the TDS understands, and with sufficient metadata to allow the data to be read and searched in a standard way. The core idea is then to utilize a Common Data Model (CDM), a unified conceptual model that describes different datatypes within each dataset. More specifically, Unidata (www.unidata.ucar.edu) has developed CDM specifications for many of the different kinds of data used by the scientific community, such as grids, profiles, time series, swath data. These datatypes are aligned the NetCDF Climate and Forecast (CF) Metadata Conventions and with Climate Science Modelling Language (CSML); CF-compliant NetCDF files and GRIB files can be read directly with no modification, while non compliant files can be modified to meet appropriate metadata requirements. Once standardized in the CDM, the TDS makes datasets available through a series of web services such as OPeNDAP or Open Geospatial Consortium Web Coverage Service (WCS), allowing the data users to easily obtain small subsets from large datasets, and to quickly visualize their content by using tools such as GODIVA2 or Integrated Data Viewer (IDV). In addition, an ISO metadata service is available through the TDS that can be harvested by catalogue broker services (e.g. GI-cat) to enable distributed search across federated data servers. Example of TDS datasets can be accessed at the CNR-ISMAR Venice site http://tds.ve.ismar.cnr.it:8080/thredds/catalog.html.

Semi-automated feature-tracking of East Antarctic sea ice from Envisat ASAR imagery

While feature tracking of sea ice using cross-correlation methods on pairs of satellite Synthetic Aperture Radar (SAR) images has been extensively carried out in the Arctic, this is not the case in the Antarctic. This is due to the dynamic nature of Antarctic pack ice, its microwave signature, the tendency for SAR swath paths to be poorly aligned with the often narrow sea ice zone around the continent and inadequate satellite sampling. A semi-automated system, known as IPADS (IMCORR [IMageCORRelation] Processing, Analysis and Display System), has been developed to map fast ice and pack ice in Antarctica using multiple pairs of SAR images. The software processing pipeline uses overlapping image pairs which are geocoded and roughly registered using only data contained in the image headers. Next, fast ice maps are rapidly generated using zero motion features located within ocean regions. This also provides precise image registration. Finally, the same image pairs are re-examined for pack ice motion in a slower off-line batch process. The pack and fast ice are identified using a cluster-based search method which compares both location and motion information. Each image pair generates a NetCDF file which adds to a growing database of Antarctic sea ice motion and ice roughness. Five image-pair examples are presented to illustrate the methods used as well as their strengths and limitations. Substantial pack ice motion can often be detected in the marginal ice zone on SAR images only a few days apart.

A data delivery system for IMOS, the Australian Integrated Marine Observing System

Abstract. The Integrated Marine Observing System (IMOS, www.imos.org.au), an AUD $150 m 7-year project (2007–2013), is a distributed set of equipment and data-information services which, among many applications, collectively contribute to meeting the needs of marine climate research in Australia. The observing system provides data in the open oceans around Australia out to a few thousand kilometres as well as the coastal oceans through 11 facilities which effectively observe and measure the 4-dimensional ocean variability, and the physical and biological response of coastal and shelf seas around Australia. Through a national science rationale IMOS is organized as five regional nodes (Western Australia – WAIMOS, South Australian – SAIMOS, Tasmania – TASIMOS, New SouthWales – NSWIMOS and Queensland – QIMOS) surrounded by an oceanic node (Blue Water and Climate). Operationally IMOS is organized as 11 facilities (Argo Australia, Ships of Opportunity, Southern Ocean Automated Time Series Observations, Australian National Facility for Ocean Gliders, Autonomous Underwater Vehicle Facility, Australian National Mooring Network, Australian Coastal Ocean Radar Network, Australian Acoustic Tagging and Monitoring System, Facility for Automated Intelligent Monitoring of Marine Systems, eMarine Information Infrastructure and Satellite Remote Sensing) delivering data. IMOS data is freely available to the public. The data, a combination of near real-time and delayed mode, are made available to researchers through the electronic Marine Information Infrastructure (eMII). eMII utilises the Australian Academic Research Network (AARNET) to support a distributed database on OPeNDAP/THREDDS servers hosted by regional computing centres. IMOS instruments are described through the OGC Specification SensorML and where-ever possible data is in CF compliant netCDF format. Metadata, conforming to standard ISO 19115, is automatically harvested from the netCDF files and the metadata records catalogued in the OGC GeoNetwork Metadata Entry and Search Tool (MEST). Data discovery, access and download occur via web services through the IMOS Ocean Portal (http://imos.aodn.org.au) and tools for the display and integration of near real-time data are in development.

A web interface for griding arbitrarily distributed in situ data based on Data-Interpolating Variational Analysis (DIVA)

Abstract. Spatial interpolation of observations on a regular grid is a common task in many oceanographic disciplines (and geosciences in general). It is often used to create climatological maps for physical, biological or chemical parameters representing e.g. monthly or seasonally averaged fields. Since instantaneous observations can not be directly related to a field representing an average, simple spatial interpolation of observations is in general not acceptable. DIVA (Data-Interpolating Variational Analysis) is an analysis tool which takes the error in the observations and the typical spatial scale of the underlying field into account. Barriers due to the coastline and the topography in general and also currents estimates (if available) are used to propagate the information of a given observation spatially. DIVA is a command-line driven application written in Fortran and Shell Scripts. To make DIVA easier to use, a web interface has been developed (http://gher-diva.phys.ulg.ac.be). Installation and compilation of DIVA is therefore not required. The user can directly upload the data in ASCII format and enter several parameters for the analysis. The analyzed field, location of the observations, and the error mask are presented as different layers using the Web Map Service protocol. They are visualized in the browser using the Javascript library OpenLayers allowing the user to interact with layers (for example zooming and panning). Finally, the results can be downloaded as a NetCDF file, Matlab/Octave file and Keyhole Markup Language (KML) file for visualization in applications such as Google Earth.

PyPnetCDF: A high level framework for parallel access to netCDF files

A Python tool for manipulating netCDF files in a parallel infrastructure is proposed. The parallel interface, PyPnetCDF, manages netCDF properties in a similar way to the serial version from ScientificPython, but hiding parallelism to the user. Implementations details and capabilities of the developed interfaces are given. Numerical experiments that show the friendly use of the interfaces and their behaviour compared with the native routines, are presented.

Sharing electronic structure and crystallographic data with ETSF_IO

We present a library of routines whose main goal is to read and write exchangeable files (NetCDF file format) storing electronic structure and crystallographic information. It is based on the specification agreed inside the European Theoretical Spectroscopy Facility (ETSF). Accordingly, this library is nicknamed ETSF_IO. The purpose of this article is to give both an overview of the ETSF_IO library and a closer look at its usage. ETSF_IO is designed to be robust and easy to use, close to Fortran read and write routines. To facilitate its adoption, a complete documentation of the input and output arguments of the routines is available in the package, as well as six tutorials explaining in detail various possible uses of the library routines. Program summary Program title: ETSF_IO Catalogue identifier: AEBG_v1_0 Program summary URL: http://cpc.cs.qub.ac.uk/summaries/AEBG_v1_0.html Program obtainable from: CPC Program Library, Queen's University, Belfast, N. Ireland Licensing provisions: Gnu Lesser General Public License No. of lines in distributed program, including test data, etc.: 63 156 No. of bytes in distributed program, including test data, etc.: 363 390 Distribution format: tar.gz Programming language: Fortran 95 Computer: All systems with a Fortran95 compiler Operating system: All systems with a Fortran95 compiler Classification: 7.3, 8 External routines: NetCDF, http://www.unidata.ucar.edu/software/netcdf Nature of problem: Store and exchange electronic structure data and crystallographic data independently of the computational platform, language and generating software Solution method: Implement a library based both on NetCDF file format and an open specification ( http://etsf.eu/index.php?page=standardization)

Accurate mass error correction in liquid chromatography time-of-flight mass spectrometry based metabolomics

Compound identification and annotation in (untargeted) metabolomics experiments based on accurate mass require the highest possible accuracy of the mass determination. Experimental LC/TOF-MS platforms equipped with a time-to-digital converter (TDC) give the best mass estimate for those mass signals with an intensity similar to that of the lock-mass used for internal calibration. However, they systematically underestimate the mass obtained at higher signal intensity and overestimate it at low signal intensities compared to that of the lock-mass. To compensate for these effects, specific tools are required for correction and automation of accurate mass calculations from LC/MS signals. Here, we present a computational procedure for the derivation of an intensity-dependent mass correction function. The chromatographic mass signals for a set of known compounds present in a large number of samples were reconstructed over consecutive scans for each sample. It was found that the mass error is a linear function of the logarithm of the signal intensity adjusted to the associated lock-mass intensity. When applied to all mass data points, the correction function reduced the mass error for the majority of the tested compounds to ≤1 ppm over a wide range of signal intensities. The mass correction function has been implemented in a Python 2.4 script, which accepts raw data in NetCDF format as input, corrects the detected masses and returns the corrected NetCDF files for subsequent (automated) processing, such as mass signal alignment and database searching.

The mgd77 supplement to the generic mapping tools

Marine geophysical data collected by government and academic vessels are archived at the US National Geophysical Data Center in Boulder, Colorado. Data exchanges between NGDC and source institutions use an ASCII, punch-card format known as the MGD77 format, reflecting a style of file design common in the 1970s. We have developed a set of new software tools that can convert between this exchange format and a new COARDS-compliant, netCDF-based, architecture-independent file format that we call the MGD77+ format. The new mgd77 tools allow the data to be manipulated in a variety of ways useful for marine research. These tools are written in POSIX-compliant C, are distributed as a supplement to the Generic Mapping Tools, and can be installed on any computer platform. Because the new format is COARDS and CF-1.0 compliant, the files can be read by any general-purpose program capable of decoding these standards; a welcome side effect. One such program is the Java application ncBrowse developed by NOAA. Furthermore, the more compact netCDF files have file sizes that are, on average, only 30% of the original sizes. Because procedural changes at NGDC and source institutions necessarily occur infrequently, it is expected that the MGD77 format will remain the official exchange format for underway geophysical data for some time, whereas the new MGD77+ format offers users much needed flexibility in how they use the data.

Stable Isotope Tagging of Epitopes

Identification of peptides presented in major histocompatibility complex (MHC) class I molecules after viral infection is of strategic importance for vaccine development. Until recently, mass spectrometric identification of virus-induced peptides was based on comparative analysis of peptide pools isolated from uninfected and virus-infected cells. Here we report on a powerful strategy aiming at the rapid, unambiguous identification of naturally processed MHC class I-associated peptides, which are induced by viral infection. The methodology, stable isotope tagging of epitopes (SITE), is based on metabolic labeling of endogenously synthesized proteins during infection. This is accomplished by culturing virus-infected cells with stable isotope-labeled amino acids that are expected to be anchor residues (i.e. residues of the peptide that have amino acid side chains that bind into pockets lining the peptide-binding groove of the MHC class I molecule) for the human leukocyte antigen allele of interest. Subsequently these cells are mixed with an equal number of non-infected cells, which are cultured in normal medium. Finally peptides are acid-eluted from immunoprecipitated MHC molecules and subjected to two-dimensional nanoscale LC-MS analysis. Virus-induced peptides are identified through computer-assisted detection of characteristic, binomially distributed ratios of labeled and unlabeled molecules. Using this approach we identified novel measles virus and respiratory syncytial virus epitopes as well as infection-induced self-peptides in several cell types, showing that SITE is a unique and versatile method for unequivocal identification of disease-related MHC class I epitopes.

Design and implementation of netCDF markup language (NcML) and its GML-based extension (NcML-G ML)

The Network Common Data Form (netCDF) is one of the primary methods of self-documenting data storage and access in the international geosciences research and education community and beyond. NetCDF was designed for use in a networked environment. The recent evolution toward web services approaches to data exchange has focused attention on communication via messages in the defacto standard XML language. XML is a text-based language while netCDF is based on a binary file storage mechanism; thus NcML is a natural augmentation of netCDF with extensions encapsulating descriptions of the structure and content of netCDF objects in an XML form. Since netCDF was designed to be self-documenting, the XML representation of internal netCDF documentation is a natural augmentation of the original netCDF concept. In fact, the netCDF Markup Language (NcML) and NcML-G (NcML-Geography) extensions described in this article have applications beyond merely representing the internal netCDF documentation in the XML language. The NcML coordinate system makes it possible to describe the coordinate system used to represent the netCDF dataset. Furthermore the NcML dataset is a tool for describing “virtual netCDF” files that may be aggregations of data from several existing netCDF files, or it can represent a target dataset to be created by transforming existing netCDF files into a new form described in the NcML language. The NcML-G extension provides a means for fusing the data models of the traditional netCDF atmospheric science community with those of the GIS community which is of the utmost importance. Bringing the data models and data systems of those communities together will foster an era of interdisciplinary research and education within the geosciences subdisciplines. It will also encourage closer interactions between the geosciences and the societal impacts community. The design and software implementation of the core NcML specification and its extensions are presented and discussed.