NetCDF Research Articles

MetAlignID: A high-throughput software tool set for automated detection of trace level contaminants in comprehensive LECO two-dimensional gas chromatography time-of-flight mass spectrometry data

A new alternative data processing tool set, metAlignID, is developed for automated pre-processing and library-based identification and concentration estimation of target compounds after analysis by comprehensive two-dimensional gas chromatography with mass spectrometric detection. The tool set has been developed for and tested on LECO data. The software is developed to run multi-threaded (one thread per processor core) on a standard PC (personal computer) under different operating systems and is as such capable of processing multiple data sets simultaneously. Raw data files are converted into netCDF (network Common Data Form) format using a fast conversion tool. They are then preprocessed using previously developed algorithms originating from metAlign software. Next, the resulting reduced data files are searched against a user-composed library (derived from user or commercial NIST-compatible libraries) (NIST=National Institute of Standards and Technology) and the identified compounds, including an indicative concentration, are reported in Excel format. Data can be processed batch wise. The overall time needed for conversion together with processing and searching of 30 raw data sets for 560 compounds is routinely within an hour. The screening performance is evaluated for detection of pesticides and contaminants in raw data obtained after analysis of soil and plant samples. Results are compared to the existing data-handling routine based on proprietary software (LECO, ChromaTOF). The developed software tool set, which is freely downloadable at www.metalign.nl, greatly accelerates data-analysis and offers more options for fine-tuning automated identification toward specific application needs. The quality of the results obtained is slightly better than the standard processing and also adds a quantitative estimate. The software tool set in combination with two-dimensional gas chromatography coupled to time-of-flight mass spectrometry shows great potential as a highly-automated and fast multi-residue instrumental screening method.

NetCDF based data archiving system applied to ITER Fast Plant System Control prototype

EURATOM/CIEMAT and Technical University of Madrid (UPM) have been involved in the development of a FPSC [1] (Fast Plant System Control) prototype for ITER, based on PXIe (PCI eXtensions for Instrumentation). One of the main focuses of this project has been data acquisition and all the related issues, including scientific data archiving. Additionally, a new data archiving solution has been developed to demonstrate the obtainable performances and possible bottlenecks of scientific data archiving in Fast Plant System Control.The presented system implements a fault tolerant architecture over a GEthernet network where FPSC data are reliably archived on remote, while remaining accessible to be redistributed, within the duration of a pulse. The storing service is supported by a clustering solution to guaranty scalability, so that FPSC management and configuration may be simplified, and a unique view of all archived data provided. All the involved components have been integrated under EPICS [2] (Experimental Physics and Industrial Control System), implementing in each case the necessary extensions, state machines and configuration process variables. The prototyped solution is based on the NetCDF-4 [3,4] (Network Common Data Format) file format in order to incorporate important features, such as scientific data models support, huge size files management, platform independent codification, or single-writer/multiple-readers concurrency.In this contribution, a complete description of the above mentioned solution is presented, together with the most relevant results of the tests performed, while focusing in the benefits and limitations of the applied technologies.

MET-IDEA version 2.06; improved efficiency and additional functions for mass spectrometry-based metabolomics data processing

Metabolomics Ion-based Data Extraction Algorithm (MET-IDEA) is a computer program for processing large-scale metabolomics data. MET-IDEA utilizes network Common Data Form (netCDF) data files available from a diversity of chromatographically coupled mass spectrometry (MS) systems, utilizes the sensitivity and selectivity associated with selected ion quantification, and greatly reduces the time and effort necessary to obtain large-scale organized data. This article reports on recent improvements to MET-IDEA which include new visualization of peak integrations, display of mass spectra associated with integrated peaks, and optional manual peak integration. The computational performance of MET-IDEA has also been improved to avoid memory overflow during the processing of large data sets and the software made compatible with 64 bit CPUs and operating systems. These new functions improve the performance of MET-IDEA, and they allow users to visualize peak integrations and curate the results through manual integration if desired. The improved version of MET-IDEA better facilitates the quantitative analysis of complex MS-based metabolomics data. MET-IDEA is freely available for academic and non commercial use at ( http://bioinfo.noble.org/gateway/index.php?option=com_wrapper&Itemid=57 ). Commercial use is available via licensing agreement.

An Objective Algorithm for Detecting and Tracking Tropical Cloud Clusters: Implications for Tropical Cyclogenesis Prediction

Abstract An algorithm to detect and track global tropical cloud clusters (TCCs) is presented. TCCs are organized large areas of convection that form over warm tropical waters. TCCs are important because they are the “seedlings” that can evolve into tropical cyclones. A TCC satisfies the necessary condition of a “preexisting disturbance,” which provides the required latent heat release to drive the development of tropical cyclone circulations. The operational prediction of tropical cyclogenesis is poor because of weaknesses in the observational network and numerical models; thus, past studies have focused on identifying differences between “developing” (evolving into a tropical cyclone) and “nondeveloping” (failing to do so) TCCs in the global analysis fields to produce statistical forecasts of these events. The algorithm presented here has been used to create a global dataset of all TCCs that formed from 1980 to 2008. Capitalizing on a global, Gridded Satellite (GridSat) infrared (IR) dataset, areas of persistent, intense convection are identified by analyzing characteristics of the IR brightness temperature (Tb) fields. Identified TCCs are tracked as they move around their ocean basin (or cross into others); variables such as TCC size, location, convective intensity, cloud-top height, development status (i.e., developing or nondeveloping), and a movement vector are recorded in Network Common Data Form (NetCDF). The algorithm can be adapted to near-real-time tracking of TCCs, which could be of great benefit to the tropical cyclone forecast community.

Open radar interferometry software for mapping surface Deformation

Over the past 15 years, software for processing interferometric synthetic aperture radar (InSAR) data into maps of surface deformation has been developed and refined. The InSAR technique is commonly used to investigate deformation associated with earthquakes, volcanoes, withdrawal of crustal fluids, and coherent ice motions [Massonnet and Feigl, 1998]. The software, called Generic Mapping Tools Synthetic Aperture Radar (GMTSAR), is an open‐source (GNU General Public License) InSAR processing system designed for users familiar with Generic Mapping Tools (GMT) [Wessel and Smith, 1998]. The GMTSAR code is written in the C programing language and will run on any UNIX® computer. It requires installation of GMT and Network Common Data Format (NetCDF) and supports several fast Fourier transform libraries.

Globally Gridded Satellite Observations for Climate Studies

Geostationary satellites have provided routine, high temporal resolution Earth observations since the 1970s. Despite the long period of record, use of these data in climate studies has been limited for numerous reasons, among them that no central archive of geostationary data for all international satellites exists, full temporal and spatial resolution data are voluminous, and diverse calibration and navigation formats encumber the uniform processing needed for multisatellite climate studies. The International Satellite Cloud Climatology Project (ISCCP) set the stage for overcoming these issues by archiving a subset of the full-resolution geostationary data at ~10-km resolution at 3-hourly intervals since 1983. Recent efforts at NOAA's National Climatic Data Center to provide convenient access to these data include remapping the data to a standard map projection, recalibrating the data to optimize temporal homogeneity, extending the record of observations back to 1980, and reformatting the data for broad public distribution. The Gridded Satellite (GridSat) dataset includes observations from the visible, infrared window, and infrared water vapor channels. Data are stored in Network Common Data Format (netCDF) using standards that permit a wide variety of tools and libraries to process the data quickly and easily. A novel data layering approach, together with appropriate satellite and file metadata, allows users to access GridSat data at varying levels of complexity based on their needs. The result is a climate data record already in use by the meteorological community. Examples include reanalysis of tropical cyclones, studies of global precipitation, and detection and tracking of the intertropical convergence zone.

A New Global 0.5° Gridded Dataset (1901–2006) of a Multiscalar Drought Index: Comparison with Current Drought Index Datasets Based on the Palmer Drought Severity Index

Abstract A monthly global dataset of a multiscalar drought index is presented and compared in terms of spatial and temporal variability with the existing continental and global drought datasets based on the Palmer drought severity index (PDSI). The presented dataset is based on the standardized precipitation evapotranspiration index (SPEI). The index was obtained using the Climatic Research Unit (CRU) TS3.0 dataset at a spatial resolution of 0.5°. The advantages of the new dataset are that (i) it improves the spatial resolution of the unique global drought dataset at a global scale; (ii) it is spatially and temporally comparable to other datasets, given the probabilistic nature of the SPEI; and, in particular, (iii) it enables the identification of various drought types, given the multiscalar character of the SPEI. The dataset is freely available on the Web page of the Spanish National Research Council (CSIC) in three different formats [network Common Data Form (netCDF), binary raster, and plain text].

Metabolite Differentiation and Discovery Lab (MeDDL): A New Tool for Biomarker Discovery and Mass Spectral Visualization

The goal of this work was to design and implement a prototype software tool for the visualization and analysis of small molecule metabolite GC-MS and LC-MS data for biomarker discovery. The key features of the Metabolite Differentiation and Discovery Lab (MeDDL) software platform include support for the manipulation of large data sets, tools to provide a multifaceted view of the individual experimental results, and a software architecture amenable to modification and addition of new algorithms and software components. The MeDDL tool, through its emphasis on visualization, provides unique opportunities by combining the following: easy use of both GC-MS and LC-MS data; use of both manufacturer specific data files as well as netCDF (network Common Data Form); preprocessing (peak registration and alignment in both time and mass); powerful visualization tools; and built in data analysis functionality.

An Extended and Improved Special Sensor Microwave Imager (SSM/I) Period of Record

Abstract The National Oceanic and Atmospheric Administration National Climatic Data Center has served as the archive of the Defense Meteorological Satellite Program Special Sensor Microwave Imager (SSM/I) data from the F-8, F-10, F-11, F-13, F-14, and F-15 platforms covering the period from July 1987 to the present. Passive microwave satellite measurements from SSM/I have been used to generate climate products in support of national and international programs. The SSM/I temperature data record (TDR) and sensor data record (SDR) datasets have been reprocessed and stored as network Common Data Form (netCDF) 3-hourly files. In addition to reformatting the data, a normalized anomaly (z score) for each footprint temperature value was calculated by subtracting each radiance value with the corresponding monthly 1° grid climatological mean and dividing it by the associated climatological standard deviation. Threshold checks were also used to detect radiance, temporal, and geolocation values that were outside the expected ranges. The application of z scores and threshold parameters in the form of embedded quality flags has improved the fidelity of the SSM/I TDR/SDR period of record for climatological applications. This effort has helped to preserve and increase the data maturity level of the longest satellite passive microwave period of record while completing a key first step before developing a homogenized and intercalibrated SSM/I climate data record in the near future.

Oceanographic Data Quality Control By Means Of A Computational Tool.

This paper describes an application of Information Technologies (IT) in the data-processing and implementation of quality tests to oceanographic data measured by a wave directional ocean buoy with the goal of having reliable information for forecasting in the field of oceanography. The development of computational algorithms and the allocation of qualification flags are integrated into the development of a computational tool built under the Rational Unified Process which, according to the standards of the program International Exchange and Ocean Information (IODE - COI),validate the data measured by this kind devices: Waverider (wave) and embedded thermistor (temperature), and ADCP (current), and the allocation of quality flags from rules and algorithm developedfrom research and consulting enhancements documents of the area that meet the recommendations and experiences of the international oceanographic community in this area. As a result of this research was a software tool called -Y´ZuNaMyi which ensures that data measured by this kind of sensors that investors are currently at anchor of the Colombian coast, are subjected to every step that includes oceanographic data control quality (including verification of their qualifications and consistency), in this context will ensure interoperability under the format NetCDF (Network Common Data Form) widely used and accepted by the international community to exchange oceanographic measurements.

Design and implementation of netCDF markup language (NcML) and its GML-based extension (NcML-G ML)

The Network Common Data Form (netCDF) is one of the primary methods of self-documenting data storage and access in the international geosciences research and education community and beyond. NetCDF was designed for use in a networked environment. The recent evolution toward web services approaches to data exchange has focused attention on communication via messages in the defacto standard XML language. XML is a text-based language while netCDF is based on a binary file storage mechanism; thus NcML is a natural augmentation of netCDF with extensions encapsulating descriptions of the structure and content of netCDF objects in an XML form. Since netCDF was designed to be self-documenting, the XML representation of internal netCDF documentation is a natural augmentation of the original netCDF concept. In fact, the netCDF Markup Language (NcML) and NcML-G (NcML-Geography) extensions described in this article have applications beyond merely representing the internal netCDF documentation in the XML language. The NcML coordinate system makes it possible to describe the coordinate system used to represent the netCDF dataset. Furthermore the NcML dataset is a tool for describing “virtual netCDF” files that may be aggregations of data from several existing netCDF files, or it can represent a target dataset to be created by transforming existing netCDF files into a new form described in the NcML language. The NcML-G extension provides a means for fusing the data models of the traditional netCDF atmospheric science community with those of the GIS community which is of the utmost importance. Bringing the data models and data systems of those communities together will foster an era of interdisciplinary research and education within the geosciences subdisciplines. It will also encourage closer interactions between the geosciences and the societal impacts community. The design and software implementation of the core NcML specification and its extensions are presented and discussed.

NCWin — A Component Object Model (COM) for Processing and Visualizing NetCDF Data

NetCDF (Network Common Data Form) is a data sharing protocol and library that is commonly used in large-scale atmospheric and environmental data archiving and modeling. The NetCDF tool described here, named NCWin and coded with Borland C + + Builder, was built as a standard executable as well as a COM (component object model) for the Microsoft Windows environment. COM is a powerful technology that enhances the reuse of applications (as components). Environmental model developers from different modeling environments, such as Python, JAVA, VISUAL FORTRAN, VISUAL BASIC, VISUAL C + +, and DELPHI, can reuse NCWin in their models to read, write and visualize NetCDF data. Some Windows applications, such as ArcGIS and Microsoft PowerPoint, can also call NCWin within the application. NCWin has three major components: 1) The data conversion part is designed to convert binary raw data to and from NetCDF data. It can process six data types (unsigned char, signed char, short, int, float, double) and three spatial data formats (BIP, BIL, BSQ); 2) The visualization part is designed for displaying grid map series (playing forward or backward) with simple map legend, and displaying temporal trend curves for data on individual map pixels; and 3) The modeling interface is designed for environmental model development by which a set of integrated NetCDF functions is provided for processing NetCDF data. To demonstrate that the NCWin can easily extend the functions of some current GIS software and the Office applications, examples of calling NCWin within ArcGIS and MS PowerPoint for showing NetCDF map animations are given.

Climate Data File Management for Agricultural Modeling

This article presents a method to effectively manage climate data information for use in agricultural modeling. The method utilizes the Network Common Data Form (netCDF) system developed by the University Consortium for Atmospheric Research (UCAR).

NetCDF: an interface for scientific data access

The network common data form (NetCDF), a data abstraction for storing and retrieving multidimensional data, is described. NetCDF is distributed as a software library that provides a concrete implementation of that abstraction. The implementation provides a machine-independent format for representing scientific data. Together, the abstraction, library, and data format support the creation, access, and sharing of scientific information. NetCDF is useful for supporting objects that contain dissimilar kinds of data in a heterogeneous network environment and for writing application software that does not depend on application-specific formats. Independence from particular machine representations is achieved by using a nonproprietary standard for external data representation. The discussion covers NetCDF data abstraction and interface; dimensions, variables, and attributes; direct access and hyperslab access, the NetCDF library; the data format; ncdump and ncgen utilities; experience, usability, and performance; limitations of NetCDF; and future plans.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>