A scientific data processing framework for time series NetCDF data

Abstract

The Atmospheric Radiation Measurement (ARM) Data Integrator (ADI) is a framework designed to streamline the development of scientific algorithms that analyze, and models that use time-series NetCDF data. ADI automates the process of retrieving and preparing data for analysis, provides a modular, flexible framework that simplifies software development, and supports a data integration workflow. Algorithm and model input data, preprocessing, and output data specifications are defined through a graphical interface. ADI includes a library of software modules to support the workflow, and a source code generator that produces C, IDL®, and Python™ templates to jump start development. While developed for processing climate data, ADI can be applied to any time-series data. This paper discusses the ADI framework, and how ADI's capabilities can decrease the time and cost of implementing scientific algorithms allowing modelers and scientists to focus their efforts on their research rather than preparing and packaging data.