Data management and archiving for the Pacific 2001 Air Quality Study

Abstract

The data management and archiving activities of the Pacific 2001 Air Quality Study were handled by the Pacific 2001 Data Centre which was run by the National Atmospheric Chemistry (NAtChem) Database and Analysis Facility of Environment Canada. To ensure that the Pacific 2001 Air Quality Study data were archived in a common way, the NARSTO Data Exchange Standard (DES) was used as the mandatory format for the data files, partially because it allowed for the inclusion of metadata within the data files and partially because it provided the necessary flexibility for handling the many measurement types used in the study. Described in detail in the paper, the DES is now readily available to the scientific community. After each DES data file was submitted to the Data Centre, a read-and-verify program was run to check its conformity to the DES and to detect incorrect and problematic data. The errors detected by the read-and-verify program were automatically documented and an error report was sent to the data originators for data correction and resubmission. Statistical summaries and data plots were created for all data files and subsequently sent to the data originators for review and further error detection. Of the 125 data files submitted to the Data Centre, only 5 were error-free upon first submission. A test of 17 randomly selected files determined that all but two required at least four iterations of the submission–error checking–resubmission cycle in order to produce final error-free files. It was therefore concluded that both data originators and data centres alike should assume that errors exist in all submitted data files until proven differently by a set of automated error-checking programs. It was also concluded that data visualization plots and statistical summaries are highly effective tools for detecting errors in data files. Metadata associated with the measurement data were documented in Quality Assurance Project Plans that were archived in the Data Centre with the DES data files.