Challenging problems of quality assurance and quality control (QA/QC) of meteorological time series data

B Faybishenko,C Varadharajan,G Pastorello,R Versteeg,D Agarwal,D Dwivedi

doi:10.1007/s00477-021-02106-w

Abstract

Representativeness and quality of collected meteorological data impact accuracy and precision of climate, hydrological, and biogeochemical analyses and predictions. We developed a comprehensive Quality Assurance (QA) and Quality Control (QC) statistical framework, consisting of three major phases: Phase I—Preliminary data exploration, i.e., processing of raw datasets, with the challenging problems of time formatting and combining datasets of different lengths and different time intervals; Phase II—QA of the datasets, including detecting and flagging of duplicates, outliers, and extreme data; and Phase III—the development of time series of a desired frequency, imputation of missing values, visualization and a final statistical summary. The paper includes two use cases based on the time series data collected at the Billy Barr meteorological station (East River Watershed, Colorado), and the Barro Colorado Island (BCI, Panama) meteorological station. The developed statistical framework is suitable for both real-time and post-data-collection QA/QC analysis of meteorological datasets.

Highlights

We developed a comprehensive Quality Assurance (QA) and Quality Control (QC) statistical framework, consisting of three major phases: Phase I—Preliminary data exploration, i.e., processing of raw datasets, with the challenging problems of time formatting and combining datasets of different lengths and different time intervals; Phase II—QA of the datasets, including detecting and flagging of duplicates, outliers, and extreme data; and Phase III—the development of time series of a desired frequency, imputation of missing values, visualization and a final statistical summary
The developed statistical framework is suitable for both real-time and post-data-collection QA/QC analysis of meteorological datasets
Motivation Quality Assurance (QA) and Quality Control (QC) procedures are commonly used to verify and control environmental monitoring activities to ensure the resulting data provide a representative evaluation of environmental conditions, which are used for ecohydrological modeling and model validation

Summary

Introduction

Motivation Quality Assurance (QA) and Quality Control (QC) procedures are commonly used to verify and control environmental monitoring activities to ensure the resulting data provide a representative evaluation of environmental conditions, which are used for ecohydrological modeling and model validation (van der Heijde and Elnawawy, 1992; QA Guide, 2013). Examples of meteorological data collected in real time by automated, streaming sensors are temperature, barometric pressure, solar radiation, rainfall, relative humidity, wind speed and wind direction, evapotranspiration, runoff, and soil moisture content. Collected time series data are commonly processed and structured in a unique way, distinct for each type of observations and instrumentation. Most modern data loggers faithfully record time series data, collected data are usually irregular, and subject to several types of errors: errors of commission, such as incorrect or inaccurate data entered, mistyped data, and malfunctioning of instrumentation, as well as errors of omission, because data or metadata are not properly recorded, for example, due to inadequate documentation, human errors, or anomalies in the field data collection.

Objectives

Methods

Findings

Conclusion