Handling missing data in near real-time environmental monitoring: A system and a review of selected methods

Abstract

High-frequency water quality monitoring systems provide valuable measurements for predicting the trend of water quality, warning of abnormal activities or operating hydrological models. However, missing values are prevalent due to network miscommunication, device replacement or failure. Applying datasets with missing values can lead to biased results in statistical analysis or hydrological modelling work. We develop a cloud-based data processing system combining advanced algorithms to impute monitoring data in near real-time. The system provides high compatibility for supporting different water quality variables, imputation algorithms and extensive scalability to support numerous data streams. Based on the proposed approach, we review various imputation methods which can be applied to water quality data. Overall, this work provides a systematic design of a water quality data imputation system, explores the advantages and limitations of selected data imputation methods and analyses the imputation performance of two real-time water quality monitoring systems located in both the USA and Australia. The results provide practical guidelines for data imputation applications in water quality data.