SaQC: Empowering Hydrological Data Integrity through Automated Quality Control  

Abstract

Global and regional hydrological databases, as well as domain-agnostic repositories, play a crucial role in advancing scientific research and decision-making processes. With new and existing data infrastructures such as TERENO and eLTER, as well as governmental monitoring initiatives, efforts to enhance the size, capabilities, and accessibility of these services are underway. However, a key challenge persists across large-scale data collections - the need for rigorous harmonization of diverse data from various sources.  This challenge extends beyond obvious considerations like numerical precision and date formats, encompassing more nuanced aspects such as data quality and its representation. Hydrological time series data, often acquired from remote sensors and monitoring stations, are susceptible to errors arising from sensor malfunctions, anomalies, and environmental fluctuations. Unchecked, these inaccuracies can lead to erroneous results and compromise decision-making processes.  Addressing this critical issue, the System for Automated Quality Control - SaQC emerges as a pioneering solution, offering a comprehensive tool/framework for automated and customizable quality control and processing of time series data. SaQC empowers researchers and practitioners in the hydrological sciences, providing a convenient and efficient means to identify and rectify data anomalies. In addition to a large body of built-in routines and algorithms, the framework's extensibility allows users to implement custom quality check routines and schemes, tailoring the quality control process to specific research objectives and the evolving needs of data services.  This presentation delves into the core principles of SaQC, showcasing its flexibility in handling diverse data types and adapting to various hydrological monitoring scenarios. Through real-world examples of fully automatized quality control and data processing workflows, we highlight the benefits of SaQC in enhancing data integrity, reducing manual intervention, and expediting the analysis pipeline. SaQC not only identifies anomalies but also provides a systematic and transparent approach to data quality assurance, contributing to the overall reliability of hydrological datasets.    Lennart Schmidt, David Schäfer, Juliane Geller, Peter Lünenschloss, Bert Palm, Karsten Rinke, Corinna Rebmann, Michael Rode, Jan Bumberger, System for automated Quality Control (SaQC) to enable traceable and reproducible data streams in environmental science, Environmental Modelling & Software, Volume 169, 2023, 105809, ISSN 1364-8152, https://doi.org/10.1016/j.envsoft.2023.105809.