Potential of Deep Learning based quality control methods for soil moisture time series in an operational data service

Abstract

Soil moisture and its measurements are important in various fields and applications, from agriculture and hydrology to climate modelling, ecology and ecosystem health management. The monitoring of soil moisture gained widespread recognition in the early 2000s as an integral component of the hydrological and meteorological observation systems. This momentum was accelerated with the establishment of several soil moisture monitoring networks. The collection of data from different networks with a diversity of sensors and data formats, harmonization, quality control, archiving of in situ soil moisture data and ensuring the free accessibility of this data for end-users underline the motivation behind the foundation of the International Soil Moisture Network (ISMN) in 2009. All in situ data sourced from the different providers undergo two quality checks. First, a visual inspection of the data excluding near real-time data and second, a rule-based automatic quality control procedure before inclusion in the ISMN database to ensure high quality research-ready soil moisture data for end-user. Thirteen different plausibility checks are applied to every singular hourly observation, which is flagged then as dubious if one of these checks fail, otherwise as “good”. These plausibility checks can be categorized into: i) a geophysical range verification, detecting the exceedance of certain thresholds (e.g., soil moisture values below 0 Vol.-%), ii) geophysical consistency methods, taking either ancillary in situ data if available or NASA’s GLDAS Noah data into account. An example is the flagging of soil moisture where soil temperature is negative). And iii) spectrum-based approaches, using the first and second derivatives of the entire soil moisture timeseries to detect dubious soil moisture patterns (i.e., spikes, breaks, and plateaus). Publications in recent years point to the great potential of Deep Learning (DL) based methods for identifying anomalies in time series data. In this study, the potential of Long Short-Term Memory (LSTM) and Transformer models for anomaly detection in soil moisture time series is being investigated. Therefore, randomly selected time series from the ISMN are manually (visually) quality flagged (labelled). In order to be able to label these data we developed a guidance how to visually quality control in situ soil moisture data. Different Deep Learning methods in combination with varying external data sets (e.g. precipitation time series) are validated against the manually labelled data and compared to the previously implemented flagging method. The method will be further developed and evaluated for its use in ISMN operations. The incorporation of additional flagging information, especially when enhanced by Deep Learning methods, is anticipated to lead to a better usability of soil moisture data, as well as promoting a more robust quality control by the ISMN for its users in the future.