Validation of satellite hydrological products: which goodness-of-fit to use?

Abstract

Satellite products of hydrological variables are essential to understanding the spatial and temporal variability of the earth's system of systems, for example, the hydrosphere and its intricate interactions with the biosphere and the atmosphere. Hydrological products from multiple satellites are regularly harmonised to produce long-term synoptic climate data records. The fidelity of these satellite-based climate records forms the backbone to quantify and analyse the variability of the water cycle and the effect of climate change over extended temporal and spatial scales. This fidelity is assessed with statistical metrics that measure the goodness of fit (GoF) between satellite products and in-situ measurements. Commonly used GoF metrics include: the slope and intercept of type-II regression, determination coefficient R2, and difference metrics like bias, root means squared differences, mean absolute differences (MAD) and their relative measures. These metrics do not need to be in harmony, for example, a high R2 value is not necessarily associated with a close-to-unity slope, or a low bias is not ineludibly translated to a low MAD. Presenting these metrics in a table for comparing various retrieval models makes it even more challenging to draw clear conclusions. In part, this confusion could be mitigated by using statistical charts, for example, Taylor or radar diagrams. These diagrams offer the capability to graphically summarise how closely satellite products match the measurements. Nonetheless, there is no unique measure that can be used to describe the GoF. In this essay, we develop a universal methodology with the capability of providing the scientific community with a quantitative and holistic measure of GoF of satellite products. The method ingests statistical validation metrics commonly employed in hydrology or any specific discipline of geoscience, transforms them into unity scalars with the same direction (0 is low and one is high accuracy) and projects them into a unity circle.  The resulting area is then calculated and normalised to the maximum expected area for a percentage GoF measure. As the projection is equiangular, each unique sequence of the employed metrics will give a different answer. With permutation, the GoF values are calculated from all possible sequences. The maximum possible accuracy and the largest probable uncertainty are subsequently derived from the resulting population. This procedure results in a unique GoF that integrates all used validation metrics and provides a collective measure of accuracy. Although it was developed for satellite-derived hydrological products, the proposed method can be applied to any statistical metrics used to measure the goodness of fit between modelled and measured biophysical variables.