Abstract
Abstract. Probabilistic predictions are becoming increasingly popular in hydrology. Equally important are methods to test such predictions, given the topical debate on uncertainty analysis in hydrology. Also in the special case of hydrological forecasting, there is still discussion about which scores to use for their evaluation. In this paper, we propose to use information theory as the central framework to evaluate predictions. From this perspective, we hope to shed some light on what verification scores measure and should measure. We start from the ''divergence score'', a relative entropy measure that was recently found to be an appropriate measure for forecast quality. An interpretation of a decomposition of this measure provides insight in additive relations between climatological uncertainty, correct information, wrong information and remaining uncertainty. When the score is applied to deterministic forecasts, it follows that these increase uncertainty to infinity. In practice, however, deterministic forecasts tend to be judged far more mildly and are widely used. We resolve this paradoxical result by proposing that deterministic forecasts either are implicitly probabilistic or are implicitly evaluated with an underlying decision problem or utility in mind. We further propose that calibration of models representing a hydrological system should be the based on information-theoretical scores, because this allows extracting all information from the observations and avoids learning from information that is not there. Calibration based on maximizing utility for society trains an implicit decision model rather than the forecasting system itself. This inevitably results in a loss or distortion of information in the data and more risk of overfitting, possibly leading to less valuable and informative forecasts. We also show this in an example. The final conclusion is that models should preferably be explicitly probabilistic and calibrated to maximize the information they provide.
Highlights
Over the last decades, probabilistic prediction has become increasingly important in the field of hydrology
Examples are the Hydrological Ensemble Prediction EXperiment (HEPEX), the European Flood Alert System (EFAS) and MAP D-PHASE, which is aimed at flood prediction in the Alps
By using a decomposition recently developed by Weijs et al (2010) in combination with some results from information theory, we provide insights into what evaluation scores measure and what, in our opinion, they should measure
Summary
Probabilistic prediction has become increasingly important in the field of hydrology. The development of methods for evaluating such forecasts, has not kept pace with the development of methods of generating them (Laio and Tamea, 2007; Brocker and Smith, 2007) This is an important problem, given the fact that science is required to make testable predictions and needs unambiguous methods for testing those predictions. Examples are the Hydrological Ensemble Prediction EXperiment (HEPEX), the European Flood Alert System (EFAS) and MAP D-PHASE, which is aimed at flood prediction in the Alps These initiatives seek to optimize hydrological (flood) forecasts by addressing the characterization of various uncertainties and by bringing together experience from the meteorological and hydrological communities and endusers. Weijs et al.: Information theory for evaluation of hydrological predictions that gives testable predictions of e.g. precipitation, temperature (the meteorological component) or streamflow, snowpack (the hydrological component) This is achieved by evaluating the forecasts. The most important insights are that deterministic forecasts are not testable without additional assumptions and that the purpose of a model should not influence the measure that is used for its calibration
Sign-in/Register to view highlights & summary 
Published Version (
Free)
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call