Investigating bias in impact observation sources and implications for impact-based forecast evaluation

Abstract

Impact-based forecasts and warnings (IbFWs) are used by meteorological agencies to forecast the likelihood and severity of impacts resulting from hydrometeorological hazards. Despite progress developing IbFW approaches the evaluation of IbFWs performance is challenging, partly due to the lack of impact observations. Impact data may provide the observations required to evaluate IbFWs, and publicly accessible impact data are available from a number of diverse sources. However, differences in the purpose, format, and method of collection across agencies has led to a fragmented and inconsistent picture of impacts to date. Furthermore, impact data can suffer from bias. To understand the implications of using impact data for IbF evaluation, we interrogate a range of publicly available impact data sources including structured databases, disaster reports, and dedicated web and news sites. Bias within publicly available impact data was assessed, and the implication of bias on IbFW evaluation considered. We find that each data source investigated has bias, although the type and magnitude of bias varies depending on the data source. Observed biases affect the coverage, severity, timing, geographic scale, and accuracy of impact observations. Furthermore, inconsistences in hazard terminology, poor adoption of unique event identifiers, and the inability to relate impact observations to specific hazardous events presents challenges when using impact data for IbFW evaluation. Despite these challenges, using a range of data sources together can provide a more holistic observation of the impacts of hydrometeorological events. Consolidating impact observations from a range of sources may produce the data required for IbFW evaluation.