An adaptive Markov Chain approach for probabilistic forecasting of categorical events

Abstract

Abstract Categorical probabilistic prediction is widely used for terrestrial and space weather forecasting as well as for other environmental forecasts. One example is a warning system for geomagnetic disturbances caused by space weather, which are often classified on a 10-level scale. The simplest approach assumes that the transition probabilities are stationary in time – the Homogeneous Markov Chain (HMC). We extend this approach by developing a flexible Non-Homogeneous Markov Chain (NHMC) model using Bayesian non-parametric estimation to describe the time-varying transition probabilities. The transition probabilities are updated using a modified Bayes’ rule that gradually forgets transitions in the distant past, with a tunable memory parameter. The approaches were tested by making daily geomagnetic state forecasts at lead times of 1-4 days and verified over the period 2000-2019 using the Rank Probability Score (RPS). Both HMC and NHMC models were found to be skilful at all lead times when compared with climatological forecasts. The NHMC forecasts with an optimal memory parameter of ~100 days were found to be substantially more skilful than the HMC forecasts, with an RPS skill for the NHMC of 10.5% and 5.6% for lead times of 1 and 4 days ahead, respectively. The NHMC is thus a viable alternative approach for forecasting geomagnetic disturbances, and could provide a new benchmark for producing operational forecasts. The approach is generic and applicable to other forecasts including discrete weather regimes or hydrological conditions, e.g. wet and dry days.