Exploring the Usefulness of Machine Learning Severe Weather Guidance in the Warn-on-Forecast System: Results from the 2022 NOAA Hazardous Weather Testbed Spring Forecasting Experiment

Abstract

Abstract Artificial intelligence (AI) is gaining popularity for severe weather forecasting. Recently, the authors developed an AI system using machine learning (ML) to produce probabilistic guidance for severe weather hazards, including tornadoes, large hail, and severe winds, using the National Severe Storms Laboratory’s (NSSL) Warn-on-Forecast System as input (WoFS). Known as WoFS-ML-Severe, it performed well in retrospective cases, but its operational usefulness had yet to be determined. To examine the potential usefulness of the ML guidance, we conducted a control and treatment (experimental) group experiment during the 2022 NOAA Hazardous Weather Testbed Spring Forecasting Experiment (HWT-SFE). The control group had full access to WoFS, while the experimental group had access to WoFS and ML products. Explainability graphics were also integrated into the WoFS web viewer. Both groups issued 1-hr convective outlooks for each hazard. After issuing their forecasts, we surveyed participants on their confidence, the number of products viewed, and the usefulness of the ML guidance. We found the ML-based outlooks outperformed non-ML-based outlooks for multiple verification metrics for all three hazards and were rated subjectively higher by the participants. However, the difference in confidence between the two groups was not significant, and the experimental group self-reported viewing more products than the control group. Participants had mixed sentiments towards explainability products as it improved their understanding of the input/output relationships, but viewing them added to their workload. Although the experiment demonstrated the usefulness of ML guidance for severe weather forecasting, there are avenues to improve upon the ML guidance, and more training and exposure are needed to exploit its benefits fully.