Exploring the Usefulness of Machine Learning Severe Weather Guidance in the Warn-on-Forecast System: Results from the 2022 NOAA Hazardous Weather Testbed Spring Forecasting Experiment

Abstract

Abstract Artificial intelligence (AI) is gaining popularity for severe weather forecasting. Recently, the authors developed an AI system using machine learning (ML) to produce probabilistic guidance for severe weather hazards, including tornadoes, large hail, and severe winds, using the National Severe Storms Laboratory’s (NSSL) Warn-on-Forecast System (WoFS) as input. Known as WoFS-ML-Severe, it performed well in retrospective cases, but its operational usefulness had yet to be determined. To examine the potential usefulness of the ML guidance, we conducted a control and treatment (experimental) group experiment during the 2022 NOAA Hazardous Weather Testbed Spring Forecasting Experiment (HWT-SFE). The control group had full access to WoFS, while the experimental group had access to WoFS and ML products. Explainability graphics were also integrated into the WoFS web viewer. Both groups issued 1-h convective outlooks for each hazard. After issuing their forecasts, we surveyed participants on their confidence, the number of products viewed, and the usefulness of the ML guidance. We found the ML-based outlooks outperformed non-ML-based outlooks for multiple verification metrics for all three hazards and were rated subjectively higher by the participants. However, the difference in confidence between the two groups was not significant, and the experimental group self-reported viewing more products than the control group. Participants had mixed sentiments toward explainability products as it improved their understanding of the input/output relationships, but viewing them added to their workload. Although the experiment demonstrated the usefulness of ML guidance for severe weather forecasting, there are avenues to improve upon the ML guidance, and more training and exposure are needed to exploit its benefits fully. Significance Statement We developed an artificial intelligence (AI) system to predict tornadoes, large hail, and damaging straight-line winds. The AI system was leveraged in real time during the 2022 NOAA Hazardous Weather Testbed Spring Forecasting Experiment. This study reveals that forecasters using AI guidance produced more reliable and spatially accurate outlooks than those without. While AI and complementary explainability products did not reduce forecaster workload, both demonstrated great potential for improving severe weather forecasting. This research also highlights the importance of user feedback in refining AI tools for severe weather forecasting.