Hazardous Weather Testbed Research Articles

Using Machine Learning to Predict Convection-Allowing Ensemble Forecast Skill: Evaluation with the NSSL Warn-on-Forecast System

Abstract Forecasters routinely calibrate their confidence in model forecasts. Ensembles inherently estimate forecast confidence but are often underdispersive, and ensemble spread does not strongly correlate with ensemble-mean error. The misalignment between ensemble spread and skill motivates new methods for “forecasting forecast skill” so that forecasters can better utilize ensemble guidance. We have trained logistic regression and random forest models to predict the skill of composite reflectivity forecasts from the NSSL Warn-on-Forecast System (WoFS), a 3-km ensemble that generates rapidly updating forecast guidance for 0–6-h lead times. The forecast skill predictions are valid at 1-, 2-, or 3-h lead times within localized regions determined by the observed storm locations at analysis time. We use WoFS analysis and forecast output and NSSL Multi-Radar/Multi-Sensor composite reflectivity for 106 cases from the 2017 to 2021 NOAA Hazardous Weather Testbed Spring Forecasting Experiments. We frame the prediction task as a multiclassification problem, where the forecast skill labels are determined by averaging the extended fraction skill scores (eFSSs) for several reflectivity thresholds and verification neighborhoods and then converting to one of three classes based on where the average eFSS ranks within the entire dataset: POOR (bottom 20%), FAIR (middle 60%), or GOOD (top 20%). Initial machine learning (ML) models are trained on 323 predictors; reducing to 10 or 15 predictors in the final models only modestly reduces skill. The final models substantially outperform carefully developed persistence- and spread-based models and are reasonably explainable. The results suggest that ML can be a valuable tool for guiding user confidence in convection-allowing (and larger-scale) ensemble forecasts. Significance Statement Some numerical weather prediction (NWP) forecasts are more likely to verify than others. Forecasters often recognize situations where NWP output should be trusted more or less than usual, but objective methods for “forecasting forecast skill” are notably lacking for thunderstorm-scale models. Better estimates of forecast skill can benefit society through more accurate forecasts of high-impact weather. Machine learning (ML) provides a powerful framework for relating forecast skill to the characteristics of model forecasts and available observations over many previous cases. ML models can leverage these relationships to predict forecast skill for new cases in real time. We demonstrate the effectiveness of this approach to forecasting forecast skill using a cutting-edge thunderstorm prediction system and logistic regression and random forest models. Based on this success, we recommend the adoption of similar ML-based methods for other prediction models.

Exploring the Usefulness of Machine Learning Severe Weather Guidance in the Warn-on-Forecast System: Results from the 2022 NOAA Hazardous Weather Testbed Spring Forecasting Experiment

Abstract Artificial intelligence (AI) is gaining popularity for severe weather forecasting. Recently, the authors developed an AI system using machine learning (ML) to produce probabilistic guidance for severe weather hazards, including tornadoes, large hail, and severe winds, using the National Severe Storms Laboratory’s (NSSL) Warn-on-Forecast System (WoFS) as input. Known as WoFS-ML-Severe, it performed well in retrospective cases, but its operational usefulness had yet to be determined. To examine the potential usefulness of the ML guidance, we conducted a control and treatment (experimental) group experiment during the 2022 NOAA Hazardous Weather Testbed Spring Forecasting Experiment (HWT-SFE). The control group had full access to WoFS, while the experimental group had access to WoFS and ML products. Explainability graphics were also integrated into the WoFS web viewer. Both groups issued 1-h convective outlooks for each hazard. After issuing their forecasts, we surveyed participants on their confidence, the number of products viewed, and the usefulness of the ML guidance. We found the ML-based outlooks outperformed non-ML-based outlooks for multiple verification metrics for all three hazards and were rated subjectively higher by the participants. However, the difference in confidence between the two groups was not significant, and the experimental group self-reported viewing more products than the control group. Participants had mixed sentiments toward explainability products as it improved their understanding of the input/output relationships, but viewing them added to their workload. Although the experiment demonstrated the usefulness of ML guidance for severe weather forecasting, there are avenues to improve upon the ML guidance, and more training and exposure are needed to exploit its benefits fully. Significance Statement We developed an artificial intelligence (AI) system to predict tornadoes, large hail, and damaging straight-line winds. The AI system was leveraged in real time during the 2022 NOAA Hazardous Weather Testbed Spring Forecasting Experiment. This study reveals that forecasters using AI guidance produced more reliable and spatially accurate outlooks than those without. While AI and complementary explainability products did not reduce forecaster workload, both demonstrated great potential for improving severe weather forecasting. This research also highlights the importance of user feedback in refining AI tools for severe weather forecasting.

Making Social Science Actionable for the NWS: The Brief Vulnerability Overview Tool (BVOT)

Abstract This paper provides an introduction to a new tool that is designed to provide operationally useful vulnerability information to National Weather Service (NWS) Weather Forecasting Offices (WFOs). The Brief Vulnerability Overview Tool (BVOT) is a shapefile containing local known, spatially specific, and weather-hazard-related vulnerabilities in a format that is easily integrated into the existing forecasting, warning, and decision support responsibilities and tasks of NWS WFO meteorologists. The methods for gathering vulnerability data and then building a BVOT for a WFO leverage and strengthen the relationships that NWS WFOs already have with their local emergency managers (EMs) and core partners to work together to identify operationally useful, local vulnerability knowledge. The BVOT is populated with discrete, known vulnerabilities to provide NWS meteorologists spatial situational awareness of those people, places, and things of greatest concern to their core partners. Crucially, the BVOT is a subsample of all potential vulnerabilities; its primary purpose is to make meteorologists aware of those weather-hazard-specific vulnerabilities that, as we posed to them, “keep them awake at night.” Here, we describe the development of the BVOT as a social science–informed operational tool; how the BVOT methods have evolved and how it can be integrated into the culture of the NWS as a tool for building and maintaining relationships with partners; and how the BVOT is designed to be used and its impact on operational decision-making as observed in NOAA’s Hazardous Weather Testbed.

WoFS and the Wisdom of the Crowd: The Impact of the Warn-on-Forecast System on Hourly Forecasts during the 2021 NOAA Hazardous Weather Testbed Spring Forecasting Experiment

Abstract During the 2021 Spring Forecasting Experiment (SFE), the usefulness of the experimental Warn-on-Forecast System (WoFS) ensemble guidance was tested with the issuance of short-term probabilistic hazard forecasts. One group of participants used the WoFS guidance, while another group did not. Individual forecasts issued by two NWS participants in each group were evaluated alongside a consensus forecast from the remaining participants. Participant forecasts of tornadoes, hail, and wind at lead times of ∼2–3 h and valid at 2200–2300, 2300–0000, and 0000–0100 UTC were evaluated subjectively during the SFE by participants the day after issuance, and objectively after the SFE concluded. These forecasts exist between the watch and the warning time frame, where WoFS is anticipated to be particularly impactful. The hourly probabilistic forecasts were skillful according to objective metrics like the fractions skill score. While the tornado forecasts were more reliable than the other hazards, there was no clear indication of any one hazard scoring highest across all metrics. WoFS availability improved the hourly probabilistic forecasts as measured by the subjective ratings and several objective metrics, including increased POD and decreased FAR at high probability thresholds. Generally, expert forecasts performed better than consensus forecasts, though expert forecasts overforecasted. Finally, this work explored the appropriate construction of practically perfect fields used during subjective verification, which participants frequently found to be too small and precise. Using a Gaussian smoother with σ = 70 km is recommended to create hourly practically perfect fields in future experiments. Significance Statement This work explores the impact of cutting-edge numerical weather prediction ensemble guidance (the Warn-on-Forecast System) on severe thunderstorm hazard outlooks at watch-to-warning time scales, typically between 1 and 6 h of lead time. Real-time forecast products in this time frame are currently provided on an as-needed basis, and the transition to continuous probabilistic forecast products across scales requires targeted research. Results showed that hourly probabilistic participant forecasts were skillful subjectively and statistically, and that the experimental guidance improved the forecasts. These results are promising for the implementation and value of the Warn-on-Forecast System to provide improved hazard timing and location guidance within severe weather watches. Suggestions are made to aid future subjective evaluations of watch-to-warning-scale probabilistic forecasts.

Collaborative Exploration of Storm-Scale Probabilistic Guidance for NWS Forecast Operations

Abstract The operational utility of the NOAA National Severe Storm Laboratory’s storm-scale probabilistic Warn-on-Forecast System (WoFS) was examined across the watch-to-warning time frame in a virtual NOAA Hazardous Weather Testbed (HWT) experiment. Over four weeks, 16 NWS forecasters from local Weather Forecast Offices, the Storm Prediction Center, and the Weather Prediction Center participated in simulated forecasting tasks and focus groups. Bringing together multiple NWS entities to explore new guidance impacts on the broader forecast process is atypical of prior NOAA HWT experiments. This study therefore provides a framework for designing such a testbed experiment, including methodological and logistical considerations necessary to meet the needs of both local office and national center NWS participants. Furthermore, this study investigated two research questions: 1) How do forecasters envision WoFS guidance fitting into their existing forecast process? and 2) How could WoFS guidance be used most effectively across the current watch-to-warning forecast process? Content and thematic analyses were completed on flowcharts of operational workflows, real-time simulation interactions, and focus group activities and discussions. Participants reported numerous potential applications of WoFS, including improved coordination and consistency between local offices and national centers, enhanced hazard messaging, and improved operations planning. Challenges were also reported, including the knowledge and training required to incorporate WoFS guidance effectively and forecasters’ trust in new guidance and openness to change. The solutions identified to these challenges will take WoFS one step closer to transition, and in the meantime, improve the capabilities of WoFS for experimental use within the operational community. Significance Statement A first-of-its-kind experiment brought together forecasters from local weather forecast offices and national centers to examine the experimental Warn-on-Forecast System’s (WoFS’s) potential applications across watch-to-warning scales. This experiment demonstrated that WoFS can provide great benefit to forecasters, though a few challenges remain. Benefits provided by WoFS frequently overlap roles and responsibilities at local and national scales, suggesting the potential for enhanced cross-office collaboration. The challenges anticipated for WoFS operational use are far fewer than the benefits, and some solutions to these challenges are now being implemented. Finally, the mixed-methods experimental framework described herein also provides guidance for future collaborative experiments in testbed research that examine impacts of new technologies across NWS entities.

Biases and Skill of Four Two-Moment Bulk Microphysics Schemes in Convection-Allowing Forecasts for the 2018 Hazardous Weather Testbed Spring Forecasting Experiment Period

Abstract A proof-of-concept systematic evaluation of convective hazards is applied to short-term (1–6 h) forecasts using the Morrison, National Severe Storms Laboratory (NSSL), Predicted Particle Properties (P3), and Thompson two-moment microphysics schemes for the 2018 NOAA Hazardous Weather Testbed Spring Forecasting Experiment (HWT SFE) period (hereafter “MORR,” “NSSL,” “P3,” and “THOM” experiments, respectively). Four convective line cases are highlighted to elaborate on relative experiment biases/skill. Composite reflectivity and 1-h accumulated precipitation are examined to determine storm coverage/precipitation biases/skill utilizing point-based verification with a neighborhood. Simulated 1–6-km updraft helicity and observed 3–6-km azimuthal shear and MESH are examined to consider simulated rotation and hail core prediction with object-based scores. Over the full season, MORR displays little overall storm coverage bias relative to NSSL, P3, and THOM underprediction. The equitable threat score (ETS) and fractions skill score (FSS) of P3 are lower than the other experiments. P3 and THOM underpredict convective regions with intense reflectivity relative to MORR and NSSL overprediction. All experiments underpredict precipitation amounts. P3 light precipitation FSS is lower than other experiments. Rotation object verification exhibits sensitivity to microphysics experiments, as microphysics has an indirect influence on storm dynamics. While P3 has the largest hail object underprediction, all experiments grossly overpredict the number of hail objects in convective line cases despite forecast objects defined with the same product (MESH) and threshold as observations. The importance of microphysics ice parameterization and ongoing scheme updates highlight the need to apply this verification framework to optimal/updated schemes before optimizing ensemble design.

The 2021 Hazardous Weather Testbed Experimental Warning Program Radar Convective Applications Experiment: A Forecaster Evaluation of the Tornado Probability Algorithm and the New Mesocyclone Detection Algorithm

Abstract Developed as part of a larger effort by the National Weather Service (NWS) Radar Operations Center to modernize their suite of single-radar severe weather algorithms for the WSR-88D network, the Tornado Probability Algorithm (TORP) and the New Mesocyclone Detection Algorithm (NMDA) were evaluated by operational forecasters during the 2021 National Oceanic and Atmospheric Administration (NOAA) Hazardous Weather Testbed (HWT) Experimental Warning Program Radar Convective Applications experiment. Both TORP and NMDA leverage new products and advances in radar technology to create rotation-based objects that interrogate single-radar data, providing important summary and trend information that aids forecasters in issuing time-critical and potentially life-saving weather products. Utilizing virtual resources like Google Workspace and cloud instances on Amazon Web Services, 18 forecasters from the NOAA/NWS and the U.S. Air Force participated remotely over three weeks during the spring of 2021, providing valuable feedback on the efficacy of the algorithms and their display in an operational warning environment, serving as a critical step in the research-to-operations process for the development of TORP and NMDA. This article will discuss the details of the virtual HWT experiment and the results of each algorithm’s evaluation during the testbed. Significance Statement Before transitioning newly developed radar-based severe weather applications to forecasting operations, an experiment simulating the use of these tools by end users issuing severe weather warnings is helpful to identify both how they are best utilized and address any needed improvements to increase their operational readiness. Conducted in 2021, this study describes the forecaster evaluation of the single-radar Tornado Probability Algorithm (TORP) and the New Mesocyclone Detection Algorithm (NMDA) in one of the first completely virtual Hazardous Weather Testbed (HWT) experiments. Participants stated both TORP and NMDA offered marked improvement over the currently available algorithms by helping the operational forecaster build their confidence when issuing severe weather warnings and increasing their overall situational awareness of storms within their domain.

Verification and Model Configuration Sensitivity of Simulated ABI Radiance Forecasts With the FV3‐LAM Model

AbstractThis study evaluates simulated radiance forecasts from a series of controlled experiments consisting of FV3‐LAM forecasts with different configurations of model physics and vertical resolution. The forecasts were produced during the 2020 Hazardous Weather Testbed Spring Forecasting Experiments on the same forecast cases. The evaluation includes grid‐point, neighborhood‐based and object‐based verification. The experiments include forecasts that were identical except for the physics (EMC‐LAM vs. EMC‐LAMx), vertical resolution (EMC‐LAMx vs. NSSL‐LAM), or combined initial conditions, physics and vertical resolution (GSL‐LAM). It is found that the EMC‐LAM generally provided better simulated radiance forecasts than the other three configurations at most forecast lead times, due to its unique physics configuration. All configurations generally over‐forecasted high level clouds. EMC‐LAM reduced the over‐forecasting of high clouds, but also under‐forecasted the coverage of mid‐level clouds. In contrast, at early lead times the EMC‐LAM had relatively poor performance relative to the other forecasts. Furthermore, EMC‐LAM was an outlier in terms of the vertical structure of clouds. It is also found that the NSSL‐LAM consistently improved upon the EMC‐LAMx, which had fewer vertical levels than NSSL‐LAM. Compared to EMC‐LAMx, NSSL‐LAM had less cloud over‐forecasting bias, especially with small cloud objects, and less overall error. The differences between EMC‐LAMx and GSL‐LAM were generally much smaller than the differences between EMC‐LAMx and EMC‐LAM/NSSL‐LAM. Finally, it is found that a non‐linear bias correction conditioned on symmetric brightness temperature reduced the overall root‐mean‐square error by about a factor of 2 while improving the unrealistic vertical structure of clouds in the EMC‐LAM.

Valid Time Shifting for an Experimental RRFS Convection-Allowing EnVar Data Assimilation and Forecast System: Description and Systematic Evaluation in Real Time

Abstract This study describes a real-time implementation of valid time shifting (VTS) within the Gridpoint Statistical Interpolation–based ensemble-variational (EnVar) data assimilation system, developed at the Multi-Scale Data Assimilation and Predictability Laboratory. This system, featuring data assimilation of mesoscale conventional observations and storm-scale radar reflectivity observations and interfaced with the next-generation Finite Volume Cubed Sphere Dynamical Core limited-area model (FV3-LAM), was run in real-time during the 2021 Hazardous Weather Testbed Spring Forecast Experiment. The VTS method efficiently increases ensemble size by incorporating ensemble forecast output before and after the central analysis. Two configurations were examined to systematically evaluate VTS: a baseline 36-member system with hourly data assimilation (NOVTS), and an experiment testing VTS for the radar analysis step. Verification across 22 cases shows statistically significant benefits of VTS to increase ensemble spread and better fit first guesses to observations. Control member forecasts launched at 0000 UTC have consistently higher skill, lower bias, and higher reliability in VTS than in NOVTS throughout the 18-h forecast evaluation period, especially from severe cases often featuring upscale growth into mesoscale convective systems. Verification of updraft helicity-based ensemble surrogate severe probabilistic forecasts against observed storm reports shows higher skill of VTS when verifying on finer scales, with benefits to constraining higher probabilities over report locations and reducing probabilities over no-report locations. This study is a first step toward the next-generation Rapid Refresh Forecast System (RRFS), demonstrating the feasibility of such a real-time system and the potential benefits of VTS implementation.

Integrating NASA Aqua AIRS in a Real‐Time NUCAPS Science‐To‐Applications System to Support Severe Weather Forecasting

AbstractIn recent years, National Oceanic and Atmospheric Administration (NOAA) Unique Combined Atmospheric Processing System (NUCAPS) hyperspectral infrared satellite sounding retrievals derived from Joint Polar Satellite System (JPSS) polar‐orbiting satellites have been documented as observations that add value to weather forecasting applications. NUCAPS is currently the operational algorithm delivering JPSS satellite sounding retrievals to the NOAA National Weather Service (NWS) and is based on the heritage Atmospheric Infrared Sounder (AIRS) Science Team algorithm for processing vertical temperature, moisture, and trace gas retrievals. For the Special Collection on “Twenty Years of Observations from AIRS,” we highlight the methodology we implemented to develop a prototype science‐to‐applications system to enable real‐time processing of AIRS satellite sounding retrievals through the NUCAPS algorithm (i.e., NUCAPS‐Aqua) to support weather forecasting applications. The addition of NUCAPS‐Aqua to experimental real‐time pathways alongside operational JPSS NUCAPS soundings, facilitated assessment of NUCAPS‐Aqua at the 2022 Hazardous Weather Testbed (HWT) Spring Experiment. Development of NUCAPS‐Aqua described in this technical report includes preservation of microwave observations and calculation of a‐priori regression coefficients. Additionally, the real‐time processing and challenges with implementing a science‐to‐applications system are discussed. Two illustrative pre‐convective forecasting examples analyzed by NWS forecasters during the 2022 HWT Spring Experiment are highlighted to demonstrate the benefit of NUCAPS‐Aqua as (a) special afternoon soundings, (b) an additional observation to assess temporal trends using multiple satellites, and (c) a complement to observational and model analysis.

Just What Is “Good”? Musings on Hail Forecast Verification through Evaluation of FV3-HAILCAST Hail Forecasts

Abstract Hail forecasts produced by the CAM-HAILCAST pseudo-Lagrangian hail size forecasting model were evaluated during the 2019, 2020, and 2021 NOAA Hazardous Weather Testbed (HWT) Spring Forecasting Experiments (SFEs). As part of this evaluation, HWT SFE participants were polled about their definition of a “good” hail forecast. Participants were presented with two different verification methods conducted over three different spatiotemporal scales, and were then asked to subjectively evaluate the hail forecast as well as the different verification methods themselves. Results recommended use of multiple verification methods tailored to the type of forecast expected by the end-user interpreting and applying the forecast. The hail forecasts evaluated during this period included an implementation of CAM-HAILCAST in the Limited Area Model of the Unified Forecast System with the Finite Volume 3 (FV3) dynamical core. Evaluation of FV3-HAILCAST over both 1- and 24-h periods found continued improvement from 2019 to 2021. The improvement was largely a result of wide intervariability among FV3 ensemble members with different microphysics parameterizations in 2019 lessening significantly during 2020 and 2021. Overprediction throughout the diurnal cycle also lessened by 2021. A combination of both upscaling neighborhood verification and an object-based technique that only retained matched convective objects was necessary to understand the improvement, agreeing with the HWT SFE participants’ recommendations for multiple verification methods. Significance Statement “Good” forecasts of hail can be determined in multiple ways and must depend on both the performance of the guidance and the perspective of the end-user. This work looks at different verification strategies to capture the performance of the CAM-HAILCAST hail forecasting model across three years of the Spring Forecasting Experiment (SFE) in different parent models. Verification strategies were informed by SFE participant input via a survey. Skill variability among models decreased in SFE 2021 relative to prior SFEs. The FV3 model in 2021, compared to 2019, provided improved forecasts of both convective distribution and 38-mm (1.5 in.) hail size, as well as less overforecasting of convection from 1900 to 2300 UTC.

Results from a Pseudo-Real-Time Next-Generation 1-km Warn-on-Forecast System Prototype

Abstract Since 2017, the Warn-on-Forecast System (WoFS) has been tested and evaluated during the Hazardous Weather Testbed Spring Forecasting Experiment (SFE) and summer convective seasons. The system has shown promise in predicting high temporal and spatial specificity of individual evolving thunderstorms. However, this baseline version of the WoFS has a 3-km horizontal grid spacing and cannot resolve some convective processes. Efforts are under way to develop a WoFS prototype at a 1-km grid spacing (WoFS-1km) with the hope to improve forecast accuracy. This requires extensive changes to data assimilation specifications and observation processing parameters. A preliminary version of WoFS-1km nested within WoFS at 3 km (WoFS-3km) was developed, tested, and run during the 2021 SFE in pseudo–real time. Ten case studies were successfully completed and provided simulations of a variety of convective modes. The reflectivity and rotation storm objects from WoFS-1km are verified against both WoFS-3km and 1-km forecasts initialized from downscaled WoFS-3km analyses using both neighborhood- and object-based techniques. Neighborhood-based verification suggests WoFS-1km improves reflectivity bias but not spatial placement. The WoFS-1km object-based reflectivity forecast accuracy is higher in most cases, leading to a net improvement. Both the WoFS-1km and downscaled forecasts have ideal reflectivity object frequency biases while the WoFS-3km overpredicts the number of reflectivity objects. The rotation object verification is ambiguous as many cases are negatively impacted by 1-km data assimilation. This initial evaluation of a WoFS-1km prototype is a solid foundation for further development and future testing. Significance Statement This study investigates the impacts of performing data assimilation directly on a 1-km WoFS model grid. Most previous studies have only initialized 1-km WoFS forecasts from coarser analyses. The results demonstrate some improvements to reflectivity forecasts through data assimilation on a 1-km model grid although finer resolution data assimilation did not improve rotation forecasts.

Scale-Dependent Verification of the OU MAP Convection Allowing Ensemble Initialized with Multi-Scale and Large-Scale Perturbations during the 2019 NOAA Hazardous Weather Testbed Spring Forecasting Experiment

Given the large range of resolvable space and time scales in large-domain convection-allowing for ensemble forecasts, there is a need to better understand optimal initial-condition perturbation strategies to sample the forecast uncertainty across these space and time scales. This study investigates two initial-condition perturbation strategies for CONUS-domain ensemble forecasts that extend into the two-day forecast lead time using traditional and object-based verification methods. Initial conditions are perturbed either by downscaling perturbations from a coarser resolution ensemble (i.e., LARGE) or by adopting the analysis perturbations from a convective-scale, EnKF system (i.e., MULTI). It was found that MULTI had more ensemble spread than LARGE across all scales initially, while LARGE’s perturbation energy surpassed that of MULTI after 3 h and continued to maintain a surplus over MULTI for the rest of the 36h forecast period. Impacts on forecast bias were mixed, depending on the forecast lead time and forecast threshold. However, MULTI was found to be significantly more skillful than LARGE at early forecast hours for the meso-gamma and meso-beta scales (1–9h), which is a result of a larger and better-sampled ensemble spread at these scales. Despite having a smaller ensemble spread, MULTI was also significantly more skillful than LARGE on the meso-alpha scale during the 20–24h period due to a better spread-skill relation. MULTI’s performance on the meso-alpha scale was slightly worse than LARGE’s performance during the 6–12h period, as LARGE’s ensemble spread surpassed that of MULTI. The advantages of each method for different forecast aspects suggest that the optimal perturbation strategy may require a combination of both the MULTI and LARGE techniques for perturbing initial conditions in a large-domain, convection-allowing ensemble.

Model Configuration versus Driving Model: Influences on Next-Day Regional Convection-Allowing Model Forecasts during a Real-Time Experiment

Abstract As part of NOAA’s Hazardous Weather Testbed Spring Forecasting Experiment (SFE) in 2020, an international collaboration yielded a set of real-time convection-allowing model (CAM) forecasts over the contiguous United States in which the model configurations and initial/boundary conditions were varied in a controlled manner. Three model configurations were employed, among which the Finite Volume Cubed-Sphere (FV3), Unified Model (UM), and Advanced Research version of the Weather Research and Forecasting (WRF-ARW) Model dynamical cores were represented. Two runs were produced for each configuration: one driven by NOAA’s Global Forecast System for initial and boundary conditions, and the other driven by the Met Office’s operational global UM. For 32 cases during SFE2020, these runs were initialized at 0000 UTC and integrated for 36 h. Objective verification of model fields relevant to convective forecasting illuminates differences in the influence of configuration versus driving model pertinent to the ongoing problem of optimizing spread and skill in CAM ensembles. The UM and WRF configurations tend to outperform FV3 for forecasts of precipitation, thermodynamics, and simulated radar reflectivity; using a driving model with the native CAM core also tends to produce better skill in aggregate. Reflectivity and thermodynamic forecasts were found to cluster more by configuration than by driving model at lead times greater than 18 h. The two UM configuration experiments had notably similar solutions that, despite competitive aggregate skill, had large errors in the diurnal convective cycle.

The Creation of a Research Television Studio to Test Probabilistic Hazard Information with Broadcast Meteorologists in NOAA’s Hazardous Weather Testbed

Abstract Broadcast meteorologists play an essential role in communicating severe weather information from the National Weather Service to the public. Because of their importance, researchers incorporated broadcast meteorologists in the development of probabilistic hazard information (PHI) in NOAA’s Hazardous Weather Testbed. As part of Forecasting a Continuum of Environmental Threats (FACETs), PHI is meant to bring additional context to severe weather warnings through the inclusion of probability information. Since this information represents a shift in the current paradigm of solely deterministic NWS warnings, understanding end user needs is paramount to create usable and accessible products that result in their intended outcome to serve the public. This paper outlines the establishment of “K-Probabilistic Hazard Information Television” (KPHI-TV), a research infrastructure under the Hazardous Weather Testbed created to study broadcast meteorologists and PHI. A description of the design of KPHI-TV and methods used by researchers are presented, including displaced real-time cases and semistructured interviews. Researchers completed an analysis of the 2018 experiment, using a quantitative analysis of television coverage decisions with PHI, and a thematic analysis of semistructured interviews. Results indicate that no clear probabilistic decision thresholds for PHI emerged among the participants. Other themes arose, including the relationship between PHI and the warning polygon, and communication challenges. Overall, broadcast participants preferred a system that includes PHI over the warning polygon alone, but raised other concerns, suggesting iterative research in the design and implementation of PHI should continue. Significance Statement Broadcast meteorologists are the primary source of severe weather warning information for the U.S. public. As a result, researchers at NOAA’s National Severe Storms Laboratory and the Cooperative Institute for Mesoscale Meteorological Studies developed a mock television studio to allow broadcast meteorologists to use and communicate experimental products “on air” as part of the research-and-development process. Feedback provided by broadcasters is incorporated into products through an iterative process. Since 2016, 18 broadcasters have tested probabilistic hazard information at the warning time scale (0–1 h) for severe wind and hail, tornadoes, and lightning.

Exploring the Watch-to-Warning Space: Experimental Outlook Performance during the 2019 Spring Forecasting Experiment in NOAA’s Hazardous Weather Testbed

Abstract During the 2019 Spring Forecasting Experiment in NOAA’s Hazardous Weather Testbed, two NWS forecasters issued experimental probabilistic forecasts of hail, tornadoes, and severe convective wind using NSSL’s Warn-on-Forecast System (WoFS). The aim was to explore forecast skill in the time frame between severe convective watches and severe convective warnings during the peak of the spring convective season. Hourly forecasts issued during 2100–0000 UTC, valid from 0100 to 0200 UTC demonstrate how forecasts change with decreasing lead time. Across all 13 cases in this study, the descriptive outlook statistics (e.g., mean outlook area, number of contours) change slightly and the measures of outlook skill (e.g., fractions skill score, reliability) improve incrementally with decreasing lead time. WoFS updraft helicity (UH) probabilities also improve slightly and less consistently with decreasing lead time, though both the WoFS and the forecasters generated skillful forecasts throughout. Larger skill differences with lead time emerge on a case-by-case basis, illustrating cases where forecasters consistently improved upon WoFS guidance, cases where the guidance and the forecasters recognized small-scale features as lead time decreased, and cases where the forecasters issued small areas of high probabilities using guidance and observations. While forecasts generally “honed in” on the reports with slightly smaller contours and higher probabilities, increased confidence could include higher certainty that severe weather would not occur (e.g., lower probabilities). Long-range (1–5 h) WoFS UH probabilities were skillful, and where the guidance erred, forecasters could adjust for those errors and increase their forecasts’ skill as lead time decreased. Significance Statement Forecasts are often assumed to improve as an event approaches and uncertainties resolve. This work examines the evolution of experimental forecasts valid over one hour with decreasing lead time issued using the Warn-on-Forecast System (WoFS). Because of its rapidly updating ensemble data assimilation, WoFS can help forecasters understand how thunderstorm hazards may evolve in the next 0–6 h. We found slight improvements in forecast and WoFS performance as a function of lead time over the full experiment; the first forecasts issued and the initial WoFS guidance performed well at long lead times, and good performance continued as the event approached. However, individual cases varied and forecasters frequently combined raw model output with observed mesoscale features to provide skillful small-scale forecasts.

Exploring the Usefulness of Downscaling Free Forecasts from the Warn-on-Forecast System

Abstract The National Severe Storms Laboratory (NSSL) Warn-on-Forecast System (WoFS) is an experimental real-time rapidly updating convection-allowing ensemble that provides probabilistic short-term thunderstorm forecasts. This study evaluates the impacts of reducing the forecast model horizontal grid spacing Δx from 3 to 1.5 km on the WoFS deterministic and probabilistic forecast skill, using 11 case days selected from the 2020 NOAA Hazardous Weather Testbed (HWT) Spring Forecasting Experiment (SFE). Verification methods include (i) subjective forecaster impressions; (ii) a deterministic object-based technique that identifies forecast reflectivity and rotation track storm objects as contiguous local maxima in the composite reflectivity and updraft helicity fields, respectively, and matches them to observed storm objects; and (iii) a recently developed algorithm that matches observed mesocyclones to mesocyclone probability swath objects constructed from the full ensemble of rotation track objects. Reducing Δx fails to systematically improve deterministic skill in forecasting reflectivity object occurrence, as measured by critical success index (CSIDET), a metric that incorporates both probability of detection (PODDET) and false alarm ratio (FARDET). However, compared to the Δx = 3 km configuration, the Δx = 1.5 km WoFS shows improved midlevel mesocyclone detection, as evidenced by its statistically significant (i) higher CSIDET for deterministic midlevel rotation track objects and (ii) higher normalized area under the performance diagram curve (NAUPDC) score for probability swath objects. Comparison between Δx = 3 km and Δx = 1.5 km reflectivity object properties reveals that the latter have 30% stronger mean updraft speeds, 17% stronger median 80-m winds, 67% larger median hail diameter, and 28% higher median near-storm-maximum 0–3-km storm-relative helicity.

The Experimental Warning Program of NOAA’s Hazardous Weather Testbed

Abstract NOAA’s Hazardous Weather Testbed (HWT) is a physical space and research framework to foster collaboration and evaluate emerging tools, technology, and products for NWS operations. The HWT’s Experimental Warning Program (EWP) focuses on research, technology, and communication that may improve severe and hazardous weather warnings and societal response. The EWP was established with three fundamental hypotheses: 1) collaboration with operational meteorologists increases the speed of the transition process and rate of adoption of beneficial applications and technology, 2) the transition of knowledge between research and operations benefits both the research and operational communities, and 3) including end users in experiments generates outcomes that are more reliable and useful for society. The EWP is designed to mimic the operations of any NWS Forecast Office, providing the opportunity for experiments to leverage live and archived severe weather activity anywhere in the United States. During the first decade of activity in the EWP, 15 experiments covered topics including new radar and satellite applications, storm-scale numerical models and data assimilation, total lightning use in severe weather forecasting, and multiple social science and end-user topics. The experiments range from exploratory and conceptual research to more controlled experimental design to establish statistical patterns and causal relationships. The EWP brought more than 400 NWS forecasters, 60 emergency managers, and 30 broadcast meteorologists to the HWT to participate in live demonstrations, archive events, and data-denial experiments influencing today’s operational warning environment and shaping the future of warning research, technology, and communication for years to come.

Assessment of Storm‐Scale Real Time Assimilation of GOES‐16 GLM Lightning‐Derived Water Vapor Mass on Short Term Precipitation Forecasts During the 2020 Spring Forecast Experiment

AbstractThis study assesses the impact of assimilating pseudo‐observations for water vapor mass derived from the Geostationary Operational Environmental Satellites GOES‐16 lightning data on short‐term quantitative precipitation forecasts (QPFs) over the contiguous United States (CONUS), with an emphasis given to regions characterized by an overall poor radar coverage. The GOES‐16/17 provide nearly uniform and high temporal resolution total lightning observations over most of the Americas. To leverage this information for convective scale weather forecasts, a three‐dimensional variational data assimilation (DA) package developed by the National Severe Storm Laboratory was employed to assimilate these data. To mimic operational settings, the Weather Research and Forecasting Model configuration used in NOAA Global Systems Laboratory's High‐Resolution Rapid Refresh Model version 4 was utilized. During the 2020 NOAA Hazardous Weather Testbed Spring Forecasting Experiment, four experiments were run in real time during a 29‐day period over CONUS and surrounding territories to assess the added value of GOES‐16 lightning over conventional radar data. Overall, the lightning DA (LDA) showed benefit in improving QPFs up to 6 hr, with the best improvements seen during the first 3 hr. Owing to notably larger lightning activity over the eastern CONUS, the most noticeable impacts from the LDA were seen there. Case‐by‐case analysis revealed that positive impacts from the LDA were seen over areas characterized by both good (eastern two thirds of CONUS) and poor radar coverage (western one third of CONUS). Highlighting the inherent difficulties in developing an optimal observation operator based on moisture for LDA applications, the GLM‐based DA experiments systematically overestimated rainfall over the eastern two thirds of CONUS while underestimating it over the western one third of CONUS.

Evaluation of an experimental Warn‐on‐Forecast 3DVAR analysis and forecast system on quasi‐real‐time short‐term forecasts of high‐impact weather events

AbstractThe purpose of this study is to demonstrate the capability of an experimental, weather‐adaptive, high‐resolution, deterministic Warn‐on‐Forecast (WoF) analysis and forecast system (WoF3DVAR‐AFS) for predicting high‐impact severe weather events that occurred during the Hazardous Weather Testbed 2019 Spring Forecast Experiments. WoF3DVAR‐AFS uses a three‐dimensional variational (3DVAR) method as its core data assimilation system and the Advanced Research Version of the Weather Research and Forecasting (WRF‐ARW) model as its forward model. Surface measurements provided in meteorological aviation reports and the Oklahoma Mesonet, Doppler radar data, and spaceborne total lightning observations provided by the Geostationary Lightning Mapper are assimilated at 15‐min frequency over a target domain determined by the “Day 1” Convective Outlook product from the Storm Prediction Center. The chief goal of this system is to complement probabilistic forecasts generated by ensemble analysis and forecast systems, such as the experimental Warn‐on‐Forecast System (WoFS) with a higher‐resolution deterministic member to aid forecasters' decision‐making. We performed both qualitative and quantitative evaluations on 0–6 hr forecasts launched hourly from 1900 to 0300 UTC the next day for each of the 12 cases. Aggregated subjective forecast evaluation metrics from each individual case, as well as detailed comparison against available verification datasets, suggest that the forecasts are generally skillful in terms of composite reflectivity fields, quantitative precipitation forecasts, and the strength and location of rotation tracks and damaging winds. This study presents initial efforts to assess the performance of WoF3DVAR‐AFS and provides possible directions for further improvements, including the development of a weather‐adaptive, dual‐resolution analysis and forecast system hybrid with an ensemble system, such as the experimental Warn‐on‐Forecast system.