NOAA Hazardous Weather Testbed Spring Research Articles

Using Machine Learning to Predict Convection-Allowing Ensemble Forecast Skill: Evaluation with the NSSL Warn-on-Forecast System

Abstract Forecasters routinely calibrate their confidence in model forecasts. Ensembles inherently estimate forecast confidence but are often underdispersive, and ensemble spread does not strongly correlate with ensemble-mean error. The misalignment between ensemble spread and skill motivates new methods for “forecasting forecast skill” so that forecasters can better utilize ensemble guidance. We have trained logistic regression and random forest models to predict the skill of composite reflectivity forecasts from the NSSL Warn-on-Forecast System (WoFS), a 3-km ensemble that generates rapidly updating forecast guidance for 0–6-h lead times. The forecast skill predictions are valid at 1-, 2-, or 3-h lead times within localized regions determined by the observed storm locations at analysis time. We use WoFS analysis and forecast output and NSSL Multi-Radar/Multi-Sensor composite reflectivity for 106 cases from the 2017 to 2021 NOAA Hazardous Weather Testbed Spring Forecasting Experiments. We frame the prediction task as a multiclassification problem, where the forecast skill labels are determined by averaging the extended fraction skill scores (eFSSs) for several reflectivity thresholds and verification neighborhoods and then converting to one of three classes based on where the average eFSS ranks within the entire dataset: POOR (bottom 20%), FAIR (middle 60%), or GOOD (top 20%). Initial machine learning (ML) models are trained on 323 predictors; reducing to 10 or 15 predictors in the final models only modestly reduces skill. The final models substantially outperform carefully developed persistence- and spread-based models and are reasonably explainable. The results suggest that ML can be a valuable tool for guiding user confidence in convection-allowing (and larger-scale) ensemble forecasts. Significance Statement Some numerical weather prediction (NWP) forecasts are more likely to verify than others. Forecasters often recognize situations where NWP output should be trusted more or less than usual, but objective methods for “forecasting forecast skill” are notably lacking for thunderstorm-scale models. Better estimates of forecast skill can benefit society through more accurate forecasts of high-impact weather. Machine learning (ML) provides a powerful framework for relating forecast skill to the characteristics of model forecasts and available observations over many previous cases. ML models can leverage these relationships to predict forecast skill for new cases in real time. We demonstrate the effectiveness of this approach to forecasting forecast skill using a cutting-edge thunderstorm prediction system and logistic regression and random forest models. Based on this success, we recommend the adoption of similar ML-based methods for other prediction models.

Exploring the Usefulness of Machine Learning Severe Weather Guidance in the Warn-on-Forecast System: Results from the 2022 NOAA Hazardous Weather Testbed Spring Forecasting Experiment

Abstract Artificial intelligence (AI) is gaining popularity for severe weather forecasting. Recently, the authors developed an AI system using machine learning (ML) to produce probabilistic guidance for severe weather hazards, including tornadoes, large hail, and severe winds, using the National Severe Storms Laboratory’s (NSSL) Warn-on-Forecast System (WoFS) as input. Known as WoFS-ML-Severe, it performed well in retrospective cases, but its operational usefulness had yet to be determined. To examine the potential usefulness of the ML guidance, we conducted a control and treatment (experimental) group experiment during the 2022 NOAA Hazardous Weather Testbed Spring Forecasting Experiment (HWT-SFE). The control group had full access to WoFS, while the experimental group had access to WoFS and ML products. Explainability graphics were also integrated into the WoFS web viewer. Both groups issued 1-h convective outlooks for each hazard. After issuing their forecasts, we surveyed participants on their confidence, the number of products viewed, and the usefulness of the ML guidance. We found the ML-based outlooks outperformed non-ML-based outlooks for multiple verification metrics for all three hazards and were rated subjectively higher by the participants. However, the difference in confidence between the two groups was not significant, and the experimental group self-reported viewing more products than the control group. Participants had mixed sentiments toward explainability products as it improved their understanding of the input/output relationships, but viewing them added to their workload. Although the experiment demonstrated the usefulness of ML guidance for severe weather forecasting, there are avenues to improve upon the ML guidance, and more training and exposure are needed to exploit its benefits fully. Significance Statement We developed an artificial intelligence (AI) system to predict tornadoes, large hail, and damaging straight-line winds. The AI system was leveraged in real time during the 2022 NOAA Hazardous Weather Testbed Spring Forecasting Experiment. This study reveals that forecasters using AI guidance produced more reliable and spatially accurate outlooks than those without. While AI and complementary explainability products did not reduce forecaster workload, both demonstrated great potential for improving severe weather forecasting. This research also highlights the importance of user feedback in refining AI tools for severe weather forecasting.

WoFS and the Wisdom of the Crowd: The Impact of the Warn-on-Forecast System on Hourly Forecasts during the 2021 NOAA Hazardous Weather Testbed Spring Forecasting Experiment

Abstract During the 2021 Spring Forecasting Experiment (SFE), the usefulness of the experimental Warn-on-Forecast System (WoFS) ensemble guidance was tested with the issuance of short-term probabilistic hazard forecasts. One group of participants used the WoFS guidance, while another group did not. Individual forecasts issued by two NWS participants in each group were evaluated alongside a consensus forecast from the remaining participants. Participant forecasts of tornadoes, hail, and wind at lead times of ∼2–3 h and valid at 2200–2300, 2300–0000, and 0000–0100 UTC were evaluated subjectively during the SFE by participants the day after issuance, and objectively after the SFE concluded. These forecasts exist between the watch and the warning time frame, where WoFS is anticipated to be particularly impactful. The hourly probabilistic forecasts were skillful according to objective metrics like the fractions skill score. While the tornado forecasts were more reliable than the other hazards, there was no clear indication of any one hazard scoring highest across all metrics. WoFS availability improved the hourly probabilistic forecasts as measured by the subjective ratings and several objective metrics, including increased POD and decreased FAR at high probability thresholds. Generally, expert forecasts performed better than consensus forecasts, though expert forecasts overforecasted. Finally, this work explored the appropriate construction of practically perfect fields used during subjective verification, which participants frequently found to be too small and precise. Using a Gaussian smoother with σ = 70 km is recommended to create hourly practically perfect fields in future experiments. Significance Statement This work explores the impact of cutting-edge numerical weather prediction ensemble guidance (the Warn-on-Forecast System) on severe thunderstorm hazard outlooks at watch-to-warning time scales, typically between 1 and 6 h of lead time. Real-time forecast products in this time frame are currently provided on an as-needed basis, and the transition to continuous probabilistic forecast products across scales requires targeted research. Results showed that hourly probabilistic participant forecasts were skillful subjectively and statistically, and that the experimental guidance improved the forecasts. These results are promising for the implementation and value of the Warn-on-Forecast System to provide improved hazard timing and location guidance within severe weather watches. Suggestions are made to aid future subjective evaluations of watch-to-warning-scale probabilistic forecasts.

Biases and Skill of Four Two-Moment Bulk Microphysics Schemes in Convection-Allowing Forecasts for the 2018 Hazardous Weather Testbed Spring Forecasting Experiment Period

Abstract A proof-of-concept systematic evaluation of convective hazards is applied to short-term (1–6 h) forecasts using the Morrison, National Severe Storms Laboratory (NSSL), Predicted Particle Properties (P3), and Thompson two-moment microphysics schemes for the 2018 NOAA Hazardous Weather Testbed Spring Forecasting Experiment (HWT SFE) period (hereafter “MORR,” “NSSL,” “P3,” and “THOM” experiments, respectively). Four convective line cases are highlighted to elaborate on relative experiment biases/skill. Composite reflectivity and 1-h accumulated precipitation are examined to determine storm coverage/precipitation biases/skill utilizing point-based verification with a neighborhood. Simulated 1–6-km updraft helicity and observed 3–6-km azimuthal shear and MESH are examined to consider simulated rotation and hail core prediction with object-based scores. Over the full season, MORR displays little overall storm coverage bias relative to NSSL, P3, and THOM underprediction. The equitable threat score (ETS) and fractions skill score (FSS) of P3 are lower than the other experiments. P3 and THOM underpredict convective regions with intense reflectivity relative to MORR and NSSL overprediction. All experiments underpredict precipitation amounts. P3 light precipitation FSS is lower than other experiments. Rotation object verification exhibits sensitivity to microphysics experiments, as microphysics has an indirect influence on storm dynamics. While P3 has the largest hail object underprediction, all experiments grossly overpredict the number of hail objects in convective line cases despite forecast objects defined with the same product (MESH) and threshold as observations. The importance of microphysics ice parameterization and ongoing scheme updates highlight the need to apply this verification framework to optimal/updated schemes before optimizing ensemble design.

Scale-Dependent Verification of the OU MAP Convection Allowing Ensemble Initialized with Multi-Scale and Large-Scale Perturbations during the 2019 NOAA Hazardous Weather Testbed Spring Forecasting Experiment

Given the large range of resolvable space and time scales in large-domain convection-allowing for ensemble forecasts, there is a need to better understand optimal initial-condition perturbation strategies to sample the forecast uncertainty across these space and time scales. This study investigates two initial-condition perturbation strategies for CONUS-domain ensemble forecasts that extend into the two-day forecast lead time using traditional and object-based verification methods. Initial conditions are perturbed either by downscaling perturbations from a coarser resolution ensemble (i.e., LARGE) or by adopting the analysis perturbations from a convective-scale, EnKF system (i.e., MULTI). It was found that MULTI had more ensemble spread than LARGE across all scales initially, while LARGE’s perturbation energy surpassed that of MULTI after 3 h and continued to maintain a surplus over MULTI for the rest of the 36h forecast period. Impacts on forecast bias were mixed, depending on the forecast lead time and forecast threshold. However, MULTI was found to be significantly more skillful than LARGE at early forecast hours for the meso-gamma and meso-beta scales (1–9h), which is a result of a larger and better-sampled ensemble spread at these scales. Despite having a smaller ensemble spread, MULTI was also significantly more skillful than LARGE on the meso-alpha scale during the 20–24h period due to a better spread-skill relation. MULTI’s performance on the meso-alpha scale was slightly worse than LARGE’s performance during the 6–12h period, as LARGE’s ensemble spread surpassed that of MULTI. The advantages of each method for different forecast aspects suggest that the optimal perturbation strategy may require a combination of both the MULTI and LARGE techniques for perturbing initial conditions in a large-domain, convection-allowing ensemble.

Assessment of Storm‐Scale Real Time Assimilation of GOES‐16 GLM Lightning‐Derived Water Vapor Mass on Short Term Precipitation Forecasts During the 2020 Spring Forecast Experiment

AbstractThis study assesses the impact of assimilating pseudo‐observations for water vapor mass derived from the Geostationary Operational Environmental Satellites GOES‐16 lightning data on short‐term quantitative precipitation forecasts (QPFs) over the contiguous United States (CONUS), with an emphasis given to regions characterized by an overall poor radar coverage. The GOES‐16/17 provide nearly uniform and high temporal resolution total lightning observations over most of the Americas. To leverage this information for convective scale weather forecasts, a three‐dimensional variational data assimilation (DA) package developed by the National Severe Storm Laboratory was employed to assimilate these data. To mimic operational settings, the Weather Research and Forecasting Model configuration used in NOAA Global Systems Laboratory's High‐Resolution Rapid Refresh Model version 4 was utilized. During the 2020 NOAA Hazardous Weather Testbed Spring Forecasting Experiment, four experiments were run in real time during a 29‐day period over CONUS and surrounding territories to assess the added value of GOES‐16 lightning over conventional radar data. Overall, the lightning DA (LDA) showed benefit in improving QPFs up to 6 hr, with the best improvements seen during the first 3 hr. Owing to notably larger lightning activity over the eastern CONUS, the most noticeable impacts from the LDA were seen there. Case‐by‐case analysis revealed that positive impacts from the LDA were seen over areas characterized by both good (eastern two thirds of CONUS) and poor radar coverage (western one third of CONUS). Highlighting the inherent difficulties in developing an optimal observation operator based on moisture for LDA applications, the GLM‐based DA experiments systematically overestimated rainfall over the eastern two thirds of CONUS while underestimating it over the western one third of CONUS.

Using Machine Learning to Generate Storm-Scale Probabilistic Guidance of Severe Weather Hazards in the Warn-on-Forecast System

AbstractA primary goal of the National Oceanic and Atmospheric Administration Warn-on-Forecast (WoF) project is to provide rapidly updating probabilistic guidance to human forecasters for short-term (e.g., 0–3 h) severe weather forecasts. Postprocessing is required to maximize the usefulness of probabilistic guidance from an ensemble of convection-allowing model forecasts. Machine learning (ML) models have become popular methods for postprocessing severe weather guidance since they can leverage numerous variables to discover useful patterns in complex datasets. In this study, we develop and evaluate a series of ML models to produce calibrated, probabilistic severe weather guidance from WoF System (WoFS) output. Our dataset includes WoFS ensemble forecasts available every 5 min out to 150 min of lead time from the 2017–19 NOAA Hazardous Weather Testbed Spring Forecasting Experiments (81 dates). Using a novel ensemble storm-track identification method, we extracted three sets of predictors from the WoFS forecasts: intrastorm state variables, near-storm environment variables, and morphological attributes of the ensemble storm tracks. We then trained random forests, gradient-boosted trees, and logistic regression algorithms to predict which WoFS 30-min ensemble storm tracks will overlap a tornado, severe hail, and/or severe wind report. To provide rigorous baselines against which to evaluate the skill of the ML models, we extracted the ensemble probabilities of hazard-relevant WoFS variables exceeding tuned thresholds from each ensemble storm track. The three ML algorithms discriminated well for all three hazards and produced more reliable probabilities than the baseline predictions. Overall, the results suggest that ML-based postprocessing of dynamical ensemble output can improve short-term, storm-scale severe weather probabilistic guidance.

Exploring Convection-Allowing Model Evaluation Strategies for Severe Local Storms Using the Finite-Volume Cubed-Sphere (FV3) Model Core

AbstractVerification methods for convection-allowing models (CAMs) should consider the finescale spatial and temporal detail provided by CAMs, and including both neighborhood and object-based methods can account for displaced features that may still provide useful information. This work explores both contingency table–based verification techniques and object-based verification techniques as they relate to forecasts of severe convection. Two key fields in severe weather forecasting are investigated: updraft helicity (UH) and simulated composite reflectivity. UH is used to generate severe weather probabilities called surrogate severe fields, which have two tunable parameters: the UH threshold and the smoothing level. Probabilities computed using the UH threshold and smoothing level that give the best area under the receiver operating curve result in very high probabilities, while optimizing the parameters based on the Brier score reliability component results in much lower probabilities. Subjective ratings from participants in the 2018 NOAA Hazardous Weather Testbed Spring Forecasting Experiment (SFE) provide a complementary evaluation source. This work compares the verification methodologies in the context of three CAMs using the Finite-Volume Cubed-Sphere Dynamical Core (FV3), which will be the foundation of the U.S. Unified Forecast System (UFS). Three agencies ran FV3-based CAMs during the five-week 2018 SFE. These FV3-based CAMs are verified alongside a current operational CAM, the High-Resolution Rapid Refresh version 3 (HRRRv3). The HRRR is planned to eventually use the FV3 dynamical core as part of the UFS; as such evaluations relative to current HRRR configurations are imperative to maintaining high forecast quality and informing future implementation decisions.

Neighborhood- and Object-Based Probabilistic Verification of the OU MAP Ensemble Forecasts during 2017 and 2018 Hazardous Weather Testbeds

Abstract An object-based probabilistic (OBPROB) forecasting framework is developed and applied, together with a more traditional neighborhood-based framework, to convection-permitting ensemble forecasts produced by the University of Oklahoma (OU) Multiscale data Assimilation and Predictability (MAP) laboratory during the 2017 and 2018 NOAA Hazardous Weather Testbed Spring Forecasting Experiments. Case studies from 2017 are used for parameter tuning and demonstration of methodology, while the 2018 ensemble forecasts are systematically verified. The 2017 case study demonstrates that the OBPROB forecast product can provide a unique tool to operational forecasters that includes convective-scale details such as storm mode and morphology, which are typically lost in neighborhood-based methods, while also providing quantitative ensemble probabilistic guidance about those details in a more easily interpretable format than the more commonly used paintball plots. The case study also demonstrates that objective verification metrics reveal different relative performance of the ensemble at different forecast lead times depending on the verification framework (i.e., object versus neighborhood) because of the different features emphasized by object- and neighborhood-based evaluations. Both frameworks are then used for a systematic evaluation of 26 forecasts from the spring of 2018. The OBPROB forecast verification as configured in this study shows less sensitivity to forecast lead time than the neighborhood forecasts. Both frameworks indicate a need for probabilistic calibration to improve ensemble reliability. However, lower ensemble discrimination for OBPROB than the neighborhood-based forecasts is also noted.

Comparisons of QPFs Derived from Single- and Multicore Convection-Allowing Ensembles

Abstract This study compares ensemble precipitation forecasts from 10-member, 3-km grid-spacing, CONUS domain single- and multicore ensembles that were a part of the 2016 Community Leveraged Unified Ensemble (CLUE) that was run for the 2016 NOAA Hazardous Weather Testbed Spring Forecasting Experiment. The main results are that a 10-member ARW ensemble was significantly more skillful than a 10-member NMMB ensemble, and a 10-member MIX ensemble (5 ARW and 5 NMMB members) performed about the same as the 10-member ARW ensemble. Skill was measured by area under the relative operating characteristic curve (AUC) and fractions skill score (FSS). Rank histograms in the ARW ensemble were flatter than the NMMB ensemble indicating that the envelope of ensemble members better encompassed observations (i.e., better reliability) in the ARW. Rank histograms in the MIX ensemble were similar to the ARW ensemble. In the context of NOAA’s plans for a Unified Forecast System featuring a CAM ensemble with a single core, the results are positive and indicate that it should be possible to develop a single-core system that performs as well as or better than the current operational CAM ensemble, which is known as the High-Resolution Ensemble Forecast System (HREF). However, as new modeling applications are developed and incremental changes that move HREF toward a single-core system are made possible, more thorough testing and evaluation should be conducted.

The Influence of Vertical Advection Discretization in the WRF-ARW Model on Capping Inversion Representation in Warm-Season, Thunderstorm-Supporting Environments

Abstract Previous studies have suggested that the Advanced Research version of the Weather Research and Forecasting (WRF-ARW) Model is unable, in its default configuration, to adequately resolve the capping inversions that are commonly found in the warm-season, thunderstorm-supporting environments of the central United States. Since capping inversions typically form in environments of synoptic-scale subsidence, this study tests the hypothesis that this degradation results, in part, from implicit numerical damping of shorter-wavelength features associated with the model-default third-order-accurate vertical advection finite-differencing scheme. To aid in testing this hypothesis, two short-range, deterministic, convection-allowing model forecasts, one using the default third-order-accurate vertical advection finite-differencing scheme and another using a fourth-order-accurate differencing scheme (which lacks implicit damping but is numerically dispersive), are conducted for 25 days during the 2017 NOAA Hazardous Weather Testbed Spring Forecasting Experiment. Model-derived vertical profiles at lead times of 11 and 23 h are validated against available rawinsonde observations released in regions located in the Storm Prediction Center’s 0600 UTC day 1 convection outlook’s “general thunderstorm” forecast area. The fourth-order-accurate vertical advection finite-differencing scheme is shown to not result in statistically significant improvements to model-forecast capping inversions or, more generally, the vertical thermodynamic profile in the lower troposphere. Instead, the fourth-order-accurate differencing scheme primarily impacts the representation of longer-wavelength features already reasonably well resolved by the model. The analysis does, however, provide quantitative evidence over a large sample that, on average, the WRF-ARW model forecasts capping inversions that are too weak, with negative buoyancy spread out over too deep of a vertical layer, compared to observations.

Practical Predictability of Supercells: Exploring Ensemble Forecast Sensitivity to Initial Condition Spread

Abstract As convection-allowing ensembles are routinely used to forecast the evolution of severe thunderstorms, developing an understanding of storm-scale predictability is critical. Using a full-physics numerical weather prediction (NWP) framework, the sensitivity of ensemble forecasts of supercells to initial condition (IC) uncertainty is investigated using a perfect model assumption. Three cases are used from the real-time NSSL Experimental Warn-on-Forecast System for Ensembles (NEWS-e) from the 2016 NOAA Hazardous Weather Testbed Spring Forecasting Experiment. The forecast sensitivity to IC uncertainty is assessed by repeating the simulations with the initial ensemble perturbations reduced to 50% and 25% of their original magnitudes. The object-oriented analysis focuses on significant supercell features, including the mid- and low-level mesocyclone, and rainfall. For a comprehensive analysis, supercell location and amplitude predictability of the aforementioned features are evaluated separately. For all examined features and cases, forecast spread is greatly reduced by halving the IC spread. By reducing the IC spread from 50% to 25% of the original magnitude, forecast spread is still substantially reduced in two of the three cases. The practical predictability limit (PPL), or the lead time beyond which the forecast spread exceeds some prechosen threshold, is case and feature dependent. Comparing to past studies reveals that practical predictability of supercells is substantially improved by initializing once storms are well established in the ensemble analysis.

More on the Scale Dependence of the Predictability of Precipitation Patterns: Extension to the 2009–13 CAPS Spring Experiment Ensemble Forecasts

AbstractThis paper analyzes the scale and case dependence of the predictability of precipitation in the Storm-Scale Ensemble Forecast (SSEF) system run by the Center for Analysis and Prediction of Storms (CAPS) during the NOAA Hazardous Weather Testbed Spring Experiments of 2008–13. The effect of different types of ensemble perturbation methodologies is quantified as a function of spatial scale. It is found that uncertainties in the large-scale initial and boundary conditions and in the model microphysical parameterization scheme can result in the loss of predictability at scales smaller than 200 km after 24 h. Also, these uncertainties account for most of the forecast error. Other types of ensemble perturbation methodologies were not found to be as important for the quantitative precipitation forecasts (QPFs). The case dependences of predictability and of the sensitivity to the ensemble perturbation methodology were also analyzed. Events were characterized in terms of the extent of the precipitation coverage and of the convective-adjustment time scale , an indicator of whether convection is in equilibrium with the large-scale forcing. It was found that events characterized by widespread precipitation and small values (representative of quasi-equilibrium convection) were usually more predictable than nonequilibrium cases. No significant statistical relationship was found between the relative role of different perturbation methodologies and precipitation coverage or .

Comparison of Next-Day Probabilistic Severe Weather Forecasts from Coarse- and Fine-Resolution CAMs and a Convection-Allowing Ensemble

Abstract Given increasing computing power, an important question is whether additional computational resources would be better spent reducing the horizontal grid spacing of a convection-allowing model (CAM) or adding members to form CAM ensembles. The present study investigates this question as it applies to CAM-derived next-day probabilistic severe weather forecasts created by using forecast updraft helicity as a severe weather proxy for 63 days of the 2010 and 2011 NOAA Hazardous Weather Testbed Spring Forecasting Experiments. Forecasts derived from three sets of Weather Research and Forecasting Model configurations are tested: a 1-km deterministic model, a 4-km deterministic model, and an 11-member, 4-km ensemble. Forecast quality is evaluated using relative operating characteristic (ROC) curves, attributes diagrams, and performance diagrams, and forecasts from five representative cases are analyzed to investigate their relative quality and value in a variety of situations. While no statistically significant differences exist between the 4- and 1-km deterministic forecasts in terms of area under ROC curves, the 4-km ensemble forecasts offer weakly significant improvements over the 4-km deterministic forecasts over the entire 63-day dataset. Further, the 4-km ensemble forecasts generally provide greater forecast quality relative to either of the deterministic forecasts on an individual day. Collectively, these results suggest that, for purposes of improving next-day CAM-derived probabilistic severe weather forecasts, additional computing resources may be better spent on adding members to form CAM ensembles than on reducing the horizontal grid spacing of a deterministic model below 4 km.

Sensitivity of 24-h Forecast Dryline Position and Structure to Boundary Layer Parameterizations in Convection-Allowing WRF Model Simulations

Abstract Recent NOAA Hazardous Weather Testbed Spring Forecasting Experiments have emphasized the sensitivity of forecast sensible weather fields to how boundary layer processes are represented in the Weather Research and Forecasting (WRF) Model. Thus, since 2010, the Center for Analysis and Prediction of Storms has configured at least three members of their WRF-based Storm-Scale Ensemble Forecast (SSEF) system specifically for examination of sensitivities to parameterizations of turbulent mixing, including the Mellor–Yamada–Janjić (MYJ); quasi-normal scale elimination (QNSE); Asymmetrical Convective Model, version 2 (ACM2); Yonsei University (YSU); and Mellor–Yamada–Nakanishi–Niino (MYNN) schemes (hereafter PBL members). In postexperiment analyses, significant differences in forecast boundary layer structure and evolution have been observed, and for preconvective environments MYNN was found to have a superior depiction of temperature and moisture profiles. This study evaluates the 24-h forecast dryline positions in the SSEF system PBL members during the period April–June 2010–12 and documents sensitivities of the vertical distribution of thermodynamic and kinematic variables in near-dryline environments. Main results include the following. Despite having superior temperature and moisture profiles, as indicated by a previous study, MYNN was one of the worst-performing PBL members, exhibiting large eastward errors in forecast dryline position. During April–June 2010–11, a dry bias in the North American Mesoscale Forecast System (NAM) initial conditions largely contributed to eastward dryline errors in all PBL members. An upgrade to the NAM and assimilation system in October 2011 apparently fixed the dry bias, reducing eastward errors. Large sensitivities of CAPE and low-level shear to the PBL schemes were found, which were largest between 1.0° and 3.0° to the east of drylines. Finally, modifications to YSU to decrease vertical mixing and mitigate its warm and dry bias greatly reduced eastward dryline errors.

Object-Based Evaluation of the Impact of Horizontal Grid Spacing on Convection-Allowing Forecasts

Abstract Forecasts generated by the Center for Analysis and Prediction of Storms with 1- and 4-km grid spacing using the Advanced Research Weather Research and Forecasting Model (ARW-WRF; ARW1 and ARW4, respectively) for the 2009–11 NOAA Hazardous Weather Testbed Spring Experiments are compared and verified. Object-based measures, including average values of object attributes, the object-based threat score (OTS), and the median of maximum interest (MMI) are used for the verification. Verification was first performed against observations at scales resolvable by each forecast model and then performed at scales resolvable by both models by remapping ARW1 to the ARW4 grid (ARW1to4). Thirty-hour forecasts of 1-h accumulated precipitation initialized at 0000 UTC on 22, 36, and 33 days during the spring of 2009, 2010, and 2011, respectively, are evaluated over a domain covering most of the central and eastern United States. ARW1, ARW1to4, and ARW4 all significantly overforecasted the number of objects during diurnal convection maxima. The overforecasts by ARW1 and ARW1to4 were more pronounced than ARW4 during the first convection maximum at 1-h lead time. The average object area and aspect ratio were closer to observations for ARW1 and ARW1to4 than for ARW4. None of the models showed a significant advantage over the others for average orientation angle and centroid location. Increased accuracy for ARW1, compared to ARW4, was statistically significant for the MMI but not the OTS. However, ARW1to4 had similar MMI and OTS as ARW4 at most lead times. These results are consistent with subjective evaluations that the greatest impact of grid spacing is on the smallest resolvable objects.

Forecasting Tornado Pathlengths Using a Three-Dimensional Object Identification Algorithm Applied to Convection-Allowing Forecasts

Abstract A three-dimensional (in space and time) object identification algorithm is applied to high-resolution forecasts of hourly maximum updraft helicity (UH)—a diagnostic that identifies simulated rotating storms—with the goal of diagnosing the relationship between forecast UH objects and observed tornado pathlengths. UH objects are contiguous swaths of UH exceeding a specified threshold. Including time allows tracks to span multiple hours and entire life cycles of simulated rotating storms. The object algorithm is applied to 3 yr of 36-h forecasts initialized daily from a 4-km grid-spacing version of the Weather Research and Forecasting Model (WRF) run in real time at the National Severe Storms Laboratory (NSSL), and forecasts from the Storm Scale Ensemble Forecast (SSEF) system run by the Center for Analysis and Prediction of Storms for the 2010 NOAA Hazardous Weather Testbed Spring Forecasting Experiment. Methods for visualizing UH object attributes are presented, and the relationship between pathlengths of UH objects and tornadoes for corresponding 18- or 24-h periods is examined. For deterministic NSSL-WRF UH forecasts, the relationship of UH pathlengths to tornadoes was much stronger during spring (March–May) than in summer (June–August). Filtering UH track segments produced by high-based and/or elevated storms improved the UH–tornado pathlength correlations. The best ensemble results were obtained after filtering high-based and/or elevated UH track segments for the 20 cases in April–May 2010, during which correlation coefficients were as high as 0.91. The results indicate that forecast UH pathlengths during spring could be a very skillful predictor for the severity of tornado outbreaks as measured by total pathlength.

Object-Based Evaluation of a Storm-Scale Ensemble during the 2009 NOAA Hazardous Weather Testbed Spring Experiment

Abstract Object-based verification of deterministic forecasts from a convection-allowing ensemble for the 2009 NOAA Hazardous Weather Testbed Spring Experiment is conducted. The average of object attributes is compared between forecasts and observations and between forecasts from subensembles with different model dynamics. Forecast accuracy for the full ensemble and the subensembles with different model dynamics is also evaluated using two object-based measures: the object-based threat score (OTS) and the median of maximum interest (MMI). Forecast objects aggregated from the full ensemble are generally more numerous, have a smaller average area, more circular average aspect ratio, and more eastward average centroid location than observed objects after the 1-h lead time. At the 1-h lead time, forecast objects are less numerous than observed objects. Members using the Advanced Research Weather Research and Forecasting Model (ARW) have fewer objects, more linear average aspect ratio, and smaller average area than members using the Nonhydrostatic Mesoscale Model (NMM). The OTS aggregated from the full ensemble is more consistent with the diurnal cycles of the traditional equitable threat score (ETS) than the MMI because the OTS places more weight on large objects, while the MMI weights all objects equally. The group of ARW members has higher OTS than the group of NMM members except at the 1-h lead time when the group of NMM members has more accurate maintenance and evolution of initially present precipitation systems provided by radar data assimilation. The differences between the ARW and NMM accuracy are more pronounced with the OTS than the MMI and the ETS.

Hierarchical Cluster Analysis of a Convection-Allowing Ensemble during the Hazardous Weather Testbed 2009 Spring Experiment. Part II: Ensemble Clustering over the Whole Experiment Period

Abstract Twenty-member real-time convection-allowing storm-scale ensemble forecasts with perturbations to model physics, dynamics, initial conditions (IC), and lateral boundary conditions (LBC) during the NOAA Hazardous Weather Testbed Spring Experiment provide a unique opportunity to study the relative impact of different sources of perturbation on convection-allowing ensemble diversity. In Part II of this two-part study, systematic similarity/dissimilarity of hourly precipitation forecasts among ensemble members from the spring season of 2009 are identified using hierarchical cluster analysis (HCA) with a fuzzy object-based threat score (OTS), developed in Part I. In addition to precipitation, HCA is also performed on ensemble forecasts using the traditional Euclidean distance for wind speed at 10 m and 850 hPa, and temperature at 500 hPa. At early lead times (3 h, valid at 0300 UTC) precipitation forecasts cluster primarily by data assimilation and model dynamic core, indicating a dominating impact of models, with secondary clustering by microphysics. There is an increasing impact of the planetary boundary layer (PBL) scheme on clustering relative to the microphysics scheme at later lead times. Forecasts of 10-m wind speed cluster primarily by the PBL scheme at early lead times, with an increasing impact of LBC at later lead times. Forecasts of midtropospheric variables cluster primarily by IC at early lead times and LBC at later lead times. The radar and Mesonet data assimilation (DA) show its impact, with members without DA in a distinct cluster, through the 12-h lead time (valid at 1200 UTC) for both precipitation and nonprecipitation variables. The implication for optimal ensemble design for storm-scale forecasts is also discussed.

Improving Probabilistic Ensemble Forecasts of Convection through the Application of QPF–POP Relationships

Abstract Four new approaches of postprocessing quantitative precipitation forecasts (QPFs) from model ensemble output were used to generate probability of precipitation (POP) tables in order to develop a forecasting method that could outperform a traditional method that relies upon calibration of POP forecasts derived using equal weighting of ensemble members. Early warm season 10-member ensemble output from the NOAA Hazardous Weather Testbed Spring Experiments was used, with 29 cases serving as a training set to create the POP tables and 20 cases used as a test set. The new approaches use QPF–POP relationships based on two properties termed precipitation amount characteristic (PAC) and ensemble member agreement. Exploratory results are presented for 20-km grid spacing and selectively for 4-km grid spacing. In the first approach, POPs were based on a binned PAC and the number of ensemble members with 6-h precipitation accumulations greater than given thresholds. In a second approach, a neighborhood method was used to find the number of points in a given neighborhood area around each of the domain grid points with precipitation amounts greater than a given threshold, while also considering the binned PAC representative of the neighborhood. A third approach synthesized the previous methods and led to an increase in skill relative to the individual methods, and a fourth approach using a combination of methods produced forecasts with even greater skill. All of the forecasts from the four approaches were improved statistically significantly compared to the calibrated traditional method’s forecasts at 20-km grid spacing. The second approach on its own showed skill comparable to that obtained by a traditional calibrated 10-member ensemble, so adopting this approach alone could potentially save computer resources that could then be used for model refinements, only sacrificing the increased skill that could have been obtained by using the fourth approach.