Method For Object-based Diagnostic Evaluation Research Articles

Regional Cloud Forecast Verification Using Standard, Spatial, and Object-Oriented Methods

Abstract This study assesses the accuracy of the Coupled Ocean–Atmosphere Mesoscale Prediction System (COAMPS) forecasts for clouds within stable and unstable environments (thereafter refers as “stable” and “unstable” clouds). This evaluation is conducted by comparing these forecasts against satellite retrievals through a combination of traditional, spatial, and object-based methods. To facilitate this assessment, the Model Evaluation Tools (MET) community tool is employed. The findings underscore the significance of fine-tuning the MET parameters to achieve a more accurate representation of the features under scrutiny. The study’s results reveal that when employing traditional pointwise statistics (e.g., frequency bias and equitable threat score), there is consistency in the results whether calculated from Method for Object-Based Diagnostic Evaluation (MODE)-based objects or derived from the complete fields. Furthermore, the object-based statistics offer valuable insights, indicating that COAMPS generally predicts cloud object locations accurately, though the spread of these predicted locations tends to increase with time. It tends to overpredict the object area for unstable clouds while underpredicting it for stable clouds over time. These results are in alignment with the traditional pointwise bias scores for the entire grid. Overall, the spatial metrics provided by the object-based verification methods emerge as crucial and practical tools for the validation of cloud forecasts. Significance Statement As the general Navy meteorological and oceanographic (METOC) community engages in collaboration with the broader scientific community, our goal is to harness community tools like MET for the systematic evaluation of weather forecasts, with a specific focus on variables crucial to the Navy. Clouds, given their significant impact on visibility, hold particular importance in our investigations. Cloud forecasts pose unique challenges, primarily attributable to the intricate physics governing cloud development and the complexity of representing these processes within numerical models. Cloud observations are also constrained by limitations, arising from both top-down satellite measurements and bottom-up ground-based measurements. This study illustrates that, with a comprehensive understanding of community tools, cloud forecasts can be consistently verified. This verification encompasses traditional evaluation methods, measuring general qualities such as bias and root-mean-squared error, as well as newer techniques like spatial and object-based methods designed to account for displacement errors.

Simulations of a tornadic supercell event in the south of Spain: Sensitivity to initial and boundary conditions and microphysics parameterizations

A tornadic supercell near the village of Campillos (Malaga province) occurred on 26 August 2019 with several associated damages. In this study, we analyze the sensitivity of the Weather Research and Forecasting (WRF) model to initial and boundary conditions and microphysics parameterizations when reproducing this particular event. A total of 20 cases were simulated with four initial and boundary conditions (ERA5, GDAS/FNL, GFS, HRES-IFS) and five microphysics parameterizations (WDM6, Goddard, Morrison, Thompson, WSM6). Forecasts are quantitatively compared with 10-min observations at 153 weather stations for the following variables: temperature at 2 m (T2), relative humidity at 2 m (RH2), wind speed at 10 m (U10) and accumulated precipitation in 3 h (PCP3h). Similarly, the spatial verification method MODE (Method for Object-Based Diagnostic Evaluation) has been used to evaluate the reproduction of convective storms by high-resolution numerical simulations. Our results indicate that microphysics plays a key role in this event, being more relevant than initial and boundary conditions. In particular, Goddard configuration combined with ERA5 or IFS initial and boundary conditions has shown the best overall performance. Simulation based on Goddard and ERA5 captured features to diagnose the development of severe convection. Therefore, WRF model, with the correct parameter sets, was able to reproduce the environmental conditions for supercell formation. The spatial resolution used in the simulations was appropriate to explicitly generate a supercellular storm, although it was not sufficiently high to resolve the formation of tornadoes. Nevertheless, the model showed signs of an environment favourable to tornadogenesis.

Methods for Validating HRRR Simulated Cloud Properties for Different Weather Phenomena Using Satellite and Radar Observations

Abstract In this study, we evaluate the ability of the High-Resolution Rapid Refresh (HRRR) model to forecast cloud characteristics through comparison of observed and simulated satellite brightness temperatures (BTs) and radar reflectivity during different weather phenomena in December 2021: the Mayfield, Kentucky, tornado on 11 December, a heavy snow event in Minnesota from 10 to 11 December, and the Midwest derecho on 15 December. This is done to illustrate the importance of examining model accuracy across a range of weather phenomena. Observation and forecast objects were created using the Method for Object-Based Diagnostic Evaluation (MODE). HRRR accurately depicted the spatial displacements between observation cloud (defined using BTs) and radar reflectivity objects, namely, the centers of cloud objects are to the east of the radar objects for the tornado and derecho events, and generally west of the radar objects for the snow event. However, HRRR had higher (less intense) simulated BTs and higher (more intense) radar reflectivity than the observations for the tornado event. Simulated radar reflectivity is higher and BTs are lower than the observations during the middle of the snow event. Also, simulated radar reflectivity is higher and BTs are lower than the observations during the derecho event. Of the three weather events, the HRRR forecasts are most accurate for the snow event, based on the object-based threat score, followed by the derecho and tornado events. The tornado event has lower accuracy because matches between paired simulated and observation objects are worse than for the snow event, with less similarity in size forecast objects and greater distance between paired object centers. Significance Statement The purpose of this study is to assess the accuracy of forecast cloud and radar objects, defined using simulated satellite brightness temperatures and radar reflectivity, from the High-Resolution Rapid Refresh (HRRR) model. This assessment was conducted for a tornado, snow, and derecho event from December 2021. Results from these three events indicate that the HRRR model accurately represents the observed displacement between the center of cloud and radar objects for the tornado and derecho events, and is the most accurate overall for the snow event.

Verification of multiresolution model forecasts of heavy rainfall events from 23 to 26 August 2017 over Nigeria

AbstractThe study uses numerical weather prediction models to predict the occurrence of heavy convective rainfall associated with the passage of the African Easterly Wave (AEW) during the period 23–26 August 2017 over Nigeria. Fraction skill score (FSS) and method for object‐based diagnostic evaluation (MODE) verification techniques were applied to verify how well the models predict the high‐impact event and to demonstrate how these tools can support operational forecasting. Ensemble model forecasts at a convective scale from UK Met Office Unified Model (MetUM) and a one‐way nested weather research and forecasting (WRF) model were compared with the integrated multisatellite retrievals for global precipitation measurement (IMERG GPM). The purpose is to examine skills of improved model resolution and ensemble in reproducing rainfall forecasts on useful scales and how the skill varies with spatial scale. WRF 2 and 6 km model forecasts show comparable skill at smaller grid scales. The skill of MetUM improves dramatically when the verification statistics are applied to the ensemble mean of the binary fields of the individual member forecast. The object‐based analysis reveals a similar structure as observed, although displaced eastwards. Most improvement occurred for heavier rainfall events associated with the passage of the AEW. WRF 6 km compares reasonably well with WRF 2 km in terms of shape and structure of rainfall underscoring the ability of the model to reasonably represent convection at 6 km horizontal resolution. The ensemble members in MetUM explicitly reproduce convection at 4 km resolution but are displaced at about 166 km behind observed rainfall.

Precipitation Nowcasting Based on Deep Learning over Guizhou, China

Accurate precipitation nowcasting (lead time: 0–2 h), which requires high spatiotemporal resolution data, is of great relevance in many weather-dependent social and operational activities. In this study, we are aiming to construct highly accurate deep learning (DL) models to directly obtain precipitation nowcasting at 6-min intervals for the lead time of 0–2 h. The Convolutional Long Short-Term Memory (ConvLSTM) and Predictive Recurrent Neural Network (PredRNN) models were used as comparative DL models, and the Lucas–Kanade (LK) Optical Flow method was selected as a traditional extrapolation baseline. The models were trained with high-quality datasets (resolution: 1 min) created from precipitation observations recorded by automatic weather stations in Guizhou Province (China). A comprehensive evaluation of the precipitation nowcasting was performed, which included consideration of the root mean square error, equitable threat score (ETS), and probability of detection (POD). The evaluation indicated that the reduction of the number of missing values and data normalization boosted training efficiency and improved the forecasting skill of the DL models. Increasing the time series length of the training set and the number of training samples both improved the POD and ETS of the DL models and enhanced nowcasting stability with time. Training with the Hea-P dataset further improved the forecasting skill of the DL models and sharply increased the ETS for thresholds of 2.5, 8, and 15 mm, especially for the 1-h lead time. The PredRNN model trained with the Hea-P dataset (time series length: 8 years) outperformed the traditional LK Optical Flow method for all thresholds (0.1, 1, 2.5, 8, and 15 mm) and obtained the best performance of all the models considered in this study in terms of ETS. Moreover, the Method for Object-Based Diagnostic Evaluation on a rainstorm case revealed that the PredRNN model, trained well with high-quality observation data, could both capture complex nonlinear characteristics of precipitation more accurately than achievable using the LK Optical Flow method and establish a reasonable mapping network during drastic changes in precipitation. Thus, its results more closely matched the observations, and its forecasting skill for thresholds exceeding 8 mm was improved substantially.

Object-based evaluation of rainfall forecasts over eastern China by eight cumulus parameterization schemes in the WRF model

Traditional pointwise verification is not fully accurate for the evaluation of rainfall forecasts due to the double-penalty problem. An alternative object-based metric, Method for Object-based Diagnostic Evaluation (MODE), was used in this study to evaluate the rainfall simulation performance of eight cumulus parameterization (CP) schemes in the Weather Research and Forecasting model. The average rainfall from May to August (MJJA) over eastern China from 2009 to 2018 and the sub-seasonal evolution of rainfall systems were mainly verified. The results showed that the heavy rainfall objects simulated by KFCuP, New Tiedtke and KSAS schemes are the closest to the observations over South China, Yangtze River Basin and North China, respectively. MODE could highlight the advantages of these schemes in the simulation of rainband physical attributes, such as centroid position, orientation, rainfall area, etc. At the sub-seasonal scale, KFCuP had the highest skill score for mimicking the observed heavy rainfall objects. To explore the reasons for the differences in rainfall simulations, the evaporation, water vapour inflow and precipitation efficiency were calculated for each scheme. It was found that members with higher precipitation generally corresponded to larger water vapour inflow and precipitation efficiency, and the water vapour inflow mainly came from the southern boundary. Further analysis found that there existed a significant positive correlation between the meridional wind and the water vapour inflow on southern boundary, which indicated that the meridional wind was an important factor affecting the regional rainfall simulation.

Dynamic downscaling ensemble forecast of an extreme rainstorm event in South China by COSMO EPS

In this study, the high-resolution ensemble prediction system COSMO (Consortium for Small Scale) EPS is used to predict the extreme rainstorm that occurred from 27 to 31 August 2018 in Guangdong Province, China, which leads to intensities exceeding historical extreme values. COSMO EPS is run with a 2.8-km grid spacing, allowing for an explicit treatment of deep convection, and 24 members of the EPS are initialized and laterally driven by the ICON (ICOsahedral Nonhydrostatic) global model. We compare the predictions of COSMO EPS against observations derived from the global precipitation measurement (GPM) and with ensemble forecasts of both mesoscale EPS and global EPS provided by GRAPES (Global and Regional Assimilation and PrEdiction System), and with the deterministic forecasts of global models ICON and ECMWF (European Centre for Medium-Range Weather Forecasts). Model performances are evaluated both by gridpoint-based scores, such as the equitable threat score (ETS), and by the Method for Object-based Diagnostic Evaluation (MODE) for spatial verification. According to our results, COSMO EPS could perform better forecasts for the rainstorms taken place in eastern Guangdong than other models. However, the location and coverage area of its predicted rainstorm is eastward and smaller in contrast with the observations. Therefore, COSMO EPS exhibits relative high performance by object-based spatial evaluations, while it could not display evident superiority in terms of the gridpoint-based scores. The cause analysis of this extreme rainstorm shows that Guangdong Province of China is mainly affected by monsoon depression. Southwesterly and southerly winds continuously transport water vapor from the South China Sea to Guangdong Province. The southwest monsoon low-level jet advances northward over time, which promotes the occurrence and development of continuous heavy precipitation in the coastal areas of Guangdong. In an additional experiment, we investigate the benefit of assimilation of radar data, by applying the latent heat nudging (LHN) approach based on surface-based radar observations to the COSMO EPS. Subsequently, the prediction by assimilation of radar data more reasonably reproduces the spatial distribution of precipitation observations, while the coverage and intensity of the rainstorm in eastern Guangdong are still not reflected satisfactorily.

Diagnostic Analysis of Multimodel Rainstorm Forecast for Cases Based on MODE Method

It is very important to analyze the performance of the model rainstorm forecast for understanding the intensity and position deviation of the model precipitation and improving the forecast ability. This paper uses classical scoring and the MODE (Method for Object-Based Diagnostic Evaluation) method to evaluate the forecast performance of different products. The forecast and observation data used in the article mainly include CMA (China Meteorological Administration) multi-source merged precipitation analysis, the precipitation forecast of ECMWF (European Centre for Medium-Range Weather Forecasts), CMA-Meso (Mesoscale model forecast of CMA) and SCMOC (National Meteorological Center grid precipitation forecast guidance product) data. At the same time, the possible correction method of heavy rainfall area is explored by using the high and low-level circulation configuration of the ECMWF model. The main conclusions are as follows: ① MODE spatial verification shows that the number, intensity, area and location of ECMWF rainstorm precipitation objects match the observed precipitation best, which is obviously better than SCMOC and CMA-Meso precipitation forecasts. There are significantly fewer SCMOC rainstorm precipitation objects, and the area of each single precipitation object is significantly larger, which often fails to report the small area objects of convective precipitation. ② The reason for the high TS score of SCMOC is that it reduces the number of small area rainstorm objects and avoids the “double punishment” phenomenon caused by position forecast error, which leads to the failure of SCMOC in local rainstorm forecasts. ③ Analyzing the relationship between the circulation situation of the ECMWF model and the location of rainstorm forecasts by the model, it is found that the location of the rainstorm area is consistent with the upper circulation system, especially with the strong rising area of the vertical velocity of 700 hPa and the high value area of the specific humidity of 850 hPa. When the rainstorm area coincides with the upper air system but is not consistent with the ground convergence area and the high value area of velocity potential, the rainstorm location often has a large deviation. The location of the surface convergence area can be used as a reference to improve the performance of the rainstorm forecast.

Verification of tropical cyclones (TC) wind structure forecasts from global NWP models and ensemble prediction systems (EPSs)

Forecasting wind structure of tropical cyclone (TC) is vital in assessment of impact due to high winds using Numerical Weather Prediction (NWP) model. The usual verification technique on TC wind structure forecasts are based on grid-to-grid comparisons between forecast field and the actual field. However, precision of traditional verification measures is easily affected by small scale errors and thus cannot well discriminate the accuracy or effectiveness of NWP model forecast. In this study, the Method for Object-Based Diagnostic Evaluation (MODE), which has been widely adopted in verifying precipitation fields, is utilized in TC's wind field verification for the first time. The TC wind field forecast of deterministic NWP model and Ensemble Prediction System (EPS) of the European Centre for Medium-Range Weather Forecasts (ECMWF) over the western North Pacific and the South China Sea in 2020 were evaluated. A MODE score of 0.5 is used as a threshold value to represent a skillful (or good) forecast. It is found that the R34 (radius of 34 knots) wind field structure forecasts within 72 h are good regardless of DET or EPS. The performance of R50 and R64 is slightly worse but the R50 forecasts within 48 h remain good, with MODE exceeded 0.5. The R64 forecast within 48 h are worth for reference as well with MODE of around 0.5. This study states that the TC wind field structure forecast by ECMWF is skillful for TCs over the western North Pacific and the South China Sea.

Spatial verification of a regional ensemble precipitation forecasting system at the Hydrometeorological Research Center of the Russian Federation using a free verification package, MET

Ensemble weather forecasting is a necessary tool for assessing the probabilities of weather events, alternative scenarios of their development, and forecast uncertainties. At the Hydrometeorological Research Center of the Russian Federation, a regional high-resolution ensemble system is under development with a grid mesh of 2.2 km for the central region of Russia. Consequently, developing a state-of-the-art ensemble verification system is of high priority. In this work, preliminary results of a high-resolution ensemble verification system are given for precipitation as an example. The system includes both standard probability scores and some novel spatial scores. A method for object-based diagnostic evaluation, MODE, is used as the main object-based approach. The system is based on a free verification package, METplus. Examples of scores are given.

Object-Based Verification of Atmospheric River Predictions in the Northeast Pacific

AbstractAccurate forecasts of atmospheric rivers (ARs) provide advance warning of flood and landslide hazards and greatly aid effective water management. It is, therefore, critical to evaluate the skill of AR forecasts in numerical weather prediction (NWP) models. A new verification framework is proposed that leverages freely available software and metrics previously used for different applications. Specifically, AR detection and statistics are computed for the first time using the Method for Object-Based Diagnostic Evaluation (MODE). In addition, the measure of effectiveness (MoE) is introduced as a new metric for understanding AR forecast skill in terms of size and location. The MoE provides a quantitative measure of the position of an entire forecast AR relative to observation, regardless of whether the AR is making landfall. In addition, the MoE can provide qualitative information about the evolution of a forecast by lead time, with implications about the predictability of an AR. We analyze AR forecast verification and skill using 11 years of cold-season forecasts from two NWP models: one global and one regional. Four different thresholds of integrated vapor transport (IVT) are used in the verification, revealing differences in forecast skill that are based on the strength of an AR. In addition to MoE, AR forecast skill is also addressed in terms of intensity error, landfall position error, and contingency-table metrics.

Evaluating the Impact of Planetary Boundary Layer, Land Surface Model, and Microphysics Parameterization Schemes on Cold Cloud Objects in Simulated GOES‐16 Brightness Temperatures

AbstractInfrared brightness temperatures (BTs) from the Geostationary Observing Environmental Satellite‐16 Advanced Baseline Imager are used to examine the ability of several microphysics and planetary boundary layer (PBL) schemes, as well as land surface models (LSM) and surface layers, to simulate upper‐level clouds. Six parameterization configurations were evaluated. Cloud objects are identified using the Method for Object‐Based Diagnostic Evaluation (MODE) and analyzed using the object‐based threat score, mean‐error distance, and pixel‐based metrics including the mean absolute error and mean bias error (MBE) for matched objects where the displacement between objects has been removed. Objects are identified using either a fixed BT threshold of 235 K or the 6.5th percentile of BTs for each model configuration. Analysis of the MODE‐identified cloud objects shows that, compared to a configuration with the Thompson microphysics scheme, Mellor‐Yamanda‐Nakanishi‐Niino (MYNN) PBL, Global Forecasting System (GFS) surface layer, and Noah LSM, the configuration employing the National Severe Storms Laboratory microphysics produced more cloud objects with higher BTs. Changing the PBL from MYNN to Shin‐Hong or Eddy‐Diffusivity Mass‐Flux also resulted in a slightly lower accuracy, though these changes result in configurations which more accurately reproduced the number of observation cloud objects and slightly reduced the high MBE. Changing the LSM from Noah to RUC reduces forecast accuracy by producing too many cloud objects with too low BTs. As the forecast hour increases, this accuracy reduction increases at a greater rate than occurred when changing the microphysics or PBL scheme and is further enhanced when using the MYNN surface layer rather than the GFS.

Impact of WRF Parameterization Schemes on Track and Intensity of Extremely Severe Cyclonic Storm “Fani”

Weather Research and Forecasting (WRF) model version 4.1.3 is configured in the present study to understand the impact of microphysical parameterization schemes on the track and intensity of extremely severe cyclonic storm (ESCS) “Fani” that occurred during 26 April to 4 May 2019 in the Bay of Bengal. The model is customized with six different microphysical parameterization schemes along with Kain–Fritsch cumulus for convection and the Yonsei University planetary boundary-layer scheme for computation of heat and moisture transport. The simulations are conducted at 27-km horizontal resolution with 41 levels in the vertical for 150 h (00UTC 28 April to 06UTC 4 May 2019). Model-simulated features such as the track, track error, minimum central sea-level pressure (MSLP), maximum sustained surface wind (MSW), and precipitation are validated against India Meteorological Department (IMD) and Tropical Rainfall Measuring Mission (TRMM) observation data. Although all the schemes predict track patterns similar to observation, the MP3 scheme shows the least track error even after landfall, and its track length is maximum in comparison with other schemes, albeit less than the observed track. The MP3 scheme exhibits a better track for “Fani,” followed by MP14. The lowest mean direct position error (along-track error) of 85 km (68.5 km) is noted for the MP3 scheme, followed by 112.5 km (82.3 km) for the MP8 scheme. Analysis of the MSLP and MSW using the t-test suggests that the MP2 scheme has better forecast ability, followed by MP6. Analysis of the quantitative/spatial matching of the rainfall forecasts and TRMM observations using the Method for Object-Based Diagnostic Evaluation (MODE) tool suggests that the MP8 (MP14) scheme is capable of simulating the rainfall beyond (up to) 48 h. Analysis of the zonal cross-section of horizontal and vertical winds shows that intense convection is well simulated by the MP2, MP3, and MP4 schemes during the change of intensity from cyclonic storm to severe cyclonic storm (SCS), by MP6 and MP14 from SCS to very severe cyclonic storm (VSCS) and from VSCS to ESCS, and by MP8 at ESCS level. All the schemes can capture the eyewall and simulate the westerly that allowed the system to move north-northeastwards. The synoptic flow at different pressure levels is reproduced by all the schemes. However, assessing the impact of the microphysics parameterizations on the synoptic flow of “Fani” at low resolution was difficult. Analysis of Qc and Qr indicates definite impacts of the microphysical parameterizations on the vertical structure of “Fani” in producing rainfall. The MP3 scheme exhibits a better cloud pattern up to upper troposphere level, followed by the MP2 scheme. Analysis of the latent heat flux predicted using the different microphysics schemes suggests that the latent heat flux release at VSCS intensity level or above contributes to further intensification of the TC, and this feature is very well simulated by the MP2 scheme.

Object-based tracking of precipitation systems in western Canada: the importance of temporal resolution of source data

Object-based algorithm provides additional spatiotemporal information of precipitation, besides traditional aspects such as amount and intensity. Using the Method for Object-based Diagnostic Evaluation with Time Dimension (MODE-TD, or MTD), precipitation features in western Canada have been analyzed comprehensively based on the Canadian Precipitation Analysis, North American Regional Reanalysis, Multi-Source Weighted-Ensemble Precipitation, and a convection-permitting climate model. We found light precipitation occurs frequently in the interior valleys of western Canada while moderate to heavy precipitation is rare there. The size of maritime precipitation system near the coast is similar to the continental precipitation system on the Prairies for moderate to heavy precipitation while light precipitation on the Prairies is larger in size than that occurs near the coast. For temporal features, moderate to heavy precipitation lasts longer than light precipitation over the Pacific coast, and precipitation systems on the Prairies generally move faster than the coastal precipitation. For annual cycle, the west coast has more precipitation events in cold seasons while more precipitation events are identified in warm seasons on the Prairies due to vigorous convection activities. Using two control experiments, the way how the spatiotemporal resolution of source data influences the MTD results has been examined. Overall, the spatial resolution of source data has little influence on MTD results. However, MTD driven by dataset with coarse temporal resolution tend to identify precipitation systems with relatively large size and slow propagation speed. This kind of precipitation systems normally have short track length and relatively long lifetime. For a typical precipitation system (0.7 $$\sim $$ 2 $$\times $$ 10 $$^{4}$$ km $$^{2}$$ in size) in western Canada, the maximum propagation speed that can be identified by 6-h data is approximately 25 km h $$^{-1}$$ , 33 km h $$^{-1}$$ for 3-h, and 100 km h $$^{-1}$$ for hourly dataset. Since the propagation speed of precipitation systems in North America is basically between 0 and 80 km h $$^{-1}$$ , we argue that precipitation features can be identified properly by MTD only when dataset with hourly or higher temporal resolution is used.

Assessing the Impact of Stochastic Perturbations in Cloud Microphysics using GOES-16 Infrared Brightness Temperatures

AbstractIn this study, infrared brightness temperatures (BTs) are used to examine how applying stochastic perturbed parameter (SPP) methodology to the widely used Thompson–Eidhammer cloud microphysics scheme impacts the cloud field in high-resolution forecasts. Modifications are made to add stochastic perturbations to three parameters controlling cloud generation and dissipation processes. Two five-member ensembles are generated, one using the microphysics parameter perturbations (SPP-MP) and another where white noise perturbations were added to potential temperature fields at initialization time (Control). The impact of the SPP method was assessed using simulated and observed GOES-16 BTs. This analysis uses pixel-based and object-based methods to assess the impact on the cloud field. Pixel-based methods revealed that the SPP-MP BTs are slightly more accurate than the Control BTs. However, too few pixels with a BT lower than 270 K result in a positive bias compared to the observations. A negative bias compared to the observations is observed when only analyzing lower BTs. The spread of the ensemble BTs was analyzed using the continuous ranked probability score differences, with the SPP-MP ensemble BTs having less (more) spread during May (January) compared to the Control. Object-based analysis using the Method for Object-Based Diagnostic Evaluation revealed the upper-level cloud objects are smaller in the SPP-MP ensemble than the Control but a lower bias exists in the SPP-MP BTs compared to the Control BTs when overlapping matching objects. However, there is no clear distinction between the SPP-MP and Control ensemble members during the evolution of objects, Overall, the SPP-MP perturbations result in lower BTs compared to the Control ensemble and more cloudy pixels.

Multimodel Ensemble Forecasts of Precipitation Based on an Object-Based Diagnostic Evaluation

Abstract We analyzed 24-h accumulated precipitation forecasts over the 4-month period from 1 May to 31 August 2013 over an area located in East Asia covering the region 15.05°–58.95°N, 70.15°–139.95°E generated with the ensemble prediction systems (EPS) from ECMWF, NCEP, UKMO, JMA, and CMA contained in the TIGGE dataset. The forecasts are first evaluated with the Method for Object-Based Diagnostic Evaluation (MODE). Then a multimodel ensemble (MME) forecast technique that is based on weights derived from object-based scores is investigated and compared with the equally weighted MME and the traditional gridpoint-based MME forecast using weights derived from the point-to-point metric, mean absolute error (MAE). The object-based evaluation revealed that attributes of objects derived from the ensemble members of the five individual EPS forecasts and the observations differ consistently. For instance, their predicted centroid location is more southwestward, their shape is more circular, and their orientation is more meridional than in the observations. The sensitivity of the number of objects and their attributes to methodological parameters is also investigated. An MME prediction technique that is based on weights computed from the object-based scores, median of maximum interest, and object-based threat score is explored and the results are compared with the ensemble forecasts of the individual EPS, the equally weighted MME forecast, and the traditional superensemble forecast. When using MODE statistics for the forecast evaluation, the object-based MME prediction outperforms all other predictions. This is mainly because of a better prediction of the objects’ centroid locations. When using the precipitation-based fractions skill score, which is not used in either of the weighted MME forecasts, the object-based MME forecasts are slightly better than the equally weighted MME forecasts but are inferior to the traditional superensemble forecast that is based on weights derived from the point-to-point metric MAE.

Verification of Extreme Rainfall in Typhoon Soudelor(1513) with the Model Evaluation Tools(MET)

ABSTRACT The extreme rainfall forecast performances of both of ECMWF-IFS and ECMWF-EPS with MET version 5.1 were examined in landing Typhoon Soudelor(1513) with 60h lead times. In this study the programs for converting ECMWF's forecast data (both of ECMWF-IFS and ECMWF-EPS) format into that needed by MET were developed. Also, during landfall, the observed maximum 6-hour accumulated rainfall was investigated, and then the verification of extreme rainfall in Soudelor was carried out. Results showed that while traditional verification gives relatively low scores, by the method for object-based diagnostic evaluation (MODE) the significant rainy areas were well predicted in this case study.

Validation of Snow Multibands in the Comma Head of an Extratropical Cyclone Using a 40-Member Ensemble

Abstract This paper investigates the ability of the Weather Research and Forecasting (WRF) Model in simulating multiple small-scale precipitation bands (multibands) within the extratropical cyclone comma head using four winter storm cases from 2014 to 2017. Using the model output, some physical processes are explored to investigate band prediction. A 40-member WRF ensemble was constructed down to 2-km grid spacing over the Northeast United States using different physics, stochastic physics perturbations, different initial/boundary conditions from the first five perturbed members of the Global Forecast System (GFS) Ensemble Reforecast (GEFSR), and a stochastic kinetic energy backscatter scheme (SKEBS). It was found that 2-km grid spacing is adequate to resolve most snowbands. A feature-based verification is applied to hourly WRF reflectivity fields from each ensemble member and the WSR-88D radar reflectivity at 2-km height above sea level. The Method for Object-Based Diagnostic Evaluation (MODE) tool is used for identifying multibands, which are defined as two or more bands that are 5–20 km in width and that also exhibit a >2:1 aspect ratio. The WRF underpredicts the number of multibands and has a slight eastward position bias. There is no significant difference in frontogenetical forcing, vertical stability, moisture, and vertical shear between the banded versus nonbanded members. Underpredicted band members tend to have slightly stronger frontogenesis than observed, which may be consolidating the bands, but overall there is no clear linkage in ambient condition errors and band errors, thus leaving the source for the band underprediction motivation for future work.

Performance assessment of WRF model in simulating the very severe cyclonic storm “TITLI” in the Bay of Bengal: A case study

The present study has been conducted to assess the performance of the Weather Research and Forecasting – Advanced Research (WRF-ARW, hereinafter named as WRF) model in simulating the very severe cyclonic storm (VSCS) “Titli” which formed over the Bay of Bengal during 08–13 October, 2018. The simulated meteorological variables viz. 10-m wind vectors, surface latent heat flux, rainfall, track and intensity based on minimum sea level pressure (MSLP), maximum sustained surface wind (MSW) are assessed to evaluate the model performance. The results are compared with India Meteorological Department (IMD) observations along with verification analyses [FiNaL analysis (FNL)]. The model simulated 24- h accumulation of rainfall verification is carried out by using the Method for Object based Diagnostic Evaluation (MODE) tool of Model Evaluation Tool Version 8 (MET-V8). The various statistical skill scores obtained from the MODE clearly indicates that the overall performance of WRF model in predicting the rainfall is reasonably good, mainly at lower forecast length (36–60 hr). The intensity and spatial distribution of rainfall are well matched with the verified analysis as well as with the Tropical Rainfall Measuring Mission (TRMM) observations. The analysis of maximum latent heat flux in the region of strongest surface wind suggests that the circulation is maintained due to increased transport of heat and moisture from the ocean. The track error analysis shows more error during the model initialization, which gradually decreases with the advance of forecast time. Efficacy of the WRF model in simulating the tropical cyclone (TC) is very distinct from the present study.

Statistics of multi‐year droughts from the method for object‐based diagnostic evaluation

This study uses the method for object‐based diagnostic evaluation (MODE) technique to examine and compare the statistics of drought attributes over the upper Colorado River basin (UCRB). The drought objects are based on the standardized precipitation index (SPI) and the standardized precipitation evapotranspiration index (SPEI) on a 36‐month timescale (SPI36 and SPEI36, respectively). The drought indicators are calculated using monthly precipitation as well as minimum and maximum temperatures from the Precipitation–Elevation Regression on Independent Slopes Model data sets from 1948 to 2012. MODE uses paired object attributes such as centroid distance, orientation angle, area ratio, and intersection area and a combination of parameter thresholds to determine the number of objects identified and retained in the merging and matching process in the two fields. Using MODE run with convolution radius of zero (no smoothing) and an area threshold of four grid points, this study computes and analyses object statistics including centroid locations, areas, and intensity percentiles. Results of the analysis show that SPI36 produces more drought objects than SPEI36. Although the spatial patterns are roughly similar leading up to almost similar statistics of object attributes, such as locations of the object centroids, the SPI36 produces higher percentile intensity of drought objects than does SPEI36, which is clearly obvious in the 90th percentile intensity of drought objects. The largest difference between SPEI36 and SPI36 occurs in the area of drought objects during the early 2000s when the region experienced multi‐year drought resulting from increased warming of the atmosphere. This study demonstrates the use of MODE as a tool to evaluate and monitor drought event over the UCRB.