Object-oriented Verification Research Articles

Listening to Stakeholders III: Potential Users Evaluate Product Content and Design for Subseasonal Extreme Precipitation Forecasts

Abstract Extreme precipitation over a 2-week period can cause significant impacts to life and property. Trustworthy and easy-to-understand forecasts of these extreme periods on the subseasonal-to-seasonal time frame may provide additional time for planning. The Prediction of Rainfall Extremes at Subseasonal to Seasonal Periods (PRES2iP) project team conducted three workshops over 6 years to engage with stakeholders to learn what is needed for decision-making for subseasonal precipitation. In this study, experimental subseasonal-to-seasonal (S2S) forecast products were designed, using knowledge gained from previous stakeholder workshops, and shown to decision-makers to evaluate the products for two 14-day extreme precipitation period scenarios. Our stakeholders preferred a combination of products that covered the spatial extent, regional daily values, with associated uncertainty, and text narratives with anticipated impacts for planning within the S2S time frame. When targeting longer extremes, having information regarding timing of expected impacts was seen as crucial for planning. We found that there is increased uncertainty tolerance with stakeholders when using products at longer lead times that typical skill metrics, such as critical success index or anomaly correlation coefficient, do not capture. Therefore, the use of object-oriented verification that allows for more flexibility in spatial uncertainty might be beneficial for evaluating S2S forecasts. These results help to create a foundation for the design, verification, and implementation of future operational forecast products with longer lead times, while also providing an example for future workshops that engage both researchers and decision-makers.

Assessing Variations in the Predictive Skill of Ensemble Snowband Forecasts with Object-Oriented Verification and Self-Organizing Maps

Abstract We used object-oriented verification and self-organizing maps (SOMs) to identify patterns in environmental parameters correlating with mesoscale snowband predictive skill by the High-Resolution Ensemble Forecast (HREF) system between 2017 and 2022. First, HREF snowband forecasts for 305 banding events were verified based on similarities between forecast and observed feature properties. HREF members performed comparably, demonstrating large positional errors, but the non-time-lagged High-Resolution Rapid Refresh member demonstrated the greatest overall skill. Observed banding events were clustered by 500-hPa geopotential height anomalies, mean sea level pressure, vertical velocity, frontogenesis, and saturation equivalent potential vorticity from the European Centre for Medium-Range Weather Forecasts Reanalysis version 5 using SOMs. Clusters reaffirmed the presence of midlevel frontogenesis, ascent, and reduced stability in most banding cases, and the predominant synoptic environments conducive to band development. Clusters were compared to determine whether patterns in the variables were correlated with predictive skill. Strength of upward motion was correlated with skill, with the strongest upward motion cases verifying 10% better than the weakest upward motion cases due to smaller positional error. Additionally, events with a single region of strong upward motion verified better than events with disorganized, but comparably intense, upward motion. The magnitude of frontogenesis was uncorrelated with skill, but events with more upright frontogenesis collocated with the band centroid were better predicted than events with shallower slopes and low-level frontogenesis displaced toward warmer air. The skill variance associated with different vertical motion magnitudes could assist forecasters in modulating forecast confidence, while the most common types of errors documented here may be beneficial to model developers in refining HREF member snowfall forecasts. Significance Statement High-resolution numerical weather prediction (NWP) models generally have limited predictive skill for mesoscale snowband forecasts. Even so, some snowbands are forecast by NWP models with much greater skill than others. In this work, we apply artificial intelligence to group snowband events based on atmospheric conditions and then determine whether different groups are easier or harder for models to predict. Identification of these groups could help forecasters know when to trust or be skeptical of NWP output and help developers improve snowband formation processes in NWP models.

Опыт пространственной верификации радиолокационного наукастинга осадков: определение и статистика объектов, ситуаций и условных выборок

Statistical analysis was performed using methods of the extreme value theory for spatial objects and specified situations identified for object-oriented verification of precipitation regions with substantial and maximal areas. We made an estimation of the effect of missing values at field points and of different observation-forecast pairs construction on volumes and on statistical characteristics of samples retrieved for spatial verification purposes. We used spatial quantile functions and geographical representations in regular coordinates to illustrate particular aspects of composite fields built on about three dozen radars' data over the European territory of Russia. Keywords: spatial forecast verification, radar precipitation nowcasting, extreme value theory, missing data, conditional verification sampling, spatial quantiles

Improvement of mesoscale numerical weather prediction WRF-ARW in the Republic of Belarus by assimilation of radar wind and reflectivity data

The forecasts of severe weather events obtained with the WRF numerical mesoscale model with the adapted system for assimilation of reflectivity and radial velocity data from the network of Belarusian Doppler weather radars used in Belhydromet in 2019 are analysed. A description of the system for the echo quality control based on the radar dual-polarisation characteristics and the method for three-dimensional variational assimilation (3D-VAR) used to assimilate data in the WRF model are described. The results of case studies on the simulation of precipitation and strong wind for various circulation types in Belarus with and without radar data assimilation are given. The statistical and object-oriented verification of these forecasts is provided. The results of the comprehensive assessment reveal a decrease in the forecast error for 10-m wind speed for the early forecast hours (+6 h) by 1.34 m/s, as well as a more accurate forecast of the location, orientation of the cloud systems and precipitation zones, and a decrease in the number of false alarms in the version with assimilation. A preliminary conclusion on the possibility of using the forecast results in nowcasting systems is also made.

Evaluation of Rainfall Forecasts by Three Mesoscale Models during the Mei-Yu Season of 2008 in Taiwan. Part III: Application of an Object-Oriented Verification Method

In this study, the performances of Mei-yu (May–June) quantitative precipitation forecasts (QPFs) in Taiwan by three mesoscale models: the Cloud-Resolving Storm Simulator (CReSS), the Central Weather Bureau (CWB) Weather Research and Forecasting (WRF), and the CWB Non-hydrostatic Forecast System (NFS) are explored and compared using an newly-developed object-oriented verification method, with particular focus on the various properties or attributes of rainfall objects identified. Against a merged dataset from ~400 rain gauges in Taiwan and the Tropical Rainfall Measuring Mission (TRMM) data in the 2008 season, the object-based analysis is carried out to complement the subjective analysis in a parallel study. The Mei-yu QPF skill is seen to vary with different aspects of rainfall objects among the three models. The CReSS model has a total rainfall production closest to the observation but a large number of smaller objects, resulting in more frequent and concentrated rainfall. In contrast, both WRF and NFS tend to under-forecast the number of objects and total rainfall, but with a higher proportion of bigger objects. Location errors inferred from object centroid locations appear in all three models, as CReSS, NFS, and WRF exhibit a tendency to simulate objects slightly south, east, and northwest with respect to the observation. Most rainfall objects are aligned close to an E–W direction in CReSS, in best agreement with the observation, but many towards the NE–SW direction in both WRF and NFS. For each model, the objects are matched with the observed ones, and the results of the matched pairs are also discussed. Overall, though preliminarily, the CReSS model, with a finer grid size, emerges as best performing model for Mei-yu QPFs.

An Evaluation of Snowband Predictability in the High-Resolution Rapid Refresh

Abstract Narrow regions of intense, banded snowfall present hazardous travel conditions due to rapid onset, high precipitation rates, and lowered visibility. Despite their importance, there are few verification studies of snowbands in operational forecast models. The objective of this study is to evaluate the ability of the High-Resolution Rapid Refresh (HRRR) model to predict snowbands in the United States east of the Rocky Mountains. An automated band-detection algorithm was applied to a 3-yr period of simulated and observed radar reflectivity to compare snowband climatologies. This algorithm uses the distributions of reflectivities in contiguous precipitation regions to determine a band intensity threshold. The predictability of snowbands on a case-by-case basis was also evaluated using an object-oriented approach. The distribution of HRRR forecast banding resembles that of the observations, but with a significant positive frequency bias. This may partially be due to underrepresentation of observed bands in our verification dataset due to limited radar coverage in portions of the central United States. On a case-by-case basis, traditional skill metrics indicate limited predictability, but allowing for small timing discrepancies dramatically improves scores. Object-oriented verification yields mixed results, with 30% of forecasts receiving a score indicative of a well-predicted event. However, 69% of cases have at least one forecast lead demonstrating skill, suggesting the HRRR is successful in depicting environments conducive to band formation. These results suggest adopting a probabilistic, ensemble approach, and indicate that the deterministic HRRR is best suited for the identification of regions of elevated snowband risk and not precise timing or location information.

New Algorithm for Rain Cell Identification and Tracking in Rainfall Event Analysis

This study proposes a new algorithm termed rain cell identification and tracking (RCIT) to identify and track rain cells from high resolution weather radar data. Previous algorithms have limitations when tracking non-consequent rain cells owing to their use of maximum correlation coefficient methods and their lack of an alternative way to handle the variation stages of rain cells during their life cycles. To address these deficiencies, various methods are implemented in the new algorithm. These include the particle image velocimetry (PIV) method for motion estimation and the rain cell matching rule to obtain the stage changes of rain cells. High resolution (5 min and 1 km) radar data from three rainy days over the German federal state North Rhine Westphalia (NRW) are used in this study. The performance of the identification module for the new algorithm is accessed by two object-oriented verification methods: structure–amplitude–location (SAL) and geometric index, while the performance of the tracking module is compared with TREC and SCOUT tracking algorithms and evaluated by the contingency table verification approach. Results suggest that the performance of the new algorithm is better than reference tracking method. Application of the RCIT algorithm to the selected cases shows that the inner structure of rainfall events in the experimental region present extreme value distributions, with most rainfall events having a short duration with less intensity. The new algorithm can effectively capture the stage changes of rain cells during their life cycles. The proposed algorithm can serve as the basis for further hydro-meteorological applications such as spatial and temporal analysis of rainfall events and short-term flood forecasting.

Impact of Rainfall Assimilation on High-Resolution Hydrometeorological Forecasts over Liguria, Italy

Abstract The autumn of 2014 was characterized by a number of severe weather episodes over Liguria (northern Italy) associated with floods and remarkable damage. This period is selected as a test bed to evaluate the performance of a rainfall assimilation scheme based on the nudging of humidity profiles and applied to a convection-permitting meteorological model at high resolution. The impact of the scheme is assessed in terms of quantitative precipitation forecast (QPF) applying an object-oriented verification methodology that evaluates the structure, amplitude, and location (SAL) of the precipitation field, but also in terms of hydrological discharge prediction. To attain this aim, the meteorological model is coupled with the operational hydrological forecasting chain of the Ligurian Hydrometeorological Functional Centre, and the whole system is implemented taking operational requirements into account. The impact of rainfall data assimilation is large during the assimilation period and still relevant in the following 3 h of the free forecasts, but hardly lasts more than 6 h. However, this can improve the hydrological predictions. Moreover, the impact of the assimilation is dependent on the environment characteristics, being more effective when nonequilibrium convection dominates, and thus an accurate prediction of the local triggering for the development of the precipitation system is required.

Research on HW/SW Co-Verification Platform

The high complexity of very deep-submicron design makes their verification extremely challenging task. Various methodologies such as assertion-based verification and formal verification increase the confidence that a circuit is working according to specification, but in most of cases to guarantee a circuit is 100% compliant to its specifications is very difficulty . In this paper we present a platform for HW/SW Co-verification which takes both the advantages of hardware accelerators and the merit of reusable object-oriented verification method. According to the complex circuits, the experiment results show that this verification platform enhances the verification efficiency and function coverage.

Evaluation of the QPF of convective flash flood rainfalls over the Czech territory in 2009

In this paper, quantitative precipitation forecasts (QPF) are evaluated using several verification techniques and analysis of the results from these techniques. The forecasts were produced by two limited-area numerical weather prediction models: the ALADIN-CZ model operated by the Czech Hydro-Meteorological Institute (CHMI) and the COSMO model operated by the German Weather Service (DWD). Each model was run using two horizontal resolutions over the domain covering the Czech Republic. The ALADIN-CZ model outputs were obtained using resolutions of approximately 9km and 4.7km, and the COSMO model outputs were obtained using resolutions of approximately 7km and 2.8km.The forecast quality is studied for the flash flood period that occurred in June and July of 2009, when convective rainfalls with durations of 1 to 3h and a return period of more than 100years caused devastating floods in many Czech localities. The radar-based rainfalls used to verify the forecasts were produced by the CHMI operational product MERGE, which merges radar-derived rainfalls with the rainfalls that are measured by ground rain gauges.A series of 56 consecutive forecasts of 3-h rainfalls were verified using traditional and spatial verification techniques, and the results from these analyses were compared. The verification was performed using traditional verification scores based on a contingency table, spatial verification by the fractions skill score (FSS) and the SAL (structure–amplitude–location) technique. The FSS represents a fuzzy verification technique and compares the fractional coverage of precipitation grids over a threshold in spatial windows around the observations and forecasts. The SAL is a spatial object-oriented verification technique used to evaluate the structure, amplitude, and location of a precipitation field.The quality of QPF depends strongly on the scale of convective precipitation, and all models provide good forecast quality for extended rainfall systems. The opposite is true for the local and more or less chaotic convection during the final part of the time period. The FSS indicates how the results depend on the threshold and scale of precipitation. The COSMO 2.8 model is able to determine the largest local rainfall values, but models with lower resolution, such as the ALADIN 9km and COSMO 7km, provide better results for lower thresholds and larger scales. The use of more verification techniques is suitable for a modeller-oriented evaluation of different aspects of forecast quality.

McGill algorithm for precipitation nowcasting by lagrangian extrapolation (MAPLE) applied to the South Korean radar network. Part II: Real-time verification for the summer season

The MAPLE system has been implemented in real-time in Korea since June 2008, producing forecasts up to 6 hours every 10 minutes. An object-oriented verification method has been applied for the summer season (June–July–August) over the Korean Peninsula to evaluate and understand the characteristics of the forecast results. The CRA (contiguous rain area) approach is used to decompose the total error into the different error components; location, pattern, and volume errors. The mean displacement error is smaller than 20 km up to the 3-h forecasts and increases with forecast time. The ratio between the displacement (location) error and the total error is less than 7% even for a 3-h forecast. This result indicates that MAPLE produces reliable forecast in terms of precipitation location. However, the pattern error is larger than 90% of the total error. Contingency scores that are defined with different categories of rain intensity and displacement errors show the outstanding performance up to 2.5 hours. MAPLE overpredicts rain areas with the threshold of 1 mm h−1 rain intensity throughout forecast periods. However, the heavy rainfall events are poorly predicted due to the inherent limitation of extrapolation-based nowcasting technique.

Three Spatial Verification Techniques: Cluster Analysis, Variogram, and Optical Flow

Abstract Three spatial verification techniques are applied to three datasets. The datasets consist of a mixture of real and artificial forecasts, and corresponding observations, designed to aid in better understanding the effects of global (i.e., across the entire field) displacement and intensity errors. The three verification techniques, each based on well-known statistical methods, have little in common and, so, present different facets of forecast quality. It is shown that a verification method based on cluster analysis can identify “objects” in a forecast and an observation field, thereby allowing for object-oriented verification in the sense that it considers displacement, missed forecasts, and false alarms. A second method compares the observed and forecast fields, not in terms of the objects within them, but in terms of the covariance structure of the fields, as summarized by their variogram. The last method addresses the agreement between the two fields by inferring the function that maps one to the other. The map—generally called optical flow—provides a (visual) summary of the “difference” between the two fields. A further summary measure of that map is found to yield useful information on the distortion error in the forecasts.

JStar

In this paper we introduce a novel methodology for verifying a large set of Java programs which builds on recent theoretical developments in program verification: it combines the idea of abstract predicate families and the idea of symbolic execution and abstraction using separation logic. The proposed technology has been implemented in a new automatic verification system, called jStar, which combines theorem proving and abstract interpretation techniques. We demonstrate the effectiveness of our methodology by using jStar to verify example programs implementing four popular design patterns (subject/observer, visitor, factory, and pooling). Although these patterns are extensively used by object-oriented developers in real-world applications, so far they have been highly challenging for existing object-oriented verification techniques.

An Object-Oriented Verification of Three NWP Model Formulations via Cluster Analysis: An Objective and a Subjective Analysis

Abstract Recently, an object-oriented verification scheme was developed for assessing errors in forecasts of spatial fields. The main goal of the scheme was to allow the automatic and objective evaluation of a large number of forecasts. However, processing speed was an obstacle. Here, it is shown that the methodology can be revised to increase efficiency, allowing for the evaluation of 32 days of reflectivity forecasts from three different mesoscale numerical weather prediction model formulations. It is demonstrated that the methodology can address not only spatial errors, but also intensity and timing errors. The results of the verification are compared with those performed by a human expert. For the case when the analysis involves only spatial information (and not intensity), although there exist variations from day to day, it is found that the three model formulations perform comparably, over the 32 days examined and across a wide range of spatial scales. However, the higher-resolution model formulation appears to have a slight edge over the other two; the statistical significance of that conclusion is weak but nontrivial. When intensity is included in the analysis, it is found that these conclusions are generally unaffected. As for timing errors, although for specific dates a model may have different timing errors on different spatial scales, over the 32-day period the three models are mostly “on time.” Moreover, although the method is nonsubjective, its results are shown to be consistent with an expert’s analysis of the 32 forecasts. This conclusion is tentative because of the focused nature of the data, spanning only one season in one year. But the proposed methodology now allows for the verification of many more forecasts.

Cluster Analysis for Object-Oriented Verification of Fields: A Variation

Abstract In a recent paper, a statistical method referred to as cluster analysis was employed to identify clusters in forecast and observed fields. Further criteria were also proposed for matching the identified clusters in one field with those in the other. As such, the proposed methodology was designed to perform an automated form of what has been called object-oriented verification. Herein, a variation of that methodology is proposed that effectively avoids (or simplifies) the criteria for matching the objects. The basic idea is to perform cluster analysis on the combined set of observations and forecasts, rather than on the individual fields separately. This method will be referred to as combinative cluster analysis (CCA). CCA naturally lends itself to the computation of false alarms, hits, and misses, and therefore, to the critical success index (CSI). A desirable feature of the previous method—the ability to assess performance on different spatial scales—is maintained. The method is demonstrated on reflectivity data and corresponding forecasts for three dates using three mesoscale numerical weather prediction model formulations—the NCEP/NWS Nonhydrostatic Mesoscale Model (NMM) at 4-km resolution (nmm4), the University of Oklahoma’s Center for Analysis and Prediction of Storms (CAPS) Weather Research and Forecasting Model (WRF) at 2-km resolution (arw2), and the NCAR WRF at 4-km resolution (arw4). In the small demonstration sample herein, model forecast quality is efficiently differentiated when performance is assessed in terms of the CSI. In this sample, arw2 appears to outperform the other two model formulations across all scales when the cluster analysis is performed in the space of spatial coordinates and reflectivity. However, when the analysis is performed only on spatial data (i.e., when only the spatial placement of the reflectivity is assessed), the difference is not significant. This result has been verified both visually and using a standard gridpoint verification, and seems to provide a reasonable assessment of model performance. This demonstration of CCA indicates promise in quickly evaluating mesoscale model performance while avoiding the subjectivity and labor intensiveness of human evaluation or the pitfalls of non-object-oriented automated verification.

Rapid engineering and re-configuration of automation objects aided by formal modelling and verification

This paper is about the engineering of the information processing and control part of intelligent manufacturing systems. A number of recently developed technologies can bring qualitative benefits to the users and vendors of such systems. In particular, component distributed architectures (IEC 61499), mechatronic ontologies, object-oriented design methods (UML) and formal verification, if properly combined, can provide real plug and play of intelligent machines, and thus drastically improve the flexibility of manufacturing systems. The paper surveys important relevant works in the area of intelligent automation and presents new ideas on object-oriented development of automation systems. The ideas presented in this paper reflect current progress of the ongoing R&D project VAIAS, 'Validatable Architectures for Industrial Automation Systems'.

CORBA based design and implementation of universal personal computing

Universal personal computing (UPC) supports nomadic computing at user mobility and at terminal mobility levels in a user-friendly way. That is, a user can access computing resources anywhere on the Internet, using any available mobile or stationary terminal attached to any subnet supporting UPC services. These services are provided via agents and enable a personalized computing environment that is familiar to or customized by the user and independent of the terminal and subnet, utilizing locally available facilities. Hence, UPC requires a homogenous distributed processing environment. We selected CORBA as a natural platform-independent framework for UPC, and Java for the implementation of client side objects and agents of UPC. We developed our design of UPC services in SDL, the formal description technique of the ITU-T, and we used ObjectGEODE which is a commercially available tool set supporting object-oriented design and verification of systems. For the implementation of the prototype we selected Borland's VisiBroker which supports Java implementation of the client objects.

Verification of precipitation in weather systems: determination of systematic errors

An object-oriented verification procedure is presented for gridded quantitative precipitation forecasts (QPFs). It is carried out within the framework of “contiguous rain areas” (CRAs), whereby a weather system is defined as a region bounded by a user-specified isopleth of precipitation in the union of the forecast and observed rain fields. The horizontal displacement of the forecast is determined by translating the forecast rain field until the total squared difference between the observed and forecast fields is minimized. This allows a decomposition of total error into components due to: (a) location; (b) rain volume and (c) pattern. Results are first presented for a Monte Carlo simulation of 40,000 synthetic CRAs in order to determine the accuracy of the verification procedure when the rain systems are only partially observed due to the presence of domain boundaries. Verification is then carried out for operational 24-h forecasts from the Australian Bureau of Meteorology LAPS numerical weather prediction model over a four-year period. Forty-five percent of all rain events were well forecast by the model, with small location and intensity errors. Location error was generally the dominant source of QPF error, with the directions of most frequent displacement varying by region. Forty-five percent of extreme rainfall events (>100 mm d −1) were well forecast, but in this case the model's underestimation of rain intensity was the most frequent source of error.