Weather Forecast Verification Research Articles

Challenges of Operational Weather Forecast Verification and Evaluation

Abstract Operational agencies face significant challenges related to the verification and evaluation of weather forecasts. These challenges were investigated in a series of online workshops and polls engaging operational personnel from six countries. Five key themes emerged: inadequate verification approaches for both existing and emerging products; incomplete and uncertain observations; difficulties in accurately capturing users’ real-world experiences using simplified metrics; poor communication and understanding of forecasts and complex verification information; and institutional factors such as limited resources, evolving meteorologist roles, and concerns over reputational damage. We identify nearly 50 operationally relevant scientific questions and suggest calls to action. Addressing these needs includes designing forecast systems with verification as a central consideration, enhancing the availability of observations, and developing and adopting community software systems. Additionally, we propose the establishment of an international community comprising environmental and social science researchers, statisticians, verification practitioners, and users to provide sustained support for this collective endeavor.

Impacts and Climate Change Adaptation of Agrometeorological Services among the Maize Farmers of West Tamil Nadu

Climate change is often linked with record-breaking heavy or poor rainfall events, unprecedented storms, extreme day and night time temperatures, etc. It may have a marked impact on climate-sensitive sectors and associated livelihoods. Block-level weather forecasting is a new-fangled dimension of agrometeorological services (AAS) in the country and is getting popularized as a climate-smart farming strategy. Studies on the economic impact of these microlevel advisories are uncommon. Agromet advisory services (AAS) play a critical role as an early warning service and preparedness among the maize farmers in the Parambikulam–Aliyar Basin, as this area still needs to widen and deepen its AWS network to reach the village level. In this article, the responses of the maize farmers of Parambikulam–Aliyar Basin on AAS were analyzed. AAS were provided to early and late Rabi farmers during the year 2020–2022. An automatic weather station was installed at the farmers’ field to understand the real-time weather. Forecast data from the India Meteorological Department (IMD) were used to provide agromet advisory services. Therefore, the present study deserves special focus. Social media and other ICT tools were used for AAS dissemination purposes. A crop simulation model (CSM), DSSAT4.7cereal maize, was used for assessing maize yield in the present scenario and under the elevated GHGs scenario under climate change. Our findings suggest that the AAS significantly supported the farmers in sustaining production. The AAS were helpful for the farmers during the dry spells in the late samba (2021–2022) to provide critical irrigation and during heavy rainfall events at the events of harvest during early and late Rabi (2021–22). Published research articles on the verification of weather forecasts from South India are scanty. This article also tries to understand the reliability of forecasts. Findings from the verification suggest that rainfall represented a fairly good forecast for the season, though erratic, with an accuracy score or HI score of 0.77 and an HK score of 0.60, and the probability of detection (PoD) of hits was found to be 0.91. Verification shows that the forecasted relative humidity observed showed a fairly good correlation, with an R2 value of 0.52. These findings suggest that enhancing model forecast accuracy can enhance the reliability and utility of AAS as a climate-smart adaptation option. This study recommends that AAS can act as a valuable input to alleviate the impacts of hydrometeorological disasters on maize crop production in the basin. There is a huge demand for quality weather forecasts with respect to accuracy, resolution, and lead time, which is increasing across the country. Externally funded research studies such as ours are an added advantage to bridge the gap in AAS dissemination to a great extent.

Verification of solar irradiance probabilistic forecasts

We propose a framework for evaluating the quality of solar irradiance probabilistic forecasts. The verification framework is based on visual diagnostic tools and a set of scoring rules mostly originating from the weather forecast verification community. Two types of probabilistic forecasts are used as a basis to illustrate the application of these verification approaches. The first one consists in ensemble forecasts commonly provided by national or international meteorological centres. The second one originates from statistical methods and produces a set of discrete quantile forecasts, the nominal proportions of which span the unit interval. These probabilistic forecasts are evaluated for two selected sites that experience very different climatic conditions. The first site is located in the continental US while the second one is situated on La Réunion Island. Although visual diagnostic tools can help identify deficiencies in generated forecasts, it is recommended that a set of numerical scores be used to assess the quality of probabilistic forecasts. In particular, the Continuous Ranked Probability Score (CRPS) seems to have all the features needed to evaluate a probabilistic forecasting system and, as such, may become a standard for verifying solar irradiance probabilistic forecasts and by extension probabilistic forecasts of solar power generation.

A new approach to testing forecast predictive accuracy

The Diebold–Mariano test for predictive accuracy has been used widely and adapted for economic forecasts, but has not seen much activity in weather forecast verification. The technique is applied to both simulated verification sets as well as weather data at eight stations in Utah, and a loss function based on dynamic time warping (DTW) is used. Results of the simulation experiment show that the DTW technique can be useful if timing errors are the concern. Real test cases demonstrate the difficulty in automating some of the more advanced methods proposed here, but also show the utility in even the most basic test, which is an improvement over similar tests that do not account for temporal and/or contemporaneous correlation.

Use of past precipitation data for regionalisation of hourly rainfall in the low mountain ranges of Saxony, Germany

Abstract. Within the context of flood forecasting we deal with the improvement of regionalisation methods for the generation of highly resolved (1 h, 1×1km2) precipitation fields, which can be used as input for rainfall-runoff models or for verification of weather forecasts. Although radar observations of precipitation are available in many regions, it might be necessary to apply regionalisation methods near real-time for the cases that radar is not available or observations are of low quality. The aim of this paper is to investigate whether past precipitation information can be used to improve regionalisation of rainfall. Within a case study we determined typical precipitation Background-Fields (BGF) for the mountainous and hilly regions of Saxony using hourly and daily rain gauge data. Additionally, calibrated radar data served as past information for the BGF generation. For regionalisation of precipitation we used de-trended kriging and compared the results with another kriging based regionalisation method and with Inverse Distance Weighting (IDW). The performance of the methods was assessed by applying cross-validation, by inspection and by evaluation with rainfall-runoff simulations. The regionalisation of rainfall yielded better results in case of advective events than in case of convective events. The performance of the applied regionalisation methods showed no significant disagreement for different precipitation types. Cross-validation results were rather similar in most cases. Subjectively judged, the BGF-method reproduced best the structures of rain cells. Precipitation input derived from radar or kriging resulted in a better matching between observed and simulated flood hydrographs. Simple techniques like IDW also deliver satisfying results in some occasions. Implementation of past radar data into the BGF-method rendered no improvement, because of data shortages. Thus, no method proved to outperform the others generally. The decision, which method is appropriate for an event, should be made objectively using cross-validation, but also subjectively, using the expert knowledge of the forecaster.

A review on aspects of climate simulation assessment

This paper reviews some aspects of evaluation of climate simulation, including the ITCZ, the surface air temperature (SAT), and the monsoon. A brief introduction of some recently proposed approaches in weather forecast verification is followed by a discussion on their possible application to evaluation of climate simulation. The authors suggest five strategies to extend the forecast verification methods to climate simulation evaluation regardless significant differences between the forecasts and climate simulations. It is argued that resolution, convection scheme, stratocumulus cloud cover, among other processes in the atmospheric general circulation model (AGCM) and the ocean-atmosphere feedback are the potential causes for the double ITCZ problem in coupled models and AGCM simulations, based on the system- and component-level evaluations as well as the downscaling strategies in some recent research. Evaluations of simulated SAT and monsoons suggest that both coupled models and AGCMs show good performance in representing the SAT evolution and its variability over the past century in terms of correlation and wavelet analysis but poor at reproducing rainfall, and in addition, the AGCM alone is not suitable for monsoon regions due to the lack of air-sea interactions.

On the appropriateness of inappropriate VaR models

The Value-at-Risk calculation reduces the dimensionality of the risk factor space. The main reasons for such simplifications are, e. g., technical efficiency, the logic and statistical appropriateness of the model. In Chapter 2 we present three simple mappings: the mapping on the market index, the principal components models and the model with equally correlated risk factors. The comparison of these models in Chapter 3 is based on the literature on the verification of weather forecasts (Murphy and Winkler, 1992; Murphy, 1997). Some considerations on the quantitative analysis are presented in the fourth chapter. In the last chapter, we present empirical analysis of the DAX data using XploRe.

On the Appropriateness of Inappropriate VAR Models

The Value-at-Risk calculation reduces the dimensionality of the risk factor space. The main reasons for such simplifications are, e.g., technical efficiency, the logic and statistical appropriateness of the model. In Chapter 2 we present three simple mappings: the mapping on the market index, the principal components model and the model with equally correlated risk factors. The comparison of these models in Chapter 3 is based on the literatere on the verification of weather forecasts (Murphy and Winkler 1992, Murphy 1997). Some considerations on the quantitative analysis are presented in the fourth chapter. In the last chapter, we present empirical analysis of the DAX data using XploRe.

The Application of a Simple Method for the Verification of Weather Forecasts and Seasonal Variations in Forecast Accuracy

The evaluation of weather forecast accuracy has always been a difficult subject to address for many reasons. In this study, a simple semiobjective method is used to examine the accuracy of zone forecasts issued by the Weldon Spring (Saint Louis) National Weather Service (NWS) Office for mid-Missouri over a period of 416 days with the goal of demonstrating the utility of this method. Zone forecasts were chosen because these forecasts are typically what the public will receive either directly or indirectly from various media outlets. Not surprising, the evaluation method used here demonstrates that forecasts issued by the NWS and the Nested Grid Model (NGM) model output statistics (MOS) represent a considerable improvement over persistence or climatological baseline forecasts. NWS forecasts were slightly better than NGM MOS forecasts, especially when considering temperature and precipitation only. All forecasts showed distinct seasonal variability. The NWS winter-season forecasts were superior to those issued in the summer season, and this superiority was found to be a function of the precipitation forecast parameter. This technique might represent an easily understandable and concise method for providing weather forecast performance information to the general public in such a way that it would instill or reinforce public confidence in the accuracy of weather forecasts.

The quality and accuracy of a sample of public and commercial weather forecasts in the UK

AbstractAt a time when the commercialisation of weather forecasting is proceeding at an alarming rate around the world, it is essential to review the quality and accuracy of weather forecasts to ensure that current standards are at least maintained. The privatisation of meteorological services in some countries (e.g. New Zealand) and the vigorous commercialisation of established Met. Offices such as in the United Kingdom, France and Sweden suggests that customer satisfaction, increased income and reduced expenditure will dominate the weather forecasting scene over the next decade. It is important that the quality of weather forecasts is closely monitored before and after commercialisation to ensure that the accuracy of weather forecasts continues to improve and that public weather forecasting does not suffer. The improved technologies for data collection such as weather satellites, weather radar, wind profilers, road weather information systems, sferics, etc. should make it easier to assess forecast accuracy ‐ but with the exception of road weather forecasts there is little sign that the new technology has been used for this purpose. The current accuracy of a sample of weather forecasts in the UK is discussed and a framework for publishing future weather forecast verification statistics is presented.

A Comparison of Three Radar-Based Severe-Storm-Detection Algorithms on Colorado High Plains Thunderstorms

Volumetric reflectivity data collected with the National Center for Atmospheric Research's (NCAR) 10-cm Doppler radar during the Program for Regional Observing and Forecasting Services (PROFS) 1985 Real-Time Forecast Exercise (RT-85) have been used to evaluate the Next Generation Weather Radar (NEXRAD) hail algorithm, a modified version of the Radar Data Processor (RADAP) II-based Severe Weather Probability (SWP) algorithm, tuned for Colorado High Plains thunderstorms, and a Precipitation Type/Intensity (PTI) hail detection algorithm that uses dual-polarization of echo return. Comparative analyses of algorithm performance are presented using two statistical scoring techniques. The results are consistent for the two methods. The SWP algorithm exhibited the best performance when a 0.10 probability threshold was used for the hail/no hail decision criterion. Probability of detection, false alarm rate and critical success index scores for the SWP algorithm were 0.58, 0.71 and 0.23, respectively. Application of signal detection theory (SDT) analysis indicated SWP to be statistically better at forecasting hail than either the PTI or NEXRAD algorithms at the 0.95 confidence level. A discussion of SDT and its application to severe weather forecast verification and the sensitivity of SDT to sample selection is presented.

Verification and Further Development of an Operational Model for Forecasting the Probability of Frozen Precipitation

Abstract An automated system for forecasting the conditional probability of frozen precipitation was put into operation by the National Weather Service in November 1972. The Model Output Statistics (MOS) concept was used to develop the system, and both teletypewriter and facsimile products have been distributed to field offices twice daily. In this paper, guidance forecasts from this system are compared to subjective (local) forecasts prepared at Weather Service Forecast Offices. The local forecasts have been archived since September 1973 as part of a combined aviation/public weather forecast verification program within the National Weather Service. The comparative verification between the guidance and locals for two different data samples shows the guidance has produced better forecasts for the 18, 30, and 42 h projections. In attempting to improve the operational system, we experimented with the Regression Estimation of Event Probabilities (REEP) screening technique. The operational system had been deve...

Discussion on the Papers by Professor Tversky and by Professor Suppes

Journal of the Royal Statistical Society: Series B (Methodological)Volume 36, Issue 2 p. 175-191 DiscussionFree Access Discussion on the Papers by Professor Tversky and by Professor Suppes First published: January 1974 https://doi.org/10.1111/j.2517-6161.1974.tb00998.xAboutPDF ToolsExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat References in the Discussion Beach, L. R. and Scopp, T. S. (1968). Intuitive statistical inferences about variances. Organ. Behav. and Hum. Perf., 3, 109– 123. Brier, G. W. and Allen, R. A. (1951). Verification of weather forecasts. In Compendium of Meteorology ( T. F. Malone, ed.), pp. 841– 848. Boston, Mass.: American Meteorological Society. De Finetti, B. (1965). Methods of discriminating levels of partial knowledge concerning a test item. Brit. J. Math. Statist. Psychol., 13, 87– 123. Dickey, J. M. and Walrath, Judy (1971). Computers and Bayesian statistical inference in the analysis of clinical data. Res. Rep. No. 56, Dept of Statistics, State University of New York at Buffalo. Godambe, V. P. and Thompson, Mary E. (1971). Bayes, fiducial and frequency aspects of statistical inference in regression analysis and survey-sampling. J. R. Statist. Soc. B, 33, 361– 390. Godambe, V. P. and Thompson, Mary E. (1973). Philosophy of survey-sampling practice. Paper delivered at the London (Ontario) meetings (to be published). Krantz, D. H., Luce, R. D., Suppes, P. and Tversky, A. (1971). Foundations of Measurement, Vol. 1. New York: Academic Press. Rapoport, A. (1973). Research paradigms for studying dynamic decision behaviour. Paper delivered at the Fourth Research Conference on Subjective Probability, Utility and Decisionmaking (to be published). Salway, J. (1971). Unpublished paper, with the Department of Psychology, University of Liverpool. Sanders, F. (1958). The evaluation of subjective probability forecasts. Scientific Report No. 5, Contract No. AFCRC-TN-58-465, M.I.T., Cambridge, Mass. Shuford, E. H. (1964). Some Bayesian learning processes. In Human Judgments and Optimality ( M. S. Shelly and G. L. Bryan, eds), pp. 127– 152. New York: Wiley. Shuford, E. H., Albert, A. and Massengill, H. (1966). Admissible probability measurement procedures. Psychometrika, 31, 125– 145. Slovic, P. and Lichtenstein, S. C. (1968). The relative importance of probabilities and payoffs in risk taking. J. Exper. Psychol. Mono. Suppl., 78, No. 3, Part 2. Suppes, P. (1973). New Foundations of objective probability: Axioms for propensities. In Logic, Methodology and Philosophy of Science ( P. Suppes et al., eds). Amsterdam: North-Holland. Toda, M. (1962). The design of a fungus-eater: A model of human behaviour in an unsophisticated environment. Behav. Sci., 7, 164– 183. Volume36, Issue2January 1974Pages 175-191 This article also appears in:Discussion Papers ReferencesRelatedInformation

Scientific Serials

THE American Meteorological Journal for September contains (1) an article by Prof. H. A. Hazen on cloud formation. The author considers that theories of storm generation and cloud formation are unsatisfactory. He has made a number of experiments, both with dry and damp air, on the formation of cloud by the cooling produced by exhaustion, the result being that he finds the amount of the latter to be only about one-fourth of what theory would indicate. The results obtained with dry and moist air were almost exactly the same. (2) State tornado charts for Alabama and Ohio, by Lieutenant Finley. (3) An article on the verification of weather forecasts, by H. Helm Clayton. He points out that all the methods adopted admit of considerable latitude in interpreting phenomena, so that the same forecasts, when verified by different persons, may differ widely as to the percentage of success, and he suggests an arrangement which may be applied to areas as well as to single stations, and which, at the same time, might be useful in studying the relative frequency of each phenomenon. (4) The distribution of wind velocities in the United States, by Dr. F. Waldo, deduced from the records of the self-recording anemometers at the statiors of the Signal Service. In this article, which is unfinished, the author deals exclusively with the values of the constants derived from various anemometrical experiments in Germany and Russia.