Use Of Forecasts Research Articles

Aims, challenges and progress of the Hydrological Ensemble Prediction Experiment (HEPEX) following the third HEPEX workshop held in Stresa 27 to 29 June 2007

AbstractSince several years, users of weather forecasts have begun to realize the benefit of quantifying the uncertainty associated with forecasts rather than relying on single value forecasts. At the same time, hydrologists and water managers have begun to explore the potential benefit of ensemble prediction systems (EPS) for hydrological applications. The Hydrologic Ensemble Prediction Experiment (HEPEX) is an international project that aims to foster the development of probabilistic hydrological forecasting and corresponding decision making tools. Since 2004, HEPEX has provided discussion opportunities for hydrological and meteorological scientists involved in the development, testing, and operational management of forecasting systems, and end users. Copyright © 2008 Royal Meteorological Society

How much does simplification of probability forecasts reduce forecast quality?

AbstractProbability forecasts from an ensemble are often discretized into a small set of categories before being distributed to the users. This study investigates how such simplification can affect the forecast quality of probabilistic predictions as measured by the Brier score (BS). An example from the European Centre for Medium‐Range Weather Forecasts (ECMWF) operational seasonal ensemble forecast system is used to show that the simplification of the forecast probabilities reduces the Brier skill score (BSS) by as much as 57% with respect to the skill score obtained with the full set of probabilities issued from the ensemble. This is more obvious for a small number of probability categories and is mainly due to a decrease in forecast resolution of up to 36%. The impact of the simplification as a function of the ensemble size is also discussed. The results suggest that forecast quality should be made available for the set of probabilities that the forecast user has access to as well as for the complete set of probabilities issued by the ensemble forecasting system. Copyright © 2008 Royal Meteorological Society

Forecast verification: current status and future directions

AbstractResearch and development of new verification strategies and reassessment of traditional forecast verification methods has received a great deal of attention from the scientific community in the last decade. This scientific effort has arisen from the need to respond to changes encompassing several aspects of the verification process, such as the evolution of forecasting systems, or the desire for more meaningful verification approaches that address specific forecast user requirements. Verification techniques that account for the spatial structure and the presence of features in forecast fields, and which are designed specifically for high‐resolution forecasts have been developed. The advent of ensemble forecasts has motivated the re‐evaluation of some of the traditional scores and the development of new verification methods for probability forecasts. The expected climatological increase of extreme events and their potential socio‐economical impacts have revitalized research studies addressing the challenges concerning extreme event verification. Verification issues encountered in the operational forecasting environment have been widely discussed, verification needs for different user communities have been identified, and models to assess the forecast value for specific users have been proposed. Proper verification practice and correct interpretation of verification statistics has been extensively promoted with recent publications and books, tutorials and workshops, and the development of open‐source software and verification tools. This paper addresses some of the current issues in forecast verification, reviews some of the most recently developed verification techniques, and provides recommendations for future research. Copyright © 2008 Royal Meteorological Society and Crown in the right of Canada.

Multiple horizons and information in USDA production forecasts

AbstractUnited States Department of Agriculture (USDA) livestock production forecasts are evaluated for their information content across multiple forecast horizons using the direct test developed by Vuchelen and Gutierrez (2005). Forecasts are explicitly tested for rationality (unbiased and efficient) as well as for incremental information out to three‐quarters ahead. The results suggest that although the forecasts are often not rational, they typically do provide the forecast user with unique information at each horizon. Turkey and milk production forecasts are found to provide the most consistent performance, while beef production forecasts provide little information beyond the two‐quarter horizon. [C53, Q13] © 2008 Wiley Periodicals, Inc.

A Practical Approach to Sequential Estimation of Systematic Error on Near-Surface Mesoscale Grids

Abstract Statistical analysis arguments are used to construct an estimation algorithm for systematic error of near-surface temperatures on a mesoscale grid. The systematic error is defined as the observed running-mean error, and an averaging length of 7 days is shown to be acceptable. Those errors are spread over a numerical weather prediction model grid via the statistical analysis equation. Two covariance models are examined: 1) a stationary, isotropic function tuned with the observed running-mean errors and 2) dynamic estimates derived from a recent history of running-mean forecasts. Prediction of error is possible with a diurnal persistence model, where the error at one time of day can be estimated from data with lags of 24-h multiples. The approach is tested on 6 months of 6-h forecasts with the fifth-generation Pennsylvania State University–NCAR Mesoscale Model (MM5) over New Mexico. Results show that for a quantity such as 2-m temperature, the systematic component of error can be effectively predicted on the grid. The gridded estimates fit the observed running-mean errors well. Cross validation shows that predictions of systematic error result in a substantial error reduction where observations are not available. The error estimates show a diurnal evolution, and are not strictly functions of terrain elevation. Observation error covariances, localization operators, and covariance functions in the isotropic case must be tuned for a specific forecast system and observing network, but the process is straightforward. Taken together, the results suggest an effective method for systematic error estimation on near-surface mesoscale grids in the absence of a useful ensemble. Correction for those errors may provide benefits to forecast users.

Methods to Elicit Forecasts from Groups: Delphi and Prediction Markets Compared

Traditional groups meetings are an inefficient and ineffective method for making forecasts and decisions. We compare two structured alternatives to traditional meetings: the Delphi technique and prediction markets. Delphi is relatively simple and cheap to implement and has been adopted for diverse applications in business and government since its origins in the 1950s. It can be used for nearly any forecasting, estimation, or decision making problem not barred by complexity or ignorance. While prediction markets were used more than a century ago, their popularity waned until more recent times. Prediction markets can be run continuously, and they motivate participation and participants to reveal their true beliefs. On the other hand, they need many participants and clear outcomes in order to determine pay-offs. Moreover, translating knowledge into a price is not intuitive to everyone and constructing contracts that will provide a useful forecast may not be possible for some problems. It is difficult to maintain confidentiality with markets and they are vulnerable to manipulation. Delphi is designed to reveal panelists' knowledge and opinions via their forecasts and the reasoning they provide. This format allows testing of knowledge and learning by panelists as they refine their forecasts but may also lead to conformity due to group pressure. The reasoning provided as an output of the Delphi process is likely to be reassuring to forecast users who are uncomfortable with the black box nature of prediction markets. We consider that, half a century after its original development, Delphi is under-utilized.

Ensemble Calibration of 500-hPa Geopotential Height and 850-hPa and 2-m Temperatures Using Reforecasts

Abstract An examination of the benefits of ensemble forecast calibration was performed for three variables: 500-hPa geopotential height (Z500), 850-hPa temperature (T850), and 2-m temperature (T2M). A large reforecast dataset was used for the calibration. Two calibration methods were examined: a correction for a gross bias in the forecast and an analog method that implicitly adjusted for bias, spread, and applied a downscaling where appropriate. The characteristics of probabilistic forecasts from the raw ensemble were also considered. Forecasts were evaluated using rank histograms and the continuous ranked probability skill score. T2M rank histograms showed a high population of extreme ranks at all leads, and a correction for model bias alleviated this only slightly. The extreme ranks of Z500 rank histograms were slightly underpopulated at short leads, though slightly overpopulated at longer leads. T850 had characteristics in between those of T2M and Z500. Accordingly, Z500 was the most skillful variable without calibration and the variable least improved by calibration, and the bias correction achieved most of the improvement in skill. For T850, there was a more substantial additional increase in skill relative to the bias correction when the analog technique was applied. For T2M forecasts, probabilistic forecasts from the raw ensemble were the least skillful, the application of a bias correction substantially increased the skill, and the application of the analog technique produced the largest further increase in skill relative to the bias correction. Hence, reforecast datasets may be particularly helpful in the improvement of probabilistic forecasts of the variables that are most directly relevant to many forecast users (i.e., the sensible surface-weather variables).

Use of Information by National Weather Service Forecasters and Emergency Managers during CALJET and PACJET-2001

Abstract Winter storms making landfall in western North America can generate heavy precipitation and other significant weather, leading to floods, landslides, and other hazards that cause significant damage and loss of life. To help alleviate these negative impacts, the California Land-falling Jets (CALJET) and Pacific Land-falling Jets (PACJET) experiments took extra meteorological observations in the coastal region to investigate key research questions and aid operational West Coast 0–48-h weather forecasting. This article presents results from a study of how information provided by CALJET and PACJET was used by National Weather Service (NWS) forecasters and forecast users. The primary study methodology was analysis of qualitative data collected from observations of forecasters and from interviews with NWS personnel, CALJET–PACJET researchers, and forecast users. The article begins by documenting and discussing the many types of information that NWS forecasters combine to generate forecasts. Within this context, the article describes how forecasters used CALJET–PACJET observations to fill in key observational gaps. It then discusses researcher–forecaster interactions and examines how weather forecast information is used in emergency management decision making. The results elucidate the important role that forecasters play in integrating meteorological information and translating forecasts for users. More generally, the article illustrates how CALJET and PACJET benefited forecasts and society in real time, and it can inform future efforts to improve human-generated weather forecasts and future studies of the use and value of meteorological information.

The Future of Medium–Extended-Range Weather Prediction: User Perspectives

AMERICAN METEOROLOGICAL SOCIETY | MAY 2007 | 635 634 A long with the increased skill of the short-term (1–3 day) weather forecasts, the prospects for accurate 3–14 day forecasts have improved, thereby opening up new possibilities for forecast users. In the United States, the National Weather Service (NWS)’s Hydrometeorological Prediction Center (HPC) currently provides forecast products and guidance for the 3–7-day range, including maximum and minimum temperature, surface circulation and fronts, and precipitation probabilities, while the NWS Climate Prediction Center is responsible for longer-range forecasts such as the 6–10and 8–14-day probability outlooks for temperature and precipitation. The potential and current uses of these products for agriculture, aviation, utilities, and commodities are numerous, but can only be fully and productively realized if the economic, educational, and decision-making needs of specific users are taken into account. Discussions of user needs with our colleagues have yielded an outline of some of these opportunities and challenges of mediumand extended-range forecasts.1

Population forecast accuracy: does the choice of summary measure of error matter?

Population projections are judged primarily by their accuracy. The most commonly used measure for the precision component of accuracy is the mean absolute percent error (MAPE). Recently, the MAPE has been criticized for overstating forecast error and other error measures have been proposed. This study compares the MAPE with two alternative measures of forecast error, the Median APE and an M-estimator. In addition, the paper also investigates forecast bias. The analysis extends previous studies of forecast error by examining a wide range of trend extrapolation techniques using a dataset that spans a century for a large sample of counties in the US. The main objective is to determine whether the choice of summary measure of error makes a difference from a practitioner’s standpoint. The paper finds that the MAPE indeed produces error values that exceed the robust measures. However, except for situations where extreme outliers rendered the MAPE meaningless, and which are rare in real world applications, there was not a single instance where using an alternative summary measure of error would have led to a fundamentally different evaluation of the projections. Moreover, where differences existed, it was not always clear that the values and patterns provided by the robust measures were necessarily more correct than those obtained with the MAPE. While research into refinements and alternatives to the MAPE and mean algebraic percent error are worthwhile, consideration of additional evaluation procedures that go beyond a single criterion might provide more benefits to producers and users of population forecasts.

ViLab: A Virtual Laboratory for Collaborative Research on Wind Power Forecasting

The consortium of the European Coordination Action ‘Prediction of Waves, Wakes and Offshore Wind’ (POW'WOW) has established a Virtual Laboratory (ViLab) for the development and operational use of wind power prediction systems. The objectives are to stimulate research in this field, to tighten the collaboration between forecasters and forecast users, and to follow and communicate the state of the art in short term prediction of wind generation. The present note clarifies the benefits for the participants to the ViLab. Also, it describes the methodology employed and some practical issues related to data and confidentiality.

AHPSVER: A web-based system for hydrologic forecast verification

Hydrologic forecast verification is a complicated endeavor due to the size and multi-dimensionality of the data sets involved. Finding an effective way to examine and interpret these data sets in ways that are meaningful to forecasters and forecast users is a central challenge in determining how good the forecasts are. We have developed the (AHPSVER) system to address these issues. The AHPSVER system consists of two components. The first is a comprehensive archive of verification data. The second component is a fully dynamic, relational database back-ended website that allows users to view the verification data in individual, customized ways. The system uses a maps- and forms-based interface to make interaction with users easy. The system outputs several types of graphs complemented with context-sensitive help. These graphs are pre-computed when possible (saving time and computation resources) and otherwise generated by the system on demand. These allow geographically diverse users to conduct individual, customized, and interactive analyses of the forecast verification data set.

Predicting magnetopause crossings at geosynchronous orbit during the Halloween storms

In late October and early November of 2003, the Sun unleashed a powerful series of events known as the Halloween storms. The coronal mass ejections launched by the Sun produced several severe compressions of the magnetosphere that moved the magnetopause inside of geosynchronous orbit. Such events are of interest to satellite operators, and the ability to predict magnetopause crossings along a given orbit is an important space weather capability. In this paper we compare geosynchronous observations of magnetopause crossings during the Halloween storms to crossings determined from the Lyon‐Fedder‐Mobarry global magnetohydrodynamic simulation of the magnetosphere as well to predictions of several empirical models of the magnetopause position. We calculate basic statistical information about the predictions as well as several standard skill scores. We find that the current Lyon‐Fedder‐Mobarry simulation of the storm provides a slightly better prediction of the magnetopause position than the empirical models we examined for the extreme conditions present in this study. While this is not surprising, given that conditions during the Halloween storms were well outside the parameter space of the empirical models, it does point out the need for physics‐based models that can predict the effects of the most extreme events that are of significant interest to users of space weather forecasts.

Measuring the Ensemble Spread–Error Relationship with a Probabilistic Approach: Stochastic Ensemble Results

Abstract One widely accepted measure of the utility of ensemble prediction systems is the relationship between ensemble spread and deterministic forecast accuracy. Unfortunately, this relationship is often characterized by spread–error linear correlations, which oversimplify the true spread–error relationship and ignore the possibility that some end users have categorical sensitivities to forecast error. In the present paper, a simulation study is undertaken to estimate the idealized spread–error statistics for stochastic ensemble prediction systems of a finite size. Under a variety of spread–error metrics, the stochastic ensemble spread–error joint distributions are characterized by increasing scatter as the ensemble spread grows larger. A new method is introduced that recognizes the inherent nonlinearity of spread–error joint distributions and capitalizes on the fact that the probability of large forecast errors increases with ensemble spread. The ensemble spread–error relationship is measured by the skill of probability forecasts that are constructed from a history of ensemble-mean forecast errors using only cases with similar ensemble spread. Thus, the value of ensemble spread information is quantified by the ultimate benefit that is realized by end users of the probability forecasts based on these conditional-error climatologies. It is found that the skill of conditional-error-climatology forecasts based on stochastic ensemble spread is nearly equal to the skill of probability forecasts constructed directly from the raw ensemble statistics. The skill is largest for cases with anomalous spread and smallest for cases with near-normal spread. These results reinforce the findings of earlier studies and affirm that the temporal variability of ensemble spread controls its potential value as a predictor. Additionally, it is found that the skill of spread-based error probability forecasts is maximized when the chosen spread metric is consistent with the end user’s cost function.

Generalised Rational Bias in Financial Forecasts

Laster et al. (Q J Econ 114(1):293–318, 1999) built an economic model in which forecasters have incentives to generate forecasts that differ form the consensus. It is shown that the dispersion of the equilibrium distribution of forecasters, depends on the relative importance given on the intensive forecast users’ loss versus the publicity gain from occasional users. These results depend heavily on the assumption of symmetry for the loss and density functions. In this paper we examine the effects of generalising loss preferences and probability densities to allow for asymmetries through the LinEx loss and the Skewed Normal density, respectively. We derive the generalised equilibrium distribution of forecasts which contains the results of Laster et al. as a special case. The presence of asymmetric preferences is shown to cause a movement of the distribution away from the conditional mean, towards the optimal forecast under loss asymmetry. Furthermore, forecasts now tend to cluster around this quantity in an asymmetric way. These effects tend to be further strengthened or partially offset by the presence of skewness in the distribution of data, a result consistent with the conclusions of Christodoulakis (Finan Res Lett 2:227–233, 2005).

Verification and value of the Australian Bureau of Meteorology township seasonal rainfall forecasts in Australia, 1997–2005

AbstractWe verified the Bureau of Meteorology's seasonal rainfall forecasts for 262 townships throughout Australia, from its inception in June 1997 to May 2005. The results indicate that the forecasting system had low skill. Brier Skill Score and the receiver operating characteristic values were uniformly close to the no skill value. Forecast variances were consistently small. The overall observed variance was 0.0048, 2.1% of the variance of a perfect system. The estimate of the gradient of the outcome against forecast was 0.42 and was imprecise. Definitive statements about bias cannot be made. The value of the forecasts for decision‐makers was estimated using value score curves, calculated for six forecast scenarios. All curves indicated that no economic benefit could have been reliably derived by users of the seasonal rainfall forecasts, with the exception of users with decisions triggered by a small shift in the forecast from climatology, in which case small economic gains may have occurred. Small value scores were associated with the observed forecast variance, not the observed bias. We examined the expected change in value scores associated with any future increase in forecast variance. This showed that a moderate increase from the observed variance would bring limited benefits. Substantial value to a broad range of users will only occur with a large increase in forecast variance. To deliver this, new lead indicators with markedly better predictive characteristics may need to be developed for the seasonal rainfall forecasting system. Copyright © 2005 Royal Meteorological Society

Choosing words in computer-generated weather forecasts

One of the main challenges in automatically generating textual weather forecasts is choosing appropriate English words to communicate numeric weather data. A corpus-based analysis of how humans write forecasts showed that there were major differences in how individual writers performed this task, that is, in how they translated data into words. These differences included both different preferences between potential near-synonyms that could be used to express information, and also differences in the meanings that individual writers associated with specific words. Because we thought these differences could confuse readers, we built our SumTime- Mousam weather-forecast generator to use consistent data-to-word rules, which avoided words which were only used by a few people, and words which were interpreted differently by different people. An evaluation by forecast users suggested that they preferred SumTime-Mousam's texts to human-generated texts, in part because of better word choice; this may be the first time that an evaluation has shown that nlg texts are better than human-authored texts.

Do the OECD 24 month horizon growth forecasts for the G7–countries contain information?

The OECD produces two–year–ahead growth forecasts for the G7–countries since 1987; these forecasts have never been evaluated. A regression is developed that tests for the information content of the forecasts. The idea is that this content is the added value forecasters incorporate in their forecasts. The information content is defined relative to the forecast for the previous year. In the end, the added value contained in the current year forecast is calculated relative to the last observation. The test consists in checking whether the information content reduces the forecasts error. 1 The authors of the present study refer to Mincer and Zarnowitz (1969), McNees (1988), Stekler (1991) and Fildes and Stekler (2002) for a discussion of the evaluation techniques of forecasts. The study begins with a calculation of the usual accuracy statistics. These indicate an extreme low quality for the forecasts. The regression tests support this conclusion although the forecasts for Japan do possess some information. Alarming for users of forecasts is that there are no obvious alternatives.

The Uncertain Value of Perfect ENSO Phase Forecasts: Stochastic Agricultural Prices and Intra-Phase Climatic Variations

Climate and crop yield variability associated with El Nino—Southern Oscillation (ENSO) are now predictable within limits. This predictability suggests a potential to tailor agricultural management to mitigate impacts of adverse conditions and to take advantage of favorable conditions. However, improved climate predictions may benefit society only with parallel advances in our ability to use this knowledge. We show that the value that will accrue to any given actor from an ENSO phase forecast should be viewed not as a known number but instead as a random draw from a distribution, even when the forecast is always correct. Forecast value depends on the highly variable contexts in which forecasts are used. Randomness in forecast value has significant implications for choices made by forecasters, forecast users and policy makers. To show randomness, we estimate potential economic values of ENSO forecasts for agricultural producers based on two realistic assumptions: the crop prices farmers receive are uncertain; and within an ENSO phase, the actual climate is variable in ways that affect profits. The use of synthetic weather and crop price series, with crop simulation models, helps show the range and likelihood of climate forecast value.

The 1998 Yangtze Floods: The Use of Short-Term Forecasts in the Context of Seasonal to Interannual Water Resource Management

This paper reviews changes in the use of short-term climate information for water management in China after the 1998 Great Flood in the Yangtze River basin. This devastating flood is now believed to have been caused mainly by the 1997–98 El Nino event. Although the short-term climate forecasts and weather forecasts are considered to be useful in planning for flood prevention activities and for making key decisions during combating floods, the gap between the meteorological services (producers of climate forecasts) and water management agencies (users of climate forecasts) has grown in terms of credibility given to climate forecasting: weather services put more efforts on improving the technology for increasing forecast accuracy, whereas water managers put their efforts and investment into upstream ecological restoration and flood control systems. By reviewing the published and ‘gray’ (unpublished) literature, we found that assessments of the 1998 Great Flood in the Yangtze River basin really helped the central government and water resources agencies to recognize the weaknesses of the existing flood control system, the mismanagement in the ecological systems, and the need for developing a national water resource management plan to deal with the problems of ‘too much water, too little water, and very polluted water’.