Sources Of Forecast Error Research Articles

Evaluation of Land–Atmosphere Coupling Processes and Climatological Bias in the UFS Global Coupled Model

Abstract This study investigates the performance of the latter NCEP Unified Forecast System (UFS) Coupled Model prototype simulations (P5–P8) during boreal summer 2011–17 in regard to coupled land–atmosphere processes and their effect on model bias. Major land physics updates were implemented during the course of model development. Namely, the Noah land surface model was replaced with Noah-MP and the global vegetation dataset was updated starting with P7. These changes occurred along with many other UFS improvements. This study investigates UFS’s ability to simulate observed surface conditions in 35-day predictions based on the fidelity of model land surface processes. Several land surface states and fluxes are evaluated against flux tower observations across the globe, and segmented coupling processes are also diagnosed using process-based multivariate metrics. Near-surface meteorological variables generally improve, especially surface air temperature, and the land–atmosphere coupling metrics better represent the observed covariance between surface soil moisture and surface fluxes of moisture and radiation. Moreover, this study finds that temperature biases over the contiguous United States are connected to the model’s ability to simulate the different balances of coupled processes between water-limited and energy-limited regions. Sensitivity to land initial conditions is also implicated as a source of forecast error. Above all, this study presents a blueprint for the validation of coupled land–atmosphere behavior in forecast models, which is a crucial model development task to assure forecast fidelity from day one through subseasonal time scales.

Warm conveyor belts as amplifiers of forecast uncertainty

AbstractWarm conveyor belts (WCBs) are rapidly ascending air streams associated with extratropical cyclones. WCBs exert a substantial influence on the evolution of the large‐scale midlatitude flow and have previously been related to increased forecast uncertainty in case studies. This study provides a first systematic investigation of the role of WCBs for errors in medium‐range ensemble forecasts in the Atlantic–European region. The study is enabled through a unique data set, allowing for a Lagrangian detection of WCBs in three years of operational ensemble forecasts from the European Centre for Medium‐Range Weather Forecasts. By analysing the relationship between commonly used error metrics of variables that characterise the large‐scale flow and WCBs, the study aims to shed light on the question of the extent to which WCBs act as a source of forecast errors and as an amplifier of pre‐existing errors in a state‐of‐the‐art global operational numerical weather prediction model. We show that forecasts with high WCB activity are on average characterised by an amplified Rossby‐wave pattern and anticyclonic flow anomalies downstream. We find that the forecast skill is generally reduced when the WCB activity is high, and that WCB activity is particularly increased when the error growth is largest. To establish a causal relationship, we employ two composite approaches. The first focuses on the time of largest error growth and the second calculates normalised forecast error fields centered on WCB objects. Both approaches yield a consistent picture: anomalously high errors are initially associated with misrepresentations of an upstream trough. In regions of WCB ascent and outflow, the errors grow rapidly in terms of magnitude and scale and are projected onto the upper‐level large‐scale circulation. We also find indications that WCBs can cause errors even when the upstream flow is well represented. Notwithstanding, evidence is robust for WCBs acting as an amplifier of forecast uncertainty.

Sources of Forecast Errors for Rainstorms in the South China Monsoon Region

The possible sources of forecast errors associated with rainstorms in the South China monsoon region were investigated based on Weather Research and Forecasting (WRF) model forecasts for 19 rainstorm cases that occurred in the past 13 years. Two datasets were separately selected as the initial fields of WRF with the same physical parameterization schemes. By investigating the improvement rate of the forecast when using one set of data rather than the other, the important degree of the initial conditions with respect to the forecasts for each case has been obtained. For those initial errors are the important sources of forecast errors, we further explored the source of the initial errors by comparing the two initial conditions. It was found that, interestingly, the significant differences between two initial conditions are all located upstream the rainfall area, with a distance of 5° of longitude away and an area of about 4° × 4°. Based on this, we developed a new method (which we refer to as the “guide flow method”) to identify the sensitive area for rainstorm forecasts in the South China monsoon region and then examined the efficiency of the sensitive areas. It was found that reducing the initial errors in the sensitive areas leads to better forecast results than doing the same in other areas. Thus, the sensitive areas are the source areas of forecast errors for rainstorms in the South China monsoon region.

Comparison of different representations of model error in ensemble forecasts

The use of analysis increments to represent model error in the Met Office ensemble prediction system is compared with the use of stochastic parametrizations. Since analysis increments can take into account more possible sources of forecast error than stochastic parametrizations which only represent specific sources of error, the spread of the ensemble and the reliability are markedly improved. There is an increase in the rms error of the ensemble mean for some fields. This may be because analysis increments cannot represent state‐dependent statistics, but may also result from the use of initial condition perturbations from the operational ETKF rather than an ensemble data assimilation with a consistent treatment of model error.

Possible Sources of Forecast Errors Generated by the Global/Regional Assimilation and Prediction System for Landfalling Tropical Cyclones. Part II: Model Uncertainty

Possible Sources of Forecast Errors Generated by the Global/Regional Assimilation and Prediction System for Landfalling Tropical Cyclones. Part II: Model Uncertainty

Sensorless PV Power Forecasting in Grid-Connected Buildings through Deep Learning.

Existing works in photovoltaic (PV) power generation focus on accurately predicting the PV power output on a forecast horizon. As the solar power generation is heavily influenced by meteorological conditions such as solar radiation, the weather forecast is a critical input in the prediction performance. However, the weather forecast is traditionally considered to have coarse granularity, so many are compelled to use on-site meteorological sensors to complement it. However, the approach involving on-site sensors has several issues. First, it incurs the cost in the installation, operation, and management of the sensors. Second, the physical model of the sensor dynamics itself can be a source of forecast errors. Third, it requires an accumulation of sensory data that represent all seasonal variations, which takes time to collect. In this paper, we take an alternative approach to use a relatively large deep neural network (DNN) instead of the on-site sensors to cope with the coarse-grained weather forecast. With historical PV output power data from our grid-connected building with a rooftop PV power generation facility and the publicly available weather forecast history data, we demonstrate that we can train a six-layer feedforward DNN for the day-ahead forecast. It achieves the average mean absolute error (MAE) of 2.9%, comparable to that of the conventional model, but without involing the on-site sensors.

Possible sources of forecast errors generated by the global/regional assimilation and prediction system for landfalling tropical cyclones. Part I: Initial uncertainties

This paper investigates the possible sources of errors associated with tropical cyclone (TC) tracks forecasted using the Global/Regional Assimilation and Prediction System (GRAPES). The GRAPES forecasts were made for 16 landfalling TCs in the western North Pacific basin during the 2008 and 2009 seasons, with a forecast length of 72 hours, and using the default initial conditions (“initials”, hereafter), which are from the NCEP-FNL dataset, as well as ECMWF initials. The forecasts are compared with ECMWF forecasts. The results show that in most TCs, the GRAPES forecasts are improved when using the ECMWF initials compared with the default initials. Compared with the ECMWF initials, the default initials produce lower intensity TCs and a lower intensity subtropical high, but a higher intensity South Asia high and monsoon trough, as well as a higher temperature but lower specific humidity at the TC center. Replacement of the geopotential height and wind fields with the ECMWF initials in and around the TC center at the initial time was found to be the most efficient way to improve the forecasts. In addition, TCs that showed the greatest improvement in forecast accuracy usually had the largest initial uncertainties in TC intensity and were usually in the intensifying phase. The results demonstrate the importance of the initial intensity for TC track forecasts made using GRAPES, and indicate the model is better in describing the intensifying phase than the decaying phase of TCs. Finally, the limit of the improvement indicates that the model error associated with GRAPES forecasts may be the main cause of poor forecasts of landfalling TCs. Thus, further examinations of the model errors are required.

Master optimization process based on neural networks ensemble for 24-h solar irradiance forecast

In the paper two models implemented to forecast the hourly solar irradiance with a day in advance are described. The models, based on Artificial Neural Networks (ANN), are generated by a master optimization process that defines the best number of neurons and selects a suitable ensemble of ANN.The two models consist of a Statistical (ST) model that uses only local measured data and a Model Output Statistics (MOS) that corrects Numerical Weather Prediction (NWP) data. ST and MOS are tested for the University of Rome “Tor Vergata” site. The models are trained and validated using one year data. Through a cross training procedure, the dependence of the models on the training year is also analyzed.The performance of ST, NWP and MOS models, together with the benchmark Persistence Model (PM), are compared. The ST model and the NWP model exhibit similar results. Nevertheless different sources of forecast errors between ST and NWP models are identified. The MOS model gives the best performance, improving the forecast of approximately 29% with respect to the PM.

Extinction debt and the species–area relationship: a neutral perspective

AbstractAimTo estimate the magnitude of delayed relative to imminent extinctions and to assess the importance of delay as a potential source of error in forecasts of extinction. To formulate a simple mechanistic model using neutral theory that links extinction debt with the species–area relationship (SAR).LocationWorld‐wide.MethodsWe use the neutral model of biodiversity to describe how a community subject to immigration responds to an insular contraction. We investigate the species richness at different times after the habitat‐loss event. We compare this with observed species losses in avian studies.ResultsFrom the model, two SARs emerge: one with a shallow slope for a habitat area before habitat loss and another with a steeper slope for the habitats that remain after habitat loss. From these curves, the first predicts imminent extinctions while the second predicts total extinctions. The difference between the two curves gives the delayed extinctions, namely the number of species that are lost during the relaxation of the community to equilibrium. The model agrees well with observed relaxation rates in communities of birds. The lag times for relaxation are often very large, with half‐lives in the order of thousands of years for remnant areas above 5000 km2. In many parameter combinations explored, the majority of extinctions are delayed extinctions, and may exceed imminent extinctions by orders of magnitude.Main conclusionsExtinction debt is a major reason for failures to observe extinctions following habitat loss. Our modelling approach supports the view that a significant proportion of extinctions are delayed, so that the predictions of SARs (as currently applied) are liable to underestimate total extinctions. SARs are a valuable instrument for conservation but must be used with caution.

Forecasts of Earnings by Takeover Bidders

The paper analyzes the earnings forecasts issued by bidders' executives in the biggest deals announced on the New York Stock Exchange in the years 1998-2001. It explores the role of the forecasts as an information source for investors at the time of the bid announcement. We investigate the different types of forecast made - precise point forecasts and more general directional forecasts - and the characteristics of the various types of forecaster. We compare forecast with out-turn. The majority of bidders fail to deliver their projections in earnings growth and there is evidence of systematic misjudgement about the future outcome of the proposed acquisitions. We explore the sources of management forecast error and find evidence of managerial over-optimism after controlling for exogenous economic shocks in earnings. Then the paper considers the relationship between forecasts and stock returns at announcement and presents prima facie evidence of market inefficiencies. We develop a model using publicly available information at the time of the bid which is able to explain a major source of the forecast error variance and which discriminates between successful and unsuccessful acquisitions. Finally, the paper analyzes the motives for forecasting, in particular its role in the pricing of bids and the competition for control.

Short‐term forecasts of euro area real GDP growth: an assessment of real‐time performance based on vintage data

AbstractForecasters commonly predict real gross domestic product growth from monthly indicators such as industrial production, retail sales and surveys, and therefore require an assessment of the reliability of such tools. While forecast errors related to model specification and unavailability of data in real time have been assessed, the impact of data revisions on forecast accuracy has seldom been evaluated, especially for the euro area. This paper proposes to evaluate the contributions of these three sources of forecast error using a set of data vintages for the euro area. The results show that gains in accuracy of forecasts achieved by using monthly data on actual activity rather than surveys or financial indicators are offset by the fact that the former set of monthly data is harder to forecast and less timely than the latter set. These results provide a benchmark which future research may build on as more vintage datasets become available. Copyright © 2008 John Wiley & Sons, Ltd.

Comparison of 10-m Wind Forecasts from a Regional Area Model and QuikSCAT Scatterometer Wind Observations over the Mediterranean Sea

Abstract Surface wind forecasts from a limited-area model [the Quadrics Bologna Limited-Area Model (QBOLAM)] covering the entire Mediterranean area at 0.1° grid spacing are verified against Quick Scatterometer (QuikSCAT) wind observations. Only forecasts within the first 24 h in coincidence with satellite overpasses are used. Two years of data, from 1 October 2000 to 31 October 2002, have been considered, allowing for an adequate statistical assessment under different wind conditions. This has been carried out by analyzing the fields of the mean wind vectors, wind speed bias, correlation, difference standard deviation, steadiness, gustiness, and mean wind direction difference, in order to investigate spatial variability. Statistics have been computed on a seasonal basis. A comparison of satellite and forecast winds with measurements from three buoys was also performed. Some critical areas of the Mediterranean Sea where wind forecast quality is lower than average have been identified. Such areas correspond to semienclosed basins surrounded by important orography and to small regions at the lee side of the main islands. In open-sea regions the model underestimates wind strength from about 0.5 m s−1 in spring and summer to 1.0 m s−1 in winter, as evidenced by the existing biases against scatterometer data. Also, a wind direction bias (scatterometer minus model) generally between 5° and 15° exists. A survey of the identified and likely sources of forecast error is performed, indicating that orography representation plays an important role. Numerical damping is identified as a likely factor reducing forecast wind strength. The need for a correction scheme is envisaged to provide more accurate forcing for numerical sea state forecasting models, wind energy evaluation, and latent and/or sensible heat exchanges.

Evaluating Macro-Modelling Systems: An Application to the Area Wide Model

This paper considers approaches to considering the validity of the overall structure of macro-econometric models - specifically, the Area Wide Model of the European Central Bank. By structure, we refer to the dynamic (business-cycle) and steady-state features that the model purports to capture. This leads to two types of tests. The first, drawing on the DSGE literature, is concerned with whether the model matches business-cycle (high-frequency) data characteristics. This is implemented by a Cholesky bootstrap whereby the steady state of the model is stochastically simulated using historically consistent covariances. The generated data is analysed for stylised-facts fitting and, similarly, using the model's implied spectral characteristics, for congruence with the data in terms of persistence, periodicity and spectral fit. Moments matching, however, is only one aspect of overall model evaluation. Consequently, we move to tests that combine high-frequency aspects (short-horizon forecasts) with long-run features (such as the existence and identification of steady states and trends). Recursive forecasting tests form the second part. The forecasts attempt to measure the accuracy of model-based forecasts both simulated out-of-sample and in an in-sample exercise. The out-of-sample exercise analyses the 1- to 8-step-ahead forecasting ability of the model. For this, the model is re-estimated each time on a subset of the original sample, and the re-estimated model is used to generate a forecast over the remaining sample. The in-sample exercise omits the re-estimation step but performs a more thorough exercise, covering 1- to 12-steps-ahead forecasts over a larger part of the original sample. For this latter exercise, a thorough analysis of sources of forecast error is made. Both exercises incorporate alternative residual-projection methods, in order to assess the importance of unaccounted-for breaks in forecast accuracy. Conclusions reached are that, on the one hand, in-sample exercises should be preferred with systems of this size and typical samples; and, on the other, that mechanical residual adjustment or model re-estimation should be avoided except under strong evidence of mis-specification. The paper considers the testing procedure to be one applicable to the class of large, macro models and therefore of general interest.

Comparison of a single-model EPS with a multi-model ensemble consisting of a few operational models

Since the introduction of operational ensemble forecasts in Numerical Weather Prediction(NWP) more than 5 years ago, the dispute on how to best determine the initial perturbationshas largely dominated the direction of research in the field of ensemble prediction. While it isimportant to consider uncertainties in the initial condition, errors due to model physics or themodel numerics and truncation provide another source of forecast errors and might also beconsidered in ensemble prediction. In this study, we compare the performance of 2 fundamentallydifferent ensemble schemes. First, the ensemble prediction system (EPS) of the European Centrefor Medium Range Forecasts is taken as a representative of the single-model approach basedon the perfect model assumption and thus taking only the uncertainty in the observations intoaccount. Second, a virtual ensemble comprised of the operational forecasts of 4 NWP centersas a “gratis” candidate of the multi-model approach which, in addition, takes model errors intoaccount. The comparison is based on forecasts of 500 hPa fields over Europe for a summer anda winter period in 1997 and on diagnostics ranging from various measures for the performanceof the ensemble means to the statistical consistency and discrimination properties of theensembles. The different sizes of both ensembles poses the main difficulty for the interpretationof the results. If the ensemble size is not considered as a criterion for the evaluation, the resultslead to controversial conclusions; but when penalizing for an overly large and inefficientensemble the results are for the most part consistent, and one has to conclude that the multimodelensemble performs better in most forecast aspects.

As assessment of the singular-vector approach to targeted observing using the FASTEX dataset

In this study, we investigate whether the results of assimilating special targeted observations from the Fronts and Atlantic Storm-Track EXperiment (FASTEX) in an operational forecast model support the underlying principles of the singular-vector (SV) approach to targeted observing. A simple framework is presented that allows explicit examination of the changes made to the analysis in the subspace of the leading SVs from assimilation of the observations. the impact of this component on the forecast provides a key measure of the effectiveness of SV-based targeting. Results confirm that the impact of the additional observations occurs primarily as a result of changes to the analysis in the subspace of the leading SVs. These changes account for a small fraction of the total targeting increment at initial time, but explain a large fraction of the response of the forecast at the verification time. the results also confirm that analysis errors in the middle and lower troposphere are an important source of error in forecasts of extratropical cyclones. While moist processes can play an important role in the forecast-error evolution, SVs that exclude these processes can remain an effective targeting tool. This is because the location of maximum sensitivity will not necessarily differ from that identified by the dry SVs. It is also shown that the locations of the leading (target) SVs can be computed accurately with lead times of up to 48 hours, allowing ample time for the deployment of observational resources.

As assessment of the singular‐vector approach to targeted observing using the FASTEX dataset

AbstractIn this study, we investigate whether the results of assimilating special targeted observations from the Fronts and Atlantic Storm‐Track EXperiment (FASTEX) in an operational forecast model support the underlying principles of the singular‐vector (SV) approach to targeted observing. A simple framework is presented that allows explicit examination of the changes made to the analysis in the subspace of the leading SVs from assimilation of the observations. the impact of this component on the forecast provides a key measure of the effectiveness of SV‐based targeting.Results confirm that the impact of the additional observations occurs primarily as a result of changes to the analysis in the subspace of the leading SVs. These changes account for a small fraction of the total targeting increment at initial time, but explain a large fraction of the response of the forecast at the verification time. the results also confirm that analysis errors in the middle and lower troposphere are an important source of error in forecasts of extratropical cyclones.While moist processes can play an important role in the forecast‐error evolution, SVs that exclude these processes can remain an effective targeting tool. This is because the location of maximum sensitivity will not necessarily differ from that identified by the dry SVs. It is also shown that the locations of the leading (target) SVs can be computed accurately with lead times of up to 48 hours, allowing ample time for the deployment of observational resources.

The Relationship between Ensemble Spread and Ensemble Mean Skill

Statistical considerations suggest that 1) even for a perfect ensemble (one in which all sources of forecast error are sampled correctly) there need not be a high correlation between spread and skill, 2) the correlation between spread and skill should be larger where the day-to-day variability of spread is large, and 3) the spread is likely to be most useful as a predictor of skill when it is “extreme,” that is, when it is either very large or very small compared to its climatological mean value. The authors investigate the relationship between spread and skill in an operational setting by analyzing ensemble predictions produced by the National Centers for Environmental Prediction. The geographical dependence of the spread–skill relationship is found to be related to the geographical dependence of day-to-day variability of spread. Dynamical mechanisms for spread variability are investigated using a linear quasigeostrophic model. Problems associated with the sample size needed to define what constitutes an extreme value of spread at a given location are discussed.

An Adjustment Procedure for Predicting Betas When Thin Trading is Present: Canadian Evidence

We have incorporated effects of the process that generates true betas for TSE stocks, as well as thin trading effects, into the beta adjustment model. We note the Blume and Dimson and Marsh beta adjustment techniques aim at eliminating beta forecast error through regression tendency bias. Effects of other sources of forecast error have been ignored. We show the process generating security betas affects both cross‐sectional correlation coefficient and order bias, while thin trading affects only cross‐sectional correlation coefficient. We demonstrate that when OLS beta estimates are used to forecast their future risk levels, order bias accounts for 86% of forecast error, while thin trading effects account for 14% of forecast error. A beta regression tendency model which properly accounts for effects of cross‐sectional correlation (which is a function of thin trading) and order bias completely abates forecast error. Our results have implications for the use of correlation coefficient to measure stability of betas across time, for beta adjustment models proposed in the literature, and for event study methodologies that rely on prediction errors.

Forecast accuracy of Australian subnational population projections.

The authors assess demographic projections made over the last 20 years for Australia. "This paper first considers the role of accuracy amongst other objectives of projection activity. Accepting accuracy as a legitimate goal, we then assess the performance of 48 sets of population projections and forecasts for states and territories of Australia prepared since 1970. Projection accuracy is assessed by reference to length of forecast horizon, population size and rate of growth. We also examine the main sources of forecast error in selected projections for each state and compare the performance of past projections with alternatives based on simple extrapolation of contemporary population trends."

The sources of forecast errors: Experimental evidence

In Muth's original theory, the passage of time is the only source of forecast error. Persistent rejections led Muth to surmise problems in the data also cause forecast error; Keane and Runkle argue that many rejections of Muth's theory are reversed if data problems are properly considered. Experimental evidence allows a particularly clean test because the data are inherently simpler. The continued rejection of Muth's hypothesis suggests model uncertainty as a source of forecast error.