Model Forecast Errors Research Articles

Toward the Probabilistic Simulation of Storm Surge and Inundation in a Limited-Resource Environment

Abstract To create more useful storm surge and inundation forecast products, probabilistic elements are being incorporated. To achieve the highest levels of confidence in these products, it is essential that as many simulations as possible are performed during the limited amount of time available. This paper develops a framework by which probabilistic storm surge and inundation forecasts within the Curvilinear Hydrodynamics in 3D (CH3D) Storm Surge Modeling System and the Southeastern Universities Research Association Coastal Ocean Observing and Prediction Program’s forecasting systems are initiated with specific focus on the application of these methods in a limited-resource environment. Ensemble sets are created by dividing probability density functions (PDFs) of the National Hurricane Center model forecast error into bins, which are then grouped into priority levels (PLs) such that each subsequent level relies on results computed earlier and has an increasing confidence associated with it. The PDFs are then used to develop an ensemble of analytic wind and pressure fields for use by storm surge and inundation models. Using this approach applied with official National Hurricane Center (OFCL) forecast errors, an analysis of Hurricane Charley is performed. After first validating the simulation of storm surge, a series of ensemble simulations are performed representing the forecast errors for the 72-, 48-, 24-, and 12-h forecasts. Analysis of the aggregated products shows that PL4 (27 members) is sufficient to resolve 90% of the inundation within the domain and appears to represent the best balance between accuracy and timeliness of computed products for this case study. A 5-day forecast using the PL4 set is shown to complete in 83 min, while the intermediate PL2 and PL3 products, representing slightly less confidence, complete in 14 and 28 min, respectively.

Weather Forecasts by the WRF-ARW Model with the GSI Data Assimilation System in the Complex Terrain Areas of Southwest Asia

Abstract This paper will first describe the forecasting errors encountered from running the National Center for Atmospheric Research (NCAR) mesoscale model (the Advanced Research Weather Research and Forecasting model; ARW) in the complex terrain of southwest Asia from 1 to 31 May 2006. The subsequent statistical evaluation is designed to assess the model’s surface and upper-air forecast accuracy. Results show that the model biases caused by inadequate parameterization of physical processes are relatively small, except for the 2-m temperature, as compared to the nonsystematic errors resulting in part from the uncertainty in the initial conditions. The total model forecast errors at the surface show a substantial spatial heterogeneity; the errors are relatively larger in higher mountain areas. The performance of 2-m temperature forecasts is different from the other surface variables’ forecasts; the model forecast errors in 2-m temperature forecasts are closely related to the terrain configuration. The diurnal cycle variation of these near-surface temperature forecasts from the model is much smaller than what is observed. Second, in order to understand the role of the initial conditions in relation to the accuracy of the model forecasts, this study assimilated a form of satellite radiance data into this model through the Joint Center for Satellite Data Assimilation (JCSDA) analysis system called the Gridpoint Statistical Interpolation (GSI). The results indicate that on average over a 30-day experiment for the 24- and 48-h (second 24 h) forecasts, the satellite data provide beneficial information for improving the initial conditions and the model errors are reduced to some degree over some of the study locations. The diurnal cycle for some forecasting variables can be improved after satellite data assimilation; however, the improvement is very limited.

Comparison of Model Forecast Skill of Sea Level Pressure along the East and West Coasts of the United States

Abstract Despite recent advances in numerical weather prediction, major errors in short-range forecasts still occur. To gain insight into the origin and nature of model forecast errors, error frequencies and magnitudes need to be documented for different models and different regions. This study examines errors in sea level pressure for four operational forecast models at observation sites along the east and west coasts of the United States for three 5-month cold seasons. Considering several metrics of forecast accuracy, the European Centre for Medium-Range Weather Forecasts (ECMWF) model outperformed the other models, while the North American Mesoscale (NAM) model was least skillful. Sea level pressure errors on the West Coast are greater than those on the East Coast. The operational switch from the Eta to the Weather Research and Forecasting Nonhydrostatic Mesoscale Model (WRF-NMM) at the National Centers for Environmental Prediction (NCEP) did not improve forecasts of sea level pressure. The results also suggest that the accuracy of the Canadian Meteorological Centre’s Global Environmental Mesoscale model (CMC-GEM) improved between the first and second cold seasons, that the ECMWF experienced improvement on both coasts during the 3-yr period, and that the NCEP Global Forecast System (GFS) improved during the third cold season on the West Coast.

Representing energy drain in numerical weather prediction models induced by boundary layer sub‐grid scale processes

AbstractThe representation of sub‐grid scale boundary layer processes is central to understanding the errors in numerical weather prediction model forecasts. Of particular importance are statistics associated with convective turbulence, notably the temporal and spatial variations of kinetic energy dissipation rate. In this paper we outline how a 1.5‐micron Doppler lidar system may be used in this context, and propose an operational network of such systems for use in numerical forecasting. Copyright © 2009 Royal Meteorological Society

Correlations between Analyses and Forecasts of Banded Heavy Snow Ingredients and Observed Snowfall

Abstract North American Mesoscale (NAM) model forecasts of the occurrence, magnitude, depth, and persistence of ingredients previously shown to be useful in the diagnosis of banded and/or heavy snowfall potential are examined for a broad range of 25 snow events, with event total snowfall ranging from 10 cm (4 in.) to over 75 cm (30 in.). The ingredients examined are frontogenetical forcing, weak moist symmetric stability, saturation, and microphysical characteristics favorable for the production of dendritic snow crystals. It is shown that these ingredients, previously identified as being critical indicators for heavy and/or banded snowfall in major storms, are often found in smaller snowfall events. It is also shown that the magnitude, depth, and persistence of these ingredients, or combinations of these ingredients, appear to be good predictors of event total snowfall potential. In addition, a relationship is demonstrated between temporal trends associated with one of the ingredients (saturated, geostrophic equivalent potential vorticity) and event total snowfall. Correlations between forecast values of these ingredients and observed snowfall are shown to decrease substantially as forecast lead time increases beyond 12 h. It is hypothesized that model forecast positioning and timing errors are primarily responsible for the lower correlations associated with longer-lead forecasts. This finding implies that the best forecasts beyond 12 h may be produced by examining the diagnostics of heavy snow ingredients from a single, high-resolution model to determine snowfall potential, then using ensemble forecasting approaches to determine the most probable location and timing of any heavy snow.

West African Monsoon observed with ground‐based GPS receivers during African Monsoon Multidisciplinary Analysis (AMMA)

A ground‐based GPS network has been established over West Africa in the framework of African Monsoon Multidisciplinary Analysis (AMMA) in tight cooperation between French and African institutes. The experimental setup is described and preliminary highlights are given for different applications using these data. Precipitable water vapor (PWV) estimates from GPS are used for evaluating numerical weather prediction (NWP) models and radiosonde humidity data. Systematic tendency errors in model forecasts are evidenced. Correlated biases in NWP model analyses and radiosonde data are evidenced also, which emphasize the importance of radiosonde humidity data in this region. PWV and precipitation are tightly correlated at seasonal and intraseasonal timescales. Almost no precipitation occurs when PWV is smaller than 30 kg m−2. This limit in PWV also coincides well with the location of the intertropical discontinuity. Five distinct phases in the monsoon season are determined from the GPS PWV, which correspond either to transition or stationary periods of the West African Monsoon system. They may serve as a basis for characterizing interannual variability. Significant oscillations in PWV are observed with 10‐ to 15‐day and 15‐ to 20‐day periods, which suggest a strong impact of atmospheric circulation on moisture and precipitation. The presence of a diurnal cycle oscillation in PWV with marked seasonal evolutions is found. This oscillation involves namely different phasing of moisture fluxes in different layers implying the low‐level jet, the return flow, and the African Easterly Jet. The broad range of timescales observed with the GPS systems shows a high potential for investigating many atmospheric processes of the West African Monsoon.

Tactical Industry Allocation and Model Uncertainty

AbstractWe use Bayesian model averaging to analyze industry return predictability in the presence of model uncertainty. The posterior analysis shows the importance of inflation and earnings yield in predicting industry returns. The out‐of‐sample performance of the Bayesian approach is, in general, superior to that of other statistical model selection criteria. However, the out‐of‐sample forecasting power of a naive i.i.d. forecast is similar to the Bayesian forecast. A variance decomposition into model risk, estimation risk, and forecast error shows that model risk is less important than estimation risk.

Prediction of the Diurnal Change Using a Multimodel Superensemble. Part I: Precipitation

Abstract Modeling the geographical distribution of the phase and amplitude of the diurnal change is a challenging problem. This paper addresses the issues of modeling the diurnal mode of precipitation over the Tropics. Largely an early morning precipitation maximum over the oceans and an afternoon rainfall maximum over land areas describe the first-order diurnal variability. However, large variability in phase and amplitude prevails even within the land and oceanic areas. This paper addresses the importance of a multimodel superensemble for much improved prediction of the diurnal mode as compared to what is possible from individual models. To begin this exercise, the skills of the member models, the ensemble mean of the member models, a unified cloud model, and the superensemble for the prediction of total rain as well as its day versus night distribution were examined. Here it is shown that the distributions of total rain over the earth (tropical belt) and over certain geographical regions are predicted reasonably well (RMSE less than 18%) from the construction of a multimodel superensemble. This dataset is well suited for addressing the diurnal change. The large errors in phase of the diurnal modes in individual models usually stem from numerous physical processes such as the cloud radiation, shallow and deep cumulus convection, and the physics of the planetary boundary layer. The multimodel superensemble is designed to reduce such systematic errors and provide meaningful forecasts. That application for the diurnal mode appears very promising. This paper examines some of the regions such as the Tibetan Plateau, the eastern foothills of the Himalayas, and the Amazon region of South America that are traditionally difficult for modeling the diurnal change. In nearly all of these regions, errors in phase and amplitude of the diurnal mode of precipitation increase with the increased length of forecasts. Model forecast errors on the order of 6–12 h for phase and 50% for the amplitude are often seen from the member models. The multimodel superensemble reduces these errors and provides a close match (RMSE < 6 h) to the observed phase. The percent of daily rain and their phases obtained from the multimodel superensemble at 3-hourly intervals for different regions of the Tropics showed a closer match (pattern correlation about 0.4) with the satellite estimates. This is another area where the individual member models conveyed a much lower skill.

Four‐dimensional data assimilation experiments with International Consortium for Atmospheric Research on Transport and Transformation ozone measurements

Ozone measurements by various platforms during the International Consortium for Atmospheric Research on Transport and Transformation (ICARTT) operations in the summer of 2004 are assimilated into the STEM regional chemical transport model (CTM). Under the four‐dimensional variational data assimilation (4D‐Var) framework, the model forecast (background) error covariance matrix is constructed using both the so‐called NMC (National Meteorological Center, now National Centers for Environmental Prediction) method and the observational (Hollingworth‐Lönnberg) method. The inversion of the covariance matrix is implemented using truncated singular value decomposition (TSVD) approach. The TSVD approach is numerically stable even with severely ill conditioned vertical correlation covariance matrix and large horizontal correlation distances. Ozone observations by different platforms (aircraft, surface, and ozonesondes) are first assimilated separately. The impacts of the various measurements are evaluated on their ability to improve the predictions, defined as the information content of the observations under the current framework. In the end, all observations are assimilated into the CTM. The final analysis matches well with observations from all platforms. Assessed with all the observations throughout the boundary layer and midtroposphere, the model bias is reduced from 11.3 ppbv for the base case to −1.5 ppbv. A reduction of 10.3 ppbv in root mean square error is also seen. In addition, the potential of improving air quality forecasts by chemical data assimilation is demonstrated. The effect of assimilating ozone observations on model predictions of other species is also shown.

Tactical Industry Allocation and Model Uncertainty

We use Bayesian model averaging to analyze the sample evidence on industry return predictability within the U.S. stock market in the presence of model uncertainty. The posterior analysis shows the importance of inflation and earnings yield in predicting industry returns. The out-of-sample performance of the Bayesian approach is, in general, superior to that of other statistical model selection criteria. However, the out-of-sample forecasting power of a naive iid forecast is similar to the Bayesian forecast. A variance decomposition into model risk, estimation risk, and forecast error shows that model risk is less important than estimation risk.

Multivariate assimilation in MERCATOR project: New statistical parameters from forecast error estimation

The new operational prototype of Mercator (french Global Ocean Data Assimilation Experiment contribution) is composed of a North Atlantic primitive equation ocean model OPA (Ocean Parallel Algorithm between 20°S and 70°N, [Madec, G., P. Delecluse, M. Imbard and C. Lévy (1998). OPA8.1 ocean general circulation model reference manuel. Notes du pôle de modélisation IPSL. n°11: 91p]) and of a multivariate and multidata assimilation scheme [De Mey, P. and M. Benkiran (2002). “A multivariate reduced-order optimal interpolation method and its application in Mediterranean basin-scale circulation.” Ocean Forecasting : Conceptual basis and application, Pinardi, N., Springer Verlag.] This system has already given some significant improvements from previous Mercator configurations (M. Benkiran, personal communication). However some biases on ocean state still remain in the tropics where the reduced-order optimal interpolation scheme is suspected to be ill-parameted in the model forecast error. Indeed the guess error covariance matrix is decomposed into an error variance value and a spatio-temporal correlation function which are assumed to have some “good” properties (spatial homogeneity of the correlation function, constant ratio between signal and error variance). This study shows how we can use ensemble methods to validate these assumptions. We can see that the correlation function can reach negative values locally, mostly in regions of high variability contradictory with the homogeneous hypothesis. The reduced space used in the operational configuration is based on the signal seasonal Empirical Orthogonal Functions (EOFs). An empirical relationship between signal and error variance has been set and the correlation function is the same on every dimension of the reduced space. By projection of the estimated guess error variance onto the reduced space, we find a repartition of this quantity quite different to what was set in the system. The error statistics is found to be inhomogeneous compared to hypothesis made in the assimilation scheme. These two new parameters tested separately in the assimilation scheme gives significant improvements of the forecast and analysis results. This is particularly obvious in the tropics. But relationship between signal and error statistics (as assumed in the optimal interpolation) is found to be complex.

Multivariate Error Covariance Estimates by Monte Carlo Simulation for Assimilation Studies in the Pacific Ocean

Abstract One of the most difficult aspects of ocean-state estimation is the prescription of the model forecast error covariances. The paucity of ocean observations limits our ability to estimate the covariance structures from model–observation differences. In most practical applications, simple covariances are usually prescribed. Rarely are cross covariances between different model variables used. Here a comparison is made between a univariate optimal interpolation (UOI) scheme and a multivariate OI algorithm (MvOI) in the assimilation of ocean temperature profiles. In the UOI case only temperature is updated using a Gaussian covariance function. In the MvOI, salinity, zonal, and meridional velocities as well as temperature are updated using an empirically estimated multivariate covariance matrix. Earlier studies have shown that a univariate OI has a detrimental effect on the salinity and velocity fields of the model. Apparently, in a sequential framework it is important to analyze temperature and salinity together. For the MvOI an estimate of the forecast error statistics is made by Monte Carlo techniques from an ensemble of model forecasts. An important advantage of using an ensemble of ocean states is that it provides a natural way to estimate cross covariances between the fields of different physical variables constituting the model-state vector, at the same time incorporating the model’s dynamical and thermodynamical constraints as well as the effects of physical boundaries. Only temperature observations from the Tropical Atmosphere–Ocean array have been assimilated in this study. To investigate the efficacy of the multivariate scheme, two data assimilation experiments are validated with a large independent set of recently published subsurface observations of salinity, zonal velocity, and temperature. For reference, a control run with no data assimilation is used to check how the data assimilation affects systematic model errors. While the performance of the UOI and MvOI is similar with respect to the temperature field, the salinity and velocity fields are greatly improved when the multivariate correction is used, as is evident from the analyses of the rms differences between these fields and independent observations. The MvOI assimilation is found to improve upon the control run in generating water masses with properties close to the observed, while the UOI fails to maintain the temperature and salinity structure.

Requirements of a global near-surface soil moisture satellite mission: accuracy, repeat time, and spatial resolution

Soil moisture satellite mission accuracy, repeat time and spatial resolution requirements are addressed through a numerical twin data assimilation study. Simulated soil moisture profile retrievals were made by assimilating near-surface soil moisture observations with various accuracy (0, 1, 2, 3, 4, 5 and 10%v/v standard deviation) repeat time (1, 2, 3, 5, 10, 15, 20 and 30 days), and spatial resolution (0.5, 6, 12 18, 30, 60 and 120 arc-min). This study found that near-surface soil moisture observation error must be less than the model forecast error required for a specific application when used as data assimilation input, else slight model forecast degradation may result. It also found that near-surface soil moisture observations must have an accuracy better than 5%v/v to positively impact soil moisture forecasts, and that daily near-surface soil moisture observations achieved the best soil moisture and evapotranspiration forecasts for the repeat times assessed, with 1–5 day repeat times having the greatest impact. Near-surface soil moisture observations with a spatial resolution finer than the land surface model resolution (∼30 arc-min) produced the best results, with spatial resolutions coarser than the model resolution yielding only a slight degradation. Observations at half the land surface model spatial resolution were found to be appropriate for our application. Moreover, it was found that satisfying the spatial resolution and accuracy requirements was much more important than repeat time.

A History of Western North Pacific Tropical Cyclone Track Forecast Skill

Abstract The tropical cyclone (TC) track forecasting skill of operational numerical weather prediction (NWP) models and their consensus is examined for the western North Pacific from 1992 to 2002. The TC track forecasting skill of the operational NWP models is steadily improving. For the western North Pacific, the typical 72-h model forecast error has decreased from roughly 600 km to roughly 400 km over the past ten years and is now comparable to the typical 48-h model forecast error of 10 years ago. In this study the performance of consensus aids that are formed whenever the TC track forecasts from at least two models from a specified pool of operational NWP models are available is examined. The 72-h consensus forecast error has decreased from about 550 km to roughly 310 km over the past ten years and is now better than the 48-h consensus forecast error of 10 years ago. For 2002, the 72-h forecast errors for a consensus computed from a specified pool of two, five, seven, and eight models were 357, 342, 3...

Besser geht’s nicht – Genauigkeitsgrenzen von Konjunkturprognosen / As Good as it Gets – Limits of Accuracy of Macroeconomic Short Term Forecasts

Zusammenfassung Die Arbeit versucht, für Deutschland obere Grenzen der Genauigkeit („Fehler-Minima“) von Konjunkturprognosen zu bestimmen. Grundlage sind Simulationsrechnungen mit dem RWIKonjunkturmodell. Eine erste Grenze lässt sich unter der Annahme richtiger prädeterminierter Variablen und der Gültigkeit der Zeitstabilität der Modellbeziehungen bestimmen. Weitere Grenzen ergeben sich bei Zerlegung des verbleibenden Fehlers in „Einzelgleichungs-“, „Modell-“ und „Dynamikfehler“. Die Simulationsrechnungen zeigen exemplarisch, dass die Grenze für die Treffsicherheit der Wachstums- und Inflationsprognosen nicht weit von den gegenwärtig beobachteten Fehlern der Konjunkturprognosen entfernt ist. Das Potenzial zur Verbesserung der Modellprognosen ist vor allem auf der Einzelgleichungsebene zu suchen.

Phase-Correcting Data Assimilation and Application to Storm-Scale Numerical Weather Prediction. Part II: Application to a Severe Storm Outbreak

Abstract A scheme to correct phase errors in numerical model forecasts using Doppler radar, radiosonde, profiler, and surface data is demonstrated to improve forecasts in a complex severe thunderstorm situation. The technique is designed to directly address forecast phase errors or initial position errors as part of a data assimilation strategy. In the demonstration the phase error correction is applied near the time of initial cell development and the forecast results are compared to the uncorrected forecast and forecasts made using an analysis created at the time of the observations. Forecasts are verified qualitatively for the position of thunderstorm cells and quantitatively for accumulated precipitation. It is shown that the scheme can successfully correct errors in thunderstorm locations and it has a positive influence on the subsequent forecast. The advantage of the phase correction over the control lasts for about 3 h despite storm dissipation and regeneration, and interactions among multiple storms.

Space-time structure and dynamics of the forecast error in a coastal circulation model of the Gulf of Lions

The probability density function (pdf) of forecast errors due to several possible error sources is investigated in a coastal ocean model driven by the atmosphere and a larger-scale ocean solution using an Ensemble (Monte Carlo) technique. An original method to generate dynamically adjusted perturbation of the slope current is proposed. The model is a high-resolution 3D primitive equation model resolving topographic interactions, river runoff and wind forcing. The Monte Carlo approach deals with model and observation errors in a natural way. It is particularly well-adapted to coastal non-linear studies. Indeed higher-order moments are implicitly retained in the covariance equation. Statistical assumptions are made on the uncertainties related to the various forcings (wind stress, open boundary conditions, etc.), to the initial state and to other model parameters, and randomly perturbed forecasts are carried out in accordance with the a priori error pdf. The evolution of these errors is then traced in space and time and the a posteriori error pdf can be explored. Third- and fourth-order moments of the pdf are computed to evaluate the normal or Gaussian behaviour of the distribution. The calculation of Central Empirical Orthogonal Functions (Ceofs) of the forecast Ensemble covariances eventually leads to a physical description of the model forecast error subspace in model state space. The time evolution of the projection of the Reference forecast onto the first Ceofs clearly shows the existence of specific model regimes associated to particular forcing conditions. The Ceofs basis is also an interesting candidate to define the Reduced Control Subspace for assimilation and in particular to explore transitions in model state space. We applied the above methodology to study the penetration of the Liguro-Provençal Catalan Current over the shelf of the Gulf of Lions in north-western Mediterranean together with the discharge of the Rhône river. This region is indeed well-known for its intense topographic and atmospheric forcings.

Feature calibration and alignment to represent model forecast errors: Empirical regularization

AbstractA method for characterizing errors for weather forecasts and analyses, called feature calibration and alignment (FCA), decomposes model forecast error into phase error, amplitude error and residual error. Phase and amplitude errors are represented by large‐scale displacements and biases of a field. In this study FCA is used to represent errors in realistic univariate two‐dimensional analysis and forecast fields. In previous work FCA was tuned subjectively. Here, a methodology to determine an empirical constraint that regularizes the FCA solutions is developed. This version of FCA still has some tunable parameters, but the new method greatly reduces the subjective nature of FCA. The data used are European Centre for Medium‐Range Weather Forecasts analyses and one‐ to ten‐day forecasts of 500 hPa geopotential height. A stepwise procedure is used to derive the statistics of the error components for ideally chosen spectral truncations. The stepwise FCA statistics are then used to define a constraint that controls the size of the spectral coefficients of the FCA representation of the phase and amplitude errors. A large fraction (∼90%) of the 500 hPa geopotential‐height forecast‐error variance is explained by FCA, demonstrating that a large part of the forecast error can be described as large‐scale phase and bias errors. The fraction of variance explained increases with forecast length, and is slightly larger in winter than in summer. The constrained FCA solutions are more efficient and exhibit less of a tendency for multiple solutions when compared to the unconstrained FCA solutions. Copyright © 2003 Royal Meteorological Society

The Long-term Ecosystem Observatory: an integrated coastal observatory

An integrated ocean observatory has been developed and operated in the coastal waters off the central coast of New Jersey, USA. One major goal for the Long-term Ecosystem Observatory (LEO) is to develop a real-time capability for rapid environmental assessment and physical/biological forecasting in coastal waters. To this end, observational data are collected from satellites, aircrafts, ships, fixed/relocatable moorings and autonomous underwater vehicles. The majority of the data are available in real-time allowing for adaptive sampling of episodic events and are assimilated into ocean forecast models. In this observationally rich environment, model forecast errors are dominated by uncertainties in the model physics or future boundary conditions rather than initial conditions. Therefore, ensemble forecasts with differing model parameterizations provide a unique opportunity for model refinement and validation. The system has been operated during three annual coastal predictive skill experiments from 1998 through 2000. To illustrate the capabilities of the system, case studies on coastal upwelling and small-scale biological slicks are discussed. This observatory is one part of the expanding network of ocean observatories that will form the basis of a national observation network.

A non-linear neural network technique for updating of river flow forecasts

Abstract. A non-linear Auto-Regressive Exogenous-input model (NARXM) river flow forecasting output-updating procedure is presented. This updating procedure is based on the structure of a multi-layer neural network. The NARXM-neural network updating procedure is tested using the daily discharge forecasts of the soil moisture accounting and routing (SMAR) conceptual model operating on five catchments having different climatic conditions. The performance of the NARXM-neural network updating procedure is compared with that of the linear Auto-Regressive Exogenous-input (ARXM) model updating procedure, the latter being a generalisation of the widely used Auto-Regressive (AR) model forecast error updating procedure. The results of the comparison indicate that the NARXM procedure performs better than the ARXM procedure. Keywords: Auto-Regressive Exogenous-input model, neural network, output-updating procedure, soil moisture accounting and routing (SMAR) model