Statistical Post-processing Techniques Research Articles

Analog‐Based Postprocessing of Navigation‐Related Hydrological Ensemble Forecasts

AbstractInland waterway transport benefits from probabilistic forecasts of water levels as they allow to optimize the ship load and, hence, to minimize the transport costs. Probabilistic state‐of‐the‐art hydrologic ensemble forecasts inherit biases and dispersion errors from the atmospheric ensemble forecasts they are driven with. The use of statistical postprocessing techniques like ensemble model output statistics (EMOS) allows for a reduction of these systematic errors by fitting a statistical model based on training data. In this study, training periods for EMOS are selected based on forecast analogs, i.e., historical forecasts that are similar to the forecast to be verified. Due to the strong autocorrelation of water levels, forecast analogs have to be selected based on entire forecast hydrographs in order to guarantee similar hydrograph shapes. Custom‐tailored measures of similarity for forecast hydrographs comprise hydrological series distance (SD), the hydrological matching algorithm (HMA), and dynamic time warping (DTW). Verification against observations reveals that EMOS forecasts for water level at three gauges along the river Rhine with training periods selected based on SD, HMA, and DTW compare favorably with reference EMOS forecasts, which are based on either seasonal training periods or on training periods obtained by dividing the hydrological forecast trajectories into runoff regimes.

Estimating predictive hydrological uncertainty by dressing deterministic and ensemble forecasts; a comparison, with application to Meuse and Rhine

Two statistical post-processing approaches for estimation of predictive hydrological uncertainty are compared: (i) ‘dressing’ of a deterministic forecast by adding a single, combined estimate of both hydrological and meteorological uncertainty and (ii) ‘dressing’ of an ensemble streamflow forecast by adding an estimate of hydrological uncertainty to each individual streamflow ensemble member. Both approaches aim to produce an estimate of the ‘total uncertainty’ that captures both the meteorological and hydrological uncertainties. They differ in the degree to which they make use of statistical post-processing techniques. In the ‘lumped’ approach, both sources of uncertainty are lumped by post-processing deterministic forecasts using their verifying observations. In the ‘source-specific’ approach, the meteorological uncertainties are estimated by an ensemble of weather forecasts. These ensemble members are routed through a hydrological model and a realization of the probability distribution of hydrological uncertainties (only) is then added to each ensemble member to arrive at an estimate of the total uncertainty.The techniques are applied to one location in the Meuse basin and three locations in the Rhine basin. Resulting forecasts are assessed for their reliability and sharpness, as well as compared in terms of multiple verification scores including the relative mean error, Brier Skill Score, Mean Continuous Ranked Probability Skill Score, Relative Operating Characteristic Score and Relative Economic Value. The dressed deterministic forecasts are generally more reliable than the dressed ensemble forecasts, but the latter are sharper. On balance, however, they show similar quality across a range of verification metrics, with the dressed ensembles coming out slightly better. Some additional analyses are suggested. Notably, these include statistical post-processing of the meteorological forecasts in order to increase their reliability, thus increasing the reliability of the streamflow forecasts produced with ensemble meteorological forcings.

Evaluation of solar irradiation estimates of the BRAMS and development of a statistical post-processing technique for Northern Brazil

Este trabalho propoe uma metodologia para obtencao de dados de irradiacao solar incidente na superficie da regiao Norte do Brasil utilizando o modelo numerico regional de mesoescala BRAMS em conjunto com uma tecnica de remocao de vies. Este tipo de solucao se faz necessario em virtude dos erros sistematicos inerentes as simulacoes, ocasionados, por exemplo, pela dificuldade na representacao da variabilidade da cobertura de nuvens. A tecnica estatistica de remocao de vies foi desenvolvida utilizando dados observacionais de irradiacao solar de estacoes meteorologicas do Instituto Nacional de Meteorologia (INMET) operando na area de estudo ao longo do ano de 2009. Verificou-se que o modelo BRAMS e capaz de reproduzir satisfatoriamente os ciclos mensais e diarios da irradiacao solar, mas existe uma clara tendencia de superestimar os valores em relacao aos dados observados. A validacao com dados de superficie mostrou que a tecnica estatistica de remocao do vies conseguiu reduzir significativamente estes erros, aproximando os valores simulados daqueles observados em superficie. Normal 0 21 false false false PT-BR X-NONE X-NONE /* Style Definitions */ table.MsoNormalTable {mso-style-name:Tabela normal; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin:0cm; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:Times New Roman,serif;} a b s t r a c t This study presents a methodology to get surface incoming solar irradiation at the Brazilian Northern region by using the regional mesoscale model BRAMS together with a statistical correction for bias removal. This methodology is required to get solar irradiation forecasts due to the systematic errors in solar data provided by numerical simulations related to the cloud variability. The statistical technique proposed for bias removal was developed using observational data acquired at automated weather stations of operated by the Brazilian Institute of Meteorology (INMET) in the study area for the whole year of 2009. The results showed that BRAMS model can simulate monthly and daily cycles of solar radiation satisfactorily, but there is a clear tendency to overestimate values when compared to the observational data. Validations from surface data showed that this statistical technique can significantly reduce these errors, approaching the simulated values to those observed in surface. Keywords : Solar Energy, Solar Irradiation, BRAMS.

Comparison of Different Configurations of Quantile Regression in Estimating Predictive Hydrological Uncertainty

Three different configurations of quantile regression as a statistical post processing technique to predict hydrological uncertainty in flood forecasts are investigated. The first configuration applies quantile regression of forecast errors conditioned on forecast discharge in original domain (QR-ORI) while the second configuration includes normal quantile transformation of forecast discharge and forecast errors prior to quantile regression (QR-NQT) to take into account the heteroscedasticity of the forecast errors. A weighted quantile regression is introduced as the third configuration (QR-WT) where weights are assigned during calibration of the quantile regression model to put more emphasis on higher flows, which is the regime of interest for flood forecasting systems. Probabilistic forecasts derived using these three different configurations are compared using different verification measures for the flood forecasting system of Sava River in Slovenia. Results show an improvement of QR-NQT over QR-ORI, largely depending on the size of the training and validation data set and the range of discharges in both data sets. It is also shown that QR-WT outperforms QR-ORI when the highest 25% discharges are considered.

A statistical approach for post-processing residential building energy simulation output

Residential building energy simulation (RBES) software plays an important role in evaluating the energy consumption and efficiency potential of homes. These physics-based models are commonly used to assess the energy performance of homes and to predict benefits of making energy-saving improvements to homes a priori. However, software may produce biased estimates of energy consumption for a variety of reasons, including: errors in the measurement and observation of building characteristics; differences in the assumed versus actual occupant behavior; and errors in the physical models and algorithms used in the software. In order to evaluate and improve the accuracy of RBES software, the National Renewable Energy Laboratory (NREL) has assembled a set of approximately 1,250 U.S. homes for which measured energy consumption and audit-collected household energy characteristics are available. Algorithms have also been developed that automatically translate the data from each home into RBES input files so that model predictions of annual electricity and natural gas consumption can be compared to measured values. To assess and improve upon the accuracy of these predictions, we first cluster the homes using weighted, independent linear combinations of these variables and then build multiple linear regressions within clusters of similar homes to model the difference between measured and predicted energy consumption based on the recorded features of the homes. The statistical post-processing techniques that we develop for RBES models have the following benefits: (1) they can identify variables and algorithms that may be causing inaccuracies in the RBES process and (2) they can be used to adjust and improve the RBES predictions.

Application of a statistical post-processing technique to a gridded, operational, air quality forecast

An automated air quality forecast bias correction scheme based on the short-term persistence of model bias with respect to recent observations is described. The scheme has been implemented in the operational Met Office five day regional air quality forecast for the UK. It has been evaluated against routine hourly pollution observations for a year-long hindcast. The results demonstrate the value of the scheme in improving performance. For the first day of the forecast the post-processing reduces the bias from 7.02 to 0.53 μg m−3 for O3, from −4.70 to −0.63 μg m−3 for NO2, from −4.00 to −0.13 μg m−3 for PM2.5 and from −7.70 to −0.25 μg m−3 for PM10. Other metrics also improve for all species. An analysis of the variation of forecast skill with lead-time is presented and demonstrates that the post-processing increases forecast skill out to five days ahead.

Uncertainty Quantification in Complex Simulation Models Using Ensemble Copula Coupling

Critical decisions frequently rely on high-dimensional output from complex computer simulation models that show intricate cross-variable, spatial and temporal dependence structures, with weather and climate predictions being key examples. There is a strongly increasing recognition of the need for uncertainty quantification in such settings, for which we propose and review a general multi-stage procedure called ensemble copula coupling (ECC), proceeding as follows: 1. Generate a raw ensemble, consisting of multiple runs of the computer model that differ in the inputs or model parameters in suitable ways. 2. Apply statistical postprocessing techniques, such as Bayesian model averaging or nonhomogeneous regression, to correct for systematic errors in the raw ensemble, to obtain calibrated and sharp predictive distributions for each univariate output variable individually. 3. Draw a sample from each postprocessed predictive distribution. 4. Rearrange the sampled values in the rank order structure of the raw ensemble to obtain the ECC postprocessed ensemble. The use of ensembles and statistical postprocessing have become routine in weather forecasting over the past decade. We show that seemingly unrelated, recent advances can be interpreted, fused and consolidated within the framework of ECC, the common thread being the adoption of the empirical copula of the raw ensemble. Depending on the use of Quantiles, Random draws or Transformations at the sampling stage, we distinguish the ECC-Q, ECC-R and ECC-T variants, respectively. We also describe relations to the Schaake shuffle and extant copula-based techniques. In a case study, the ECC approach is applied to predictions of temperature, pressure, precipitation and wind over Germany, based on the 50-member European Centre for Medium-Range Weather Forecasts (ECMWF) ensemble.

Spread Calibration of Ensemble MOS Forecasts

Abstract Ensemble forecasting systems often contain systematic biases and spread deficiencies that can be corrected by statistical postprocessing. This study presents an improvement to an ensemble statistical postprocessing technique, called ensemble kernel density model output statistics (EKDMOS). EKDMOS uses model output statistics (MOS) equations and spread–skill relationships to generate calibrated probabilistic forecasts. The MOS equations are multiple linear regression equations developed by relating observations to ensemble mean-based predictors. The spread–skill relationships are one-term linear regression equations that predict the expected accuracy of the ensemble mean given the ensemble spread. To generate an EKDMOS forecast, the MOS equations are applied to each ensemble member. Kernel density fitting is used to create a probability density function (PDF) from the ensemble MOS forecasts. The PDF spread is adjusted to match the spread predicted by the spread–skill relationship, producing a calibrated forecast. The improved EKDMOS technique was used to produce probabilistic 2-m temperature forecasts from the North American Ensemble Forecast System (NAEFS) over the period 1 October 2007–31 March 2010. The results were compared with an earlier spread adjustment technique, as well as forecasts generated by rank sorting the bias-corrected ensemble members. Compared to the other techniques, the new EKDMOS forecasts were more reliable, had a better calibrated spread–error relationship, and showed increased day-to-day spread variability.

Probabilistic Visibility Forecasting Using Bayesian Model Averaging

Bayesian model averaging (BMA) is a statistical postprocessing technique that has been used in probabilistic weather forecasting to calibrate forecast ensembles and generate predictive probability density functions (PDFs) for weather quantities. The authors apply BMA to probabilistic visibility forecasting using a predictive PDF that is a mixture of discrete point mass and beta distribution components. Three approaches to developing predictive PDFs for visibility are developed, each using BMA to postprocess an ensemble of visibility forecasts. In the first approach, the ensemble is generated by a translation algorithm that converts predicted concentrations of hydrometeorological variables into visibility. The second approach augments the raw ensemble visibility forecasts with model forecasts of relative humidity and quantitative precipitation. In the third approach, the ensemble members are generated from relative humidity and precipitation alone. These methods are applied to 12-h ensemble forecasts from 2007 to 2008 and are tested against verifying observations recorded at Automated Surface Observing Stations in the Pacific Northwest. Each of the three methods produces predictive PDFs that are calibrated and sharp with respect to both climatology and the raw ensemble.

Dynamical, Statistical–Dynamical, and Multimodel Ensemble Forecasts of Australian Spring Season Rainfall

Abstract The prediction skill of the Australian Bureau of Meteorology dynamical seasonal forecast model Predictive Ocean Atmosphere Model for Australia (POAMA) is assessed for probabilistic forecasts of spring season rainfall in Australia and the feasibility of increasing forecast skill through statistical postprocessing is examined. Two statistical postprocessing techniques are explored: calibrating POAMA prediction of rainfall anomaly against observations and using dynamically predicted mean sea level pressure to infer regional rainfall anomaly over Australia (referred to as “bridging”). A “homogeneous” multimodel ensemble prediction method (HMME) is also introduced that consists of the combination of POAMA’s direct prediction of rainfall anomaly together with the two statistically postprocessed predictions. Using hindcasts for the period 1981–2006, the direct forecasts from POAMA exhibit skill relative to a climatological forecast over broad areas of eastern and southern Australia, where El Niño and the Indian Ocean dipole (whose behavior POAMA can skillfully predict at short lead times) are known to exert a strong influence in austral spring. The calibrated and bridged forecasts, while potentially offering improvement over the direct forecasts because of POAMA’s ability to predict the main drivers of springtime rainfall (e.g., El Niño and the Southern Oscillation), show only limited areas of improvement, mainly because strict cross-validation limits the ability to capitalize on relatively modest predictive signals with short record lengths. However, when POAMA and the two statistical–dynamical rainfall forecasts are combined in the HMME, higher deterministic and probabilistic skill is achieved over any of the single models, which suggests the HMME is another useful method to calibrate dynamical model forecasts.

The Effects of Imputing Missing Data on Ensemble Temperature Forecasts

A major issue for developing post-processing methods for NWP forecasting systems is the need to obtain complete training datasets. Without a complete dataset, it can become difficult, if not impossible, to train and verify statistical post-processing techniques, including ensemble consensus forecasting schemes. In addition, when ensemble forecast data are missing, the real-time use of the consensus forecast weighting scheme becomes difficult and the quality of uncertainty information derived from the ensemble is reduced. To ameliorate these problems, an analysis of the treatment of missing data in ensemble model temperature forecasts is performed to determine which method of replacing the missing data produces the lowest Mean Absolute Error (MAE) of consensus forecasts while preserving the ensemble calibration. This study explores several methods of replacing missing data, including ones based on persistence, a Fourier fit to capture seasonal variability, ensemble member mean substitution, three day mean deviation, and an Artificial Neural Network (ANN). The analysis is performed on 48-hour temperature forecasts for ten locations in the Pacific Northwest. The methods are evaluated according to their effect on the forecast performance of two ensemble post-processing forecasting methods, specifically an equal-weight consensus forecast and a ten day performance-weighted window. The methods are also assessed using rank histograms to determine if they preserve the calibration of the ensembles. For both post- processing techniques all imputation methods, with the exception of the ensemble mean substitution, produce mean absolute errors not significantly different from the cases when all ensemble members are available. However, the three day mean deviation and ANN have rank histograms similar to that for the baseline of the non-imputed cases (i.e. the ensembles are appropriately calibrated) for all locations, while persistence, ensemble mean, and Fourier substitution do not consistently produce appropriately calibrated ensembles. The three day mean deviation has the advantage of being computationally efficient in a real-time forecasting environment.

Calibrating Multimodel Forecast Ensembles with Exchangeable and Missing Members Using Bayesian Model Averaging

Abstract Bayesian model averaging (BMA) is a statistical postprocessing technique that generates calibrated and sharp predictive probability density functions (PDFs) from forecast ensembles. It represents the predictive PDF as a weighted average of PDFs centered on the bias-corrected ensemble members, where the weights reflect the relative skill of the individual members over a training period. This work adapts the BMA approach to situations that arise frequently in practice; namely, when one or more of the member forecasts are exchangeable, and when there are missing ensemble members. Exchangeable members differ in random perturbations only, such as the members of bred ensembles, singular vector ensembles, or ensemble Kalman filter systems. Accounting for exchangeability simplifies the BMA approach, in that the BMA weights and the parameters of the component PDFs can be assumed to be equal within each exchangeable group. With these adaptations, BMA can be applied to postprocess multimodel ensembles of any composition. In experiments with surface temperature and quantitative precipitation forecasts from the University of Washington mesoscale ensemble and ensemble Kalman filter systems over the Pacific Northwest, the proposed extensions yield good results. The BMA method is robust to exchangeability assumptions, and the BMA postprocessed combined ensemble shows better verification results than any of the individual, raw, or BMA postprocessed ensemble systems. These results suggest that statistically postprocessed multimodel ensembles can outperform individual ensemble systems, even in cases in which one of the constituent systems is superior to the others.

Statistical Postprocessing of NOGAPS Tropical Cyclone Track Forecasts

Abstract A statistical postprocessing technique is developed and tested to reduce the U.S. Navy global model (NOGAPS) track forecast errors for a sample of western North Pacific tropical cyclones during 1992–96. The key piece of information is the offset of the initial NOGAPS position relative to an updated (here best-track) position that will be known by 6 h after the synoptic times, which is when the NOGAPS forecast is actually available for use by the forecaster. In addition to the basic storm characteristics, the set of 24 predictors includes various segments in the 0–36-h NOGAPS forecast track as well as a 0–36-h backward extrapolation that is compared with the known recent track positions. As statistically significant regressions are only found for 12–36 h, the original 36-h to 72-h NOGAPS track segment is simply translated to the adjusted 36-h position. For the development sample, the adjusted NOGAPS track errors are reduced by about 51 n mi (95 km) at 12 h, 35 n mi (65 km) at 36 h, and 28 n mi (52...