Verification Metrics Research Articles

Rad-cGAN v1.0: Radar-based precipitation nowcasting model with conditional generative adversarial networks for multiple dam domains

Abstract. Numerical weather prediction models and probabilistic extrapolation methods using radar images have been widely used for precipitation nowcasting. Recently, machine-learning-based precipitation nowcasting models have also been actively developed for relatively short-term precipitation predictions. This study was aimed at developing a radar-based precipitation nowcasting model using an advanced machine-learning technique, conditional generative adversarial network (cGAN), which shows high performance in image generation tasks. The cGAN-based precipitation nowcasting model, named Rad-cGAN, developed in this study was trained with the radar reflectivity data of the Soyang-gang Dam basin in South Korea with a spatial domain of 128 × 128 pixels, spatial resolution of 1 km, and temporal resolution of 10 min. The model performance was evaluated using previously developed machine-learning-based precipitation nowcasting models, namely convolutional long short-term memory (ConvLSTM) and U-Net. In addition, Eulerian persistence model and pySTEPS, a radar-based deterministic nowcasting system, are used as baseline models. We demonstrated that Rad-cGAN outperformed reference models at 10 min lead time prediction for the Soyang-gang Dam basin based on verification metrics: Pearson correlation coefficient (R), root mean square error (RMSE), Nash–Sutcliffe efficiency (NSE), critical success index (CSI), and fraction skill scores (FSS) at an intensity threshold of 0.1, 1.0, and 5.0 mm h−1. However, unlike low rainfall intensity, the CSI at high rainfall intensity in Rad-cGAN deteriorated rapidly beyond the lead time of 10 min; however, ConvLSTM and baseline models maintained better performances. This observation was consistent with the FSS calculated at high rainfall intensity. These results were qualitatively evaluated using typhoon Soulik as an example, and through this, ConvLSTM maintained relatively higher precipitation than the other models. However, for the prediction of precipitation area, Rad-cGAN showed the best results, and the advantage of the cGAN method to reduce the blurring effect was confirmed through radially averaged power spectral density (PSD). We also demonstrated the successful implementation of the transfer learning technique to efficiently train the model with the data from other dam basins in South Korea, such as the Andong Dam and Chungju Dam basins. We used the pre-trained model, which was completely trained in the Soyang-gang Dam basin. Furthermore, we analyzed the amount of data to effectively develop the model for the new domain through the transfer learning strategies applying the pre-trained model using data for additional dam basins. This study confirmed that Rad-cGAN can be successfully applied to precipitation nowcasting with longer lead times and using the transfer learning approach showed good performance in dam basins other than the originally trained basin.

A New Validation Metric for Models With Correlated Responses Using Limited Experimental and Simulation Data

Abstract Model validation is the process of determining the degree to which a model is an accurate representation of the real object. Most of the existing model verification metrics rely on massive data, which are expensive to obtain in complex engineering problems. This paper first proposes a new enhanced Mahalanobis distance (EMD) metric by multiplying the original Mahalanobis distance with a direction angle to incorporate the correlation information using limited experimental and simulation data. Combining with EMD, an angle metric is developed as an alternative of the area metric to reduce the misjudgment rate of model validation. In order to quantify the uncertainty due to insufficient experimental and simulation data, the angle metric is further extended to a new interval angle metric, namely the interval EMD-pooling angle metric as the ultimate metric, for validating models at multiple sites. The proposed interval EMD-pooling angle metric is compared with other existing metrics through three numerical case studies to demonstrate its advantages when both experimental and simulation data are insufficient. An engineering example regarding ultrasonic welding is also provided to demonstrate the effectiveness of the proposed metric for practical model validation problems.

Multi-objective optimization of energy conservation and emission reduction in China's iron and steel industry based on dimensionality reduction

With increasingly stringent environmental restrictions, the iron and steel industry is faced with multiple targets such as energy conservation, emission reduction, and cost control, hence, making industrial-environmental management a multi-objective optimization (MOO) problem. Existing studies adopt MOO models to address this issue, however, their methods result in high-dimension problems that increase computational cost and the difficulty to formulate optimal management schemes. This study builds a constrained MOO model in China's iron and steel industry, and introduces a dimensionality reduction technique to identify redundant objectives which can be omitted without altering the Pareto dominance structure of the original high-dimension problem. NSGA-II is then applied on the resulting lower-dimension problem to optimize the application of five types of Energy Conservation and Emission Reduction (ECER) measures, consisting of 23-process equipment and 55-ECER technologies. Results show that: (1) The high-dimension problem can be reduced to a four objective optimization problem. The four key-objectives are cost, energy, CO2, and PM intensity control. Hence, ECER policies should focus on these four key-objectives since optimizing them will synergistically minimize redundant objectives like NOx and SO2 emissions; (2) For the resulting lower-dimension problem, solution quality increases as reflected by algorithm verification metrics (spacing metric and hypervolume indicator), while computational cost is reduced by 73% and the decision making process is simplified significantly; (3) By-production reutilization and cleaner production technologies are the most attractive measures, contributing about 60% of ECER potential and only 30% of total cost. Equipment upgrading also has good ECER effects, but requires strong economic incentives, while end-of-pipe and renewable energy technologies have the lowest ECER potential. In sum, this study proposed a methodology to reduce the complexity of ECER optimization under multiple objectives, and provided suggestions to enhance industrial-environmental management under increasing environmental restrictions.

Classification of remote sensing time series and similarity metrics for crop type verification

One of the greatest challenges in many machine learning applications is to build robust classification models that reduce calibration efforts, ensure model transferability to unseen data distributions, and handle high semantic resolution datasets with a relatively large number of target classes. We present an operational domain adaptation framework for crop type verification from remote sensing data for integrated administration and control and farm management systems. It is part of the ag|knowledge crop monitoring platform and demonstrates a more robust spatiotemporal model transferability than conventional supervised crop type classification approaches, which use multitemporal and multispectral features to classify the specific characteristics of crop phenology patterns. The proposed method is based on a machine learning model that has been trained on particular similarity metrics between all target crop types. The metrics are derived by quantifying the similarity of vegetation index time series between the observed parcel and the aggregated time series of reference objects for each parcel and crop type. This method achieves higher classification results than a model trained on the pure remote sensing time series data, as the algorithm does not learn the temporal vegetation index patterns, but instead, the adversarial characteristics and the differences between each crop type. Moreover, it also reduces dimensionality as the time series are summarized through the respective similarity metrics. We compare both approaches through different classification algorithms. Overall, the achieved classification accuracy for more than 67 crop types scores more than 80%.

Spatial verification of ensemble rainfall forecasts over India

Increased resolution of Numerical Weather Prediction (NWP) models can lead to improved prediction of heavy rains; however, the forecasts often suffer from location mismatch and intensity errors which can lead to poor traditional verification scores. Spatial verification can provide more realistic statistics to ascertain the quality of forecasts. This study applies contiguous rain areas (CRA), an object based spatial verification method, to evaluate ensemble rainfall forecasts from the NCMRWF Ensemble Prediction System (NEPS-G) over Indian land region for recent three monsoon seasons (2018–2020) using four thresholds 10, 20, 40 and 80 mm/d. The main aim in this study is (a) to quantify the spatial errors and contribution from error components (displacement, volume and pattern errors) which is achieved by the analysis of results from rainfall forecasts during three monsoon seasons over India; (b) to determine if spread-skill relationship observed using traditional measures is also evident in all or some of the object parameters / forecast attributes (followingGallus, 2010); (c) to assess the skill of probabilistic forecast of the object parameters / forecast attributes. This study assesses the skill in the probabilistic forecasts of attributes. Verification of these probabilistic forecasts of attributes is presented in terms of standard verification metrics like Spread Vs Skill, BS, reliability diagram and ROC curve. The analysis of contribution from displacement, volume and pattern errors to the total error shows that the volume contributes the least followed by displacement and finally pattern (~70%). For heavier rainfall events, the pattern is better matched and contribution from displacement and volume is higher. Spread-skill relationship shows that the uncertainties in the NEPS system are better represented for the rainfall area and volume over the west coast and for maximum rainfall in the core monsoon region. Error in the mean of all the attributes is seen to be lower in the west coast as compared to the core monsoon region. Higher rainfall intensity, smaller areas and higher volume are better predicted in terms of having lower BS, reliability values (as a component of BS) and higher BSS and AROC. Similarly these attributes also have ROC curve more aligned towards the top left diagonal and the reliability curve more aligned along the diagonal line of perfect reliability. Spatial verification results indicate that rainfall area and volume demonstrate higher reliability and skill in the forecasts and intensity of rainfall is still a challenging attribute to predict.

Cover Image, Volume 9, Issue 2

The cover image is based on the Advanced Review Predictive performance of ensemble hydroclimatic forecasts: Verifi cation metrics, diagnostic plots and forecast attributes by Zeqing Huang and Tongtiegang Zhao https://doi.org/10.1002/wat2.1580.image

Predictive performance of ensemble hydroclimatic forecasts: Verification metrics, diagnostic plots and forecast attributes

AbstractPredictive performance is one of the most important issues for practical applications of ensemble hydroclimatic forecasts. While different forecasting studies tend to use different combinations of verification metrics and diagnostic plots, this paper presents a literature survey of popular verification metrics, diagnostic plots and their corresponding forecast attributes. Twenty metrics to quantify predictive performance of ensemble, deterministic and categorical forecasts are detailed; six types of diagnostic plots to visualize the relationship between ensemble forecasts and observations are presented; and the correspondence relationships of verification metrics and diagnostic plots with ten forecast attributes are illustrated. Numerical experiments are devised for raw ensemble precipitation forecasts generated by the Climate Forecast System version 2 (CFSv2) to illustrate the verification metrics, diagnostic plots and forecast attributes. The results suggest that verification metrics that pay attention to some similar attributes can be related to one another. Meanwhile, the metrics are not necessarily in full accordance as they tend to reflect different attributes. Time series plots visualize forecasts over a period of interest and graphically illustrate forecast bias, reliability and skill. Quantile range plots provide an overall diagnosis for the association of ensemble forecasts with observations. Overall, verification metrics can be illustrated by diagnostic plots for individual grid cells and also be screened by spatial plots at regional and global scales. The survey presented in this article can serve as a roadmap for the selection of verification metrics, diagnostic plots and forecast attributes for verifying ensemble hydroclimatic forecasts.This article is categorized under: Science of Water > Hydrological Processes

Evaluation of GPM-IMERG rainfall estimates at multiple temporal and spatial scales over Greece

Accurate precipitation data in space and time are among the most important variables for analyzing, modelling and forecasting many hydrometeorological applications and hazards. For countries like Greece, which have a complex terrain and great spatial variability of precipitation, satellite precipitation products (SPPs) are an alternative source of precipitation information that could complement the sparse rain gauge network. The objective of this study is to evaluate the Integrated Multi-satellitE Retrievals for Global Precipitation Measurement (IMERG) V06B Final-run product over Greece for a 5-year period (January 2015 to December 2019) using as reference measurements from 263 rain gauges. The evaluation of IMERG is performed at multiple temporal (annual, monthly, daily) and spatial (country, region, grid) scales by employing several statistical and graphical methods. The rainfall detection capability of IMERG is examined with categorical verification metrics and the accuracy of the estimated rainfall amounts with continuous verification metrics. In addition, the performance of IMERG under different rainfall intensities, elevations and seasons is examined.On the monthly and annual scale evaluation the intercomparison between the calibrated (IMERGcal) and uncalibrated (IMERGuncal) IMERG estimates revealed the superiority of IMERGcal, but on the same time it highlighted the weakness of the GPCC gauge-based calibration product over Greece. The monthly IMERGcal estimates could reproduce the annual cycle of precipitation with a slight overestimation of monthly precipitation (RBIAS = 5.1%). On a daily scale IMERGcal performed well on detecting rainfall events (POD = 0.77), but it also produced many false alarm events (FAR = 0.40) which are primarily found over the islands of Aegean and Ionian Sea and along the coastline of mainland Greece. The assessment of daily rainfall intensities by IMERGcal showed a moderate correlation across Greece (0.49), with overestimation of light and moderate precipitation events (0.4–20 mm/day) and underestimation of heavy rainfall events (>20 mm/day). On a regional scale, IMERGcal tends to underestimate the daily rainfall over the rainfall-prone western part of Greece because in addition to the smoothing of high rainfall values by the GPCC calibration, satellite sensors fail to detect the magnitude of orographic rainfall. The complete opposite situation was found over the Aegean and Ionian Sea and many areas of the coastline, with great overestimation of rainfall. The results of our study highlight the potentials and limitations of IMERG rainfall estimates over Greece and they can be very helpful for future users of IMERG in Greece.

Probabilistic Predictions from Deterministic Atmospheric River Forecasts with Deep Learning

Abstract Deep-learning (DL) postprocessing methods are examined to obtain reliable and accurate probabilistic forecasts from single-member numerical weather predictions of integrated vapor transport (IVT). Using a 34-yr reforecast, based on the Center for Western Weather and Water Extremes West-WRF mesoscale model of North American West Coast IVT, the dynamically/statistically derived 0–120-h probabilistic forecasts for IVT under atmospheric river (AR) conditions are tested. These predictions are compared with the Global Ensemble Forecast System (GEFS) dynamic model and the GEFS calibrated with a neural network. In addition, the DL methods are tested against an established, but more rigid, statistical–dynamical ensemble method (the analog ensemble). The findings show, using continuous ranked probability skill score and Brier skill score as verification metrics, that the DL methods compete with or outperform the calibrated GEFS system at lead times from 0 to 48 h and again from 72 to 120 h for AR vapor transport events. In addition, the DL methods generate reliable and skillful probabilistic forecasts. The implications of varying the length of the training dataset are examined, and the results show that the DL methods learn relatively quickly and ∼10 years of hindcast data are required to compete with the GEFS ensemble.

Problems and solutions regarding generalized functional safety in cyberspace

The common endogenous security problems in cyberspace and related attack threats have posed subversive challenges to conventional theories and methods of functional safety. In the current design of the cyber physical system (CPS), functional safety and cyber security are increasingly intertwined and inseparable, which evolve into the generalized functional safety (S&S) problem. The conventional reliability and cybersecurity technologies are unable to provide security assurance with quantifiable design and verification metrics in response to the cyberattacks in hardware and software with common endogenous security problems, and the functional safety of CPS facilities or device has become a frightening ghost. The dynamic heterogeneity redundancy (DHR) architecture and coding channel theory (CCT) proposed by the cyberspace endogenous security paradigm could handle random failures and uncertain network attacks in an integrated manner, and its generalized robust control mechanism can solve the universal problem of quantitative design for functional safety under probability or improbability perturbation. As a generalized functional safety enabling structure, DHR opens up a new direction to solve the common endogenous security problems in the cross-disciplinary fields of cyberspace.

APS2-ACCESS-C2: the first Australian operational NWP convection-permitting model

The Australian Bureau of Meteorology’s ‘Australian Parallel Suite’ (APS) operational numerical weather prediction regional Australian Community Climate and Earth-System Simulator (ACCESS) city-based system (APS1 ACCESS-C1) was updated in August 2017 with the commissioning of the APS2 ACCESS-C2. ACCESS-C2 runs over six regional domains. Significant upgrade changes included implementation of Unified Model 8.2 code; nesting in the 12 km resolution APS2 ACCESS-R2 regional model; and, importantly, an increased horizontal resolution from 4 to 1.5 km, enabling C2 to become the first Australian operational convection-permitting model (CPM). Traditional rainfall verification metrics and Fractions Skill Score show C2 forecast skill over ACCESS-C domains in summer and winter was generally, and in many cases, significantly better than C1. Case studies showed that C2 forecasts had better-detailed wind and precipitation fields, particularly at longer forecast ranges and higher rain rates. The improvements in C2 forecasts were principally due to its CPM ability to simulate high temporal and spatial resolution features, which continue to be of great interest to forecasters. C2 also laid the groundwork for the present day APS3 ACCESS-C forecast C3 and ensemble CE3 models and further development of higher resolution (down to 300 m) fire weather and urban models.

Lagrangian Integro-Difference Equation Model for Precipitation Nowcasting

Abstract Delivering reliable nowcasts (short-range forecasts) of severe rainfall and the resulting flash floods is important in densely populated urban areas. The conventional method is advection-based extrapolation of radar echoes. However, during rapidly evolving convective rainfall this so-called Lagrangian persistence (LP) approach is limited to deterministic and very short-range nowcasts. To address these limitations in the 1-h time range, a novel extension of LP, called Lagrangian Integro-Difference equation model with Autoregression (LINDA), is proposed. The model consists of five components: 1) identification of rain cells, 2) advection, 3) autoregressive process describing growth and decay of the cells, 4) convolution describing loss of predictability at small scales, and 5) stochastic perturbations to simulate forecast uncertainty. Advection is separated from the other components that are applied in the Lagrangian coordinates. The reliability of LINDA is evaluated using the NEXRAD WSR-88D radar that covers the Dallas–Fort Worth metropolitan area, as well as the NEXRAD mosaic covering the continental United States. This is done with two different configurations: LINDA-D for deterministic and LINDA-P for probabilistic nowcasts. The validation dataset consists of 11 rainfall events during 2018–20. For predicting moderate to heavy rainfall (5–20 mm h−1), LINDA outperforms the previously proposed LP-based approaches. The most significant improvement is seen for the ETS and POD statistics with the 5 mm h−1 threshold. For 30-min nowcasts, they show 15% and 16% increases, respectively, to the second-best method and 48% and 34% increases compared to LP. For the 5 mm h−1 threshold, the increase in the relative operating characteristic (ROC) skill score of 30-min nowcasts from the second-best method is 10%. Significance Statement Delivering reliable forecasts of severe rainfall for the next few hours has a major societal importance. This is particularly true for densely populated urban areas, where flash floods can cause property damage and loss of lives. Such forecasts are conventionally produced by direct extrapolation of weather radar measurements. However, for intense localized rainfall this approach has low prediction ability beyond 30 min. To extend this limit, we propose a novel method that combines machine vision with a statistical model for growth and decay of rainfall. The method is designed for predicting highly localized rain cells and bands. In addition, a stochastic extension for producing probabilistic forecasts is developed. Using several verification metrics, we demonstrate that for predicting moderate to heavy rainfall (5–20 mm h−1), the proposed method has significantly improved forecast skill compared to the reference methods. The evaluation is done by using the NEXRAD WSR-88D that covers the Dallas–Fort Worth urban metroplex with a population of over 7 million. Demonstration of the applicability of LINDA in a larger domain is done by using the NEXRAD radar network that covers the continental United States.

HUP‐BMA: An Integration of Hydrologic Uncertainty Processor and Bayesian Model Averaging for Streamflow Forecasting

AbstractUncertainty quantification and providing probabilistic streamflow forecasts are of particular interest for water resource management. The hydrologic uncertainty processor (HUP) is a well‐known Bayesian approach used to quantify hydrologic uncertainty based on observations and deterministic forecasts. This uncertainty quantification is model‐specific; however, utilizing information from multiple hydrologic models should be advantageous and should lead to better probabilistic forecasts. Using seven, structurally different, conceptual models, this study first aims at evaluating the effects of implementing different hydrologic models on HUP performance. Second, using the concepts of the Bayesian Model Averaging (BMA) approach, a multimodel HUP‐based Bayesian postprocessor (HUP‐BMA) is proposed where the combination of posterior distributions derived from HUP with different hydrologic models are used to better quantify the hydrologic uncertainty. All postprocessing approaches are applied for medium‐range daily streamflow forecasting (1–14 days ahead) in two watersheds located in Ontario, Canada. The results indicate that the HUP forecasts for short lead‐times are negligibly affected by implementing different hydrologic models, while with increasing lead‐time and flow magnitude, they significantly depend on the quality of the deterministic forecast. Moreover, the superiority of the proposed HUP‐BMA method over HUP is demonstrated based on various verification metrics in both watersheds. Additionally, HUP‐BMA outperformed the original BMA in quantifying hydrologic uncertainty for short lead‐times. However, by increasing lead‐time, considering the effects of initially observed flow on HUP‐BMA formulation may not be beneficial. So, its modified version unconditioned on initial observations is preferred.

Many-objective optimization of energy conservation and emission reduction in China’s cement industry

Industrial Energy Conservation and Emission Reduction (ECER) management is challenging due to the large number of objectives involved, and the complex synergies/conflicts amongst them. Most studies do not quantify these synergies/conflicts, resulting in high dimension ECER management problems. This study builds a 10-objective problem to optimize 5 heavy metals (Cd, Hg, Pb, As, Cr) and 5 conventional ECER-objectives (economic cost, energy, CO2, PM and NOx) in China’s cement industry. First, NSGA-III is used to calculate optimal objective values under technology application constraints. Second, synergies/conflicts amongst objectives are quantified using Spearman’s correlation. Finally, TOPSIS, Conservation Supply Curves and Quadrant methods are used to generate optimal ECER management policies. Results show that: (1) Optimal solutions are reliable as reflected by algorithm verification metrics – Error rate, Spacing metric and Hypervolume indicator; (2) The 10-objective problem is reduced to a 4-objective problem based on synergies. The four main-objectives are economic cost, energy consumption, CO2 and PM emission control. Therefore, ECER policies should focus on these objectives since optimizing them will synergistically improve all other objectives; (3) The average reduction potential for HMs and conventional objectives is 20% and 25% respectively, with an increase in economic cost of about 44–83 CNY/t; (4) From 72-ECER strategies assessed, 11-key strategies have better economic and environmental performances. They are mostly circular economy strategies utilizing industrial wastes, indicating that ECER policies should shift from traditional end-of-pipe approaches to a more circular economy approach. In sum, the proposed methodology can reduce the complexity of many-objective ECER management through a synergic control.

Evaluation of the Skill of Monthly Precipitation Forecasts from Global Prediction Systems over the Greater Horn of Africa

AbstractThe skill of precipitation forecasts from global prediction systems has a strong regional and seasonal dependence. Quantifying the skill of models for different regions and time scales is important, not only to improve forecast skill, but to enhance the effective uptake of forecast information. The Subseasonal to Seasonal Prediction project (S2S) database contains near-real-time forecasts and reforecasts from 11 operational centers and provides a great opportunity to evaluate and compare the skill of operational S2S systems. This study evaluates the skill of these state-of-the-art global prediction systems in predicting monthly precipitation over the Greater Horn of Africa. This comprehensive evaluation was performed using deterministic and probabilistic forecast verification metrics. Results from the analysis showed that the prediction skill varies with months and region. Generally, the models show high prediction skill during the start of the rainfall season in March and lower prediction skill during the peak of the rainfall in April. ECCC, ECMWF, KMA, NCEP, and UKMO show better prediction skill over the region for most of the months compared with the rest of the models. Conversely, BoM, CMA, HMCR, and ISAC show poor prediction skill over the region. Overall, the ECMWF model performs best over the region among the 11 models analyzed. Importantly, this study serves as a baseline skill assessment with the findings helping to inform how a subset of models could be selected to construct an objectively consolidated multimodel ensemble of S2S forecast products for the Greater Horn of Africa region, as recommended by the World Meteorological Organization.

Forecast verification: Relating deterministic and probabilistic metrics

AbstractThe philosophy of forecast verification is rather different between deterministic and probabilistic verification metrics: generally speaking, deterministic metrics measure differences, whereas probabilistic metrics assess reliability and sharpness of predictive distributions. This article considers the root‐mean‐square error (RMSE), which can be seen as a deterministic metric, and the probabilistic metric Continuous Ranked Probability Score (CRPS), and demonstrates that under certain conditions, the CRPS can be mathematically expressed in terms of the RMSE when these metrics are aggregated. One of the required conditions is the normality of distributions. The other condition is that, while the forecast ensemble need not be calibrated, any bias or over/underdispersion cannot depend on the forecast distribution itself. Under these conditions, the CRPS is a fraction of the RMSE, and this fraction depends only on the heteroscedasticity of the ensemble spread and the measures of calibration. The derived CRPS–RMSE relationship for the case of perfect ensemble reliability is tested on simulations of idealised two‐dimensional barotropic turbulence. Results suggest that the relationship holds approximately despite the normality condition not being met.

Postprocessing continental-scale, medium-range ensemble streamflow forecasts in South America using Ensemble Model Output Statistics and Ensemble Copula Coupling

Probabilistic hydrological forecasting and ensemble techniques have leveraged streamflow prediction at regional to continental scales up to several weeks in advance. However, ensembles that only account for meteorological forecast uncertainty are typically biased and subject to dispersion errors, thus limiting their use for rational decision-making and optimization systems. Statistical postprocessing is therefore necessary to convert ensemble forecasts into calibrated and sharp predictive distributions, and it should also account for dependencies between lead times to enable realistic forecast trajectories. This work provides a continental-scale assessment of the use of statistical postprocessing on medium-range, ensemble streamflow forecasts over South America (SA). These forecasts were produced through a large-scale hydrologic–hydrodynamic model forced with a global precipitation dataset and ECMWF reforecast data. The Ensemble Model Output Statistics (EMOS) technique was used to generate conditional predictive distributions in 488 locations at each forecast lead time, while the Ensemble Copula Coupling method with the transformation scheme (ECC-T) was applied to derive ensemble traces from EMOS distributions. Postprocessed streamflow forecasts were cross-validated for the period from 1996 to 2014 using a range of verification metrics. Results showed that the skill and reliability of EMOS forecasts substantially improve over the raw ensembles, and that EMOS leads to skillful predictions relative to discharge climatology and persistence forecasts up to 15 days in advance in most locations. Furthermore, EMOS results in predictive distributions that are generally sharper than the climatology. Limitations in depicting autocorrelations of forecast trajectories were observed in rivers for which the raw ensemble spread is very low and EMOS has to largely increase dispersion, especially at short lead times. The study’s findings suggest that combining a continental-scale hydrological model with EMOS and ECC-T methods can lead to skillful predictions and realistic ensemble traces in several locations in SA, even if in situ hydrometeorological observations are not available in real time.

WRF Precipitation Performance and Predictability for Systematically Varied Parameterizations over Complex Terrain

AbstractPhysics parameterizations in the Weather Research and Forecasting (WRF) Model are systematically varied to investigate precipitation forecast performance over the complex terrain of southwest British Columbia (BC). Comparing a full year of modeling data from over 100 WRF configurations to station observations reveals sensitivities of precipitation intensity, season, location, grid resolution, and accumulation window. The choice of cumulus and microphysics parameterizations is most important. The WSM5 microphysics scheme yields competitive verification scores when compared to more sophisticated and computationally expensive parameterizations. Although the scale-aware Grell–Freitas cumulus parameterization performs better for summertime convective precipitation, the conventional Kain–Fritsch parameterization better simulates wintertime frontal precipitation, which contributes to the majority of the annual precipitation in southwest BC. Finer grid spacings have lower relative biases and a more realistic spread in precipitation intensity distribution, yet higher relative standard deviations of their errors—they produce finer spatial differences and local extrema. Finer resolutions produce the best fraction of correct-to-incorrect forecasts across all precipitation intensities, whereas the coarser 27-km domain yields the highest hit rates and equitable threat scores. Verification metrics improve greatly with longer accumulation windows—hourly precipitation values are prone to double-penalty issues, while longer accumulation windows compensate for timing errors but lose information about short-term precipitation intensities. This study provides insights regarding WRF precipitation performance in complex terrain across a wide variety of configurations, using metrics important to a range of end users.

Optimizing Precipitation Forecasts for Hydrological Catchments in Ethiopia Using Statistical Bias Correction and Multi‐Modeling

AbstractAccurate rainfall forecasts on timescales ranging from a few hours to several weeks are needed for many hydrological applications. This study examines bias, skill and reliability of four ensemble forecast systems (from Canada, UK, Europe, and the United States) and a multi‐model ensemble as applied to Ethiopian catchments. By verifying these forecasts on hydrological catchments, we focus on spatial scales that are relevant to many actual water forecasting applications, such as flood forecasting and reservoir optimization. By most verification metrics tested, the bias corrected European model is the best individual model at predicting daily rainfall variations, while the Canadian model shows the most realistic ensemble spread and thus the most reliable forecast probabilities, including those of extreme events. The skill of the multi‐model ensemble outperforms individual models by most metrics, and is skillful up to 9 days ahead. Skill is higher for the 0–5 day model accumulation than for the first 24 h, suggesting that timing errors strongly penalize the skill of forecasts with shorter accumulation periods. Due to seasonality in the model biases, bias correction is best applied to each month individually. Forecasting extreme rainfall is a challenge for Ethiopia, especially over mountainous regions where positive skill is only reached after bias correction. Compared to individual models, the multi‐model ensemble has a higher probability of detecting extreme rainfall and a lower false alarm rate, with usable skill at 24 h lead times.

Probabilistic Robustness Quantification of Neural Networks

Safety properties of neural networks are critical to their application in safety-critical domains. Quantification of their robustness against uncertainties is an upcoming area of research. In this work, we propose an approach for providing probabilistic guarantees on the performance of a trained neural network. We present two novel metrics for probabilistic verification on training data distribution and test dataset. Given a trained neural network, we quantify the probability of the model to make errors on a random sample drawn from the training data distribution. Second, from the output logits of a sample test point, we measure its p-value on the learned logit distribution to quantify the confidence of the model at this test point. We compare our results with softmax based metric using the black-box adversarial attacks on a simple CNN architecture trained for MNIST digit classification.