Brier Skill Score Research Articles

Skill of large-scale seasonal drought impact forecasts

Abstract. Forecasting of drought impacts is still lacking in drought early-warning systems (DEWSs), which presently do not go beyond hazard forecasting. Therefore, we developed drought impact functions using machine learning approaches (logistic regression and random forest) to predict drought impacts with lead times up to 7 months ahead. The observed and forecasted hydrometeorological drought hazards – such as the standardized precipitation index (SPI), standardized precipitation evaporation index (SPEI), and standardized runoff index (SRI) – were obtained from the The EU-funded Enhancing Emergency Management and Response to Extreme Weather and Climate Events (ANYWHERE) DEWS. Reported drought impact data, taken from the European Drought Impact Report Inventory (EDII), were used to develop and validate drought impact functions. The skill of the drought impact functions in forecasting drought impacts was evaluated using the Brier skill score and relative operating characteristic metrics for five cases representing different spatial aggregation and lumping of impacted sectors. Results show that hydrological drought hazard represented by SRI has higher skill than meteorological drought represented by SPI and SPEI. For German regions, impact functions developed using random forests indicate a higher discriminative ability to forecast drought impacts than logistic regression. Moreover, skill is higher for cases with higher spatial resolution and less lumped impacted sectors (cases 4 and 5), with considerable skill up to 3–4 months ahead. The forecasting skill of drought impacts using machine learning greatly depends on the availability of impact data. This study demonstrates that the drought impact functions could not be developed for certain regions and impacted sectors, owing to the lack of reported impacts.

Prognostic utilization of models based on the APACHE II, APACHE IV, and SAPS II scores for predicting in-hospital mortality in emergency department

BackgroundThis study was designed to evaluate and compare the prognostic value of the APACHE II, APACHE IV, and SAPSII scores for predicting in-hospital mortality in the ED on a large sample of patients. Earlier studies in the ED setting have either used a small sample or focused on specific diagnoses. MethodsA prospective study was conducted to include patients with higher risk of mortality from March 2016 to March 2017 in the ED of Emam Reza Hospital, northeast of Iran. Logistic regression was used to develop three models. Evaluation was performed in terms of the overall performance (Brier Score, BS, and Brier Skill Score, BSS), discrimination (Area Under the Curve, AUC), and calibration (calibration graph). ResultsA total of 2205 patients met the study criteria (53% male and median age of 64, IQR: 50–77). In-hospital mortality amounted to 19%. For APACHE II, APACHE IV, and SAPS II the BS was 0.132, 0.125 and 0.133 and the BSS was 0.156, 0.2, and 0.144, respectively. The AUC was 0.755 (0.74 to 0.779) for APACHE II, 0.794 (0.775 to 0.818) for APACHE IV, and 0.751 (0.727 to 0.776) for SAPS II. The APACHE IV showed significantly greater AUC in comparison to the APACHE II and SAPS II. The graphical evaluation revealed good calibration of the APACHE IV model. ConclusionAPACHEIV outperformed APACHEII and SAPSII in terms of discrimination and calibration. More validation is needed for using these models for decision-making about individual patients, although they would perform best at a cohort level.

External validation of existing prediction models of 30-day mortality after Transcatheter Aortic Valve Implantation (TAVI) in the Netherlands Heart Registration

BackgroundSeveral mortality prediction models (MPM) are used for predicting early (30-day) mortality following transcatheter aortic valve implantation (TAVI). Little is known about their predictive performance in external TAVI populations. We aim to externally validate established MPMs on a large TAVI dataset from the Netherlands Heart Registration (NHR). MethodsWe included data from NHR-patients who underwent TAVI during 2013–2017. We calculated the predicted mortalities per MPM. We assessed the predictive performance by discrimination (Area Under Receiver Operating-characteristic Curve, AU-ROC); the Area Under Precision-Recall Curve, AU-PRC; calibration (using calibration-intercept and calibration-slope); Brier Score and Brier Skill Score. We also assessed the predictive performance among subgroups: tertiles of mortality-risk for non-survivors, gender, and access-route. ResultsWe included 6177 TAVI-patients with an observed early-mortality rate of 4.5% (n = 280). We applied seven MPMs (STS, EuroSCORE-I, EuroSCORE-II, ACC-TAVI, FRANCE-2, OBSERVANT, and German-AV) on our cohort. The highest AU-ROCs were 0.64 (95%CI 0.61–0.67) for ACC-TAVI and 0.63 (95%CI 0.60–0.67) for FRANCE-2. All MPMs had a very low AU-PRC of ≤0.09. ACC-TAVI had the best calibration-intercept and calibration-slope. Brier Score values ranged between 0.043 and 0.063. Brier Skill Score ranged between −0.47 and 0.004. ACC-TAVI and FRANCE-2 predicted high mortality-risk better than other MPMs. ACC-TAVI outperformed other MPMs in different subgroups. ConclusionThe ACC-TAVI model has relatively the best predictive performance. However, all models have poor predictive performance. Because of the poor discrimination, miscalibration and limited accuracy of the models there is a need to update the existing models or develop new TAVI-specific models for local populations.

Improvements in tropical cyclone forecasting through ensemble prediction system at NCMRWF in India

This paper deals with the comparison of cyclone forecasts from the two versions of the operational global ensemble prediction system (EPS) at the National Centre for Medium Range Weather Forecasting (NEPS). The previous version had a horizontal resolution of 33 km with 44 ensemble members (NEPS) whereas the updated version of this EPS has a resolution of 12 km with 11 members (NEPS-UP). The ensemble mean forecasts from both the models are compared using the direct position (DPE), along (ATE) and cross track (CTE) errors. For the verification of strike probability, Brier Score (BS), Brier Skill Score (BSS), Reliability Diagram, Relative Operating Characteristic (ROC) Curve and Root Mean Square Error (RMSE) in mean Vs Spread in members are used. For verification of intensity, RMSE in maximum wind speed from the ensemble mean forecasts are compared.Comparison of ensemble mean tracks from both models showed lower errors in NEPS-UP for all forecast lead times. The decrease in the DPE, ATE and CTE in NEPS-UP was around 38%, 48% and 15% respectively. NEPS-UP showed lower BS and higher BSS values indicating a better match between observed frequencies and forecast probabilities as well as higher prediction skills. The reliability diagram showed higher accuracy for NEPS-UP as compared to NEPS. The ROC curves showed that for forecasts with higher probabilities the hit rate was high in NEPS-UP. There was a greater consensus between the RMSE and Spread for NEPS-UP at all lead times. It was also seen that the RMSE in mean showed a 41% decrease from NEPS to NEPS-UP. On comparing maximum wind, it was found that for all lead times the RMSE in maximum wind speed for NEPS-UP was lower than NEPS.

Subseasonal Predictions of Tropical Cyclone Occurrence and ACE in the S2S Dataset

Abstract Probabilistic tropical cyclone (TC) occurrence, at lead times of week 1–4, in the Subseasonal to Seasonal (S2S) dataset are examined here. Forecasts are defined over 15° in latitude × 20° in longitude regions, and the prediction skill is measured using the Brier skill score with reference to climatological reference forecasts. Two types of reference forecasts are used: a seasonally constant one and a seasonally varying one, with the latter used for forecasts of anomalies from the seasonal climatology. Models from the European Centre for Medium-Range Weather Forecasts (ECMWF), Australian Bureau of Meteorology, and Météo-France/Centre National de Recherche Météorologiques have skill in predicting TC occurrence four weeks in advance. In contrast, only the ECMWF model is skillful in predicting the anomaly of TC occurrence beyond one week. Errors in genesis prediction largely limit models’ skill in predicting TC occurrence. Three calibration techniques, removing the mean genesis and occurrence forecast biases, and a linear regression method, are explored here. The linear regression method performs the best and guarantees a higher skill score when applied to the in-sample dataset. However, when applied to the out-of-sample data, especially in areas where the TC sample size is small, it may reduce the models’ prediction skill. Generally speaking, the S2S models are more skillful in predicting TC occurrence during favorable Madden–Julian oscillation phases. Last, we also report accumulated cyclone energy predictions skill using the ranked probability skill score.

Calibration Of Rainfall Ensemble Prediction Of ECMWF System 4 Using Bayesian Model Averaging

Bayesian model averaging (BMA) is a statistical post-processing method for producing probabilistic forecasts from ensembles in the form of predictive PDFs. It is known that BMA is able to improve the reliability of probabilistic forecast of short range and medium range rainfall forecast. This study aims to develop the application of BMA to calibrate seasonal forecast (long range) in order to improved quality of seasonal forecast in Indonesia. The seasonal forecast used is monthly rainfall from the output of the ensemble prediction system European Center for Medium-Range Weather Forecasts (ECMWF) system 4 model (ECS4) and it is calibrated against observational data at 26 stations of the Agency for Meteorology Climatology and Geophysics of Republic of Indonesia (BMKG) in Java Island in 1981 – 2018. BMA predictive PDFs is generated with Gamma distribution approach which is obtained based on sequential training windows (JTS) and conditionals training windows (JTC). BMA-JTS approach is done by selecting the width of the 30-month training window as the optimal training period while the BMA-JTC is carried out with a cross-validation scheme for each month. In general, Both of BMA-JTS and BMA-JTC better than RAW models. BMA-JTC calibration results are varying according to spatial and temporal, but in general the result is better in the dry season and during the El Nino phase. BMA is able to improve the distribution characteristics of the RAW model ECS4 prediction which is shown by: a smaller value of Continuous Rank Probability Score (CRPS), a larger value of the Continuous Rank Probability Skill Score (CRPSS) and more flat form of the Verification Rank Histogram (VRH) than the RAW model. BMA also increases the skill, esolution and reliability of prediction of probability Below Normal (BN) and Above Normal (AN), which is known from the increasing Brier Skill Score (BSS), and the increasing area under curve of Relative Operating Characteristics (ROC) compared to the RAW model. Furthermore, the reliability of BN and AN of BMA results also has the category of “still very useful” and “perfect” compared to RAW models that are in the “dangerous”, “not useful” and “marginally useful” categories. The reliability of BMA results with the category “still very useful” and “perfect” show that the probabilistic forecast of BN and AN events can be used in making decisions related to seasonal forecast.

Performance of the HWRF Rapid Intensification Analog Ensemble (HWRF RI-AnEn) during the 2017 and 2018 HFIP Real-Time Demonstrations

Abstract The performance of the Hurricane Weather Research and Forecasting (HWRF) Model Rapid Intensification Analog Ensemble (RI-AnEn) is evaluated for real-time forecasts made during the National Oceanic and Atmospheric Administration (NOAA)’s 2018 Hurricane Forecast Improvement Program (HFIP) demonstration. Using a variety of assessment tools (Brier skill score, reliability diagrams, ROC curves, ROC skill scores), RI-AnEn is shown to perform competitively compared to both the deterministic HWRF and current operational probabilistic RI forecast aids. The assessment is extended to include forecasts from the 2017 HFIP demonstration and shows that RI-AnEn is the only model with significant RI forecast skill at all lead times in the Atlantic and eastern Pacific basins. Though RI-AnEn is overconfident in its RI forecasts, it is generally well calibrated for all lead times. Furthermore, significance testing indicates that for the 2017–18 Atlantic and eastern Pacific sample, RI-AnEn is more skillful than HWRF at all lead times and better than most of the other probabilistic guidance at 48 and 72 h. ROC curves reveal that RI-AnEn offers a good combination of sensitivity and specificity, performing comparably to SHIPS-RII at all lead times in both basins. With respect to specific high-impact cases from the 2018 Atlantic season, performance of RI-AnEn ranges from excellent (Hurricane Michael) to poor (Hurricane Florence). The multiyear assessment and results for two high-impact case studies from 2018 indicate that, while promising, RI-AnEn requires further work to refine its performance as well as to accurately situate its effectiveness relative to other RI forecasts aids.

Probabilistic Heat Wave Forecast Based on a Large-Scale Circulation Pattern Using the TIGGE Data

Abstract Forecasting extreme events is important for having more time to prepare and mitigate high-impact events because those are expected to become more frequent, intense, and persistent around the globe in the future under the warming atmosphere. This study evaluates the probabilistic predictability of the heat wave index (HWI) associated with large-scale circulation patterns for predicting heat waves over South Korea. The HWI, reflecting heat waves over South Korea, was defined as the vorticity difference at 200 hPa between the South China Sea and northeast Asia. The forecast of up to 15 days from five ensemble prediction systems and the multimodel ensemble has been used to predict the probabilistic HWI during the summers of 2011–15. The ensemble prediction systems consist of different five operational centers, and the forecast skill of the probability of heat waves occurrence was assessed using the Brier skill score (BSS), relative operating characteristics (ROC), and reliability diagram. It was found that the multimodel ensemble is capable of better predicting the large-scale circulation patterns leading to heat waves over South Korea than any other single ensemble system through all forecast lead times. We concluded that the probabilistic forecast of the HWI has promise as a tool to take appropriate and timely actions to minimize the loss of lives and properties from imminent heat waves.

Calibration and assessment of process-based numerical models for beach profile evolution in southern California

Cross-shore profile response to energetic wave events is challenging to predict because the physics are poorly understood and wave and topographic data are often sparse or unknown. Six events with varying duration, wave intensity and beach slopes are used to calibrate and assess two process-based cross-shore models, CShore and XBeach, at two southern California beaches. Model performance is quantitatively evaluated using high resolution spatiotemporal survey observations along with Brier Skill Scores (BSS) and a Beach Elevation Change Error (BECE) metric that evaluates alongshore-averaged upper beach volume predictions. XBeach is tested with default and site-calibrated parameters. Calibration improved skill for all profiles and events, however XBeach skill scores often remained low and in no case correctly predicted the offshore bar formation. Notably, XBeach is sensitive to the beach profile’s calibration depth extent. Upper beach calibration produced significantly different skill than when the full profile was considered. CShore was tested using both the ’Atlantic’ and ’Pacific’ parameters. Both models predict profile change with limited skill. In their present forms, CShore and XBeach are unable to accurately predict beach profile change on these typical southern California beaches, but when calibrated may provide qualitatively useful beach face erosion estimates.

Early Identification of Trauma-induced Coagulopathy: Development and Validation of a Multivariable Risk Prediction Model.

The aim of this study was to develop and validate a risk prediction tool for trauma-induced coagulopathy (TIC), to support early therapeutic decision-making. TIC exacerbates hemorrhage and is associated with higher morbidity and mortality. Early and aggressive treatment of TIC improves outcome. However, injured patients that develop TIC can be difficult to identify, which may compromise effective treatment. A Bayesian Network (BN) prediction model was developed using domain knowledge of the causal mechanisms of TIC, and trained using data from 600 patients recruited into the Activation of Coagulation and Inflammation in Trauma (ACIT) study. Performance (discrimination, calibration, and accuracy) was tested using 10-fold cross-validation and externally validated on data from new patients recruited at 3 trauma centers. Rates of TIC in the derivation and validation cohorts were 11.8% and 11.0%, respectively. Patients who developed TIC were significantly more likely to die (54.0% vs 5.5%, P < 0.0001), require a massive blood transfusion (43.5% vs 1.1%, P < 0.0001), or require damage control surgery (55.8% vs 3.4%, P < 0.0001), than those with normal coagulation. In the development dataset, the 14-predictor BN accurately predicted this high-risk patient group: area under the receiver operating characteristic curve (AUROC) 0.93, calibration slope (CS) 0.96, brier score (BS) 0.06, and brier skill score (BSS) 0.40. The model maintained excellent performance in the validation population: AUROC 0.95, CS 1.22, BS 0.05, and BSS 0.46. A BN (http://www.traumamodels.com) can accurately predict the risk of TIC in an individual patient from standard admission clinical variables. This information may support early, accurate, and efficient activation of hemostatic resuscitation protocols.

Verification of high resolution (12 km) Global Ensemble Prediction System

Abstract Ensemble Prediction Systems (EPSs) are used to estimate the uncertainty in a weather forecast as well as the most likely outcome. To improve weather forecasts and services, the global EPS at the NCMRWF has been upgraded to 12 km horizontal resolution (with 22 perturbed +1 control members). Although the upgraded version of EPS had become operational from 1st June 2018 in the newly acquired high-performance computing system (HPCS), the previous EPS at 33 km horizontal resolution (with 44 perturbed +1 control members) was also running in the old HPCS till 16th July 2018. As compared to the old EPS in NCMRWF, the new EPS includes perturbations in initial state of sea surface temperature, soil moisture content and deep soil temperature. The analysis of the new EPS also includes some more sources of observations. The perturbed 22 members long forecast of the new EPS provided at 00 UTC is the combination of 11 members from 00 UTC cycle and lagged 11 members from 12 UTC cycle of previous day. In the present study, the performances of the new and old EPS are compared for the period from 1st June to 16th July 2018. Performances of both the systems are assessed considering different attributes of probabilistic forecasting. The verification of precipitation, geopotential height at 500 hPa and zonal wind at 850 hPa has been carried out over the northern hemisphere, southern hemisphere and a smaller tropical region which includes India. The metrics used for verification are ensemble-spread, ensemble root-mean-square error, brier score, brier skill score, relative operating characteristics, reliability, sharpness diagrams, rank histogram, ranked probability score, continuous ranked probability score and continuous ranked probability skill score. Two heavy rainfall events have also been considered for subjective comparison of probabilistic prediction capabilities of both the systems. The results show that the new EPS is more skilful in probabilistic forecasts especially over the Indian region and it significantly outperforms the old EPS in probabilistic forecasts of the selected extreme weather events.

Bias Correction of Short-Range Ensemble Forecasts of Daily Maximum Temperature Using Decaying Average

In this study, we assessed the performance of the decaying average bias correction method in removing the systematic error in daily maximum temperature (dTmax) ensemble forecasts. We applied the technique to a short-range high-resolution limited-area ensemble prediction system of the Korea Meteorological Administration, which shows under-predictive and under-dispersive characteristics for dTmax. The bias correction was applied to the grid of the model using spatial interpolation of the decaying average bias from surrounding reference points. The method was verified by evaluating the accuracy of the ensemble mean, spread-skill relationship, and the performance of the probabilistic forecasts. The results showed that the decaying average technique minimized the systematic error in the ensemble mean and improved the performance of the probabilistic forecasts. The overall mean absolute error of the ensemble mean was lowered from 2.2 to 1.2 ∘C and the root-mean-square error from 2.5 to 1.6 ∘C. The continuous ranked probability score decreased from 1.9 to 1.0 ∘C. The reliability of three dichotomous events also improved and the Brier skill scores increased. However, the bias correction only slightly affected the ensemble spread, and the system remained under-dispersive.

Verification Results from the 2017 HMT–WPC Flash Flood and Intense Rainfall Experiment

AbstractThe Flash Flood and Intense Rainfall (FFaIR) Experiment developed within the Hydrometeorology Testbed (HMT) of the Weather Prediction Center (WPC) is a pseudo-operational platform for participants from across the weather enterprise to test emerging flash flood forecasting tools and issue experimental forecast products. This study presents the objective verification portion of the 2017 edition of the experiment, which examines the performance from a variety of guidance tools (deterministic models, ensembles, and machine-learning techniques) and the participants’ forecasts, with occasional reference to the participants’ subjective ratings. The skill of the model guidance used in the FFaIR Experiment is evaluated using performance diagrams verified against the Stage IV analysis. The operational and FFaIR Experiment versions of the excessive rainfall outlook (ERO) are evaluated by assessing the frequency of issuances, probabilistic calibration, Brier skill score (BSS), and area under relative operating characteristic (AuROC). An ERO first-guess field called the Colorado State University Machine-Learning Probabilities method (CSU-MLP) is also evaluated in the FFaIR Experiment. Among convection-allowing models, the Met Office Unified Model generally performed optimally throughout the FFaIR Experiment when using performance diagrams (at the 0.5- and 1-in. thresholds; 1 in. = 25.4 mm), whereas the High-Resolution Rapid Refresh (HRRR), version 3, performed best subjectively. In terms of subjective and objective ensemble scores, the HRRR ensemble scored optimally. The CSU-MLP overpredicted lower risk categories and underpredicted higher risk categories, but it shows future promise as an ERO first-guess field. The EROs issued by the FFaIR Experiment forecasters had improved BSS and AuROC relative to the operational ERO, suggesting that the experimental guidance may have aided forecasters.

Assessment of the Sea Surface Temperature Predictability Based on Multimodel Hindcasts

Abstract Based on historical forecasts of four individual forecasting systems, we conducted multimodel ensembles (MME) to predict the sea surface temperature anomaly (SSTA) variability and assessed these methods from a deterministic and probabilistic point of view. To investigate the advantages and drawbacks of different deterministic MME methods, we used simple averaged MME with equal weighs (SCM) and the stepwise pattern projection method (SPPM). We measured the probabilistic forecast accuracy by Brier skill score (BSS) combined with its two components: reliability (Brel) and resolution (Bres). The results indicated that SCM showed a high predictability in the tropical Pacific Ocean, with a correlation exceeding 0.8 with a 6-month lead time. In general, the SCM outperformed the SPPM in the tropics, while the SPPM tend to show some positive effect on the correction when at long lead times. Corrections occurred for the spring predictability barrier of ENSO, in particular for improvements when the correlation was low or the RMSE was large using the SCM method. These qualitative results are not susceptible to the selection of the hindcast periods, it is as a rule rather by chance of these individual systems. Performance of our probabilistic MME was better than the Climate Forecast System version2 (CFSv2) forecasts in forecasting COLD, NEUTRAL, and WARM SSTA categories for most regions, mainly due to the contribution of Brel, indicating more adequate ensemble construction strategies of the MME system superior to the CFSv2.

Postprocessing Next-Day Ensemble Probabilistic Precipitation Forecasts Using Random Forests

Abstract Most ensembles suffer from underdispersion and systematic biases. One way to correct for these shortcomings is via machine learning (ML), which is advantageous due to its ability to identify and correct nonlinear biases. This study uses a single random forest (RF) to calibrate next-day (i.e., 12–36-h lead time) probabilistic precipitation forecasts over the contiguous United States (CONUS) from the Short-Range Ensemble Forecast System (SREF) with 16-km grid spacing and the High-Resolution Ensemble Forecast version 2 (HREFv2) with 3-km grid spacing. Random forest forecast probabilities (RFFPs) from each ensemble are compared against raw ensemble probabilities over 496 days from April 2017 to November 2018 using 16-fold cross validation. RFFPs are also compared against spatially smoothed ensemble probabilities since the raw SREF and HREFv2 probabilities are overconfident and undersample the true forecast probability density function. Probabilistic precipitation forecasts are evaluated at four precipitation thresholds ranging from 0.1 to 3 in. In general, RFFPs are found to have better forecast reliability and resolution, fewer spatial biases, and significantly greater Brier skill scores and areas under the relative operating characteristic curve compared to corresponding raw and spatially smoothed ensemble probabilities. The RFFPs perform best at the lower thresholds, which have a greater observed climatological frequency. Additionally, the RF-based postprocessing technique benefits the SREF more than the HREFv2, likely because the raw SREF forecasts contain more systematic biases than those from the raw HREFv2. It is concluded that the RFFPs provide a convenient, skillful summary of calibrated ensemble output and are computationally feasible to implement in real time. Advantages and disadvantages of ML-based postprocessing techniques are discussed.

Experimental Subseasonal‐to‐Seasonal (S2S) Forecasting of Atmospheric Rivers Over the Western United States

AbstractA multimodel evaluation of subseasonal‐to‐seasonal (S2S) hindcast skill of atmospheric rivers (ARs) out to 4‐week lead over the western United States is presented for three operational hindcast systems: European Centre for Medium‐Range Weather Forecasts (ECMWF; Europe), National Centers for Environmental Prediction (NCEP; U.S.), and Environment and Canada Climate Change (ECCC; Canada). Ensemble mean biases and Brier Skill Scores are examined for no, moderate, and high levels of AR activity (0, 1–2, and 3–7 AR days/week, respectively). All hindcast systems are more skillful in predicting no and high AR activity relative to moderate activity. There are isolated regions of skill at week‐3 over 150–125°W, 25–35°N for the no and high AR activity levels, with larger magnitude and spatial extent of the skill in ECMWF and ECCC compared to NCEP. The spatial pattern of this skill suggests that for high AR activity, a southwest‐to‐northeast orientation is more predictable at subseasonal lead times than other orientations, and for no AR activity, more skill exists in the subtropical North Pacific, upstream of central and southern California. AR hindcast skill along the western U.S. is most strongly increased in hindcasts initialized during Madden‐Julian Oscillation (MJO) Phases 1 and 8, and hindcast skill is substantially decreased over California in hindcasts initialized during MJO Phase 4. Skill modulations in the ECMWF hindcast system conditioned on El Niño‐Southern Oscillation phase are weaker than those conditioned on particular MJO phases. This work provides hindcast skill benchmarks and uncertainty quantification for experimental real‐time forecasts of AR activity during winters 2019–2021 as part of the S2S Prediction Project Real‐time Pilot Initiative in collaboration with the California Department of Water Resources.

Ensemble Forecasting of Major Solar Flares with Short-, Mid-, and Long-term Active Region Properties

Abstract We apply an ensemble technique for major flare prediction by considering short-, mid-, and long-term active region (AR) properties and their relative contributions. For this, we consider magnetic parameters from Solar Dynamics Observatory/Helioseismic and Magnetic Imager and flare lists from Geostationary Operational Environmental Satellites. In this study, we simultaneously consider flaring rates during short- (1 day), mid- (several days), and long-term (several years) timeframes. In our model, the predicted rate is given by a weighted combination of the three rates such that the sum of their weights is 1. We calculate the Brier skill scores (BSSs) for investigating prediction performance and weights of these three terms to provide optimal results. The BSS (0.22) of the model with only long-term properties is higher than that with only short-term (0.07) or mid-term (0.08) properties. When short-/mid-term properties are additionally considered, the BSS is improved to 0.28/0.24. Our model has the best performance (BSS = 0.29) when all terms are considered, and their relative contributions to the short-, mid-, and long-term rates are 20%, 20%, and 60%, respectively. In addition, the model with three terms is more effective at predicting major flares in strong ARs. In view of the energy storage and release process, long-term magnetic properties may indicate the storage of magnetic free energy, while short- and mid-term flare history may reflect a recent trend of energy release process. Our results suggest that the performances of other existing flare models based on long-term properties should be improved by considering short- and/or mid-term flare history.

Comparison of statistical post-processing methods for probabilistic NWP forecasts of solar radiation

The increased usage of solar energy places additional importance on forecasts of solar radiation. Solar panel power production is primarily driven by the amount of solar radiation and it is therefore important to have accurate forecasts of solar radiation. Accurate forecasts that also give information on the forecast uncertainties can help users of solar energy to make better solar radiation based decisions related to the stability of the electrical grid. To achieve this, we apply statistical post-processing techniques that determine relationships between observations of global radiation (made within the KNMI network of automatic weather stations in the Netherlands) and forecasts of various meteorological variables from the numerical weather prediction (NWP) model HARMONIE-AROME (HA) and the atmospheric composition model CAMS. Those relationships are used to produce probabilistic forecasts of global radiation. We compare 7 different statistical post-processing methods, consisting of two parametric and five non-parametric methods. We find that all methods are able to generate probabilistic forecasts that improve the raw global radiation forecast from HA according to the root mean squared error (on the median) and the potential economic value. Additionally, we show how important the predictors are in the different regression methods. We also compare the regression methods using various probabilistic scoring metrics, namely the continuous ranked probability skill score, the Brier skill score and reliability diagrams. We find that quantile regression and generalized random forests generally perform best. In (near) clear sky conditions the non-parametric methods have more skill than the parametric ones.

Comparative Study on the Numerical Simulation of Bathymetric Changes under Storm Condition

Cho, M.; Yoon, H.-D.; Do, K.; Son, S., and Kim, I.–H., 2019. Comparative study on the numerical simulation o bathymetric changes under storm condition. In: Lee, J.L.; Yoon, J.-S.; Cho, W.C.; Muin, M., and Lee, J. (eds.), The 3rd International Water Safety Symposium. Journal of Coastal Research, Special Issue No. 91, pp. 106-110. Coconut Creek (Florida), ISSN 0749-0208.Topographic changes in the nearshore mainly occur in strong storm condition. To understand bathymetric changes, attempts were made using several numerical models. Nowadays, Delft3D and XBeach are one of the most popular numerical models for predictions of morphodynamics in the nearshore. In this study, a comparative study of Delft3D and XBeach was performed to understand bathymetric changes under storm condition. The model parameters were calibrated with field observations. Storm surge and storm-induced wave field were re-created to simulate morphodynamics modelling. Bathymetric evolution, as well as shoreline changes were in a good agreement with field observations. BSS (Brier Skill Score) evaluation was applied to compare the accuracy of predictions of both models.

Two-Phase Flow Simulation of Tunnel and Lee-Wake Erosion of Scour below a Submarine Pipeline

This paper presents a numerical investigation of the scour phenomenon around a submarine pipeline. The numerical simulations are performed using SedFoam, a two-phase flow model for sediment transport implemented in the open source Computational Fluid Dynamics (CFD) toolbox OpenFOAM. The paper focuses on the sensitivity of the granular stress model and the turbulence model with respect to the predictive capability of the two-phase flow model. The quality of the simulation results is estimated using a statistical estimator: the Brier Skill Score. The numerical results show no sensitivity to the granular stress model. However, the results strongly depend on the choice of the turbulence model, especially through the different implementations of the cross-diffusion term in the dissipation equation between the k − ε and the k − ω 2006 models. The influence of the cross-diffusion term tends to indicate that the sediment transport layer behaves more as a shear layer than as a boundary layer, for which the k − ε model is more suitable.