Mean Square Error Skill Score Research Articles

Measuring and modelling nearshore recovery of an eroded beach in Lake Michigan, USA

Erosion by storms and high-water levels impacts large enclosed basins; however there have been few attempts to numerically model cumulative impacts in large lakes. Antecedent morphology is a large determinant of coastal sensitivity to storms, so capturing the beach recovery is important for overall vulnerability assessment. To study beach recovery, we apply the numerical model XBeach to simulate a period of low to moderate wave energy when beach recovery typically occurs. Surveys were conducted one month apart during summer of 2020 on the west coast of Lake Michigan and used to initiate model runs and evaluate model performance. XBeach was used to propagate offshore wave conditions from a Great Lakes Coastal Forecasting System (GLCFS) node ∼1 km offshore into the nearshore, and results were compared to measurements from a nearshore pressure sensor. We tested for the optimal value of the asymmetry/skewness parameter (facua) for model-data convergence. We evaluated model skill using a Mean Square Error Skill Score (MSESS) and a decomposition. In our repeat surveys we observed slight landward migration of longshore bars and the initiation of bar welding to the shoreline but, overall, changes in bathymetry were small. We found that XBeach transforms offshore waves well and sediment transport volume was accurately predicted by the model. However, XBeach did not capture the morphologic evolution under low energy conditions, preventing simulation of beach recovery. Overall, higher values of facua resulted in improved skill scores and modeled nearshore morphology that was more similar to the morphology measured in our surveys.

Evaluating storm surge predictability on subseasonal timescales for flood forecasting applications: A case study for Hurricane Isabel and Katrina

Coastal flooding operational forecasting in the US is limited to short-range temporal scales (3–7 days), which limits the response time for emergency preparation and planning. The sub-seasonal prediction project (SubX), which produces weather forecasts with a lead time of up to four weeks, provides an opportunity to assess the potential for creating probabilistic flood forecasts with longer lead times. Using the ADCIRC hydrodynamic model for coastal storm surge, two major hurricanes, Isabel (2003) and Katrina (2005), were used as case studies to test coastal flood predictions induced by wind and pressure fields generated from five global weather models within SubX. The storm surges simulations are forced by Sea Level Pressure (SLP) and 10 m winds fields from SubX models for a lead-time of up to 30 days before storm landfall. The subseasonal surge forecasts are evaluated temporally and spatially at 1–4 weeks lead-time against the NOAA tide gages observations and a verification dataset derived by forcing the storm surge model with wind and pressure fields from the NCEP-Reanalysis. The results are evaluated in terms of lead-time and forecast skill metrics. The storm surge forecast skill is measured using the mean square error skill score (MSESS) relative to the verification dataset and an approximate of the climatology. A skill score greater than 0.55 is considered here useful for flood forecasting. The multi-model ensemble (MME) mean surge forecasts driven by several members of SubX models demonstrate skill greater than 0.55 up to a 4-day and 10-day lead for Katrina and Isabel, respectively. A sharper decrease in MSESS was noted from week 1 to week 3 lead-times for Katrina, in comparison to Isabel. Some ensemble members forecasted hurricanes and storm surges as early as 3–4 weeks lead-time. However, due to the offsets developed in the timing and magnitude of the peak at these lead-times, and based on a sample size of only two events, it is hard to establish the significance of these longer lead-time results. While a follow up study involving flood reforecasts over the entire SubX reforecast period (1999–2015) is needed to support more robust statistics of the forecast skill, our case studies demonstrate the feasibility of probabilistic flood forecasting at subseasonal timescales using the SubX models.

Attribution of Large‐Scale Climate Patterns to Seasonal Peak‐Flow and Prospects for Prediction Globally

AbstractFlood‐related fatalities and impacts on society surpass those from all other natural disasters globally. While the inclusion of large‐scale climate drivers in streamflow (or high‐flow) prediction has been widely studied, an explicit link to global‐scale long‐lead prediction is lacking, which can lead to an improved understanding of potential flood propensity. Here we attribute seasonal peak‐flow to large‐scale climate patterns, including the El Niño Southern Oscillation (ENSO), Pacific Decadal Oscillation (PDO), North Atlantic Oscillation (NAO), and Atlantic Multidecadal Oscillation (AMO), using streamflow station observations and simulations from PCR‐GLOBWB, a global‐scale hydrologic model. Statistically significantly correlated climate patterns and streamflow autocorrelation are subsequently applied as predictors to build a global‐scale season‐ahead prediction model, with prediction performance evaluated by the mean squared error skill score (MSESS) and the categorical Gerrity skill score (GSS). Globally, fair‐to‐good prediction skill (20% ≤ MSESS and 0.2 ≤ GSS) is evident for a number of locations (28% of stations and 29% of land area), most notably in data‐poor regions (e.g., West and Central Africa). The persistence of such relevant climate patterns can improve understanding of the propensity for floods at the seasonal scale. The prediction approach developed here lays the groundwork for further improving local‐scale seasonal peak‐flow prediction by identifying relevant global‐scale climate patterns. This is especially attractive for regions with limited observations and or little capacity to develop flood early warning systems.

Short-term forecasting of daily crop evapotranspiration using the ‘Kc-ETo’ approach and public weather forecasts

ABSTRACTShort-term forecasting of daily crop evapotranspiration (ETc) is essential for real-time irrigation management. This study proposed a methodology to forecast short-term daily ETc using the ‘Kc-ETo’ approach and public weather forecasts. Daily reference evapotranspiration (ETo) forecasts were obtained using a locally calibrated version of the Hargreaves-Samani (HS) model and temperature forecasts, while the crop coefficient (Kc) was estimated from observed daily ETo and ETc. The methodology was evaluated by comparing the daily ETc forecasts with measured ETc values from a field irrigation experiment during 2012–2014 in Yongkang Irrigation Experimental Station, China. The overall average of the statistical indices was in the range of 0.96–1.27 mm d−1 for the mean absolute error (MAE), 1.53–2.55 mm d−1 for the mean square error (MSE), 1.77–2.30 mm d−1 for the normalized mean square error (NMSE), 27.5–29.4% for the mean relative error (MRE), 0.71–0.44 for the correlation coefficient (R) and 0.46–0.05 for the mean square error skill score (MSESS). Sources of error werewere Kc estion, temperature forecasts and HS model that does not consider wind speed and humidity, and.the largesourceof error is Kc determination, which suggested that care should be taken when forecasting ETc with estimated Kc values in the study area.

Development of statistical prediction models for Changma precipitation: An ensemble approach

An ensemble statistical forecast scheme with a one-month lead is developed to predict year-to-year variations of Changma rainfall over the Korean peninsula. Spring sea surface temperature (SST) anomalies over the North Atlantic, the North Pacific and the tropical Pacific Ocean have been proposed as useful predictors in a previous study. Through a forward-stepwise regression method, four additional springtime predictors are selected: the northern Indian Ocean (NIO) SST, the North Atlantic SST change (NAC), the snow cover anomaly over the Eurasian continent (EUSC), and the western North Pacific outgoing longwave radiation anomaly (WNP (OLR)). Using these, three new prediction models are developed. A simple arithmetic ensemble mean produces much improved forecast skills compared to the original prediction model of Lee and Seo (2013). Skill scores measured by temporal correlation and MSSS (mean square error skill score) are improved by about 9% and 17%, respectively. The GMSS (Gerrity skill score) and hit rate based on a tercile prediction validation scheme are also enhanced by about 19% and 13%, respectively. The reversed NIO, reversed WNP (OLR), and reversed NAC are all related to the enhancement of a cyclonic circulation anomaly to the south or southwest of the Korean peninsula, which induces southeasterly moisture flux into the peninsula and increasing Changma precipitation. The EUSC predictor induces an enhancement of the Okhotsk Sea high downstream and thus strengthening of Changma front.

Deterministic skill of ENSO predictions from the North American Multimodel Ensemble

Hindcasts and real-time predictions of the east-central tropical Pacific sea surface temperature (SST) from the North American Multimodel Ensemble (NMME) system are verified for 1982–2015. Skill is examined using two deterministic verification measures: mean squared error skill score (MSESS) and anomaly correlation. Verification of eight individual models shows somewhat differing skills among them, with some models consistently producing more successful predictions than others. The skill levels of MME predictions are approximately the same as the two best performing individual models, and sometimes exceed both of them. A decomposition of the MSESS indicates the presence of calibration errors in some of the models. In particular, the amplitudes of some model predictions are too high when predictability is limited by the northern spring ENSO predictability barrier and/or when the interannual variability of the SST is near its seasonal minimum. The skill of the NMME system is compared to that of the MME from the IRI/CPC ENSO prediction plume, both for a comparable hindcast period and also for a set of real-time predictions spanning 2002–2011. Comparisons are made both between the MME predictions of each model group, and between the average of the skills of the respective individual models in each group. Acknowledging a hindcast versus real-time inconcsistency in the 2002–2012 skill comparison, the skill of the NMME is slightly higher than that of the prediction plume models in all cases. This result reflects well on the NMME system, with its large total ensemble size and opportunity for possible complementary contributions to skill.

Deployment of artificial neural network for short-term forecasting of evapotranspiration using public weather forecast restricted messages

Near-future irrigation demand forecasting is important information for anticipating decisions on crop irrigation scheduling and planning water allocation in large irrigation command areas of Texas. The key determinant that is required for estimating irrigation demand in advance is toward the evapotranspiration forecast. Normally, in rich data environment, current reference evapotranspiration (ETo) is estimated by the well-known FAO56 PM method which requires bunch of observed climatic data. In poor data environment for either current or future estimation, this well-known method application is restricted. Indeed, the correctness of ETo forecast remains a challenging computational task, since inaccurate weather variables can alter the forecast accuracy. Therefore, this study aims to employ artificial neural network (ANN) methodology for forecasting near future ETo values by using restricted climate information messages retrieved from public weather forecast source. Four ANNs learning algorithms including the Generalized Feedforward (GFF), Linear Regression (LR), Multilayer Perceptron (MLP) and Probabilistic Neural Network (PNN) are applied with three sets of inputs combination composed of minimum (Tmin) and maximum (Tmax) daily air temperatures, extraterrestrial radiation (Ra) and net solar radiation (Rs) to forecast ETo in Dallas. The coefficient of correlation (CC), mean square error (MSE), normalized mean square error (NMSE), mean absolute error (MAE) and mean square error skill score (MSESS) were used for the models evaluation. Statistically, in comparison with FAO56 PM, the performances of ANNs models using only Tmax and Tmin predictors were inferior to those of Tmax, Tmin and Ra. With Tmax, Tmin and Rs input-sets, MLP yielded the highest accuracies (CC=0.926; MSE=0.770mm/day, NMSE=0.143mm/day; MAE=0.708mm/day). Tmax is an important ETo forecast predictor, and the performance improvement relies mostly on Rs accuracy. With precise weather forecast information, ANN made ETo forecast possible (Average CC=0.860, MSESS=0.738). These results can assist irrigation districts to accommodate in advance their crop water demand to near-future irrigation requirement.

On the perception of morphodynamic model skill

The quality of morphodynamic predictions is generally expressed by an overall grid-point based skill score, which measures the relative accuracy of a morphological prediction over a prediction of zero morphological change, using the mean-squared error (MSE) as the accuracy measure. Through a generic ranking for morphodynamic model predictions, this MSE based skill score (MSESS) aims at making model performance comparable across different prediction situations (geographical locations, forcing conditions, time periods, internal dynamics). The implicit assumptions underlying this approach are that the MSE is an appropriate measure of correspondence for morphological predictions and that the accuracy of the initial bed as the reference correctly reflects the inherent difficulty or ease of prediction situations. This paper presents a thorough analysis of the perception of model skill through the MSE skill score. Using synthetic examples, an example from literature and a long-yearly Delft3D model simulation, we demonstrate that unexpected skill may be reported due to a violation of either of the above assumptions. It is shown that the accuracy of the reference fails to reflect the relative difficulty of prediction situations with a different morphological development prior to the evaluation time (for instance trend, cyclic/seasonal, episodic, speed of the development). We further demonstrate that the MSESS tends to favor model results that underestimate the variance of cumulative bed changes, a feature inherited from the MSE. As a consequence of these limitations, the MSESS may report a relative ranking of predictions not matching the intuitive judgment of experts. Guidelines are suggested for how to adjust calibration and validation procedures to be more in line with a morphologist's expert judgment.

Comparison of two analog-based downscaling methods for regional reference evapotranspiration forecasts

The objective of this study was to compare the performance of natural analog (NA) and constructed analog (CA) methods to produce both probabilistic and deterministic downscaled daily reference evapotranspiration (ETo) forecasts in the southeastern United States. The 1–15 day, 15-member ETo forecasts were produced from 1979 to 2009 using the Penman–Monteith equation and a forecast analog approach with a combination of the Global Forecast System (GFS) reforecasts and NCEP-DOE Reanalysis 2 climatology, and were downscaled using the North American Regional Reanalysis (NARR). The Pearson correlation coefficient (R), mean squared error skill score (MSESS), and Bias were used to evaluate the skill of downscaled deterministic forecasts. The Linear Error in Probability Space (LEPS) skill score, Brier Skill Score (BSS), relative operating characteristic, and reliability diagrams were used to evaluate the skill of downscaled probabilistic forecasts. Overall, CA showed slightly higher skill than NA in terms of the metrics for deterministic forecasts, while for probabilistic forecasts NA showed higher skill than CA regarding the BSS in five categories (terciles, and 10th and 90th percentiles) and lower skill than CA regarding the LEPS skill score. Both CA and NA produced skillful deterministic results in the first 3 lead days, while the skill was higher for CA than for NA. Probabilistic NA forecasts exhibited higher resolution and reliability than CA, likely due to a larger ensemble size. Forecasts by both methods showed the lowest skill in the Florida peninsula and in mountainous areas, likely due to the fact that these features were not well-resolved in the model forecast.