Forecast Verification Research Articles

On the verification of ensemble precipitation forecasts over the Godavari River basin

The ensemble forecasts from Numerical Weather Prediction (NWP) models are considered as essential elements for operational hydrologic forecasting. However, these forecasts are burdened with significant inherent biases and cannot be used directly for hydrologic forecasting. Therefore, the present study examines the capability of ensemble precipitation forecasts obtained from two NWP models i.e., a 20-member ensemble from the National Center for Environmental Prediction (NCEP) and a 50-member ensemble from the European center for Medium-term Weather Forecasts (ECMWF). The skill of 70-member multi-model grand ensemble (MME) generated from the NCEP and ECMWF forecasts is also verified in this study. Furthermore, post-processing of the raw ensemble members was carried out using two methods (Quantile Mapping (QM) and Quantile Regression Forests (QRF)) in order to minimize the inherent biases. The analysis was performed over subbasins of an Indian river basin namely the Godavari River Basin. Here, a set of deterministic and probabilistic verification measures (forecast error box plots, correlation coefficient, Relative Mean Error, mean Continuous Ranked Probability Score, spread-skill relationship, rank histograms, reliability diagram, Area under Relative Operating Characteristic Curve) were employed to assess various quality attributes of the forecasts. The findings from the study suggests that QRF post-processed NCEP and MME forecasts are comparatively better than other forecasts in terms of all employed verification measures over all subbasins. The QRF post-processed forecasts were found to be superior than QM post-processed and raw forecasts over all subbasins. The adopted post-processing methods resulted in improved forecasts at shorter lead times. However, the skill of both raw and post-processed forecasts declines at higher lead times over all subbasins.

Evaluation and bias correction of probabilistic volcanic ash forecasts

Abstract. Satellite retrievals of column mass loading of volcanic ash are incorporated into the HYSPLIT transport and dispersion modeling system for source determination, bias correction, and forecast verification of probabilistic ash forecasts of a short eruption of Bezymianny in Kamchatka. The probabilistic forecasts are generated with a dispersion model ensemble created by driving HYSPLIT with 31 members of the NOAA global ensemble forecast system (GEFS). An inversion algorithm is used for source determination. A bias correction procedure called cumulative distribution function (CDF) matching is used to very effectively reduce bias. Evaluation is performed with rank histograms, reliability diagrams, fractions skill score, and precision recall curves. Particular attention is paid to forecasting the end of life of the ash cloud when only small areas are still detectable in satellite imagery. We find indications that the simulated dispersion of the ash cloud does not represent the observed dispersion well, resulting in difficulty simulating the observed evolution of the ash cloud area. This can be ameliorated with the bias correction procedure. Individual model runs struggle to capture the exact placement and shape of the small areas of ash left near the end of the clouds lifetime. The ensemble tends to be overconfident but does capture the range of possibilities of ash cloud placement. Probabilistic forecasts such as ensemble-relative frequency of exceedance and agreement in percentile levels are suited to strategies in which areas with certain concentrations or column mass loadings of ash need to be avoided with a chosen amount of confidence.

Impacts and Climate Change Adaptation of Agrometeorological Services among the Maize Farmers of West Tamil Nadu

Climate change is often linked with record-breaking heavy or poor rainfall events, unprecedented storms, extreme day and night time temperatures, etc. It may have a marked impact on climate-sensitive sectors and associated livelihoods. Block-level weather forecasting is a new-fangled dimension of agrometeorological services (AAS) in the country and is getting popularized as a climate-smart farming strategy. Studies on the economic impact of these microlevel advisories are uncommon. Agromet advisory services (AAS) play a critical role as an early warning service and preparedness among the maize farmers in the Parambikulam–Aliyar Basin, as this area still needs to widen and deepen its AWS network to reach the village level. In this article, the responses of the maize farmers of Parambikulam–Aliyar Basin on AAS were analyzed. AAS were provided to early and late Rabi farmers during the year 2020–2022. An automatic weather station was installed at the farmers’ field to understand the real-time weather. Forecast data from the India Meteorological Department (IMD) were used to provide agromet advisory services. Therefore, the present study deserves special focus. Social media and other ICT tools were used for AAS dissemination purposes. A crop simulation model (CSM), DSSAT4.7cereal maize, was used for assessing maize yield in the present scenario and under the elevated GHGs scenario under climate change. Our findings suggest that the AAS significantly supported the farmers in sustaining production. The AAS were helpful for the farmers during the dry spells in the late samba (2021–2022) to provide critical irrigation and during heavy rainfall events at the events of harvest during early and late Rabi (2021–22). Published research articles on the verification of weather forecasts from South India are scanty. This article also tries to understand the reliability of forecasts. Findings from the verification suggest that rainfall represented a fairly good forecast for the season, though erratic, with an accuracy score or HI score of 0.77 and an HK score of 0.60, and the probability of detection (PoD) of hits was found to be 0.91. Verification shows that the forecasted relative humidity observed showed a fairly good correlation, with an R2 value of 0.52. These findings suggest that enhancing model forecast accuracy can enhance the reliability and utility of AAS as a climate-smart adaptation option. This study recommends that AAS can act as a valuable input to alleviate the impacts of hydrometeorological disasters on maize crop production in the basin. There is a huge demand for quality weather forecasts with respect to accuracy, resolution, and lead time, which is increasing across the country. Externally funded research studies such as ours are an added advantage to bridge the gap in AAS dissemination to a great extent.

Developing and evaluating week 2 and weeks 3-4 outlook tools for extratropical storminess

Extratropical cyclones give rise to most of the high impact weather in the mid-to high-latitudes during the cool seasons, including heavy precipitation and strong winds. Thus it is important for stakeholders to be informed of approaching periods of increased or decreased cyclone activity. While individual cyclone tracks can be predicted out to about a week or so, from week 2 on, statistics summarizing cyclone activity, or storminess, are more useful. Storminess can be defined based on Lagrangian cyclone tracking or by Eulerian variance statistics. The outlook includes a combination of both methods. Lagrangian cyclone tracks provide information about where cyclones pass through and are more intuitive to users, while Eulerian variance statistics have been shown to be highly correlated with cyclone-related weather and are expected to be more predictable given that they are not as noisy. In this paper, we evaluate a storminess outlook tool developed based on dynamical model forecasts in the week-2 and weeks 3-4 time ranges. The outlook uses two 6-hourly subseasonal ensemble forecasts–the Global Ensemble Forecast System version 12 (GEFSv12), and the coupled Climate Forecast System version 2 (CFSv2). Hindcasts and operational forecasts from 1999–2016 are used to assess the prediction skill. Our results show that the GEFSv12 and CFSv2 combined ensemble has higher skill than either individual ensemble. The combined ensemble shows some skill in predicting cyclone amplitude and frequency out to weeks 3-4, with highest skill in winter, and lowest skill in summer. Models also show some skill in predicting the statistics of deep cyclones for week 2. The prediction skills for an Eulerian sea level pressure variance storminess metric is significantly higher than those for Lagrangian track statistics. Our results also show that GEFSv12 performs better than its predecessor GEFSv11. Correlations between the storminess indices and surface weather, including precipitation and high winds, are examined. A publicly accessible web page has been developed to display the subseasonal predictions in real time. The web page also contains information on climatology and forecast verification to enable users to make more informed use of the outlook.

Adapting to climate change through anticipatory action: The potential use of weather-based early warnings

As a crucially-needed adaptation to climate change, the United Nations plans to expand Early Warning Systems (EWS) for extreme weather to cover everyone on Earth. Given the growing interest in this climate change adaptation solution, we assess how well weather early warnings perform for extreme events in different parts of the world. First, we carry out a forecast verification for weather forecasts from the National Oceanic and Atmospheric Administration (NOAA) for 95th percentile extreme heat and extreme precipitation globally at 0.5° resolution, with three days of lead time. We present the results alongside similar verification results from ECMWF forecasts and a CHIRPS-GEFS forecast, to identify regions of the world with consistent forecast skill. We then overlay the skill of these short-term weather forecasts on top of climate change projections for the increasing frequency of the extreme events themselves. Based on these results, we offer policy implications for EWS investments in different regions. We find that in much of the tropics, weather forecasts have relatively poor skill in forecasting extreme temperature and precipitation events, calling for further investments in predictability. In the extra-tropics, most extreme heat and extreme precipitation events can be correctly forecasted, with better results for multi-day events and shorter lead-times. While there is room to improve predictability, end-to-end investments in EWS in these regions can focus on the use of existing skillful forecasts. Finally, most of the world's land area is projected to see an increase in the magnitude of extreme heat and precipitation events with climate change, and EWS investments in these regions should prepare for unprecedented extremes and changing vulnerabilities. These results provide a foundation for localized research on EWS in different parts of the world as well as evidence for policy and donors on how best to invest in EWS in different regions.

Contribution of lagged members to the performance of a global ensemble prediction system

National Centre for Medium Range Weather Forecasting (NCMRWF), India runs operationally a global ensemble prediction system (NEPS-G) which has 23 ensemble members. The control and 11 perturbed ensemble members have the initial conditions of 00 UTC. The start time of the other 11 perturbed members is 12 UTC of the previous day. The objective of the present paper is to study the effect of the lagged ensemble members on the forecast quality of NEPS-G. For this, we have carried out an inter-comparison of the relative skills of (i) 11-member ensemble starting from 00 UTC i.e., E00_11, (ii) 11 lagged ensemble members starting from 12 UTC i.e., E12_11 and the (iii) operational 23-member ensemble i.e. E23 using the verification of temperature, wind and geopotential height forecasts for 7-day forecast lead times for April–May-June 2019 (AMJ) of summer and December 2019–January 2020 (DJ) of winter. The temperature at 850 hPa and geopotential height at 500 hPa are verified over the northern hemisphere. The zonal and meridional winds at 850 hPa and 200 hPa are verified over the region 0oN-40oN, 60°E-110°E. The metrics used for validation are ensemble-spread and root-mean-square error (RMSE) relationship, brier skill score (BSS), reliability diagram, outliers statistics, relative operating characteristics (ROC) score and ranked probability score (RPS). For brevity, only the results for Z500, T850 and U850 have been discussed in detail here.The results show that in both seasons, the RMSE-spread relationship is better for E23 than for E00_11 and E12_11. For T850 and Z500 over NH, both E23 and E00_11 have higher RMSE and higher ensemble spread in DJ season as compared to AMJ season. In case of wind over the Indian region, both RMSE and ensemble spread exhibit lower values during DJ as compared to AMJ for both ensemble configurations. For both seasons, the BSS of E23 is positive and the ROC score is >0.5 beyond 7 days forecast times for all the variables for all three ensembles and the percentage of outliers in E23 is also less. Overall, the contribution of 11 lagged ensemble members improves the skill of NEPS-G in both seasons and skill is better during winter (DJ season). The error in the mean forecast track for tropical cyclone ‘Fani’ for E23 is found to be less than for E00_11.We have also compared the forecast skill of E23 with that of the ensemble formed by 22 members, all running from 00 UTC i.e. E00_22 for the same set of variables as well as precipitation for one month forecast data from each season, namely June 2019 from summer and January 2020 from winter. Results show that the RMSE-spread relationship is very much similar for both configurations and both seasons. For E00_22 and E23, the ensemble spread is closer to the RMSE of ensemble mean for all the variables in January compared to June. BSS was slightly higher for E00_22 but was not significant at all lead time for all the variables except Z500 in January which shows significant improvement in BSS of E00_22 from day 2 to day 5. In precipitation forecast verification, RPS for E00_22 is slightly less (though significant) than E23 at all forecast lead times in June while in January there was no significant difference. So, E23 dataset of NEPS-G which demands less computational resources at the same time offers more skilful forecast than E00_11 over a longer forecast lead time and is nearly as skilful as E00_22.

Rainfall Forecast Verification in Different Blocks of Banka District of Bihar

Accuracy in rainfall forecast plays an important role in farming operations. Here, a study has been done to verify the forecast and actual rainfall data of 11 blocks of Banka district at KVK, Banka. To verify the accuracy of rainfall forecast for the financial year 2020-21 several statistical tools has been used i.e., Forecast Accuracy (ACC) or Ratio Score or Hit Score, Hanssen and Kuipers Scores or True Skill Score (HK score), Probability of detection (POD), Heidke Skill Score (HSS), False alarm ratio, Critical Success index (CSI), Missing Rate (MR) and BIAS score. Forecast Accuracy (ACC) ranged from 0.81 to 0.87 and the highest accuracy of rainfall prediction was observed in Katoriya block and lowest accuracy observed in Amarpur, Dhoraiya and Rajoun block. Similarly, Critical Success Index (CSI) was recorded with the values range of 0.4 to 0.6 for respective blocks of Banka district. Heidke Skill Score (HSS) was accounted as 0.50 for Amarpur, Banka, Barahat, Bounsi, Belhar, Dhoraiya, Rajoun and Shambhuganj and for rest of the blocks it was 0.6. The higher accuracy (87.1 per cent) was recorded in Katoriya block followed by Fullidumar (86.3 per cent) and Chandan (85.4 per cent). The highest no. of rainy days was observed in Katoriya block (75 no.) followed by Chandan (74 no.). POD value ranges between 0.8-0.9, FAR value between 0.3-0.5, MR and BIAS value ranges between 0.8-0.9 and 1.3 -2.0 respectively.

Comparison of sequential and variational assimilation methods to improve hydrological predictions in snow dominated mountainous catchments

• Assimilation of streamflow and snow data using variational and sequential techniques in a hydrological model. • Satellite snow coverage products are assimilated together with streamflow data. • Variational techniques outperform sequential techniques in two mountainous basins. • Average relative gains in variational assimilation are 49% for streamflow and 28% for snow data. Snowmelt modeling is vital issue for proper operational flow forecasting systems but still challenging in upstream mountainous basins due to limited observations and associated uncertainties. Data assimilation is a valuable tool to improve model state estimates to improve hydrological forecast. While assimilation of snow has been increasingly implemented in research, particularly using sequential data assimilation methods, less attention has been dedicated to compare different types of assimilation techniques. This study aims at comparing the well-known Ensemble Kalman Filter (EnKF) sequential data assimilation and the less known Moving Horizon Estimation (MHE) variational assimilation. Initial conditions of the hydrological model states are improved by assimilating in-situ streamflow and remotely sensed snow covered area data into the Hydrologiska Byråns Vattenbalansavdelning (HBV) conceptual rainfall-runoff model. The study is applied to improve runoff predictions over two mountainous basins: one in the uppermost catchment of the Karasu river, in Eastern Turkey, and the second in an alpine basin located in the headwaters of the Genil River in the northern flank of the Sierra Nevada Mountain in Southern Spain. The forecast models are tested in a hindcasting condition, i.e. closed loop environment, which simulates real-time forecasting systems. The results are analyzed using forecast verification metrics. Results showed that streamflow and snow state predictions using MHE outperform the commonly used EnKF based on the continuous ranked probability score (CRPS).

A Two‐Stage Framework for Bias and Reliability Tests of Ensemble Hydroclimatic Forecasts

AbstractThe popular probability integral transform (PIT) uniform plot presents informative empirical illustrations of five types of ensemble forecasts, that is, reliable, under‐confident, over‐confident, negatively biased and positively biased. This paper has built a novel two‐stage framework upon the PIT uniform plot to quantitatively examine the forecast attributes of bias and reliability. The first stage utilizes the test statistic on bias to examine whether the mean of PIT values is equal to the theoretical mean of standard uniform distribution. Then, the second stage uses the test statistic on reliability to examine whether the mean squared deviation from the theoretical mean is equal to the theoretical variance of standard uniform distribution. Therefore, by using the two‐tailed bootstrap hypothesis testing, the first stage identifies unbiased ensemble forecasts, negatively biased forecasts and positively biased forecasts; the second stage focuses on unbiased ensemble forecasts to furthermore identify reliable forecasts, under‐confident forecasts and over‐confident forecasts. Numerical experiments are devised for the National Centers for Environmental Prediction's Climate Forecast System version 2 ensemble forecasts of global precipitation. The results highlight the existence of various shapes of the PIT uniform plots. Due to extreme values of observed precipitation, the PIT uniform plots in some cases can substantially deviate from the 1:1 line even though the mean and variance of ensemble forecasts are respectively in accordance with the mean and variance of observations. Nevertheless, the two‐stage framework along with the two test statistics serves as a robust tool for the verification of ensemble hydroclimatic forecasts.

The High-Resolution Rapid Refresh (HRRR): An Hourly Updating Convection-Allowing Forecast Model. Part II: Forecast Performance

Abstract The High-Resolution Rapid Refresh (HRRR) is a convection-allowing implementation of the Advanced Research version of the Weather Research and Forecast (WRF-ARW) Model that covers the conterminous United States and Alaska and runs hourly (for CONUS; every 3 h for Alaska) in real time at the National Centers for Environmental Prediction. The high-resolution forecasts support a variety of user applications including aviation, renewable energy, and prediction of many forms of severe weather. In this second of two articles, forecast performance is documented for a wide variety of forecast variables and across HRRR versions. HRRR performance varies across geographical domain, season, and time of day depending on both prevalence of particular meteorological phenomena and the availability of both conventional and nonconventional observations. Station-based verification of surface weather forecasts (2-m temperature and dewpoint temperature, 10-m winds, visibility, and cloud ceiling) highlights the ability of the HRRR to represent daily planetary boundary layer evolution and the development of convective and stratiform cloud systems, while gridded verification of simulated composite radar reflectivity and quantitative precipitation forecasts reveals HRRR predictive skill for summer and winter precipitation systems. Significant improvements in performance for specific forecast problems are documented for the upgrade versions of the HRRR (HRRRv2, v3, and v4) implemented in 2016, 2018, and 2020, respectively. Development of the HRRR model data assimilation and physics paves the way for future progress with operational convective-scale modeling. Significance Statement NOAA’s operational hourly updating convection-allowing model, the High-Resolution Rapid Refresh (HRRR), is a key tool for short-range weather forecasting and situational awareness. Improvements in assimilation of weather observations, as well as in physics parameterizations, has led to improvements in simulated radar reflectivity and quantitative precipitation forecasts since the initial implementation of HRRR in September 2014. Other targeted development has focused on improved representation of the diurnal cycle of the planetary boundary layer, resulting in improved near-surface temperature and humidity forecasts. Additional physics and data assimilation changes have led to improved treatment of the development and erosion of low-level clouds, including subgrid-scale clouds. The final version of HRRR features storm-scale ensemble data assimilation and explicit prediction of wildfire smoke plumes.

Performance of Regional Climate Model (WRF 4.3) in Medium Range Rainfall Forecast (MRRF) for Tamil Nadu, India

The weather events are highly dynamic and fluctuating for the next few days due to enormous processes carried out by nature and physics and it is even more highly variable in tropics. The Medium Range Weather Forecast is incredibly helpful and trustworthy for agricultural purposes and rainfall is one of the most imminent events determining productivity. The Medium Range Rainfall Forecast (MRRF) given by Weather Research and Forecast model (WRF v 4.3) is verified using forecast verification scores including Ratio of Root Mean Square Error (RMSE) to the standard deviation of the observations (RSR), Nash-Sutcliffe Efficiency (NSE), Percent Bias (PBIAS), Kling-Gupta Efficiency (KGE), and Root Mean Square Error (RMSE). Scores were computed by comparing forecast generated using two microphysics options viz., WRF Single Moment scheme (WSM-3) and Kessler scheme during South West Monsoon (SWM) and North East Monsoon (NEM) of the year 2021 for five different physiographic regions of Tamil Nadu. WSM-3 microphysics scheme outperformed in predicting MRRF for all the five regions and during both the monsoons.

XCast: A python climate forecasting toolkit

Climate forecasts, both experimental and operational, are often made by calibrating Global Climate Model (GCM) outputs with observed climate variables using statistical and machine learning models. Often, machine learning techniques are applied to gridded data independently at each gridpoint. However, the implementation of these gridpoint-wise operations is a significant barrier to entry to climate data science. Unfortunately, there is a significant disconnect between the Python data science ecosystem and the gridded earth data ecosystem. Traditional Python data science tools are not designed to be used with gridded datasets, like those commonly used in climate forecasting. Heavy data preprocessing is needed: gridded data must be aggregated, reshaped, or reduced in dimensionality in order to fit the strict formatting requirements of Python's data science tools. Efficiently implementing this gridpoint-wise workflow is a time-consuming logistical burden which presents a high barrier to entry to earth data science. A set of high-performance, easy-to-use Python climate forecasting tools is needed to bridge the gap between Python's data science ecosystem and its gridded earth data ecosystem. XCast, an Xarray-based climate forecasting Python library developed by the authors, bridges this gap. XCast wraps underlying two-dimensional data science methods, like those of Scikit-Learn, with data structures that allow them to be applied to each gridpoint independently. XCast uses high-performance computing libraries to efficiently parallelize the gridpoint-wise application of data science utilities and make Python's traditional data science toolkits compatible with multidimensional gridded data. XCast also implements a diverse set of climate forecasting tools including traditional statistical methods, state-of-the-art machine learning approaches, preprocessing functionality (regridding, rescaling, smoothing), and postprocessing modules (cross validation, forecast verification, visualization). These tools are useful for producing and analyzing both experimental and operational climate forecasts. In this study, we describe the development of XCast, and present in-depth technical details on how XCast brings highly parallelized gridpoint-wise versions of traditional Python data science tools into Python's gridded earth data ecosystem. We also demonstrate a case study where XCast was used to generate experimental real-time deterministic and probabilistic forecasts for South Asian Summer Monsoon Rainfall in 2022 using different machine learning-based multi-model ensembles.

Significant Improvement in Rainfall Forecast over Delhi: Annual and Seasonal Verification

Regional Weather Forecasting Centre (RWFC) New Delhi has the responsibility to issue and disseminate rainfall forecast for Delhi. So it is very important to scientifically verify the rainfall forecast issued by RWFC. In this study rainfall forecast verification of Delhi has been carried out annually and season wise for the period 2011 to 2021. Various statistical parameters such as Percentage Correct (PC), Probability of Detection (POD), Missing Ratio (MR), False Alarm Ratio (FAR), Critical Success Index (CSI), True Skill Statistics (TSS) and Heidke Skill Score (HSS) have been calculated for season wise and annually. A forecast is considered to be improved if PC, POD, CSI, TSS and HSS increase and FAR and MR decrease over a period of time. The author can conclude that annual accuracy of forecast has increased significantly over the period of time from 2011 to 2021, as PC, POD, CSI, TSS and HSS increase and FAR and MR decrease over a period of time. Maximum contribution in the improved forecast has observed in transition season (pre-monsoon season followed by post-monsoon, having rainfall activity mainly in association with thunderstorms), when FAR and MR have decreased drastically.

Impact of the tilt angle, inverter sizing factor and row spacing on the photovoltaic power forecast accuracy

The growth of the installed photovoltaic (PV) capacity calls for accurate power forecasts, which are commonly calculated from irradiance forecasts using physical model chains. This study analyses the impact of the PV plant design parameters on the additional error of the irradiance-to-power conversion using three approaches. First, PV power curves are plotted for different design parameters to illustrate the amplification of errors and to enhance the understanding of the presented phenomena. Second, six forecast accuracy metrics are calculated as a function of the design parameters for the irradiance observation and forecast data of ten stations from five climate zones and two forecast providers. Third, the Pareto-optimal tradeoff between the annual energy production and the absolute forecast errors are obtained by the NSGA-II multi-objective optimization algorithm. The results reveal that the design parameters highly influence the PV power forecast errors, which is important to consider in all studies presenting PV power forecast verification. A tilt angle of 45° increases the error metrics up to 49% compared to a horizontal plane, and undersizing the inverters by a 1.5 sizing factor lead to an average error decrease up to 25%. Including the expected forecast errors as an objective during their design optimization process is important to maximize the net revenues of PV plants, especially if the unit imbalance costs are more than 50% of the electricity prices.

Correlogram, predictability error growth, and bounds of mean square error of solar irradiance forecasts

Solar forecast verification must incorporate a notion of “relativity.” Particularly for deterministic forecasts, one wishes to know the upper and lower bounds of the mean square error (MSE) for the forecasting situation of concern, such that the performance of some forecasts of interest can be quantified relative to these bounds. This work proposes a method for estimating these bounds. For the upper bound of MSE, the theory builds upon the premise that no forecast worse than that provided by the standard of reference should be used. Under some mild assumptions, the MSE of the standard of reference, which is herein defined to be the optimal convex combination of climatology and persistence, varies solely with the lag-h autocorrelation of the clear-sky index. Then, by fitting a theoretical isotropic correlogram with a nugget effect to the empirical autocorrelations, the upper bound of MSE, as a function of forecast horizon, can be derived. For the lower bound of MSE, it can be approximated by the predictability error growth, which is the difference between the controlled and perturbed forecasts from a perfect dynamical weather model. The empirical part of this work considers the irradiance forecasts from the European Centre for Medium-Range Weather Forecasts, at seven locations in United States, over a period of two years (2019–2020). Various MSEs reported in the literature are put in perspective of the estimated bounds of MSE. The proposed bounds have a profound impact on predictability quantification and skill score computation, which are essential for comparative forecast verification.

Опыт пространственной верификации радиолокационного наукастинга осадков: определение и статистика объектов, ситуаций и условных выборок

Statistical analysis was performed using methods of the extreme value theory for spatial objects and specified situations identified for object-oriented verification of precipitation regions with substantial and maximal areas. We made an estimation of the effect of missing values at field points and of different observation-forecast pairs construction on volumes and on statistical characteristics of samples retrieved for spatial verification purposes. We used spatial quantile functions and geographical representations in regular coordinates to illustrate particular aspects of composite fields built on about three dozen radars' data over the European territory of Russia. Keywords: spatial forecast verification, radar precipitation nowcasting, extreme value theory, missing data, conditional verification sampling, spatial quantiles

Common Issues in Verification of Climate Forecasts and Projections

With increased interest in climate forecasts and projections, it is important to understand more about their sources and levels of skill. A starting point here is to describe the nature of the skill associated with forecasts and projections. Climate forecasts and projections typically both include time varying forcing of the climate, but only forecasts have initial conditions set close to the observed climate state. Climate forecasts therefore derive skill from both initial conditions and from forcing. The character of the initial condition skill and forcing skill is different. Skill from initial conditions results in a narrowing of expectations relative to a climatological distribution and points toward a more favoured part of the distribution. Forcing skill could result from a shift in the preferred parts of the climatological distribution in response to forcing, or it could result from a shift in the entire distribution, or both. Assessments of forcing skill require time averages of the target variable that are long enough so that the contributions from internal variations are small compared to the forced response. The assessment of skill of climate forecasts and projections is inherently partial because of the small number of repeated trials possible on typical climate time scales but is nonetheless the only direct measure of their performance.

Calibration of deterministic NWP forecasts and its impact on verification

Deterministic forecasts (as opposed to ensemble or probabilistic forecasts) issued by numerical weather prediction (NWP) models require post-processing. Such corrective procedure can be viewed as a form of calibration. It is well known that, based on different objective functions, e.g., minimizing the mean square error or the mean absolute error, the calibrated forecasts have different impacts on verification. In this regard, this paper investigates how a calibration directive can affect various aspects of forecast quality outlined in the Murphy–Winkler distribution-oriented verification framework. It is argued that the correlation coefficient is the best measure for the potential performance of NWP forecast verification when linear calibration is involved, because (1) it is not affected by the directive of linear calibration, (2) it can be used to compute the skill score of the linearly calibrated forecasts, and (3) it can avoid the potential deficiency of using squared error to rank forecasts. Since no single error metric can fully represent all aspects of forecast quality, forecasters need to understand the trade-offs between different calibration strategies. To echo the increasing need to bridge atmospheric sciences, renewable energy engineering, and power system engineering, as to move toward the grand goal of carbon neutrality, this paper first provides a brief introduction to solar forecasting, and then revolves its discussion around a solar forecasting case study, such that the readers of this journal can gain further understanding on the subject and thus potentially contribute to it.

Representation Uncertainty in the Earth Sciences

AbstractThe first Joint Workshop on Representation Uncertainty in the Earth Sciences was held in March 2021. This brought together the Earth observation, data assimilation and forecast verification and post‐processing communities, alongside metrologists to discuss the definition and quantification of representation uncertainty within the Earth sciences. The aim of the workshop was to facilitate cross‐disciplinary discussion, establishing where existing methodologies could be shared and to foster future collaboration. A key outcome of the workshop was a working definition of representation uncertainty applicable across all the Earth science communities, which is presented in this white paper. The cross‐disciplinary discussions at the workshop highlighted the need for scientists to work with metrologists to establish a common vocabulary for uncertainties, accessible to Earth science applications. Further recommendations included regular workshops to discuss progress in defining and quantifying representation uncertainty and awareness of cross‐disciplinary funding opportunities to further address representation uncertainty issues.

On the ROC Area of Ensemble Forecasts for Rare Events

Abstract The relative operating characteristic (ROC) curve is a popular diagnostic tool in forecast verification, with the area under the ROC curve (AUC) used as a verification metric measuring the discrimination ability of a forecast. Along with calibration, discrimination is deemed as a fundamental probabilistic forecast attribute. In particular, in ensemble forecast verification, AUC provides a basis for the comparison of potential predictive skill of competing forecasts. While this approach is straightforward when dealing with forecasts of common events (e.g., probability of precipitation), the AUC interpretation can turn out to be oversimplistic or misleading when focusing on rare events (e.g., precipitation exceeding some warning criterion). How should we interpret AUC of ensemble forecasts when focusing on rare events? How can changes in the way probability forecasts are derived from the ensemble forecast affect AUC results? How can we detect a genuine improvement in terms of predictive skill? Based on verification experiments, a critical eye is cast on the AUC interpretation to answer these questions. As well as the traditional trapezoidal approximation and the well-known binormal fitting model, we discuss a new approach that embraces the concept of imprecise probabilities and relies on the subdivision of the lowest ensemble probability category.