Quantile Regression Neural Networks Research Articles

Evaluating Wind Speed Forecasting Models: A Comparative Study of CNN, DAN2, Random Forest and XGBOOST in Diverse South African Weather Conditions

The main source of electricity worldwide stems from fossil fuels, contributing to air pollution, global warming, and associated adverse effects. This study explores wind energy as a potential alternative. Nevertheless, the variable nature of wind introduces uncertainty in its reliability. Thus, it is necessary to identify an appropriate machine learning model capable of reliably forecasting wind speed under various environmental conditions. This research compares the effectiveness of Dynamic Architecture for Artificial Neural Networks (DAN2), convolutional neural networks (CNN), random forest and XGBOOST in predicting wind speed across three locations in South Africa, characterised by different weather patterns. The forecasts from the four models were then combined using quantile regression averaging models, generalised additive quantile regression (GAQR) and quantile regression neural networks (QRNN). Empirical results show that CNN outperforms DAN2 in accurately forecasting wind speed under different weather conditions. This superiority is likely due to the inherent architectural attributes of CNNs, including feature extraction capabilities, spatial hierarchy learning, and resilience to spatial variability. The results from the combined forecasts were comparable with those from the QRNN, which was slightly better than those from the GAQR model. However, the combined forecasts were more accurate than the individual models. These results could be useful to decision-makers in the energy sector.

Uncertainty estimation of machine learning spatial precipitation predictions from satellite data

Abstract Merging satellite and gauge data with machine learning produces high-resolution precipitation datasets, but uncertainty estimates are often missing. We addressed the gap of how to optimally provide such estimates by benchmarking six algorithms, mostly novel even for the more general task of quantifying predictive uncertainty in spatial prediction settings. On 15 years of monthly data from over the contiguous United States (CONUS), we compared quantile regression (QR), quantile regression forests (QRF), generalized random forests (GRF), gradient boosting machines (GBM), light gradient boosting machines (LightGBM), and quantile regression neural networks (QRNN). Their ability to issue predictive precipitation quantiles at nine quantile levels (0.025, 0.050, 0.100, 0.250, 0.500, 0.750, 0.900, 0.950, 0.975), approximating the full probability distribution, was evaluated using quantile scoring functions and the quantile scoring rule. Predictors at a site were nearby values from two satellite precipitation retrievals, namely PERSIANN (Precipitation Estimation from Remotely Sensed Information using Artificial Neural Networks) and IMERG (Integrated Multi-satellitE Retrievals), and the site’s elevation. The dependent variable was the monthly mean gauge precipitation. With respect to QR, LightGBM showed improved performance in terms of the quantile scoring rule by 11.10%, also surpassing QRF (7.96%), GRF (7.44%), GBM (4.64%) and QRNN (1.73%). Notably, LightGBM outperformed all random forest variants, the current standard in spatial prediction with machine learning. To conclude, we propose a suite of machine learning algorithms for estimating uncertainty in spatial data prediction, supported with a formal evaluation framework based on scoring functions and scoring rules.

Ultra-short-term wind power probabilistic forecasting based on an evolutionary non-crossing multi-output quantile regression deep neural network

Ultra-short-term wind power probabilistic forecasting is of significance for stable power grid operation; however, it is still challenging due to the inherent nonlinearity and uncertainty. Most state-of-the-art methods have focused on achieving quantile prediction using a combination of linear quantile regression and nonlinear complex deep neural networks. Yet, these methods struggle with several dilemmas. Quantile regression deep neural networks require a complete training once for each quantile. The multi-training mode and complex structure of quantile regression deep neural network can lead to extremely high computational complexity. Most of the training of quantile regression deep neural networks are guided by the loss of each quantile, and the weights are adjusted by gradient descent in which the gradient explosion and quantile crossover may be encountered. Therefore, this paper proposes a non-crossing multi-output quantile regression deep neural network optimized by chaotic particle swarm optimization. It designs a multi-output deep neural network to output all quantile estimations simultaneously through one training, effectively solving the structural complexity problem of traditional quantile regression deep neural networks. Since quantile regression produces a non-differentiable loss function which significantly hinders model training, the proposed neural network is trained by chaotic particle swarm optimization. It not only achieves the effect of optimizing all quantile losses simultaneously, but also can significantly alleviate the dilemma of training in traditional neural network weight optimization. In addition, several non-crossing constraints are designed for avoiding quantile crossover. The proposed model is trained and tested on two real-world wind power case studies. The experiment results show that the proposed model shows superiority in performance criteria, training speed, and avoiding quantile crossover.

Neural networks for quantile claim amount estimation: a quantile regression approach

AbstractIn this paper, we discuss the estimation of conditional quantiles of aggregate claim amounts for non-life insurance embedding the problem in a quantile regression framework using the neural network approach. As the first step, we consider the quantile regression neural networks (QRNN) procedure to compute quantiles for the insurance ratemaking framework. As the second step, we propose a new quantile regression combined actuarial neural network (Quantile-CANN) combining the traditional quantile regression approach with a QRNN. In both cases, we adopt a two-part model scheme where we fit a logistic regression to estimate the probability of positive claims and the QRNN model or the Quantile-CANN for the positive outcomes. Through a case study based on a health insurance dataset, we highlight the overall better performances of the proposed models with respect to the classical quantile regression one. We then use the estimated quantiles to calculate a loaded premium following the quantile premium principle, showing that the proposed models provide a better risk differentiation.

Dynamic line rating forecasting algorithm for a secure power system network

This research aimed to enhance the capacity of transmission lines by developing an algorithm to predict Dynamic Line Rating (DLR) to avert the curtailment of renewable energy sources. This will help meet electricity demand, prevent power outages, reduce costs, and protect equipment and personnel. The proposed algorithm was trained by analyzing the correlation between sequential hourly DLR calculations derived from historical weather data and line physical parameters. The proposed algorithm used k-means clustering and Monte Carlo simulation to predict hourly DLR, considering the temporal correlation of historical DLR values for each month. The model's accuracy was verified through statistical tests and was compared to other forecasting methods such as ensemble forecasting, quantile regression, and recurrent neural networks. It was found that the proposed model demonstrated performance that is comparable or superior to existing methods, as seen in forecast skill ranging from 91 to 98%, a Continuous Ranked Probability Score (CRPS) between 0.02 and 0.05, and probabilistic ratings that are 48–70% higher than traditional Static Line Rating (SLR).

Short-term forecasting of confirmed daily COVID-19 cases in the Southern African Development Community region.

The coronavirus pandemic has resulted in complex challenges worldwide, and the Southern African Development Community (SADC) region has not been spared. The region has become the epicentre for coronavirus in the African continent. Combining forecasting techniques can help capture other attributes of the series, thus providing crucial information to address the problem. To formulate an effective model that timely predicts the spread of COVID-19 in the SADC region. Using the Quantile regression approaches; linear quantile regression averaging (LQRA), monotone composite quantile regression neural network (MCQRNN), partial additive quantile regression averaging (PAQRA), among others, we combine point forecasts from four candidate models namely, the ARIMA (p, d, q) model, TBATS, Generalized additive model (GAM) and a Gradient Boosting machine (GBM). Among the single forecast models, the GAM provides the best model for predicting the spread of COVID-19 in the SADC region. However, it did not perform well in some periods. Combined forecasts models performed significantly better with the MCQRNN being the best (Theil's U statistic=0.000000278). The findings present an insightful approach in monitoring the spread of COVID-19 in the SADC region. The spread of COVID-19 can best be predicted using combined forecasts models, particularly the MCQRNN approach.

Modeling Censored Mobility Demand Through Censored Quantile Regression Neural Networks

Shared mobility services require accurate demand models for effective service planning. On the one hand, modeling the full probability distribution of demand is advantageous because the entire uncertainty structure preserves valuable information for decision-making. On the other hand, demand is often observed through the usage of the service itself, so that the observations are censored, as they are inherently limited by available supply. Since the 1980s, various works on Censored Quantile Regression models have performed well under such conditions. Further, in the last two decades, several papers have proposed to implement these models flexibly through Neural Networks. However, the models in current works estimate the quantiles individually, thus incurring a computational overhead and ignoring valuable relationships between the quantiles. We address this gap by extending current Censored Quantile Regression models to learn multiple quantiles at once and apply these to synthetic baseline datasets and datasets from two shared mobility providers in the Copenhagen metropolitan area in Denmark. The results show that our extended models yield fewer quantile crossings and less computational overhead without compromising model performance.

Ice water path retrievals from Meteosat-9 using quantile regression neural networks

Abstract. The relationship between geostationary radiances and ice water path (IWP) is complex, and traditional retrieval approaches are not optimal. This work applies machine learning to improve the IWP retrieval from Meteosat-9 observations, with a focus on low latitudes, training the models against retrievals based on CloudSat. Advantages of machine learning include avoiding explicit physical assumptions on the data, an efficient use of information from all channels, and easily leveraging spatial information. Thermal infrared (IR) retrievals are used as input to achieve a performance independent of the solar angle. They are compared with retrievals including solar reflectances as well as a subset of IR channels for compatibility with historical sensors. The retrievals are accomplished with quantile regression neural networks. This network type provides case-specific uncertainty estimates, compatible with non-Gaussian errors, and is flexible enough to be applied to different network architectures. Spatial information is incorporated into the network through a convolutional neural network (CNN) architecture. This choice outperforms architectures that only work pixelwise. In fact, the CNN shows a good retrieval performance by using only IR channels. This makes it possible to compute diurnal cycles, a problem that CloudSat cannot resolve due to its limited temporal and spatial sampling. These retrievals compare favourably with IWP retrievals in CLAAS, a dataset based on a traditional approach. These results highlight the possibilities to overcome limitations from physics-based approaches using machine learning while providing efficient, probabilistic IWP retrieval methods. Moreover, they suggest this first work can be extended to higher latitudes as well as that geostationary data can be considered as a complement to the upcoming Ice Cloud Imager mission, for example, to bridge the gap in temporal sampling with respect to space-based radars.

Uncertainty quantification for virtual diagnostic of particle accelerators

Virtual diagnostic (VD) is a computational tool based on deep learning that can be used to predict a diagnostic output. VDs are especially useful in systems where measuring the output is invasive, limited, costly or runs the risk of altering the output. Given a prediction, it is necessary to relay how reliable that prediction is, i.e., quantify the uncertainty of the prediction. In this paper, we use ensemble methods and quantile regression neural networks to explore different ways of creating and analyzing prediction's uncertainty on experimental data from the Linac Coherent Light Source at SLAC National Lab. We aim to accurately and confidently predict the current profile or longitudinal phase space images of the electron beam. The ability to make informed decisions under uncertainty is crucial for reliable deployment of deep learning tools on safety-critical systems as particle accelerators.

Quantile Regression Neural Networks: A Bayesian Approach

This article introduces a Bayesian neural network estimation method for quantile regression assuming an asymmetric Laplace distribution (ALD) for the response variable. It is shown that the posterior distribution for feedforward neural network quantile regression is asymptotically consistent under a misspecified ALD model. This consistency proof embeds the problem from density estimation domain and uses bounds on the bracketing entropy to derive the posterior consistency over Hellinger neighborhoods. This consistency result is shown in the setting where the number of hidden nodes grow with the sample size. The Bayesian implementation utilizes the normal-exponential mixture representation of the ALD density. The algorithm uses Markov chain Monte Carlo (MCMC) simulation technique - Gibbs sampling coupled with Metropolis-Hastings algorithm. We have addressed the issue of complexity associated with the afore-mentioned MCMC implementation in the context of chain convergence, choice of starting values, and step sizes. We have illustrated the proposed method with simulation studies and real data examples.

Can machine learning correct microwave humidity radiances for the influence of clouds?

Abstract. A methodology based on quantile regression neural networks (QRNNs) is presented that identifies and corrects the cloud impact on microwave humidity sounder radiances at 183 GHz. This approach estimates the posterior distributions of noise-free clear-sky (NFCS) radiances, providing nearly bias-free estimates of clear-sky radiances with a full posterior error distribution. It is first demonstrated by application to a present sensor, the MicroWave Humidity Sounder 2 (MWHS-2); then the applicability to sub-millimetre (sub-mm) sensors is also analysed. The QRNN results improve upon what operational cloud filtering techniques like a scattering index can achieve but are ultimately imperfect due to limited information content on cirrus impact from traditional microwave channels – the negative departures associated with high cloud impact are successfully corrected, but thin cirrus clouds cannot be fully corrected. In contrast, when sub-mm observations are used, QRNN successfully corrects most cases with cloud impact, with only 2 %–6 % of the cases left partially corrected. The methodology works well even if only one sub-mm channel (325 GHz) is available. When using sub-mm observations, cloud correction usually results in error distributions with a standard deviation less than typical channel noise values. Furthermore, QRNN outputs predicted quantiles for case-specific uncertainty estimates, successfully representing the uncertainty of cloud correction for each observation individually. In comparison to deterministic correction or filtering approaches, the corrected radiances and attendant uncertainty estimates have great potential to be used efficiently in assimilation systems due to being largely unbiased and adding little further uncertainty to the measurements.

An ensemble methodology for hierarchical probabilistic electric vehicle load forecasting at regular charging stations

Transportation electrification is a valid option for supporting decarbonization efforts but, at the same time, the growing number of electric vehicles will produce new and unpredictable load conditions for the electrical networks. Accurate electric vehicle load forecasting becomes essential to reduce adverse effects of electric vehicle integration into the grid. In this paper, a methodology dedicated to probabilistic electric vehicle load forecasting for different geographic regions is presented. The hierarchical approach is applied to decompose the problem into sub-problems at low-level regions, which are resolved through standard probabilistic models such as gradient boosted regression trees, quantile regression forests and quantile regression neural networks, coupled with principal component analysis to reduce the dimensionality of the sub-problems. The hierarchical perspective is then finalized to forecast the aggregate load at a high-level geographic region through an ensemble methodology based on a penalized linear quantile regression model. This paper brings, as relevant contributions, the development of hierarchical probabilistic forecasting framework, its comparison with non-hierarchical frameworks, and the assessment of the role of data dimensionality refduction. Extensive experimental results based on actual electric vehicle load data are presented which confirm that the hierarchical approaches increase the skill of probabilistic forecasts up to 9.5% compared with non-hierarchical approaches.

Forecasting of Landslide Displacement Using a Probability-Scheme Combination Ensemble Prediction Technique.

Data-driven models have been extensively employed in landslide displacement prediction. However, predictive uncertainty, which consists of input uncertainty, parameter uncertainty, and model uncertainty, is usually disregarded in deterministic data-driven modeling, and point estimates are separately presented. In this study, a probability-scheme combination ensemble prediction that employs quantile regression neural networks and kernel density estimation (QRNNs-KDE) is proposed for robust and accurate prediction and uncertainty quantification of landslide displacement. In the ensemble model, QRNNs serve as base learning algorithms to generate multiple base learners. Final ensemble prediction is obtained by integration of all base learners through a probability combination scheme based on KDE. The Fanjiaping landslide in the Three Gorges Reservoir area (TGRA) was selected as a case study to explore the performance of the ensemble prediction. Based on long-term (2006–2018) and near real-time monitoring data, a comprehensive analysis of the deformation characteristics was conducted for fully understanding the triggering factors. The experimental results indicate that the QRNNs-KDE approach can perform predictions with perfect performance and outperform the traditional backpropagation (BP), radial basis function (RBF), extreme learning machine (ELM), support vector machine (SVM) methods, bootstrap-extreme learning machine-artificial neural network (bootstrap-ELM-ANN), and Copula-kernel-based support vector machine quantile regression (Copula-KSVMQR). The proposed QRNNs-KDE approach has significant potential in medium-term to long-term horizon forecasting and quantification of uncertainty.

Probabilistic Hydrological Post-Processing at Scale: Why and How to Apply Machine-Learning Quantile Regression Algorithms

We conduct a large-scale benchmark experiment aiming to advance the use of machine-learning quantile regression algorithms for probabilistic hydrological post-processing “at scale” within operational contexts. The experiment is set up using 34-year-long daily time series of precipitation, temperature, evapotranspiration and streamflow for 511 catchments over the contiguous United States. Point hydrological predictions are obtained using the Génie Rural à 4 paramètres Journalier (GR4J) hydrological model and exploited as predictor variables within quantile regression settings. Six machine-learning quantile regression algorithms and their equal-weight combiner are applied to predict conditional quantiles of the hydrological model errors. The individual algorithms are quantile regression, generalized random forests for quantile regression, generalized random forests for quantile regression emulating quantile regression forests, gradient boosting machine, model-based boosting with linear models as base learners and quantile regression neural networks. The conditional quantiles of the hydrological model errors are transformed to conditional quantiles of daily streamflow, which are finally assessed using proper performance scores and benchmarking. The assessment concerns various levels of predictive quantiles and central prediction intervals, while it is made both independently of the flow magnitude and conditional upon this magnitude. Key aspects of the developed methodological framework are highlighted, and practical recommendations are formulated. In technical hydro-meteorological applications, the algorithms should be applied preferably in a way that maximizes the benefits and reduces the risks from their use. This can be achieved by (i) combining algorithms (e.g., by averaging their predictions) and (ii) integrating algorithms within systematic frameworks (i.e., by using the algorithms according to their identified skills), as our large-scale results point out.

Quantile regression neural network‐based fault detection scheme for wind turbines with application to monitoring a bearing

AbstractUnder the framework of normal behavior modeling, this paper develops a novel scheme for fault detection via quantile regression neural networks (QRNNs). The QRNN model is a combination of quantile regressions and neural networks. It is able to identify the normal status or extract the normal behavior data accurately and quickly through lower and upper regression quantiles. Additionally, it is flexible to explore the potential nonlinear patterns contained in the normal status by taking advantage of neural networks. Finally, we monitor the residuals produced from QRNN to detect faults by using the exponentially weighted moving average (EWMA) control chart. The utility of our scheme is illustrated by empirical analyses of bearing fault detection based on Supervisory Control and Data Acquisition (SCADA) data from a wind turbine. We find that the QRNN model outperforms the multiple linear regression (MLR) and back propagation neural networks (BPNNs) in terms of mean absolute error (MAE). Besides, the obtained relationship between the width of control limits in EWMA and the number of alarms provides an important and convenient way for practical applications.

A neural network approach to estimating a posteriori distributions of Bayesian retrieval problems

Abstract. A neural-network-based method, quantile regression neural networks (QRNNs), is proposed as a novel approach to estimating the a posteriori distribution of Bayesian remote sensing retrievals. The advantage of QRNNs over conventional neural network retrievals is that they learn to predict not only a single retrieval value but also the associated, case-specific uncertainties. In this study, the retrieval performance of QRNNs is characterized and compared to that of other state-of-the-art retrieval methods. A synthetic retrieval scenario is presented and used as a validation case for the application of QRNNs to Bayesian retrieval problems. The QRNN retrieval performance is evaluated against Markov chain Monte Carlo simulation and another Bayesian method based on Monte Carlo integration over a retrieval database. The scenario is also used to investigate how different hyperparameter configurations and training set sizes affect the retrieval performance. In the second part of the study, QRNNs are applied to the retrieval of cloud top pressure from observations by the Moderate Resolution Imaging Spectroradiometer (MODIS). It is shown that QRNNs are not only capable of achieving similar accuracy to standard neural network retrievals but also provide statistically consistent uncertainty estimates for non-Gaussian retrieval errors. The results presented in this work show that QRNNs are able to combine the flexibility and computational efficiency of the machine learning approach with the theoretically sound handling of uncertainties of the Bayesian framework. Together with this article, a Python implementation of QRNNs is released through a public repository to make the method available to the scientific community.

Statistical downscaling of precipitation using long short-term memory recurrent neural networks

Hydrological impacts of global climate change on regional scale are generally assessed by downscaling large-scale climatic variables, simulated by General Circulation Models (GCMs), to regional, small-scale hydrometeorological variables like precipitation, temperature, etc. In this study, we propose a new statistical downscaling model based on Recurrent Neural Network with Long Short-Term Memory which captures the spatio-temporal dependencies in local rainfall. The previous studies have used several other methods such as linear regression, quantile regression, kernel regression, beta regression, and artificial neural networks. Deep neural networks and recurrent neural networks have been shown to be highly promising in modeling complex and highly non-linear relationships between input and output variables in different domains and hence we investigated their performance in the task of statistical downscaling. We have tested this model on two datasets—one on precipitation in Mahanadi basin in India and the second on precipitation in Campbell River basin in Canada. Our autoencoder coupled long short-term memory recurrent neural network model performs the best compared to other existing methods on both the datasets with respect to temporal cross-correlation, mean squared error, and capturing the extremes.

Post-Processing of Stream Flows in Switzerland with an Emphasis on Low Flows and Floods

Post-processing has received much attention during the last couple of years within the hydrological community, and many different methods have been developed and tested, especially in the field of flood forecasting. Apart from the different meanings of the phrase “post-processing” in meteorology and hydrology, in this paper, it is regarded as a method to correct model outputs (predictions) based on meteorological (1) observed input data, (2) deterministic forecasts (single time series) and (3) ensemble forecasts (multiple time series) and to derive predictive uncertainties. So far, the majority of the research has been related to floods, how to remove bias and improve the forecast accuracy and how to minimize dispersion errors. Given that global changes are driving climatic forces, there is an urgent need to improve the quality of low-flow predictions, as well, even in regions that are normally less prone to drought. For several catchments in Switzerland, different post-processing methods were tested with respect to low stream flow and flooding conditions. The complexity of the applied procedures ranged from simple AR processes to more complex methodologies combining wavelet transformations and Quantile Regression Neural Networks (QRNN) and included the derivation of predictive uncertainties. Furthermore, various verification methods were tested in order to quantify the possible improvements that could be gained by applying these post-processing procedures based on different stream flow conditions. Preliminary results indicate that there is no single best method, but with an increase of complexity, a significant improvement of the quality of the predictions can be achieved.

Quantile Regression Neural Networks Based Prediction of Drug Activities

<p class="Abstract">QSAR (quantitative structure-activity relationship) modeling is one of the well developed areas in drug development through computational chemistry. Similar molecules with just a slight variation in their structure can have quit different biological activity. This kind of relationship between molecular structure and change in biological activity is center of focus for QSAR Modeling. Predictions of property and/or activity of interest have the potential to save time, money and minimize the use of expensive experimental designs, such as, for example, animal testing. Intelligent machine learning techniques are important tools for QSAR analysis, as a result, they are integrated into the drug production process. The effective learnable model can reduce the cost of drug design significantly. The quantile estimation via neural network structure technique introduced in this paper is used to predict activity of pyrimidines based on the structure–activity relationship of these compounds which assist for finding potential treatment agents for serious disease. In comparison with statistical quantile regression, the qrnn significantly reduce the prediction error.</p>

Estimating Distribution of Concrete Strength Using Quantile Regression Neural Networks

This paper is aimed at demonstrating the possibilities of adaptingQuantile Regression Neural Network (QRNN) to estimate the distribution ofcompressive strength of high performance concrete (HPC). The databasecontaining 1030 compressive strength data were used to evaluate QRNN. Each dataincludes the amounts of cement, blast furnace slag, fly ash, water,superplasticizer, coarse aggregate, fine aggregate (in kilograms per cubicmeter), the age, and the compressive strength. This study led to the followingconclusions: (1) The Quantile Regression Neural Networks can buildaccurate quantile models and estimate the distribution of compressive strengthof HPC. (2) The various distributions of prediction of compressive strength of HPCshow that the variance of the error is inconstant across observations, whichimply that the prediction is heteroscedastic. (3) The logarithmic normaldistribution may be more appropriate than normal distribution to fit thedistribution of compressive strength of HPC. Since engineers should not assumethat the variance of the error of prediction of compressive strength isconstant, the ability of estimating the distribution of compressive strength ofHPC is an important advantage of QRNN.