Prediction Interval Estimation Research Articles

Estimation of Prediction Intervals for Performance Assessment of Building Using Machine Learning.

This study utilizes artificial neural networks (ANN) to estimate prediction intervals (PI) for seismic performance assessment of buildings subjected to long-term ground motion. To address the uncertainty quantification in structural health monitoring (SHM), the quality-driven lower upper bound estimation (QD-LUBE) has been opted for global probabilistic assessment of damage at local and global levels, unlike traditional methods. A distribution-free machine learning model has been adopted for enhanced reliability in quantifying uncertainty and ensuring robustness in post-earthquake probabilistic assessments and early warning systems. The distribution-free machine learning model is capable of quantifying uncertainty with high accuracy as compared to previous methods such as the bootstrap method, etc. This research demonstrates the efficacy of the QD-LUBE method in complex seismic risk assessment scenarios, thereby contributing significant enhancement in building resilience and disaster management strategies. This study also validates the findings through fragility curve analysis, offering comprehensive insights into structural damage assessment and mitigation strategies.

An integrated skip convolutional network with residual learning and feature extraction for point and interval prediction of solar radiation

Spinning reserve based on solar radiation prediction can ensure the secure operation of large-scale grid-connected photovoltaic systems, but irrational spinning reserve can lead to substantial economic losses and even grid collapse. Therefore, it is imperative to identify characteristic patterns of solar radiation and establish an effective solar radiation prediction model. However, existing hybrid models often struggle to efficiently recognize and leverage input features, compromising the robustness of the model. To address this limitation, this study proposes an integrated skip-convolutional network with residual learning and feature extraction (InSCNet), which enhances the capability to represent information data and thereby improves the stability and accuracy of solar radiation prediction by employing a series of feature extraction and prediction blocks with residual learning. InSCNet includes a sophisticated feature extraction module to effectively address the underutilization of feature information, and it incorporates residual learning, skip-convolution, and recursive networks in the prediction module, reducing the risk of gradient vanishing or gradient explosion while enhancing prediction accuracy. Additionally, an interval estimation method with adaptively chosen distributions is introduced, which extends the prediction interval estimation method. The proposed InSCNet is rigorously evaluated using datasets from three major cities in Pakistan. Experimental results demonstrate that InSCNet outperforms existing solutions in both point and interval predictions.

Lightweight Regression Model with Prediction Interval Estimation for Computer Vision-based Winter Road Surface Condition Monitoring

Lightweight Regression Model with Prediction Interval Estimation for Computer Vision-based Winter Road Surface Condition Monitoring

Conformal Prediction for Time Series.

We present a general framework for constructing distribution-free prediction intervals for time series. We establish explicit bounds on the conditional and marginal coverage gaps of estimated prediction intervals, which asymptotically converge to zero under additional assumptions. We also provide similar bounds on the size of set differences between oracle and estimated prediction intervals. To implement this framework, we introduce an efficient algorithm called EnbPI, which utilizes ensemble predictors and is closely related to conformal prediction (CP) but does not require data exchangeability. Unlike other methods, EnbPI avoids data-splitting and is computationally efficient by avoiding retraining, making it scalable for sequentially producing prediction intervals. Extensive simulation and real-data analyses demonstrate the effectiveness of EnbPI compared to existing methods.

Uncertainty Quantification of Deep Learning–Based Statistical Downscaling of Climatic Parameters

Abstract This study investigated the uncertainty involved in statistical downscaling of hydroclimatic time series obtained by artificial neural networks (ANNs). Phase 6 of the Coupled Model Intercomparison Project (CMIP6) general circulation model (GCM) Canadian Earth System Model, version 5 (CanESM5), was used as large-scale predictor data for downscaling temperature and precipitation parameters. Two ANNs, feed forward and long short-term memory (LSTM), were utilized for statistical downscaling. To quantify the uncertainty of downscaling, prediction intervals (PIs) were estimated via the lower upper bound estimation (LUBE) method. To assess performance of proposed models in different climate regimes, data from the Tabriz and Rasht stations in Iran were employed. The calibrated models via historical GCM data were used for future projections via the high-forcing and fossil fuel–driven development scenario shared socioeconomic pathway (SSP) 5-8.5. Projections were compared with the Canadian Regional Climate Model 4 (Can-RCM4) projections via the same scenario. Results indicated that both LSTM-based point predictions and PIs are more accurate than the feedforward neural network (FFNN)-based predictions, with an average of 55% higher Nash–Sutcliffe efficiency (NSE) for point predictions and 25% lower coverage width criterion (CWC) for PIs. Projections suggested that Tabriz is going to experience a warmer climate with an increase in average temperature of 2° and 5°C for near and far futures, respectively, and a drier climate with a 20% decrease in precipitation until 2100. Future projections for the Rasht station, however, suggested a more uniform climate with less seasonal variability. Average precipitation will increase by up to 25% and 70% until near and far future periods, respectively. Ultimately, point predictions show that the average temperature in Rasht will increase by 1°C until the near future and then be a constant average temperature until the far future. Significance Statement The downscaling of hydroclimatic parameters is subject to uncertainty. The best way is to provide an area with the highest contingency of them as a prediction interval. The reduction width of such an interval leads to increased confidence in explaining and predicting these processes. We proposed and applied a deep learning–based machine learning method for both point prediction and prediction interval estimation of temperature and precipitation parameters for the future over two different climatic regions. The results show the superiority of such a machine learning–based prediction interval estimation for quantification of the downscaling uncertainty.

Estimation of prediction intervals for uncertainty assessment of artificial neural network based wastewater treatment plant effluent modeling

Uncertainty qualification of estimated biological variables of Tabriz wastewater treatment plant (WWTP) in Iran, was conducted by artificial intelligence-based modeling. The pollution variables at entrance of sedimentation section and its output were considered as inputs and outputs of the modeling. Non-linear mutual information (MI) and linear correlation coefficient (CC) were used to select inputs. Modeling was conducted via two feed-forward neural networks (FFNN) models for point prediction (PP) and lower and upper bound estimation (LUBE) to provide prediction intervals (PIs) of the computed variables as representative of the modeling uncertainty. In the proposed LUBE, an FFNN was trained with two outputs as upper and lower limits of the predictions. The results showed that selecting the inputs through MI led to narrower PI bound and less uncertainty. Computed determination coefficient (DC)train and DCTest for modeling effluent biological oxygen demand (BODeff) were respectively as 0.9 and 0.7, whereas the values for modeling effluent chemical oxygen demand (CODeff) were 0.8 and 0.7. Comparison of non-linear machine learning (ML) models of FFNN, support vector machine (SVR), adaptive neuro-fuzzy inference system (ANFIS), and linear autoregressive integrated moving average (ARIMA) model demonstrated the superiority of FFNN method for PP task over others. In the LUBE method, the best relationship between pollution parameters was obtained via the MI method. PICP and CWC values were obtained for BOD as 94.08 % and 0.152 and for COD as 86.41 % and 0.1769, respectively. The innovation of this research is to examine the uncertainties in neural network-based analysis of WWTP modeling.

Uncertainty quantification of the virial black hole mass with conformal prediction

ABSTRACT Precise measurements of the black hole mass are essential to gain insight on the black hole and host galaxy co-evolution. A direct measure of the black hole mass is often restricted to nearest galaxies and instead, an indirect method using the single-epoch virial black hole mass estimation is used for objects at high redshifts. However, this method is subjected to biases and uncertainties as it is reliant on the scaling relation from a small sample of local active galactic nuclei. In this study, we propose the application of conformalized quantile regression (CQR) to quantify the uncertainties of the black hole predictions in a machine learning setting. We compare CQR with various prediction interval techniques and demonstrated that CQR can provide a more useful prediction interval indicator. In contrast to baseline approaches for prediction interval estimation, we show that the CQR method provides prediction intervals that adjust to the black hole mass and its related properties. That is it yields a tighter constraint on the prediction interval (hence more certain) for a larger black hole mass, and accordingly, bright and broad spectral line width source. Using a combination of neural network model and CQR framework, the recovered virial black hole mass predictions and uncertainties are comparable to those measured from the Sloan Digital Sky Survey. The code is publicly available.

Spatial–temporal information model-based load current interval prediction for transmission lines

Load current interval prediction (LCIP) plays an increasingly significant role in transmission lines load demand uncertainty assessment and becomes necessary for power system dispatch and management. However, the stochastic, volatility and non-stationary characteristics of load demand bring challenges to high-quality prediction intervals (PIs) estimation. In this article, a hybrid LCIP model based on Spatial–Temporal Attention integrating Temporal Convolutional Network (STATCN) and residual estimation is proposed. First, to diminish the uncertainty in the prediction process, a spatial-partitioned method and a temporal-phased method are proposed for decoupling the spatial and temporal correlations among transmission lines. Then, a spatial–temporal attention mechanism is fused in temporal convolutional networks to enhance the sensitivity of ensemble model for important information from the temporal and spatial feature dimensions. Finally, in the STATCN interval prediction model, a residual estimation optimization strategy is designed to eliminate quantile crossings and ensure the validity of the prediction interval results. The performance of the proposed prediction framework is verified by actual numerical simulation. The results demonstrate that our proposed method has higher quality PIs compared to the existing state-of-the-art interval prediction methods.

Prediction interval estimation of sinter drum index based on light gradient boosting machine and kernel density estimation

ABSTRACT Owing to the uncertainty operation in the sintering process, it is easy to produce uncertain prediction errors in the single drum index prediction model, which makes the prediction results lack certain reliability. Accurate and reliable prediction of the drum index can help improve the drum index. In this paper, a prediction interval estimation method of drum index based on a light gradient boosting machine (LightGBM) and kernel density estimation (KDE) is proposed. LightGBM can obtain accurate points prediction of drum index, and then use the KDE method to obtain the estimated prediction interval of drum index. The comparison results of different methods show that LightGBM has high prediction performance, and KDE can well quantify the prediction error of drum index, which verifies the effectiveness of the prediction interval estimation method combined with LightGBM and KDE, and provides more reliable decision-making information for the optimisation of sintering process parameters.

Prediction Interval Estimation and Deterministic Forecasting Model Using Ground-Based Sky Image

Due to the explosive penetration of photovoltaic (PV) systems in the power grid, accurate and well-informed solar PV power forecasting has become exceptionally crucial, while accurate irradiance prediction is the basis of PV power forecasting. Yet most research on PV power/irradiance forecasting focus on deterministic forecasting, whereas it is difficult to express the credibility of the forecasting results, especially in the scene of instantaneous fluctuation of irradiance and photovoltaic power caused by cloud movement. Therefore, in order to accurately estimate and track the rapid fluctuations of solar irradiance, this paper takes the ground-based sky image information into consideration and obtains the probability-level prediction results along with the deterministic point forecasting, proposes a comprehensive evaluation function constructed with the average coverage and the average width as the fitness function to assist in the optimization of the deterministic forecasting results, and then constructs a three-output model, naming it DenLUBE. In addition, a framework for irradiance variation pattern prediction is proposed to provide pattern information for the forecasting time period to further improve the performance of the ultra-short-term forecasting model. The results show that the proposed model can achieve high quality and strongly robust deterministic forecasting results while producing tight prediction intervals independent of weather conditions. In addition, the performance of our model is greatly enhanced compared to the traditional approach of taking the middle position of the interval as the deterministic forecasting result, as well as other benchmark models.

Reliability-based design considering prediction interval estimation to optimize composite patches

A modified reliability-based design optimization (RBDO) technique was developed and applied to optimal designs of composite patches of a perforated base plate. The modified RBDO technique uses adaptive sampling strategies for the response surface method (RSM) to enhance the computational efficiency of the traditional RSM-based RBDO technique. Optimal design of composite patches was conducted for a perforated base plate using the newly proposed failure criteria which state that two conditions must be satisfied simultaneously for failure to occur. One condition is based on stress magnitude while the second condition is associated with the stress gradient. In other words, the failure criteria suggests that failure is more likely for higher stresses and lower stress gradients. Therefore, the design optimization of the composite patch was conducted to decrease the stress and to increase the stress gradient simultaneously at the edge of the perforated hole of the base plate in order to optimize the probability of failure of the base plate with composite patches.

Distribution‐free predictive inference for partial least squares regression with applications to molecular descriptors datasets

AbstractIn statistical modeling, partial least squares (PLS) regression is one of the most popular techniques for prediction problems. An important but often overlooked problem is that the estimation of prediction confidence intervals always contains an unobserved response value with a specified probability. Therefore, in the present work, we studied how to estimate prediction intervals in PLS regression without any distributional assumptions on data. First, a recently proposed method, jackknife+, is introduced to PLS regression for uncertainty quantification. Second, a novel approach is developed to construct distribution‐free prediction intervals. Finally, empirical studies on simulations and three real‐world datasets show that the proposed method has better coverage properties than other state‐of‐the‐art methods.

Interventions Associated With Reduced Loneliness and Social Isolation in Older Adults

Loneliness and social isolation are public health concerns faced by older adults due to physical, cognitive, and psychosocial changes that develop with aging. Loneliness and social isolation are associated with increased morbidity and mortality. To evaluate interventions, targeting older adults, associated with a reduction in loneliness and social isolation. OVID, CINAHL, CENTRAL, Embase, PsychINFO, Web of Science, and Scopus were searched from inception to March 2020. Peer-reviewed randomized clinical trials measuring loneliness and social isolation or support in adults aged 65 years or older. Only English language articles were included. Two independent reviewers screened studies, extracted data, and assessed risk of bias. Random-effects models were performed to pool the overall effect size by intervention. Statistical heterogeneity was evaluated with the I2 statistic and by estimating prediction intervals. Data were analyzed from November 2021 to September 2022. Quantitative measures of loneliness, social isolation, or social support based on an effect size of standardized mean differences. Seventy studies were included in the systematic review (8259 participants); 44 studies were included in the loneliness meta-analysis (33 in the community with 3535 participants; 11 in long-term care with 1057 participants), with participants' ages ranging from 55 to 100 years. Study sizes ranged from 8 to 741 participants. Interventions included animal therapy, psychotherapy or cognitive behavioral therapy, multicomponent, counseling, exercise, music therapy, occupational therapy, reminiscence therapy, social interventions, and technological interventions. Most interventions had a small effect size. Animal therapy in long-term care, when accounting for studies with no active controls, had the largest effect size on loneliness reduction (-1.86; 95% CI, -3.14 to -0.59; I2 = 86%) followed by technological interventions (videoconferencing) in long-term care (-1.40; 95% CI, -2.37 to -0.44; I2 = 70%). In this study, animal therapy and technology in long-term care had large effect sizes, but also high heterogeneity, so the effect size's magnitude should be interpreted with caution. The small number of studies per intervention limits conclusions on sources of heterogeneity. Overall quality of evidence was very low. Future studies should consider measures of social isolation in long-term care and identify the contextual components that are associated with a reduction in loneliness.

Robust prediction interval estimation for Gaussian processes by cross-validation method

Probabilistic regression models typically use the Maximum Likelihood Estimation or Cross-Validation to fit parameters. These methods can give an advantage to the solutions that fit observations on average, but they do not pay attention to the coverage and the width of Prediction Intervals. A robust two-step approach is used to address the problem of adjusting and calibrating Prediction Intervals for Gaussian Processes Regression. First, the covariance hyperparameters are determined by a standard Cross-Validation or Maximum Likelihood Estimation method. A Leave-One-Out Coverage Probability is introduced as a metric to adjust the covariance hyperparameters and assess the optimal type II Coverage Probability to a nominal level. Then a relaxation method is applied to choose the hyperparameters that minimize the Wasserstein distance between the Gaussian distribution with the initial hyperparameters (obtained by Cross-Validation or Maximum Likelihood Estimation) and the proposed Gaussian distribution with the hyperparameters that achieve the desired Coverage Probability. The method gives Prediction Intervals with appropriate coverage probabilities and small widths.

Direct estimation of prediction intervals for solar and wind regional energy forecasting with deep neural networks

Deep neural networks (DNN) are becoming increasingly relevant for probabilistic forecasting because of their ability to estimate prediction intervals (PIs). Two different ways for estimating PIs with neural networks stand out: quantile estimation for posterior PI construction and direct PI estimation. The former first estimates quantiles, which are then used to construct PIs, while the latter directly obtains the lower and upper PI bounds by optimizing some loss functions, with the advantage that PI width is directly considered in the optimization process and thus may result in narrower intervals. In this work, two different DNN-based models are studied for direct PI estimation, and compared with DNN for quantile estimation in the context of solar and wind regional energy forecasting. The first approach is based on the recent quality-driven loss and is formulated to estimate multiple PIs with a single model. The second is a novel approach that employs hypernetworks (HN), where direct PI estimation is formulated as a multi-objective problem, returning a Pareto front of solutions that contains all possible coverage-width optimal trade-offs. This formulation allows HN to obtain optimal PIs for all possible coverages without increasing the number of network outputs or adjusting additional hyperparameters, as opposed to the first direct model. Results show that prediction intervals from direct estimation are narrower (up to 20%) than those of quantile estimation, for target coverages 70%–80% for all regions, and also 85%, 90%, and 95% depending on the region, while HN always achieves the required coverage for the higher target coverages.

Valid prediction intervals for regression problems

Over the last few decades, various methods have been proposed for estimating prediction intervals in regression settings, including Bayesian methods, ensemble methods, direct interval estimation methods and conformal prediction methods. An important issue is the calibration of these methods: the generated prediction intervals should have a predefined coverage level, without being overly conservative. In this work, we review the above four classes of methods from a conceptual and experimental point of view. Results on benchmark data sets from various domains highlight large fluctuations in performance from one data set to another. These observations can be attributed to the violation of certain assumptions that are inherent to some classes of methods. We illustrate how conformal prediction can be used as a general calibration procedure for methods that deliver poor results without a calibration step.

Adult Skeletal Age-at-Death Estimation through Deep Random Neural Networks: A New Method and Its Computational Analysis.

Simple SummaryAge-at-death is of paramount importance in forensic analysis of skeletal remains. In addition to sex, stature, and population affinity, it constitutes baseline information in the identification process of deceased individuals. Despite its long tradition, in anthropological research age-at-death estimation poses many challenges and unanswered questions. It is undisputedly among the most difficult tasks of the forensic anthropologist and its results are often subject to a lackluster performance. In this study, we assessed computationally the efficiency of a holistic approach to skeletal age estimation based on a new proposal for macroscopic examination and the use of machine learning-based models for data analysis. Our results suggest that this approach is key for accurate and efficient age-at-death estimation based on skeletal remains analysis.Age-at-death assessment is a crucial step in the identification process of skeletal human remains. Nonetheless, in adult individuals this task is particularly difficult to achieve with reasonable accuracy due to high variability in the senescence processes. To improve the accuracy of age-at-estimation, in this work we propose a new method based on a multifactorial macroscopic analysis and deep random neural network models. A sample of 500 identified skeletons was used to establish a reference dataset (age-at-death: 19–101 years old, 250 males and 250 females). A total of 64 skeletal traits are covered in the proposed macroscopic technique. Age-at-death estimation is tackled from a function approximation perspective and a regression approach is used to infer both point and prediction interval estimates. Based on cross-validation and computational experiments, our results demonstrate that age estimation from skeletal remains can be accurately (~6 years mean absolute error) inferred across the entire adult age span and informative estimates and prediction intervals can be obtained for the elderly population. A novel software tool, DRNNAGE, was made available to the community.

Uncertainty assessment of LSTM based groundwater level predictions

ABSTRACT Due to the underlying uncertainty in groundwater level (GWL) modelling, point prediction of GWLs does not provide sufficient information. Moreover, the insufficiency of data on subjects such as illegal exploitation wells and wastewater pounds, which are untraceable, underlines the importance of evolved uncertainty in the groundwaters of the Ardabil plain. Thus, estimating prediction intervals (PIs) for groundwater modelling can be an important step. In this paper, PIs were estimated for GWLs of selected piezometers of the Ardebil plain in Iran using the artificial neural network (ANN)-based lower upper bound estimation (LUBE) method. The classic feedforward neural network (FFNN) and deep-learning-based long short-term memory (LSTM) were used. GWL data of piezometers and hydrological data (1992–2018) were applied for modelling. The results indicate that LSTM outperforms FFNN in both PI and point prediction tasks. LSTM-based LUBE was found to be superior to FFNN-based LUBE, providing an average 25% lower coverage width criterion (CWC). PIs estimated for piezometers with high transmissivity resulted in 50% lower CWC than PIs estimated for piezometers in areas with lower transmissivity.

Fractional stochastic configuration networks-based nonstationary time series prediction and confidence interval estimation

Time series prediction is an important topic in the field of data analytics for real industrial production. However, the time series from real system usually has strong nonstationarity, which affects the generalization ability of the prediction model. An improved forecasting technique, named as fractional stochastic configuration networks (FSCN), is proposed for the prediction of nonstationary time series. FSCN is built on the basis of traditional stochastic configuration network by introducing fractional differential operator. The fractional differential technique avoids the challenge of infinite variance caused by modeling nonstationary series and the over-difference problem caused by traditional integer order difference. First, several data analysis methods are introduced to find the tendency, periodicity and probability density distribution characteristics hidden in the raw industrial data. Hurst exponent is calculated to determine the order of fractional difference to eliminate the nonstationarity of the raw data. Then FSCN network is constructed to model and forecast the sequential data. An explicit prediction uncertainty is derived to provide the confidence interval for the FSCN prediction. The proposed method is tested on a nonstationary time series benchmark dataset and a real cooling system. The experiment result demonstrates that it has a good potential prediction performance compared with several traditional prediction methods.

Estimating timber volume loss due to storm damage in Carinthia, Austria, using ALS/TLS and spatial regression models

A spatial regression model framework is presented to predict growing stock volume loss due to storm Adrian which caused heavy forest damage in the upper Gail valley in Carinthia, Austria, in October 2018. Model parameters were estimated using growing stock volume measured with a terrestrial laser scanner on 62 sample plots distributed across five sub-regions. Predictor variables were derived from high resolution vegetation height measurements collected during an airborne laser scanning campaign. Non-spatial and spatial candidate models were proposed and assessed based on fit to observed data and out-of-sample prediction. Spatial Gaussian processes associated model intercepts and regression coefficients were used to capture spatial dependence. Results show a spatially-varying coefficient model, which allowed the intercept and regression coefficients to vary spatially, yielded the best fit and prediction. Two approaches were considered for prediction over blowdown areas: 1) an areal approach that viewed each blowdown as a single prediction unit indexed by its centroid; and 2) a block approach where each blowdown was partitioned into smaller prediction units to better align with sample plots’ spatial support. Joint prediction was used to acknowledge spatial dependence among block units. Results demonstrated the block approach is preferable as it mitigated change-of-support issues encountered in the areal approach. Despite the small sample size, predictions for 55% of the total 564 blowdown areas, accounting for 93% of the total loss, had a coefficient of variation less than 25%. Key advantages of the proposed regression framework and chosen Bayesian inferential paradigm, were the ability to quantify uncertainty in spatial covariance parameters, propagate parameter uncertainty through to prediction, and provide statistically valid prediction point and interval estimates for individual blowdowns and collections of blowdowns at the sub-region and region scale via posterior predictive distribution summaries.