Computing Prediction Intervals Research Articles

Methods to Compute Prediction Intervals: A Review and New Results

The purpose of this paper is to review both classic and modern methods for constructing prediction intervals. We focus, primarily, on model-based non-Bayesian methods for the prediction of a scalar random variable, but we also include Bayesian methods with objective prior distributions. Our review of non-Bayesian methods follows two lines: general methods based on (approximate) pivotal quantities and methods based on non-Bayesian predictive distributions. The connection between these two types of methods is described for distributions in the (log-)location-scale family. We also discuss extending the general prediction methods to data with complicated dependence structures as well as some nonparametric prediction methods (e.g., conformal prediction).

A data-driven approach based on quantile regression forest to forecast cooling load for commercial buildings

• Data-driven approach based on quantile regression forest for forecasting cooling load. • Input variables selection by applying mutual information and recursive feature elimination. • Uncertainty quantification by computing Prediction Intervals (PIs). Reliable prediction of thermal load is essential for implementing an efficient and economic energy management plan in commercial buildings. While previous research has been concerned with point forecasts , in this study we focus on forecasting prediction intervals for building thermal load. Prediction Intervals (PIs) are more useful than point forecasts as they can quantify the uncertainty associated with energy consumption in the buildings and enable the market participants to exploit current energy market benefits more effectively. We present a data-driven approach for forecasting PIs for building cooling load which first uses machine learning feature selection methods to identify a small but informative set of variables. It then applies quantile regression forest as the prediction algorithm that uses the selected features as inputs to compute the upper and lower boundaries of PIs. We evaluate the performance of the proposed approach using real data sets from two commercial buildings: a large shopping centre and an office building. The results show that the proposed approach can generate narrow and reliable PIs while satisfying the pre-specified coverage probabilities. The approach is fast to train and significantly outperforms traditional Gradient Boosting Regression (GBR) model in terms of reliability of the generated PIs.

Forecasting Crowd Counts With Wi-Fi Systems: Univariate, Non-Seasonal Models

Recently, event organizers and researchers have advocated the development of novel technologies supporting crowd control, notably for public events. This paper presents a crowd monitoring system based on probe requests (PRs), which are Wi-Fi packets smartphones send periodically. By estimating the global rate at which nearby smartphones send PRs, Wi-Fi sensors can estimate crowd counts. The core contribution of this paper is a computationally tractable method that forecasts crowd counts up to thirty minutes in the future, with forecasts becoming available as soon as two hours of data are available. The forecasting method relies on autoregressive integrated moving average (ARIMA) models. Contributions also include two methods that compute prediction intervals associated with the forecasts, one of which is based upon generalized autoregressive conditional heteroskedasticity (GARCH) models. Recent real-world data from Winter Wonders 2018/2019 (an event that took place in Brussels, Belgium) notably demonstrate that the proposed forecasting method outperforms its immediate variations as well as baseline models (i.e., random walk models).

Learning Prediction Intervals for Model Performance

Understanding model performance on unlabeled data is a fundamental challenge of developing, deploying, and maintaining AI systems. Model performance is typically evaluated using test sets or periodic manual quality assessments, both of which require laborious manual data labeling. Automated performance prediction techniques aim to mitigate this burden, but potential inaccuracy and a lack of trust in their predictions has prevented their widespread adoption. We address this core problem of performance prediction uncertainty with a method to compute prediction intervals for model performance. Our methodology uses transfer learning to train an uncertainty model to estimate the uncertainty of model performance predictions. We evaluate our approach across a wide range of drift conditions and show substantial improvement over competitive baselines. We believe this result makes prediction intervals, and performance prediction in general, significantly more practical for real-world use.

Predictive Distribution Modeling Using Transformation Forests

Regression models for supervised learning problems with a continuous response are commonly understood as models for the conditional mean of the response given predictors. This notion is simple and therefore appealing for interpretation and visualization. Information about the whole underlying conditional distribution is, however, not available from these models. A more general understanding of regression models as models for conditional distributions allows much broader inference, for example, the computation of prediction intervals or probabilistic predictions for exceeding certain thresholds. Several random forest-type algorithms aim at estimating conditional distributions, most prominently quantile regression forests. We propose a novel approach based on a parametric family of distributions characterized by their transformation function. A dedicated novel “transformation tree” algorithm able to detect distributional changes is developed. Based on these transformation trees, we introduce “transformation forests” as an adaptive local likelihood estimator of conditional distribution functions. The resulting predictive distributions are fully parametric yet very general and allow inference procedures, such as likelihood-based variable importances, to be applied in a straightforward way. Supplemental files for this article are available online.

Bayesian inference and uncertainty propagation using efficient fractional-order viscoelastic models for dielectric elastomers

Dielectric elastomers are employed for a wide variety of adaptive structures. Many of these soft elastomers exhibit significant rate-dependencies in their response. Accurately quantifying this viscoelastic behavior is non-trivial and in many cases a nonlinear modeling framework is required. Fractional-order operators have been applied to modeling viscoelastic behavior for many years, and recent research has shown fractional-order methods to be effective for nonlinear frameworks. This implementation can become computationally expensive to achieve an accurate approximation of the fractional-order derivative. Accurate estimation of the elastomer’s viscoelastic behavior to quantify parameter uncertainty motivates the use of Markov Chain Monte Carlo (MCMC) methods. Since MCMC is a sampling based method, requiring many model evaluations, efficient estimation of the fractional derivative operator is crucial. In this paper, we demonstrate the effectiveness of using quadrature techniques to approximate the Riemann–Liouville definition for fractional derivatives in the context of estimating the uncertainty of a nonlinear viscoelastic model. We also demonstrate the use of parameter subset selection techniques to isolate parameters that are identifiable in the sense that they are uniquely determined by measured data. For those identifiable parameters, we employ Bayesian inference to compute posterior distributions for parameters. Finally, we propagate parameter uncertainties through the models to compute prediction intervals for quantities of interest.

Going beyond mean effect size: Presenting prediction intervals for on-farm network trial analyses

Abstract The aim of on-farm research is to identify and test a new technology, product or management practice (e.g. more efficient seeding rate, enhanced row spacing, better disease management treatment, etc.) suited to local conditions by comparing it to a standard farmer practice across several farmers’ fields. Typically, each trial includes two treatments (new practice vs. standard control practice) replicated at least three times in each field. The statistical analysis of yield data collected in such trials provides growers with useful information about the effectiveness of the tested farming practice on crop productivity and its uncertainty. We used a random-effects model to i) estimate the performance of a treatment compared to a control in individual trials, ii) estimate the overall mean yield response across all trials, iii) compute prediction intervals describing a range of plausible yield response for a new (out-of-sample) field at the trial level, and iv) compute the probability that the tested management practice will be ineffective in a new field. We used frequentist (classical) and Bayesian approaches for data collected in 26 on-farm trial categories managed by the Iowa Soybean Association. Depending on the level of between-trial variability, we found that prediction intervals were 2.2–12.1 times larger than confidence intervals for the estimated mean yield responses for all tested management practices. We conclude that prediction intervals should be systematically reported to provide additional information about future trials or experiments with associated uncertainties. Nevertheless, prediction intervals should be interpreted with caution when the between-trial variance is small. Using prediction intervals and, when appropriate, the probability of ineffective treatment will prevent farmers from overoptimistic expectations that a significant effect at the overall population level will lead with high certainty to a yield gain on their own farms.

A general framework for prediction in penalized regression

There are two main approaches to carrying out prediction in the context of penalized regression: with low-rank basis and penalties or through the smooth mixed models. In this article, we give further insight in the case of P-splines showing the influence of the penalty on the prediction. In the context of mixed models, we can connect the new predicted values to the observed values through a joint normal distribution, which allows us to compute prediction intervals. In this work, we propose an alternative approach, called the extended mixed model approach, that allows us to fit and predict data simultaneously. The methodology is illustrated with two real datasets, one of them on aboveground biomass and the other on monthly sulphur dioxide ([Formula: see text]) levels in a selection of monitoring sites in Europe.

The role and value of replication in empirical software engineering results

ContextConcerns have been raised from many quarters regarding the reliability of empirical research findings and this includes software engineering. Replication has been proposed as an important means of increasing confidence. ObjectiveWe aim to better understand the value of replication studies, the level of confirmation between replication and original studies, what confirmation means in a statistical sense and what factors modify this relationship. MethodWe perform a systematic review to identify relevant replication experimental studies in the areas of (i) software project effort prediction and (ii) pair programming. Where sufficient details are provided we compute prediction intervals. ResultsOur review locates 28 unique articles that describe replications of 35 original studies that address 75 research questions. Of these 10 are external, 15 internal and 3 internal-same-article replications. The odds ratio of internal to external (conducted by independent researchers) replications of obtaining a ‘confirmatory’ result is 8.64. We also found incomplete reporting hampered our ability to extract estimates of effect sizes. Where we are able to compute replication prediction intervals these were surprisingly large. ConclusionWe show that there is substantial evidence to suggest that current approaches to empirical replications are highly problematic. There is a consensus that replications are important, but there is a need for better reporting of both original and replicated studies. Given the low power and incomplete reporting of many original studies, it can be unclear the extent to which a replication is confirmatory and to what extent it yields additional knowledge to the software engineering community. We recommend attention is switched from replication research to meta-analysis.

Germany's Bundesliga: Does Money Score Goals?

During one of the most nerve-wracking football matches of the 2012–2013 Bundesliga season, life-long friends Franz Dully and Max Vogel begin arguing about whether the wealth of a football club determines its success during the season. In order to disprove Vogel's claim that “money scores goals,” Dully must analyze the Bundesliga's current market values, points earned, and mid-season leader data.After analyzing the case, students will be able to compute prediction intervals, develop regression models, and interpret data. The development of the regression models asks students to choose the relevant set of independent variables, as well as determine an appropriate functional form for the regression equation. The models derived have to be evaluated as well as compared to one another. Further, the students have to interpret the quantitative findings in the context of the application.

Calculation of solar irradiation prediction intervals combining volatility and kernel density estimates

In order to integrate solar energy into the grid it is important to predict the solar radiation accurately, where forecast errors can lead to significant costs. Recently, the increasing statistical approaches that cope with this problem is yielding a prolific literature. In general terms, the main research discussion is centred on selecting the “best” forecasting technique in accuracy terms. However, the need of the users of such forecasts require, apart from point forecasts, information about the variability of such forecast to compute prediction intervals. In this work, we will analyze kernel density estimation approaches, volatility forecasting models and combination of both of them in order to improve the prediction intervals performance. The results show that an optimal combination in terms of prediction interval statistical tests can achieve the desired confidence level with a lower average interval width. Data from a facility located in Spain are used to illustrate our methodology.

A derivation of prediction intervals for gamma regression

ABSTRACTGamma regression (GR) is a member of generalized liner models and often used when the phenomenon under study is skewed and the mean is proportional to the standard deviation. It can find applications in several areas such as life-testing problems, forecasting cancer incidences, weather extremes and quality control. Also it is a natural candidate when modelling the variance and it has been increasingly used over the past decade. When appropriate, studies encouraged its use over log transformation, Lewis et al. [Examples of designed experiments with nonnormal responses. J Qual Technol. 2001;33:265–278] and Hardin and Hilbe [Generalized linear models and extensions, 2nd ed. Stata Press; 2007]. Diagnostic tools and various inferential procedures such as confidence intervals and hypothesis testing were developed for GR models. An exception is prediction intervals for a future value of the process understudy. They are rarely discussed despite of its apparent importance for a practitioner. In this study I describe a simple but yet an effective procedure to compute prediction intervals for Gamma models. Four real data sets coming from medical and industrial experiments are studied to demonstrate the implementations and the usefulness of the suggested intervals. Extensive simulations are performed to investigate their advertised coverage under a wide range of sample sizes and several model designs showing consistency and satisfactory performance even for small samples.

Combined Analysis of Accelerated Fixed Stress Lab and Varying Stress Field Data

ABSTRACTThis article performs a combined analysis of lab and field data. The lab data are obtained by testing the specimens at high but fixed temperatures. The specimens in the field are subjected to a varying temperature profile. We use an accelerated aging model for the lab data and a cumulative damage version of this model for the field data. A Bayesian analysis provides the necessary quantities to compute prediction intervals for specimens in the field for many years into the future.

Predicting hourly air pollutant levels using artificial neural networks coupled with uncertainty analysis by Monte Carlo simulations

Recent progress in developing artificial neural network (ANN) metamodels has paved the way for reliable use of these models in the prediction of air pollutant concentrations in urban atmosphere. However, improvement of prediction performance, proper selection of input parameters and model architecture, and quantification of model uncertainties remain key challenges to their practical use. This study has three main objectives: to select an ensemble of input parameters for ANN metamodels consisting of meteorological variables that are predictable by conventional weather forecast models and variables that properly describe the complex nature of pollutant source conditions in a major city, to optimize the ANN models to achieve the most accurate hourly prediction for a case study (city of Tehran), and to examine a methodology to analyze uncertainties based on ANN and Monte Carlo simulations (MCS). In the current study, the ANNs were constructed to predict criteria pollutants of nitrogen oxides (NOx), nitrogen dioxide (NO2), nitrogen monoxide (NO), ozone (O3), carbon monoxide (CO), and particulate matter with aerodynamic diameter of less than 10 μm (PM10) in Tehran based on the data collected at a monitoring station in the densely populated central area of the city. The best combination of input variables was comprehensively investigated taking into account the predictability of meteorological input variables and the study of model performance, correlation coefficients, and spectral analysis. Among numerous meteorological variables, wind speed, air temperature, relative humidity and wind direction were chosen as input variables for the ANN models. The complex nature of pollutant source conditions was reflected through the use of hour of the day and month of the year as input variables and the development of different models for each day of the week. After that, ANN models were constructed and validated, and a methodology of computing prediction intervals (PI) and probability of exceeding air quality thresholds was developed by combining ANNs and MCSs based on Latin Hypercube Sampling (LHS). The results showed that proper ANN models can be used as reliable metamodels for the prediction of hourly air pollutants in urban environments. High correlations were obtained with R (2) of more than 0.82 between modeled and observed hourly pollutant levels for CO, NOx, NO2, NO, and PM10. However, predicted O3 levels were less accurate. The combined use of ANNs and MCSs seems very promising in analyzing air pollution prediction uncertainties. Replacing deterministic predictions with probabilistic PIs can enhance the reliability of ANN models and provide a means of quantifying prediction uncertainties.

A comparison of methods for estimating prediction intervals in NIR spectroscopy: Size matters

In this article we demonstrate that, when evaluating a method for determining prediction intervals, interval size matters more than coverage because the latter can be fixed at a chosen confidence level with good reliability. To achieve the desired coverage, we employ a simple non-parametric method to compute prediction intervals by calibrating estimated prediction errors, and we extend the basic method with a continuum correction to deal with small data sets. In our experiments on a collection of several NIR data sets, we evaluate several existing methods of computing prediction intervals for partial least-squares (PLS) regression. Our results show that, when coverage is fixed at a chosen confidence level, and the number of PLS components is selected to minimize squared error of point estimates, interval estimation based on the classic ordinary least-squares method produces the narrowest intervals, outperforming the U-deviation method and linearization, regardless of the confidence level that is chosen.

Improved Prediction Intervals and Distribution Functions

Abstract. The plug‐in solution is usually not entirely adequate for computing prediction intervals, as their coverage probability may differ substantially from the nominal value. Prediction intervals with improved coverage probability can be defined by adjusting the plug‐in ones, using rather complicated asymptotic procedures or suitable simulation techniques. Other approaches are based on the concept of predictive likelihood for a future random variable. The contribution of this paper is the definition of a relatively simple predictive distribution function giving improved prediction intervals. This distribution function is specified as a first‐order unbiased modification of the plug‐in predictive distribution function based on the constrained maximum likelihood estimator. Applications of the results to the Gaussian and the generalized extreme‐value distributions are presented.

Nonparametric Forecasting in Time Series—A Comparative Study

The problem of predicting a future value of a time series is considered in this article. If the series follows a stationary Markov process, this can be done by nonparametric estimation of the autoregression function. Two forecasting algorithms are introduced. They only differ in the nonparametric kernel-type estimator used: the Nadaraya-Watson estimator and the local linear estimator. There are three major issues in the implementation of these algorithms: selection of the autoregressor variables, smoothing parameter selection, and computing prediction intervals. These have been tackled using recent techniques borrowed from the nonparametric regression estimation literature under dependence. The performance of these nonparametric algorithms has been studied by applying them to a collection of 43 well-known time series. Their results have been compared to those obtained using classical Box-Jenkins methods. Finally, the practical behavior of the methods is also illustrated by a detailed analysis of two data sets.

Triplet Ultrasound Growth Parameters

To create ultrasound growth curves for normal growth of fetal triplets using statistical methodology that properly accounts for similarities of growth of fetuses within a mother as well as repeated measurements over time for each fetus. In this longitudinal study, all triplet pregnancies managed at a single tertiary center from 1992-2004 were reviewed. Fetuses with major anomalies, prior selective reduction, or fetal demise were excluded. Data from early and late gestation in which there were fewer than 30 fetal measurements available for analysis were excluded. We used multilevel models to account for variation in growth within a single fetus over time, variations in growth between multiple fetuses within a single mother, and variations in fetal growth between mothers. Medians (50th), 10th, and 90th percentiles were estimated by the creation of multiple quadratic growth models from bootstrap samples adapting a previously published method to compute prediction intervals. Estimated fetal weight was derived from Hadlock's formula. One hundred fifty triplet pregnancies were identified. Twenty-seven pregnancies were excluded for the following reasons: missing records (23), fetal demise (3), and fetal anomaly (1). The study group consisted of 123 pregnancies. The gestational age range was restricted to 14-34 weeks. Figures and tables were developed showing medians, 10th and 90th percentiles for estimated fetal weight, femur length, biparietal diameter, abdominal circumference, and head circumference. Growth curves for triplet pregnancies were derived. These may be useful for identification of abnormal growth in triplet fetuses. III.

Bootstrap prediction intervals for power-transformed time series

In this paper, we propose a bootstrap procedure to construct prediction intervals for future values of a variable after a linear ARIMA model has been fitted to its power transformation. The procedure is easy to implement and provides a useful tool in empirical applications given that it is often the case that, for example, the log-transformation is modeled when the variable of interest for prediction is the original one. The advantages over existing methods for computing prediction intervals of power-transformed time series are that the proposed bootstrap intervals incorporate the variability due to parameter estimation and do not rely on distributional assumptions neither on the original variable nor on the transformed one. We derive the asymptotic distribution and show the good behavior of the bootstrap approach versus alternative procedures by means of Monte Carlo experiments. Finally, the procedure is illustrated by analyzing three real-time series data sets.

Time Series Based Errors and Empirical Errors in Fertility Forecasts in the Nordic Countries*

Summary We use ARCH time series models to derive model based prediction intervals for the Total Fertility Rate (TFR) in Norway, Sweden, Finland, and Denmark up to 2050. For the short term (5–10 yrs), expected TFR‐errors are compared with empirical forecast errors observed in historical population forecasts prepared by the statistical agencies in these countries since 1969. Medium‐term and long‐term (up to 50 years) errors are compared with error patterns based on so‐called naïve forecasts, i.e. forecasts that assume that recently observed TFR‐levels also apply for the future.