Width Of Prediction Intervals Research Articles

Geographic and temporal validity of prediction models: different approaches were useful to examine model performance

ObjectiveValidation of clinical prediction models traditionally refers to the assessment of model performance in new patients. We studied different approaches to geographic and temporal validation in the setting of multicenter data from two time periods. Study Design and SettingWe illustrated different analytic methods for validation using a sample of 14,857 patients hospitalized with heart failure at 90 hospitals in two distinct time periods. Bootstrap resampling was used to assess internal validity. Meta-analytic methods were used to assess geographic transportability. Each hospital was used once as a validation sample, with the remaining hospitals used for model derivation. Hospital-specific estimates of discrimination (c-statistic) and calibration (calibration intercepts and slopes) were pooled using random-effects meta-analysis methods. I2 statistics and prediction interval width quantified geographic transportability. Temporal transportability was assessed using patients from the earlier period for model derivation and patients from the later period for model validation. ResultsEstimates of reproducibility, pooled hospital-specific performance, and temporal transportability were on average very similar, with c-statistics of 0.75. Between-hospital variation was moderate according to I2 statistics and prediction intervals for c-statistics. ConclusionThis study illustrates how performance of prediction models can be assessed in settings with multicenter data at different time periods.

Lower Upper Bound Estimation Method Considering Symmetry for Construction of Prediction Intervals in Flood Forecasting

It is widely accepted that Prediction Interval (PI) can provide more accurate and precise information than deterministic forecast when the uncertainty level increases in flood forecasting. Coverage probability and PI width are two main criteria used to assess the constructed PI, rarely has there been an index to quantify the symmetry between target value and PI. This study extends a newly proposed PI estimation method called Lower Upper Bound Estimation (LUBE) method, which adopts an Artificial Neural Network (ANN) with two outputs to directly generate the upper and lower bounds of PI without making any assumption about the data distribution. A new Prediction Interval Symmetry (PIS) index is introduced and a new objective function is developed for the comprehensive evaluation of PI considering their coverage probability, width and symmetry. Furthermore, Shuffled Complex Evolution algorithm (SCE-UA) is used to minimize the objective function and optimize ANN parameters in the LUBE method. The proposed method is applied to a real world flood forecasting case study of the upper Yangtze River Watershed. The result shows that the SCE-UA based LUBE method with new objective function is very efficient, meanwhile, the midpoint forecasting of the PI obtains excellent performance by evidently improving the symmetry of PI.

Assessing uncertainty in soil organic carbon modeling across a highly heterogeneous landscape

To understand if soil carbon acts as a sink or source in the global carbon cycle it is not only important to make reliable estimates, but also determine upper and lower prediction bounds through uncertainty analysis that represent best and worst case conditions. In this study, the Bayesian geostatistics was applied to assess the uncertainty associated with the predictive models of SOC (top soils) in a large region — Florida, USA. Results showed that the Bayesian estimates of model parameters were comparable to the conventional geostatistical methods especially the restricted maximum likelihood (REML). The Bayesian prediction uncertainty assessment was encouragingly accurate based on the validation of 50 and 95% prediction intervals with the validation dataset. Generally, the width of prediction intervals increased with the posterior mean SOC predictions — large prediction intervals were found in the Everglades Agricultural Area (Histosols) and the wetland areas in the Suwannee River Basin. The Bayesian constant mean model (high model inadequacy) had marked prediction uncertainty which was reduced by accounting for the effects of environmental covariates in the Bayesian linear trend model (low model inadequacy), indicating that model inadequacy had a negative impact on prediction uncertainty. Analyses of factors impacting SOC prediction uncertainty suggest that effects that explained more of the SOC variance contributed more uncertainty to the SOC prediction. These findings are critical to quantify SOC stocks in the southeastern USA where a heterogeneous mosaic of high and low carbon in soils occurs ranging from 0.45 to 34.15kgm−2. Although this study considers only the topsoil, the results are valuable for global carbon cycling research. The uncertainty of SOC predictions not only enables identification of “hot” and “cold” spots in a landscape to mitigate and adapt to global climate change, but also informs scenario assessment to imagine possible carbon- rich, -neutral, and -poor futures.

An interval type-2 fuzzy logic system-based method for prediction interval construction

This paper introduces a new non-parametric method for uncertainty quantification through construction of prediction intervals (PIs). The method takes the left and right end points of the type-reduced set of an interval type-2 fuzzy logic system (IT2FLS) model as the lower and upper bounds of a PI. No assumption is made in regard to the data distribution, behaviour, and patterns when developing intervals. A training method is proposed to link the confidence level (CL) concept of PIs to the intervals generated by IT2FLS models. The new PI-based training algorithm not only ensures that PIs constructed using IT2FLS models satisfy the CL requirements, but also reduces widths of PIs and generates practically informative PIs. Proper adjustment of parameters of IT2FLSs is performed through the minimization of a PI-based objective function. A metaheuristic method is applied for minimization of the non-linear non-differentiable cost function. Performance of the proposed method is examined for seven synthetic and real world benchmark case studies with homogenous and heterogeneous noise. The demonstrated results indicate that the proposed method is capable of generating high quality PIs. Comparative studies also show that the performance of the proposed method is equal to or better than traditional neural network-based methods for construction of PIs in more than 90% of cases. The superiority is more evident for the case of data with a heterogeneous noise.

Prediction intervals for the Gaussian autoregressive processes following the unit root tests

Recently, Diebold and Kilian (9), Niwitpong (21,22), and Niwitpong and Panichkitkosolkul (23) indicated that the preliminary unit root test was a useful tool for improving the accuracy of a one-step-ahead predictor and prediction interval for an AR(1) process. This paper extends these mentioned concepts to the prediction interval for the Gaussian autoregressive processes. We propose the methods to construct the simple prediction interval based on the residual model, PIa, and the prediction intervals following the unit root tests, PIfi. The unit root tests applied in this paper consisted of the augmented Dickey-Fuller test, the Phillips-Perron test, and the Elliott-Rothenberg-Stock test. In addition, an expression of the coverage probability is derived and we found that the structure of the coverage probability is independent from the parameter of a random error, but it is a function of the autoregressive parameters only. The coverage probability and average width of prediction intervals are compared through Monte Carlo simulation studies. Simulation results have shown that the proposed prediction intervals have minimum coverage probabilities 0.95 for almost all situations. Furthermore, the average widths of prediction intervals PIfi are shorter than that of a prediction interval PIa when the first-order autoregressive parameter value approaches one. A comparison of the proposed methods is also illustrated by using an empirical application.

Improved closed-form prediction intervals for binomial and Poisson distributions

Abstract The problems of constructing prediction intervals for the binomial and Poisson distributions are considered. Available approximate, exact and conditional methods for both distributions are reviewed and compared. Simple approximate prediction intervals based on the joint distribution of the past samples and the future sample are proposed. Exact coverage studies and expected widths of prediction intervals show that the new prediction intervals are comparable to or better than the available ones in most cases. The methods are illustrated using two practical examples.

Optimization of a product performance using mixture experiments including process variables

This article presents a case study of a chemical compound used in the delay mechanism to start a rocket engine. The compound consists in a three-component mixture. Besides the components proportions, two process variables are considered. The aim of the study is to investigate the mix components proportions and the levels of process variables that set the expected delay time as close as possible to the target value and, at the same time, minimize the width of prediction interval for the response. A linear regression model with normal responses was fitted. Through the model developed, the optimal components proportions and the levels of the process variables were determined. For the model selection, the use of the backward method with an information criterion proved to be efficient in the case under study.

Assessing Uncertainty in Mass Balance Calculation of River Nonpoint Source Loads

Several sources of uncertainty are considered in using field data to perform mass balance calculations to estimate nonpoint source (NPS) loads of total dissolved solids (TDS) and selenium (Se) to two reaches along the Lower Arkansas River in Colorado. This approach renders stochastic models of the mass balance equations for each river reach, where the input variables and their associated parameters are treated as random variables described by probability distributions. A data set collected in an intensive field effort conducted over several years is used in developing the models. Monte Carlo simulation solves the stochastic mass balance equations to describe distributions of possible values of the NPS loads. Results indicate that uncertainty in these calculated loads is sizeable. Annual average coefficient of variation (CV) in calculated total NPS TDS loads for sample periods within the two reaches ranges between 0.15 and 2.76, averaging about 1.1. For the Se mass balance along the downstream reach, the average CV in calculated loads is 0.23. The 90% prediction interval width for total NPS TDS load averaged over the 58 sample periods along the upstream reach is 13,691 (kg/day)/km, compared to an overall average mean load of 8,383 (kg/day)/km. It is 8,545 (kg/day)/km averaged over the 61 sample periods downstream, compared to an overall average mean load of 11,183 (kg/day)/km. For the Se load, the overall average 90% prediction interval width is also substantial compared to the overall average mean: 0.028 (kg/day)/km compared to 0.038 (kg/day)/km. Change in stored solute mass within the river over sample periods is found to be a major contributing factor to the calculation of NPS loads. Also, sensitivity analyses are performed that yield information on the relative influence that the degree of uncertainty in each random parameter has on the uncertainty in the calculated solute loads.

Predicting event times in clinical trials when randomization is masked and blocked

Timing for interim or final analysis of data in an event-based trial is often determined by the accrual of events during the study. Existing Bayesian methods may be used to predict the date of the landmark event using observed enrollment, event, and loss times when treatment arm information is masked. For event-based trials with a blocked randomization, knowledge of blocks in which patients are enrolled can provide additional information to improve predictions versus models that only assume a known treatment allocation proportion. We therefore propose to incorporate blocking information into existing methods for prediction. We derive a latent variable (LV) extension of a mixture model used in Donovan JM, Elliott MR, Heitjan DF. Predicting Event Times in Clinical Trials When Treatment Arm is Masked. J Biopharmaceut Stat 2006; 16:343-56 to incorporate block randomization (constrained LV) for predicting the landmark event and compare this model with (a) methods where blocking information is ignored (unconstrained LV), and (b) methods assuming a single population (SP). Comparison of the constrained and unconstrained LV models in our application shows that the constrained LV model has narrower prediction intervals. Simulation studies show that the constrained LV model can have better coverage probabilities for the prediction intervals than SP models if a treatment effect is present, and prediction intervals from the constrained LV model are narrower than those for the unconstrained LV model. These differences varied by block size and prediction time. We have limited focus to the exponential model for events. Coverage for the LV models may be somewhat reduced if no treatment effect is present. Extra information provided by knowledge of blocking can be used to decrease prediction interval width versus the unconstrained LV model, while providing better coverage properties than the SP model if a treatment effect is present.

Precision, bias, and uncertainty for state population forecasts: an exploratory analysis of time series models

Many researchers have used time series models to construct population forecasts and prediction intervals at the national level, but few have evaluated the accuracy of their forecasts or the out-of-sample validity of their prediction intervals. Fewer still have developed models for subnational areas. In this study, we develop and evaluate six ARIMA time series models for states in the United States. Using annual population estimates from 1900 to 2000 and a variety of launch years, base periods, and forecast horizons, we construct population forecasts for four states chosen to reflect a range of population size and growth rate characteristics. We compare these forecasts with population counts for the corresponding years and find precision, bias, and the width of prediction intervals to vary by state, launch year, model specification, base period, and forecast horizon. Furthermore, we find that prediction intervals based on some ARIMA models provide relatively accurate forecasts of the distribution of future population counts but prediction intervals based on other models do not. We conclude that there is some basis for optimism regarding the possibility that ARIMA models might be able to produce realistic prediction intervals to accompany population forecasts, but a great deal of work remains to be done before we can draw any firm conclusions.

A validation/evaluation of five postoperative prognostic tools for prostate cancer

5117 Background: Prognostic nomograms, scoring tools, and risk tables are rapidly accumulating in the prostate cancer (PCa) literature. It is not always clear to whom these tools should apply and how well they predict the outcomes they were designed to forecast. There is need for independent evaluation of these tools. Methods: We used the Mayo Clinic Radical Prostatectomy Registry, a prospective database of radical prostatectomy (RP) outcomes, to assess tools designed to predict RP outcomes. The validation set included 13,313 RP patients from 1990–2005. There were 3,256 PSA failures, 566 metastases, and 1,599 deaths (301 were due to PCa). We assessed the discrimination, calibration and overall accuracy of prediction tools identified through a structured Pubmed search. Results: Tools varied greatly in terms of complexity, width of prediction interval, and method of presentation. Several tools included non-standard variables and were therefore unevaluable, despite an extensive dataset, leaving the 1999 and 2005 Kattan nomograms, the 1998 and 2001 CDPR scores, and the 2001 GPSM score for analysis. Discrimination (quantitated by the c index) was better for PCa-specific survival and metastases than for PSA failure ( Table ). Kaplan-Meier plots demonstrated clustering of risk groups in most tools, most severely in the higher risk groups of the Kattan nomograms. The calibration plots of most tools (excepting GPSM) had a serious discordance between observed and predicted outcomes in the lower probability ranges. This meant that most tools gravely overestimated the probability of RP failure in high-risk patients, by up to 4- fold. Conclusions: The tools showed moderate discriminatory ability for PSA failure but performed much better for non-surrogate outcomes. Most tools (excepting GPSM) were miscalibrated in high risk patients and dramatically underestimated the efficacy of surgery in this cohort. Prognostic tools may not be as accurate as previously reported. [Table: see text] No significant financial relationships to disclose.

A comparison of low volume, Draize and in vitro eye irritation test data. III. Surfactant-based formulations

The third phase in a series of investigations of the relationship between low volume eye test (LVET) data, Draize eye irritation test data, and comparable data from in vitro assay protocols is presented. These investigations utilize Draize eye test and in vitro endpoint data generated previously as part of the CTFA Evaluation of Alternatives Program. LVET data were generated de novo using the same 25 representative surfactant-based personal-care formulations. In general, these formulations were minimally to moderately irritating. The linear correlation between maximum average score as determined by the Draize test (MAS) and the LVET (LVET-MAS) was 0.87; LVET-MAS values were typically about 30% lower then corresponding MAS values. Comparison of in vitro assay performance with that of the LVET was determined by statistical analysis of the relationship between LVET-MAS and in vitro endpoint. Regression modelling was the primary means of enabling such a comparison, the objective being to predict LVET-MAS for a given test material (and to place upper and lower prediction bounds on the range in which the LVET-MAS is anticipated to fall with high probability) based on observation of an in vitro score for that material. The degree of 95% confidence in prediction is quantified in terms of the relative widths of prediction intervals constructed about the fitted regression curves. Twenty in vitro endpoints were shown to have the greatest agreement with the LVET (these endpoints included those with low discordance rates relative to the Draize test) and were therefore selected for regression modelling. Although prediction interval widths tended to be narrower when predicting LVET-MAS compared with predicting MAS, the confidence with which the selected in vitro endpoints predicted both LVET-MAS and MAS for surfactant-based formulations was greatest when values were close to the lower or upper limits of the observed irritation range (i.e. 95% prediction interval widths were most narrow in these areas). Overall precision of LVET-MAS prediction for surfactant-based formulations was similar to that previously reported for hydroalcoholic formulations and considerably better than was reported for oil/water emulsions.

A Comparison of Low Volume, Draize and in vitro eye irritation test data. II. Oil/water emulsions

The second phase in a series of investigations of the relationship between low volume eye test (LVET) data, Draize eye irritation test data, and comparable data from in vitro eye irritation test protocols is presented. These investigations utilize Draize eye test and in vitro endpoint data generated previously as part of the CTFA Evaluation of Alternatives Program. LVET data were generated de novo using the same 18 representative oil/water based personal-care formulations. In general, these formulations were minimally to mildly irritating; only three were classified as moderate eye irritants. The linear correlation between maximum average score as determined by the Draize test (MAS) and the LVET (LVET-MAS) was 0.85; LVET-MAS values were typically about half the corresponding MAS values. Comparison of in vitro assay performance with that of the LVET was determined by statistical analysis of the relationship between LVET-MAS and each in vitro endpoint. Regression modelling was the primary means of enabling such a comparison, the objective being to predict LVET-MAS for a given test material (and to place upper and lower 95% prediction bounds on the range in which the LVET-MAS is anticipated to fall with high probability) based on observation of an in vitro score for that material. The degree of confidence in prediction is quantified in terms of the relative widths of prediction intervals constructed about the fitted regression curves. Sixteen endpoints were shown to have the greatest agreement with the LVET (all but two were selected for modelling when compared with the Draize procedure). While the lower maximum average scores values (compared with the Draize test) in the LVET led to lower variability in LVET-MAS compared to MAS, the upper and lower bounds on predicted LVET-MAS values conditional on observed in vitro scores were still wide. Because there was overlap in the range of scores determined by the prediction bounds for many formulations, each of the selected endpoints was frequently unable to distinguish between test formulations in terms of statistically different predicted LVET-MAS values. In summary, none of the in vitro endpoints evaluated were able to reliably predict values of LVET-MAS among the set of oil/water emulsions considered here.

Comparison of low-volume, draize and in vitro eye irritation test data. I. Hydroalcoholic formulations

The first phase in a series of investigations of the relationship between low-volume eye test (LVET) data, Draize eye irritation test data, and comparable data from 25 in vitro assay protocols is presented. These investigations utilize Draize eye test and in vitro assay data generated previously as part of the Cosmetic, Toiletry and Fragrance Association (CTFA) Evaluation of Alternatives Program. LVET data were generated de novo using the same 10 representative hydroalcoholic personal-care formulations. The linear correlation between maximum average score (MAS) as determined by the Draize test and the LVET (LVET-MAS) was 0.93. Comparison of in vitro assay performance with that of the LVET was determined by statistical analysis of the relationship between LVET-MAS and in vitro endpoint. As in the CTFA program, regression modelling is the primary means of enabling such a comparison. The objective is to predict LVET-MAS for a given test material (and to place upper and lower prediction interval bounds in the range in which the LVET-MAS is anticipated to fall with high probability) conditional on observing an in vitro assay score for that material. The degree of confidence in prediction is quantified in terms of the relative widths of prediction intervals constructed about the fitted regression curves. Four assays [EYTEX™ MPA (membrane partition assay), HET-CAM (hen's egg test-chorioallantoic membrane HET-CAM) I, neutral red release and HET-CAM II] were shown to have the greatest agreement with the LVET. These assays were also among those with low discordance rates relative to the Draize test. Prediction of LVET-MAS values from experimentally determined in vitro scores was more accurate for hydroalcoholic formulations with lower rather than higher irritancy potential.

The CFTA evaluation of alternatives program: An evaluation of in vitro alternatives to the draize primary eye irritation test. (Phase III) surfactant-based formulations

The CTFA Evaluation of Alternatives Program is an evaluation of the relationship between data from the Draize primary eye irritation test and comparable data from a selection of promising in vitro eye irritation tests. In Phase III, data from the Draize test and 41 in vitro endpoints on 25 representative surfactant-based personal care formulations were compared. As in Phase I and Phase II, regression modelling of the relationship between maximum average Draize score (MAS) and in vitro endpoint was the primary approach adopted for evaluating in vitro assay performance. The degree of confidence in prediction of MAS for a given in vitro endpoint is quantified in terms of the relative widths of prediction intervals constructed about the fitted regression curve. Prediction intervals reflect not only the error attributed to the model but also the material-specific components of variation in both the Draize and the in vitro assays. Among the in vitro assays selected for regression modelling in Phase III, the relationship between MAS and in vitro score was relatively well defined. The prediction bounds on MAS were most narrow for materials at the lower or upper end of the effective irritation range (MAS = 0–45), where variability in MAS was smallest. Thus, the confidence with which the MAS of surfactant-based formulations is predicted is greatest when MAS approaches zero or when MAS approaches 45 (no comment is made on prediction of MAS > 45 since extrapolation beyond the range of observed data is not possible). No single in vitro endpoint was found to exhibit relative superiority with regard to prediction of MAS. Variability associated with Draize test outcome (e.g. in MAS values) must be considered in any future comparisons of in vivo and in vitro test results if the purpose is to predict in vivo response using in vitro data.

The CTFA evaluation of alternatives program: An evaluation of in vitro alternatives to the draize primary eye irritation test. (Phase II) oil/water emulsions

The Cosmetic, Toiletry and Fragrance Association (CTFA) Evaluation of Alternatives Program is an evaluation of the relationship between Draize ocular safety test data and comparable data from a selection of in vitro tests. In Phase II, 18 representative oil/water-based personal-care formulations were subjected to the Draize primary eye safety test and 30 in vitro assay protocols (14 different types of in vitro endpoints were evaluated; the remainder were protocol variations). Correlation of in vitro with in vivo data was evaluated using analysis of sensitivity/specificity and statistical analysis of the relationship between maximum average Draize score (MAS) and in vitro endpoint. Regression modelling is the primary approach adopted in the CTFA Program for evaluating in vitro assay performance. The objective of regression analysis is to predict MAS for a given test material (and to place upper and lower prediction interval bounds on the range in which the MAS is anticipated to fall with high probability) conditional on observing an in vitro assay score for that material. The degree of confidence in prediction is quantified in terms of the relative widths of prediction intervals constructed about the fitted regression curves: the narrower the prediction interval, the more predictive of the Draize score is the in vitro test result. 16 assays were shown to have the greatest agreement with the Draize procedure and were therefore selected for regression analysis. Based on the magnitude of the 95% prediction bounds of each of the 16 selected assays over the range of test data, it may be inferred that prediction of MAS values from experimentally determined in vitro scores is more accurate for oil/water-based formulations with lower rather than higher irritancy potential. The assays selected for modelling in Phase II generally exhibited weaker relationships with MAS than those selected in Phase I (evaluated using hydroalcoholic formulations), even though several assays were common to both Phases.

Probabilistic forecasts. The case of population projections

Probabilistic models for assessing the random error and the bias of population forecasts are discussed. The predictions of the models reflect the “expert opinion” about the future vital rates, and the widths of the prediction intervals are calibrated using the performance of past forecasts. An application with the U.S. data is presented and comparisons with the situation in Nordic countries are made. It is found that the interval between the high and the low birth forecasts of the national statistical agencies in the United States, Finland, and Norway are narrower than our 67% prediction intervals. The U.S. intervals are by far the narrowest.