Model Error Variance Research Articles

Linear mixed model approach to network meta-analysis for continuous outcomes in periodontal research.

Analysing continuous outcomes for network meta-analysis by means of linear mixed models is a great challenge, as it requires statistical software packages to specify special patterns of model error variance and covariance structure. This article demonstrates a non-Bayesian approach to network meta-analysis for continuous outcomes in periodontal research with a special focus on the adjustment of data dependency. Seventeen studies on guided tissue regeneration were used to illustrate how the proposed linear mixed models for network meta-analysis of continuous outcomes. Arm-based network meta-analysis use treatment arms from each study as the unit of analysis; when patients are randomly assigned to each arm, data are deemed independent and therefore no adjustment is required for multi-arm trials. Trial-based network meta-analysis use treatment contrasts as the unit of analysis, and therefore treatment contrasts within a multi-arm trial are not independent. This data dependency occurs also in split-mouth studies, and adjustments for data dependency are therefore required. Arm-based analysis is the preferred approach to network meta-analysis, when all included studies use the parallel group design and some compare more than two treatment arms. When included studies used designs that yield dependent data, the trial-based analysis is the preferred approach.

Small Sample Behaviors of the Delete-<i>d</i> Cross Validation Statistic

Built upon an iterative process of resampling without replacement and out-of-sample prediction, the delete-d cross validation statistic CV(d) provides a robust estimate of forecast error variance. To compute CV(d), a dataset consisting of n observations of predictor and response values is systematically and repeatedly partitioned (split) into subsets of size n – d (used for model training) and d (used for model testing). Two aspects of CV(d) are explored in this paper. First, estimates for the unknown expected value E[CV(d)] are simulated in an OLS linear regression setting. Results suggest general formulas for E[CV(d)] dependent on σ2 (“true” model error variance), n – d (training set size), and p (number of predictors in the model). The conjectured E[CV(d)] formulas are connected back to theory and generalized. The formulas break down at the two largest allowable d values (d = n – p – 1 and d = n – p, the 1 and 0 degrees of freedom cases), and numerical instabilities are observed at these points. An explanation for this distinct behavior remains an open question. For the second analysis, simulation is used to demonstrate how the previously established asymptotic conditions {d/n → 1 and n – d → ∞ as n → ∞} required for optimal linear model selection using CV(d) for model ranking are manifested in the smallest sample setting, using either independent or correlated candidate predictors.

Eliciting prior distributions for extra parameters in some generalized linear models

To elicit an informative prior distribution for a normal linear model or a gamma generalized linear model (GLM), expert opinion must be quantified about both the regression coefficients and the extra parameters of these models. The latter task has attracted comparatively little attention. In this article, we introduce two elicitation methods that aim to complete the prior structure of the normal and gamma GLMs. First, we develop a method of assessing a conjugate prior distribution for the error variance in normal linear models. The method quantifies an expert's opinions through assessments of a median and conditional medians. Second, we propose a novel method for eliciting a lognormal prior distribution for the scale parameter of gamma GLMs. Given the mean value of a gamma distributed response variable, the method is based on conditional quartile assessments. It can also be used to quantify an expert's opinion about the prior distribution for the shape parameter of any gamma random variable, if the mean of the distribution has been elicited or is assumed to be known. In the context of GLMs, the mean value is determined by the regression coefficients. Interactive graphics is the medium through which assessments for the two proposed methods are elicited. Examples illustrating use of the methods are given. Computer programs that implement both methods are available.

How well can we estimate error variance of satellite precipitation data around the world?

Providing error information associated with existing satellite precipitation estimates is crucial to advancing applications in hydrologic modeling. In this study, we present a method of estimating the square difference prediction of satellite precipitation (hereafter used synonymously with “error variance”) using regression model for three satellite precipitation products (3B42RT, CMORPH, and PERSIANN-CCS) using easily available geophysical features and satellite precipitation rate. Building on a suite of recent studies that have developed the error variance models, the goal of this work is to explore how well the method works around the world in diverse geophysical settings. Topography, climate, and seasons are considered as the governing factors to segregate the satellite precipitation uncertainty and fit a nonlinear regression equation as a function of satellite precipitation rate. The error variance models were tested on USA, Asia, Middle East, and Mediterranean region. Rain-gauge based precipitation product was used to validate the error variance of satellite precipitation products. The regression approach yielded good performance skill with high correlation between simulated and observed error variances. The correlation ranged from 0.46 to 0.98 during the independent validation period. In most cases (~85% of the scenarios), the correlation was higher than 0.72. The error variance models also captured the spatial distribution of observed error variance adequately for all study regions while producing unbiased residual error. The approach is promising for regions where missed precipitation is not a common occurrence in satellite precipitation estimation. Our study attests that transferability of model estimators (which help to estimate the error variance) from one region to another is practically possible by leveraging the similarity in geophysical features. Therefore, the quantitative picture of satellite precipitation error over ungauged regions can be discerned even in the absence of ground truth data.

Calibrating short‐range ensemble forecasts using estimated model error variances

This study investigates the impact of a calibration technique using estimated global model error variances on the performance of short‐range ensemble forecasts. The calibration technique is rather simple as it consists of adding a random white noise of zero mean and variance equal to estimated model error variances. Two calibrated ensemble forecasts are generated, one using a homogeneous calibration factor and another one using a heterogeneous calibration factor. Based on probabilistic scores, the qualities of these two ensemble forecasts are evaluated against a non‐calibrated ensemble and the operational ensemble prediction system of Météo‐France. The results show that this calibration technique significantly improves the reliability and, to a less extent, the resolution of ensemble forecasts. Better reliability results are found using a spatially heterogeneous calibration factor instead of using the homogeneous calibration factor at forecast ranges beyond 48 and 72 h.

Obsessive–compulsive symptoms in a normative Chinese sample of youth: Prevalence, symptom dimensions, and factor structure of the Leyton Obsessional Inventory—Child Version

BackgroundChinese adolescents face life stresses from multiple sources, with higher levels of stress predictive of adolescent mental health outcomes, including in the area of obsessive–compulsive disorders (OCD). Valid assessment of OCD among this age group is therefore a critical need in China. This study aims to standardise the Chinese version of the Leyton short version scale for adolescents of secondary schools in order to assess this condition. MethodsStratified randomly selected adolescents were selected from four high schools located in Beijing, China. The Chinese version of the Leyton scale was administered to 3221 secondary school students aged between 12 and 18 years. A high response rate was achieved, with 3185 adolescents responding to the survey (98.5 percent). Exploratory factor analysis (EFA) extracted four factors from the scale: compulsive thoughts, concerns of cleanliness, lucky number, repetitiveness and repeated checking. The four-factor structures were confirmed using Confirmatory Factor Analysis (CFA). ResultsOverall the four-factor structure had a good model fit and high levels of reliability for each individual dimension and reasonable content validity. Invariance analyses in unconstrained, factor loading, and error variance models demonstrated that the Leyton scale is invariant in relation to the presence or absence OCD, age and gender. Discriminant validity analysis demonstrated that the four-factor structure scale also had excellent ability to differentiate between OCD and non-OCD students, male and female students, and age groups. LimitationsThe dataset was a non-clinical sample of high school students, rather than a sample of individuals with OCD. Future research may examine symptom structure in clinical populations to assess whether this structure fits into both clinical and community population. ConclusionsThe structure derived from the Leyton short version scale in a non-clinical secondary school sample of adolescents, suggests that a four-factor solution can be utilised as a screening tool to assess adolescents׳ psychopathological symptoms in the area of OCD in mainland Chinese non-clinical secondary school students.

Estimation of Satellite Rainfall Error Variance Using Readily Available Geophysical Features

The present study addresses the estimation of error variance (mean square error, MSE) of three satellite rainfall products: i) Tropical Rainfall Measuring Mission (TRMM) Multi-satellite Precipitation Analysis (TMPA) product of 3B42RT; ii) Climate Prediction Center (CPC) Morph (CMORPH); and iii) Precipitation Estimation from Remotely Sensed Information using Artificial Neural Networks-Cloud Classification System (PERSIANN-CCS). Nonlinear regression model is used to fit the response variable (satellite rainfall error variance) with explanatory variable (satellite rainfall rate) by grouping them as function of three key geophysical features: topography, climate, and season. The results of the study suggest that the error variance of a rainfall product is strongly correlated with rainfall rate and can be expressed as a power-law function. The geophysical feature based error classification analysis helps in achieving superior accuracy for prognostic error variance quantification in the absence of ground truth data. The multiple correlation coefficients between the estimated and observed error variance over an independent validation region (Upper Mississippi River basin) and time period (2007-2010) are found to be 0.75, 0.86, and 0.87 for 3B42RT, CMORPH, and PERSIANN-CCS products, respectively. In another validation region (Arkansas-Red River basin), the correlation coefficients are 0.59, 0.89, and 0.92 for the same products, respectively. Results of the assessment of error variance models reveal that the type of error component present in a satellite rainfall product directly impacts the accuracy of estimated error variance. The model estimates the error variance more accurately when the precipitation error components are mostly hit bias or false precipitation, while for a product with extensive missed precipitation, the accuracy of estimated error variance is significantly compromised. The study clearly demonstrates the feasibility of quantifying the error variance of satellite rainfall products in a spatially and temporally varying manner using readily available geophysical features and rainfall rate. The study is a path finder to a globally applicable and operationally feasible methodology for error variance estimation at high spatial and temporal scales for advancing satellite rainfall applications in ungauged basins.

Estimating and diagnosing model error variances in the Météo‐France global NWP model

AbstractA methodology for estimating model error statistics is proposed. Its application to the global operational model ARPEGE of Météo‐France provides valuable insights into the spatio‐temporal dynamics of model error variances. In particular larger model errors are found in the midlatitude storm tracks (high cyclonic activity) for dynamical variables such as 500 hPa geopotential height and the 850 hPa wind speed.The average model errors over both hemispheres show a linear growth until they reach saturation. Model errors are also shown to grow more rapidly than predictability errors; this leads to a crossover time beyond which model error contribution to forecast error starts to play the dominant role. Moreover, model errors saturate more rapidly than predictability errors. On the other hand, spectral analysis shows an upscale energy transfer and a faster growth at synoptic scales for both model errors and predictability errors. This indicates that, for dynamical variables, the growth of both errors is most likely driven by baroclinic instability.The results found in this study could provide valuable information for a future implementation of a stochastic physics approach to account for model errors in the operational ensemble prediction system.

Impact of multicollinearity on small sample hydrologic regression models

[1] Often hydrologic regression models are developed with ordinary least squares (OLS) procedures. The use of OLS with highly correlated explanatory variables produces multicollinearity, which creates highly sensitive parameter estimators with inflated variances and improper model selection. It is not clear how to best address multicollinearity in hydrologic regression models. Here a Monte Carlo simulation is developed to compare four techniques to address multicollinearity: OLS, OLS with variance inflation factor screening (VIF), principal component regression (PCR), and partial least squares regression (PLS). The performance of these four techniques was observed for varying sample sizes, correlation coefficients between the explanatory variables, and model error variances consistent with hydrologic regional regression models. The negative effects of multicollinearity are magnified at smaller sample sizes, higher correlations between the variables, and larger model error variances (smaller R2). The Monte Carlo simulation indicates that if the true model is known, multicollinearity is present, and the estimation and statistical testing of regression parameters are of interest, then PCR or PLS should be employed. If the model is unknown, or if the interest is solely on model predictions, is it recommended that OLS be employed since using more complicated techniques did not produce any improvement in model performance. A leave-one-out cross-validation case study was also performed using low-streamflow data sets from the eastern United States. Results indicate that OLS with stepwise selection generally produces models across study regions with varying levels of multicollinearity that are as good as biased regression techniques such as PCR and PLS.

Hidden Error Variance Theory. Part II: An Instrument That Reveals Hidden Error Variance Distributions from Ensemble Forecasts and Observations

Abstract In Part I of this study, a model of the distribution of true error variances given an ensemble variance is shown to be defined by six parameters that also determine the optimal weights for the static and flow-dependent parts of hybrid error variance models. Two of the six parameters (the climatological mean of forecast error variance and the climatological minimum of ensemble variance) are straightforward to estimate. The other four parameters are (i) the variance of the climatological distribution of the true conditional error variances, (ii) the climatological minimum of the true conditional error variance, (iii) the relative variance of the distribution of ensemble variances given a true conditional error variance, and (iv) the parameter that defines the mean response of the ensemble variances to changes in the true error variance. These parameters are hidden because they are defined in terms of condition-dependent forecast error variance, which is unobservable if the condition is not sufficiently repeatable. Here, a set of equations that enable these hidden parameters to be accurately estimated from a long time series of (observation minus forecast, ensemble variance) data pairs is presented. The accuracy of the equations is demonstrated in tests using data from long data assimilation cycles with differing model error variance parameters as well as synthetically generated data. This newfound ability to estimate these hidden parameters provides new tools for assessing the quality of ensemble forecasts, tuning hybrid error variance models, and postprocessing ensemble forecasts.

Hidden Error Variance Theory. Part I: Exposition and Analytic Model

Abstract A conundrum of predictability research is that while the prediction of flow-dependent error distributions is one of its main foci, chaos fundamentally hides flow-dependent forecast error distributions from empirical observation. Empirical estimation of such error distributions requires a large sample of error realizations given the same flow-dependent conditions. However, chaotic elements of the flow and the observing network make it impossible to collect a large enough conditioned error sample to empirically define such distributions and their variance. Such conditional variances are “hidden.” Here, an exposition of the problem is developed from an ensemble Kalman filter data assimilation system applied to a 10-variable nonlinear chaotic model and 25 000 replicate models. The 25 000 replicates reveal the error variances that would otherwise be hidden. It is found that the inverse-gamma distribution accurately approximates the posterior distribution of conditional error variances given an imperfect ensemble variance and provides a reasonable approximation to the prior climatological distribution of conditional error variances. A new analytical model shows how the properties of a likelihood distribution of ensemble variances given a true conditional error variance determine the posterior distribution of error variances given an ensemble variance. The analytically generated distributions are shown to satisfactorily fit empirically determined distributions. The theoretical analysis yields a rigorous interpretation and justification of hybrid error variance models that linearly combine static and flow-dependent estimates of forecast error variance; in doing so, it also helps justify and inform hybrid error covariance models.

Improved robust Bayes estimators of the error variance in linear models

We consider the problem of estimating the error variance in a general linear model when the error distribution is assumed to be spherically symmetric, but not necessary Gaussian. In particular we study the case of a scale mixture of Gaussians including the particularly important case of the multivariate-t distribution. Under Stein's loss, we construct a class of estimators that improve on the usual best unbiased (and best equivariant) estimator. Our class has the interesting double robustness property of being simultaneously generalized Bayes (for the same generalized prior) and minimax over the entire class of scale mixture of Gaussian distributions.

Bayesian Monte Carlo for evaluation of uncertainty in hydrodynamic models of coastal systems

Camacho, R.A. and Martin, J.L., 2013. Bayesian Monte Carlo for evaluation of uncertainty in hydrodynamic models of coastal systems. In: Conley, D.C., Masselink, G., Russell, P.E. and O’Hare, T.J. (eds.), Proceedings 12 th International Coastal Symposium (Plymouth, England), Journal of Coastal Research, Special Issue No. 65, pp. 886-891, ISSN 07490208. Uncertainty analysis constitutes the set of procedures and strategies conducted to identify, quantify, and report the impacts of different sources of errors on the predictions of a numerical model. Although during the last couple of decades several investigations in different areas of water resources have explicitly discussed the importance of a new modeling paradigm where model predictions are reported along with uncertainty estimates, uncertainty analysis remains an emerging topic in the field of hydrodynamic modeling. Presently, several unresolved issues remain with regard to the applicability, benefits, and limitations of existing strategies for quantification of uncertainty, and the identification of the principal sources of uncertainty in practical applications. In this document we apply the Bayesian Monte Carlo (BMC) method as a strategy to perform uncertainty analysis in coastal modeling studies. BMC is based on robust principles of Bayesian inference and Monte Carlo simulations, and has been successfully applied for studies of groundwater modeling, hydrologic modeling, and water quality modeling in the past. The procedural aspects of the method are discussed in detail, and the implementation of the method is demonstrated for the hydrodynamic model of the St. Louis Bay estuary, Mississippi (USA) for the evaluation of the impacts of input data errors. Results indicate that BMC is an effective strategy for uncertainty analyses, although in professional practice its use may be limited by computational requirements. We also point out that the effectiveness of the method may be affected by the correct selection of the likelihood function and the model error variance.

Analyzing the uncertainty of suspended sediment load prediction using sequential data assimilation

► Estimated snow melt and streamflow using SNOW-17 and SAC-SMA models. ► Applied ensemble data assimilation to improve predictions and estimate uncertainty. ► Developed an approach to dynamically adjust model error variance scaling factors. ► Linked streamflow ensemble to non-linear regression model of suspended sediment. ► Estimated annual suspended sediment load for watershed in the Lake Tahoe Basin. A first step in understanding the impacts of sediment and controlling the sources of sediment is to quantify the mass loading. Since mass loading is the product of flow and concentration, the quantification of loads first requires the quantification of runoff volume. Using the National Weather Service’s SNOW-17 and the Sacramento Soil Moisture Accounting (SAC-SMA) models, this study employed particle filter based Bayesian data assimilation methods to predict seasonal snow water equivalent (SWE) and runoff within a small watershed in the Lake Tahoe Basin located in California, USA. A procedure was developed to scale the variance multipliers (a.k.a hyperparameters) for model parameters and predictions based on the accuracy of the mean predictions relative to the ensemble spread. In addition, an online bias correction algorithm based on the lagged average bias was implemented to detect and correct for systematic bias in model forecasts prior to updating with the particle filter. Both of these methods significantly improved the performance of the particle filter without requiring excessively wide prediction bounds. The flow ensemble was linked to a non-linear regression model that was used to predict suspended sediment concentrations (SSCs) based on runoff rate and time of year. Runoff volumes and SSC were then combined to produce an ensemble of suspended sediment load estimates. Annual suspended sediment loads for the 5 years of simulation were finally computed along with 95% prediction intervals that account for uncertainty in both the SSC regression model and flow rate estimates. Understanding the uncertainty associated with annual suspended sediment load predictions is critical for making sound watershed management decisions aimed at maintaining the exceptional clarity of Lake Tahoe. The computational methods developed and applied in this research could assist with similar studies where it is important to quantify the predictive uncertainty of pollutant load estimates.

A Comparison of Error Variance Estimates in Nonparametric Mixed Models

The author provides three kinds of estimates (six estimators) of the error variance in nonparametric mixed models (NMMs) without any distribution assumptions about random effects and random errors. Their asymptotic mean square errors are investigated. Different from nonparametric regression (NR) with independent homoscedastic case, error variance estimators by a nonparametric fit with the form of Y τ M σ Y/tr(M σ), which are consistent in NR (Dette et al., 1998), are inconsistent in NMMs. Besides, the equivalence of GCV proposed by Gu and Ma (2005) and GCV by Wang (1998b) is also found. A simulation study is conducted to investigate the performance of these estimators.

Regional flood frequency analysis in eastern Australia: Bayesian GLS regression-based methods within fixed region and ROI framework – Quantile Regression vs. Parameter Regression Technique

Summary In this article, an approach using Bayesian Generalised Least Squares (BGLS) regression in a region-of-influence (ROI) framework is proposed for regional flood frequency analysis (RFFA) for ungauged catchments. Using the data from 399 catchments in eastern Australia, the BGLS-ROI is constructed to regionalise the flood quantiles (Quantile Regression Technique (QRT)) and the first three moments of the log-Pearson type 3 (LP3) distribution (Parameter Regression Technique (PRT)). This scheme firstly develops a fixed region model to select the best set of predictor variables for use in the subsequent regression analyses using an approach that minimises the model error variance while also satisfying a number of statistical selection criteria. The identified optimal regression equation is then used in the ROI experiment where the ROI is chosen for a site in question as the region that minimises the predictive uncertainty. To evaluate the overall performances of the quantiles estimated by the QRT and PRT, a one-at-a-time cross-validation procedure is applied. Results of the proposed method indicate that both the QRT and PRT in a BGLS-ROI framework lead to more accurate and reliable estimates of flood quantiles and moments of the LP3 distribution when compared to a fixed region approach. Also the BGLS-ROI can deal reasonably well with the heterogeneity in Australian catchments as evidenced by the regression diagnostics. Based on the evaluation statistics it was found that both BGLS-QRT and PRT-ROI perform similarly well, which suggests that the PRT is a viable alternative to QRT in RFFA. The RFFA methods developed in this paper is based on the database available in eastern Australia. It is expected that availability of a more comprehensive database (in terms of both quality and quantity) will further improve the predictive performance of both the fixed and ROI based RFFA methods presented in this study, which however needs to be investigated in future when such a database is available.

A Bayesian view of model complexity

This article addresses the problem of formally defining the ‘effective number of parameters’ in a Bayesian model which is assumed to be given by a sampling distribution and a prior distribution for the parameters. The problem occurs in the derivation of information criteria for model comparison which often trade off ‘goodness of fit’ and ‘model complexity’. It also arises in (frequentist) attempts to estimate the error variance in regression models with informative priors on the regression coefficients, for example, in smoothing. It is argued that model complexity can be conceptualized as a feature of the joint distribution of the observed variables and the random parameters and might be formally described by a measure of dependence. The universal and accurate estimation of terms of model complexity is a challenging problem in practice. Several well‐known criteria for model comparison are interpreted and discussed along these lines.

Regional flood frequency analysis using Bayesian generalized least squares: a comparison between quantile and parameter regression techniques

AbstractRegression‐based regional flood frequency analysis (RFFA) methods are widely adopted in hydrology. This paper compares two regression‐based RFFA methods using a Bayesian generalized least squares (GLS) modelling framework; the two are quantile regression technique (QRT) and parameter regression technique (PRT). In this study, the QRT focuses on the development of prediction equations for a flood quantile in the range of 2 to 100 years average recurrence intervals (ARI), while the PRT develops prediction equations for the first three moments of the log Pearson Type 3 (LP3) distribution, which are the mean, standard deviation and skew of the logarithms of the annual maximum flows; these regional parameters are then used to fit the LP3 distribution to estimate the desired flood quantiles at a given site. It has been shown that using a method similar to stepwise regression and by employing a number of statistics such as the model error variance, average variance of prediction, Bayesian information criterion and Akaike information criterion, the best set of explanatory variables in the GLS regression can be identified. In this study, a range of statistics and diagnostic plots have been adopted to evaluate the regression models. The method has been applied to 53 catchments in Tasmania, Australia. It has been found that catchment area and design rainfall intensity are the most important explanatory variables in predicting flood quantiles using the QRT. For the PRT, a total of four explanatory variables were adopted for predicting the mean, standard deviation and skew. The developed regression models satisfy the underlying model assumptions quite well; of importance, no outlier sites are detected in the plots of the regression diagnostics of the adopted regression equations. Based on ‘one‐at‐a‐time cross validation’ and a number of evaluation statistics, it has been found that for Tasmania the QRT provides more accurate flood quantile estimates for the higher ARIs while the PRT provides relatively better estimates for the smaller ARIs. The RFFA techniques presented here can easily be adapted to other Australian states and countries to derive more accurate regional flood predictions. Copyright © 2011 John Wiley & Sons, Ltd.

The Effect of Media Advertising on Brand Consideration and Choice

The nature of the effect of media advertising on brand choice is investigated in two product categories in analyses that combine household scanner panel data with media exposure information. Alternative model specifications are tested in which advertising is assumed to directly affect brand utility, model error variance, and brand consideration. We find strong support for advertising effects on choice through an indirect route of consideration set formation that does not directly affect brand utility. Implications for media buying and advertising effects are explored.

Model-based inference for biomass estimation in a LiDAR sample survey in Hedmark County, NorwayThis article is one of a selection of papers from Extending Forest Inventory and Monitoring over Space and Time.

In forest inventories, regression models are often applied to predict quantities such as biomass at the level of sampling units. In this paper, we propose a model-based inference framework for combining sampling and model errors in the variance estimation. It was applied to airborne laser (LiDAR) data sets from Hedmark County, Norway, where the model error proportion of the total variance was found to be large for both scanning (airborne laser scanning) and profiling LiDAR when biomass was estimated. With profiling LiDAR, the model error variance component for the entire county was as large as 71% whereas for airborne laser scanning, it was 43% of the total variance. Partly, this reflects the better accuracy of the pixel-based regression models estimated from scanner data as compared with the models estimated from profiler data. The framework proposed in our study can be applied in all types of sample surveys where model-based predictions are made at the level of individual sampling units. Especially, it should be useful in cases where model-assisted inference cannot be applied due to the lack of a probability sample from the target population or due to problems of correctly matching observations of auxiliary and target variables.