Model Selection Uncertainty Research Articles

Estimation of a multivariate mean under model selection uncertainty

Model selection uncertainty would occur if we selected a model based on one data set and subsequently applied it for statistical inferences, because the correct model would not be selected with certainty. When the selection and inference are based on the same dataset, some additional problems arise due to the correlation of the two stages (selection and inference). In this paper model selection uncertainty is considered and model averaging is proposed. The proposal is related to the theory of James and Stein of estimating more than three parameters from independent normal observations. We suggest that a model averaging scheme taking into account the selection procedure could be more appropriate than model selection alone. Some properties of this model averaging estimator are investigated; in particular we show using Stein's results that it is a minimax estimator and can outperform Stein-type estimators.

P values are only an index to evidence: 20th- vs. 21st-century statistical science.

Early statistical methods focused on pre-data probability statements (i.e., data as random variables) such as P values; these are not really inferences nor are P values evidential. Statistical science clung to these principles throughout much of the 20th century as a wide variety of methods were developed for special cases. Looking back, it is clear that the underlying paradigm (i.e., testing and P values) was weak. As Kuhn (1970) suggests, new paradigms have taken the place of earlier ones: this is a goal of good science. New methods have been developed and older methods extended and these allow proper measures of strength of evidence and multimodel inference. It is time to move forward with sound theory and practice for the difficult practical problems that lie ahead. Given data the useful foundation shifts to post-data probability statements such as model probabilities (Akaike weights) or related quantities such as odds ratios and likelihood intervals. These new methods allow formal inference from multiple models in the a prior set. These quantities are properly evidential. The past century was aimed at finding the "best" model and making inferences from it. The goal in the 21st century is to base inference on all the models weighted by their model probabilities (model averaging). Estimates of precision can include model selection uncertainty leading to variances conditional on the model set. The 21st century will be about the quantification of information, proper measures of evidence, and multi-model inference. Nelder (1999:261) concludes, "The most important task before us in developing statistical science is to demolish the P-value culture, which has taken root to a frightening extent in many areas of both pure and applied science and technology".

Identifying disease candidate genes via large-scale gene network analysis.

Gene Regulatory Networks (GRN) provide systematic views of complex living systems, offering reliable and large-scale GRNs to identify disease candidate genes. A reverse engineering technique, Bayesian Model Averaging-based Networks (BMAnet), which ensembles all appropriate linear models to tackle uncertainty in model selection that integrates heterogeneous biological data sets is introduced. Using network evaluation metrics, we compare the networks that are thus identified. The metric 'Random walk with restart (Rwr)' is utilised to search for disease genes. In a simulation our method shows better performance than elastic-net and Gaussian graphical models, but topological quantities vary among the three methods. Using real-data, brain tumour gene expression samples consisting of non-tumour, grade III and grade IV are analysed to estimate networks with a total of 4422 genes. Based on these networks, 169 brain tumour-related candidate genes were identified and some were found to relate to 'wound', 'apoptosis', and 'cell death' processes.

New Criteria of Model Selection and Model Averaging in Linear Regression Models

Model selection is an important part of any statistical analysis. Many tools are suggested for selecting the best model including frequentist and Bayesian perspectives. There is often a considerable uncertainty in the selection of a particular model to be the best approximating model. Model selection uncertainty arises when the data are used for both model selection and parameter estimation. Bias in estimators of model parameters often arise when data based selection has been done. Therefore, model averaging of the parameter estimators will be done to alleviate the bias in model selection in a set of candidate models, by combining the information from a set of candidate models. This paper is two-fold, new criteria of model selection are proposed based on different averages of AIC, BIC, AICc, and HQC. Also, model averaging is introduced to compare the parameter estimators in model averaging with the ones in model selection. Two Simulation studies are considered, the first is for model selection and showed that the new proposed criteria are lies between some of the known criteria such as AIC, BIC, AICc, and HQC, and so they can be used as new criteria of model selection. The second simulation study is for model averaging and showed that the parameter estimators have less bias and less predicted mean square error (PMSE) compared with the parameter estimators in model selection.

Incorporating detection probability into northern Great Plains pronghorn population estimates

ABSTRACTPronghorn (Antilocapra americana) abundances commonly are estimated using fixed‐wing surveys, but these estimates are likely to be negatively biased because of violations of key assumptions underpinning line‐transect methodology. Reducing bias and improving precision of abundance estimates through use of detection probability and mark‐resight models may allow for more responsive pronghorn management actions. Given their potential application in population estimation, we evaluated detection probability and mark‐resight models for use in estimating pronghorn population abundance. We used logistic regression to quantify probabilities that detecting pronghorn might be influenced by group size, animal activity, percent vegetation, cover type, and topography. We estimated pronghorn population size by study area and year using mixed logit‐normal mark‐resight (MLNM) models. Pronghorn detection probability increased with group size, animal activity, and percent vegetation; overall detection probability was 0.639 (95% CI = 0.612–0.667) with 396 of 620 pronghorn groups detected. Despite model selection uncertainty, the best detection probability models were 44% (range = 8–79%) and 180% (range = 139–217%) greater than traditional pronghorn population estimates. Similarly, the best MLNM models were 28% (range = 3–58%) and 147% (range = 124–180%) greater than traditional population estimates. Detection probability of pronghorn was not constant but depended on both intrinsic and extrinsic factors. When pronghorn detection probability is a function of animal group size, animal activity, landscape complexity, and percent vegetation, traditional aerial survey techniques will result in biased pronghorn abundance estimates. Standardizing survey conditions, increasing resighting occasions, or accounting for variation in individual heterogeneity in mark‐resight models will increase the accuracy and precision of pronghorn population estimates. © 2013 The Wildlife Society.

Estimating marginal properties of quantitative real‐time PCR data using nonlinear mixed models

A unified modeling framework based on a set of nonlinear mixed models is proposed for flexible modeling of gene expression in real-time PCR experiments. Focus is on estimating the marginal or population-based derived parameters: cycle thresholds and ΔΔc(t), but retaining the conditional mixed model structure to adequately reflect the experimental design. Additionally, the calculation of model-average estimates allows incorporation of the model selection uncertainty. The methodology is applied for estimating the differential expression of a phosphate transporter gene OsPT6 in rice in comparison to a reference gene at several states after phosphate resupply. In a small simulation study the performance of the proposed method is evaluated and compared to a standard method.

Information-theoretic model averaging: a useful method for modelling exploratory secondary endpoints in clinical trials

In clinical trials, highly exploratory secondary endpoints are common and the number of measured variables plausibly associated with response can be numerous, particularly if biomarkers are considered. Automated, single-best-model selection procedures are widely employed in such instances. Known issues are: 1) the precision of parameter estimates are estimated as if the selected model was prespecified and 2) useful competing models are disregarded. By contrast, information-theoretic approaches to model averaging (IT-MA) incorporate model selection uncertainty and combine parameter estimates across models. Two IT-MA estimators exist: the ‘standard’ and the shrinkage estimator. The latter has been considered previously in the context of linear regression. We extend this by exploring IT-MA estimator properties in logistic regression simulation studies and compare results to minimum AIC selection (mAIC). In the extreme case where no explanatory variables are related to response, the confidence interval coverage of both IT-MA estimators is near-nominal in sharp contrast to mAIC, even with a prudent number of events-per-variable. In the presence of collinearity we demonstrate that for r>0.7, both IT-MA estimators exhibit bias but have considerably lower variances than the full model. Interestingly, for true zero effects the standard IT-MA estimator is preferable whilst for non-zero effects the shrinkage estimator is preferable. In conclusion, confidence interval coverage for IT-MA is superior to mAIC and both IT-MA estimators perform well even in the presence of collinearity. These results emphasise the usefulness of IT-MA as a tool for modelling exploratory secondary endpoints in clinical trials.

CHull as an alternative to AIC and BIC in the context of mixtures of factor analyzers

Mixture analysis is commonly used for clustering objects on the basis of multivariate data. When the data contain a large number of variables, regular mixture analysis may become problematic, because a large number of parameters need to be estimated for each cluster. To tackle this problem, the mixtures-of-factor-analyzers (MFA) model was proposed, which combines clustering with exploratory factor analysis. MFA model selection is rather intricate, as both the number of clusters and the number of underlying factors have to be determined. To this end, the Akaike (AIC) and Bayesian (BIC) information criteria are often used. AIC and BIC try to identify a model that optimally balances model fit and model complexity. In this article, the CHull (Ceulemans & Kiers, 2006) method, which also balances model fit and complexity, is presented as an interesting alternative model selection strategy for MFA. In an extensive simulation study, the performances of AIC, BIC, and CHull were compared. AIC performs poorly and systematically selects overly complex models, whereas BIC performs slightly better than CHull when considering the best model only. However, when taking model selection uncertainty into account by looking at the first three models retained, CHull outperforms BIC. This especially holds in more complex, and thus more realistic, situations (e.g., more clusters, factors, noise in the data, and overlap among clusters).

On the Use of Bayesian Model Averaging for Covariate Selection in Epidemiological Modeling

Bayesian model averaging (BMA) is a powerful technique to address model selection uncertainty and recent computational advances have led to a proliferation of usage. BMA methods are of particular interest in environmental health risk assessment because of the high degree of uncertainty that typically arises in that context. In this article, we review a variety of approaches to conducting BMA and compare four implementations in a setting where there are a number of potential predictors. We then use these four methods to calculate risk assessment measures that account for the uncertainty involved in modeling environmental exposures. These methods are used to reexamine data from a study conducted by Walkowiak et al. (2001) to investigate the effects of maternal polychlorinated biphenyl exposure on cognitive development in early childhood. This case study reveals that different strategies for implementing BMA can yield varying risk assessment results. We conclude with some practical recommendations.

Model uncertainty and multimodel inference in reliability estimation within a longitudinal framework

Laenen, Alonso, and Molenberghs (2007) and Laenen, Alonso, Molenberghs, and Vangeneugden (2009) proposed a method to assess the reliability of rating scales in a longitudinal context. The methodology is based on hierarchical linear models, and reliability coefficients are derived from the corresponding covariance matrices. However, finding a good parsimonious model to describe complex longitudinal data is a challenging task. Frequently, several models fit the data equally well, raising the problem of model selection uncertainty. When model uncertainty is high one may resort to model averaging, where inferences are based not on one but on an entire set of models. We explored the use of different model building strategies, including model averaging, in reliability estimation. We found that the approach introduced by Laenen et al. (2007, 2009) combined with some of these strategies may yield meaningful results in the presence of high model selection uncertainty and when all models are misspecified, in so far as some of them manage to capture the most salient features of the data. Nonetheless, when all models omit prominent regularities in the data, misleading results may be obtained. The main ideas are further illustrated on a case study in which the reliability of the Hamilton Anxiety Rating Scale is estimated. Importantly, the ambit of model selection uncertainty and model averaging transcends the specific setting studied in the paper and may be of interest in other areas of psychometrics.

The problem of model selection uncertainty in structural equation modeling.

Model selection in structural equation modeling (SEM) involves using selection criteria to declare one model superior and treating it as a best working hypothesis until a better model is proposed. A limitation of this approach is that sampling variability in selection criteria usually is not considered, leading to assertions of model superiority that may not withstand replication. We illustrate that selection decisions using information criteria can be highly unstable over repeated sampling and that this uncertainty does not necessarily decrease with increases in sample size. Methods for addressing model selection uncertainty in SEM are evaluated, and implications for practice are discussed.

Occupancy rates of primary burrowing crayfish in natural and disturbed large river bottomlands

ABSTRACT Among crayfish, primary burrowing species are the least understood ecologically. Many primary burrowing crayfish inhabit floodplains where forested landscapes have been fragmented by agricultural, industrial, or residential uses. In this study, site occupancy rates (ψ) were modeled for two primary burrowing crayfish, Fallicambarus fodiens (Cottle, 1863) and Cambarus thomai Jezerinac, 1993, from Ohio and Kanawha river floodplains in West Virginia, U.S.A. Fallicambarus fodiens is one of West Virginia's rarest crayfish, while C. thomai is prevalent in most wetlands along both river floodplains. Occupancy rate modeling incorporated four environmental covariates (forest age, soil type, tree frequency, and land use). Based on presence/absence data, forests with tree ages > 100 years (ΔQAICc = 0) and sites with loam soils (ΔQAICc = 1.80) were most likely to harbor F. fodiens populations. For C. thomai, several models were supported owing to model selection uncertainty, but those with the land use covari...

Estimation of stochastic frontier models based on multimodel inference

In most empirical studies, once the best model has been selected according to a certain criterion, subsequent analysis is conducted conditionally on the chosen model. In other words, the uncertainty of model selection is ignored once the best model has been chosen. However, the true data-generating process is in general unknown and may not be consistent with the chosen model. In the analysis of productivity and technical efficiencies in the stochastic frontier settings, if the estimated parameters or the predicted efficiencies differ across competing models, then it is risky to base the prediction on the selected model. Buckland et al. (Biometrics 53:603–618, 1997) have shown that if model selection uncertainty is ignored, the precision of the estimate is likely to be overestimated, the estimated confidence intervals of the parameters are often below the nominal level, and consequently, the prediction may be less accurate than expected. In this paper, we suggest using the model-averaged estimator based on the multimodel inference to estimate stochastic frontier models. The potential advantages of the proposed approach are twofold: incorporating the model selection uncertainty into statistical inference; reducing the model selection bias and variance of the frontier and technical efficiency estimators. The approach is demonstrated empirically via the estimation of an Indian farm data set.

Combining day-ahead forecasts for British electricity prices

This paper considers how well the approach of combining forecasts extends to the context of electricity prices. With the increasing popularity of regime switching and time-varying parameter models for predicting power prices, the multi model and evolutionary considerations that usually support the combining of simpler time series methods may be less applicable when the individual models incorporate these features. We address this question with a backtesting analysis on British day-ahead prices. Furthermore, given the volatility of power prices and concerns about accurate forecasting under extreme price excursions, we evaluate the results using various error metrics including expected shortfall. The comparisons are furthermore carefully simulated to consider model selection uncertainty in order to realistically test the value of combining as an ex ante policy. Overall, our results support combining for both accurate operational planning and risk management.

An improved multi‐scale approach to modeling habitat occupancy of northern bobwhite

AbstractPredicting species presence requires knowledge of detection of individuals, scale of model variables, model selection uncertainty, and spatial autocorrelation. Our objective was to incorporate recent modeling advances to predict potential habitat occupancy of northern bobwhite (Colinus virginianus). From 15 May–15 August 2008 and 2009, we conducted repeat‐visit surveys at 360 sites within Delaware to sample presence of bobwhite. We randomly selected half the data to model scale‐dependent relationships of bobwhite presence with metrics of landscape‐ and site‐scale habitat composition and configuration. The final averaged habitat‐occupancy model fit the remainder testing dataset with an area under the receiver operating characteristic curve value of 0.62. At the site scale, bobwhite presence was negatively related to interspersion and juxtaposition of early successional habitat (ESH; grassland and shrubland), ESH to forest edge density, and agriculture to forest edge density, though relative effect sizes were weak to moderate after accounting for model selection uncertainty. At the landscape scale, bobwhite presence was negatively related to patch cohesion of human development within 2.5 km and positively related to patch cohesion of ESH within 2.0 km, with both variables exerting strong effects. Bobwhite presence was also weakly and positively related to percentage of shrubland habitat within 1.0 km of the sampling point. We applied our habitat occupancy model to map the predicted presence of breeding bobwhite within the Delmarva Peninsula, USA. The modeling results and distribution map will provide guidance to State and Federal private land management programs in the Mid‐Atlantic to identify where habitat management efforts will be most effective. Our methodology can also serve as a basis for future habitat modeling of bobwhite and other grassland–shrubland species across their range. © 2011 The Wildlife Society.

Estimates of live-tree carbon stores in the Pacific Northwest are sensitive to model selection

BackgroundEstimates of live-tree carbon stores are influenced by numerous uncertainties. One of them is model-selection uncertainty: one has to choose among multiple empirical equations and conversion factors that can be plausibly justified as locally applicable to calculate the carbon store from inventory measurements such as tree height and diameter at breast height (DBH). Here we quantify the model-selection uncertainty for the five most numerous tree species in six counties of northwest Oregon, USA.ResultsThe results of our study demonstrate that model-selection error may introduce 20 to 40% uncertainty into a live-tree carbon estimate, possibly making this form of error the largest source of uncertainty in estimation of live-tree carbon stores. The effect of model selection could be even greater if models are applied beyond the height and DBH ranges for which they were developed.ConclusionsModel-selection uncertainty is potentially large enough that it could limit the ability to track forest carbon with the precision and accuracy required by carbon accounting protocols. Without local validation based on detailed measurements of usually destructively sampled trees, it is very difficult to choose the best model when there are several available. Our analysis suggests that considering tree form in equation selection may better match trees to existing equations and that substantial gaps exist, in terms of both species and diameter ranges, that are ripe for new model-building effort.

Model Averaging in Factor Analysis: An Analysis of Olympic Decathlon Data

This article presents a multivariate analysis of Olympic decathlon data based on maximum likelihood factor analysis. All results explicitly account for model selection uncertainty, which is inherent in any data-based selection process but mostly ignored in reports related to multivariate sports data. For this purpose, some well-established frequentist procedures that have so far been applied almost exclusively to regression analysis are adopted and transferred to the factor analytical context. The findings support the claim that decathlon contests consist of three dimensions. These dimensions seem to be similar to, but not exactly the same, as those found by Cox and Dunn (2002) via hierarchical cluster analysis.

Making better sense of monitoring data from low density species using a spatially explicit modelling approach

Summary1. Wildlife managers are limited in the inferences they can draw about low density populations. These limits are imposed by biases in monitoring data not regularly accounted for.2. We developed a Bayesian hierarchical model to correct biases arising from imperfect detection and spatial autocorrelation. Our analysis incorporated model selection uncertainty by treating model probabilities as parameters to be estimated in the context of model fitting. We fitted our model to count data from a monitoring programme for the mountain plover Charadrius montanus, a low density bird species in Nebraska, USA.3. Our results demonstrated that previous accounts of the abundance and distribution of plovers in Nebraska were impacted by low detection probabilities (∼5–20%). Uncorrected relative abundance estimates showed that the average number of birds per agricultural section increased over time, whereas corrected estimates showed that average abundance was stable.4. Our method spatially interpolated relative abundance to produce distribution maps. These predictions suggested that birds were selecting some sites more frequently than others based on some habitat feature not explored in our study. Variation in mountain plover abundance appeared more heavily influenced by changes in the number of individuals occupying a few high quality sites, rather than from changes in abundance across many sites. Thus, conservation efforts may not be as efficient when focusing on low to moderate quality sites.5. Synthesis and applications. Managers who must make decisions based on data‐poor systems should adopt rigorous statistical approaches for drawing inferences. Spatial predictions provide information for deciding where to implement management, which is just as important as knowing what kind of management to apply. Our approach provides a step in the direction of making the biological signal in data‐poor monitoring programmes more informative for conservation and management.

Resource heterogeneity influences home range area in the swamp wallaby Wallabia bicolor

Home range area is influenced by the spatio-temporal distribution of multiple resources, but current theoretical frameworks such as the habitat productivity and resource dispersion hypotheses fail to account for this adequately. We propose a conceptual framework for quantifying the influence of multiple resources on home range area where separate resources interact to form a continuous, multi-dimensional resource surface exhibiting an emergent environmental pattern. We adopt the term resource heterogeneity to describe this pattern, and predict that home range area will be more closely correlated with resource heterogeneity than with simple additive or interactive effects of separate resources. Initially, we quantified resource heterogeneity as the geometric mean of resource abundance, and tested the performance of this index against simulated low, moderate and high heterogeneity scenarios. In all cases, the geometric mean reflected the modelled scenarios well, demonstrating its utility as a heterogeneity index. We then quantified shelter and food resources within the home ranges of swamp wallabies Wallabia bicolor, and used these data in a generalised linear mixed modelling framework to test our prediction. Based on Akaike's information criterion (AIC c ), the best model included the main effect of sex and the heterogeneity index. Although model selection uncertainty was relatively high (largest Akaike weight =0.37), the AIC c value associated with the best model was 2.41 units less than the highest ranked model including shelter and food, indicating support for our prediction. Home range area was negatively correlated with resource heterogeneity, and males had larger home ranges than females (37.6 (28.5, 49.7) ha compared to 17.2 (13.6, 21.7) ha; mean and 95% CLs). We suggest that statistical models of home range area parameterised using an index of resource heterogeneity will often be superior to those that focus on a single resource, or fail to take the heterogeneity of multiple resources into account.

Multimodel inference and adaptive management

Ecology is an inherently complex science coping with correlated variables, nonlinear interactions and multiple scales of pattern and process, making it difficult for experiments to result in clear, strong inference. Natural resource managers, policy makers, and stakeholders rely on science to provide timely and accurate management recommendations. However, the time necessary to untangle the complexities of interactions within ecosystems is often far greater than the time available to make management decisions. One method of coping with this problem is multimodel inference. Multimodel inference assesses uncertainty by calculating likelihoods among multiple competing hypotheses, but multimodel inference results are often equivocal. Despite this, there may be pressure for ecologists to provide management recommendations regardless of the strength of their study’s inference. We reviewed papers in the Journal of Wildlife Management (JWM) and the journal Conservation Biology (CB) to quantify the prevalence of multimodel inference approaches, the resulting inference (weak versus strong), and how authors dealt with the uncertainty. Thirty-eight percent and 14%, respectively, of articles in the JWM and CB used multimodel inference approaches. Strong inference was rarely observed, with only 7% of JWM and 20% of CB articles resulting in strong inference. We found the majority of weak inference papers in both journals (59%) gave specific management recommendations. Model selection uncertainty was ignored in most recommendations for management. We suggest that adaptive management is an ideal method to resolve uncertainty when research results in weak inference.