Traditional Model Selection Approaches Research Articles

Bayesian Model Averaging as an Alternative to Model Selection for Multilevel Models

We investigated the Bayesian model averaging (BMA) technique as an alternative method to the traditional model selection approaches for multilevel models (MLMs). BMA synthesizes the information derived from all possible models and comes up with a weighted estimate. A simulation study compared BMA with additional modeling techniques, including the single “best” model approach, Bayesian MLM using informative, diffuse, and inaccurate priors, and restricted maximum likelihood. A two-level random intercept and random slope model was examined with these modeling techniques. Generated data used two types of true models: a full MLM and a reduced MLM. Findings of the simulation study suggested that BMA was a trustworthy alternative to traditional model comparison and selection approaches through the Bayesian and the frequentist frameworks. We also include an empirical example highlighting the extension of MLMs into the BMA framework, as well as model interpretation.

Self-reported health behaviors and longitudinal cognitive performance in late middle age: Results from the Wisconsin Registry for Alzheimer's Prevention.

BackgroundStudies have suggested associations between self-reported engagement in health behaviors and reduced risk of cognitive decline. Most studies explore these relationships using one health behavior, often cross-sectionally or with dementia as the outcome. In this study, we explored whether several individual self-reported health behaviors were associated with cognitive decline when considered simultaneously, using data from the Wisconsin Registry for Alzheimer’s Prevention (WRAP), an Alzheimer’s disease risk-enriched cohort who were non-demented and in late midlife at baseline.MethodWe analyzed longitudinal cognitive data from 828 participants in WRAP, with a mean age at baseline cognitive assessment of 57 (range = 36–78, sd = 6.8) and an average of 6.3 years (standard deviation = 1.9, range = 2–10) of follow-up. The primary outcome was a multi-domain cognitive composite, and secondary outcomes were immediate/delayed memory and executive function composites. Predictors of interest were self-reported measures of physical activity, cognitive activity, adherence to a Mediterranean-style diet (MIND), and interactions with each other and age. We conducted linear mixed effects analyses within an Information-theoretic (IT) model averaging (MA) approach on a set of models including covariates and combinations of these 2- and 3-way interactions. The IT approach was selected due to the large number of interactions of interest and to avoid pitfalls of traditional model selection approaches.ResultsModel-averaged results identified no significant self-reported health behavior*age interactions in relationship to the primary composite outcome. In secondary outcomes, higher MIND diet scores associated with slower decline in executive function. Men showed faster decline than women on delayed memory, independent of health behaviors. There were no other significant interactions among any other health behaviors and cognitive trajectories.ConclusionsWhen multiple covariates and health behaviors were considered simultaneously, there were limited weak associations with cognitive decline in this age range. These results may be explained alone or in combination by three alternative explanations: 1) the range of cognitive decline is in middle age is too small to observe relationships with health behaviors, 2) the putative associations of these health behaviors on cognition may not be robust in this age range, or 3) the self-reported measures of the health behaviors may not be optimal for predicting cognitive decline. More study may be needed that incorporates sensitive measures of health behaviors, AD biomarker profiles, and/or other disease comorbidities.

Bacterial spore levels in bulk tank raw milk are influenced by environmental and cow hygiene factors

Sporeforming bacteria are responsible for the spoilage of several dairy products including fluid milk, cheese, and products manufactured using dried dairy powders as ingredients. Sporeforming bacteria represent a considerable challenge for the dairy industry because they primarily enter the dairy product continuum at the farm, survive processing hurdles, and subsequently grow in finished products. As such, strategies to reduce spoilage due to this group of bacterial contaminants have focused on understanding the effect of farm level factors on the presence of spores in bulk tank raw milk with the goal of reducing spore levels in raw milk, as well as understanding processing contributions to spore levels and outgrowth in finished products. The goal of the current study was to investigate sources of spores in the farm environment and survey farm management practices to identify variables using multimodel inference, a model averaging approach that eliminates the uncertainty of traditional model selection approaches, that affect the presence and levels of spores in bulk tank raw milk. To this end, environmental samples including feed, bedding, manure, soil, water, and so on, and bulk tank raw milk were collected twice from 17 upstate New York dairy farms over a 19-mo period and the presence and levels of various spore types (e.g., psychrotolerant, mesophilic, thermophilic, highly heat resistant thermophilic, specially thermoresistant thermophilic, and anaerobic butyric acid bacteria) were assessed. Manure had the highest level of spores for 4 out of 5 aerobic spore types with mean counts of 5.87, 5.22, 4.35, and 3.68 log cfu/g of mesophilic, thermophilic, highly heat resistant thermophilic, and specially thermoresistant thermophilic spores, respectively. In contrast, bulk tank raw milk had mean spore levels below 1 log cfu/mL across spore types. Multimodel inference was used to determine variables (i.e., management factors, environmental spore levels, and meteorological data from each sampling) that were important for presence or levels of each spore type in bulk tank raw milk. Analyses indicated that variables of importance for more than one spore type included the residual level of spores in milk from individual cows after thorough teat cleaning and forestripping, udder hygiene, clipping or flaming of udders, spore level in feed commodities, spore level in parlor air, how often bedding was topped up or changed, the use of recycled manure bedding, and the use of sawdust bedding. These results improve our understanding of how spores transfer from environmental sources into bulk tank raw milk and provide information that can be used to design intervention trials aimed at reducing spore levels in raw milk.

Ordinal regression models for zero-inflated and/or over-dispersed count data

Count data commonly arise in natural sciences but adequately modeling these data is challenging due to zero-inflation and over-dispersion. While multiple parametric modeling approaches have been proposed, unfortunately there is no consensus regarding how to choose the best model. In this article, we propose a ordinal regression model (MN) as a default model for count data given that this model is shown to fit well data that arise from several types of discrete distributions. We extend this model to allow for automatic model selection (MN-MS) and show that the MN-MS model generates superior inference when compared to using the full model or more traditional model selection approaches. The MN-MS model is used to determine how human biting rate of mosquitoes, known to be able to transmit malaria, are influenced by environmental factors in the Peruvian Amazon. The MN-MS model had one of the best fit and out-of-sample predictive skill amongst all models. While A. darlingi is strongly associated with highly anthropized landscapes, all the other mosquito species had higher mean biting rates in landscapes with a lower fraction of exposed soil and urban area, revealing a striking shift in species composition. We believe that the MN and MN-MS models are valuable additions to the modelling toolkit employed by environmental modelers and quantitative ecologists.

A general framework for frequentist model averaging

Model selection strategies have been routinely employed to determine a model for data analysis in statistics, and further study and inference then often proceed as though the selected model were the true model that were known a priori. Model averaging approaches, on the other hand, try to combine estimators for a set of candidate models. Specifically, instead of deciding which model is the `right' one, a model averaging approach suggests to fit a set of candidate models and average over the estimators using data adaptive weights. In this paper we establish a general frequentist model averaging framework that does not set any restrictions on the set of candidate models. It broadens the scope of the existing methodologies under the frequentist model averaging development. Assuming the data is from an unknown model, we derive the model averaging estimator and study its limiting distributions and related predictions while taking possible modeling biases into account. We propose a set of optimal weights to combine the individual estimators so that the expected mean squared error of the average estimator is minimized. Simulation studies are conducted to compare the performance of the estimator with that of the existing methods. The results show the benefits of the proposed approach over traditional model selection approaches as well as existing model averaging methods.

Characterizing the Effects of Sex, APOE ɛ4, and Literacy on Mid-life Cognitive Trajectories: Application of Information-Theoretic Model Averaging and Multi-model Inference Techniques to the Wisconsin Registry for Alzheimer's Prevention Study.

Prior research has identified numerous genetic (including sex), education, health, and lifestyle factors that predict cognitive decline. Traditional model selection approaches (e.g., backward or stepwise selection) attempt to find one model that best fits the observed data, risking interpretations that only the selected predictors are important. In reality, several predictor combinations may fit similarly well but result in different conclusions (e.g., about size and significance of parameter estimates). In this study, we describe an alternative method, Information-Theoretic (IT) model averaging, and apply it to characterize a set of complex interactions in a longitudinal study on cognitive decline. Here, we used longitudinal cognitive data from 1256 late-middle aged adults from the Wisconsin Registry for Alzheimer's Prevention study to examine the effects of sex, apolipoprotein E (APOE) ɛ4 allele (non-modifiable factors), and literacy achievement (modifiable) on cognitive decline. For each outcome, we applied IT model averaging to a set of models with different combinations of interactions among sex, APOE, literacy, and age. For a list-learning test, model-averaged results showed better performance for women versus men, with faster decline among men; increased literacy was associated with better performance, particularly among men. APOE had less of an association with cognitive performance in this age range (∼40-70 years). These results illustrate the utility of the IT approach and point to literacy as a potential modifier of cognitive decline. Whether the protective effect of literacy is due to educational attainment or intrinsic verbal intellectual ability is the topic of ongoing work. (JINS, 2019, 25, 119-133).

Objective, Bayesian Analysis of Fluorescence Correlation Spectroscopy Data

Fluctuations in fluorescence intensity provide quantitative insight into a range of molecular processes including diffusion, active transport, and binding kinetics in living cells and tissues. Classical fluorescence correlation spectroscopy (FCS) measures fluctuations in fluorescence intensity in a small detection volume to infer molecular transport properties from governing continuum diffusion-convection equations. Essential to the interpretation of FCS data is the use of an underlying mathematical model that governs the molecular process under study. Autocorrelations in fluctuations in fluorescence intensity may then be fit to the analytical solution of the model, and corresponding molecular properties may be inferred. While the choice of physical process may be unambiguous for special cases of free molecular diffusion or convective transport in solution, the correct choice of model becomes considerably less unambiguous in the application of FCS to complex biological processes in living cells or extracellular matrices. For this reason, an objective and unbiased approach to model selection and parameter estimation is of interest. The method of Bayesian inference provides such a framework. At the level of parameter estimation, Bayesian inference is similar to Maximum Likelihood Estimation when flat priors are chosen for initial parameter estimates. At the level of model selection, however, Bayesian inference assigns explicit model probabilities that are proportional to their marginal likelihoods by considering the full range of parameter values and their posterior probability distributions rather than only using point-wise, Maximum Likelihood estimates, thereby appropriately penalizing model complexity and preventing over-fitting of experimental data. Here we illustrate application of Bayesian inference to the problem of model selection and parameter estimation of molecular transport properties from FCS data, comparing it to traditional model selection approaches.

Model selection in demographic time series using VC-bounds

The problem of distinguishing density-independent (DI) from density-dependent (DD) demographic time series is important for understanding the mechanisms that regulate populations of animals and plants. We address this problem in a novel way by means of Statistical Learning Theory (SLT); SLT is built around the idea of VC-dimension, a complexity index for classes of parameterized functions. Though VC-dimensions of nonlinear models are generally unknown, in the linear case VC-dimension actually corresponds to the number of free parameters; this allows one to straightforwardly apply the model selection framework developed within SLT, and called Structural Risk Minimization (SRM). We generate noisy artificial time series, both DI and DD, and use SRM to recognize the model underlying the data, choosing among a suite of both density-dependent and independent demographies. We show that SRM significantly outperforms traditional model selection approaches, such as the Schwartz Information Criterion and Final Prediction Error in recognizing both density-dependence and independence.

Model selection for environmental data

AbstractA practical approach is proposed for model selection and discrimination among nested and non‐nested probability distributions. Some existing problems with traditional model selection approaches are addressed, including standard testing of a null hypothesis against a more general alternative and the use of some well‐known discrimination criteria for non‐nested distributions. A generalized information criterion (GIC) is used to choose from two or more model structures or probability distributions. For each set of random samples, all model structures that do not perform significantly worse than other candidates are selected. The two‐and three‐parameter gamma, Weibull and lognormal distributions are used to compare the discrimination procedures with traditional approaches. Monte Carlo experiments are employed to examine the performances of the criteria and tests over large sets of finite samples. For each distribution, the Monte Carlo procedure is undertaken for various representative sets of parameter values which are encountered in fitting environmental quality data.