Formal Model Selection Research Articles

Shedding light on uncertainties in machine learning: formal derivation and optimal model selection

Shedding light on uncertainties in machine learning: formal derivation and optimal model selection

To Sample or Not to Sample: Retrieving Exoplanetary Spectra with Variational Inference and Normalizing Flows

Current endeavours in exoplanet characterization rely on atmospheric retrieval to quantify crucial physical properties of remote exoplanets from observations. However, the scalability and efficiency of said technique are under strain with increasing spectroscopic resolution and forward model complexity. The situation has become more acute with the recent launch of the James Webb Space Telescope and other upcoming missions. Recent advances in machine learning provide optimization-based variational inference as an alternative approach to perform approximate Bayesian posterior inference. In this investigation we developed a normalizing-flow-based neural network, combined with our newly developed differentiable forward model, Diff-τ, to perform Bayesian inference in the context of atmospheric retrievals. Using examples from real and simulated spectroscopic data, we demonstrate the advantages of our proposed framework: (1) training our neural network does not require a large precomputed training set and can be trained with only a single observation; (2) it produces high-fidelity posterior distributions in excellent agreement with sampling-based retrievals; (3) it requires up to 75% fewer forward model calls to converge to the same result; and (4) this approach allows formal Bayesian model selection. We discuss the computational efficiencies of Diff-τ in relation to TauREx3's nominal forward model and provide a “lessons learned” account of developing radiative transfer models in differentiable languages. Our proposed framework contributes toward the latest development of neural network–powered atmospheric retrieval. Its flexibility and significant reduction in forward model calls required for convergence holds the potential to be an important addition to the retrieval tool box for large and complex data sets along with sampling-based approaches.

A model selection approach to structural equation modelling: A critical evaluation and a road map for ecologists

Abstract Structural equation modelling (SEM) can illuminate complex interaction networks of the sort found in ecology. However, selecting optimally complex, data‐supported SEM models and quantifying their uncertainty are difficult processes. To this end, we recommend a formal model selection approach (MSA) that uses information criteria. Using a suite of numerical simulations, we compare MSA‐SEM against two traditional methods. We find that MSA‐SEM exhibits superior, unbiased results under the suboptimal realistic conditions characteristic of ecological studies. We then provide a road map for MSA‐SEM and demonstrate its use via a case study. We illustrate the unique abilities of SEM to confirm a network structure within the realm of known causal pathways and delineate the boundaries within which MSA‐SEM should be applied.

Model comparison via simplicial complexes and persistent homology.

In many scientific and technological contexts, we have only a poor understanding of the structure and details of appropriate mathematical models. We often, therefore, need to compare different models. With available data we can use formal statistical model selection to compare and contrast the ability of different mathematical models to describe such data. There is, however, a lack of rigorous methods to compare different models a priori. Here, we develop and illustrate two such approaches that allow us to compare model structures in a systematic way by representing models as simplicial complexes. Using well-developed concepts from simplicial algebraic topology, we define a distance between models based on their simplicial representations. Employing persistent homology with a flat filtration provides for alternative representations of the models as persistence intervals, which represent model structure, from which the model distances are also obtained. We then expand on this measure of model distance to study the concept of model equivalence to determine the conceptual similarity of models. We apply our methodology for model comparison to demonstrate an equivalence between a positional-information model and a Turing-pattern model from developmental biology, constituting a novel observation for two classes of models that were previously regarded as unrelated.

Model selection in time series analysis: using information criteria as an alternative to hypothesis testing

PurposeThe issue of model selection in applied research is of vital importance. Since the true model in such research is not known, which model should be used from among various potential ones is an empirical question. There might exist several competitive models. A typical approach to dealing with this is classic hypothesis testing using an arbitrarily chosen significance level based on the underlying assumption that a true null hypothesis exists. In this paper, the authors investigate how successful the traditional hypothesis testing approach is in determining the correct model for different data generating processes using time series data. An alternative approach based on more formal model selection techniques using an information criterion or cross-validation is also investigated.Design/methodology/approachMonte Carlo simulation experiments on various generating processes are used to look at the response surfaces resulting from hypothesis testing and response surfaces resulting from model selection based on minimizing an information criterion or the leave-one-out cross-validation prediction error.FindingsThe authors find that the minimization of an information criterion can work well for model selection in a time series environment, often performing better than hypothesis-testing strategies. In such an environment, the use of an information criterion can help reduce the number of models for consideration, but the authors recommend the use of other methods also, including hypothesis testing, to determine the appropriateness of a model.Originality/valueThis paper provides an alternative approach for selecting the best potential model among many for time series data. It demonstrates how minimizing an information criterion can be useful for model selection in a time-series environment in comparison to some standard hypothesis testing strategies.

Assessing the Significance of Model Selection in Ecology

Model Selection is a key part of many ecological studies, with Akaike’s Information Criterion (AIC) being by far the most commonly used technique for this purpose. Typically, a number of candidate models are defined a priori and ranked according to their expected out-of-sample performance. Model selection, however, only assesses the relative performance of the models and, as pointed out in a recent paper, a large proportion of ecology papers that use model selection do not assess the absolute fit of the ‘best’ model. In this paper, it is argued that assessing the absolute fit of the ‘best’ model alone does not go far enough. This is because a model that appears to perform well under model selection is also likely to appear to perform well under measures of absolute fit, even when there is no predictive value. A model selection permutation test is proposed that assesses the probability that the model selection statistic of the ‘best’ model could have occurred by chance alone, whilst taking account of dependencies between the models. It is argued that this test should always be performed as a part of formal model selection. The test is demonstrated on two real population modelling examples of ibex in northern Italy and wild reindeer in Norway.

Assessing model mismatch and model selection in a Bayesian uncertainty quantification analysis of a fluid-dynamics model of pulmonary blood circulation.

This study uses Bayesian inference to quantify the uncertainty of model parameters and haemodynamic predictions in a one-dimensional pulmonary circulation model based on an integration of mouse haemodynamic and micro-computed tomography imaging data. We emphasize an often neglected, though important source of uncertainty: in the mathematical model form due to the discrepancy between the model and the reality, and in the measurements due to the wrong noise model (jointly called ‘model mismatch’). We demonstrate that minimizing the mean squared error between the measured and the predicted data (the conventional method) in the presence of model mismatch leads to biased and overly confident parameter estimates and haemodynamic predictions. We show that our proposed method allowing for model mismatch, which we represent with Gaussian processes, corrects the bias. Additionally, we compare a linear and a nonlinear wall model, as well as models with different vessel stiffness relations. We use formal model selection analysis based on the Watanabe Akaike information criterion to select the model that best predicts the pulmonary haemodynamics. Results show that the nonlinear pressure–area relationship with stiffness dependent on the unstressed radius predicts best the data measured in a control mouse.

Model selection for the North American Breeding Bird Survey.

The North American Breeding Bird Survey (BBS) provides data that can be used in complex, multiscale analyses of population change, while controlling for scale-specific nuisance factors. Many alternative models can be fit to the data, but most model selection procedures are not appropriate for hierarchical models. Leave-one-out cross-validation (LOOCV), in which relative model fit is assessed by omitting an observation and assessing the prediction of a model fit using the remainder of the data, provides a reasonable approach for assessing models, but is time consuming and not feasible to apply for all observations in large data sets. We report the first large-scale formal model selection for BBS data, applying LOOCV to stratified random samples of observations from BBS data. Our results are for 548 species of North American birds, comparing the fit of four alternative models that differ in year effect structures and in descriptions of extra-Poisson overdispersion. We use a hierarchical model among species to evaluate posterior probabilities that models are best for individual species. Models in which differences in year effects are conditionally independent (D models) were generally favored over models in which year effects are modeled by a slope parameter and a random year effect (S models), and models in which extra-Poisson overdispersion effects are independent and t-distributed (H models) tended to be favored over models where overdispersion was independent and normally distributed. Our conclusions lead us to recommend a change from the conventional S model to D and H models for the vast majority of species (544/548). Comparison of estimated population trends based on the favored model relative to the S model currently used for BBS summaries indicates no consistent differences in estimated trends. Of the 18 species that showed large differences in estimated trends between models, estimated trends from the default S model were more extreme, reflecting the influence of the slope parameter in that model for species that are undergoing large population changes. WAIC, a computationally simpler alternative to LOOCV, does not appear to be a reliable alternative to LOOCV.

Credit spread and employment growth – a time-varying relationship?

ABSTRACT In this article, I explore time variation in the relationship between the credit spread and employment growth. Using a Bayesian VAR framework and formal model selection, it is concluded that the relationship is best modelled with constant parameters, but that heteroscedasticity needs to be taken into account. An interesting change in the dynamics is uncovered, where volatility in employment shocks induces more of the dynamics earlier in the sample and the role of volatility in credit-spread shocks is more pronounced in the latter parts.

The relation between the corporate bond-yield spread and the real economy: Stable or time-varying?

In this paper we assess whether the relation between the corporate bond-yield spread and the real economy has been stable over time. Using quarterly US data from 1953Q1 to 2018Q2, we estimate Bayesian VAR models which allow for drifting parameters and/or stochastic volatility and conduct formal model selection in a Bayesian setting. Our results indicate that the relation between the variables has been stable; we do, however, find strong support for stochastic volatility. We conclude that the corporate bond-yield spread’s usefulness for predicting real economic activity has not changed to a relevant extent after the Great Recession.

Biased pain reports through vicarious information: A computational approach to investigate the role of uncertainty

Expectations about an impeding pain stimulus strongly shape its perception, yet the degree that uncertainty might affect perception is far less understood. To explore the influence of uncertainty on pain ratings, we performed a close replication of the study of Yoshida, Seymour, Koltzenburg, and Dolan (2013), who manipulated vicarious information about upcoming heat pain and found evidence for uncertainty-induced hyperalgesia. In our study, we presented eight fictitious ratings of previous participants prior the delivery of electrocutaneous pain. The vicarious information was either biased to over- or underreport pain levels based on the participant’s psychometric function. We induced uncertainty by manipulating the variation of the vicarious information. As in Yoshida et al. (2013), four computational models were formulated, such that each model represented a different way of how the pain ratings might have been generated by the physical stimulus and the vicarious information. The four competing models were tested against the data of each participant separately. Using a formal model selection criterion, the best model was selected and interpreted. Contrary to the original study, the preferred model for the majority of participants suggested that pain ratings were biased towards the average vicarious information, ignoring the degree of uncertainty. Possible reasons for these diverging results are discussed.

Model selection with estimated factors and idiosyncratic components

SummaryThis paper provides consistent information criteria for the selection of forecasting models that use a subset of both the idiosyncratic and common factor components of a big dataset. This hybrid model approach has been explored by recent empirical studies to relax the strictness of pure factor‐augmented model approximations, but no formal model selection procedures have been developed. The main difference to previous factor‐augmented model selection procedures is that we must account for estimation error in the idiosyncratic component as well as the factors. Our main contribution is to show the conditions required for selection consistency of a class of information criteria that reflect this additional source of estimation error. We show that existing factor‐augmented model selection criteria are inconsistent in circumstances where N is of larger order than , where N and T are the cross‐section and time series dimensions of the dataset respectively, and that the standard Bayesian information criterion is inconsistent regardless of the relationship between N and T. We therefore propose a new set of information criteria that guarantee selection consistency in the presence of estimated idiosyncratic components. The properties of these new criteria are explored through a Monte Carlo simulation study. The paper concludes with an empirical application to long‐horizon exchange rate forecasting using a recently proposed model with country‐specific idiosyncratic components from a panel of global exchange rates.

On the emergence of heavy-tailed streamflow distributions

The right tail of streamflow distributions quantifies the occurrence probability of high flows, which play an important role in the dynamics of many eco-hydrological processes and eventually contribute to shape riverine environments. In this paper, the ability of a mechanistic analytical model for streamflow distributions to capture the statistical features of high flows has been investigated. The model couples a stochastic description of soil moisture dynamics with a simplified hydrologic response based on a catchment-scale storage–discharge relationship. Different types of relations between catchment water storage and discharge have been investigated, and alternative methods for parameter estimation have been compared using informal performance metrics and formal model selection criteria. The study highlights the pivotal role of non-linear storage–discharge relations in reproducing observed frequencies of high flows, and reveals the importance of analyzing the behavior of individual events for a reliable characterization of recession parameters. The emergence of heavy-tailed streamflow distributions is mechanistically linked to the degree of non-linearity of the catchment hydrologic response, with implications for the understanding of rivers’ flooding potential and related ecologic and morphological processes.

Kinetic model selection and the Hill model in geochemistry

This study introduces the Hill model for modelling sorption kinetics and illustrates its efficacy using formal model selection procedures. Although the coefficient of determination (R 2) value can be used to ensure goodness-of-fit of a particular model to a set of data, a measure that incorporates the number of model parameters, such as the Bayesian information criterion, law of iterated logarithm criterion, Akaike information criterion, and the extra sum-of-squares F-test is necessary to undertake reliable model selection. This study undertook a formal model selection procedure based on these criteria, in combination with the coefficient of determination, for a range of candidate kinetics sorption models [the pseudo-second order (PSO), the intra-particle diffusion (IPD), Power, Elovich, and the four and five parameter Hill models]. It was found that the model order that statistically best described the data was Hill 5 > Hill 4 > pseudo-second order > Elovich > Power > IPD as given by the evidence ratios, information criterion weights and Bayes factors. The extra sum-of-squares F-test, only applicable to nested models, confirms the best fit order as Hill 5 > Hill 4 > PSO models. Two new parameters (T and k Hill) have also been defined that give the Hill models a mechanistic meaning directly comparable to the well-defined parameters of the pseudo-second-order model, namely, the pseudo-second-order rate constant k pso and the instantaneous sorption coefficient h pso. In the field of geochemistry the use of the four and five parameter Hill models for describing sorption kinetics has been overlooked with many studies favouring the PSO model. It is strongly recommended that the Hill model be considered more widely as a general predictive tool in geochemistry.

Pathogen growth in insect hosts: inferring the importance of different mechanisms using stochastic models and response-time data.

Pathogen population dynamics within individual hosts can alter disease epidemics and pathogen evolution, but our understanding of the mechanisms driving within-host dynamics is weak. Mathematical models have provided useful insights, but existing models have only rarely been subjected to rigorous tests, and their reliability is therefore open to question. Most models assume that initial pathogen population sizes are so large that stochastic effects due to small population sizes, so-called demographic stochasticity, are negligible, but whether this assumption is reasonable is unknown. Most models also assume that the dynamic effects of a host's immune system strongly affect pathogen incubation times or "response times," but whether such effects are important in real host-pathogen interactions is likewise unknown. Here we use data for a baculovirus of the gypsy moth to test models of within-host pathogen growth. By using Bayesian statistical techniques and formal model-selection procedures, we are able to show that the response time of the gypsy moth virus is strongly affected by both demographic stochasticity and a dynamic response of the host immune system. Our results imply that not all response-time variability can be explained by host and pathogen variability, and that immune system responses to infection may have important effects on population-level disease dynamics.

Model selection criterion for causal parameters in structural mean models based on a quasi‐likelihood

Structural mean models (SMMs) have been proposed for estimating causal parameters in the presence of non-ignorable non-compliance in clinical trials. To obtain a valid causal estimate, we must impose several assumptions. One of these is the correct specification of the structural model. Building on Pan's work (2001, Biometrics 57, 120-125) on developing a model selection criterion for generalized estimating equations, we propose a new approach for model selection of SMMs based on a quasi-likelihood. We provide a formal model selection criterion that is an extension of Akaike's information criterion. Using subset selection of baseline covariates, our method allows us to understand whether the treatment effect varies across the available baseline covariate levels, and/or to quantify the treatment effect on a specific covariates level to target specific individuals to maximize treatment benefit. We present simulation results in which our method performs reasonably well compared to other testing methods in terms of both the probability of selecting the correct model and the predictive performances of the individual treatment effects. We use a large randomized clinical trial of pravastatin as a motivation.

Assessing the State of Substitution Models Describing Noncoding RNA Evolution

Phylogenetic inference is widely used to investigate the relationships between homologous sequences. RNA molecules have played a key role in these studies because they are present throughout life and tend to evolve slowly. Phylogenetic inference has been shown to be dependent on the substitution model used. A wide range of models have been developed to describe RNA evolution, either with 16 states describing all possible canonical base pairs or with 7 states where the 10 mismatched nucleotides are reduced to a single state. Formal model selection has become a standard practice for choosing an inferential model and works well for comparing models of a specific type, such as comparisons within nucleotide models or within amino acid models. Model selection cannot function across different sized state spaces because the likelihoods are conditioned on different data. Here, we introduce statistical state-space projection methods that allow the direct comparison of likelihoods between nucleotide models and 7-state and 16-state RNA models. To demonstrate the general applicability of our new methods, we extract 287 RNA families from genomic alignments and perform model selection. We find that in 281/287 families, RNA models are selected in preference to nucleotide models, with simple 7-state RNA models selected for more conserved families with shorter stems and more complex 16-state RNA models selected for more divergent families with longer stems. Other factors, such as the function of the RNA molecule or the GC-content, have limited impact on model selection. Our models and model selection methods are freely available in the open-source PHASE 3.0 software.

Development of an actuarial static risk model suitable for automatic scoring for predicting juvenile recidivism

ObjectivesTo test the feasibility of an actuarial model for juvenile offending suitable for automatic scoring.DesignWe identified a nationally representative sample of 936 young persons aged 13–17 (745 male, 191 female) who received a juvenile justice intake in 2002 in New Zealand.MethodsBest‐subsets logistic regression and a formal model selection criterion were used to generate a predictive model for reoffending, and a conservative estimate of accuracy was obtained with cross‐validation.ResultsRecidivism during a 1‐year follow‐up was significantly higher for male (60.8%) compared to female (46.6%) delinquents. The model showed that young persons who were male, younger at their first social welfare intake, and had more prior court dates and a greater frequency of contact with police, were more likely to re‐offend. The accuracy of the model was moderately high (area under the receiver operating characteristic curve = .710). A model developed specifically for the female cases failed to provide a significant increase in predictive accuracy.ConclusionsThese results demonstrate the feasibility of an actuarial model for juvenile offending that is suitable for automatic scoring. Although male delinquents pose a higher absolute risk of juvenile offending than female delinquents, a common set of items related to history of contact with police and social welfare agencies provide a similarly accurate measure of relative risk for both sexes.

Biotic and abiotic correlates of tissue quality for common scleractinian corals

MEPS Marine Ecology Progress Series Contact the journal Facebook Twitter RSS Mailing List Subscribe to our mailing list via Mailchimp HomeLatest VolumeAbout the JournalEditorsTheme Sections MEPS 438:119-128 (2011) - DOI: https://doi.org/10.3354/meps09271 Biotic and abiotic correlates of tissue quality for common scleractinian corals M. O. Hoogenboom1,*, S. R. Connolly1,2, K. R. N. Anthony3 1ARC Centre of Excellence for Coral Reef Studies, James Cook University, Townsville, Queensland 4811, Australia 2School of Marine and Tropical Biology, James Cook University, Townsville, Queensland 4811, Australia 3ARC Centre of Excellence for Coral Reef Studies, and School of Biological Sciences, University of Queensland, Brisbane 4072, Australia *Email: mia.hoogenboom@gmail.com ABSTRACT: Reef-building corals provide critical ecosystem functions but are under pressure from a range of stressors, and there is a growing need to understand how environmental factors affect the physiological condition of corals. This study explored the relationships between tissue quality (i.e. energy content per unit surface area) and light intensity, water flow velocity, temperature, colony size and competition for 4 common coral species. A formal model selection procedure revealed that different corals, even species that are closely related and morphologically similar, respond to their local environments in fundamentally different ways. Overall, colony size was the most consistent predictor of tissue quality, and the observed increase in tissue energy content with size may partially explain field observations of reduced mortality for large colonies. However, competitive interactions, light intensity, water flow and temperature also contributed to variation in tissue properties. Warming reduced the tissue quality of massive and foliose corals more than that of branching corals, despite the latter generally being recognised as more susceptible to thermal bleaching. These findings identify species- and habitat-specific responses to baseline environmental variables that can be used to anticipate shifts in coral health under environmental change. KEY WORDS: Light intensity · Water flow · Colony size · Temperature · Competition · Bioenergetics · Statistical modelling · Model-average regression coefficient Full text in pdf format Supplementary material PreviousNextCite this article as: Hoogenboom MO, Connolly SR, Anthony KRN (2011) Biotic and abiotic correlates of tissue quality for common scleractinian corals. Mar Ecol Prog Ser 438:119-128. https://doi.org/10.3354/meps09271 Export citation RSS - Facebook - Tweet - linkedIn Cited by Published in MEPS Vol. 438. Online publication date: October 05, 2011 Print ISSN: 0171-8630; Online ISSN: 1616-1599 Copyright © 2011 Inter-Research.

Determining Epidermal Disposition Kinetics for Use in an Integrated Nonanimal Approach to Skin Sensitization Risk Assessment

Development of risk assessment methods for skin sensitization in the absence of toxicological data generated in animals represents a major scientific and technical challenge. The first step in human skin sensitization induction is the transport of sensitizer from the applied dose on the skin surface to the epidermis, where innate immune activation occurs. Building on the previous development of a time course in vitro human skin permeation assay, new kinetic data for 10 sensitizers and 2 nonsensitizers are reported. Multicompartmental modeling has been applied to analyze the data and determine candidate dose parameters for use in integrated risk assessment methods: the area under the curve (AUC) and maximum concentration (C(max)) in the epidermis. A model with two skin compartments, representing the stratum corneum and viable skin (epidermis and dermis), was chosen following a formal model selection process. Estimates of the uncertainty, as well as average values of the epidermal disposition kinetics parameters, were made by fitting to the time course skin permeation data from individual skin donors. A potential reduced time course method is proposed based on two time points at 4 and 24 h, which gives results close to those from the full time course for the current data sets. The time course data presented in this work have been provided as a resource for development of predictive in silico skin permeation models.