Model Comparison Methods Research Articles

Statistical model comparison applied to common network motifs.

BackgroundNetwork motifs are small modules that show interesting functional and dynamic properties, and are believed to be the building blocks of complex cellular processes. However, the mechanistic details of such modules are often unknown: there is uncertainty about the motif architecture as well as the functional form and parameter values when converted to ordinary differential equations (ODEs). This translates into a number of candidate models being compatible with the system under study. A variety of statistical methods exist for ranking models including maximum likelihood-based and Bayesian methods. Our objective is to show how such methods can be applied in a typical systems biology setting.ResultsWe focus on four commonly occurring network motif structures and show that it is possible to differentiate between them using simulated data and any of the model comparison methods tested. We expand one of the motifs, the feed forward (FF) motif, for several possible parameterizations and apply model selection on simulated data. We then use experimental data on three biosynthetic pathways in Escherichia coli to formally assess how current knowledge matches the time series available. Our analysis confirms two of them as FF motifs. Only an expanded set of FF motif parameterisations using time delays is able to fit the third pathway, indicating that the true mechanism might be more complex in this case.ConclusionsMaximum likelihood as well as Bayesian model comparison methods are suitable for selecting a plausible motif model among a set of candidate models. Our work shows that it is practical to apply model comparison to test ideas about underlying mechanisms of biological pathways in a formal and quantitative way.

Cognitive mechanisms underlying risky decision-making in chronic cannabis users

Chronic cannabis users are known to be impaired on a test of decision-making, the Iowa Gambling Task (IGT). Computational models of the psychological processes underlying this impairment have the potential to provide a rich description of the psychological characteristics of poor performers within particular clinical groups. We used two computational models of IGT performance, the Expectancy Valence Learning model (EVL) and the Prospect Valence Learning model (PVL), to assess motivational, memory, and response processes in 17 chronic cannabis abusers and 15 control participants. Model comparison and simulation methods revealed that the PVL model explained the observed data better than the EVL model. Results indicated that cannabis abusers tended to be under-influenced by loss magnitude, treating each loss as a constant and minor negative outcome regardless of the size of the loss. In addition, they were more influenced by gains, and made decisions that were less consistent with their expectancies relative to non-using controls.

Dynamic causal modelling: A critical review of the biophysical and statistical foundations

The goal of dynamic causal modelling (DCM) of neuroimaging data is to study experimentally induced changes in functional integration among brain regions. This requires (i) biophysically plausible and physiologically interpretable models of neuronal network dynamics that can predict distributed brain responses to experimental stimuli and (ii) efficient statistical methods for parameter estimation and model comparison. These two key components of DCM have been the focus of more than thirty methodological articles since the seminal work of Friston and colleagues published in 2003. In this paper, we provide a critical review of the current state-of-the-art of DCM. We inspect the properties of DCM in relation to the most common neuroimaging modalities (fMRI and EEG/MEG) and the specificity of inference on neural systems that can be made from these data. We then discuss both the plausibility of the underlying biophysical models and the robustness of the statistical inversion techniques. Finally, we discuss potential extensions of the current DCM framework, such as stochastic DCMs, plastic DCMs and field DCMs.

Parameter estimation for fractional transport: A particle‐tracking approach

Space‐fractional advection‐dispersion models provide attractive alternatives to the classical advection‐dispersion equation for model applications that exhibit early arrivals and plume skewness. This paper develops a flexible method for estimating the parameters of the fractional transport model on the basis of spatial plume snapshots or temporal breakthrough curve data. A particle‐tracking approach provides error bars for the parameter estimates and a general method for model fitting and comparison via optimal weighted least squares. A simple model of concentration variance, based on the particle‐tracking approach, identifies the optimal weights.

Evaluation of the performance of three diffuse hourly irradiation models on tilted surfaces according to the utilizability concept

The performance of three diffuse hourly irradiation models on tilted surfaces was evaluated by making a database of hourly global and diffuse solar irradiation on a horizontal surface, as well as global solar irradiation on a tilted surface, recorded in a solar radiation station located at Córdoba University (Spain). The method for a comparison of the performance of these models was developed from a study of the “utilizable energy” statistics, a value representing, for a specific period of time, the mean monthly radiation that exceeded a critical level of radiation. This model comparison method seemed to us to be highly suitable since it provides a way of comparing the capacity of these models to estimate, however, much energy is incident on a tilted surface above a critical radiation level. Estimated and measured values were compared using the normalized RMBE and RRMSE statistics. According to the results of the method let us verify that, of the three models evaluated, one isotropic and two anisotropic, the Reindl et al. anisotropic model was the one giving the best results.

Non‐linear structural equation models with correlated continuous and discrete data

Structural equation models (SEMs) have been widely applied to examine interrelationships among latent and observed variables in social and psychological research. Motivated by the fact that correlated discrete variables are frequently encountered in practical applications, a non-linear SEM that accommodates covariates, and mixed continuous, ordered, and unordered categorical variables is proposed. Maximum likelihood methods for estimation and model comparison are discussed. One real-life data set about cardiovascular disease is used to illustrate the methodologies.

Comparing fire spread algorithms using equivalence testing and neutral landscape models

We demonstrate a method to evaluate the degree to which a meta-model approximates spatial disturbance processes represented by a more detailed model across a range of landscape conditions, using neutral landscapes and equivalence testing. We illustrate this approach by comparing burn patterns produced by a relatively simple fire spread algorithm with those generated by a more detailed fire behavior model from which the simpler algorithm was derived. Equivalence testing allows objective comparisons of the output of simple and complex models, to determine if the results are significantly similar. Neutral landscape models represent a range of landscape conditions that the model may encounter, allowing evaluation of the sensitivity and behavior of the model to different landscape compositions and configurations. We first tested the model for universal applicability, then narrowed the testing to assess the practical domain of applicability. As a whole, the calibrated simple model passed the test for significant equivalence using the 25% threshold. When applied to a range of landscape conditions different from the calibration scenarios, the model failed the tests for equivalence. Although our particular model failed the tests, the neutral landscape models were helpful in determining an appropriate domain of applicability and in assessing the model sensitivity to landscape changes. Equivalence testing provides an effective method for model comparison, and coupled with neutral landscapes, our approach provides an objective way to assess the domain of applicability of a spatial model.

Bayesian models identify specific lymphoproliferative disorders associated with hepatitis C virus infection

Bayesian models identify specific lymphoproliferative disorders associated with hepatitis C virus infection

Evaluation of quantitative models of the effect of insulin on lipolysis and glucose disposal

The effects of insulin on the suppression of lipolysis are neither fully understood nor quantified. We examined a variety of mathematical models analogous to the minimal model of glucose disposal (MMG) to quantify the combined influence of insulin on lipolysis and glucose disposal during an insulin-modified frequently sampled intravenous glucose tolerance test. The tested models, which include two previously published ones, consisted of separate compartments for plasma free fatty acids (FFA), glucose, and insulin. They differed in the number of compartments and in the action of insulin to suppress lipolysis that decreased the plasma FFA level. In one category of models, a single insulin compartment acted on both glucose and FFA simultaneously. In a second category, there were two insulin compartments, each acting on FFA and glucose independently. For each of these two categories, we tested 11 variations of how insulin suppressed lipolysis. We also tested a model with an additional glucose compartment that acted on FFA. These 23 models were fit to the plasma FFA and glucose concentrations of 102 subjects individually. Using Bayesian model comparison methods, we selected the model that best balanced fit and minimized model complexity. In the best model, insulin suppressed lipolysis via a Hill function through a remote compartment that acted on both glucose and FFA simultaneously, and glucose dynamics obeyed the classic MMG.

Learning, forecasting and structural breaks

AbstractWe provide a general methodology for forecasting in the presence of structural breaks induced by unpredictable changes to model parameters. Bayesian methods of learning and model comparison are used to derive a predictive density that takes into account the possibility that a break will occur before the next observation. Estimates for the posterior distribution of the most recent break are generated as a by‐product of our procedure. We discuss the importance of using priors that accurately reflect the econometrician's opinions as to what constitutes a plausible forecast. Several applications to macroeconomic time‐series data demonstrate the usefulness of our procedure. Copyright © 2008 John Wiley & Sons, Ltd.

The ΛCDM Model in the Lead—A Bayesian Cosmological Model Comparison

Recent astronomical observations indicate that our universe is undergoing a period of accelerated expansion. While there are many cosmological models that explain this phenomenon, the main question remains which is the best one in light of available data. We consider 10 cosmological models of the accelerating universe and select the best one using the Bayesian model comparison method. We demonstrate that the ΛCDM model is most favored by the Bayesian statistical analysis of the SNIa, CMB, baryon acoustic oscillation, and H(z) data.

Evaluation of the models handling heterotachy in phylogenetic inference

BackgroundThe evolutionary rate at a given homologous position varies across time. When sufficiently pronounced, this phenomenon – called heterotachy – may produce artefactual phylogenetic reconstructions under the commonly used models of sequence evolution. These observations have motivated the development of models that explicitly recognize heterotachy, with research directions proposed along two main axes: 1) the covarion approach, where sites switch from variable to invariable states; and 2) the mixture of branch lengths (MBL) approach, where alignment patterns are assumed to arise from one of several sets of branch lengths, under a given phylogeny.ResultsHere, we report the first statistical comparisons contrasting the performance of covarion and MBL modeling strategies. Using simulations under heterotachous conditions, we explore the properties of three model comparison methods: the Akaike information criterion, the Bayesian information criterion, and cross validation. Although more time consuming, cross validation appears more reliable than AIC and BIC as it directly measures the predictive power of a model on 'future' data. We also analyze three large datasets (nuclear proteins of animals, mitochondrial proteins of mammals, and plastid proteins of plants), and find the optimal number of components of the MBL model to be two for all datasets, indicating that this model is preferred over the standard homogeneous model. However, the covarion model is always favored over the optimal MBL model.ConclusionWe demonstrated, using three large datasets, that the covarion model is more efficient at handling heterotachy than the MBL model. This is probably due to the fact that the MBL model requires a serious increase in the number of parameters, as compared to two supplementary parameters of the covarion approach. Further improvements of the both the mixture and the covarion approaches might be obtained by modeling heterogeneous behavior both along time and across sites.

Statistical inference for atmospheric transport models using process convolutions

AbstractA computer simulator for atmospheric concentrations of chemical species, or chemical transport model, is used to simulate global ozone concentrations. Two different wind forcings are considered: one is a combination of a numerical weather prediction model and observational data, the other is obtained as output from a climate model. The goal is to study the impact of meteorological variability on ozone. The statistical approach that we consider consists on learning the spatial structure of ozone concentrations by using process convolutions. We use several Bayesian model comparison methods to determine if the two simulations can be considered as realizations of the same random field. The methods provide a quantification of the differences for each of the computer model grid cells. Copyright © 2007 John Wiley & Sons, Ltd.

Spatial Growth Regressions: Model Specification, Estimation and Interpretation

This paper uses Bayesian model comparison methods to simultaneously specify both the spatial weight structure and explanatory variables for a spatial growth regression involving 255 NUTS 2 regions across 25 European countries. In addition, a correct interpretation of the spatial regression parameter estimates that takes into account the simultaneous feedback nature of the spatial autoregressive model is provided. Our findings indicate that incorporating model uncertainty in conjunction with appropriate parameter interpretation decreased the importance of explanatory variables traditionally thought to exert an important influence on regional income growth rates.

Bayesian analysis of structural equation models with multinomial variables and an application to type 2 diabetic nephropathy

There is now increasing evidence proving that many complex diseases can be significantly influenced by correlated phenotype and genotype variables, as well as their interactions. Effective and rigorous assessment of such influence is difficult, because the number of phenotype and genotype variables of interest may not be small, and a genotype variable is an unordered categorical variable that follows a multinomial distribution. To address the problem, we establish a novel nonlinear structural equation model for analysing mixed continuous and multinomial data that can be missing at random. A confirmatory factor analysis model with Kronecker product is proposed for grouping the manifest continuous and multinomial variables into latent variables according to their functions; and a nonlinear structural equation is formulated to assess the linear and interaction effects of the independent latent variables to the dependent latent variables. Bayesian methods for estimation and model comparison are developed through Markov chain Monte Carlo techniques and path sampling. The newly developed methodologies are applied to a case-control cohort of type 2 diabetic patients with nephropathy.

Multivariate Stochastic Volatility: A Review

The literature on multivariate stochastic volatility (MSV) models has developed significantly over the last few years. This paper reviews the substantial literature on specification, estimation, and evaluation of MSV models. A wide range of MSV models is presented according to various categories, namely, (i) asymmetric models, (ii) factor models, (iii) time-varying correlation models, and (iv) alternative MSV specifications, including models based on the matrix exponential transformation, the Cholesky decomposition, and the Wishart autoregressive process. Alternative methods of estimation, including quasi-maximum likelihood, simulated maximum likelihood, and Markov chain Monte Carlo methods, are discussed and compared. Various methods of diagnostic checking and model comparison are also reviewed.

Scanned Three-Dimensional Model Matching and Comparison Algorithms for Manufacturing Applications

In recent years the increased use of 3D scanning hardware has introduced a new type of data to the design and manufacturing field. In many design and manufacturing applications (e.g., part refurbishing or remanufacturing) a scanned 3D model may be provided as an input to a shape matching system to search the database for related or identical models with the purpose of extracting useful information. The introduction of scanned 3D models restricts the use of the CAD-based 3D model search and comparison methods due to significant differences in model representations. The CAD models provide structured and high-level representation of the part features, whereas the scanned 3D models usually come in a polygonal mesh representation, which does not directly reveal engineering features of the part. These differences require new algorithms for comparing the shapes of scanned 3D models, ones that are robust against different scanning technologies and can be adjusted to work with different representations of the models. In this paper, a new approach and algorithms for scanned 3D shape matching and comparison are presented. Given the scanned 3D model as an input the approach first uses general-purpose shape matching methods to identify a small list of likely matches (i.e., candidate models) for more detailed shape comparison. To perform detailed comparison of the shapes each candidate model is geometrically adjusted (i.e., rotated and translated) with the input using one of two new viewpoint algorithms developed in this paper. Once the candidate models are adjusted they are compared to the input to identify the similarities and differences between their shapes. To accomplish this task a new 3D shape matching algorithm is developed. The relevance of the methodology developed in this paper is illustrated with the application of scanned 3D shape matching and comparison algorithms in rapid manufacturing of broken parts.

A maximum likelihood framework for protein design

BackgroundThe aim of protein design is to predict amino-acid sequences compatible with a given target structure. Traditionally envisioned as a purely thermodynamic question, this problem can also be understood in a wider context, where additional constraints are captured by learning the sequence patterns displayed by natural proteins of known conformation. In this latter perspective, however, we still need a theoretical formalization of the question, leading to general and efficient learning methods, and allowing for the selection of fast and accurate objective functions quantifying sequence/structure compatibility.ResultsWe propose a formulation of the protein design problem in terms of model-based statistical inference. Our framework uses the maximum likelihood principle to optimize the unknown parameters of a statistical potential, which we call an inverse potential to contrast with classical potentials used for structure prediction. We propose an implementation based on Markov chain Monte Carlo, in which the likelihood is maximized by gradient descent and is numerically estimated by thermodynamic integration. The fit of the models is evaluated by cross-validation. We apply this to a simple pairwise contact potential, supplemented with a solvent-accessibility term, and show that the resulting models have a better predictive power than currently available pairwise potentials. Furthermore, the model comparison method presented here allows one to measure the relative contribution of each component of the potential, and to choose the optimal number of accessibility classes, which turns out to be much higher than classically considered.ConclusionAltogether, this reformulation makes it possible to test a wide diversity of models, using different forms of potentials, or accounting for other factors than just the constraint of thermodynamic stability. Ultimately, such model-based statistical analyses may help to understand the forces shaping protein sequences, and driving their evolution.

A Maximum Likelihood Approach for Multisample Nonlinear Structural Equation Models With Missing Continuous and Dichotomous Data

Structural equation models are widely appreciated in social-psychological research and other behavioral research to model relations between latent constructs and manifest variables and to control for measurement error. Most applications of SEMs are based on fully observed continuous normal data and models with a linear structural equation. However, discrete nonnormal data and missing data are rather common, and sometimes it is necessary to incorporate nonlinear structural equations for assessing the impact of nonlinear terms of the exogenous latent variables to the endogenous latent variables. Moreover, to study the behaviors of different populations, it is necessary to extend from a single sample model to a multisample model. In this article, a maximum likelihood (ML) approach is developed for analyzing a multisample nonlinear structural equation model, in the context of mixed continuous and dichotomous data that involve data that are missing at random. The article demonstrates the newly developed methods for estimation and model comparison by a simulation study, a synthetic data application, and a real application to study latent psychological constructs of job attitude, emotion, and benefit attitude that are related to organizational and management research.

Bayesian model comparison and selection in energy decay analysis of acoustically coupled spaces

The use of Bayesian inference in problems of parameter estimation from noisy data and in uncertain environments has been well discussed in the acoustical signal processing literature. In many acoustical problems where it is uncertain which suitable model among a set of competing ones should be used, the model comparison and selection become crucial prior to the actual parameter estimation. However, tools for model selection in Bayesian inference have received less attention. In problems of model comparison and selection, the Bayesian methodology is most different from orthodox statistical methods. We present several methods for model comparison and selection from a Bayesian viewpoint and demonstrate their application to the problem of estimating the number of decay rates present in acoustically coupled spaces.