Model Selection Issues Research Articles

Bayesian Lassos for spatial durbin error model with smoothness prior: Application to detect spillovers of China's treaty ports

In this paper we consider a generalized spatial durbin error model with spatial spillovers that are flexible and subject to abrupt change arising from zero spillover effects. We propose a new class of Bayesian lasso-type prior, the Bayesian elastic net with smoothness prior, to both tackle the multicollinearity among spatial lags of explanatory variables and capture both zero and nonzero spillover effects. We develop a computationally tractable Markov Chain Monte Carlo (MCMC) algorithm to estimate the model under the new prior. We also study the corresponding model selection issue among the generalized spatial durbin error model and its two special cases: 1) the spatial error model and 2) the spatial autoregressive type model. Simulation results suggest that the new prior can outperform many existing Bayesian priors in terms of in-sample predictive performance. We apply the model with the new prior to investigate the influence and spillover effects of China's historical treaty ports in the 19th century on prefectures' population and economic growth in the long-run. We find that prefectures with treaty ports tend to grow faster in terms of population size and GDP level. We also detect positive and significant spillovers when one's neighbors become treaty ports.

Reliability Analysis Using Adaptive Kriging Surrogates with Multimodel Inference

AbstractThis work addresses the issue of model selection in adaptive kriging-based Monte Carlo reliability analysis. It is shown that arbitrary model selection (kriging trend and correlation) can l...

Model selection and parameter estimation of dynamical systems using a novel variant of approximate Bayesian computation

Abstract Model selection is a challenging problem that is of importance in many branches of the sciences and engineering, particularly in structural dynamics. By definition, it is intended to select the most likely model among a set of competing models that best matches the dynamic behaviour of a real structure and better predicts the measured data. The Bayesian approach which is based essentially on the evaluation of a likelihood function is one of the most popular approach to deal with model selection and parameter estimation issues. However, in some circumstances, the likelihood function is either intractable or not available even in a closed form. To overcome this issue, the likelihood-free or approximate Bayesian computation (ABC) algorithm has been introduced in the literature, which relaxes the need for an explicit likelihood function to measure the level of agreement between model predictions and measurements. However, ABC algorithms suffer from a low acceptance rate of samples which is actually a common problem with the traditional Bayesian methods. To overcome this shortcoming and alleviate the computational burden, a new variant of the ABC algorithm based on an ellipsoidal Nested Sampling (NS) technique is introduced in this paper; it has been called ABC-NS. Through this paper, it will be shown how the new algorithm is a promising alternative to deal with parameter estimation and model selection issues. It promises drastic speedups and provides a good approximation of the posterior distributions. To demonstrate its robust computational efficiency, four illustrative examples are given. Firstly, the efficiency of the algorithm is demonstrated to deal with parameter estimation. Secondly, two examples based on simulated and real data are given to demonstrate the efficiency of the algorithm to deal with model selection in structural dynamics.

Sequential Dirichlet process mixtures of multivariate skew $t$-distributions for model-based clustering of flow cytometry data

Flow cytometry is a high-throughput technology used to quantify multiple surface and intracellular markers at the level of a single cell. This enables to identify cell sub-types, and to determine their relative proportions. Improvements of this technology allow to describe millions of individual cells from a blood sample using multiple markers. This results in high-dimensional datasets, whose manual analysis is highly time-consuming and poorly reproducible. While several methods have been developed to perform automatic recognition of cell populations, most of them treat and analyze each sample independently. However, in practice, individual samples are rarely independent (e.g. longitudinal studies). Here, we propose to use a Bayesian nonparametric approach with Dirichlet process mixture (DPM) of multivariate skew $t$-distributions to perform model based clustering of flow-cytometry data. DPM models directly estimate the number of cell populations from the data, avoiding model selection issues, and skew $t$-distributions provides robustness to outliers and non-elliptical shape of cell populations. To accommodate repeated measurements, we propose a sequential strategy relying on a parametric approximation of the posterior. We illustrate the good performance of our method on simulated data, on an experimental benchmark dataset, and on new longitudinal data from the DALIA-1 trial which evaluates a therapeutic vaccine against HIV. On the benchmark dataset, the sequential strategy outperforms all other methods evaluated, and similarly, leads to improved performance on the DALIA-1 data. We have made the method available for the community in the R package NPflow.

Heteroscedasticity and Model Selection via Partitioning in Fisheries Data

Selecting a proper model for a data set is a challenging task. In this article, an attempt was made to answer and to find a suitable model for a given data set. A general linear model (GLM) was introduced along with three different methods for estimating the parameters of the model. The three estimation methods considered in this paper were ordinary least squares (OLS), generalized least squares (GLS), and feasible generalized least squares (FGLS). In the case of GLS, two different weights were selected for improving the severity of heteroscedasticity and the proper weight (s) was deployed. The third weight was selected through the application of FGLS. Analyses showed that only two of the three weights including the FGLS were effective in improving or reducing the severity of heteroscedasticity. In addition, each data set was divided into Training, Validation, and Testing producing a more reliable set of estimates for the parameters in the model. Partitioning data is a relatively new approach is statistics borrowed from the field of machine learning. Stepwise and forward selection methods along with a number of statistics including the average square error testing (ASE), Adj. R-Sq, AIC, AICC, and ASE validate along with proper hierarchies were deployed to select a more appropriate model(s) for a given data set. Furthermore, the response variable in both data files was transformed using the Box-Cox method to meet the assumption of normality. Analysis showed that the logarithmic transformation solved this issue in a satisfactory manner. Since the issues of heteroscedasticity, model selection, and partitioning of data have not been addressed in fisheries, for introduction and demonstration purposes only, the 2015 and 2016 shrimp data in the Gulf of Mexico (GOM) were selected and the above methods were applied to these data sets. At the conclusion, some variations of the GLM were identified as possible leading candidates for the above data sets.

Model selection in geological and geotechnical engineering in the face of uncertainty - Does a complex model always outperform a simple model?

In order to achieve higher fidelity (i.e., higher accuracy) in the model predictions, the solution models employed in geological and/or geotechnical engineering are becoming complex and sophisticated. However, simple and robust models are preferred by engineers in practice. Thus, a dilemma exists between the choice of a complex model for fidelity and that of a simple model for high robustness (i.e., lower variation in the discrepancy between model prediction and observation). This issue becomes more profound when model parameters exhibit uncertainty, which is quite common in geological and geotechnical problems. In this paper, we examine the issue of model selection in the face of uncertainty with three problems: the selection of the order of polynomial fit (i.e., lower order vs. higher order) in the development of data-driven empirical model, the selection of the level of sophistication (i.e., random variable vs. random field) in the probabilistic characterization of the detrended soil property, and the selection of the level of complexity (i.e., simple vs. complex) of the soil constitutive model in numerical modelling. The results illustrate that although the complex and sophisticated models could yield predictions that are more accurate, the simple models might yield predictions that are more robust. This paper provides an insight regarding the question, “Does a complex model always outperform a simple model?”

Big-data and machine learning to revamp computational toxicology and its use in risk assessment.

The creation of large toxicological databases and advances in machine-learning techniques have empowered computational approaches in toxicology. Work with these large databases based on regulatory data has allowed reproducibility assessment of animal models, which highlight weaknesses in traditional in vivo methods. This should lower the bars for the introduction of new approaches and represents a benchmark that is achievable for any alternative method validated against these methods. Quantitative Structure Activity Relationships (QSAR) models for skin sensitization, eye irritation, and other human health hazards based on these big databases, however, also have made apparent some of the challenges facing computational modeling, including validation challenges, model interpretation issues, and model selection issues. A first implementation of machine learning-based predictions termed REACHacross achieved unprecedented sensitivities of >80% with specificities >70% in predicting the six most common acute and topical hazards covering about two thirds of the chemical universe. While this is awaiting formal validation, it demonstrates the new quality introduced by big data and modern data-mining technologies. The rapid increase in the diversity and number of computational models, as well as the data they are based on, create challenges and opportunities for the use of computational methods.

Extended models for nosocomial infection: parameter estimation and model selection.

We consider extensions to previous models for patient level nosocomial infection in several ways, provide a specification of the likelihoods for these new models, specify new update steps required for stochastic integration, and provide programs that implement these methods to obtain parameter estimates and model choice statistics. Previous susceptible-infected models are extended to allow for a latent period between initial exposure to the pathogen and the patient becoming themselves infectious, and the possibility of decolonization. We allow for multiple facilities, such as acute care hospitals or long-term care facilities and nursing homes, and for multiple units or wards within a facility. Patient transfers between units and facilities are tracked and accounted for in the models so that direct importation of a colonized individual from one facility or unit to another might be inferred. We allow for constant transmission rates, rates that depend on the number of colonized individuals in a unit or facility, or rates that depend on the proportion of colonized individuals. Statistical analysis is done in a Bayesian framework using Markov chain Monte Carlo methods to obtain a sample of parameter values from their joint posterior distribution. Cross validation, deviance information criterion and widely applicable information criterion approaches to model choice fit very naturally into this framework and we have implemented all three. We illustrate our methods by considering model selection issues and parameter estimation for data on methicilin-resistant Staphylococcus aureus surveillance tests over 1 year at a Veterans Administration hospital comprising seven wards.

Model selection and parameter estimation in structural dynamics using approximate Bayesian computation

This paper will introduce the use of the approximate Bayesian computation (ABC) algorithm for model selection and parameter estimation in structural dynamics. ABC is a likelihood-free method typically used when the likelihood function is either intractable or cannot be approached in a closed form. To circumvent the evaluation of the likelihood function, simulation from a forward model is at the core of the ABC algorithm. The algorithm offers the possibility to use different metrics and summary statistics representative of the data to carry out Bayesian inference. The efficacy of the algorithm in structural dynamics is demonstrated through three different illustrative examples of nonlinear system identification: cubic and cubic-quintic models, the Bouc-Wen model and the Duffing oscillator. The obtained results suggest that ABC is a promising alternative to deal with model selection and parameter estimation issues, specifically for systems with complex behaviours.

Variational Bayesian Gaussian Mixture Regression for Soft Sensing Key Variables in Non-Gaussian Industrial Processes

In this brief, a variational Bayesian Gaussian mixture regression (VBGMR) method is developed for soft sensing key quality-related variables in a non-Gaussian industrial process. Traditional Gaussian mixture regression (GMR) is based on Gaussian mixture model (GMM) and can be easily stuck into the issue of model selection since the GMM model generally requires a large amount of local components so as to achieve desirable predicting performances. However, such assignation can be computationally extensive and may also result in some numerical issues since only a few components are positively adopted. In this brief, a fully Bayesian modeling method is proposed for GMR-based soft sensor development. Conjugate Bayesian prior is defined for each empirical parameter, while a Dirichlet process prior is further defined on the mixture component. The full Bayesian GMR is first validated on a numerical case study and then applied to the industrial hydrogen manufacturing units, both compared with GMR. The results demonstrate feasibility and reliability of the new soft sensor and show that VBGMR generally outperforms GMR with the requirement of only a few activated components.

Fixed-effects dynamic spatial panel data models and impulse response analysis

Real data often have complicated correlations over cross section and time. Such correlations are of particular interests in empirical studies. This paper considers using high order spatial lags and high order time lags to model complicated correlations over cross section and time. We propose to use the quasi maximum likelihood (QML) method to estimate the model. We establish the asymptotic theory of the quasi maximum likelihood estimator (QMLE), including the consistency and limiting distribution, under large N and large T setup, where N denotes the number of individuals and T the number of time periods. We investigate the problem of estimating impulse response functions and the associated (1−α)-confidence intervals. Average direct, indirect and total impacts are defined along the same spirits of LeSage and Pace (2009) under the dynamic spatial panel data setup. The estimation and inferential theory for the three impacts are studied. Model selection issue is also considered. Monte Carlo simulations confirm our theoretical results and show that the QMLE after bias correction has good finite sample performance.

Estimation and model selection of higher-order spatial autoregressive model: An efficient Bayesian approach

In this paper we consider estimation and model selection of higher-order spatial autoregressive model by an efficient Bayesian approach. Based upon the exchange algorithm, we develop an efficient MCMC sampler, which does not rely on special features of spatial weights matrices and does not require the evaluation of the Jacobian determinant in the likelihood function. We also propose a computationally simple procedure to tackle nested model selection issues of higher-order spatial autoregressive models. We find that the exchange algorithm can be utilized to simplify the computation of Bayes factor through the Savage-Dickey density ratio. We apply the efficient estimation algorithm and the model selection procedure to study the “tournament competition” across Chinese cities and the spatial dependence of county-level voter participation rates in the 1980 U.S. presidential election.

Predictive coarse-graining

We propose a data-driven, coarse-graining formulation in the context of equilibrium statistical mechanics. In contrast to existing techniques which are based on a fine-to-coarse map, we adopt the opposite strategy by prescribing a probabilistic coarse-to-fine map. This corresponds to a directed probabilistic model where the coarse variables play the role of latent generators of the fine scale (all-atom) data. From an information-theoretic perspective, the framework proposed provides an improvement upon the relative entropy method [1] and is capable of quantifying the uncertainty due to the information loss that unavoidably takes place during the coarse-graining process. Furthermore, it can be readily extended to a fully Bayesian model where various sources of uncertainties are reflected in the posterior of the model parameters. The latter can be used to produce not only point estimates of fine-scale reconstructions or macroscopic observables, but more importantly, predictive posterior distributions on these quantities. Predictive posterior distributions reflect the confidence of the model as a function of the amount of data and the level of coarse-graining. The issues of model complexity and model selection are seamlessly addressed by employing a hierarchical prior that favors the discovery of sparse solutions, revealing the most prominent features in the coarse-grained model. A flexible and parallelizable Monte Carlo – Expectation–Maximization (MC-EM) scheme is proposed for carrying out inference and learning tasks. A comparative assessment of the proposed methodology is presented for a lattice spin system and the SPC/E water model.

Model selection criteria in beta regression with varying dispersion

ABSTRACTWe address the issue of model selection in beta regressions with varying dispersion. The model consists of two submodels, namely: for the mean and for the dispersion. Our focus is on the selection of the covariates for each submodel. Our Monte Carlo evidence reveals that the joint selection of covariates for the two submodels is not accurate in finite samples. We introduce two new model selection criteria that explicitly account for varying dispersion and propose a fast two step model selection scheme which is considerably more accurate and is computationally less costly than usual joint model selection. Monte Carlo evidence is presented and discussed. We also present the results of an empirical application.

Bayesian Analysis of Spatial Panel Autoregressive Models With Time-Varying Endogenous Spatial Weight Matrices, Common Factors, and Random Coefficients

This article examines spatial panel autoregressive (SAR) models with dynamic, time-varying endogenous spatial weights matrices, common factors, and random coefficients. An empirical application is on the spillover effects of state Medicaid spending. Endogeneity of spatial weights matrices comes from the correlation of “economic distance” and the disturbances in the SAR equation. Common factors control for common shocks to all states and random coefficients may capture heterogeneity in responses. The Bayesian Markov chain Monte Carlo (MCMC) estimation is developed. Identification of factors and factor loadings, and model selection issues based upon the deviance information criterion (DIC) are explored. We find that a state’s Medicaid related spending is positively and significantly affected by those of its neighbors. Both welfare motivated move and yardstick competition are possible sources of strategic interactions among state governments. Welfare motivated move turns out to be more a driving force for the interdependence and states do exhibit heterogenous responses.

A functional regression model for inventories supported by aerial laser scanner data or photogrammetric point clouds

Abstract Forest inventories, with a probability sampling of a target variable Y and a potentially very large number of auxiliary variables (X) obtained from an aerial laser scanner or photogrammetry, are faced with the issue of model and variable selection when a model for linking Y to X is formulated. To bypass this step we propose a generic functional regression model (FRM) for use in both a design- and a model-based framework of inference. We demonstrate applications of FRM with inventory data from France, Germany, and Norway. The generic FRM achieved results that were comparable to those obtained with more traditional approaches based on model and variable selections. The proposed FRM generates interpretable regression coefficients and enables testing of practically relevant hypotheses regarding estimated models.

Methodological development for selection of significant predictors explaining fatal road accidents

Identification of the most relevant factors for explaining road accident occurrence is an important issue in road safety research, particularly for future decision-making processes in transport policy. However model selection for this particular purpose is still an ongoing research. In this paper we propose a methodological development for model selection which addresses both explanatory variable and adequate model selection issues. A variable selection procedure, TIM (two-input model) method is carried out by combining neural network design and statistical approaches. The error structure of the fitted model is assumed to follow an autoregressive process. All models are estimated using Markov Chain Monte Carlo method where the model parameters are assigned non-informative prior distributions. The final model is built using the results of the variable selection. For the application of the proposed methodology the number of fatal accidents in Spain during 2000–2011 was used. This indicator has experienced the maximum reduction internationally during the indicated years thus making it an interesting time series from a road safety policy perspective. Hence the identification of the variables that have affected this reduction is of particular interest for future decision making. The results of the variable selection process show that the selected variables are main subjects of road safety policy measures.

CASP11 refinement experiments with ROSETTA.

We report new Rosetta-based approaches to tackling the major issues that confound protein structure refinement, and the testing of these approaches in the CASP11 experiment. Automated refinement protocols were developed that integrate a range of sampling methods using parallel computation and multiobjective optimization. In CASP11, we used a more aggressive large-scale structure rebuilding approach for poor starting models, and a less aggressive local rebuilding plus core refinement approach for starting models likely to be closer to the native structure. The more incorrectly modeled a structure was predicted to be, the more it was allowed to vary during refinement. The CASP11 experiment revealed strengths and weaknesses of the approaches: the high-resolution strategy incorporating local rebuilding with core refinement consistently improved starting structures, while the low-resolution strategy incorporating the reconstruction of large parts of the structures improved starting models in some cases but often considerably worsened them, largely because of model selection issues. Overall, the results suggest the high-resolution refinement protocol is a promising method orthogonal to other approaches, while the low-resolution refinement method clearly requires further development. Proteins 2016; 84(Suppl 1):314-322. © 2015 Wiley Periodicals, Inc.

Nowcasting French GDP in real-time with surveys and “blocked” regressions: Combining forecasts or pooling information?

This paper empirically investigates two alternative combination strategies, namely forecast combination and information pooling, in the context of nowcasting French GDP in real time with monthly survey opinions. According to the encompassing paradigm, we claim that the outperformance of the forecast combination strategy reported by recent works may be related to the issues of model selection and misspecification. To address these issues, we promote the blocking modeling approach to allow us to handle mixed frequencies in a linear framework that is compatible with an automatic model selection algorithm. Selected restricted- and pooled-information models are specified and tested for forecast encompassing in order to determine the best combination strategy. The results suggest that the forecast combination strategy dominates as long as no individual (restricted) model encompasses the rivals. However, when a predictive encompassing model is obtained by pooling the information sets, this model outperforms the most accurate forecast combination scheme.

The exponential-Poisson model for recurrent event data: an application to a set of data on malaria in Brazil.

In this paper, we introduce a new model for recurrent event data characterized by a baseline rate function fully parametric, which is based on the exponential-Poisson distribution. The model arises from a latent competing risk scenario, in the sense that there is no information about which cause was responsible for the event occurrence. Then, the time of each recurrence is given by the minimum lifetime value among all latent causes. The new model has a particular case, which is the classical homogeneous Poisson process. The properties of the proposed model are discussed, including its hazard rate function, survival function, and ordinary moments. The inferential procedure is based on the maximum likelihood approach. We consider an important issue of model selection between the proposed model and its particular case by the likelihood ratio test and score test. Goodness of fit of the recurrent event models is assessed using Cox-Snell residuals. A simulation study evaluates the performance of the estimation procedure in the presence of a small and moderate sample sizes. Applications on two real data sets are provided to illustrate the proposed methodology. One of them, first analyzed by our team of researchers, considers the data concerning the recurrence of malaria, which is an infectious disease caused by a protozoan parasite that infects red blood cells.