Accurate Model Selection Research Articles

Fine-grained wireless propagation ambience sensing

The ubiquitous wireless network infrastructure and the need of people’s indoor sensing inspire the work leveraging wireless signal into broad spectrum for indoor applications, including indoor localization, human–computer interaction, and activity recognition. To provide an accurate model selection or feature template, these applications take the system reliability of the signal in line-of-sight and non-line-of-sight propagation into account. Unfortunately, these two types of signal propagation are analyzed in static or mobile scenario separately. Our question is how to use the wireless signal to estimate the signal propagation ambience to facilitate the adaptive complex environment? In this paper, we exploit the Fresnel zone theory and channel state information (CSI) to model the static and mobile ambience detectors. Considering the spatiotemporal correlation of indoor activities, the propagation ambience can be divided into three categories: line-of-sight (LOS), non-line-of-sight (NLOS), and semi-line-of-sight (SLOS), which is used to represent the intermediate state between the LOS and NLOS propagation ambience during user movement. Leveraging the hidden Markov model to estimate the dynamic propagation ambience in the mobile environment, a novel propagation ambience identification method, named Ambience Sensor (Asor), is proposed to improve the real-time performance for the upper applications. Furthermore, Asor is integrated into a localization algorithm, Asor-based localization system (Aloc), to confirm the effectiveness. We prototype Asor and Aloc based on commodity WiFi infrastructure without any hardware modification. In addition, the real-time performance of Asor is evaluated by conducting tracking experiments. The experimental results show that the median detection rate of propagation ambience is superior to the existing methods in absence of any a priori hypothesis of static or mobile scenarios.

Robust and parallel Bayesian model selection

Effective and accurate model selection is an important problem in modern data analysis. One of the major challenges is the computational burden required to handle large datasets that cannot be stored or processed on one machine. Another challenge one may encounter is the presence of outliers and contaminations that damage the inference quality. The parallel “divide and conquer” model selection strategy divides the observations of the full dataset into roughly equal subsets and perform inference and model selection independently on each subset. After local subset inference, this method aggregates the posterior model probabilities or other model/variable selection criteria to obtain a final model by using the notion of geometric median. This approach leads to improved concentration in finding the “correct” model and model parameters and also is provably robust to outliers and data contamination.

Dynamic cognitive models of intertemporal choice

Traditionally, descriptive accounts of intertemporal choice have relied on static and deterministic models that assume alternative-wise processing of the options. Recent research, by contrast, has highlighted the dynamic and probabilistic nature of intertemporal choice and provided support for attribute-wise processing. Currently, dynamic models of intertemporal choice—which account for both the resulting choice and the time course over which the construction of a choice develops—rely exclusively on the framework of evidence accumulation. In this article, we develop and rigorously compare several candidate schemes for dynamic models of intertemporal choice. Specifically, we consider an existing dynamic modeling scheme based on decision field theory and develop two novel modeling schemes—one assuming lexicographic, noncompensatory processing, and the other built on the classical concepts of random utility in economics and discrimination thresholds in psychophysics. We show that all three modeling schemes can accommodate key behavioral regularities in intertemporal choice. When empirical choice and response time data were fit simultaneously, the models built on random utility and discrimination thresholds performed best. The results also indicated substantial individual differences in the dynamics underlying intertemporal choice. Finally, model recovery analyses demonstrated the benefits of including both choice and response time data for more accurate model selection on the individual level. The present work shows how the classical concept of random utility can be extended to incorporate response dynamics in intertemporal choice. Moreover, the results suggest that this approach offers a successful alternative to the dominating evidence accumulation approach when modeling the dynamics of decision making.

Recommendations on the Sample Sizes for Multilevel Latent Class Models.

A multilevel latent class model (MLCM) is a useful tool for analyzing data arising from hierarchically nested structures. One important issue for MLCMs is determining the minimum sample sizes needed to obtain reliable and unbiased results. In this simulation study, the sample sizes required for MLCMs were investigated under various conditions. A series of design factors, including sample sizes at two levels, the distinctness and the complexity of the latent structure, and the number of indicators were manipulated. The results revealed that larger samples are required when the latent classes are less distinct and more complex with fewer indicators. This study also provides recommendations about the minimum required sample sizes that satisfied all four criteria-model selection accuracy, parameter estimation bias, standard error bias, and coverage rate-as well as rules of thumb for sample size requirements when applying MLCMs in data analysis.

Ignoring a Multilevel Structure in Mixture Item Response Models: Impact on Parameter Recovery and Model Selection.

The current study investigated the consequences of ignoring a multilevel structure for a mixture item response model to show when a multilevel mixture item response model is needed. Study 1 focused on examining the consequence of ignoring dependency for within-level latent classes. Simulation conditions that may affect model selection and parameter recovery in the context of a multilevel data structure were manipulated: class-specific ICC, cluster size, and number of clusters. The accuracy of model selection (based on information criteria) and quality of parameter recovery were used to evaluate the impact of ignoring a multilevel structure. Simulation results indicated that, for the range of class-specific ICCs examined here (.1 to .3), mixture item response models which ignored a higher level nesting structure resulted in less accurate estimates and standard errors (SEs) of item discrimination parameters when the number of clusters was larger than 24 and the cluster size was larger than six. Class-varying ICCs can have compensatory effects on bias. Also, the results suggested that a mixture item response model which ignored multilevel structure was not selected over the multilevel mixture item response model based on Bayesian information criterion (BIC) if the number of clusters and cluster size was at least 50, respectively. In Study 2, the consequences of unnecessarily fitting a multilevel mixture item response model to single-level data were examined. Reassuringly, in the context of single-level data, a multilevel mixture item response model was not selected by BIC, and its use would not distort the within-level item parameter estimates or SEs when the cluster size was at least 20. Based on these findings, it is concluded that, for class-specific ICC conditions examined here, a multilevel mixture item response model is recommended over a single-level item response model for a clustered dataset having cluster size and the number of clusters .

Protein Loop Structure Prediction Using Conformational Space Annealing.

We have developed a protein loop structure prediction method by combining a new energy function, which we call EPLM (energy for protein loop modeling), with the conformational space annealing (CSA) global optimization algorithm. The energy function includes stereochemistry, dynamic fragment assembly, distance-scaled finite ideal gas reference (DFIRE), and generalized orientation- and distance-dependent terms. For the conformational search of loop structures, we used the CSA algorithm, which has been quite successful in dealing with various hard global optimization problems. We assessed the performance of EPLM with two widely used loop-decoy sets, Jacobson and RAPPER, and compared the results against the DFIRE potential. The accuracy of model selection from a pool of loop decoys as well as de novo loop modeling starting from randomly generated structures was examined separately. For the selection of a nativelike structure from a decoy set, EPLM was more accurate than DFIRE in the case of the Jacobson set and had similar accuracy in the case of the RAPPER set. In terms of sampling more nativelike loop structures, EPLM outperformed EDFIRE for both decoy sets. This new approach equipped with EPLM and CSA can serve as the state-of-the-art de novo loop modeling method.

Bayesian Estimation of Multivariate Latent Regression Models: Gauss Versus Laplace

A latent multivariate regression model is developed that employs a generalized asymmetric Laplace (GAL) prior distribution for regression coefficients. The model is designed for high-dimensional applications where an approximate sparsity condition is satisfied, such that many regression coefficients are near zero after accounting for all the model predictors. The model is applicable to large-scale assessments such as the National Assessment of Educational Progress (NAEP), which includes hundreds of student, teacher, and school predictors of latent achievement. Monte Carlo evidence suggests that employing the GAL prior provides more precise estimation of coefficients that equal zero in comparison to a multivariate normal (MVN) prior, which translates to more accurate model selection. Furthermore, the GAL yielded less biased estimates of regression coefficients in smaller samples. The developed model is applied to mathematics achievement data from the 2011 NAEP for 175,200 eighth graders. The GAL and MVN NAEP estimates were similar, but the GAL was more parsimonious by selecting 12 fewer (i.e., 83 of the 148) variable groups. There were noticeable differences between estimates computed with a GAL prior and plausible value regressions with the AM software (beta version 0.06.00). Implications of the results are discussed for test developers and applied researchers.

Estimating daily air temperatures over the Tibetan Plateau by dynamically integrating MODIS LST data

AbstractRecently, remotely sensed land surface temperature (LST) data have been used to estimate air temperatures because of the sparseness of station measurements in remote mountainous areas. Due to the availability and accuracy of Moderate Resolution Imaging Spectroradiometer (MODIS) LST data, the use of a single term or a fixed combination of terms (e.g., Terra/Aqua night and Terra/Aqua day), as used in previous estimation methods, provides only limited practical application. Furthermore, the estimation accuracy may be affected by different combinations and variable data quality among the MODIS LST terms and models. This study presents a method that dynamically integrates the available LST terms to estimate the daily mean air temperature and simultaneously considers model selection, data quality, and estimation accuracy. The results indicate that the differences in model performance are related to the combinations of LST terms and their data quality. The spatially averaged cloud cover of ~14% for the developed product between 2003 and 2010 is much lower than the 35–54% for single LST terms. The average cross‐validation root‐mean‐square difference values are approximately 2°C. This study identifies the best LST combinations and statistical models and provides an efficient method for daily air temperature estimation with low cloud blockage over the Tibetan Plateau (TP). The developed data set and the method proposed in this study can help alleviate the problem of sparse air temperature data over the TP.

The Impact of Ignoring the Level of Nesting Structure in Nonparametric Multilevel Latent Class Models.

The multilevel latent class model (MLCM) is a multilevel extension of a latent class model (LCM) that is used to analyze nested structure data structure. The nonparametric version of an MLCM assumes a discrete latent variable at a higher-level nesting structure to account for the dependency among observations nested within a higher-level unit. In the present study, a simulation study was conducted to investigate the impact of ignoring the higher-level nesting structure. Three criteria-the model selection accuracy, the classification quality, and the parameter estimation accuracy-were used to evaluate the impact of ignoring the nested data structure. The results of the simulation study showed that ignoring higher-level nesting structure in an MLCM resulted in the poor performance of the Bayesian information criterion to recover the true latent structure, the inaccurate classification of individuals into latent classes, and the inflation of standard errors for parameter estimates, while the parameter estimates were not biased. This article concludes with remarks on ignoring the nested structure in nonparametric MLCMs, as well as recommendations for applied researchers when LCM is used for data collected from a multilevel nested structure.

Spatially explicit summary statistics for historical population genetic inference

Summary The integration of population genetics with explicit spatial analyses is crucial to address a range of evolutionary and ecological questions under realistic scenarios. Ignoring space can lead to misleading inferences, yet incorporating spatial realism leads to using complex evolutionary models that necessitate distilling raw genetic data into summary statistics that capture information relevant to the models in question. However, summary statistics derived from traditional population genetic theory overlook the valuable spatial component of genetic variation that is innate in natural systems and can be informative about historical spatio‐demographic processes. To overcome this limitation, we introduce and evaluate a new set of spatially explicit summary statistics that can be calculated from a wide range of molecular markers and that take advantage of well‐established spatial genetic algorithms to summarize the spatial distribution and autocorrelation of genetic variation. Using spatially explicit demographic simulations of SNP data, we characterize their behaviour under different realistic historical demographic scenarios and assess their relative contribution to accurate model selection and parameter estimation. We demonstrate that under a wide range of parameter values, alternate demographic histories could be best differentiated by supplementing traditional summary statistics with these new spatial statistics. We identify different subset of statistics, including Euclidean distances in spatial‐PCA space, Monmonier's identification of genetic breaks and spatial correlograms, that greatly improve discrimination of complex species histories and have sizeable potential to estimate associated parameters, although the added power is likely to be dependent on landscape models and sampling configurations. These results highlight the potential benefits of condensing spatial genetic information into informative summary statistics to substantially improve testing alternative historical demographic models that reflect the complex spatio‐temporal dynamics of species evolutionary histories. This study bridges phylogeography and landscape genetics by paving the way for phylogeographers wanting to incorporate spatial models and data as well as landscape geneticists wanting to incorporate history.

Genealogical Working Distributions for Bayesian Model Testing with Phylogenetic Uncertainty.

Marginal likelihood estimates to compare models using Bayes factors frequently accompany Bayesian phylogenetic inference. Approaches to estimate marginal likelihoods have garnered increased attention over the past decade. In particular, the introduction of path sampling (PS) and stepping-stone sampling (SS) into Bayesian phylogenetics has tremendously improved the accuracy of model selection. These sampling techniques are now used to evaluate complex evolutionary and population genetic models on empirical data sets, but considerable computational demands hamper their widespread adoption. Further, when very diffuse, but proper priors are specified for model parameters, numerical issues complicate the exploration of the priors, a necessary step in marginal likelihood estimation using PS or SS. To avoid such instabilities, generalized SS (GSS) has recently been proposed, introducing the concept of "working distributions" to facilitate--or shorten--the integration process that underlies marginal likelihood estimation. However, the need to fix the tree topology currently limits GSS in a coalescent-based framework. Here, we extend GSS by relaxing the fixed underlying tree topology assumption. To this purpose, we introduce a "working" distribution on the space of genealogies, which enables estimating marginal likelihoods while accommodating phylogenetic uncertainty. We propose two different "working" distributions that help GSS to outperform PS and SS in terms of accuracy when comparing demographic and evolutionary models applied to synthetic data and real-world examples. Further, we show that the use of very diffuse priors can lead to a considerable overestimation in marginal likelihood when using PS and SS, while still retrieving the correct marginal likelihood using both GSS approaches. The methods used in this article are available in BEAST, a powerful user-friendly software package to perform Bayesian evolutionary analyses.

Detecting Adaptive Evolution in Phylogenetic Comparative Analysis Using the Ornstein-Uhlenbeck Model.

Phylogenetic comparative analysis is an approach to inferring evolutionary process from a combination of phylogenetic and phenotypic data. The last few years have seen increasingly sophisticated models employed in the evaluation of more and more detailed evolutionary hypotheses, including adaptive hypotheses with multiple selective optima and hypotheses with rate variation within and across lineages. The statistical performance of these sophisticated models has received relatively little systematic attention, however. We conducted an extensive simulation study to quantify the statistical properties of a class of models toward the simpler end of the spectrum that model phenotypic evolution using Ornstein-Uhlenbeck processes. We focused on identifying where, how, and why these methods break down so that users can apply them with greater understanding of their strengths and weaknesses. Our analysis identifies three key determinants of performance: a discriminability ratio, a signal-to-noise ratio, and the number of taxa sampled. Interestingly, we find that model-selection power can be high even in regions that were previously thought to be difficult, such as when tree size is small. On the other hand, we find that model parameters are in many circumstances difficult to estimate accurately, indicating a relative paucity of information in the data relative to these parameters. Nevertheless, we note that accurate model selection is often possible when parameters are only weakly identified. Our results have implications for more sophisticated methods inasmuch as the latter are generalizations of the case we study.

Measurement errors should always be incorporated in phylogenetic comparative analysis

Summary The evolution of continuous traits is the central component of comparative analyses in phylogenetics, and the comparison of alternative models of trait evolution has greatly improved our understanding of the mechanisms driving phenotypic differentiation. Several factors influence the comparison of models, and we explore the effects of random errors in trait measurement on the accuracy of model selection. We simulate trait data under a Brownian motion model (BM) and introduce different magnitudes of random measurement error. We then evaluate the resulting statistical support for this model against two alternative models: Ornstein–Uhlenbeck (OU) and accelerating/decelerating rates (ACDC). Our analyses show that even small measurement errors (10%) consistently bias model selection towards erroneous rejection of BM in favour of more parameter‐rich models (most frequently the OU model). Fortunately, methods that explicitly incorporate measurement errors in phylogenetic analyses considerably improve the accuracy of model selection. Our results call for caution in interpreting the results of model selection in comparative analyses, especially when complex models garner only modest additional support. Importantly, as measurement errors occur in most trait data sets, we suggest that estimation of measurement errors should always be performed during comparative analysis to reduce chances of misidentification of evolutionary processes.

Sampling strategies for frequency spectrum-based population genomic inference.

BackgroundThe allele frequency spectrum (AFS) consists of counts of the number of single nucleotide polymorphism (SNP) loci with derived variants present at each given frequency in a sample. Multiple approaches have recently been developed for parameter estimation and calculation of model likelihoods based on the joint AFS from two or more populations. We conducted a simulation study of one of these approaches, implemented in the Python module δaδi, to compare parameter estimation and model selection accuracy given different sample sizes under one- and two-population models.ResultsOur simulations included a variety of demographic models and two parameterizations that differed in the timing of events (divergence or size change). Using a number of SNPs reasonably obtained through next-generation sequencing approaches (10,000 - 50,000), accurate parameter estimates and model selection were possible for models with more ancient demographic events, even given relatively small numbers of sampled individuals. However, for recent events, larger numbers of individuals were required to achieve accuracy and precision in parameter estimates similar to that seen for models with older divergence or population size changes. We quantify i) the uncertainty in model selection, using tools from information theory, and ii) the accuracy and precision of parameter estimates, using the root mean squared error, as a function of the timing of demographic events, sample sizes used in the analysis, and complexity of the simulated models.ConclusionsHere, we illustrate the utility of the genome-wide AFS for estimating demographic history and provide recommendations to guide sampling in population genomics studies that seek to draw inference from the AFS. Our results indicate that larger samples of individuals (and thus larger AFS) provide greater power for model selection and parameter estimation for more recent demographic events.Electronic supplementary materialThe online version of this article (doi:10.1186/s12862-014-0254-4) contains supplementary material, which is available to authorized users.

Mining system logs to learn error predictors: a case study of a telemetry system

Predicting system failures can be of great benefit to managers that get a better command over system performance. Data that systems generate in the form of logs is a valuable source of information to predict system reliability. As such, there is an increasing demand of tools to mine logs and provide accurate predictions. However, interpreting information in logs poses some challenges. This study discusses how to effectively mining sequences of logs and provide correct predictions. The approach integrates different machine learning techniques to control for data brittleness, provide accuracy of model selection and validation, and increase robustness of classification results. We apply the proposed approach to log sequences of 25 different applications of a software system for telemetry and performance of cars. On this system, we discuss the ability of three well-known support vector machines - multilayer perceptron, radial basis function and linear kernels - to fit and predict defective log sequences. Our results show that a good analysis strategy provides stable, accurate predictions. Such strategy must at least require high fitting ability of models used for prediction. We demonstrate that such models give excellent predictions both on individual applications - e.g., 1 % false positive rate, 94 % true positive rate, and 95 % precision - and across system applications - on average, 9 % false positive rate, 78 % true positive rate, and 95 % precision. We also show that these results are similarly achieved for different degree of sequence defectiveness. To show how good are our results, we compare them with recent studies in system log analysis. We finally provide some recommendations that we draw reflecting on our study.

Robust MIMO Detection Under Imperfect CSI Based on Bayesian Model Selection

A robust receiver for multiple-input multiple-output orthogonal frequency-division multiplexing (MIMO-OFDM) systems is proposed. We are interested in the scenario when only a limited number of observations in both time and frequency domains are available. For this scenario, perfect channel state information is impossible to obtain and the receiver suffers from statistical information mismatch. To overcome this limitation, we first propose the optimum receiver by performing jointly channel and data estimation. For statistical information mismatch, we construct a finite set of covariance matrices and derive a model-selection scheme based on Bayesian Model Selection. Finally, the sliding-window scheme is used in order to enhance the model selection accuracy. Simulation results are presented, showing that the proposed scheme outperforms the conventional scheme under imperfect channel knowledge.

Model driven reconstruction of roofs from sparse LIDAR point clouds

This article presents a novel, fully automatic method for the reconstruction of three-dimensional building models with prototypical roofs (CityGML LoD2) from LIDAR data and building footprints. The proposed method derives accurate results from sparse point data sets and is suitable for large area reconstruction. Sparse LIDAR data are widely available nowadays. Robust estimation methods such as RANSAC/MSAC, are applied to derive best fitting roof models in a model-driven way. For the identification of the most probable roof model, supervised machine learning methods (Support Vector Machines) are used. In contrast to standard approaches (where the best model is selected via MDL or AIC), supervised classification is able to incorporate additional features enabling a significant improvement in model selection accuracy.

Chemical cross‐linking, mass spectrometry, and in silico modeling of proteasomal 20S core particles of the haloarchaeon Haloferax volcanii

A fast and accurate method is reported to generate distance constraints between juxtaposited amino acids and to validate molecular models of halophilic protein complexes. Proteasomal 20S core particles (CPs) from the haloarchaeon Haloferax volcanii were used to investigate the quaternary structure of halophilic proteins based on their symmetrical, yet distinct subunit composition. Proteasomal CPs are cylindrical barrel-like structures of four-stacked homoheptameric rings of α- and β-type subunits organized in α(7)β(7) β(7)α(7) stoichiometry. The CPs of H. volcanii are formed from a single type of β subunit associated with α1 and/or α2 subunits. Tandem affinity chromatography and new genetic constructs were used to separately isolate α1(7)β(7)β(7)α1(7) and α2(7)β(7)β(7)α2(7) CPs from H. volcanii. Chemically cross-linked peptides of the H. volcanii CPs were analyzed by high-performance mass spectrometry and an open modification search strategy to first generate and then to interpret the resulting tandem mass spectrometric data. Distance constraints obtained by chemical cross-linking mass spectrometry, together with the available structural data of nonhalophilic CPs, facilitated the selection of accurate models of H. volcanii proteasomal CPs composed of α1-, α2-, and β-homoheptameric rings from several different possible structures from Protein Data Bank.

Accurate Model Selection of Relaxed Molecular Clocks in Bayesian Phylogenetics

Recent implementations of path sampling (PS) and stepping-stone sampling (SS) have been shown to outperform the harmonic mean estimator (HME) and a posterior simulation-based analog of Akaike's information criterion through Markov chain Monte Carlo (AICM), in bayesian model selection of demographic and molecular clock models. Almost simultaneously, a bayesian model averaging approach was developed that avoids conditioning on a single model but averages over a set of relaxed clock models. This approach returns estimates of the posterior probability of each clock model through which one can estimate the Bayes factor in favor of the maximum a posteriori (MAP) clock model; however, this Bayes factor estimate may suffer when the posterior probability of the MAP model approaches 1. Here, we compare these two recent developments with the HME, stabilized/smoothed HME (sHME), and AICM, using both synthetic and empirical data. Our comparison shows reassuringly that MAP identification and its Bayes factor provide similar performance to PS and SS and that these approaches considerably outperform HME, sHME, and AICM in selecting the correct underlying clock model. We also illustrate the importance of using proper priors on a large set of empirical data sets.

Efficient model selection in semivarying coefficient models

Varying coefficient models are useful extensions of classical linear models. In practice, some of the varying coefficients may be just constant, while other coefficients are varying. Several methods have been developed to utilize the information that some coefficient functions are constant to improve estimation efficiency. However, in order for such methods to really work, the information about which coefficient functions are constant should be given in advance. In this paper, we propose a computationally efficient method to discriminate in a consistent way the constant coefficient functions from the varying ones. Additionally, we compare the performance of our proposal with that of previous methods developed for the same purpose in terms of model selection accuracy and computing time.