Model Selection Problem Research Articles

Penalized composite likelihood for colored graphical Gaussian models

AbstractThis article proposes a penalized composite likelihood method for model selection in colored graphical Gaussian models. The method provides a sparse and symmetry‐constrained estimator of the precision matrix and thus conducts model selection and precision matrix estimation simultaneously. In particular, the method uses penalty terms to constrain the elements of the precision matrix, which enables us to transform the model selection problem into a constrained optimization problem. Further, computer experiments are conducted to illustrate the performance of the proposed new methodology. It is shown that the proposed method performs well in both the selection of nonzero elements in the precision matrix and the identification of symmetry structures in graphical models. The feasibility and potential clinical application of the proposed method are demonstrated on a microarray gene expression dataset.

Bayesian model averaging for analysis of lattice field theory results

Statistical modeling is a key component in the extraction of physical results from lattice field theory calculations. Although the general models used are often strongly motivated by physics, many model variations can frequently be considered for the same lattice data. Model averaging, which amounts to a probability-weighted average over all model variations, can incorporate systematic errors associated with model choice without being overly conservative. We discuss the framework of model averaging from the perspective of Bayesian statistics, and give useful formulae and approximations for the particular case of least-squares fitting, commonly used in modeling lattice results. In addition, we frame the common problem of data subset selection (e.g. choice of minimum and maximum time separation for fitting a two-point correlation function) as a model selection problem and study model averaging as a straightforward alternative to manual selection of fit ranges. Numerical examples involving both mock and real lattice data are given.

An efficient alpha seeding method for optimized extreme learning machine-based feature selection algorithm

Embedded feature selection algorithms, such as support vector machine based recursive feature elimination (SVM-RFE), have proven to be effective for many real applications. However, due to the model selection problem, SVM-RFE naturally suffers from a heavy computational burden as well as high computational complexity. To solve these issues, this paper proposes using an optimized extreme learning machine (OELM) model instead of SVM. This model, referred to as OELM-RFE provides an efficient active set solver for training the OELM algorithm. We also present an effective alpha seeding algorithm to efficiently solve successive quadratic programming (QP) problems inherent in OELM. One of the salient characteristics of OELM-RFE is that it has only one tuning parameter: the penalty constant C. Experimental results from work on benchmark datasets show that OELM-RFE tends to have higher prediction accuracy than SVM-RFE, and requires fewer model selection efforts. In addition, the alpha seeding method works better on more datasets.

Mixed Lasso estimator for stochastic restricted regression models

Parameters of a linear regression model can be estimated with the help of traditional methods like generalized least squares and mixed estimator. However, recent developments increased the importance of big data sets, which have much more predictors than observations where some predictors have no impact on the dependent variable. The estimation and model selection problem of big datasets can be solved using the least absolute shrinkage and selection operator (Lasso). However, to the authors’ knowledge, there is no study that incorporates stochastic restrictions, within a Lasso framework. In this paper, we propose a Mixed Lasso (M-Lasso) estimator that incorporates stochastic linear restrictions to big data sets for selecting the true model and estimating parameters simultaneously. We conduct a simulation study to compare the performance of M-Lasso with existing estimators based on mean squared error and model selection performance. Results show that M-Lasso is superior in terms of and it generally dominates compared estimators according to the model selection criteria. We employ M-Lasso to estimate parameters of a widely analysed production function under stochastic restrictions raised from economic theory. Our results show that M-Lasso can provide reasonable and more precise estimates of model parameters that are in line with the economic theory.

SvReg: Structural varying-coefficient regression to differentiate how regional brain atrophy affects motor impairment for Huntington disease severity groups.

For Huntington disease, identification of brain regions related to motor impairment can be useful for developing interventions to alleviate the motor symptom, the major symptom of the disease. However, the effects from the brain regions to motor impairment may vary for different groups of patients. Hence, our interest is not only to identify the brain regions but also to understand how their effects on motor impairment differ by patient groups. This can be cast as a model selection problem for a varying-coefficient regression. However, this is challenging when there is a pre-specified group structure among variables. We propose a novel variable selection method for a varying-coefficient regression with such structured variables and provide a publicly available R package svreg for implementation of our method. Our method is empirically shown to select relevant variables consistently. Also, our method screens irrelevant variables better than existing methods. Hence, our method leads to a model with higher sensitivity, lower false discovery rate and higher prediction accuracy than the existing methods. Finally, we found that the effects from the brain regions to motor impairment differ by disease severity of the patients. To the best of our knowledge, our study is the first to identify such interaction effects between the disease severity and brain regions, which indicates the need for customized intervention by diseaseseverity.

Bayesian model averaging in longitudinal studies using Bayesian variable selection methods

Parameter estimation is often considered as a post model selection problem, i.e., the parameters of interest are often estimated based on “the best” model. However, this approach does not take into account that “the best” model was selected from a set of possible models. Ignoring this uncertainty may lead to bias in estimation. In this paper, we present a Bayesian variable selection (BVS) approach for model averaging which would address the model uncertainty. Although averaging would be preferred approach, BVS can be used as well for model selection if the interest is to select one among the set of candidate models. The performance of Bayesian variable selection is compared with the information criterion based model averaging on real longitudinal data and through simulations study.

Bayesian Information Criterion and Selection of the Number of Factors in Factor Analysis Models

In maximum likelihood exploratory factor analysis, the estimates of unique variances can often turn out to be zero or negative, which makes no sense from a statistical point of view. In order to overcome this difficulty, we employ a Bayesian approach by specifying a prior distribution for the variances of unique factors. The factor analysis model is estimated by EM algorithm, for which we provide the expectation and maximization steps within a general framework of EM algorithms. Crucial issues in Bayesian factor analysis model are the choice of adjusted parameters including the number of factors and also the hyper-parameters for the prior distribution. The choice of these parameters can be viewed as a model selection and evaluation problem. We derive a model selection criterion for evaluating a Bayesian factor analysis model. Monte Carlo simulations are conducted to investigate the effectiveness of the proposed procedure. A real data example is also given to illustrate our procedure. We observe that our modeling procedure prevents the occurrence of improper solutions and also chooses the appropriate number of factors objectively.

Designing a covariance matrix tapered beamformer that is universal over notch width

Universal adaptive beamformers (UABFs) combine classic adaptive beamformers with ensemble methods from machine learning to design new beamformers whose asymptotic performance rivals the performance of the best beamformer in a set. The UABF array weights blend a weighted average of the competing beamformer array weights based on past performance using the softmax function. This model averaging approach avoids a difficult model selection problem by hedging performance across a family of models rather than committing to a single model. This talk presents a UABF which blends covariance matrix tapers (CMT). The CMT technique creates broader notches in beampatterns for environments with moving interferers. Interferers moving with high bearing rates cross resolution cells faster and violate the stationarity solution of the minimum variance distortionless response (MVDR) beamformer. The performance of the CMT depends on the chosen notch width. The UABF adapts to find the blend of CMT notch widths that best balances the interferer output power with white noise gain. We present simulation results illustrating the benefits of the UABF approach for CMT in the presence of moving interferers. [Work supported by ONR 321US.]

Bias-corrected Kullback–Leibler distance criterion based model selection with covariables missing at random

A model selection problem for the conditional probability function of the response variable Y given the covariable vector (X,Z) is considered under the case where X is missing at random. And two novel model selection criteria are suggested. It is shown that the model selection by these two criteria is consistent and that the population parameter estimators, corresponding to the selected model, are also consistent and asymptotically normal. Extensive simulation studies are conducted to investigate the finite-sample performances of the proposed two criteria and a thorough comparison is made with some related model selection strategies. Moreover, two real data analyses are presented for illustrating the practical application of the proposed two criteria.

Bayesian Testing of Scientific Expectations under Multivariate Normal Linear Models

The multivariate normal linear model is one of the most widely employed models for statistical inference in applied research. Special cases include (multivariate) t testing, (M)AN(C)OVA, (multivariate) multiple regression, and repeated measures analysis. Statistical criteria for a model selection problem where models may have equality as well as order constraints on the model parameters based on scientific expectations are limited however. This paper presents a default Bayes factor for this inference problem using fractional Bayes methodology. Group specific fractions are used to properly control prior information. Furthermore the fractional prior is centered on the boundary of the constrained space to properly evaluate order-constrained models. The criterion enjoys various important properties under a broad set of testing problems. The methodology is readily usable via the R package ‘BFpack’. Applications from the social and medical sciences are provided to illustrate the methodology.

Fast Bayesian Deconvolution Using Simple Reversible Jump Moves

We propose a Markov chain Monte Carlo-based deconvolution method designed to estimate the number of peaks in spectral data, along with the optimal parameters of each radial basis function. Assuming cases where the number of peaks is unknown, and a sweep simulation on all candidate models is computationally unrealistic, the proposed method efficiently searches over the probable candidates via trans-dimensional moves assisted by annealing effects from replica exchange Monte Carlo moves. Through simulation using synthetic data, the proposed method demonstrates its advantages over conventional sweep simulations, particularly in model selection problems. Application to a set of olivine reflectance spectral data with varying forsterite and fayalite mixture ratios reproduced results obtained from previous mineralogical research, indicating that our method is applicable to deconvolution on real data sets.

On the Quantification of Model Uncertainty: A Bayesian Perspective.

Issues of model selection have dominated the theoretical and applied statistical literature for decades. Model selection methods such as ridge regression, the lasso, and the elastic net have replaced ad hoc methods such as stepwise regression as a means of model selection. In the end, however, these methods lead to a single final model that is often taken to be the model considered ahead of time, thus ignoring the uncertainty inherent in the search for a final model. One method that has enjoyed a long history of theoretical developments and substantive applications, and that accounts directly for uncertainty in model selection, is Bayesian model averaging (BMA). BMA addresses the problem of model selection by not selecting a final model, but rather by averaging over a space of possible models that could have generated the data. The purpose of this paper is to provide a detailed and up-to-date review of BMA with a focus on its foundations in Bayesian decision theory and Bayesian predictive modeling. We consider the selection of parameter and model priors as well as methods for evaluating predictions based on BMA. We also consider important assumptions regarding BMA and extensions of model averaging methods to address these assumptions, particularly the method of Bayesian stacking. Simple empirical examples are provided and directions for future research relevant to psychometrics are discussed.

The Focused Information Criterion for Logistic Time Series Regression Models Under Locally Biased Estimating Functions

Statistical modeling of biomedical data arising from the cytologic samples collected during cancer trials often involves the analysis of binary time series responses indicating the status of the samples being benign or malignant. The model selection problems pertaining to such data involve exploring the Markovian dependence of the status of the cytologic samples collected in a timely manner on the potential covariates of the malignancy measured during each phase of the trial. In the present paper, we propose the focused information criterion (FIC) that facilitates the simultaneous selection of the order of the binary autoregressive response and a subset of covariates in logistic time series regression models. The FIC attempts to select the subset of covariates that minimizes the mean squared error in estimating a certain focus function of interest under the potential model misspecification. The focus function is chosen so as to adequately represent the purpose of statistical modeling for the concerned data. We derive the general expressions for the FIC using quasi-maximum likelihood estimators when the quasi-likelihood estimating functions are locally biased. In terms of minimizing the estimated mean squared error of the focus function, our FIC is seen to outperform the traditional FIC that ignores the bias in the underlying estimating functions induced by the potential model misspecification.

Competing Models: Inferring Exploration Patterns and Information Relevance via Bayesian Model Selection.

Analyzing interaction data provides an opportunity to learn about users, uncover their underlying goals, and create intelligent visualization systems. The first step for intelligent response in visualizations is to enable computers to infer user goals and strategies through observing their interactions with a system. Researchers have proposed multiple techniques to model users, however, their frameworks often depend on the visualization design, interaction space, and dataset. Due to these dependencies, many techniques do not provide a general algorithmic solution to user exploration modeling. In this paper, we construct a series of models based on the dataset and pose user exploration modeling as a Bayesian model selection problem where we maintain a belief over numerous competing models that could explain user interactions. Each of these competing models represent an exploration strategy the user could adopt during a session. The goal of our technique is to make high-level and in-depth inferences about the user by observing their low-level interactions. Although our proposed idea is applicable to various probabilistic model spaces, we demonstrate a specific instance of encoding exploration patterns as competing models to infer information relevance. We validate our technique's ability to infer exploration bias, predict future interactions, and summarize an analytic session using user study datasets. Our results indicate that depending on the application, our method outperforms established baselines for bias detection and future interaction prediction. Finally, we discuss future research directions based on our proposed modeling paradigm and suggest how practitioners can use this method to build intelligent visualization systems that understand users' goals and adapt to improve the exploration process.

Robust variable selection for model-based learning in presence of adulteration

The problem of identifying the most discriminating features when performing supervised learning has been extensively investigated. In particular, several methods for variable selection have been proposed in model-based classification. The impact of outliers and wrongly labeled units on the determination of relevant predictors has instead received far less attention, with almost no dedicated methodologies available. Two robust variable selection approaches are introduced: one that embeds a robust classifier within a greedy-forward selection procedure and the other based on the theory of maximum likelihood estimation and irrelevance. The former recasts the feature identification as a model selection problem, while the latter regards the relevant subset as a model parameter to be estimated. The benefits of the proposed methods, in contrast with non-robust solutions, are assessed via an experiment on synthetic data. An application to a high-dimensional classification problem of contaminated spectroscopic data is presented.

Graphical-model based high dimensional generalized linear models

We consider the problem of both prediction and model selection in high dimensional generalized linear models. Predictive performance can be improved by leveraging structure information among predictors. In this paper, a graphic model-based doubly sparse regularized estimator is discussed under the high dimensional generalized linear models, that utilizes the graph structure among the predictors. The graphic information among predictors is incorporated node-by-node using a decomposed representation and the sparsity is encouraged both within and between the decomposed components. We propose an efficient iterative proximal algorithm to solve the optimization problem. Statistical convergence rates and selection consistency for the doubly sparse regularized estimator are established in the ultra-high dimensional setting. Specifically, we allow the dimensionality grows exponentially with the sample size. We compare the estimator with existing methods through numerical analysis on both simulation study and a microbiome data analysis.

A Tutorial on Markov Chain Monte-Carlo and Bayesian Modeling

This tutorial provides an introduction to Bayesian modeling and Markov Chain Monte-Carlo (MCMC) algorithms including the Metropolis-Hastings and Gibbs Sampling algorithms. We discuss some of the challenges associated with running MCMC including the important question of determining when convergence to stationarity has been achieved. Several applications of Bayesian modeling are also provided including the MRP approach to modeling election outcomes, topic modeling in machine learning, and large-scale optimization and code breaking. We also discuss the important problems of Bayesian model checking and selection and provide an introduction to empirical Bayesian models. In the appendices we briefly discuss several topics including: (i) Hamiltonian Monte-Carlo (an MCMC algorithm that forms the basis for the statistical modeling language STAN) (ii) the EM algorithm and (iii) other problems in Bayesian statistics including Bayesian approaches to constructing confidence intervals, hypothesis testing and decision theory.

Technology-Enhanced Course in English Theoretical Grammar and Phonetics at Tertiary Level

The advancement of technologies and the recently forced lockdown by Covid-19 are bringing changes to the organisation of the learning process by accelerating the introduction of e-learning to create a learner-centred technology-based approach to English studies, thus stepping towards digital humanities. These trends initiated the institutional project Mobile and Desktop Software Integration in Bachelor and Master Study Programmes. The present study, using a questionnaire, elicits university students’ attitudes to the mobile applications and speech analysis software-based seminar activities in Moodle e-course in accordance with the blended learning model selected for the studies of theoretical grammar and phonetics. It is a cross-sectional, focused, and exploratory case study, comprising a description of factors, contributing to the problem of blended learning model selection. The yielded data demonstrate that students do not possess extensive prior experience with the use of software and mobile applications to study English grammar and phonetics. After completing seminar tasks, they favourably account for the integrated blended learning materials and consider that those facilitate their learning process.

The Asymptotic Behavior of Bootstrap Support Values in Molecular Phylogenetics.

The phylogenetic bootstrap is the most commonly used method for assessing statistical confidence in estimated phylogenies by non-Bayesian methods such as maximum parsimony and maximum likelihood (ML). It is observed that bootstrap support tends to be high in large genomic data sets whether or not the inferred trees and clades are correct. Here, we study the asymptotic behavior of bootstrap support for the ML tree in large data sets when the competing phylogenetic trees are equally right or equally wrong. We consider phylogenetic reconstruction as a problem of statistical model selection when the compared models are nonnested and misspecified. The bootstrap is found to have qualitatively different dynamics from Bayesian inference and does not exhibit the polarized behavior of posterior model probabilities, consistent with the empirical observation that the bootstrap is more conservative than Bayesian probabilities. Nevertheless, bootstrap support similarly shows fluctuations among large data sets, with no convergence to a point value, when the compared models are equally right or equally wrong. Thus, in large data sets strong support for wrong trees or models is likely to occur. Our analysis provides a partial explanation for the high bootstrap support values for incorrect clades observed in empirical data analysis. [Bootstrap; model selection; star-tree paradox; support value.].

The focussed information criterion for generalised linear regression models for time series

SummaryThe present paper proposes the focussed information criterion (FIC) to tackle the model selection problems pertinent to generalised linear models (GLM) for time series. As a first step towards constructing the FIC, we formally discuss the local asymptotic theory of quasi‐maximum likelihood estimation for time series GLM under potential model misspecification. The general FIC formula is derived subsequently that is useful for the simultaneous selection of the order of the autoregressive response as well as a subset of important covariates. We also develop the average FIC (AFIC) that is instrumental in selecting an overall good model for a range of covariates and time regions and establish the equivalence of the AFIC with the classical Akaike's information criterion (AIC). We demonstrate our theory with the analysis of rainfall patterns in Melbourne by means of the logistic and Gamma regression models.