Model Selection Consistency Research Articles

On model selection consistency using a kick-one-out method for selecting response variables in high-dimensional multivariate linear regression

. This article deals with the selection of non redundant response variables in normality-assumed multivariate linear regression, where the redundancy of the response variables is defined by conditional expectation. A sufficient condition for model selection consistency is obtained using a kick-one-out method based on the generalized information criterion under a high-dimensional asymptotic framework such that the sample size tends to infinity and the number of response variables and explanatory variables does not exceed the sample size but may tend to infinity. A consistent kick-one-out method using the obtained condition is proposed. Simulation results show that the proposed method has a high probability of selecting true, non redundant variables.

Selection inconsistency for factor-augmented regressions

Factor-augmented regression (FAR) is an effective tool in forming predictions in the presence of big data set. However, few studies have considered the selection of latent factors and observed covariates simultaneously in FAR. We discover that the class of information criteria suggested by Groen and Kapetanios (2013) to determine the factors and covariates fail to provide consistent model selection, especially when regressors are correlated and the cross-section dimension is small relative to the sample size. This theoretical discovery is corroborated with simulation findings that the criteria in Groen and Kapetanios (2013) tend to underestimate the factor number but overestimate the covariate number.

Robust multitask learning in high dimensions under memory constraint

AbstractWe investigate multitask learning in the context of multivariate linear regression with high dimensional covariates and heavy‐tailed noise, while under the constraint of limited memory. To tackle the computational complexity arising from the non‐smoothness of the quantile loss, we reformulate it as an equivalent least squares loss, which yields robust solutions even in the presence of heavy‐tailed noise. We incorporate a group lasso penalty into the quantile loss to produce sparse solutions, and an accelerated proximal sub‐gradient descent algorithm to speed up the computation while ensuring explicit forms for penalized solutions at each iteration. The proposed algorithm is general and can be applied to similar optimization problems. Moreover, we introduce a communication‐efficient distributed algorithm that guarantees optimal convergence rates after finite communication rounds in cases where computing resources such as memory are insufficient. We also study the theoretical properties of the resultant estimate and relax the widely used model selection consistency assumption on the initial estimate. We demonstrate the effectiveness of our proposal through extensive numerical studies.

Model selection for generalized linear models with weak factors

AbstractThe literature has witnessed an upsurge of interest in model selection in diverse fields and optimization applications. Despite the substantial progress, model selection remains a significant challenge when covariates are highly correlated, particularly within economic and financial datasets that exhibit cross‐sectional and serial dependency. In this paper, we introduce a novel methodology named factor augmented regularized model selection with weak factors (WeakFARM) for generalized linear models in the presence of correlated covariates with weak latent factor structure. By identifying weak latent factors and idiosyncratic components and employing them as predictors, WeakFARM converts the challenge from model selection with highly correlated covariates to that with weakly correlated ones. Furthermore, we develop a variable screening method based on the proposed WeakFARM method. Comprehensive theoretical guarantees including estimation consistency, model selection consistency and sure screening property are also provided. We demonstrate the effectiveness of our approach by extensive simulation studies and a real data application in economic forecasting.

Consistent skinny Gibbs in probit regression

Spike and slab priors have emerged as effective and computationally scalable tools for Bayesian variable selection in high-dimensional linear regression. However, the crucial model selection consistency and efficient computational strategies using spike and slab priors in probit regression have rarely been investigated. A hierarchical probit model with continuous spike and slab priors over regression coefficients is considered, and a highly scalable Gibbs sampler with a computational complexity that grows only linearly in the dimension of predictors is proposed. Specifically, the “Skinny Gibbs” algorithm is adapted to the setting of probit and negative binomial regression and model selection consistency for the proposed method under probit model is established, when the number of covariates is allowed to grow much larger than the sample size. Through simulation studies, the method is shown to achieve superior empirical performance compared with other state-of-the art methods. Gene expression data from 51 asthmatic and 44 non-asthmatic samples are analyzed and the performance for predicting asthma using the proposed approach is compared with existing approaches.

Bayesian Inference on Hierarchical Nonlocal Priors in Generalized Linear Models

Variable selection methods with nonlocal priors have been widely studied in linear regression models, and their theoretical and empirical performances have been reported. However, the crucial model selection properties for hierarchical nonlocal priors in high-dimensional generalized linear regression have rarely been investigated. In this paper, we consider a hierarchical nonlocal prior for high-dimensional logistic regression models and investigate theoretical properties of the posterior distribution. Specifically, a product moment (pMOM) nonlocal prior is imposed over the regression coefficients with an Inverse-Gamma prior on the tuning parameter. Under standard regularity assumptions, we establish strong model selection consistency in a high-dimensional setting, where the number of covariates is allowed to increase at a sub-exponential rate with the sample size. We implement the Laplace approximation for computing the posterior probabilities, and a modified shotgun stochastic search procedure is suggested for efficiently exploring the model space. We demonstrate the validity of the proposed method through simulation studies and an RNA-sequencing dataset for stratifying disease risk.

Subgroup detection in the heterogeneous partially linear additive Cox model

In the analysis of survival data, it is crucial to consider individual heterogeneities related to therapy, gender, and genetics as they can impact the validity of conclusions. The heterogeneous partially linear additive Cox model offers a versatile and useful approach for addressing these variations by incorporating both heterogeneous linear components and homogeneous additive components. We propose a subgroup detection method for this model under right-censoring using B-spline smoothing technique to approximate the additive functions. Without imposing any specific subgroup structure, we generate an objective function by combining a log partial likelihood function with a fusion penalty. To estimate parameters and detect subgroups simultaneously, we employ a majorized alternating direction method of multipliers (ADMM) algorithm. Furthermore, we establish oracle properties and model selection consistency for the proposed penalised estimator. Simulation studies and real data analysis on breast cancer demonstrate the performance of our method.

A Unified Framework for Tuning Hyperparameters in Clustering Problems

Selecting hyperparameters for unsupervised learning problems is challenging in general due to the lack of ground truth for validation. Despite the prevalence of this issue in statistics and machine learning, especially in clustering problems, there are not many methods for tuning these hyperparameters with theoretical guarantees. In this paper, we provide a framework with provable guarantees for selecting hyperparameters in a number of distinct models. We consider both the subgaussian mixture model and network models to serve as examples of i.i.d. and non-i.i.d. data. We demonstrate that the same framework can be used to choose the Lagrange multipliers of penalty terms in semi-definite programming (SDP) relaxations for community detection, and the bandwidth parameter for constructing kernel similarity matrices for spectral clustering. By incorporating a cross-validation procedure, we show the framework can also do consistent model selection for network models. Using a variety of simulated and real data examples, we show that our framework outperforms other widely used tuning procedures in a broad range of parameter settings.

Consistent Model Selection for the Degree Corrected Stochastic Blockmodel

Consistent Model Selection for the Degree Corrected Stochastic Blockmodel

Interpretable Machine Learning for Discovery: Statistical Challenges and Opportunities

New technologies have led to vast troves of large and complex data sets across many scientific domains and industries. People routinely use machine learning techniques not only to process, visualize, and make predictions from these big data, but also to make data-driven discoveries. These discoveries are often made using interpretable machine learning, or machine learning models and techniques that yield human-understandable insights. In this article, we discuss and review the field of interpretable machine learning, focusing especially on the techniques, as they are often employed to generate new knowledge or make discoveries from large data sets. We outline the types of discoveries that can be made using interpretable machine learning in both supervised and unsupervised settings. Additionally, we focus on the grand challenge of how to validate these discoveries in a data-driven manner, which promotes trust in machine learning systems and reproducibility in science. We discuss validation both from a practical perspective, reviewing approaches based on data-splitting and stability, as well as from a theoretical perspective, reviewing statistical results on model selection consistency and uncertainty quantification via statistical inference. Finally, we conclude byhighlighting open challenges in using interpretable machine learning techniques to make discoveries, including gaps between theory and practice for validating data-driven discoveries.

Efficient and consistent model selection procedures for time series

Efficient and consistent model selection procedures for time series

Consistent Model Selection Procedure for Random Coefficient INAR Models.

In the realm of time series data analysis, information criteria constructed on the basis of likelihood functions serve as crucial instruments for determining the appropriate lag order. However, the intricate structure of random coefficient integer-valued time series models, which are founded on thinning operators, complicates the establishment of likelihood functions. Consequently, employing information criteria such as AIC and BIC for model selection becomes problematic. This study introduces an innovative methodology that formulates a penalized criterion by utilizing the estimation equation within conditional least squares estimation, effectively addressing the aforementioned challenge. Initially, the asymptotic properties of the penalized criterion are derived, followed by a numerical simulation study and a comparative analysis. The findings from both theoretical examinations and simulation investigations reveal that this novel approach consistently selects variables under relatively relaxed conditions. Lastly, the applications of this method to infectious disease data and seismic frequency data produce satisfactory outcomes.

The cluster D-trace loss for differential network analysis

ABSTRACT A growing literature suggests that gene expression can be greatly altered in disease conditions, and identifying those changes will improve the understanding of complex diseases such as cancers or diabetes. A prevailing direction in the analysis of gene expression studies the changes in gene pathways which include sets of related genes. Therefore, introducing structured exploration to differential analysis of gene expression networks may lead to meaningful discoveries. The topic of this paper is differential network analysis, which focuses on capturing the differences between two or more precision matrices. We discuss the connection between the thresholding method and the D-trace loss method on differential network analysis in the case that the precision matrices share the common connected components. Based on this connection, we further propose the cluster D-trace loss method which directly estimates the differential network and achieves model selection consistency. Simulation studies demonstrate its improved performance and computational efficiency. Finally, the usefulness of our proposed estimator is demonstrated by a real-data analysis on non-small cell lung cancer.

On the Use of Minimum Penalties in Statistical Learning

Modern multivariate machine learning and statistical methodologies estimate parameters of interest while leveraging prior knowledge of the association between outcome variables. The methods that do allow for estimation of relationships do so typically through an error covariance matrix in multivariate regression which does not generalize to other types of models. In this article we proposed the MinPen framework to simultaneously estimate regression coefficients associated with the multivariate regression model and the relationships between outcome variables using common assumptions. The MinPen framework uses a novel penalty based on the minimum function to simultaneously detect and exploit relationships between responses. An iterative algorithm is proposed as a solution to the nonconvex optimization. Theoretical results such as high-dimensional convergence rates, model selection consistency, and a framework for post selection inference are provided. We extend the proposed MinPen framework to other exponential family loss functions, with a specific focus on multiple binomial responses. Tuning parameter selection is also addressed. Finally, simulations and two data examples are presented to show the finite sample properties of this framework. Supplemental material providing proofs, additional simulations, code, and datasets are available online.

A Bayesian Approach for Partial Gaussian Graphical Models With Sparsity

We explore various Bayesian approaches to estimate partial Gaussian graphical models. Our hierarchical structures enable to deal with single-output as well as multiple-output linear regressions, in small or high dimension, enforcing either no sparsity, sparsity, group sparsity or even sparse-group sparsity for a bi-level selection through partial correlations (direct links) between predictors and responses, thanks to spike-and-slab priors corresponding to each setting. Adaptative and global shrinkages are also incorporated in the Bayesian modeling of the direct links. An existing result for model selection consistency is reformulated to stick to our sparse and group-sparse settings, providing a theoretical guarantee under some technical assumptions. Gibbs samplers are developed and a simulation study shows the efficiency of our models which give very competitive results, especially in terms of support recovery. To conclude, a real dataset is investigated.

Variable Selection for High-Dimensional Nodal Attributes in Social Networks with Degree Heterogeneity

We consider a class of network models, in which the connection probability depends on ultrahigh-dimensional nodal covariates (homophily) and node-specific popularity (degree heterogeneity). A Bayesian method is proposed to select nodal features in both dense and sparse networks under a mild assumption on popularity parameters. The proposed approach is implemented via Gibbs sampling. To alleviate the computational burden for large sparse networks, we further develop a working model in which parameters are updated based on a dense sub-graph at each step. Model selection consistency is established for both models, in the sense that the probability of the true model being selected converges to one asymptotically, even when the dimension grows with the network size at an exponential rate. The performance of the proposed models and estimation procedures are illustrated through Monte Carlo studies and three real world examples. Supplementary materials for this article are available online.

Model selection uncertainty and stability in beta regression models: A study of bootstrap-based model averaging with an empirical application to clickstream data

Statistical model development is a central feature of many scientific investigations with a vast methodological landscape. However, uncertainty in the model development process has received less attention and is frequently resolved nonrigorously through beliefs about generalizability, practical usefulness, and computational ease. This is particularly problematic in settings of abundant data, such as clickstream data, as model selection routinely admits multiple models and imposes a source of uncertainty, unacknowledged and unknown by many, on all postselection conclusions. Regression models, based on the beta distribution, are a class of nonlinear models, attractive because of their great flexibility and potential explanatory power, but have not been investigated from the standpoint of multimodel uncertainty and model averaging. For this reason a formalized tool that can combine model selection uncertainty and beta regression modeling is presented in this work. The tool combines bootstrap model averaging, model selection, and asymptotic theory to yield a procedure that can perform joint modeling of the mean and precision parameters, capture sources of variability in the data, and achieve more accurate claims of estimate precision, variable importance, generalization performance, and model stability. Practical utility of the tool is demonstrated through a study of model selection consistency and variable importance in average exit and bounce rate statistical models. This work emphasizes the necessity of a departure from the all-too-common practice of ignoring model selection uncertainty and introduces an accessible technique to handle frequently neglected aspects of the modeling pipeline.

High-Dimensional Consistencies of KOO Methods for the Selection of Variables in Multivariate Linear Regression Models with Covariance Structures

In this paper, we consider the high-dimensional consistencies of KOO methods for selecting response variables in multivariate linear regression with covariance structures. Here, the covariance structures are considered as (1) independent covariance structure with the same variance, (2) independent covariance structure with different variances, and (3) uniform covariance structure. A sufficient condition for model selection consistency is obtained using a KOO method under a high-dimensional asymptotic framework, such that sample size n, the number p of response variables, and the number k of explanatory variables are large, as in p/n→c1∈(0,1) and k/n→c2∈[0,1), where c1+c2<1.

Predictive quantile regression with mixed roots and increasing dimensions: The ALQR approach

In this paper we propose the adaptive lasso for predictive quantile regression (ALQR). Reflecting empirical findings, we allow predictors to have various degrees of persistence and exhibit different signal strengths. The number of predictors is allowed to grow with the sample size. We study regularity conditions under which stationary, local unit root, and cointegrated predictors are present simultaneously. We next show the convergence rates, model selection consistency, and asymptotic distributions of ALQR. We apply the proposed method to the out-of-sample quantile prediction problem of stock returns and find that it outperforms the existing alternatives. We also provide numerical evidence from additional Monte Carlo experiments, supporting the theoretical results.

High-dimensional properties for empirical priors in linear regression with unknown error variance

We study full Bayesian procedures for high-dimensional linear regression. We adopt data-dependent empirical priors introduced in Martin et al. (Bernoulli 23(3):1822–1847, 2017). In their paper, these priors have nice posterior contraction properties and are easy to compute. Our paper extend their theoretical results to the case of unknown error variance . Under proper sparsity assumption, we achieve model selection consistency, posterior contraction rates as well as Bernstein von-Mises theorem by analyzing multivariate t-distribution.