High-dimensional Regression Models Research Articles

Wavelet Feature Screening

An initial screening of which covariates are relevant is a common practice in high-dimensional regression models. The classic feature screening selects only a subset of covariates correlated with the response variable. However, many important features might have a relevant albeit highly nonlinear relation with the response. One screening approach that handles nonlinearity is to compute the correlation between the response and nonparametric functions of each covariate. Wavelets are powerful tools for nonparametric and functional data analysis but are still seldom used in the feature screening literature. We propose a wavelet feature screening method that can be easily implemented, and we prove that, under suitable conditions, it captures the true covariates with high probability. Simulation results also show that our approach outperforms other screening methods in highly nonlinear models. We apply feature screening to two datasets about ozone concentration and epilepsy. In both applications, the proposed method selects features that match findings in the literature of their respective research fields, illustrating the applicability of feature screening. Supplementary material for this article is available online.

Incremental transfer learning for spatial autoregressive model with linear constraints

Transfer learning is generally regarded as a beneficial technique for utilizing external information to enhance learning performance on target tasks. However, current research on transfer learning in high-dimensional regression models does not take into account both the location information of the data and the explicit utilization of prior knowledge. In the framework of transfer learning, this study seeks to resolve the spatial autoregressive problem and investigate the impact of introducing linear constraints. In this paper, a two-step transfer learning approach and a transferable source detection algorithm based on cross-validation are proposed when the input dimensions of the source and target datasets are the same. When the input dimensions are different, this paper suggests a straightforward and workable incremental transfer learning method. Additionally, for the estimating model developed under this method, Karush–Kuhn–Tucker (KKT) conditions and degrees of freedom are determined, and a Bayesian Information Criterion (BIC) is created for choosing hyperparameters. The effectiveness of the proposed methods is proven by numerical calculations, and the performance of the model in transfer learning estimation is improved by the addition of linear constraints.

Rejoinder on: statistical inference and large-scale multiple testing for high-dimensional regression models

Rejoinder on: statistical inference and large-scale multiple testing for high-dimensional regression models

Comments on: Statistical inference and large-scale multiple testing for high-dimensional regression models

Comments on: Statistical inference and large-scale multiple testing for high-dimensional regression models

Comments on: Statistical inference and large-scale multiple testing for high-dimensional regression models

Comments on: Statistical inference and large-scale multiple testing for high-dimensional regression models

Comments on: Statistical inference and large-scale multiple testing for high-dimensional regression models

Comments on: Statistical inference and large-scale multiple testing for high-dimensional regression models

Inference in High-Dimensional Regression Models without the Exact or Lp Sparsity

Abstract We propose a new inference method in high-dimensional regression models and high-dimensional IV regression models. The method is shown to be valid without requiring the exact sparsity or Lp sparsity conditions. Simulation studies demonstrate superior performance of this proposed method over those based on the LASSO or the random forest, especially under less sparse models. We illustrate an application to production analysis with a panel of Chilean firms.

Statistical inference and large-scale multiple testing for high-dimensional regression models

Statistical inference and large-scale multiple testing for high-dimensional regression models

Fiducial variable selection for the high-dimensional regression model

Fiducial variable selection for the high-dimensional regression model

The robust desparsified lasso and the focused information criterion for high-dimensional generalized linear models

The classical lasso estimation for sparse, high-dimensional regression models is typically biased and lacks the oracle properties. The desparsified versions of the lasso have been proposed in the literature that attempt to overcome these drawbacks. In this paper, we propose the outliers-robust version of the desparsified lasso for high dimensional generalized linear models. The robustness, consistency and high dimensional asymptotics are investigated rigorously in a general framework of M-estimation under potential model misspecification. The desparsification mechanism is subsequently utilized to construct the focused information criterion (FIC) thereby facilitating robust, focused model selection in high dimensions. The applications are demonstrated with the Poisson regression under robust quasilikelihood loss function. The empirical performance of the proposed methods is examined via simulations and a real data example.

Heterogeneous robust estimation with the mixed penalty in high-dimensional regression model

In this paper, we propose a MIXED penalty for the LAD regression model, which can estimate parameters and select important variables efficiently and stably. The proposed method has a good performance in the case of dependent variable with heavy tail and outliers, so this estimator is robust and efficient for tackling the problem of heterogeniety. We show that the proposed estimator possesses the good properties by applying certain assumptions. In the part of numerical simulation, we give several simulation studies to examine the asymptotic results, which shows that the method we proposed behaves better.

High-dimensional robust approximated [formula omitted]-estimators for mean regression with asymmetric data

Asymmetry along with heteroscedasticity or contamination often occurs with the growth of data dimensionality. In ultra-high dimensional data analysis, such irregular settings are usually overlooked for both theoretical and computational convenience. In this paper, we establish a framework for estimation in high-dimensional regression models using Penalized Robust Approximated quadratic M-estimators (PRAM). This framework allows general settings such as random errors lack symmetry and homogeneity, or covariates are not sub-Gaussian. To reduce the possible bias caused by data’s irregularity in mean regression, PRAM adopts a loss function with an adaptive robustification parameter. Theoretically, we first show that, in the ultra-high dimension setting, PRAM estimators have local estimation consistency at the minimax rate enjoyed by the LS-Lasso. Then we show that PRAM with an appropriate non-convex penalty in fact agrees with the local oracle solution, and thus obtain its oracle property. Computationally, we compare the performances of six PRAM estimators (Huber, Tukey’s biweight or Cauchy loss function combined with Lasso or MCP penalty function). Our simulation studies and real data analysis demonstrate satisfactory finite sample performances of the PRAM estimator under different irregular settings.

F-test and z-test for high-dimensional regression models with a factor structure

The classic F-test and z-test can fail for high-dimensional regression models. This paper addresses this problem, especially for the case where the covariates contain a latent factor structure. We first use a new technique, the cross-section averages (CSA) of covariates, to estimate the latent factors. We then develop two F-type tests, namely, the Wald test and the F-test, to assess the overall significance of covariates. If the covariates are tested jointly significant, we next carry out a CSA-based z-test to sequentially test the significance of covariates one at a time. Compared with the existing approaches in the literature, which often use principal component analysis (PCA) to estimate the latent factors, the new tests do not depend on the accurate estimation of the unknown degrees of freedom, or on the acquisition of unknown eigenvalues. Therefore, they can reduce the uncertainty arising from the estimation of unknown quantities. We show the power and model selection consistency of these tests and propose a follow-up ratio-type test to further control the model size. Numerical simulations and a real data analysis show the competitive performance of these CSA-based tests.

Sparse high-dimensional linear regression. Estimating squared error and a phase transition

We consider a sparse high-dimensional regression model where the goal is to recover a k-sparse unknown binary vector β∗ from n noisy linear observations of the form Y=Xβ∗+W∈Rn where X∈Rn×p has i.i.d. N(0,1) entries and W∈Rn has i.i.d. N(0,σ2) entries. In the high signal-to-noise ratio regime and sublinear sparsity regime, while the order of the sample size needed to recover the unknown vector information-theoretically is known to be n∗:=2klogp/log(k/σ2+1), no polynomial-time algorithm is known to succeed unless n>nalg:=(2k+σ2)logp. In this work, we offer a series of results investigating multiple computational and statistical aspects of the recovery task in the regime n∈[n∗,nalg]. First, we establish a novel information-theoretic property of the MLE of the problem happening around n=n∗ samples, which we coin as an “all-or-nothing behavior”: when n>n∗ it recovers almost perfectly the support of β∗, while if n<n∗ it fails to recover any fraction of it correctly. Second, at an attempt to understand the computational hardness in the regime n∈[n∗,nalg], we prove that at order nalg samples there is an Overlap Gap Property (OGP) phase transition occurring at the landscape of the MLE: for constants c,C>0 when n<cnalg OGP appears in the landscape of MLE while if n>Cnalg OGP disappears. OGP is a geometric “disconnectivity” property, which initially appeared in the theory of spin glasses and is known to suggest algorithmic hardness when it occurs. Finally, using certain technical results obtained to establish the OGP phase transition, we additionally establish various novel positive and negative algorithmic results for the recovery task of interest, including the failure of LASSO with access to n<cnalg samples and the success of a simple local search method with access to n>Cnalg samples.

Improved high-dimensional regression models with matrix approximations applied to the comparative case studies with support vector machines

Nowadays, high-dimensional data appear in many practical applications such as biosciences. In the regression analysis literature, the well-known ordinary least-squares estimation may be misleading when the full ranking of the design matrix is missed. As a popular issue, outliers may corrupt normal distribution of the residuals. Thus, since not being sensitive to the outlying data points, robust estimators are frequently applied in confrontation with the issue. Ill-conditioning in high-dimensional data is another common problem in modern regression analysis under which applying the least-squares estimator is hardly possible. So, it is necessary to deal with estimation methods to tackle these problems. As known, a successful approach for high-dimension cases is the penalized scheme with the aim of obtaining a subset of effective explanatory variables that predict the response as the best, while setting the other parameters to zero. Here, we develop several penalized mixed-integer nonlinear programming models to be used in high-dimension regression analysis. The given matrix approximations have simple structures, decreasing computational cost of the models. Moreover, the models are effectively solvable by metaheuristic algorithms. Numerical tests are made to shed light on performance of the proposed methods on simulated and real world high-dimensional data sets.

Profile and Non-Profile MM Modeling of Cluster Failure Time and Analysis of ADNI Data

Motivated by the Alzheimer’s Disease Neuroimaging Initiative (ADNI) data, the objective of integration of important biomarkers for the early detection of Mild Cognitive Impairment (MCI) to Alzheimer’s disease (AD) as a therapeutic intervention is most likely to be beneficial in the early stages of disease progression. Developing predictors for MCI to AD comes down to genotype variables such that the dimension of predictors increases as the sample becomes large. Thus, we consider the sparsity concept of coefficients in a high-dimensional regression model with clustered failure time data such as ADNI, which enables enhancing predictive performances and facilitates the model’s interpretability. In this study, we propose two MM algorithms (profile and non-profile) for the shared frailty survival model firstly and then extend the two proposed MM algorithms to regularized estimation in sparse high-dimensional regression model. The convergence properties of our proposed estimators are also established. Furthermore simulation studies and analysis of ADNI data are illustrated by our proposed methods.

Variational Bayesian Inference in High-Dimensional Linear Mixed Models

In high-dimensional regression models, the Bayesian lasso with the Gaussian spike and slab priors is widely adopted to select variables and estimate unknown parameters. However, it involves large matrix computations in a standard Gibbs sampler. To solve this issue, the Skinny Gibbs sampler is employed to draw observations required for Bayesian variable selection. However, when the sample size is much smaller than the number of variables, the computation is rather time-consuming. As an alternative to the Skinny Gibbs sampler, we develop a variational Bayesian approach to simultaneously select variables and estimate parameters in high-dimensional linear mixed models under the Gaussian spike and slab priors of population-specific fixed-effects regression coefficients, which are reformulated as a mixture of a normal distribution and an exponential distribution. The coordinate ascent algorithm, which can be implemented efficiently, is proposed to optimize the evidence lower bound. The Bayes factor, which can be computed with the path sampling technique, is presented to compare two competing models in the variational Bayesian framework. Simulation studies are conducted to assess the performance of the proposed variational Bayesian method. An empirical example is analyzed by the proposed methodologies.

Improved estimators for semi-supervised high-dimensional regression model

We study a high-dimensional linear regression model in a semi-supervised setting, where for many observations only the vector of covariates X is given with no responses Y. We do not make any sparsity assumptions on the vector of coefficients, nor do we assume normality of the covariates. We aim at estimating the signal level, i.e., the amount of variation in the response that can be explained by the set of covariates. We propose an estimator, which is unbiased, consistent, and asymptotically normal. This estimator can be improved by adding zero-estimators arising from the unlabeled data. Adding zero-estimators does not affect the bias and potentially can reduce the variance. We further present an algorithm based on our approach that improves any given signal level estimator. Our theoretical results are demonstrated in a simulation study.

A nonlinear mixed–integer programming approach for variable selection in linear regression model

Modern statistical studies often encounter regression models with high dimensions in which the number of features p is greater than the sample size n. Although the theory of linear models is well–established for the traditional assumption p < n, making valid statistical inference in high dimensional cases is a considerable challenge. With recent advances in technologies, the problem appears in many biological, medical, social, industrial, and economic studies. As known, the LASSO method is a popular technique for variable selection/estimation in high dimensional sparse linear models. Here, we show that the prediction performance of the LASSO method can be improved by eliminating the structured noises through a mixed–integer programming approach. As a result of our analysis, a modified variable selection/estimation scheme is proposed for a high dimensional regression model which can be considered as an alternative of the LASSO method. Some numerical experiments are made on the classical riboflavin production and some simulated data sets to shed light on the practical performance of the suggested method.

Cross-Fitted Residual Regression for High-Dimensional Heteroscedasticity Pursuit

There is a vast amount of work on high-dimensional regression. The common starting point for the existing theoretical work is to assume the data generating model is a homoscedastic linear regression model with some sparsity structure. In reality the homoscedasticity assumption is often violated, and hence understanding the heteroscedasticity of the data is of critical importance. In this article we systematically study the estimation of a high-dimensional heteroscedastic regression model. In particular, the emphasis is on how to detect and estimate the heteroscedasticity effects reliably and efficiently. To this end, we propose a cross-fitted residual regression approach and prove the resulting estimator is selection consistent for heteroscedasticity effects and establish its rates of convergence. Our estimator has tuning parameters to be determined by the data in practice. We propose a novel high-dimensional BIC for tuning parameter selection and establish its consistency. This is the first high-dimensional BIC result under heteroscedasticity. The theoretical analysis is more involved in order to handle heteroscedasticity, and we develop a couple of interesting new concentration inequalities that are of independent interests.