High-dimensional Linear Regression Models Research Articles

Testing Many Zero Restrictions in a High Dimensional Linear Regression Setting

We propose a test of many zero parameter restrictions in a high dimensional linear iid regression model with k ≫ n regressors. The test statistic is formed by estimating key parameters one at a time based on many low dimension regression models with nuisance terms. The parsimoniously parametrized models identify whether the original parameter of interest is or is not zero. Estimating fixed low dimension sub-parameters ensures greater estimator accuracy, it does not require a sparsity assumption nor therefore a regularized estimator, it is computationally fast compared to, e.g., de-biased Lasso, and using only the largest in a sequence of weighted estimators reduces test statistic complexity and therefore estimation error. We provide a parametric wild bootstrap for p-value computation, and prove the test is consistent and has non-trivial n / { ln ( n ) M n } -local-to-null power where M n is the l ∞ covariate fourth moment.

The eigenvector LSD of information plus noise matrices and its application to linear regression model

In this paper, we investigate the limit of the eigenvector empirical spectral distribution (VESD) of large dimensional information plus noise matrices Cn=1nTn1/2(Xn+Rn)(Xn+Rn)′Tn1/2, where Xn are p×n random matrices with independent and identically distributed entries, Tn and Rn are non-random matrices. It is shown that as p/n→c∈(0,∞), the VESD of Cn has a deterministic limit under some mild conditions. The theoretical result is applied to the model selection problems in high dimensional linear regression model.

Solving multistage stochastic linear programming via regularized linear decision rules: An application to hydrothermal dispatch planning

The solution of multistage stochastic linear problems (MSLPs) represents a challenge for many applications. Long-term hydrothermal dispatch planning (LHDP) materializes this challenge in a real-world problem that affects electricity markets, economies, and natural resources worldwide. No closed-form solutions are available for MSLP and the definition of non-anticipative policies with high-quality out-of-sample performance is crucial. Linear decision rules (LDRs) provide an interesting simulation-based framework for finding high-quality policies for MSLP through two-stage stochastic models. In practical applications, however, the number of parameters to be estimated when using an LDR may be close to or higher than the number of scenarios of the sample average approximation problem, thereby generating an in-sample overfit and poor performances in out-of-sample simulations. In this paper, we propose a novel regularized LDR to solve MSLP based on the AdaLASSO (adaptive least absolute shrinkage and selection operator). The goal is to use the parsimony principle, as largely studied in high-dimensional linear regression models, to obtain better out-of-sample performance for a LDR applied to MSLP. Computational experiments show that overfitting is a non-negligible threat when using the classical non-regularized LDR to solve the LHDP, one of the most studied MSLP with relevant applications in industry. Our analysis highlights the following benefits of the proposed framework in comparison to the non-regularized benchmark: 1) significant reductions in the number of non-zero coefficients (model parsimony), 2) substantial cost reductions in out-of-sample evaluations, and 3) improved spot-price profiles.

An Orthogonal Matching Pursuit Variable Screening Algorithm for High-Dimensional Linear Regression Models

Variable selection plays an important role in data mining. It is crucial to filter useful variables and extract useful information in a high-dimensional setup when the number of predictor variables d tends to be much larger than the sample size n . Statistical inferences can be more precise after irrelevant variables are moved out by the screening method. This article proposes an orthogonal matching pursuit algorithm for variable screening under the high-dimensional setup. The proposed orthogonal matching pursuit method demonstrates good performance in variable screening. In particular, if the dimension of the true model is finite, OMP might discover all relevant predictors within a finite number of steps. Throughout theoretical analysis and simulations, it is confirmed that the orthogonal matching pursuit algorithm can identify relevant predictors to ensure screening consistency in variable selection. Given the sure screening property, the BIC criterion can be used to practically select the best candidate from the models generated by the OMP algorithm. Compared with the traditional orthogonal matching pursuit method, the resulting model can improve prediction accuracy and reduce computational cost by screening out the relevant variables.

A semi-parametric approach to feature selection in high-dimensional linear regression models

A semi-parametric approach to feature selection in high-dimensional linear regression models

On estimation error bounds of the Elastic Net when p ≫ n

We study the estimation property of the Elastic Net estimator in high-dimensional linear regression models where the number of parameters p is comparable to or larger than the sample size n. In such a situation, one often assumes sparsity of the true regression coefficient vector , i.e., assuming that belongs to an -ball with radius , , for some . In this paper, we provide -estimation error bounds for the Elastic Net and naive Elastic Net estimators under a unified framework for high-dimensional analysis of M-estimators proposed by Negahban et al. [A unified framework for high-dimensional analysis of M-estimators with decomposable regularizers. Adv Neural Inf Process Syst. 2009;22:1348–1356]. We show that for both cases of exact sparsity and weak sparsity, under the same conditions on the design matrix, the Elastic Net estimator achieves a slightly better error bound than the Lasso estimator by suitably choosing the tuning parameters.

Conditional Test for Ultrahigh Dimensional Linear Regression Coefficients

This paper is concerned with a conditional test for regression coefficients in ultrahigh dimensional linear models. Conditioning on a subset of important predictors in the model, we test the overall significance of regression coefficients of the remaining ultrahigh dimensional predictors. We first propose a conditional U-statistic test (CUT) based on an estimated U-statistic for a high dimensional linear regression model and prove that its null asymptotic distribution is normal under some mild assumptions. However, the empirical power of the proposed test is inversely affected by the dimensionality of predictors. To this end, we further propose a two-stage CUT with screening (CUTS) procedure to reduce the dimensionality under the sparsity assumption and enhance the empirical power based on random data splitting strategy. In the first stage, we divide data randomly into two equal halves and apply the conditional sure independence screening to the first half to reduce the dimensionality; In the second stage, we apply the proposed CUT test to the second half. To eliminate the effect of random data splitting and further enhance the empirical power, we also develop a powerful ensemble algorithm based on multiple splitting strategy and prove that the family-wise error rate is asymptotically controlled at a given significance level. We demonstrate the excellent finite-sample performances of the proposed conditional test via Monte Carlo simulations and a real data analysis.

Improved estimators for semi-supervised high-dimensional regression model

We study a high-dimensional linear regression model in a semi-supervised setting, where for many observations only the vector of covariates X is given with no responses Y. We do not make any sparsity assumptions on the vector of coefficients, nor do we assume normality of the covariates. We aim at estimating the signal level, i.e., the amount of variation in the response that can be explained by the set of covariates. We propose an estimator, which is unbiased, consistent, and asymptotically normal. This estimator can be improved by adding zero-estimators arising from the unlabeled data. Adding zero-estimators does not affect the bias and potentially can reduce the variance. We further present an algorithm based on our approach that improves any given signal level estimator. Our theoretical results are demonstrated in a simulation study.

High-dimensional inference robust to outliers with -norm penalization

This article studies inference in the high-dimensional linear regression model with outliers. Sparsity constraints are imposed on the vector of coefficients of the covariates. The number of outliers can grow with the sample size while their proportion goes to 0. We propose a two-step procedure for inference on the coefficients of a fixed subset of regressors. The first step is a based on several square-root lasso -norm penalized estimators, while the second step is the ordinary least squares estimator applied to a well-chosen regression. We establish asymptotic normality of the two-step estimator. The proposed procedure is efficient in the sense that it attains the semiparametric efficiency bound when applied to the model without outliers under homoscedasticity. This approach is also computationally advantageous, it amounts to solving a finite number of convex optimization programs.

An adaptive shortest-solution guided decimation approach to sparse high-dimensional linear regression

High-dimensional linear regression model is the most popular statistical model for high-dimensional data, but it is quite a challenging task to achieve a sparse set of regression coefficients. In this paper, we propose a simple heuristic algorithm to construct sparse high-dimensional linear regression models, which is adapted from the shortest-solution guided decimation algorithm and is referred to as ASSD. This algorithm constructs the support of regression coefficients under the guidance of the shortest least-squares solution of the recursively decimated linear models, and it applies an early-stopping criterion and a second-stage thresholding procedure to refine this support. Our extensive numerical results demonstrate that ASSD outperforms LASSO, adaptive LASSO, vector approximate message passing, and two other representative greedy algorithms in solution accuracy and robustness. ASSD is especially suitable for linear regression problems with highly correlated measurement matrices encountered in real-world applications.

Bayesian Dynamic Feature Partitioning in High-Dimensional Regression With Big Data

Bayesian computation of high-dimensional linear regression models using Markov chain Monte Carlo (MCMC) or its variants can be extremely slow or completely prohibitive since these methods perform costly computations at each iteration of the sampling chain. Furthermore, this computational cost cannot usually be efficiently divided across a parallel architecture. These problems are aggravated if the data size is large or data arrive sequentially over time (streaming or online settings). This article proposes a novel dynamic feature partitioned regression (DFP) for efficient online inference for high-dimensional linear regressions with large or streaming data. DFP constructs a pseudo posterior density of the parameters at every time point, and quickly updates the pseudo posterior when a new block of data (data shard) arrives. DFP updates the pseudo posterior at every time point suitably and partitions the set of parameters to exploit parallelization for efficient posterior computation. The proposed approach is applied to high-dimensional linear regression models with Gaussian scale mixture priors and spike-and-slab priors on large parameter spaces, along with large data, and is found to yield state-of-the-art inferential performance. The algorithm enjoys theoretical support with pseudoposterior densities over time being arbitrarily close to the full posterior as the data size grows, as shown in the supplementary material. Supplementary material also contains details of the DFP algorithm applied to different priors. Package to implement DFP is available in https://github.com/Rene-Gutierrez/DynParRegReg. The dataset is available in https://github.com/Rene-Gutierrez/DynParRegReg\\_Implementation.

Minimax Rates of ℓp-Losses for High-Dimensional Linear Errors-in-Variables Models over ℓq-Balls.

In this paper, the high-dimensional linear regression model is considered, where the covariates are measured with additive noise. Different from most of the other methods, which are based on the assumption that the true covariates are fully obtained, results in this paper only require that the corrupted covariate matrix is observed. Then, by the application of information theory, the minimax rates of convergence for estimation are investigated in terms of the -losses under the general sparsity assumption on the underlying regression parameter and some regularity conditions on the observed covariate matrix. The established lower and upper bounds on minimax risks agree up to constant factors when , which together provide the information-theoretic limits of estimating a sparse vector in the high-dimensional linear errors-in-variables model. An estimator for the underlying parameter is also proposed and shown to be minimax optimal in the -loss.

Strong selection consistency of Bayesian vector autoregressive models based on a pseudo-likelihood approach

Vector autoregressive (VAR) models aim to capture linear temporal interdependencies among multiple time series. They have been widely used in macroeconomics and financial econometrics and more recently have found novel applications in functional genomics and neuroscience. These applications have also accentuated the need to investigate the behavior of the VAR model in a high-dimensional regime, which will provide novel insights into the role of temporal dependence for regularized estimates of the models parameters. However, hardly anything is known regarding posterior model selection consistency for Bayesian VAR models in such regimes. In this work, we develop a pseudo-likelihood based Bayesian approach for consistent variable selection in high-dimensional VAR models by considering hierarchical normal priors on the autoregressive coefficients, as well as on the model space. We establish strong selection consistency of the proposed method, namely that the posterior probability assigned to the true underlying VAR model converges to one under high-dimensional scaling where the dimension p of the VAR system grows nearly exponentially with the sample size n. Further, the result is established under mild regularity conditions on the problem parameters. Finally, as a by-product of these results, we also establish strong selection consistency for the sparse high-dimensional linear regression model with serially correlated regressors and errors.

Robust penalized estimation via Welsch loss with group Lasso

Robust estimation is widely used for analyzing statistical inference. We investigate penalized robust estimation via Welsch loss function with group Lasso method in high-dimensional linear regression models with group structure in this paper. This penalty identifies the significant groups of predictor variables. Robust estimation with group Lasso has crucial meaning in that it accompanies the large p (the number of predictors) small n (sample size) problems. We present the updating algorithms for this group Lasso problem. Compared with other penalty functions, we carried out simulation studies in order to assess the performance of the proposed method and the real dataset was demonstrated numerically for illustration purpose.

Optimal sparsity testing in linear regression model

We consider the problem of sparsity testing in the high-dimensional linear regression model. The problem is to test whether the number of non-zero components (aka the sparsity) of the regression parameter θ∗ is less than or equal to k0. We pinpoint the minimax separation distances for this problem, which amounts to quantifying how far a k1-sparse vector θ∗ has to be from the set of k0-sparse vectors so that a test is able to reject the null hypothesis with high probability. Two scenarios are considered. In the independent scenario, the covariates are i.i.d. normally distributed and the noise level is known. In the general scenario, both the covariance matrix of the covariates and the noise level are unknown. Although the minimax separation distances differ in these two scenarios, both of them actually depend on k0 and k1 illustrating that for this composite-composite testing problem both the size of the null and of the alternative hypotheses play a key role.

High-dimensional variable selection via low-dimensional adaptive learning

A stochastic search method, the so-called Adaptive Subspace (AdaSub) method, is proposed for variable selection in high-dimensional linear regression models. The method aims at finding the best model with respect to a certain model selection criterion and is based on the idea of adaptively solving low-dimensional sub-problems in order to provide a solution to the original high-dimensional problem. Any of the usual $\ell_0$-type model selection criteria can be used, such as Akaike's Information Criterion (AIC), the Bayesian Information Criterion (BIC) or the Extended BIC (EBIC), with the last being particularly suitable for high-dimensional cases. The limiting properties of the new algorithm are analysed and it is shown that, under certain conditions, AdaSub converges to the best model according to the considered criterion. In a simulation study, the performance of AdaSub is investigated in comparison to alternative methods. The effectiveness of the proposed method is illustrated via various simulated datasets and a high-dimensional real data example.

REMI: REGRESSION WITH MARGINAL INFORMATION AND ITS APPLICATION IN GENOME-WIDE ASSOCIATION STUDIES

In this study, we consider the problem of variable selection and estimation in high-dimensional linear regression models when the complete data are not accessible, but only certain marginal information or summary statistics are available. This problem is motivated from the Genome-wide association studies (GWAS) that have been widely used to identify risk variants underlying complex human traits/diseases. With a large number of completed GWAS, statistical methods using summary statistics become more and more important because of restricted accessibility to individual-level data sets. Theoretically guaranteed methods are highly demanding to advance the statistical inference with a large amount of available marginal information. Here we propose an $\ell_1$ penalized approach, REMI, to estimate high dimensional regression coefficients with marginal information and external reference samples. We establish an upper bound on the error of the REMI estimator, which has the same order as that of the minimax error bound of Lasso with complete individual-level data. In particular, when marginal information is obtained from a large number of samples together with a small number of reference samples, REMI yields good estimation and prediction results, and outperforms the Lasso because the sample size of accessible individual-level data can be limited. Through simulation studies and real data analysis of the NFBC1966 GWAS data set, we demonstrate that REMI can be widely applicable. The developed R package and the codes to reproduce all the results are available at this https URL

Bayesian inference in high-dimensional linear models using an empirical correlation-adaptive prior

In the context of a high-dimensional linear regression model, we propose the use of an empirical correlation-adaptive prior that makes use of information in the observed predictor variable matrix to adaptively address high collinearity, determining if parameters associated with correlated predictors should be shrunk together or kept apart. Under suitable conditions, we prove that this empirical Bayes posterior concentrates around the true sparse parameter at the optimal rate asymptotically. A simplified version of a shotgun stochastic search algorithm is employed to implement the variable selection procedure, and we show, via simulation experiments across different settings and a real-data application, the favorable performance of the proposed method compared to existing methods.

Empirical Bayes oracle uncertainty quantification for regression

We propose an empirical Bayes method for high-dimensional linear regression models. Following an oracle approach that quantifies the error locally for each possible value of the parameter, we show that an empirical Bayes posterior contracts at the optimal rate at all parameters and leads to uniform size-optimal credible balls with guaranteed coverage under an “excessive bias restriction” condition. This condition gives rise to a new slicing of the entire space that is suitable for ensuring uniformity in uncertainty quantification. The obtained results immediately lead to optimal contraction and coverage properties for many conceivable classes simultaneously. The results are also extended to high-dimensional additive nonparametric regression models.

Consistent Group Selection with Bayesian High Dimensional Modeling

In many applications with high dimensional covariates, the covariates are naturally structured into different groups which can be used to perform efficient statistical inference. We propose a Bayesian hierarchical model with a spike and slab prior specification to perform group selection in high dimensional linear regression models. While several penalization methods and more recently, some Bayesian approaches are proposed for group selection, theoretical properties of Bayesian approaches have not been studied extensively. In this paper, we provide novel theoretical results for group selection consistency under spike and slab priors which demonstrate that the proposed Bayesian approach has advantages compared to penalization approaches. Our theoretical results accommodate flexible conditions on the design matrix and can be applied to commonly used statistical models such as nonparametric additive models for which very limited theoretical results are available for the Bayesian methods. A shotgun stochastic search algorithm is adopted for the implementation of our proposed approach. We illustrate through simulation studies that the proposed method has better performance for group selection compared to a variety of existing methods.