Sparse High-dimensional Models Research Articles

High-dimensional sparse predictive modeling applied to varied correlation structure via the generalized additive model

This paper explores the characteristics of a two-step procedure (dimension reduction and function approximation) in discrete choice modeling with high-dimension data. This study proposes the SS-GAM procedure which is an extension of the Super Sparse Principal Component Analysis (SSPCA) where the results are further processed with the Generalized Additive Model (GAM) in a classification problem. Moreover, the Orthogonal Sparse Principal Component Analysis with GAM (OS-GAM) is also proposed. For baseline comparison, the General Adaptive Sparse-PCA with GAM (GAS-GAM) is considered in this paper. The performance of these three sparse PCA methods are investigated with varied underlying correlation structure. In the simulation study, it is demonstrated that with varied degree of dimensionality, and levels of correlation structure, SS-GAM performed better compared to OS-GAM and GAS-GAM in terms of its predictive rate, on the average. It was observed that the OS-GAM performed best when data exhibits low correlation structure. However, with high correlation structure, OS-GAM and GAS-GAM obtained comparable result. Moreover, in terms of computational time, OS-GAM seems to be not affected by the increase of feature dimension.

A fast trans-lasso algorithm with penalized weighted score function

An efficient transfer learning algorithm for high-dimensional sparse logistic regression models is proposed using penalized weighted score function based on square root Lasso, which intends to prespecify the tuning parameter. Three different choices of the tuning parameter are considered in the case of fixed design matrix. With a novel weight construction, the estimator of the regression vector is showed to be consistent when the inequality with respect to the Karush-Kuhn-Tucker (KKT) optimality conditions holds with high probability and the sparsity assumption for the regression vectors is required. There is information from source data to fit target data such that with high probability tending to 1-α, which is sharper than the corresponding probability bound without using the auxiliary samples, the KKT optimality conditions hold for the asymptotic choice. To detect which sources are transferable, an efficient data-driven method is proposed, which helps avoid negative transfer in practice. Simulation studies are carried out to demonstrate the numerical performance of the proposed procedure and their superiority over some existing methods. The procedures are also illustrated by analyzing the China Migrants Dynamic Survey dataset with binary outcomes concerning the associations among different provinces.

Multiple-hypothesis testing rules for high-dimensional model selection and sparse-parameter estimation

We consider the problem of model selection for high-dimensional sparse linear regression models. We pose the model selection problem as a multiple-hypothesis testing problem and employ the methods of false discovery rate (FDR) and familywise error rate (FER) to solve it. We also present the reformulation of the FDR/FER-based approaches as criterion-based model selection rules and establish their relation to the extended Bayesian Information Criterion (EBIC), which is a state-of-the-art high-dimensional model selection rule. We use numerical simulations to show that the proposed FDR/FER method is well suited for high-dimensional model selection and performs better than EBIC.

Robust penalized empirical likelihood in high dimensional longitudinal data analysis

As an effective nonparametric method, empirical likelihood (EL) is appealing in combining estimating equations flexibly and adaptively for incorporating data information. To select important variables and estimating equations in a sparse high-dimensional model, a penalized EL method based on robust estimating functions is proposed for regularizing the regression parameters and the associated Lagrange multipliers simultaneously, which allows the dimensionality of both regression parameters and estimating equations to grow exponentially with the sample size. A first inspection on the robustness of estimating equations contributing to variable selection is discussed from both theoretical perspective and intuitive simulation results in this paper. The proposed method can improve the robustness and effectiveness when the data have underlying outliers or heavy tails in the response variables and/or covariates. The robustness of the estimator is measured via the bounded influence function, and the oracle properties are also established under some regularity conditions. Extensive simulation studies and yeast cell data are used to evaluate the performance of the proposed method. The numerical results reveal that the robustness of sparse estimating equations selection fundamentally enhances variable selection accuracy when the data have heavy tails and/or include underlying outliers.

Variable selection and regularization via arbitrary rectangle-range generalized elastic net

We introduce the arbitrary rectangle-range generalized elastic net penalty method, abbreviated to ARGEN, for performing constrained variable selection and regularization in high-dimensional sparse linear models. As a natural extension of the nonnegative elastic net penalty method, ARGEN is proved to have both variable selection consistency and estimation consistency under some conditions. The asymptotic behavior in distribution of the ARGEN estimators have been studied in this framework. We also propose an algorithm called MU-QP-RR-W-l_1 to efficiently solve the ARGEN problem. By conducting simulation study we show that ARGEN outperforms the elastic net in a number of settings. Finally an application of S &P 500 index tracking with constraints on the stock allocations is performed to provide general guidance for adapting ARGEN to solve real-world problems.

Two-Directional Simultaneous Inference for High-Dimensional Models

This article proposes a general two-directional simultaneous inference (TOSI) framework for high-dimensional models with a manifest variable or latent variable structure, for example, high-dimensional mean models, high-dimensional sparse regression models, and high-dimensional latent factors models. TOSI performs simultaneous inference on a set of parameters from two directions, one to test whether the assumed zero parameters indeed are zeros and one to test whether exist zeros in the parameter set of nonzeros. As a result, we can better identify whether the parameters are zeros, thereby keeping the data structure fully and parsimoniously expressed. We theoretically prove that the single-split TOSI is asymptotically unbiased and the multi-split version of TOSI can control the Type I error below the prespecified significance level. Simulations are conducted to examine the performance of the proposed method in finite sample situations and two real datasets are analyzed. The results show that the TOSI method can provide more predictive and more interpretable estimators than existing methods.

Targeted Undersmoothing: Sensitivity Analysis for Sparse Estimators

Abstract This paper proposes a procedure for assessing the sensitivity of inferential conclusions for functionals of sparse high-dimensional models following model selection. The proposed procedure is called targeted undersmoothing. Functionals considered include dense functionals that may depend on many or all elements of the high-dimensional parameter vector. The sensitivity analysis is based on systematic enlargements of an initially selected model. By varying the enlargements, one can conduct sensitivity analysis about the strength of empirical conclusions to model selection mistakes. We illustrate the procedure's performance through simulation experiments and two empirical examples.

Sticky PDMP samplers for sparse and local inference problems

We construct a new class of efficient Monte Carlo methods based on continuous-time piecewise deterministic Markov processes (PDMPs) suitable for inference in high dimensional sparse models, i.e. models for which there is prior knowledge that many coordinates are likely to be exactly 0. This is achieved with the fairly simple idea of endowing existing PDMP samplers with “sticky” coordinate axes, coordinate planes etc. Upon hitting those subspaces, an event is triggered during which the process sticks to the subspace, this way spending some time in a sub-model. This results in non-reversible jumps between different (sub-)models. While we show that PDMP samplers in general can be made sticky, we mainly focus on the Zig-Zag sampler. Compared to the Gibbs sampler for variable selection, we heuristically derive favourable dependence of the Sticky Zig-Zag sampler on dimension and data size. The computational efficiency of the Sticky Zig-Zag sampler is further established through numerical experiments where both the sample size and the dimension of the parameter space are large.

A communication-efficient method for ℓ0 regularization linear regression models

We propose a communication-efficient distributed learning algorithm for high-dimensional sparse linear regression models in the scenario that the data are stored across multiple machines. Our approach is a distributed version of the SDAR [Huang J, Jiao Y, Liu Y, et al. A constructive approach to l0 penalized regression. J Mach Learn Res. 2018;19(1):403–439] method for solving the KKT system of the regularized least squares. At each step of the proposed method, the reduced least squares are solved by the steepest descent method, which only needs to calculate the gradient vectors on each node machine and communicate them instead of the data. We refer to this as SD-SDAR for brevity. Under some regular conditions, we obtain the sharp and error bounds for the solution sequences generated by SD-SDAR algorithm. We investigate the computational complexity and show that the number of rounds of communications are bounded by and , respectively, where J is the number of important predictors, R is the relative magnitude of the non-zero target coefficients, N is the total sample size and p is the dimension of covariates. Simulation studies illustrate that SD-SDAR outperforms some existing distributed methods in accuracy, efficiency and support recovery.

Statistical Capacity Matters: The Long-Term Effects of Africa’s Slave Trade on Development Reflected by Nighttime Light Intensity

Abstract Empirical research depends on reliable data. Yet, in many countries, statistical agencies do not have the capacity to collect high-quality data on economic development. This is especially the case in Africa, where the capacity to collect such data is affected by the same historical factors that explain economic development—in particular, the slave trade. We hypothesise that the impact of the slave trade on economic development in Africa is biased because cross-country heterogeneity in statistical capacity related to the slave trade creates a non-classical measurement error in gross domestic product (GDP) per capita. Our empirical evidence supports this view. When replacing GDP per capita by nighttime light intensity per capita—an indicator of economic development unrelated to statistical capacity—the impact of the slave trade on economic development drops by a factor of 2–4 depending on model specification and estimation methodology (OLS, IV and high-dimensional sparse models). Various robustness tests further corroborate our main hypothesis.

Consistency of survival tree and forest models: splitting bias and correction

Random survival forest and survival trees are popular models in statistics and machine learning. However, there is a lack of general understanding regarding consistency, splitting rules and influence of the censoring mechanism. In this paper, we investigate the statistical properties of existing methods from several interesting perspectives. First, we show that traditional splitting rules with censored outcomes rely on a biased estimation of the within-node failure distribution. To exactly quantify this bias, we develop a concentration bound of the within-node estimation based on non i.i.d. samples and apply it to the entire forest. Second, we analyze the entanglement between the failure and censoring distributions caused by univariate splits, and show that without correcting the bias at an internal node, survival tree and forest models can still enjoy consistency under suitable conditions. In particular, we demonstrate this property under two cases: a finite-dimensional case where the splitting variables and cutting points are chosen randomly, and a high-dimensional case where the covariates are weakly correlated. Our results can also degenerate into an independent covariate setting, which is commonly used in the random forest literature for high-dimensional sparse models. However, it may not be avoidable that the convergence rate depends on the total number of variables in the failure and censoring distributions. Third, we propose a new splitting rule that compares bias-corrected cumulative hazard functions at each internal node. We show that the rate of consistency of this new model depends only on the number of failure variables, which improves from non-bias-corrected versions. We perform simulation studies to confirm that this can substantially benefit the prediction error.

Does SLOPE outperform bridge regression?

Abstract A recently proposed SLOPE estimator [6] has been shown to adaptively achieve the minimax $\ell _2$ estimation rate under high-dimensional sparse linear regression models [25]. Such minimax optimality holds in the regime where the sparsity level $k$, sample size $n$ and dimension $p$ satisfy $k/p\rightarrow 0, k\log p/n\rightarrow 0$. In this paper, we characterize the estimation error of SLOPE under the complementary regime where both $k$ and $n$ scale linearly with $p$, and provide new insights into the performance of SLOPE estimators. We first derive a concentration inequality for the finite sample mean square error (MSE) of SLOPE. The quantity that MSE concentrates around takes a complicated and implicit form. With delicate analysis of the quantity, we prove that among all SLOPE estimators, LASSO is optimal for estimating $k$-sparse parameter vectors that do not have tied nonzero components in the low noise scenario. On the other hand, in the large noise scenario, the family of SLOPE estimators are sub-optimal compared with bridge regression such as the Ridge estimator.

A nonlinear mixed–integer programming approach for variable selection in linear regression model

Modern statistical studies often encounter regression models with high dimensions in which the number of features p is greater than the sample size n. Although the theory of linear models is well–established for the traditional assumption p < n, making valid statistical inference in high dimensional cases is a considerable challenge. With recent advances in technologies, the problem appears in many biological, medical, social, industrial, and economic studies. As known, the LASSO method is a popular technique for variable selection/estimation in high dimensional sparse linear models. Here, we show that the prediction performance of the LASSO method can be improved by eliminating the structured noises through a mixed–integer programming approach. As a result of our analysis, a modified variable selection/estimation scheme is proposed for a high dimensional regression model which can be considered as an alternative of the LASSO method. Some numerical experiments are made on the classical riboflavin production and some simulated data sets to shed light on the practical performance of the suggested method.

Kernel Meets Sieve: Post-Regularization Confidence Bands for Sparse Additive Model

We develop a novel procedure for constructing confidence bands for components of a sparse additive model. Our procedure is based on a new kernel-sieve hybrid estimator that combines two most popular nonparametric estimation methods in the literature, the kernel regression and the spline method, and is of interest in its own right. Existing methods for fitting sparse additive model are primarily based on sieve estimators, while the literature on confidence bands for nonparametric models are primarily based upon kernel or local polynomial estimators. Our kernel-sieve hybrid estimator combines the best of both worlds and allows us to provide a simple procedure for constructing confidence bands in high-dimensional sparse additive models. We prove that the confidence bands are asymptotically honest by studying approximation with a Gaussian process. Thorough numerical results on both synthetic data and real-world neuroscience data are provided to demonstrate the efficacy of the theory. Supplementary materials for this article are available online.

A Bootstrap Lasso + Partial Ridge Method to Construct Confidence Intervals for Parameters in High-dimensional Sparse Linear Models

Constructing confidence intervals for the coefficients of high-dimensional sparse linear models remains a challenge, mainly because of the complicated limiting distributions of the widely used estimators, such as the lasso. Several methods have been developed for constructing such intervals. Bootstrap lasso+ols is notable for its technical simplicity, good interpretability, and performance that is comparable with that of other more complicated methods. However, bootstrap lasso+ols depends on the beta-min assumption, a theoretic criterion that is often violated in practice. Thus, we introduce a new method, called bootstrap lasso+partial ridge, to relax this assumption. Lasso+partial ridge is a two-stage estimator. First, the lasso is used to select features. Then, the partial ridge is used to refit the coefficients. Simulation results show that bootstrap lasso+partial ridge outperforms bootstrap lasso+ols when there exist small, but nonzero coefficients, a common situation that violates the beta-min assumption. For such coefficients, the confidence intervals constructed using bootstrap lasso+partial ridge have, on average, $50\%$ larger coverage probabilities than those of bootstrap lasso+ols. Bootstrap lasso+partial ridge also has, on average, $35\%$ shorter confidence interval lengths than those of the de-sparsified lasso methods, regardless of whether the linear models are misspecified. Additionally, we provide theoretical guarantees for bootstrap lasso+partial ridge under appropriate conditions, and implement it in the R package "HDCI."

High-Dimensional Adaptive Minimax Sparse Estimation With Interactions

High-dimensional linear regression with interaction effects is broadly applied in research fields such as bioinformatics and social science. In this paper, we first investigate the minimax rate of convergence for regression estimation in high-dimensional sparse linear models with two-way interactions. We derive matching upper and lower bounds under three types of heredity conditions: strong heredity, weak heredity and no heredity. From the results: (i) A stronger heredity condition may or may not drastically improve the minimax rate of convergence. In fact, in some situations, the minimax rates of convergence are the same under all three heredity conditions; (ii) The minimax rate of convergence is determined by the maximum of the total price of estimating the main effects and that of estimating the interaction effects, which goes beyond purely comparing the order of the number of non-zero main effects $r_1$ and non-zero interaction effects $r_2$; (iii) Under any of the three heredity conditions, the estimation of the interaction terms may be the dominant part in determining the rate of convergence for two different reasons: 1) there exist more interaction terms than main effect terms or 2) a large ambient dimension makes it more challenging to estimate even a small number of interaction terms. Second, we construct an adaptive estimator that achieves the minimax rate of convergence regardless of the true heredity condition and the sparsity indices $r_1, r_2$.

Estimation and variable selection for a class of quantile regression models with multiple index

Quantile regression (QR) for a groupwise additive multiple-index models and its applications are investigated. We find that quantile regression can be used to recovery the directions of the index parameter vectors, and it does not involve the nonparametric treatment completely. Based on this useful finding, a iterative-free QR estimator for the partial linear single index model and a penalized QR for variable selection in the high dimensional sparse models are proposed respectively. Because of inheriting the superiorities of quantile regression, our methods are robust and comprehensive. Simulation studies and real data analysis are included to illustrate the finite sample performance.

On Tuning Parameter Selection in Model Selection and Model Averaging: A Monte Carlo Study

Model selection and model averaging are popular approaches for handling modeling uncertainties. The existing literature offers a unified framework for variable selection via penalized likelihood and the tuning parameter selection is vital for consistent selection and optimal estimation. Few studies have explored the finite sample performances of the class of ordinary least squares (OLS) post-selection estimators with the tuning parameter determined by different selection approaches. We aim to supplement the literature by studying the class of OLS post-selection estimators. Inspired by the shrinkage averaging estimator (SAE) and the Mallows model averaging (MMA) estimator, we further propose a shrinkage MMA (SMMA) estimator for averaging high-dimensional sparse models. Our Monte Carlo design features an expanding sparse parameter space and further considers the effect of the effective sample size and the degree of model sparsity on the finite sample performances of estimators. We find that the OLS post-smoothly clipped absolute deviation (SCAD) estimator with the tuning parameter selected by the Bayesian information criterion (BIC) in finite sample outperforms most penalized estimators and that the SMMA performs better when averaging high-dimensional sparse models.

Variable Selection via SCAD-Penalized Quantile Regression for High-Dimensional Count Data

This article introduces a quantile penalized regression technique for variable selection and estimation of conditional quantiles of counts in sparse high-dimensional models. The direct estimation and variable selection of the quantile regression is not feasible due to the discreteness of the count data and non-differentiability of the objective function, therefore, some smoothness must be artificially imposed on the problem. To achieve the necessary smoothness, we use the Jittering process by adding a uniformly distributed noise to the response count variable. The proposed method is compared with the existing penalized regression methods in terms of prediction accuracy and variable selection. We compare the proposed approach in zero-inflated count data regression models and in the presence of outliers. The performance and implementation of the proposed method are illustrated by detailed simulation studies and real data applications.

Nonparametric regression with adaptive truncation via a convex hierarchical penalty.

We consider the problem of nonparametric regression with a potentially large number of covariates. We propose a convex, penalized estimation framework that is particularly well suited to high-dimensional sparse additive models and combines the appealing features of finite basis representation and smoothing penalties. In the case of additive models, a finite basis representation provides a parsimonious representation for fitted functions but is not adaptive when component functions possess different levels of complexity. In contrast, a smoothing spline-type penalty on the component functions is adaptive but does not provide a parsimonious representation. Our proposal simultaneously achieves parsimony and adaptivity in a computationally efficient way. We demonstrate these properties through empirical studies and show that our estimator converges at the minimax rate for functions within a hierarchical class. We further establish minimax rates for a large class of sparse additive models. We also develop an efficient algorithm that scales similarly to the lasso with the number of covariates and sample size.