Lasso Estimator Research Articles

Continuous-time targeted minimum loss-based estimation of intervention-specific mean outcomes

This paper generalizes the targeted minimum loss-based estimation (TMLE) framework to allow for estimating the effects of time-varying interventions in settings where both interventions, covariates, and outcome can happen at subject-specific time-points on an arbitrarily fine time-scale. TMLE is a general template for constructing asymptotically linear substitution estimators for smooth low-dimensional parameters in infinite-dimensional models. Existing longitudinal TMLE methods are developed for data where observations are made on a discrete time-grid. We consider a continuous-time counting process model where intensity measures track the monitoring of subjects, and focus on a low-dimensional target parameter defined as the intervention-specific mean outcome at the end of follow-up. To construct our TMLE algorithm for the given statistical estimation problem, we derive an expression for the efficient influence curve and represent the target parameter as a functional of intensities and conditional expectations. The high-dimensional nuisance parameters of our model are estimated and updated in an iterative manner according to separate targeting steps for the involved intensities and conditional expectations. The resulting estimator solves the efficient influence curve equation. We state a general efficiency theorem and describe a highly adaptive lasso estimator for nuisance parameters that allows us to establish asymptotic linearity and efficiency of our estimator under minimal conditions on the underlying statistical model.

Estimation of the Spatial Weighting Matrix for Spatiotemporal Data under the Presence of Structural Breaks

ABSTRACT In this article, we propose a two-stage LASSO estimation approach for the estimation of a full spatial weight matrix of spatiotemporal autoregressive models. In addition, we allow for an unknown number of structural breaks in the local means of each spatial location. These locally varying mean levels, however, can easily be mistaken as spatial dependence and vice versa. Thus, the proposed approach jointly estimates the spatial dependence, all structural breaks, and the local mean levels. For selection of the penalty parameter, we propose a completely new selection criterion based on the distance between the empirical spatial autocorrelation and the spatial dependence estimated in the model. Through simulation studies, we will show the finite-sample performance of the estimators and provide practical guidance as to when the approach could be applied. Finally, the method will be illustrated by an empirical example of intra-city monthly real-estate prices in Berlin between 1995 and 2014. The spatial units will be defined by the respective postal codes. The new approach allows us to estimate local mean levels and quantify the deviation of the observed prices from these levels due to spatial spillover effects. In doing so, the entire spatial dependence structure is estimated on a data-driven basis. Supplementary materials for this article are available online.

Comparison of Some Group Variable Selection Methods in High Dimensional Multiple Linear Regression via Simulation Study

In many applications, covariates possess a grouping structure that can be incorporated into the analysis to select important groups as well as important members of those groups. In this paper, we reviewed some group variable selection methods in the penalized regression model. This paper investigates by comparing the performance of seven previously proposed group variable selection methods; the group Lasso estimates, the group Lasso net estimates, the sparse group Lasso estimates, the group scad estimates, the group scad net estimates, the group mcp estimates, and the group gel estimate via a simulation study. The simulation study is used in determining which methods are best in all of the linear regression scenarios.

Fast Lasso-type safe screening for Fine-Gray competing risks model with ultrahigh dimensional covariates.

The Fine-Gray proportional sub-distribution hazards (PSH) model is among the most popular regression model for competing risks time-to-event data. This article develops a fast safe feature elimination method, named PSH-SAFE, for fitting the penalized Fine-Gray PSH model with a Lasso (or adaptive Lasso) penalty. Our PSH-SAFE procedure is straightforward to implement, fast, and scales well to ultrahigh dimensional data. We also show that as a feature screening procedure, PSH-SAFE is safe in a sense that the eliminated features are guaranteed to be inactive features in the original Lasso (or adaptive Lasso) estimator for the penalized PSH model. We evaluate the performance of the PSH-SAFE procedure in terms of computational efficiency, screening efficiency and safety, run-time, and prediction accuracy on multiple simulated datasets and a real bladder cancer data. Our empirical results show that the PSH-SAFE procedure possesses desirable screening efficiency and safety properties and can offer substantially improved computational efficiency as well as similar or better prediction performance in comparison to their baseline competitors.

Degrees of freedom for off-the-grid sparse estimation

The degrees of freedom quantify the number of parameters of an estimator and is central to various risk minimization procedures. Its computation and properties are well understood when dealing with finite dimensional linear models, possibly regularized using sparsity through the celebrated Lasso estimator. However, for some applications (such as super-resolution in imaging or training multi-layer perceptrons with a single hidden layer) it makes sense to rather consider “continuous” methods. For these applications, these methods avoid the discretization of the parameter space and lead to more efficient numerical solvers. Training these continuous models with a sparsity inducing prior can be achieved by solving a convex optimization problem over the infinite dimensional space of measures, which is often called the Beurling Lasso (Blasso), and is the continuous counterpart of the Lasso. Previous works (Ann. Statist. 35 (2007) 2173–2192; Statist. Sinica 23 (2013) 809–828) show that the size of the smallest solution support is an unbiased estimator of the degrees of freedom of the Lasso. Our main contribution is a proof of a continuous counterpart to this result for the Blasso. In contrast to the Lasso, our new formula shows that the empirical degrees of freedom of the Blasso is not proportional to the size of the support. Our second contribution is a detailed study of the case of sampling Fourier coefficients in 1D, which corresponds to a super-resolution problem. We show that our formula for the unbiased estimation of the degrees of freedom is valid outside of a set of measure zero of observations, which in turn justifies its use to compute an unbiased estimator of the prediction risk using the Stein Unbiased Risk Estimator (SURE). We also report numerical results for both the case of Fourier sampling and the learning of a multilayers perceptron with a single hidden layer.

Quantum Differentially Private Sparse Regression Learning

The eligibility of various advanced quantum algorithms will be questioned if they can not guarantee privacy. To fill this knowledge gap, here we devise an efficient quantum differentially private (QDP) Lasso estimator to solve sparse regression tasks. Concretely, given $N$ $d$-dimensional data points with $N\ll d$, we first prove that the optimal classical and quantum non-private Lasso requires $\Omega(N+d)$ and $\Omega(\sqrt{N}+\sqrt{d})$ runtime, respectively. We next prove that the runtime cost of QDP Lasso is \textit{dimension independent}, i.e., $O(N^{5/2})$, which implies that the QDP Lasso can be faster than both the optimal classical and quantum non-private Lasso. Last, we exhibit that the QDP Lasso attains a near-optimal utility bound $\tilde{O}(N^{-2/3})$ with privacy guarantees and discuss the chance to realize it on near-term quantum chips with advantages.

Economic survival duration of Thai workers during COVID-19

This study examines the impact of the COVID-19 pandemic on the livelihood and economic survival of Thai citizen workers, using The Asia Foundation’s survey data which were conducted in May 2020 (first round), August 2020 (second round) and November 2020 (third round). We adopt the Cox proportional-hazards regression with lasso estimation to estimate the coefficients and perform variable selection simultaneously. The model allows us to identify the vulnerable groups with risks of consumption inadequacy. The empirical results show that those workers characterized as low-educated, unemployed, unskilled, working in the tourism sector and living in the northeastern or southern regions are less likely to sustain their consumption. However, our study highlights that higher education is a crucial factor influencing the survivability of Thai workers. Regarding the role of government schemes, the result shows that that a set of cash assistance programs is less likely to increase the survivability of the non-agricultural workers.

The effect of nonpharmaceutical interventions on COVID-19 infections for lower and middle-income countries: A debiased LASSO approach.

This paper investigates the determinants of COVID-19 infection in the first 100 days of government actions. Using a debiased LASSO estimator, we explore how different measures of government nonpharmaceutical interventions affect new infections of COVID-19 for 37 lower and middle-income countries (LMCs). We find that closing schools, stay-at-home restrictions, and contact tracing reduce the growth of new infections, as do economic support to households and the number of health care workers. Notably, we find no significant effects of business closures. Finally, infections become higher in countries with greater income inequality, higher tourist inflows, poorly educated adults, and weak governance quality. We conclude that several policy interventions reduce infection rates for poorer countries. Further, economic and institutional factors are important; thereby justifying the use, and ultimately success, of economic support to households during the initial infection period.

Efficient estimation of pathwise differentiable target parameters with the undersmoothed highly adaptive lasso.

We consider estimation of a functional parameter of a realistically modeled data distribution based on observing independent and identically distributed observations. The highly adaptive lasso estimator of the functional parameter is defined as the minimizer of the empirical risk over a class of cadlag functions with finite sectional variation norm, where the functional parameter is parametrized in terms of such a class of functions. In this article we establish that this HAL estimator yields an asymptotically efficient estimator of any smooth feature of the functional parameter under a global undersmoothing condition. It is formally shown that the L 1-restriction in HAL does not obstruct it from solving the score equations along paths that do not enforce this condition. Therefore, from an asymptotic point of view, the only reason for undersmoothing is that the true target function might not be complex so that the HAL-fit leaves out key basis functions that are needed to span the desired efficient influence curve of the smooth target parameter. Nonetheless, in practice undersmoothing appears to be beneficial and a simple targeted method is proposed and practically verified to perform well. We demonstrate our general result HAL-estimator of a treatment-specific mean and of the integrated square density. We also present simulations for these two examples confirming the theory.

Relaxed Adaptive Lasso and Its Asymptotic Results

This article introduces a novel two-stage variable selection method to solve the common asymmetry problem between the response variable and its influencing factors. In practical applications, we cannot correctly extract important factors from a large amount of complex and redundant data. However, the proposed method based on the relaxed lasso and the adaptive lasso, namely, the relaxed adaptive lasso, can achieve information symmetry because the variables it selects contain all the important information about the response variables. The goal of this paper is to preserve the relaxed lasso’s superior variable selection speed while imposing varying penalties on different coefficients. Additionally, the proposed method enjoys favorable asymptotic properties, that is, consistency with a fast rate of convergence with Opn−1. The simulation demonstrates that the proper variable recovery, i.e., the number of significant variables selected, and prediction accuracy of the relaxed adaptive lasso in a limited sample is superior to the regular lasso, relaxed lasso and adaptive lasso estimators.

High-Dimensional Granger Causality Tests with an Application to VIX and News

Abstract We study Granger causality testing for high-dimensional time series using regularized regressions. To perform proper inference, we rely on heteroskedasticity and autocorrelation consistent (HAC) estimation of the asymptotic variance and develop the inferential theory in the high-dimensional setting. To recognize the time-series data structures, we focus on the sparse-group LASSO (sg-LASSO) estimator, which includes the LASSO and the group LASSO as special cases. We establish the debiased central limit theorem for low-dimensional groups of regression coefficients and study the HAC estimator of the long-run variance based on the sg-LASSO residuals. This leads to valid time-series inference for individual regression coefficients as well as groups, including Granger causality tests. The treatment relies on a new Fuk–Nagaev inequality for a class of τ-mixing processes with heavier than Gaussian tails, which is of independent interest. In an empirical application, we study the Granger causal relationship between the VIX and financial news.

Penalized estimation of panel vector autoregressive models: A panel LASSO approach

This paper proposes LASSO estimation specific for panel vector autoregressive (PVAR) models. The penalty term allows for shrinkage for different lags, for shrinkage towards homogeneous coefficients across panel units, for penalization of lags of variables belonging to another cross-sectional unit, and for varying penalization across equations. The penalty parameters therefore build on time series and cross-sectional properties that are commonly found in PVAR models. Simulation results point towards advantages of using the proposed LASSO for PVAR models over ordinary least squares in terms of forecast accuracy. An empirical forecasting application including 20 countries supports these findings.

Extraction of Continuous and Discrete Spatial Heterogeneities: Fusion Model of Spatially Varying Coefficient Model and Sparse Modelling

Geospatial phenomena often have spatial heterogeneity, which is caused by differences in the data generation process from place to place. There are two types of spatial heterogeneity: continuous and discrete, and there has been much discussion about how to analyze one type of spatial heterogeneity. Although geospatial phenomena can have both types of spatial heterogeneities, previous studies have not sufficiently discussed how to consider these two different types of spatial heterogeneity simultaneously and how to detect them separately, which may lead to biased estimates and the wrong interpretation of geospatial phenomena. This study proposes a new approach for the analysis of spatial data with both heterogeneities by combining the eigenvector spatial filtering-based spatially varying coefficient (ESF-SVC) model, which assumes the continuous spatial heterogeneity and generalized lasso (GL) estimation, which assumes discrete spatial heterogeneity and proposes the ESF-GL-SVC model. The performance of ESF-GL-SVC was evaluated through experiments based on a Monte Carlo simulation and confirms that the ESF-GL-SVC showed better performance in estimating coefficients with both types of spatial heterogeneity than the previous two models. The application of the apartment rent data showed that the ESF-GL-SVC outputs the result with the smallest BIC value, and the estimated coefficients depict continuous and discrete spatial heterogeneity in the dataset. Reasonable coefficients were estimated using the ESF-GL-SVC, although some coefficients by ESF-SVC were not.

Moving beyond sub-Gaussianity in high-dimensional statistics: applications in covariance estimation and linear regression

Abstract Concentration inequalities form an essential toolkit in the study of high-dimensional statistical methods. Most of the relevant statistics literature in this regard is, however, based on the assumptions of sub-Gaussian or sub-exponential random variables/vectors. In this paper, we first bring together, through a unified exposition, various probabilistic inequalities for sums of independent random variables under much more general exponential type (namely sub-Weibull) tail assumptions. These results extract a part sub-Gaussian tail behavior of the sum in finite samples, matching the asymptotics governed by the central limit theorem, and are compactly represented in terms of a new Orlicz quasi-norm—the Generalized Bernstein–Orlicz norm—that typifies such kind of tail behaviors. We illustrate the usefulness of these inequalities through the analysis of four fundamental problems in high-dimensional statistics. In the first two problems, we study the rate of convergence of the sample covariance matrix in terms of the maximum elementwise norm and the maximum $k$-sub-matrix operator norm that are key quantities of interest in bootstrap procedures and high-dimensional structured covariance matrix estimation, as well as in high-dimensional and post-selection inference. The third example concerns the restricted eigenvalue condition, required in high-dimensional linear regression, which we verify for all sub-Weibull random vectors through a unified analysis, and also prove a more general result related to restricted strong convexity in the process. In the final example, we consider the Lasso estimator for linear regression and establish its rate of convergence to be generally $\sqrt{k\log p/n}$, for $k$-sparse signals, under much weaker than usual tail assumptions (on the errors as well as the covariates), while also allowing for misspecified models and both fixed and random design. To our knowledge, these are the first such results for Lasso obtained in this generality. The common feature in all our results over all the examples is that the convergence rates under most exponential tails match the usual (optimal) ones obtained under sub-Gaussian assumptions. Finally, we also establish some complementary results on analogous tail bounds for the suprema of empirical processes indexed by sub-Weibull variables. All our results are finite samples.

Finite-sample results for lasso and stepwise Neyman-orthogonal Poisson estimators

High-dimensional models that include many covariates which might potentially affect an outcome are increasingly common. This paper begins by introducing a lasso-based approach and a stepwise-based approach to valid inference for a high-dimensional model. It then discusses several essential extensions to the literature that make the estimators more usable in practice. Finally, it presents Monte Carlo evidence to help applied researchers choose which of several available estimators should be used in practice. The Monte Carlo evidence shows that our extensions to the literature perform well. It also shows that a BIC-stepwise approach performs well for a data-generating process for which the lasso-based approaches and a testing-stepwise approach fail. The Monte Carlo evidence also indicates the BIC-based lasso and plugin-based lasso can produce better inferential results than the ubiquitous CV-based lasso. Easy-to-use Stata commands are available for all the methods that we discuss.

DOUBLY DEBIASED LASSO: HIGH-DIMENSIONAL INFERENCE UNDER HIDDEN CONFOUNDING.

Inferring causal relationships or related associations from observational data can be invalidated by the existence of hidden confounding. We focus on a high-dimensional linear regression setting, where the measured covariates are affected by hidden confounding and propose the Doubly Debiased Lasso estimator for individual components of the regression coefficient vector. Our advocated method simultaneously corrects both the bias due to estimation of high-dimensional parameters as well as the bias caused by the hidden confounding. We establish its asymptotic normality and also prove that it is efficient in the Gauss-Markov sense. The validity of our methodology relies on a dense confounding assumption, i.e. that every confounding variable affects many covariates. The finite sample performance is illustrated with an extensive simulation study and a genomic application.

High dimensional threshold model with a time-varying threshold based on Fourier approximation

Abstract This paper studies high-dimensional threshold models with a time-varying threshold approximated using a Fourier function. We develop a weighted LASSO estimator of regression coefficients as well as the threshold parameters. Our LASSO estimator can not only select covariates but also distinguish between linear and threshold models. We derive non-asymptotic oracle inequalities for the prediction risk, the l 1 and l ∞ bounds for regression coefficients, and provide an upper bound on the l 1 estimation error of the time-varying threshold estimator. The bounds can be translated easily into asymptotic consistency for prediction and estimation. We also establish the variable selection consistency and threshold detection consistency based on the l ∞ bounds. Through Monte Carlo simulations, we show that the thresholded LASSO works reasonably well in finite samples in terms of variable selection, and there is little harmness by the allowance for Fourier approximation in the estimation procedure even when there is no time-varying feature in the threshold. On the contrary, the estimation and variable selection are inconsistent when the threshold is time-varying but being misspecified as a constant. The model is illustrated with an empirical application to the famous debt-growth nexus.

Uniform inference in high-dimensional Gaussian graphical models

Summary Graphical models have become a popular tool for representing dependencies within large sets of variables and are crucial for representing causal structures. We provide results for uniform inference on high-dimensional graphical models, in which the number of target parameters $d$ is potentially much larger than the sample size, under approximate sparsity. Our results highlight how graphical models can be estimated and recovered using modern machine learning methods in high-dimensional complex settings. To construct simultaneous confidence regions on many target parameters, it is crucial to have sufficiently fast estimation rates of the nuisance functions. In this context, we establish uniform estimation rates and sparsity guarantees for the square-root lasso estimator in a random design under approximate sparsity conditions. These might be of independent interest for related problems in high dimensions. We also demonstrate in a comprehensive simulation study that our procedure has good small sample properties in comparison to existing methods, and we present two empirical applications.

Heterogeneous Overdispersed Count Data Regressions via Double-Penalized Estimations

Recently, the high-dimensional negative binomial regression (NBR) for count data has been widely used in many scientific fields. However, most studies assumed the dispersion parameter as a constant, which may not be satisfied in practice. This paper studies the variable selection and dispersion estimation for the heterogeneous NBR models, which model the dispersion parameter as a function. Specifically, we proposed a double regression and applied a double ℓ1-penalty to both regressions. Under the restricted eigenvalue conditions, we prove the oracle inequalities for the lasso estimators of two partial regression coefficients for the first time, using concentration inequalities of empirical processes. Furthermore, derived from the oracle inequalities, the consistency and convergence rate for the estimators are the theoretical guarantees for further statistical inference. Finally, both simulations and a real data analysis demonstrate that the new methods are effective.

Sliding Window Strategy for Convolutional Spike Sorting with Lasso

Spike sorting is a class of algorithms used in neuroscience to attribute the time occurrences of particular electric signals, called action potential or spike, to neurons. We rephrase this problem as a particular optimization problem: Lasso for convolutional models in high dimension. Lasso (i.e. least absolute shrinkage and selection operator) is a very generic tool in machine learning that help us to look for sparse solutions (here the time occurrences). However, for the size of the problem at hand in this neuroscience context, the classical Lasso solvers are failing. We present here a new and much faster algorithm. Making use of biological properties related to neurons, we explain how the particular structure of the problem allows several optimizations, leading to an algorithm with a temporal complexity which grows linearly with respect to the size of the recorded signal and can be performed online. Moreover the spatial separability of the initial problem allows to break it into subproblems, further reducing the complexity and making possible its application on the latest recording devices which comprise a large number of sensors. We provide several mathematical results: the size and numerical complexity of the subproblems can be estimated mathematically by using percolation theory. We also show under reasonable assumptions that the Lasso estimator retrieves the true time occurrences of the spikes with large probability. Finally the theoretical time complexity of the algorithm is given. Numerical simulations are also provided in order to illustrate the efficiency of our approach.