Adaptive Lasso Estimator Research Articles

Variable selection for semivarying coefficient models via local averaging

AbstractThis study aims to provide novel insights into variable selection in the semivarying coefficient model. We focus on the problem of variable selection and screening for the constant coefficient part. A common approach in the existing literature is to infer the constant coefficients by transforming the problem into a linear model scenario, utilizing a fine estimator of the varying coefficients. In this paper, we propose an approximation method for the varying coefficient functions using local averaging, which is characterized by its simplicity, rough and computational efficiency. Additionally, we introduce an adaptive lasso estimator and a forward regression algorithm specifically designed for semivarying coefficient models. Theoretical and experimental results highlight the effectiveness of the local averaging method in extending variable selection techniques from the linear model to the semivarying coefficient model. Our proposed approaches demonstrate a significant improvement in inference speed compared with baseline methods, with little loss of asymptotic efficiency.

On the adaptive Lasso estimator of AR(p) time series with applications to INAR(p) and Hawkes processes

We investigate the consistency and the rate of convergence of the adaptive Lasso estimator for the parameters of linear AR(p) time series with a white noise which is a strictly stationary and ergodic martingale difference. Roughly speaking, we prove that (i) If the white noise has a finite second moment, then the adaptive Lasso estimator is almost sure consistent (ii) If the white noise has a finite fourth moment, then the error estimate converges to zero with the same rate as the regularizing parameters of the adaptive Lasso estimator. Such theoretical findings are applied to estimate the parameters of INAR(p) time series and to estimate the fertility function of Hawkes processes. The results are validated by some numerical simulations, which show that the adaptive Lasso estimator allows for a better balancing between bias and variance with respect to the Conditional Least Square estimator and the classical Lasso estimator.

Estimation of time-specific intervention effects on continuously distributed time-to-event outcomes by targeted maximum likelihood estimation.

This work considers targeted maximum likelihood estimation (TMLE) of treatment effects on absolute risk and survival probabilities in classical time-to-event settings characterized by right-censoring and competing risks. TMLE is a general methodology combining flexible ensemble learning and semiparametric efficiency theory in a two-step procedure for substitution estimation of causal parameters. We specialize and extend the continuous-time TMLE methods for competing risks settings, proposing a targeting algorithm that iteratively updates cause-specific hazards to solve the efficient influence curve equationfor the target parameter. As part of the work, we further detail and implement the recently proposed highly adaptive lasso estimator for continuous-time conditional hazards with L1 -penalized Poisson regression. The resulting estimation procedure benefits from relying solely on very mild nonparametric restrictions on the statistical model, thus providing a novel tool for machine-learning-based semiparametric causal inference for continuous-time time-to-event data. We apply the methods to a publicly available dataset on follicular cell lymphoma where subjects are followed over time until disease relapse or death without relapse. The data display important time-varying effects that can be captured by the highly adaptive lasso. In our simulations that are designed to imitate the data, we compare our methods to a similar approach based on random survival forests and to the discrete-timeTMLE.

Continuous-time targeted minimum loss-based estimation of intervention-specific mean outcomes

This paper generalizes the targeted minimum loss-based estimation (TMLE) framework to allow for estimating the effects of time-varying interventions in settings where both interventions, covariates, and outcome can happen at subject-specific time-points on an arbitrarily fine time-scale. TMLE is a general template for constructing asymptotically linear substitution estimators for smooth low-dimensional parameters in infinite-dimensional models. Existing longitudinal TMLE methods are developed for data where observations are made on a discrete time-grid. We consider a continuous-time counting process model where intensity measures track the monitoring of subjects, and focus on a low-dimensional target parameter defined as the intervention-specific mean outcome at the end of follow-up. To construct our TMLE algorithm for the given statistical estimation problem, we derive an expression for the efficient influence curve and represent the target parameter as a functional of intensities and conditional expectations. The high-dimensional nuisance parameters of our model are estimated and updated in an iterative manner according to separate targeting steps for the involved intensities and conditional expectations. The resulting estimator solves the efficient influence curve equation. We state a general efficiency theorem and describe a highly adaptive lasso estimator for nuisance parameters that allows us to establish asymptotic linearity and efficiency of our estimator under minimal conditions on the underlying statistical model.

Fast Lasso-type safe screening for Fine-Gray competing risks model with ultrahigh dimensional covariates.

The Fine-Gray proportional sub-distribution hazards (PSH) model is among the most popular regression model for competing risks time-to-event data. This article develops a fast safe feature elimination method, named PSH-SAFE, for fitting the penalized Fine-Gray PSH model with a Lasso (or adaptive Lasso) penalty. Our PSH-SAFE procedure is straightforward to implement, fast, and scales well to ultrahigh dimensional data. We also show that as a feature screening procedure, PSH-SAFE is safe in a sense that the eliminated features are guaranteed to be inactive features in the original Lasso (or adaptive Lasso) estimator for the penalized PSH model. We evaluate the performance of the PSH-SAFE procedure in terms of computational efficiency, screening efficiency and safety, run-time, and prediction accuracy on multiple simulated datasets and a real bladder cancer data. Our empirical results show that the PSH-SAFE procedure possesses desirable screening efficiency and safety properties and can offer substantially improved computational efficiency as well as similar or better prediction performance in comparison to their baseline competitors.

Efficient estimation of pathwise differentiable target parameters with the undersmoothed highly adaptive lasso.

We consider estimation of a functional parameter of a realistically modeled data distribution based on observing independent and identically distributed observations. The highly adaptive lasso estimator of the functional parameter is defined as the minimizer of the empirical risk over a class of cadlag functions with finite sectional variation norm, where the functional parameter is parametrized in terms of such a class of functions. In this article we establish that this HAL estimator yields an asymptotically efficient estimator of any smooth feature of the functional parameter under a global undersmoothing condition. It is formally shown that the L 1-restriction in HAL does not obstruct it from solving the score equations along paths that do not enforce this condition. Therefore, from an asymptotic point of view, the only reason for undersmoothing is that the true target function might not be complex so that the HAL-fit leaves out key basis functions that are needed to span the desired efficient influence curve of the smooth target parameter. Nonetheless, in practice undersmoothing appears to be beneficial and a simple targeted method is proposed and practically verified to perform well. We demonstrate our general result HAL-estimator of a treatment-specific mean and of the integrated square density. We also present simulations for these two examples confirming the theory.

Relaxed Adaptive Lasso and Its Asymptotic Results

This article introduces a novel two-stage variable selection method to solve the common asymmetry problem between the response variable and its influencing factors. In practical applications, we cannot correctly extract important factors from a large amount of complex and redundant data. However, the proposed method based on the relaxed lasso and the adaptive lasso, namely, the relaxed adaptive lasso, can achieve information symmetry because the variables it selects contain all the important information about the response variables. The goal of this paper is to preserve the relaxed lasso’s superior variable selection speed while imposing varying penalties on different coefficients. Additionally, the proposed method enjoys favorable asymptotic properties, that is, consistency with a fast rate of convergence with Opn−1. The simulation demonstrates that the proper variable recovery, i.e., the number of significant variables selected, and prediction accuracy of the relaxed adaptive lasso in a limited sample is superior to the regular lasso, relaxed lasso and adaptive lasso estimators.

Combining data envelopment analysis and stochastic frontiers via a LASSO prior

Technical inefficiencies in stochastic frontier models can be thought of as non-negative parameters. Since, however, their number along with other parameters exceeds the sample size, an adaptive LASSO estimator is a reasonable way to overcome the problem, especially in view of the oracle properties of the estimator under broad conditions. It is shown that the adaptive LASSO estimator can be thought of as the posterior mean of a usual stochastic frontier model with a special prior that benchmarks inefficiencies on known quantities. We take these quantities from DEA scores to obtain technical inefficiencies having oracle properties. The LASSO parameters can be estimated routinely in the Bayesian context without the need for cross-validation. In an application to a data set of large U.S. banks we find that adaptive LASSO outperforms significantly traditional stochastic frontier models.

Extended Oracle Properties of Adaptive Lasso Estimators

We study the asymptotic properties of adaptive lasso estimators when some components of the parameter of interest β are strictly different than zero, while other components may be zero or may converge to zero with rate n-δ, with δ>0, where n denotes the sample size. To achieve this objective, we analyze the convergence/divergence rates of each term in the first-order conditions of adaptive lasso estimators. First, we derive conditions that allow selecting tuning parameters in order to ensure that adaptive lasso estimates of n-δ-components indeed collapse to zero. Second, in this case, we also derive asymptotic distributions of adaptive lasso estimators for nonzero components. When δ>1/2, we obtain the usual n1/2-asymptotic normal distribution, while when 0δ ≤ 1/2, we show nδ-consistency combined with (biased) n1/2-δ-asymptotic normality for nonzero components. We call these properties, Extended Oracle Properties. These results allow practitioners to exclude in their model the asymptotically negligible variables and make inferences on the asymptotically relevant variables.

An adaptive group LASSO approach for domain selection in functional generalized linear models

This paper focuses on estimation and null region detection of the coefficient function for functional generalized linear models. Traditional estimating approaches cannot serve as null regions detectors. To simultaneously estimate coefficient functions and detect corresponding important subregions in functional generalized linear models, an adaptive group LASSO approach with B-spline smoothing technique is developed. The convergence rate of the resulting estimator is obtained. The consistency of domain selection and limiting distribution of the proposed estimator are then established, which is not straightforward since the groups to be penalized are overlapping. These asymptotic properties are also supported by extensive simulation studies. The resulting estimator performs better than direct adaptive LASSO estimators and some existing functional generalized linear models estimators, which are obtained without considering the domain selection. A real data application reveals the effectiveness of the proposed method.

Ultrahigh‐dimensional generalized additive model: Unified theory and methods

AbstractGeneralized additive model is a powerful statistical learning and predictive modeling tool that has been applied in a wide range of applications. The need of high‐dimensional additive modeling is eminent in the context of dealing with high throughput data such as genetics data analysis. In this article, we studied a two‐step selection and estimation method for ultrahigh‐dimensional generalized additive models. The first step applies group lasso on the expanded bases of the functions. With high probability this selects all nonzero functions without having too much over selection. The second step uses adaptive group lasso with any initial estimators, including the group lasso estimator, that satisfies some regular conditions. The adaptive group lasso estimator is shown to be selection consistent with improved convergence rates. Tuning parameter selection is also discussed and shown to select the true model consistently under generalized information criterion procedure. The theoretical properties are supported by extensive numerical study.

Ridge parameters optimization based on minimizing model selection criterion in multivariate generalized ridge regression

A multivariate generalized ridge (MGR) regression provides a shrinkage estimator of the multivariate linear regression by multiple ridge parameters. Since the ridge parameters which adjust the amount of shrinkage of the estimator are unknown, their optimization is an important task to obtain a better estimator. For the univariate case, a fast algorithm has been proposed for optimizing ridge parameters based on minimizing a model selection criterion (MSC) and the algorithm can be applied to various MSCs. In this paper, we extend this algorithm to MGR regression. We also describe the relationship between the MGR estimator which is not sparse and a multivariate adaptive group Lasso estimator which is sparse, under orthogonal explanatory variables.

Multiple structural breaks in cointegrating regressions: a model selection approach

Abstract In this paper, we propose a new approach to model structural change in cointegrating regressions using penalized regression techniques. First, we consider a setting with known breakpoint candidates and show that a modified adaptive lasso estimator can consistently estimate structural breaks in the intercept and slope coefficient of a cointegrating regression. Second, we extend our approach to a diverging number of breakpoint candidates and provide simulation evidence that timing and magnitude of structural breaks are consistently estimated. Third, we use the adaptive lasso estimation to design new tests for cointegration in the presence of multiple structural breaks, derive the asymptotic distribution of our test statistics and show that the proposed tests have power against the null of no cointegration. Finally, we use our new methodology to study the effects of structural breaks on the long-run PPP relationship.

UNIFORM ASYMPTOTICS AND CONFIDENCE REGIONS BASED ON THE ADAPTIVE LASSO WITH PARTIALLY CONSISTENT TUNING

We consider the adaptive Lasso estimator with componentwise tuning in the framework of a low-dimensional linear regression model. In our setting, at least one of the components is penalized at the rate of consistent model selection and certain components may not be penalized at all. We perform a detailed study of the consistency properties and the asymptotic distribution which includes the effects of componentwise tuning within a so-called moving-parameter framework. These results enable us to explicitly provide a set $\mathcal {M}$ such that every open superset acts as a confidence set with uniform asymptotic coverage equal to 1, whereas removing an arbitrarily small open set along the boundary yields a confidence set with uniform asymptotic coverage equal to 0. The shape of the set $\mathcal {M}$ depends on the regressor matrix as well as the deviations within the componentwise tuning parameters. Our findings can be viewed as a broad generalization of Pötscher and Schneider (2009, Journal of Statistical Planning and Inference 139, 2775–2790; 2010, Electronic Journal of Statistics 4, 334–360), who considered distributional properties and confidence intervals based on components of the adaptive Lasso estimator for the case of orthogonal regressors.

Inferring latent heterogeneity using many feature variables supervised by survival outcome.

In cancer studies, it is important to understand disease heterogeneity among patients so that precision medicine can particularly target high-risk patients at the right time. Many feature variables such as demographic variables and biomarkers, combined with a patient's survival outcome, can be used to infer such latent heterogeneity. In this work, we propose a mixture model to model each patient's latent survival pattern, where the mixing probabilities for latent groups are modeled through a multinomial distribution. The Bayesian information criterion is used for selecting the number of latent groups. Furthermore, we incorporate variable selection with the adaptive lasso into inference so that only a few feature variables will be selected to characterize the latent heterogeneity. We show that our adaptive lasso estimator has oracle properties when the number of parameters diverges with the sample size. The finite sample performance is evaluated by the simulation study, and the proposed method is illustrated by two datasets.

The Prior Adaptive Group Lasso with an Application to Risk Factor Selection

This paper develops and presents the prior adaptive group lasso for generalized linear models. The prior adaptive group lasso is an extension of the prior lasso developed by Jiang, He, and Zhang (2016), which allows for the use of existing information from previous or similar studies in the estimation of the lasso. We demonstrate that the estimator exhibits consistent variable selection and estimation similarly to those derived in Wang and Tian (2019) under at set of similar conditions. The performance of the prior adaptive group lasso estimator is illustrated in a Monte Carlo study. Finally, the estimator is applied in selecting the set of relevant risk factors in asset pricing models conditioning on the fact that the chosen factors must be able to price the test assets as well as the remaining factors. The empirical study shows that the prior adaptive group lasso yields a set of factors that explain the time variation in the returns while delivering 𝛼 estimates close to zero. We also show how this set of factors has evolved over time. We find that the canonical factor models of Fama and French (1993), (Carhart, 1997), (Fama and French, 2015), and (Hou, Xue, and Zhang, 2015) are insufficient to price the cross section of the test assets together with the remaining traded factors, and we find that the required number of pricing factors to include at any given time is closer to 20.

Bayesian bootstrap adaptive lasso estimators of regression models

ABSTRACT This paper proposes a modified adaptive lasso method by the Bayesian bootstrap (BBAL) and approximates the posterior distributions of parameters for a linear and a logistic regression model, respectively. The BBAL estimators are proved to have asymptotic and Oracle properties and they are acquired by the coordinate descent algorithm which could get the solutions at the grid of values of the penalty parameter λ. Three numerical experiments are conducted to demonstrate the BBAL method. Test results show the consistency of the variable selection and result in more robust estimators. And we use the median coefficients of the BBAL estimators to do the prediction with a medical dataset.

A robust adaptive Lasso estimator for the independent contamination model

The Lasso has become a benchmark method for simultaneous parameter estimation and variable selection in regression analysis. It is based on the least-squares estimator and, therefore, suffers from the presence of outliers. Robust Lasso methods combine the objective function of a robust estimator with ℓ1-penalization. We address robustness for cases in which the number of observations is smaller (or not much larger) than the number of predictors. Further, we assume that the regression matrix may contain cellwise outliers. In such settings, even a few highly contaminated predictors can cause existing robust methods that are based on the commonly used rowwise contamination model to break down. Therefore, we propose a new adaptive Lasso type regularization. It takes into account cellwise outlyingness in the regression matrix and uses this information for robust variable selection. The proposed regularization term is integrated into the objective function of the MM-estimator, which yields the proposed MM-Robust Weighted Adaptive Lasso (MM-RWAL). A performance comparison to existing robust Lasso estimators is provided using Monte Carlo experiments. Further, the MM-RWAL is applied to determine the temporal releases of the European Tracer Experiment (ETEX) at the source location. This sparse and ill-conditioned linear inverse problem contains cellwise and rowwise outliers.

Automatic variable selection in a linear model on massive data

For a linear model on massive data, we propose an aggregated estimator depending on adaptive LASSO estimators. The proposed method allows the reduction of the data storage volume and the introduction of an aggregates estimator which automatically selects, with a probability converging to one, the significant explanatory variables. Moreover, the aggregated estimator, corresponding to the non null true parameters has the same asymptotic Normal law as the adaptive LASSO estimator on the all data. But, the estimator calculated on all data is practically impossible to calculate, for lack of calculation memory or storage, when the model is on massive data. Then, another interest of our method is that it can work around the data processing problem of insufficient memory allocated by statistical software when the observation number is very large. The empirical performance is investigated by a comparative simulation study. A real data example is used to illustrate the usefulness of our method.

Multiple Structural Breaks in Cointegrating Regressions: A Model Selection Approach

In this paper, we propose a new approach to model structural change in cointegrating regressions using penalized regression techniques. First, we consider a setting with known breakpoint candidates and show that a modified adaptive lasso estimator can consistently estimate structural breaks in the intercept and slope coefficient of a cointegrating regression. Second, we extend our approach to a diverging number of breakpoint candidates and provide simulation evidence that timing and magnitude of structural breaks are consistently estimated. Third, we use the adaptive lasso estimation to design new tests for cointegration in the presence of multiple structural breaks, derive the asymptotic distribution of our test statistics and show that the proposed tests have power against the null of no cointegration. Finally, we use our new methodology to study the effects of structural breaks on the long-run PPP relationship.