Lasso Estimator Research Articles

Combining data envelopment analysis and stochastic frontiers via a LASSO prior

Technical inefficiencies in stochastic frontier models can be thought of as non-negative parameters. Since, however, their number along with other parameters exceeds the sample size, an adaptive LASSO estimator is a reasonable way to overcome the problem, especially in view of the oracle properties of the estimator under broad conditions. It is shown that the adaptive LASSO estimator can be thought of as the posterior mean of a usual stochastic frontier model with a special prior that benchmarks inefficiencies on known quantities. We take these quantities from DEA scores to obtain technical inefficiencies having oracle properties. The LASSO parameters can be estimated routinely in the Bayesian context without the need for cross-validation. In an application to a data set of large U.S. banks we find that adaptive LASSO outperforms significantly traditional stochastic frontier models.

Oil-price uncertainty and the U.K. unemployment rate: A forecasting experiment with random forests using 150 years of data

We analyze the predictive role of oil-price uncertainty for changes in the UK unemployment rate using more than a century of monthly data covering the period from 1859 (when the drilling of the first oil well started at Titusville, Pennsylvania, United States) to 2020. To this end, we use a machine-learning technique known as random forests. Random forests render it possible to model the potentially nonlinear link between oil-price uncertainty and subsequent changes in the unemployment rate in an entirely data-driven way, where it is possible to control for the impact of several other macroeconomic variables and other macroeconomic and financial uncertainties. We estimate random forests on rolling-estimation windows and find evidence that oil-price uncertainty predicts out-of-sample changes in the unemployment rate, especially at longer (six and twelve months) forecast horizons. Moreover, the relative importance of oil-price uncertainty has undergone substantial swings during the history of the modern petroleum industry. Relative importance was high in the 1970s and the 1980s, and it was higher than the relative importance of changes in the oil price itself for most of the sample period. We also find that oil-price uncertainty has predictive value for changes in the unemployment rate when we use a Lasso estimator, where random forests have a superior forecasting performance relative to the Lasso forecasts.

High-dimensional VARs with common factors

This paper studies high-dimensional vector autoregressions (VARs) augmented with common factors that allow for strong cross-sectional dependence. Models of this type provide a convenient mechanism for accommodating the interconnectedness and temporal co-variability that are often present in large dimensional systems. We propose an ℓ1-nuclear-norm regularized estimator and derive the non-asymptotic upper bounds for the estimation errors as well as large sample asymptotics for the estimates. A singular value thresholding procedure is used to determine the correct number of factors with probability approaching one. Both the LASSO estimator and the conservative LASSO estimator are employed to improve estimation precision. The conservative LASSO estimates of the non-zero coefficients are shown to be asymptotically equivalent to the oracle least squares estimates. Simulations demonstrate that our estimators perform reasonably well in finite samples given the complex high-dimensional nature of the model. In an empirical illustration we apply the methodology to explore dynamic connectedness in the volatilities of financial asset prices and the transmission of ‘investor fear’. The findings reveal that a large proportion of connectedness is due to the common factors. Conditional on the presence of these common factors, the results still document remarkable connectedness due to the interactions between the individual variables, thereby supporting a common factor augmented VAR specification.

On estimation error bounds of the Elastic Net when p ≫ n

We study the estimation property of the Elastic Net estimator in high-dimensional linear regression models where the number of parameters p is comparable to or larger than the sample size n. In such a situation, one often assumes sparsity of the true regression coefficient vector , i.e., assuming that belongs to an -ball with radius , , for some . In this paper, we provide -estimation error bounds for the Elastic Net and naive Elastic Net estimators under a unified framework for high-dimensional analysis of M-estimators proposed by Negahban et al. [A unified framework for high-dimensional analysis of M-estimators with decomposable regularizers. Adv Neural Inf Process Syst. 2009;22:1348–1356]. We show that for both cases of exact sparsity and weak sparsity, under the same conditions on the design matrix, the Elastic Net estimator achieves a slightly better error bound than the Lasso estimator by suitably choosing the tuning parameters.

Self-Report Measures of Procrastination Exhibit Inconsistent Concurrent Validity, Predictive Validity, and Psychometric Properties.

Procrastination is a chronic and widespread problem; however, emerging work raises questions regarding the strength of the relationship between self-reported procrastination and behavioral measures of task engagement. This study assessed the internal reliability, concurrent validity, predictive validity, and psychometric properties of 10 self-report procrastination assessments using responses collected from 242 students. Participants’ scores on each self-report instrument were compared to each other using correlations and cluster analysis. Lasso estimation was used to test the self-report scores’ ability to predict two behavioral measures of delay (days to study completion; pacing style). The self-report instruments exhibited strong internal reliability and moderate levels of concurrent validity. Some self-report measures were predictive of days to study completion. No self-report measures were predictive of deadline action pacing, the pacing style most commonly associated with procrastination. Many of the self-report measures of procrastination exhibited poor fit. These results suggest that researchers should exercise caution in selecting self-report measures and that further study is necessary to determine the factors that drive misalignment between self-reports and behavioral measures of delay.

The variable selection methods and algorithms in the multiple linear model

The Lasso and Ridge estimators are effective variable selection methods for a simple linear model. While the multiple linear model is often used in practice, only few studies have examined its problem of variable selection. To address this gap, we study such problem by taking advantage of the Lasso and Ridge estimate. We propose two variable selection methods in the multiple linear model and introduce the multiple random simulation (MRS) algorithm, whose efficiency is similar to that of the least angle regression (LARS) algorithm in a simple linear model. We verify the performance of these two algorithms by applying them to real diabetes data and find that the LARS algorithm cannot be easily applied in a multiple linear model. We also illustrate the excellent performance of MRS by applying this algorithm on a simulated dataset.

Pathway Lasso: Pathway Estimation and Selection with High-Dimensional Mediators.

In many scientific studies, it becomes increasingly important to delineate the pathways through a large number of mediators, such as genetic and brain mediators. Structural equation modeling (SEM) is a popular technique to estimate the pathway effects, commonly expressed as the product of coefficients. However, it becomes unstable and computationally challenging to fit such models with high-dimensional mediators. This paper proposes a sparse mediation model using a regularized SEM approach, where sparsity means that a small number of mediators have a nonzero mediation effect between a treatment and an outcome. To address the model selection challenge, we innovate by introducing a new penalty called Pathway Lasso. This penalty function is a convex relaxation of the non-convex product function for the mediation effects, and it enables a computationally tractable optimization criterion to estimate and select pathway effects simultaneously. We develop a fast ADMM-type algorithm to compute the model parameters, and we show that the iterative updates can be expressed in closed form. We also prove the asymptotic consistency of our Pathway Lasso estimator for the mediation effect. On both simulated data and an fMRI data set, the proposed approach yields higher pathway selection accuracy and lower estimation bias than competing methods.

Extended Oracle Properties of Adaptive Lasso Estimators

We study the asymptotic properties of adaptive lasso estimators when some components of the parameter of interest β are strictly different than zero, while other components may be zero or may converge to zero with rate n-δ, with δ>0, where n denotes the sample size. To achieve this objective, we analyze the convergence/divergence rates of each term in the first-order conditions of adaptive lasso estimators. First, we derive conditions that allow selecting tuning parameters in order to ensure that adaptive lasso estimates of n-δ-components indeed collapse to zero. Second, in this case, we also derive asymptotic distributions of adaptive lasso estimators for nonzero components. When δ>1/2, we obtain the usual n1/2-asymptotic normal distribution, while when 0δ ≤ 1/2, we show nδ-consistency combined with (biased) n1/2-δ-asymptotic normality for nonzero components. We call these properties, Extended Oracle Properties. These results allow practitioners to exclude in their model the asymptotically negligible variables and make inferences on the asymptotically relevant variables.

Two-Stage Robust and Sparse Distributed Statistical Inference for Large-Scale Data

In this paper, we address the problem of conducting statistical inference in settings involving large-scale data that may be high-dimensional and contaminated by outliers. The high volume and dimensionality of the data require distributed processing and storage solutions. We propose a two-stage distributed and robust statistical inference procedures coping with high-dimensional models by promoting sparsity. In the first stage, known as model selection, relevant predictors are locally selected by applying robust Lasso estimators to the distinct subsets of data. The variable selections from each computation node are then fused by a voting scheme to find the sparse basis for the complete data set. It identifies the relevant variables in a robust manner. In the second stage, the developed statistically robust and computationally efficient bootstrap methods are employed. The actual inference constructs confidence intervals, finds parameter estimates and quantifies standard deviation. Similar to stage 1, the results of local inference are communicated to the fusion center and combined there. By using analytical methods, we establish the favorable statistical properties of the robust and computationally efficient bootstrap methods including consistency for a fixed number of predictors, and robustness. The proposed two-stage robust and distributed inference procedures demonstrate reliable performance and robustness in variable selection, finding confidence intervals and bootstrap approximations of standard deviations even when data is high-dimensional and contaminated by outliers.

An adaptive group LASSO approach for domain selection in functional generalized linear models

This paper focuses on estimation and null region detection of the coefficient function for functional generalized linear models. Traditional estimating approaches cannot serve as null regions detectors. To simultaneously estimate coefficient functions and detect corresponding important subregions in functional generalized linear models, an adaptive group LASSO approach with B-spline smoothing technique is developed. The convergence rate of the resulting estimator is obtained. The consistency of domain selection and limiting distribution of the proposed estimator are then established, which is not straightforward since the groups to be penalized are overlapping. These asymptotic properties are also supported by extensive simulation studies. The resulting estimator performs better than direct adaptive LASSO estimators and some existing functional generalized linear models estimators, which are obtained without considering the domain selection. A real data application reveals the effectiveness of the proposed method.

Online Debiasing for Adaptively Collected High-Dimensional Data With Applications to Time Series Analysis

Adaptive collection of data is commonplace in applications throughout science and engineering. From the point of view of statistical inference, however, adaptive data collection induces memory and correlation in the samples, and poses significant challenge. We consider the high-dimensional linear regression, where the samples are collected adaptively, and the sample size n can be smaller than p, the number of covariates. In this setting, there are two distinct sources of bias: the first due to regularization imposed for consistent estimation, for example, using the LASSO, and the second due to adaptivity in collecting the samples. We propose “online debiasing,” a general procedure for estimators such as the LASSO, which addresses both sources of bias. In two concrete contexts (i) time series analysis and (ii) batched data collection, we demonstrate that online debiasing optimally debiases the LASSO estimate when the underlying parameter θ 0 has sparsity of order . In this regime, the debiased estimator can be used to compute p-values and confidence intervals of optimal size.

Regularized estimation of high‐dimensional vector autoregressions with weakly dependent innovations

There has been considerable advance in understanding the properties of sparse regularization procedures in high‐dimensional models. In time series context, it is mostly restricted to Gaussian autoregressions or mixing sequences. We study oracle properties of LASSO estimation of weakly sparse vector‐autoregressive models with heavy tailed, weakly dependent innovations. In contrast to current literature, our innovation process satisfy an L1 mixingale type condition on the centered conditional covariance matrices. This condition covers L1‐NED sequences and strong (‐) mixing sequences as particular examples.

Identifying and Distinguishing Among Teachers’ Supplementary Curriculum Use Patterns Using the Lasso

Research has shown that officially-adopted textbooks comprise only a small part of teachers’ enacted curriculum. Teachers often supplement their core textbooks with unofficial materials, but empirical study of teacher curriculum supplementation is relatively new and underdeveloped. Grounding our work in the Teacher Curriculum Supplementation Framework, we use data from two state-representative teacher surveys to describe different supplement use patterns and explore their correlates. (We use RAND’s American Teacher Panel survey of K-12 ELA teachers, representative of Louisiana, Massachusetts, and Rhode Island, and Harvard’s National Evaluation of Curriculum Effectiveness survey of fourth and fifth grade math teachers, representative of California, Louisiana, Maryland, New Jersey, New Mexico, and Washington.) We find evidence of four distinct supplement use patterns. We then predict each pattern, producing sparse models using the lasso estimator. We find that teacher-, school-, and textbook-level characteristics are predictive of teachers’ supplement use, suggesting that it may be affected by structures and policies beyond the individual teacher. We recommend researchers use consistent measures to explore the causes and consequences of supplementation.

Improved Lasso (ILASSO) for Gene Selection and Classification in High Dimensional DNA Microarray Data

<p class="0abstract">Classification and selection of gene in high dimensional microarray data has become a challenging problem in molecular biology and genetics. Penalized Adaptive likelihood method has been employed recently for classification of cancer to address both gene selection consistency and estimation of gene coefficients in high dimensional data simultaneously. Many studies from the literature have proposed the use of ordinary least squares (OLS), maximum likelihood estimation (MLE) and Elastic net as the initial weight in the Adaptive elastic net, but in high dimensional microarray data the MLE and OLS are not suitable. Likewise, considering the Elastic net as the initial weight in Adaptive elastic yields a poor performance, because the ridge penalty in the Elastic net grouped coefficient of highly correlated genes closer to each other. As a result, the estimator fails to differentiate coefficients of highly correlated genes that have different sign being grouped together. To tackle this issue, the present study proposed Improved LASSO (ILASSO) estimator which add the ridge penalty to the original LASSO with an Adaptive weight to both and simultaneously. Results from the real data indicated that ILASSO has a better performance compared to other methods in terms of the number of gene selected, classification precision, Sensitivity and Specificity.</p>

Estimation of high-dimensional seemingly unrelated regression models

In this article, we investigate seemingly unrelated regression (SUR) models that allow the number of equations (N) to be large and comparable to the number of the observations in each equation (T). It is well known that conventional SUR estimators, for example, the feasible generalized least squares estimator from Zellner (1962) does not perform well in a high-dimensional setting. We propose a new feasible GLS estimator called the feasible graphical lasso (FGLasso) estimator. For a feasible implementation of the GLS estimator, we use the graphical lasso estimation of the precision matrix (the inverse of the covariance matrix of the equation system errors) assuming that the underlying unknown precision matrix is sparse. We show that under certain conditions, FGLasso converges uniformly to GLS even when T < N, and it shares the same asymptotic distribution with the efficient GLS estimator when We confirm these results through finite sample Monte-Carlo simulations.

The distribution of the Lasso: Uniform control over sparse balls and adaptive parameter tuning

The Lasso is a popular regression method for high-dimensional problems in which the number of parameters θ1,…,θN, is larger than the number n of samples: N>n. A useful heuristics relates the statistical properties of the Lasso estimator to that of a simple soft-thresholding denoiser, in a denoising problem in which the parameters (θi)i≤N are observed in Gaussian noise, with a carefully tuned variance. Earlier work confirmed this picture in the limit n,N→∞, pointwise in the parameters θ and in the value of the regularization parameter. Here, we consider a standard random design model and prove exponential concentration of its empirical distribution around the prediction provided by the Gaussian denoising model. Crucially, our results are uniform with respect to θ belonging to ℓq balls, q∈[0,1], and with respect to the regularization parameter. This allows us to derive sharp results for the performances of various data-driven procedures to tune the regularization. Our proofs make use of Gaussian comparison inequalities, and in particular of a version of Gordon’s minimax theorem developed by Thrampoulidis, Oymak and Hassibi, which controls the optimum value of the Lasso optimization problem. Crucially, we prove a stability property of the minimizer in Wasserstein distance that allows one to characterize properties of the minimizer itself.

Regularised orthogonal machine learning for nonlinear semiparametric models

Summary This paper proposes a Lasso-type estimator for a high-dimensional sparse parameter identified by a single index conditional moment restriction (CMR). In addition to this parameter, the moment function can also depend on a nuisance function, such as the propensity score or the conditional choice probability, which we estimate by modern machine learning tools. We first adjust the moment function so that the gradient of the future loss function is insensitive (formally, Neyman orthogonal) with respect to the first-stage regularisation bias, preserving the single index property. We then take the loss function to be an indefinite integral of the adjusted moment function with respect to the single index. The proposed Lasso estimator converges at the oracle rate, where the oracle knows the nuisance function and solves only the parametric problem. We demonstrate our method by estimating the short-term heterogeneous impact of Connecticut’s Jobs First welfare reform experiment on women’s welfare participation decision.

Reducing Model Misspecification and Bias in the Estimation of Interactions

AbstractAnalyzing variation in treatment effects across subsets of the population is an important way for social scientists to evaluate theoretical arguments. A common strategy in assessing such treatment effect heterogeneity is to include a multiplicative interaction term between the treatment and a hypothesized effect modifier in a regression model. Unfortunately, this approach can result in biased inferences due to unmodeled interactions between the effect modifier and other covariates, and including these interactions can lead to unstable estimates due to overfitting. In this paper, we explore the usefulness of machine learning algorithms for stabilizing these estimates and show how many off-the-shelf adaptive methods lead to two forms of bias: direct and indirect regularization bias. To overcome these issues, we use a post-double selection approach that utilizes several lasso estimators to select the interactions to include in the final model. We extend this approach to estimate uncertainty for both interaction and marginal effects. Simulation evidence shows that this approach has better performance than competing methods, even when the number of covariates is large. We show in two empirical examples that the choice of method leads to dramatically different conclusions about effect heterogeneity.

High‐dimensional robust inference for Cox regression models using desparsified Lasso

AbstractWe consider high‐dimensional inference for potentially misspecified Cox proportional hazard models based on low‐dimensional results by Lin and Wei (1989). A desparsified Lasso estimator is proposed based on the log partial likelihood function and shown to converge to a pseudo‐true parameter vector. Interestingly, the sparsity of the true parameter can be inferred from that of the above limiting parameter. Moreover, each component of the above (nonsparse) estimator is shown to be asymptotically normal with a variance that can be consistently estimated even under model misspecifications. In some cases, this asymptotic distribution leads to valid statistical inference procedures, whose empirical performances are illustrated through numerical examples.

Ultrahigh‐dimensional generalized additive model: Unified theory and methods

AbstractGeneralized additive model is a powerful statistical learning and predictive modeling tool that has been applied in a wide range of applications. The need of high‐dimensional additive modeling is eminent in the context of dealing with high throughput data such as genetics data analysis. In this article, we studied a two‐step selection and estimation method for ultrahigh‐dimensional generalized additive models. The first step applies group lasso on the expanded bases of the functions. With high probability this selects all nonzero functions without having too much over selection. The second step uses adaptive group lasso with any initial estimators, including the group lasso estimator, that satisfies some regular conditions. The adaptive group lasso estimator is shown to be selection consistent with improved convergence rates. Tuning parameter selection is also discussed and shown to select the true model consistently under generalized information criterion procedure. The theoretical properties are supported by extensive numerical study.