Lasso Estimator Research Articles

Forecasting the Volatility of Crude Oil: The Role of Uncertainty and Spillovers

We use a dataset for the group of G7 countries and China to study the out-of-sample predictive value of uncertainty and its international spillovers for the realized variance of crude oil (West Texas Intermediate and Brent) over the sample period from 1996Q1 to 2020Q4. Using the Lasso estimator, we found evidence that uncertainty and international spillovers had predictive value for the realized variance at intermediate (two quarters) and long (one year) forecasting horizons in several of the forecasting models that we studied. This result holds also for upside (good) and downside (bad) variance, and irrespective of whether we used a recursive or a rolling estimation window. Our results have important implications for investors and policymakers.

Lasso estimation of hierarchical interactions for analyzing heterogeneity of treatment effect.

Individuals differ in how they respond to a given treatment. In an effort to predict the treatment response and analyze the heterogeneity of treatment effect, we propose a general modeling framework by identifying treatment-covariate interactions honoring a hierarchical condition. We construct a single-step norm penalty procedure that maintains the hierarchical structure of interactions in the sense that a treatment-covariate interaction term is included in the model only when either the covariate or both the covariate and treatment have nonzero main effects. We developed a constrained Lasso approach with two parameterization schemes that enforce the hierarchical interaction restriction differently. We solved the resulting constrained optimization problem using a spectral projected gradient method. We compared our methods to the unstructured Lasso using simulation studies including a scenario that violates the hierarchical condition (misspecified model). The simulations showed that our methods yielded more parsimonious models and outperformed the unstructured Lasso for correctly identifying nonzero treatment-covariate interactions. The superior performance of our methods are also corroborated by an application to a large randomized clinical trial data investigating a drug for treating congestive heart failure (N=2569). Our methods provide a well-suited approach for doing secondary analysis in clinical trials to analyze heterogeneous treatment effects and to identify predictive biomarkers.

Uncertainty, Spillovers, and Forecasts of the Realized Variance of Gold Returns

Using data for the group of G7 countries and China for the sample period 1996Q1 to 2020Q4, we study the role of uncertainty and spillovers for the out-of-sample forecasting of the realized variance of gold returns and its upside (good) and downside (bad) counterparts. We go beyond earlier research in that we do not focus exclusively on U.S.-based measures of uncertainty, and in that we account for international spillovers of uncertainty. Our results, based on the Lasso estimator, show that, across the various model configurations that we study, uncertainty has a more systematic effect on out-of-sample forecast accuracy than spillovers. Our results have important implications for investors in terms of, for example, pricing of related derivative securities and the development of portfolio-allocation strategies.

Improving Lasso for model selection and prediction

AbstractIt is known that the Thresholded Lasso (TL), SCAD or MCP correct intrinsic estimation bias of the Lasso. In this paper we propose an alternative method of improving the Lasso for predictive models with general convex loss functions which encompass normal linear models, logistic regression, quantile regression, or support vector machines. For a given penalty we order the absolute values of the Lasso nonzero coefficients and then select the final model from a small nested family by the Generalized Information Criterion. We derive exponential upper bounds on the selection error of the method. These results confirm that, at least for normal linear models, our algorithm seems to be the benchmark for the theory of model selection as it is constructive, computationally efficient and leads to consistent model selection under weak assumptions. Constructivity of the algorithm means that, in contrast to the TL, SCAD or MCP, consistent selection does not rely on the unknown parameters as the cone invertibility factor. Instead, our algorithm only needs the sample size, the number of predictors and an upper bound on the noise parameter. We show in numerical experiments on synthetic and real‐world datasets that an implementation of our algorithm is more accurate than implementations of studied concave regularizations. Our procedure is included in the R package DMRnet and available in the CRAN repository.

Ridge parameters optimization based on minimizing model selection criterion in multivariate generalized ridge regression

A multivariate generalized ridge (MGR) regression provides a shrinkage estimator of the multivariate linear regression by multiple ridge parameters. Since the ridge parameters which adjust the amount of shrinkage of the estimator are unknown, their optimization is an important task to obtain a better estimator. For the univariate case, a fast algorithm has been proposed for optimizing ridge parameters based on minimizing a model selection criterion (MSC) and the algorithm can be applied to various MSCs. In this paper, we extend this algorithm to MGR regression. We also describe the relationship between the MGR estimator which is not sparse and a multivariate adaptive group Lasso estimator which is sparse, under orthogonal explanatory variables.

On cross-validated Lasso in high dimensions

In this paper, we derive nonasymptotic error bounds for the Lasso estimator when the penalty parameter for the estimator is chosen using K-fold cross-validation. Our bounds imply that the cross-validated Lasso estimator has nearly optimal rates of convergence in the prediction, L2, and L1 norms. For example, we show that in the model with the Gaussian noise and under fairly general assumptions on the candidate set of values of the penalty parameter, the estimation error of the cross-validated Lasso estimator converges to zero in the prediction norm with the slogp/n×log(pn) rate, where n is the sample size of available data, p is the number of covariates and s is the number of nonzero coefficients in the model. Thus, the cross-validated Lasso estimator achieves the fastest possible rate of convergence in the prediction norm up to a small logarithmic factor log(pn), and similar conclusions apply for the convergence rate both in L2 and in L1 norms. Importantly, our results cover the case when p is (potentially much) larger than n and also allow for the case of non-Gaussian noise. Our paper therefore serves as a justification for the widely spread practice of using cross-validation as a method to choose the penalty parameter for the Lasso estimator.

Sparse Laplacian Shrinkage with the Graphical Lasso Estimator for Regression Problems

This paper considers a high-dimensional linear regression problem where there are complex correlation structures among predictors. We propose a graph-constrained regularization procedure, named Sparse Laplacian Shrinkage with the Graphical Lasso Estimator (SLS-GLE). The procedure uses the estimated precision matrix to describe the specific information on the conditional dependence pattern among predictors, and encourages both sparsity on the regression model and the graphical model. We introduce the Laplacian quadratic penalty adopting the graph information, and give detailed discussions on the advantages of using the precision matrix to construct the Laplacian matrix. Theoretical properties and numerical comparisons are presented to show that the proposed method improves both model interpretability and accuracy of estimation. We also apply this method to a financial problem and prove that the proposed procedure is successful in assets selection.

Individual Data Protected Integrative Regression Analysis of High-Dimensional Heterogeneous Data

Evidence-based decision making often relies on meta-analyzing multiple studies, which enables more precise estimation and investigation of generalizability. Integrative analysis of multiple heterogeneous studies is, however, highly challenging in the ultra high-dimensional setting. The challenge is even more pronounced when the individual-level data cannot be shared across studies, known as DataSHIELD contraint. Under sparse regression models that are assumed to be similar yet not identical across studies, we propose in this paper a novel integrative estimation procedure for data-Shielding High-dimensional Integrative Regression (SHIR). SHIR protects individual data through summary-statistics-based integrating procedure, accommodates between-study heterogeneity in both the covariate distribution and model parameters, and attains consistent variable selection. Theoretically, SHIR is statistically more efficient than the existing distributed approaches that integrate debiased LASSO estimators from the local sites. Furthermore, the estimation error incurred by aggregating derived data is negligible compared to the statistical minimax rate and SHIR is shown to be asymptotically equivalent in estimation to the ideal estimator obtained by sharing all data. The finite-sample performance of our method is studied and compared with existing approaches via extensive simulation settings. We further illustrate the utility of SHIR to derive phenotyping algorithms for coronary artery disease using electronic health records data from multiple chronic disease cohorts.

High-Dimensional Interaction Detection With False Sign Rate Control

Identifying interaction effects is fundamentally important in many scientific discoveries and contemporary applications, but it is challenging since the number of pairwise interactions increases quadratically with the number of covariates and that of higher-order interactions grows even faster. Although there is a growing literature on interaction detection, little work has been done on the prediction and false sign rate on interaction detection in ultrahigh-dimensional regression models. This article fills such a gap. More specifically, in this article we establish some theoretical results on interaction selection for ultrahigh-dimensional quadratic regression models under random designs. We prove that the examined method enjoys the same oracle inequalities as the lasso estimator and further admits an explicit bound on the false sign rate. Moreover, the false sign rate can be asymptotically vanishing. These new theoretical characterizations are confirmed by simulation studies. The performance of our proposed approach is further illustrated through a real data application.

Oracle Efficient Estimation of Structural Breaks in Cointegrating Regressions

In this article, we propose an adaptive group lasso procedure to efficiently estimate structural breaks in cointegrating regressions. It is well known that the group lasso estimator is not simultaneously estimation consistent and model selection consistent in structural break settings. Hence, we use a first step group lasso estimation of a diverging number of breakpoint candidates to produce weights for a second adaptive group lasso estimation. We prove that parameter changes are estimated consistently by group lasso and show that the number of estimated breaks is greater than the true number but still sufficiently close to it. Then, we use these results and prove that the adaptive group lasso has oracle properties if weights are obtained from our first step estimation. Simulation results show that the proposed estimator delivers the expected results. An economic application to the long‐run US money demand function demonstrates the practical importance of this methodology.

The Higher-Order of Adaptive Lasso and Elastic Net Methods for Classification on High Dimensional Data

The lasso and elastic net methods are the popular technique for parameter estimation and variable selection. Moreover, the adaptive lasso and elastic net methods use the adaptive weights on the penalty function based on the lasso and elastic net estimates. The adaptive weight is related to the power order of the estimator. Normally, these methods focus to estimate parameters in terms of linear regression models that are based on the dependent variable and independent variable as a continuous scale. In this paper, we compare the lasso and elastic net methods and the higher-order of the adaptive lasso and adaptive elastic net methods for classification on high dimensional data. The classification is used to classify the categorical data for dependent variable dependent on the independent variables, which is called the logistic regression model. The categorical data are considered a binary variable, and the independent variables are used as the continuous variable. The high dimensional data are represented when the number of independent variables is higher than the sample sizes. For this research, the simulation of the logistic regression is considered as the binary dependent variable and 20, 30, 40, and 50 as the independent variables when the sample sizes are less than the number of the independent variables. The independent variables are generated from normal distribution on several variances, and the dependent variables are obtained from the probability of logit function and transforming it to predict the binary data. For application in real data, we express the classification of the type of leukemia as the dependent variables and the subset of gene expression as the independent variables. The criterion of these methods is to compare by the average percentage of predicted accuracy value. The results are found that the higher-order of adaptive lasso method is satisfied with large dispersion, but the higher-order of adaptive elastic net method outperforms on small dispersion.

Mixed Lasso estimator for stochastic restricted regression models

Parameters of a linear regression model can be estimated with the help of traditional methods like generalized least squares and mixed estimator. However, recent developments increased the importance of big data sets, which have much more predictors than observations where some predictors have no impact on the dependent variable. The estimation and model selection problem of big datasets can be solved using the least absolute shrinkage and selection operator (Lasso). However, to the authors’ knowledge, there is no study that incorporates stochastic restrictions, within a Lasso framework. In this paper, we propose a Mixed Lasso (M-Lasso) estimator that incorporates stochastic linear restrictions to big data sets for selecting the true model and estimating parameters simultaneously. We conduct a simulation study to compare the performance of M-Lasso with existing estimators based on mean squared error and model selection performance. Results show that M-Lasso is superior in terms of and it generally dominates compared estimators according to the model selection criteria. We employ M-Lasso to estimate parameters of a widely analysed production function under stochastic restrictions raised from economic theory. Our results show that M-Lasso can provide reasonable and more precise estimates of model parameters that are in line with the economic theory.

Development of a prognostic model for hepatocellular carcinoma using genes involved in aerobic respiration.

Hepatocellular carcinoma (HCC) is the second leading cause of cancer-related death worldwide. Currently, recent risk stratification has only focused on liver function and tumor characteristics. Thus, the purpose of this study was to develop a prognostic model based on genes involved in aerobic respiration. Matched tumor and normal tissues from TCGA and ICGC cohorts were analyzed to identify 15 overlapping differential expressed genes. Cox univariate analysis of the 15 genes in the TCGA cohort revealed they were all associated with disease-specific survival (DSS) in HCC patients. Using LASSO estimation and the optimal value for penalization coefficient lambda 12 genes were selected for the prognostic model, and then HCC patients in the TCGA cohort were dichotomized into low-risk and high-risk groups. Univariate and multivariate Cox analysis demonstrated patients in low-risk group had better survival. Validation of the risk score model with the ICGC cohort produces results consistent with those of the TCGA cohort. In conclusion, this study developed and validated a prognostic model of HCC through a comprehensive analysis of genes involved in aerobic respiration. This model may help develop personalized treatments for patients with HCC.

Machine Learning Time Series Regressions With an Application to Nowcasting

This article introduces structured machine learning regressions for high-dimensional time series data potentially sampled at different frequencies. The sparse-group LASSO estimator can take advantage of such time series data structures and outperforms the unstructured LASSO. We establish oracle inequalities for the sparse-group LASSO estimator within a framework that allows for the mixing processes and recognizes that the financial and the macroeconomic data may have heavier than exponential tails. An empirical application to nowcasting US GDP growth indicates that the estimator performs favorably compared to other alternatives and that text data can be a useful addition to more traditional numerical data. Our methodology is implemented in the R package midasml, available from CRAN.

Penalized regression via the restricted bridge estimator

This article is concerned with the Bridge Regression, which is a special family in penalized regression with penalty function $\sum_{j=1}^{p}|\beta_j|^q$ with $q>0$, in a linear model with linear restrictions. The proposed restricted bridge (RBRIDGE) estimator simultaneously estimates parameters and selects important variables when a prior information about parameters are available in either low dimensional or high dimensional case. Using local quadratic approximation, the penalty term can be approximated around a local initial values vector and the RBRIDGE estimator enjoys a closed-form expression which can be solved when $q>0$. Special cases of our proposal are the restricted LASSO ($q=1$), restricted RIDGE ($q=2$), and restricted Elastic Net ($1< q < 2$) estimators. We provide some theoretical properties of the RBRIDGE estimator under for the low dimensional case, whereas the computational aspects are given for both low and high dimensional cases. An extensive Monte Carlo simulation study is conducted based on different prior pieces of information and the performance of the RBRIDGE estiamtor is compared with some competitive penalty estimators as well as the ORACLE. We also consider four real data examples analysis for comparison sake. The numerical results show that the suggested RBRIDGE estimator outperforms outstandingly when the prior is true or near exact

Multiple structural breaks in cointegrating regressions: a model selection approach

Abstract In this paper, we propose a new approach to model structural change in cointegrating regressions using penalized regression techniques. First, we consider a setting with known breakpoint candidates and show that a modified adaptive lasso estimator can consistently estimate structural breaks in the intercept and slope coefficient of a cointegrating regression. Second, we extend our approach to a diverging number of breakpoint candidates and provide simulation evidence that timing and magnitude of structural breaks are consistently estimated. Third, we use the adaptive lasso estimation to design new tests for cointegration in the presence of multiple structural breaks, derive the asymptotic distribution of our test statistics and show that the proposed tests have power against the null of no cointegration. Finally, we use our new methodology to study the effects of structural breaks on the long-run PPP relationship.

UNIFORM ASYMPTOTICS AND CONFIDENCE REGIONS BASED ON THE ADAPTIVE LASSO WITH PARTIALLY CONSISTENT TUNING

We consider the adaptive Lasso estimator with componentwise tuning in the framework of a low-dimensional linear regression model. In our setting, at least one of the components is penalized at the rate of consistent model selection and certain components may not be penalized at all. We perform a detailed study of the consistency properties and the asymptotic distribution which includes the effects of componentwise tuning within a so-called moving-parameter framework. These results enable us to explicitly provide a set $\mathcal {M}$ such that every open superset acts as a confidence set with uniform asymptotic coverage equal to 1, whereas removing an arbitrarily small open set along the boundary yields a confidence set with uniform asymptotic coverage equal to 0. The shape of the set $\mathcal {M}$ depends on the regressor matrix as well as the deviations within the componentwise tuning parameters. Our findings can be viewed as a broad generalization of Pötscher and Schneider (2009, Journal of Statistical Planning and Inference 139, 2775–2790; 2010, Electronic Journal of Statistics 4, 334–360), who considered distributional properties and confidence intervals based on components of the adaptive Lasso estimator for the case of orthogonal regressors.

Inferring latent heterogeneity using many feature variables supervised by survival outcome.

In cancer studies, it is important to understand disease heterogeneity among patients so that precision medicine can particularly target high-risk patients at the right time. Many feature variables such as demographic variables and biomarkers, combined with a patient's survival outcome, can be used to infer such latent heterogeneity. In this work, we propose a mixture model to model each patient's latent survival pattern, where the mixing probabilities for latent groups are modeled through a multinomial distribution. The Bayesian information criterion is used for selecting the number of latent groups. Furthermore, we incorporate variable selection with the adaptive lasso into inference so that only a few feature variables will be selected to characterize the latent heterogeneity. We show that our adaptive lasso estimator has oracle properties when the number of parameters diverges with the sample size. The finite sample performance is evaluated by the simulation study, and the proposed method is illustrated by two datasets.

Necessary and sufficient conditions for variable selection consistency of the LASSO in high dimensions

This paper investigates conditions for variable selection consistency of the LASSO in high dimensional regression models and gives necessary and sufficient conditions for the same, potentially allowing the model dimension p to grow arbitrarily fast as a function of the sample size n. These conditions require both upper and lower bounds on the growth rate of the penalty parameter. It turns out that a variant of the irrepresentable Condition (IRC) of (J. Mach. Learn. Res. 7 (2006) 2541–2563), herein called the lower irrepresentable Condition (or LIRC), is determined by the lower bound considerations while the upper bound considerations lead to a new condition, called the upper irrepresentable Condition (or UIRC) in this paper. It is shown that the LIRC together with the UIRC is necessary and sufficient for the variable selection consistency of the LASSO, thereby settling a conjecture of (J. Mach. Learn. Res. 7 (2006) 2541–2563). Further, it is shown that under some mild regularity conditions, the penalty parameter must necessarily tend to infinity at a certain minimal rate to ensure variable selection consistency of the LASSO and that the corresponding LASSO estimators of the nonzero regression parameters cannot be n-consistent (even for individual parameters). Thus, under fairly general conditions, the LASSO with a single choice of the penalty parameter cannot achieve both variable selection consistency and n-consistency simultaneously.

Integrating across neuroimaging modalities boosts prediction accuracy of cognitive ability.

Variation in cognitive ability arises from subtle differences in underlying neural architecture. Understanding and predicting individual variability in cognition from the differences in brain networks requires harnessing the unique variance captured by different neuroimaging modalities. Here we adopted a multi-level machine learning approach that combines diffusion, functional, and structural MRI data from the Human Connectome Project (N = 1050) to provide unitary prediction models of various cognitive abilities: global cognitive function, fluid intelligence, crystallized intelligence, impulsivity, spatial orientation, verbal episodic memory and sustained attention. Out-of-sample predictions of each cognitive score were first generated using a sparsity-constrained principal component regression on individual neuroimaging modalities. These individual predictions were then aggregated and submitted to a LASSO estimator that removed redundant variability across channels. This stacked prediction led to a significant improvement in accuracy, relative to the best single modality predictions (approximately 1% to more than 3% boost in variance explained), across a majority of the cognitive abilities tested. Further analysis found that diffusion and brain surface properties contribute the most to the predictive power. Our findings establish a lower bound to predict individual differences in cognition using multiple neuroimaging measures of brain architecture, both structural and functional, quantify the relative predictive power of the different imaging modalities, and reveal how each modality provides unique and complementary information about individual differences in cognitive function.