Penalized Least-squares Estimator Research Articles

Corrected generalized cross-validation for finite ensembles of penalized estimators

Abstract Generalized cross-validation (GCV) is a widely used method for estimating the squared out-of-sample prediction risk that employs scalar degrees of freedom adjustment (in a multiplicative sense) to the squared training error. In this paper, we examine the consistency of GCV for estimating the prediction risk of arbitrary ensembles of penalized least-squares estimators. We show that GCV is inconsistent for any finite ensemble of size greater than one. Towards repairing this shortcoming, we identify a correction that involves an additional scalar correction (in an additive sense) based on degrees of freedom adjusted training errors from each ensemble component. The proposed estimator (termed CGCV) maintains the computational advantages of GCV and requires neither sample splitting, model refitting, or out-of-bag risk estimation. The estimator stems from a finer inspection of the ensemble risk decomposition and two intermediate risk estimators for the components in this decomposition. We provide a non-asymptotic analysis of the CGCV and the two intermediate risk estimators for ensembles of convex penalized estimators under Gaussian features and a linear response model. Furthermore, in the special case of ridge regression, we extend the analysis to general feature and response distributions using random matrix theory, which establishes model-free uniform consistency of CGCV.

Identify latent group structures in panel data: The classifylasso command

In this article, we introduce a new command, classifylasso, that implements the classifier-lasso method (Su, Shi, and Phillips, 2016, Econometrica 84: 2215–2264) to simultaneously identify and estimate unobserved parameter heterogeneity in panel-data models using penalized techniques. We document the functionality of this command, including 1) penalized least-squares estimation of group-specific coefficients and classification of unknown group membership under a certain number of groups; 2) two lasso-type estimators with robust standard errors, namely, classifier-lasso and postlasso; and 3) determination of the number of groups based on an information criterion. We further develop some postestimation commands to display and visualize the estimation results.

A Regularized High-Dimensional Positive Definite Covariance Estimator with High-Frequency Data

This paper proposes a novel large-dimensional positive definite covariance estimator for high-frequency data under a general factor model framework. We demonstrate an appealing connection between the proposed estimator and a weighted group least absolute shrinkage and selection operator (LASSO) penalized least-squares estimator. The proposed estimator improves on traditional principal component analysis by allowing for weak factors, whose signal strengths are weak relative to idiosyncratic components. Despite the presence of microstructure noises and asynchronous trading, the proposed estimator achieves guarded positive definiteness without sacrificing the convergence rate. To make our method fully operational, we provide an extended simultaneous alternating direction method of multipliers algorithm to solve the resultant constrained convex minimization problem efficiently. Empirically, we study the monthly high-frequency covariance structure of the stock constituents of the S&P 500 index from 2008 to 2016, using all traded stocks from the NYSE, AMEX, and NASDAQ stock markets to construct the high-frequency Fama-French four and extended eleven economic factors. We further examine the out-of-sample performance of the proposed method through vast portfolio allocations, which deliver significantly reduced out-of-sample portfolio risk and enhanced Sharpe ratios. The success of our method supports the usefulness of machine learning techniques in finance. This paper was accepted by Agostino Capponi, finance. Funding: This work was supported by the Research Grants Council, University Grants Committee [Grants 11500119, 11505522, 11505721, and 21504818] and the National Natural Science Foundation of China (NSFC) Basic Scientific Center Project [Grant 71988101], entitled as “Econometric Modelling and Economic Policy Studies”, as well as NSFC [Grants 71803166 and 72173104]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2022.04138 .

Tuning parameter selection for penalized estimation via R2

The tuning parameter selection strategy for penalized estimation is crucial to identify a model that is both interpretable and predictive. However, popular strategies (e.g., minimizing average squared prediction error via cross-validation) tend to select models with more predictors than necessary. A simple yet powerful cross validation strategy is proposed which is based on maximizing the squared correlation between the observed and predicted values, rather than minimizing squared error loss for the purposes of support recovery. The strategy can be applied to all penalized least-squares estimators and, under certain conditions, the metric implicitly performs a bias adjustment named the α-modification. When applied to the Lasso estimator, the α-modification is closely related to the relaxed Lasso estimator. The approach is demonstrated on a functional variable selection problem to identify optimal placement of surface electromyogram sensors to control a robotic hand prosthesis.

Efficient estimation in single index models through smoothing splines

We consider estimation and inference in a single index regression model with an unknown but smooth link function. In contrast to the standard approach of using kernels or regression splines, we use smoothing splines to estimate the smooth link function. We develop a method to compute the penalized least squares estimators (PLSEs) of the parametric and the nonparametric components given independent and identically distributed (i.i.d.) data. We prove the consistency and find the rates of convergence of the estimators. We establish asymptotic normality under mild assumption and prove asymptotic efficiency of the parametric component under homoscedastic errors. A finite sample simulation corroborates our asymptotic theory. We also analyze a car mileage data set and a Ozone concentration data set. The identifiability and existence of the PLSEs are also investigated.

Sorted concave penalized regression

The Lasso is biased. Concave penalized least squares estimation (PLSE) takes advantage of signal strength to reduce this bias, leading to sharper error bounds in prediction, coefficient estimation and variable selection. For prediction and estimation, the bias of the Lasso can be also reduced by taking a smaller penalty level than what selection consistency requires, but such smaller penalty level depends on the sparsity of the true coefficient vector. The sorted $\ell_{1}$ penalized estimation (Slope) was proposed for adaptation to such smaller penalty levels. However, the advantages of concave PLSE and Slope do not subsume each other. We propose sorted concave penalized estimation to combine the advantages of concave and sorted penalizations. We prove that sorted concave penalties adaptively choose the smaller penalty level and at the same time benefits from signal strength, especially when a significant proportion of signals are stronger than the corresponding adaptively selected penalty levels. A local convex approximation for sorted concave penalties, which extends the local linear and quadratic approximations for separable concave penalties, is developed to facilitate the computation of sorted concave PLSE and proven to possess desired prediction and estimation error bounds. Our analysis of prediction and estimation errors requires the restricted eigenvalue condition on the design, not beyond, and provides selection consistency under a required minimum signal strength condition in addition. Thus, our results also sharpens existing results on concave PLSE by removing the upper sparse eigenvalue component of the sparse Riesz condition.

Tensor-on-Tensor Regression

ABSTRACTI propose a framework for the linear prediction of a multiway array (i.e., a tensor) from another multiway array of arbitrary dimension, using the contracted tensor product. This framework generalizes several existing approaches, including methods to predict a scalar outcome from a tensor, a matrix from a matrix, or a tensor from a scalar. I describe an approach that exploits the multiway structure of both the predictors and the outcomes by restricting the coefficients to have reduced PARAFAC/CANDECOMP rank. I propose a general and efficient algorithm for penalized least-squares estimation, which allows for a ridge (L2) penalty on the coefficients. The objective is shown to give the mode of a Bayesian posterior, which motivates a Gibbs sampling algorithm for inference. I illustrate the approach with an application to facial image data. An R package is available at https://github.com/lockEF/MultiwayRegression.

Sparse transition matrix estimation for high-dimensional and locally stationary vector autoregressive models

We consider the estimation of the transition matrix in the high-dimensional time-varying vector autoregression (TV-VAR) models. Our model builds on a general class of locally stationary VAR processes that evolve smoothly in time. We propose a hybridized kernel smoothing and $\ell^1$-regularized method to directly estimate the sequence of time-varying transition matrices. Under the sparsity assumption on the transition matrix, we establish the rate of convergence of the proposed estimator and show that the convergence rate depends on the smoothness of the locally stationary VAR processes only through the smoothness of the transition matrix function. In addition, for our estimator followed by thresholding, we prove that the false positive rate (type I error) and false negative rate (type II error) in the pattern recovery can asymptotically vanish in the presence of weak signals without assuming the minimum nonzero signal strength condition. Favorable finite sample performances over the $\ell^2$-penalized least-squares estimator and the unstructured maximum likelihood estimator are shown on simulated data. We also provide two real examples on estimating the dependence structures on financial stock prices and economic exchange rates datasets.

Local influence in ridge semiparametric models

ABSTRACTRidge penalized least-squares estimators has been suggested as an alternative to the minimum penalized sum of squares estimates in the presence of collinearity among the explanatory variables in semiparametric regression models (SPRMs). This paper studies the local influence of minor perturbations on the ridge estimates in the SPRM. The diagnostics under the perturbation of ridge penalized sum of squares, response variable, explanatory variables and ridge parameter are considered. Some local influence diagnostics are given. A Monte Carlo simulation study and a real example are used to illustrate the proposed perturbations.

Shrinkage estimation of common breaks in panel data models via adaptive group fused Lasso

In this paper we consider estimation and inference of common breaks in panel data models via adaptive group fused Lasso. We consider two approaches—penalized least squares (PLS) for first-differenced models without endogenous regressors, and penalized GMM (PGMM) for first-differenced models with endogeneity. We show that with probability tending to one, both methods can correctly determine the unknown number of breaks and estimate the common break dates consistently. We establish the asymptotic distributions of the Lasso estimators of the regression coefficients and their post Lasso versions. We also propose and validate a data-driven method to determine the tuning parameter used in the Lasso procedure. Monte Carlo simulations demonstrate that both the PLS and PGMM estimation methods work well in finite samples. We apply our PGMM method to study the effect of foreign direct investment (FDI) on economic growth using a panel of 88 countries and regions from 1973 to 2012 and find multiple breaks in the model.

Shrinkage Estimation of Common Breaks in Panel Data Models via Adaptive Group Fused Lasso

In this paper we consider estimation and inference of common breaks in panel data models via adaptive group fused lasso. We consider two approaches -- penalized least squares (PLS) for first-differenced models without endogenous regressors, and penalized GMM (PGMM) for first-differenced models with endogeneity. We show that with probability tending to one both methods can correctly determine the unknown number of breaks and estimate the common break dates consistently. We obtain estimates of the regression coefficients via post Lasso and establish their asymptotic distributions. We also propose and validate a data-driven method to determine the tuning parameter used in the Lasso procedure. Monte Carlo simulations demonstrate that both the PLS and PGMM estimation methods work well in finite samples. We apply our PGMM method to study the effect of foreign direct investment (FDI) on economic growth using a panel of 88 countries and regions from 1973 to 2012 and find multiple breaks in the model.

Identifying Latent Structures in Panel Data

This paper provides a novel mechanism for identifying and estimating latent group structures in panel data using penalized regression techniques. We focus on linear models where the slope parameters are heterogeneous across groups but homogenous within a group and the group membership is unknown. Two approaches are considered — penalized least squares (PLS) for models without endogenous regressors, and penalized GMM (PGMM) for models with endogeneity. In both cases we develop a new variant of Lasso called classifier-Lasso (C-Lasso) that serves to shrink individual coefficients to the unknown group-specific coefficients. C-Lasso achieves simultaneous classification and consistent estimation in a single step and the classification exhibits the desirable property of uniform consistency. For PLS estimation C-Lasso also achieves the oracle property so that group-specific parameter estimators are asymptotically equivalent to infeasible estimators that use individual group identity information. For PGMM estimation the oracle property of C-Lasso is preserved in some special cases. Simulations demonstrate good finite-sample performance of the approach both in classification and estimation. An empirical application investigating the determinants of cross-country savings rates finds two latent groups among 56 countries, providing empirical confirmation that higher savings rates go in hand with higher income growth.

Penalized least-squares estimation for regression coefficients in high-dimensional partially linear models

We consider a partially linear model with diverging number of groups of parameters in the parametric component. The variable selection and estimation of regression coefficients are achieved simultaneously by using the suitable penalty function for covariates in the parametric component. An MM-type algorithm for estimating parameters without inverting a high-dimensional matrix is proposed. The consistency and sparsity of penalized least-squares estimators of regression coefficients are discussed under the setting of some nonzero regression coefficients with very small values. It is found that the root p n / n-consistency and sparsity of the penalized least-squares estimators of regression coefficients cannot be given consideration simultaneously when the number of nonzero regression coefficients with very small values is unknown, where p n and n, respectively, denote the number of regression coefficients and sample size. The finite sample behaviors of penalized least-squares estimators of regression coefficients and the performance of the proposed algorithm are studied by simulation studies and a real data example.

Complexity of penalized likelihood estimation

We show that for a class of penalty functions, finding the global optimizer in the penalized least-squares estimation is equivalent to the ‘exact cover by 3-sets’ problem, which belongs to a class of NP-hard problems. The NP-hardness result is then extended to the cases of penalized least absolute deviations regression and a special class of penalized support vector machines. We discuss its implication in statistics. To the best of our knowledge, this is the first formal documentation on the complexity of this type of problem.

Optimal model selection in density estimation

Une procédure de pénalisation en sélection de modèle repose sur la construction d’une forme pour la pénalité ainsi que sur le choix d’une constante de calibration. Dans cet article, nous étudions, pour le problème d’estimation de la densité, les pénalités obtenues par rééchantillonnage de pénalités idéales. Nous montrons l’efficacité de ces procédures pour l’estimation de la forme des pénalités en prouvant, pour les estimateurs sélectionnés, des inégalités d’oracle fines sans termes résiduelles; les résultats sont valides sous des hypothèses faibles à la fois sur la densité inconnue $s$ et sur les collections de modèles. Ces pénalités sont de plus faciles à calibrer puisque la constante asymptotiquement optimale peut être calculée en fonction des poids de rééchantillonnage. En pratique, le nombre de données est toujours fini, nous étudions donc également l’heuristique de pente et justifions l’algorithme de pente qui permet de calibrer la constante de calibration à partir des données.

Using [formula omitted]-splines to test the linearity of partially linear models

The nonparametric component in a partially linear model is estimated by a linear combination of fixed-knot cubic B -splines with a second-order difference penalty on the adjacent B -spline coefficients. The resulting penalized least-squares estimator is used to construct two Wald-type spline-based test statistics for the null hypothesis of the linearity of the nonparametric function. When the number of knots is fixed, the first test statistic asymptotically has the distribution of a linear combination of independent chi-squared random variables, each with one degree of freedom, under the null hypothesis. The smoothing parameter is determined by specifying a value for the asymptotically expected value of the test statistic under the null hypothesis. When the number of knots is fixed and under the null hypothesis, the second test statistic asymptotically has a chi-squared distribution with K = q + 2 degrees of freedom, where q is the number of knots used for estimation. The power performances of the two proposed tests are investigated via simulation experiments, and the practicality of the proposed methodology is illustrated using a real-life data set.

Estimating the joint distribution of independent categorical variables via model selection

Assume one observes independent categorical variables or, equivalently, one observes the corresponding multinomial variables. Estimating the distribution of the observed sequence amounts to estimating the expectation of the multinomial Sequence. A new estimator for this mean is proposed that is nonparametric, non-asymptotic and implementable even for large sequences. It is a penalized least-squares estimator based oil wavelets, with a penalization term inspired by papers of Birge and Massart. The estimator is proved to satisfy ail oracle inequality and to be adaptive in the minimax sense over a class of Besov bodies. The method is embedded in a general framework which allows us to recover also an existing method for segmentation. Beyond theoretical results, a simulation Study is reported and an application on real data is provided.

Generalized SURE for Exponential Families: Applications to Regularization

Stein's unbiased risk estimate (SURE) was proposed by Stein for the independent, identically distributed (i.i.d.) Gaussian model in order to derive estimates that dominate least squares (LS). Recently, the SURE criterion has been employed in a variety of denoising problems for choosing regularization parameters that minimize an estimate of the mean-squared error (MSE). However, its use has been limited to the i.i.d. case which precludes many important applications. In this paper we begin by deriving a SURE counterpart for general, not necessarily i.i.d. distributions from the exponential family. This enables extending the SURE design technique to a much broader class of problems. Based on this generalization we suggest a new method for choosing regularization parameters in penalized LS estimators. We then demonstrate its superior performance over the conventional generalized cross validation and discrepancy approaches in the context of image deblurring and deconvolution. The SURE technique can also be used to design estimates without predefining their structure. However, allowing for too many free parameters impairs the estimate's performance. To address this inherent tradeoff, we propose a regularized SURE objective, and demonstrate its use in the context of wavelet denoising.

Multiple Penalty Regression: Fitting and Extrapolating a Discrete Incomplete Multi-way Layout

The d iscrete multi-way layout is an abstract data type associated with regression, experimental designs, digital images or videos, spatial statistics, gene or protein chips, and more. The factors influencing response can be nominal or ordinal. The observed factor level combinations are finitely discrete and often incomplete or irregularly spaced. This paper develops low risk biased estimators of the means at the observed factor level combinations; and extrapolates the estimated means to larger discrete complete layouts. Candidate penalized least squares (PLS) estimators with multiple quadratic penalties express competing conjectures about each of the main effects and interactions in the analysis of variance decomposition of the means. The candidate PLS estimator with smallest estimated quadratic risk attains, asymptotically, the smallest risk over all candidate PLS estimators. In the theoretical analysis, the dimension of the regression space tends to infinity. No assumptions are made about the unknown means or about replication.

An exponential inequality under weak dependence

Doukhan and Louhichi introduced a covariance-based concept of weak dependence which is more general than classical mixing concepts. We prove a Bernstein-type inequality under this condition which is similar to the well-known inequality in the independent case. We apply this tool to derive asymptotic properties of penalized least-squares estimators in Barron's classes.