Irrelevant Regressors Research Articles

PYE: A Penalized Youden Index Estimator for selecting and combining biomarkers in high-dimensional data

In a classification context, the Youden index, in addition to being a powerful measure of the performance of diagnostic markers and identifying the optimal cut-off point, has also been shown to be an effective measure to be maximized to combine biomarkers. However, in a high-dimensional background, this method may lead to overfitting, mainly due to the noise caused by considering irrelevant regressors. Hence, to reduce this side effect and increase the robustness and interpretability of the solution, the challenge is not only to combine the features but also to select them, obtaining a sparse result. There are several methods to perform feature selection. One of the most performing solution is to adopt regularization techniques.We propose the Penalized Youden Index Estimator (PYE), a method based on the Youden index and various penalization terms, capable of searching for the best selection and combination of biomarkers at the same time, maximizing this index. Depending on the considered regularization term, L12, L1, Elastic Net, SCAD, and MCP, we propose different versions of PYE. Being the objective function possibly non-convex, we also propose a modified accelerated proximal gradient algorithm to optimize PYE. To test the effectiveness of our approach, a wide simulation study on synthetic and real high-dimensional datasets is presented, comparing the performance of our proposal with some of the most popular existing methods.The results show how PYE, in its different versions, returns alternative solutions with respect to the considered existing methods. In some cases more sparse, in others with a preferable balance among accuracy, selection, and which biomarkers have been considered relevant.

Robust kernel regression function with uncertain scale parameter for high dimensional ergodic data using $ k $-nearest neighbor estimation

<abstract><p>In this paper, we consider a new method dealing with the problem of estimating the scoring function $ \gamma_a $, with a constant $ a $, in functional space and an unknown scale parameter under a nonparametric robust regression model. Based on the $ k $ Nearest Neighbors ($ k $NN) method, the primary objective is to prove the asymptotic normality aspect in the case of a stationary ergodic process of this estimator. We begin by establishing the almost certain convergence of a conditional distribution estimator. Then, we derive the almost certain convergence (with rate) of the conditional median (scale parameter estimator) and the asymptotic normality of the robust regression function, even when the scale parameter is unknown. Finally, the simulation and real-world data results reveal the consistency and superiority of our theoretical analysis in which the performance of the $ k $NN estimator is comparable to that of the well-known kernel estimator, and it outperforms a nonparametric series (spline) estimator when there are irrelevant regressors.</p></abstract>

Shrinkage estimation of dynamic panel data models with interactive fixed effects

We consider the problem of determining the number of factors and selecting the proper regressors in linear dynamic panel data models with interactive fixed effects. Based on the preliminary estimates of the slope parameters and factors a la Bai (2009) and Moon and Weidner (2015), we propose a method for simultaneous selection of regressors and factors and estimation through the method of adaptive group Lasso (least absolute shrinkage and selection operator). We show that with probability approaching one, our method can correctly select all relevant regressors and factors and shrink the coefficients of irrelevant regressors and redundant factors to zero. Further, we demonstrate that our shrinkage estimators of the nonzero slope parameters exhibit some oracle property. We conduct Monte Carlo simulations to demonstrate the superb finite-sample performance of the proposed method. We apply our method to study the determinants of economic growth and find that in addition to three common unobserved factors selected by our method, government consumption share has negative effects, whereas investment share and lagged economic growth have positive effects on economic growth.

SLOPE-ADAPTIVE VARIABLE SELECTION VIA CONVEX OPTIMIZATION.

We introduce a new estimator for the vector of coefficients β in the linear model y = Xβ + z, where X has dimensions n × p with p possibly larger than n. SLOPE, short for Sorted L-One Penalized Estimation, is the solution to [Formula: see text]where λ1 ≥ λ2 ≥ … ≥ λ p ≥ 0 and [Formula: see text] are the decreasing absolute values of the entries of b. This is a convex program and we demonstrate a solution algorithm whose computational complexity is roughly comparable to that of classical ℓ1 procedures such as the Lasso. Here, the regularizer is a sorted ℓ1 norm, which penalizes the regression coefficients according to their rank: the higher the rank-that is, stronger the signal-the larger the penalty. This is similar to the Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B57 (1995) 289-300] procedure (BH) which compares more significant p-values with more stringent thresholds. One notable choice of the sequence {λ i } is given by the BH critical values [Formula: see text], where q ∈ (0, 1) and z(α) is the quantile of a standard normal distribution. SLOPE aims to provide finite sample guarantees on the selected model; of special interest is the false discovery rate (FDR), defined as the expected proportion of irrelevant regressors among all selected predictors. Under orthogonal designs, SLOPE with λBH provably controls FDR at level q. Moreover, it also appears to have appreciable inferential properties under more general designs X while having substantial power, as demonstrated in a series of experiments running on both simulated and real data.

Efficient estimation and variable selection in dynamic panel data partially linear varying coefficient models with incidental parameter

This paper is concerned with the statistical inference of partially linear varying coefficient dynamic panel data model with incidental parameter, including efficient estimation of the parametric and nonparametric components and consistent determination of the lagged order. For the parametric component, we propose an efficient semiparametric generalized method-of-moments (GMM) estimator and establish its asymptotic normality. For the nonparametric component, B-spline series approximation is employed to estimate the unknown coefficient functions, which are shown to achieve the optimal nonparametric convergence rate. A consistent estimator of the variance of error component is also constructed. In addition, by using the smooth-threshold GMM estimating equations, we propose a variable selection method to identify the significant order of lagged terms automatically and remove the irrelevant regressors by setting their coefficient to zeros. As a result, it can consistently determine the true lagged order and specify the significant exogenous variables. Further studies show that the resulting estimator has the same asymptotic properties as if the true lagged order and significant regressors were known prior, i.e., achieving the oracle property. Numerical experiments are conducted to evaluate the finite sample performance of our procedures. An example of application is also illustrated.

Shrinkage Estimation of Dynamic Panel Data Models with Interactive Fixed Effects

We consider the problem of determining the number of factors and selecting the proper regressors in linear dynamic panel data models with interactive fixed effects. Based on the preliminary estimates of the slope parameters and factors a la Bai and Ng (2009) and Moon and Weidner (2014a), we propose a method for simultaneous selection of regressors and factors and estimation through the method of adaptive group Lasso (least absolute shrinkage and selection operator). We show that with probability approaching one, our method can correctly select all relevant regressors and factors and shrink the coefficients of irrelevant regressors and redundant factors to zero. Further, we demonstrate that our shrinkage estimators of the nonzero slope parameters exhibit some oracle property. We conduct Monte Carlo simulations to demonstrate the superb finite-sample performance of the proposed method. We apply our method to study the determinants of economic growth and find that in addition to three common unobserved factors selected by our method, government consumption share has negative effects, whereas investment share and lagged economic growth have positive effects on economic growth.

The Value of Function Valuation Systems

Function valuation (grading) is an essential Human Resources (HR) tool with a long history in business, but a lack of empirical evidence. Based on a unique data sample of German DAX companies from 2006 to 2009, this paper is the first that provides comprehensive empirical results on the accuracy of the most prevalent grading models of the Hay Group, Mercer, Towers Watson, the German trade union IGM and of related research studies.The prevalent function valuation systems are very good HR instruments. However, similar to the according function/compensation valuation theories of academics, they still leave room for improvement. Both, practitioner and academic models, miss important variables and incorporate irrelevant regressors. A combination of practitioner and academic models can close the existing gap significantly. Overall, function valuation is a combination of company environment, company size and structure and functional characteristics.

Estimating the effect of technological factors from samples affected by collinearity: a data-weighted entropy approach

Measuring the effect of technological activities on productivity growth is an issue that attracted much attention in recent works on empirical econometric studies. Specifically, in the field of regional economics, several attempts have been made in order to quantify the contribution of R&D to labor productivity growth at a regional scale, considering both the internal R&D and the effects obtained by geographical spillovers. The results obtained, however, are characterized by a huge variability and in many cases there is no empirical evidence of positive contributions of R&D activities to productivity growth. Our argument is that this can be a consequence of dealing with samples’ affect by a high level of collinearity. This paper proposes the use of the data-weighted prior (DWP) estimator suggested by Golan (J Econom 101:165–193, 2001). The main advantage of this estimator is that it discriminates between relevant and irrelevant regressors better than other estimators when dealing with highly collinear samples. We evaluate the performance of the DWP estimator by Monte Carlo simulations and illustrate how it works by means of a real-world example.

Additive regression splines with irrelevant categorial and continuous regressors

We consider the problem of estimating a relationship using semiparametric additive regression splines when there exist both continuous and categorical regressors and some of the regressors are irrelevant however this is not known a priori. We show that choosing the spline degree, number of subintervals, and bandwidths via cross-validation can automatically remove irrelevant regressors thereby delivering 'automatic dimensionality reduction' without the need for pre-testing. Theoretical underpinnings are provided, nite- sample performance is studied, and an illustrative application demonstrates the ecacy of the proposed approach in nite-sample settings. An R package implementing the methods is available from the Comprehensive R Archive Network (Racine & Nie (2011)).

False posteriors for the long-term growth determinants

This paper shows: (1) bias may exist in Bayesian methods of selecting regressors; (2) the non-zero posterior inclusion probability of irrelevant regressors might affect the choice of regressors; and (3) how to compute the probability of falsely including a variable.

Nonparametric regression with weakly dependent data: the discrete and continuous regressor case

Data-driven methods of bandwidth selection are necessary for the sound application of kernel methods, with benefits including but not limited to automatic dimensionality reduction in the presence of irrelevant regressors [P. Hall, Q. Li, and J.S. Racine, ‘Nonparametric estimation of regression functions in the presence of irrelevant regressors, Rev. Econ. Statist. 89 (2007), pp. 784–789] and the ability to handle the mix of discrete and continuous data often encountered in applied settings without resorting to sample splitting [J.S. Racine and Q. Li, Nonparametric estimation of regression functions with both categorical and continuous data, J. Econometrics 119(1) (2004), pp. 99–130]. Many existing results have been developed under the presumption of independence, which may not hold when one deals with time-series data. This paper develops the properties of data-driven kernel regression for weakly dependent mixed discrete and continuous data. Monte Carlo simulations are undertaken to examine the finite-sample properties of the estimator, and an illustrative application is presented.

ADDING REGRESSORS TO OBTAIN EFFICIENCY

It is well known that in standard linear regression models with independent and identically distributed data and homoskedasticity, adding “irrelevant regressors” hurts (asymptotic) efficiency unless such irrelevant regressors are orthogonal to the remaining regressors. But we have found that under (conditional) heteroskedasticity “irrelevant regressors” can always be found such that one can achieve the asymptotic variance of the generalized least squares estimator by adding the “irrelevant regressors” to the model.

NONPARAMETRIC ESTIMATION OF REGRESSION FUNCTIONS WITH DISCRETE REGRESSORS

We consider the problem of estimating a nonparametric regression model containing categorical regressors only. We investigate the theoretical properties of least squares cross-validated smoothing parameter selection, establish the rate of convergence (to zero) of the smoothing parameters for relevant regressors, and show that there is a high probability that the smoothing parameters for irrelevant regressors converge to their upper bound values, thereby automatically smoothing out the irrelevant regressors. A small-scale simulation study shows that the proposed cross-validation-based estimator performs well in finite-sample settings.

K-nearest-neighbour non-parametric estimation of regression functions in the presence of irrelevant variables

Summary We show that when estimating a non-parametric regression model, the k-nearest-neighbour non-parametric estimation method has the ability to remove irrelevant variables provided one uses a product weight function with a vector of smoothing parameters, and the least-squares cross-validation method is used to select the smoothing parameters. Simulation results are consistent with our theoretical analysis and show that the performance of the k-nn estimator is comparable to the popular kernel estimator; and it dominates a non-parametric series (spline) estimator when there exist irrelevant regressors.

Nonparametric Estimation of Regression Functions in the Presence of Irrelevant Regressors

In this paper we consider a nonparametric regression model that admits a mix of continuous and discrete regressors, some of which may in fact be redundant (that is, irrelevant). We show that, asymptotically, a data-driven least squares cross-validation method can remove irrelevant regressors. Simulations reveal that this “automatic dimensionality reduction” feature is very effective in finite-sample settings.

On the finite sampling properties of the Zheng test for omitted and irrelevant variable problems

The paper conducts simple Monte Carlo simulation investigations to ascertain the finite sample properties of the Zheng test when applied to familiar specification errors in the form of omitted and irrelevant variable problems. Results indicate that the respective estimated sizes underestimate the nominal sizes. In an omitted variable setting, the Zheng test performs rather well with estimated powers being sensitive to increasing bandwidth parameters. However, the test exhibited contrasting results in examining power performance for testing irrelevant regressors.

Nonparametric Estimation of Multifactor Continuous Time Interest Rate Models

This paper studies the finite sample properties of the kernel regression method of Boudoukh, Richardson, Stanton and Whitelaw (1998) for estimating multifactor continuous {time term structure models. Monte Carlo simulations are employed, with a grid-search techniqueto find the optimal kernel bandwidth. The performance of the estimator is also studied under model misspecification. Irrelevant regressors reduce efficiency and induce additional biases in the estimates. Using Treasury bill data, I test whether the estimates produced by the non-parametric estimator are statistically distinguishable from estimates obtained under a parametric model. The kernel regressions pick up nonlinearities that the parametric model cannot capture.