Biased Regression Methods Research Articles

Regression Analysis of Collinear Data using r-k Class Estimator: Socio-Economic and Demographic Factors Affecting the Total Fertility Rate (TFR) in India

A basic assumption concerned with general linear regression model is that there is no correlation (or no multicollinearity) between the explana tory variables. When this assumption is not satisfied, the least squares estimators have large variances and become unstable and may have a wrong sign. Therefore, we resort to biased regression methods, which stabilize the parameter estimates. Ridge regression (RR) and principal component regression (PCR) are two of the most popular biased regression methods which can be used in case of multicollinearity. But the r-k class estimator, which is composed by combining the RR estimator and the PCR estimator into a single estimator gives the better estimates of the regression coefficients than the RR estimator and PCR estimator. This paper explores the multiple regression technique using r-k class estimator between TFR and other socio-economic and demographic variables and the data has been taken from the National Family Health Survey-III (NFHS-III): 29 states of India. The analysis shows that use of contraceptive devices shares the greatest impact on fertility rate followed by maternal care, use of improved water, female age at marriage and spacing between births.

The Comparison of Classical and Robust Biased Regression Methods for Determining Unemployment Rate in Turkey: Period of 1985-2012

Unemployment is one of the most important issues in macro economics. Unemployment creates many economic and social problems in the economy. The condition and qualification of labor force in a country show economical developments. In the light of these facts, a developing country should overcome the problem of unemployment. In this study, the performance of robust biased Robust Ridge Regression (RRR), Robust Principal Component Regression (RPCR) and RSIMPLS methods are compared with each other and their classical versions known as Ridge Regression (RR), Principal Component Regression (PCR) and Partial Least Squares Regression (PLSR) in terms of predictive ability by using trimmed Root Mean Squared Error (TRMSE) statistic in case of both of multicollinearity and outliers existence in an unemployment data set of Turkey. Analysis results show that RRR model is chosen as the best model for determining unemployment rate in Turkey for the period of 1985-2012. Robust biased RRR method showed that the most important independent variable effecting the unemployment rate is Purchasing Power Parities (PPP). The least important variables effecting the unemployment rate are Import Growth Rate (IMP) and Export Growth Rate (EXP). Hence, any increment in PPP cause an important increment in unemployment rate, however, any increment in IMP causes an unimportant increase in unemployment rate. Any increment in EXP causes an unimportant decrease in unemployment rate.

Characterizing multivariate calibration tradeoffs (bias, variance, selectivity, and sensitivity) to select model tuning parameters

Common methods used to form multivariate calibration models are partial least squares, principal component regression, and ridge regression. There are many aspects to forming a calibration model. The focus in this paper is determining “good” values of respective meta‐parameters (tuning parameters). In most situations, some form of cross‐validation and the resultant root mean square error of cross‐validation (RMSECV) values are evaluated. The RMSECV values can be thought of as a prediction accuracy (bias) measure and does not isolate variance information. The methods of partial least squares, principal component regression, and ridge regression (and others) are biased regression methods, and hence, there is a bias/variance tradeoff that should be considered in determining tuning parameter values. Presented in this paper are new combinations of model accuracy and fit that can be plotted with a model variance indicator for selecting appropriate tuning parameter values. For the spectroscopic and industrial data sets evaluated, these new combinations appear to remove ambiguities that can occur in RMSECV plots. Essentially, the user's preference for the degree of balance between bias and variance ultimately decides the tuning parameter selection and the underlying tradeoff between model selectivity and sensitivity. From the analysis of the selectivity/sensitivity tradeoff, a new definition of model selectivity is proposed that is able to discern when or when not a model vector deviates from the target orthogonal net analyte signal model vector as the well as the degree of deviation. A new merit is also proposed that simultaneously evaluates model selectivity and sensitivity. Copyright © 2013 John Wiley & Sons, Ltd.

Comparing Ridge Regression and Principal Components Regression by Monte Carlo Simulation Based on MSE

In multiple regression model, regression variables are usually assumed to be independent from each other. When this assumption is not established, the model would be inappropriate and therefore the results might be incorrect. So, biased regression methods are applied. Ridge regression and principal components regression are two methods of biased regression methods. In this paper, Monte Carlo simulation tests were used for estimating coefficients of ridge and principal components regression. These two methods were compared using minimum squared error (MSE).

A comparative study of principal component regression and partial least squares on and partial least squares regression with regression with application to FTIR diabetes data

In recent years, Fourier Transform Infrared (FT-IR) spectroscopy has had an increasingly important role in the field of pathology and diagnosis of disease states. The principal component regression (PCR) and the partial least squares regression (PLS) are the often proposed methods and widely used in FTIR data analysis, when the number of explanatory variable is relatively large in comparison to the samples as the least squares estimator may fail in such situations. They provide biased estimators with the relatively smaller variation than the variance of the least squares estimators. In this paper, a FTIR diabetes dataset is used in order to examine the performance of the two biased regression models on prediction. The conclusion is that for prediction PCR and PLS provides similar results which require substantial verification for any claims as to the superiority of any of the two biased regression methods. Keywords: Fourier Transform Infrared, Principal Com Keywords: ponent Regression, Partial Least Square, Diabetes Data.

A Comparative Study of Principal Component Regression and Partial least Squares Regression with Application to FTIR Diabetes Data

In recent years, Fourier Transform Infrared (FT-IR) spectroscopy has had an increasingly important role in the field of pathology and diagnosis of disease states. The principal component regression (PCR) and the partial least squares regression (PLS) are the often proposed methods and widely used in FTIR data analysis, when the number of explanatory variable is relatively large in comparison to the samples as the least squares estimator may fail in such situations. They provide biased estimators with the relatively smaller variation than the variance of the least squares estimators. In this paper, a FTIR diabetes dataset is used in order to examine the performance of the two biased regression models on prediction. The conclusion is that for prediction PCR and PLS provides similar results which require substantial verification for any claims as to the superiority of any of the two biased regression methods.

A closer look at the bias-variance trade-off in multivariate calibration

Principal component regression (PCR) and partial least squares (PLS) avoid the high variance often associated with ordinary least squares (OLS) results by allowing a small bias in the model. This paper presents a closer look at this bias–variance trade-off by discussing three practical aspects: (1) variance increases relatively slowly with increasing model complexity; (2) bias may be zero for the optimum model; (3) variance does not necessarily increase with increasing model complexity. While the first aspect is well known, the last two aspects are not. The second aspect implies that so-called biased regression methods do not necessarily yield biased predictions, while the third aspect, which is only encountered with non-linear estimation methods such as PLS, even contradicts the concept of bias–variance trade-off. The possibility of having both variance and bias decreasing with increasing PLS model complexity is illustrated using a near-infrared data set published by Fearn. Copyright © 1999 John Wiley & Sons, Ltd.

A closer look at the bias-variance trade-off in multivariate calibration

Principal component regression (PCR) and partial least squares (PLS) avoid the high variance often associated with ordinary least squares (OLS) results by allowing a small bias in the model. This paper presents a closer look at this bias–variance trade-off by discussing three practical aspects: (1) variance increases relatively slowly with increasing model complexity; (2) bias may be zero for the optimum model; (3) variance does not necessarily increase with increasing model complexity. While the first aspect is well known, the last two aspects are not. The second aspect implies that so-called biased regression methods do not necessarily yield biased predictions, while the third aspect, which is only encountered with non-linear estimation methods such as PLS, even contradicts the concept of bias–variance trade-off. The possibility of having both variance and bias decreasing with increasing PLS model complexity is illustrated using a near-infrared data set published by Fearn. Copyright © 1999 John Wiley & Sons, Ltd.

Length modified ridge regression

Biased regression methods may improve considerably on ordinary least squares regression with few or noisy data, or when the predictor variables are highly collinear. In the present work, I present a new, biased method that modifies the ordinary least squares estimate by adjusting each element of the estimated coefficient vector. The adjusting factors are found by minimizing a measure of prediction error. However, the optimal adjusting factors depend on the unknown coefficient vector as well as the variance of the noise, so in practice these are replaced by preliminary estimates. The final estimate of the coefficient vector has the same direction as the preliminary estimate, but the length is modified. Ridge regression is used as the principal method to find the preliminary estimate, and the method is called length modified ridge regression. In addition, length modified principal components regression is considered. The prediction performance of the methods are compared to other regression methods (ridge, James-Stein, partial least squares, principal components and variable subset selection) in a simulation study. Of all methods considered, length modified ridge regression shows the overall best behaviour. The improvement over ridge regression is moderate, but significant, especially when the data are few and noisy.

Propagation of measurement errors for the validation of predictions obtained by principal component regression and partial least squares

Multivariate calibration aims to model the relation between a dependent variable, e.g. analyte concentration, and the measured independent variables, e.g. spectra, for complex mixtures. The model parameters are obtained in the form of a regression vector from calibration data by regression methods such as principal component regression (PCR) or partial least squares (PLS). Subsequently, this regression vector is used to predict the dependent variable for unknown mixtures. The validation of the obtained predictions is a crucial part of the procedure, i.e. together with the point estimate an interval estimate is desired. The associated prediction intervals can be constructed from the covariance matrix of the estimated regression vector. However, currently known expressions for PCR and PLS are derived within the classical regression framework, i.e. they only take the uncertainty in the dependent variable into account. This severely limits their capability for establishing realistic prediction intervals in practical situations. In this paper, expressions are derived using the method of error propagation that also account for the measurement errors in the independent variables. An exact linear relation is assumed between the dependent and independent variables. The obtained expressions are therefore valid for the classical errors-in-variables (EIV) model. In order to make the presentation reasonably self-contained, relevant expressions are reviewed for the classical regression model as well as the classical EIV model, especially for ordinary least squares (OLS). The consequences for the limit of detection, wavelength selection, sample selection and local modeling are discussed. Diagnostics are proposed to determine the adequacy of the approximations used in the derivations. Finally, PCR and PLS are so-called biased regression methods. Compared with OLS, they yield small variance at the expense of increased bias. It follows that bias may be an important ingredient of the obtained predictions. Therefore considerable attention is paid to the quantification of bias and new stopping rules for model selection in PCR and PLS are proposed. The theoretical ideas are illustrated by the analysis of real data taken from the literature (classical regression model) as well as simulated data (classical EIV model). © 1997 John Wiley & Sons, Ltd.

On the predictive performance of biased regression methods and multiple linear regression

Kowalski, K.G., 1990. On the predictive performance of biased regression methods and multiple linear regression. Chemometrics and Intelligent Laboratory Systems, 9: 177–184. The predictive performance of three commonly used biased regression methods (ridge, principal components and partial least squares) and multiple linear regression (ordinary least squares) using classical model selection techniques are evaluated on five data sets published in the chemical and statistical literature. For these five data sets, the degree of collinearity among the regressors varies considerably. For each data set, cross-validation is performed and the prediction error sum of squares (PRESS) is computed to assess the predictive performance of each method. The results show that multiple linear regression using reduced models obtained by classical methods of model selection performed better (lower PRESS) than the three commonly used biased regression methods.

On identification of transfer function models by biased regression methods

This paper investigates a biased regression approach to the preliminary estimation of the Box-Jenkins transfer function weights. Using statistical simulation to generate time series, 14 estimators ...

Regresion espectral sesgada

A spectral regression method for dealing with regressors having negligible spectral density over narrow frequency bands is proposed. Its relationship with biased regression methods in the time domain is investigated