Principal Components Regression Estimator Research Articles

Evaluating Model-assisted Estimators: A Comparative Study in High-dimensional Survey Data

Model-assisted estimators have gained significant attention due to their ability to efficiently utilize auxiliary information during the estimation process. These estimators rely on a working model that links the survey variable to the auxiliary variables, which is then fitted to the sample data to generate predictions. These predictions are subsequently integrated into the estimation procedures. In this study, were explores various model-assisted estimators including Generalized Regression (GREG), Ridge regression, Lasso regression, CART (Classification and Regression Tree), Random Forest, Cubist and Principal Components Regression (PCR) estimator. The analysis involved 2,000 samples of size 50 (n/N ≈ 10%) and employed a stepwise variable selection method to determine the most significant auxiliary variables, incrementally adding them to the model. The performance of these estimators was assessed using relative bias (RB), relative root mean square error (RRMSE) and relative efficiency (RE). Our findings reveal that tree-based models like CART and Random Forest and penalized regression estimators such as Ridge and Lasso display robustness with increased number of auxiliary variables. Among all the estimators, Random Forest consistently yielded the lowest RRMSE, particularly with five auxiliary variables, demonstrating superior efficiency. Conversely, the GREG estimator exhibited poor performance as the number of auxiliary variables increased. This study underscores the importance of selecting suitable model-assisted estimation procedures tailored to the data characteristics and the relationship between survey and auxiliary variables within this high-dimensional dataset.

Combination of the modified Kibria–Lukman and the principal component regression estimators

Statistical inference with the ordinary least squares (OLS) estimator is frequently influenced when there is a multicollinearity in the linear regression model. In this article, to reduce these effects of multicollinearity, we generalize the modified Kibria–Lukman principal component (MKLPC) estimator in the linear regression model by combining the principal component regression (PCR) estimator and the modified Kibria–Lukman (MKL) estimator. Meanwhile, the necessary and sufficient conditions for the superiority of the MKLPC estimator over OLS, PCR, Ridge, r-k, Liu, r-d, k-d, KL, and MKL estimators in the mean squared error (MSE) criterion are derived. Furthermore, we conduct Monte Carlo simulation and empirical analysis to compare these estimators under the MSE criterion.

A New Approach of Principal Component Regression Estimator with Applications to Collinear Data

A New Approach of Principal Component Regression Estimator with Applications to Collinear Data

Principal components estimator for measurement error models

ABSTRACTIn this paper, we carry out the principal components regression approach to the measurement error models. We introduce the principal components estimator and then the restricted principal components estimator by combining the approaches principal components regression estimator and restricted least squares estimator for the measurement error models, when the reliability matrix known and unknown, separately. We investigate the asymptotic properties and matrix mean squared error performances of the new estimators. Also, we conduct a Monte Carlo simulation study and a numerical example to investigate the performances of the proposed estimators by the scalar mean squared error criterion.

Variable Selection via Biased Estimators in the Linear Regression Model

Least Absolute Shrinkage and Selection Operator (LASSO) is used for variable selection as well as for handling the multicollinearity problem simultaneously in the linear regression model. LASSO produces estimates having high variance if the number of predictors is higher than the number of observations and if high multicollinearity exists among the predictor variables. To handle this problem, Elastic Net (ENet) estimator was introduced by combining LASSO and Ridge estimator (RE). The solutions of LASSO and ENet have been obtained using Least Angle Regression (LARS) and LARS-EN algorithms, respectively. In this article, we proposed an alternative algorithm to overcome the issues in LASSO that can be combined LASSO with other exiting biased estimators namely Almost Unbiased Ridge Estimator (AURE), Liu Estimator (LE), Almost Unbiased Liu Estimator (AULE), Principal Component Regression Estimator (PCRE), r-k class estimator and r-d class estimator. Further, we examine the performance of the proposed algorithm using a Monte-Carlo simulation study and real-world examples. The results showed that the LARS-rk and LARS-rd algorithms, which are combined LASSO with r-k class estimator and r-d class estimator, outperformed other algorithms under the moderated and severe multicollinearity.

THE PREDICTIVE PERFORMANCE EVALUATION AND NUMERICAL EXAMPLE STUDY FOR THE PRINCIPAL COMPONENT TWO-PARAMETERS ESTIMATOR

In this paper, detailed comparisons are given between those estimators that can be derived from the principal component two-parameter estimator such as the ordinary least squares estimator, the principal components regression estimator, the ridge regression estimator, the Liu estimator, the r-k estimator and the r-d estimator by the prediction mean square error criterion. In addition, conditions for the superiority of the principal component two-parameter estimator over the others are obtained. Furthermore, a numerical example study is conducted to compare these estimators under the prediction mean squared error criterion.

Amendments of a Stochastic Restricted Principal Components Regression Estimator in the Linear Model

Principal component Analysis (PCA) is one of the popular methods used to solve the multicollinearity problem. Researchers in 2014 proposed an estimator to solve this problem in the linear model when there were stochastic linear restrictions on the regression coefficients. This estimator was called the stochastic restricted principal components (SRPC) regression estimator. The estimator was constructed by combining the ordinary mixed estimator (OME) and the principal components regression (PCR) estimator. It ignores the number of components (orthogonal matrix Tr) that the researchers choose to solve the multicollinearity problem in the data matrix (X). This paper proposed four different methods (Lagrange function, the same technique, the constrained principal component model, and substitute in model) to modify the (SRPC) estimator to be used in case of multicollinearity. Finally, a numerical example, an application, and simulation study have been introduced to illustrate the performance of the proposed estimator.

Stochastic Restricted Biased Estimators in Misspecified Regression Model with Incomplete Prior Information

The analysis of misspecification was extended to the recently introduced stochastic restricted biased estimators when multicollinearity exists among the explanatory variables. The Stochastic Restricted Ridge Estimator (SRRE), Stochastic Restricted Almost Unbiased Ridge Estimator (SRAURE), Stochastic Restricted Liu Estimator (SRLE), Stochastic Restricted Almost Unbiased Liu Estimator (SRAULE), Stochastic Restricted Principal Component Regression Estimator (SRPCRE), Stochastic Restricted r-k (SRrk) class estimator, and Stochastic Restricted r-d (SRrd) class estimator were examined in the misspecified regression model due to missing relevant explanatory variables when incomplete prior information of the regression coefficients is available. Further, the superiority conditions between estimators and their respective predictors were obtained in the mean square error matrix (MSEM) sense. Finally, a numerical example and a Monte Carlo simulation study were used to illustrate the theoretical findings.

Performance of Existing Biased Estimators and the Respective Predictors in a Misspecified Linear Regression Model

In this paper, the performance of existing biased estimators (Ridge Estimator (RE), Almost Unbiased Ridge Estimator (AURE), Liu Estimator (LE), Almost Unbiased Liu Estimator (AULE), Principal Component Regression Estimator (PCRE), r-k class estimator and r-d class estimator) and the respective predictors were considered in a misspecified linear regression model when there exists multicollinearity among explanatory variables. A generalized form was used to compare these estimators and predictors in the mean square error sense. Further, theoretical findings were established using mean square error matrix and scalar mean square error. Finally, a numerical example and a Monte Carlo simulation study were done to illustrate the theoretical findings. The simulation study revealed that LE and RE outperform the other estimators when weak multicollinearity exists, and RE, r-k class and r-d class estimators outperform the other estimators when moderated and high multicollinearity exist for certain values of shrinkage parameters, respectively. The predictors based on the LE and RE are always superior to the other predictors for certain values of shrinkage parameters.

On the predictive performance of the almost unbiased Liu estimator

ABSTRACTRegression models are usually used in forecasting (predicting) unknown values of the response variable y. This article considers the predictive performance of the almost unbiased Liu estimator compared to the ordinary least-squares estimator, principal component regression estimator, and Liu estimator. Finally, we present a numerical example to explain the theoretical results and we obtain a region where the almost unbiased Liu estimator is uniformly superior to the ordinary least-squares estimator, principal component regression estimator, and Liu estimator.

The small sample properties of the restricted principal component regression estimator in linear regression model

ABSTRACTIn regression analysis, to deal with the problem of multicollinearity, the restricted principal components regression estimator is proposed. In this paper, we compared the restricted principal components regression estimator, the principal components regression estimator, and the ordinary least-squares estimator with each other under the Pitman's closeness criterion. We showed that the restricted principal components regression estimator is always superior to the principal components regression estimator, under certain conditions the restricted principal components regression estimator is superior to the ordinary least-squares estimator under the Pitman's closeness criterion and under certain conditions the principal components regression estimator is superior to the ordinary least-squares estimator under the Pitman's closeness criterion.

Principal components regression estimator of the parameters in partially linear models

ABSTRACTAs a compromise between parametric regression and non-parametric regression models, partially linear models are frequently used in statistical modelling. This paper is concerned with the estimation of partially linear regression model in the presence of multicollinearity. Based on the profile least-squares approach, we propose a novel principal components regression (PCR) estimator for the parametric component. When some additional linear restrictions on the parametric component are available, we construct a corresponding restricted PCR estimator. Some simulations are conducted to examine the performance of our proposed estimators and the results are satisfactory. Finally, a real data example is analysed.

Principal Components Regression Estimation in Semiparametric Partially Linear Additive Models

<p>Partially linear additive model is useful in statistical modelling as a multivariate nonparametric fitting technique. This paper considers statistical inference for the semiparametric model in the presence of multicollinearity. Based on the profile least-squares approach, we propose a novel principal components regression estimator for the parametric component, and provide the asymptotic bias and covariance matrix of the proposed estimator. Some simulations are conducted to examine the performance of our proposed estimators and the results are satisfactory.</p>

Comparison of the r- (k, d) class estimator with some estimators for multicollinearity under the Mahalanobis loss function

In the case of ill-conditioned design matrix in linear regression model, the r - (k, d) class estimator was proposed, including the ordinary least squares (OLS) estimator, the principal component regression (PCR) estimator, and the two-parameter class estimator. In this paper, we opted to evaluate the performance of the r - (k, d) class estimator in comparison to others under the weighted quadratic loss function where the weights are inverse of the variance-covariance matrix of the estimator, also known as the Mahalanobis loss function using the criterion of average loss. Tests verifying the conditions for superiority of the r - (k, d) class estimator have also been proposed. Finally, a simulation study and also an empirical illustration have been done to study the performance of the tests and hence verify the conditions of dominance of the r - (k, d) class estimator over the others under the Mahalanobis loss function in artificially generated data sets and as well as for a real data. To the best of our knowledge, this study provides stronger evidence of superiority of the r - (k, d) class estimator over the other competing estimators through tests for verifying the conditions of dominance, available in literature on multicollinearity.

The principal correlation components estimator and its optimality

In regression analysis, the principal components regression estimator (PCRE) is often used to alleviate the effect of multicollinearity by deleting the principal components variables with smaller eigenvalues, but it has some drawbacks for subset selection. So another effective estimator based on the principal components, termed the principal correlation components estimator (PCCE), has been proposed, which selects the variables by considering the correlation coefficients between the principal components variables and the response variable. In this paper we investigate the property of the PCCE and its optimality. We prove that the PCCE is an admissible estimator and obtain the condition that the PCCE performs better than the PCRE under the balanced loss function (BLF). As an improvement of the least squares estimator (LSE), the conditions that the PCCE performs better than LSE under the BLF and the Pitman closeness criterion are derived in the paper. The empirical performance of the PCCE is demonstrated by several real and simulation data examples.

A restricted $$r{-}k$$ r - k class estimator in the mixed regression model with autocorrelated disturbances

In this paper, a new estimator is proposed by combining basically the ordinary ridge regression estimator and the principal component regression estimator for a regression model which has multicollinearity and which satisfies some a priori stochastic linear restrictions. The performance of the proposed \(r{-}k\) class estimator in this mixed regression model is compared with those of the mixed regression estimator, ridge regression estimator and the stochastic restricted ridge regression estimator in terms of the mean squared error matrix criterion. Tests for verifying the conditions of dominance of the proposed estimator over the others are also proposed. Furthermore, a Monte Carlo study and a numerical illustration are carried out to empirically study the performance of the estimators by the mean squared error values, and then to perform tests to verify if the conditions for superiority of the proposed estimator over the others hold.

Modified and Restricted r-k Class Estimators

In this article, we introduce the modified r-k class estimator and the restricted r-k class estimator. We compare the performances of the new estimators to the r-k class estimator with respect to the matrix mean square error (MSE) criterion. As a special case of the restricted r-k class estimator, we obtain the restricted principal components regression (RPCR) estimator. Finally, we conduct a Monte Carlo simulation study and a numerical example to investigate the performances of the proposed estimators by the scalar mean square error (mse) criterion.

A Class of s–K Type Principal Components Estimators in the Linear Model

In this article, we introduce a new class of estimators called the s–K type principal components estimators to combat multicollinearity, which include the principal components regression (PCR) estimator, the r–k estimator and the s–K estimator as special cases. Necessary and sufficient conditions for the superiority of the new estimator over the PCR estimator, the r–k estimator and the s–K estimator are derived in the sense of the mean squared error matrix criterion. A Monte Carlo simulation study and a numerical example are given to illustrate the performance of the proposed estimator.

Efficiency of the restricted [formula omitted] class estimator in linear regression

In this paper, a restricted r-d class estimator is introduced for the vector of parameters in a linear regression model with linear restrictions. The risk function for the restricted r-d class estimator and restricted principal component regression estimator associated with balanced loss function are given and the comparison of the restricted r-d class estimator and restricted principal component regression estimator associated with balanced loss function is also presented. The performance of the proposed estimators is evaluated by a simulated data set.

Comparison of Some Estimators under the Pitman’s Closeness Criterion in Linear Regression Model

Batah et al. (2009) combined the unbiased ridge estimator and principal components regression estimator and introduced the modifiedr-kclass estimator. They also showed that the modifiedr-kclass estimator is superior to the ordinary least squares estimator and principal components regression estimator in the mean squared error matrix. In this paper, firstly, we will give a new method to obtain the modifiedr-kclass estimator; secondly, we will discuss its properties in some detail, comparing the modifiedr-kclass estimator to the ordinary least squares estimator and principal components regression estimator under the Pitman closeness criterion. A numerical example and a simulation study are given to illustrate our findings.