Ordinary Ridge Estimator Research Articles

Ridge parameter estimation for the linear regression model under different loss functions using T-K approximation

In multiple linear regression models, the explanatory variables should be uncorrelated within each other but this assumption is violated in most of the cases. Generally, ordinary least square (OLS) estimator produces larger variances when explanatory variables are highly multicollinear. So, in this paper, we propose some new ridge parameters under Bayesian perspective relative to different loss functions, using Tierney and Kadane (T-K) approximation technique to overcome the effect of multicollinearity. We conduct the simulation study to compare the performance of the proposed estimators with OLS estimator and ordinary ridge estimator with some available best ridge parameters using mean squared error as the performance evaluation criterion. A real application is also consider to show the superiority of proposed estimators against competitive estimators. Based on the results of simulation and real application, we conclude that Bayesian ridge parameter estimated under general entropy loss function is better as compared to the OLS estimator and ordinary ridge estimator, when explanatory variables are small. This statement is also true for larger explanatory variables with small sample size. While for larger sample sizes and explanatory variables, the ordinary ridge estimator with best ridge parameter gives the better performance as others.

Restricted ridge estimator in generalized linear models: Monte Carlo simulation studies on Poisson and binomial distributed responses

ABSTRACTIt is known that collinearity among the explanatory variables in generalized linear models (GLMs) inflates the variance of maximum likelihood estimators. To overcome multicollinearity in GLMs, ordinary ridge estimator and restricted estimator were proposed. In this study, a restricted ridge estimator is introduced by unifying the ordinary ridge estimator and the restricted estimator in GLMs and its mean squared error (MSE) properties are discussed. The MSE comparisons are done in the context of first-order approximated estimators. The results are illustrated by a numerical example and two simulation studies are conducted with Poisson and binomial responses.

A Bayesian approach with generalized ridge estimation for high-dimensional regression and testing

ABSTRACTThis paper adopts a Bayesian strategy for generalized ridge estimation for high-dimensional regression. We also consider significance testing based on the proposed estimator, which is useful for selecting regressors. Both theoretical and simulation studies show that the proposed estimator can simultaneously outperform the ordinary ridge estimator and the LSE in terms of the mean square error (MSE) criterion. The simulation study also demonstrates the competitive MSE performance of our proposal with the Lasso under sparse models. We demonstrate the method using the lung cancer data involving high-dimensional microarrays.

Robust ridge estimator in restricted semiparametric regression models

In this paper, ridge and non-ridge type estimators and their robust forms are defined in the semiparametric regression model when the errors are dependent and some non-stochastic linear restrictions are imposed under a multicollinearity setting. In the context of ridge regression, the estimation of shrinkage parameter plays an important role in analyzing data. Another common problem in applied statistics is the presence of outliers in the data besides multicollinearity. In this respect, we propose some robust estimators for shrinkage parameter based on least trimmed squares (LTS) method. Given a set of n observations and the integer trimming parameter h≤n, the LTS estimator involves computing the hyperplane that minimizes the sum of the smallest h squared residuals. The LTS estimator is closely related to the well-known least median squares (LMS) estimator in which the objective is to minimize the median squared residual. Although LTS estimator has the advantage of being statistically more efficient than LMS estimator, the computational complexity of LTS is less understood than LMS. Here, we extract the robust estimators for linear and nonlinear parts of the model based on robust shrinkage estimators. It is shown that these estimators perform better than ordinary ridge estimator. For our proposal, via a Monté-Carlo simulation and a real data example, performance of the ridge type of robust estimators are compared with the classical ones in restricted semiparametric regression models.

A Generalized Stochastic Restricted Ridge Regression Estimator

In this article, we introduce a new stochastic restricted estimator for the unknown vector parameter in the linear regression model when stochastic linear restrictions on the parameters hold. We show that the new estimator is a generalization of the ordinary mixed estimator (OME), Liu estimator (LE), ordinary ridge estimator (ORR), (k-d) class estimator, stochastic restricted Liu estimator (SRLE), and stochastic restricted ridge estimator (SRRE). Performance of the new estimator in comparison to other estimators in terms of the mean squares error matrix (MMSE) is examined. Numerical example from literature have been given to illustrate the results.

Restricted Two Parameter Ridge Estimator

SummaryIn 2005 Lipovetsky and Conklin proposed an estimator, the two parameter ridge estimator (TRE), as an alternative to the ordinary least squares estimator (OLSE) and the ordinary ridge estimator (RE) in the presence of multicollinearity, and in 2006 Lipovetsky improved the two parameter model. In this paper, we introduce two new estimators, one of which is the modified two parameter ridge estimator (MTRE) defined by following Swindel's paper of 1976. The other one is the restricted two parameter ridge estimator (RTRE) which is derived by setting additional linear restrictions on the parameter vectors. This estimator is a generalization of the restricted least squares estimator (RLSE) and includes the restricted ridge estimator (RRE) proposed by Groß in 2003. A numerical example is provided and a simulation study is conducted for the comparisons of the RTRE with the OLSE, RLSE, RE, RRE and TRE.

Performances of the Positive-Rule Stein-Type Ridge Estimator in a Linear Regression Model with Spherically Symmetric Error Distributions

In this article, the positive-rule Stein-type ridge estimator (PSRE) is introduced for the parameters in a multiple linear regression model with spherically symmetric error distributions when it is suspected that the parameter vector may be restricted to a linear manifold. The bias and quadratic risk functions of the PSRE are derived and compared with some related competing estimators in literatures. Particularly, some sufficient conditions are derived for superiority of the PSRE over the ordinary ridge estimator, the restricted ridge estimator and the preliminary test ridge estimator, respectively. Furthermore, some graphical results are provided to illustrate some of the theoretical results.

On the performance of two parameter ridge estimator under the mean square error criterion

Lipovetsky and Conklin [12] proposed an estimator, two parameter ridge (ridge-2) estimator, as an alternative to the ordinary least squares (OLS) and the ordinary ridge (ridge-1) estimators in the presence of multicollinearity. Lipovetsky [13] improved the two parameter model and investigated various characteristics of ridge-2 solutions. In this paper, we compare ridge-2 estimator with the OLS, ridge-1 and contraction estimators with respect to matrix mean square error (MSE) criterion. A numerical example from the literature has been analyzed to evaluate the performance of mentioned estimators in the theoretical results.

A new ridge-type estimator in stochastic restricted linear regression

In this paper, we propose a new ridge-type estimator called the weighted mixed ridge estimator by unifying the sample and prior information in linear model with additional stochastic linear restrictions. The new estimator is a generalization of the weighted mixed estimator [B. Schaffrin and H. Toutenburg, Weighted mixed regression, Zeitschrift fur Angewandte Mathematik und Mechanik 70 (1990), pp. 735–738] and ordinary ridge estimator (ORE) [A.E. Hoerl and R.W. Kennard, Ridge regression: Biased estimation for non-orthogonal problems, Technometrics 12 (1970), pp. 55–67]. The performances of this new estimator against the weighted mixed estimator, ORE and the mixed ridge estimator [Y.L. Li and H. Yang, A new stochastic mixed ridge estimator in linear regression, Stat. Pap. (2008) (in press, DOI 10.1007/s00362-008-0169-5)] are examined in terms of the mean squared error matrix sense. Finally, a numerical example and a Monte Carlo simulation are also given to show the theoretical results.

A new estimator in multiple linear regression model

Least squares estimates of parameters of a multiple linear regression model are known to be highly variable when the matrix on independent variables has the multicollinearity problem. To circumvent this problem, several alternative methods have been suggested to improve the precision of estimators. In this paper, we introduce a new type of biased estimate so-called Al estimator to reduce the effect of multicollinearity on the estimators. By some theorems and a simulation study, we show that, this new estimator has desirable properties as the ordinary ridge estimator and it is better than the ordinary least squares estimator and Liu estimator in terms of the matrix of mean square error. A numerical example from literature is used to illustrate the results.

New Algorithm of GPS Rapid Positioning Based on Double-k-Type Ridge Estimation

In this paper, a new algorithm is employed in global positioning system (GPS) rapid positioning using several-epoch single-frequency phase data. First, we define the double- k -type ridge estimator (DKRE) based on the structure characteristics of multicollinearities of the normal equations matrix in the double-difference (DD) model, and prove when the DKRE is not worse than the least-squares estimator (LSE) in the sense of a reduced MSEM. Taking into account how the ridge parameter in the ordinary ridge estimator is confirmed based on the generalized ridge estimator (GRE), we propose a method of estimating two ridge parameters for the DKRE. Second, we improve the LAMBDA method through replacing the cofactor matrix computed by the LSE with the cofactor matrix computed by the DKRE. The ambiguities-fixed solution is found by the sequential LSE. A theorem stating that the success rate of the improved LAMBDA method is bigger than the original LAMBDA method is given. Finally, through the GPS positioning tests, ...

ROBUST-TYPE BIASED ESTIMATION IN GAUSS-MARKOV MODEL

The parameter estimation problem in Gauss-Markov model is considered when multi-collinearity and outliers exist simultaneously. A class of new estimators, robust-type generalized shrunken estimators, is proposed by grafting robust estimation technique into philosophy generalized shrunken estimation. Many useful and important estimators such as robust-type ordinary ridge estimator, robust-type principal components estimator and so on are obtained by appropriate choices of the shrinking parameter matrix. An algorithm for computing the robust-type generalized shrunken estimate is established. A numerical example is provided to illustrate that these new estimators can not only effectively overcome difficulty caused by multi-collinearity but also resist the influence of outliers.

IMPROVING THE 'HKB' ORDINARY TYPE RIDGE ESTIMATOR

The HKB estimator of Hoerl, Kennard and Baldwin is known to be an ordinary ridge type shrinkage estimator and an adjustment of degrees of freedom in the ordinary ridge estimator of Farebrother. The HKB estimator has a smaller predictive mean squared error (MSE) than the positive-part Stein-rule (PP) estimator in the wide region of the noncentral parameter when the degrees of freedom q =3 ∼ 6, but does not satisfy the sufficient condition to dominate the OLS estimator of Baranchik or Efron and Morris when q = 3. In this paper, a sufficient condition to dominate the OLS estimator is derived, and the modified HKB estimator is constructed to dominate the OLS estimator and have a smaller MSE than the PP estimator in the wide region of the noncentral parameter.

Tests of regression coefficients under ridge regression models

This paper investigates two “non-exact” t-type tests, t( k2) and t(k2), of the individual coefficients of a linear regression model, based on two ordinary ridge estimators. The reported results are built on a simulation study covering 84 different models. For models with large standard errors, the ridge-based t-tests have correct levels with considerable gain in powers over those of the least squares t-test, t(0). For models with small standard errors, t(k1) is found to be liberal and is not safe to use while, t(k2) is found to slightly exceed the nominal level in few cases. When tie two ridge tests art: not winners, the results indicate that they don't loose much against t(0).

ROBUST RIDGE REGRESSION BASED ON AN M‐ESTIMATOR

SummaryConsider the linear regression model y=β01 +Xβ+ in the usual notation. It is argued that the class of ordinary ridge estimators obtained by shrinking the least squares estimator by the matrix (X1X + kI)‐1X'X is sensitive to outliers in the ^variable. To overcome this problem, we propose a new class of ridge‐type M‐estimators, obtained by shrinking an M‐estimator (instead of the least squares estimator) by the same matrix. Since the optimal value of the ridge parameter k is unknown, we suggest a procedure for choosing it adaptively. In a reasonably large scale simulation study with a particular M‐estimator, we found that if the conditions are such that the M‐estimator is more efficient than the least squares estimator then the corresponding ridge‐type M‐estimator proposed here is better, in terms of a Mean Squared Error criteria, than the ordinary ridge estimator with k chosen suitably. An example illustrates that the estimators proposed here are less sensitive to outliers in the y‐variable than ordinary ridge estimators.

An examiniation of criticisms of ridge regression simulation methodology

Page1 (1981) claimed that there is a serious design flaw in some of the recent simulation studies of ridge estimators, in particular those in Hoerl, Kennard and Baldwin (1976) and Lawless and Wang (1976). Farebrother (1983) argued that the major criticism in Page1 (1981) is unsubstantiated. In this paper we obtain a series expansion for the mean squared error of the ordinary ridge estimator, use it to prove that Pagel's claim is incorrect and reinforce Farebrother's comments.

The existence theorem in general ridge regression

Hoerl and Kennard (1970) state that, like the ordinary ridge estimator, the general ridge estimator is also better than the least squares estimator relative to a mean square error. The proof of this result is given in this note.

On exact small sample properties of ordinary ridge estimators

In this paper, we examine the exact small sample properties of the ordinary ridge estimator and the bias-corrected ordinary ridge estimator.

Some conditions for optimality of the almost unbiased estimators of regression coefficients

The purpose of this paper is to examine the asymptotic properties of the operational almost unbiased estimator of regression coefficients which includes almost unbiased ordinary ridge estimator a s a special case. The small distrubance approximations for the bias and mean square error matrix of the estimator are derived. As a consequence, it is proved that, under certain conditions, the estimator is more efficient than a general class of estimators given by Vinod and Ullah (1981). Also it is shown that, if the ordinary ridge estimator (ORE) dominates the ordinary least squares estimator then the almost unbiased ordinary ridge estimator does not dominate ORE under the mean square error criterion.

An evaluation of biased estimators of regression coefficients—A simulation study

Different versions of generalized and ordinary ridge estimators and shrinkage estimators of regression coefficients are studied in comparison with least squares estimators using simulations. The results show that some of the biased estimators considered are better than the least squares estimator in general and the improvement is substantial in some cases.