Interrelationships of multivariate regression methods using eigenvector basis sets

Abstract

This paper provides an expository discussion of the interrelationships between least squares (LS), principal component regression (PCR), partial least squares (PLS), ridge regression (RR), generalized ridge regression (GRR), continuum regression (CR) and cyclic subspace regression (CSR) for the linear model y = Xb + e. Developed in this paper is continuum CSR (CCSR). From this study it is ascertained that GRR encompasses LS, PCR, PLS, RR, CR, CSR and CCSR. It is shown that a regression vector, regardless of its source, can be written as a linear combination of the vi eigenvectors obtained from a singular value decomposition (SVD) of X, i.e. X = UΣVT. Similarly, it is shown that calibration fitted values ŷ obtained from any linear regression method can be written as a linear combination of the ui eigenvectors obtained from an SVD of X. Formulae are provided to compute ϕ and γ, respective vectors of weights for vi and ui eigenvectors. It is recommended that the ϕ eigenvector weights be inspected to ascertain exactly what information is being used to form the regression vector for the particular modeling approach used. Analogously, the γ eigenvector weights should be inspected to determine what information is being used to form calibration fitted values. Besides assisting in prediction rank determination, both eigenvector weight plots also allow for easy comparison of models built by different methods, e.g. the PCR model versus the PLS model. It is shown that it is not the number of factors used to build a PCR or PLS model that is important, but the number of eigenvectors used, which ones, and how they are weighted to form respective regression vectors and fitted values of calibration samples. In essence, how eigenvectors are weighted dictates which GRR model is formed. From the CR, CSR and RR eigenvector weight plots of ϕ it is concluded that the optimal model will most often have a combination of PCR and PLS attributes. Copyright © 1999 John Wiley & Sons, Ltd.