Selection Of Principal Components Research Articles

Reliable estimation of externally validated prediction errors for QSAR models

In most cases of QSAR modelling the final model used to make predictions, is not known a priori but has to be selected in a data driven fashion (e.g. selection of principal components, variable selection, selection of the best mathematical modelling technique). Reliable estimation of externally validated prediction errors under this model uncertainty is still a challenge in chemoinformatics. To fulfil the standards of external validation, the test data set has to be independent not only from model building but also from model selection. There still is a controversy in the literature how the independent test data set should be chosen and how large it should be. For setting aside a test data set there are basically two different options: 1) a single test data set is set aside and 2) the test data are generated by repeatedly partitioning the available data into test and training set partitions - i.e. cross-validation. Since cross-validation uses the data more efficiently, it is to be preferred in particular for small data sets. The aforementioned cross-validation step must not be confused with a cross-validation step that might be necessary to select the model! If model selection is also done by cross-validation two loops of cross-validation are necessary [1]. In the inner loop, cross-validation is employed for model selection [2] (also referred to as internal validation) while in the outer loop of cross-validation different test data sets are generated repeatedly that are used to assess the readily selected models (external validation). In this contribution double cross-validation is evaluated for its ability to estimate prediction errors under model uncertainty. Depending on how double cross-validation is parameterized (test set size, number of repetitions), it either yields biased or highly variable estimates of the prediction error. The sources of bias and variability will be highlighted and recommendations are provided how to determine the test set size in order to obtain a favourable bias-variability trade-off.

Multichannel ECG Data Compression Based on Multiscale Principal Component Analysis

In this paper, multiscale principal component analysis (MSPCA) is proposed for multichannel electrocardiogram (MECG) data compression. In wavelet domain, principal components analysis (PCA) of multiscale multivariate matrices of multichannel signals helps reduce dimension and remove redundant information present in signals. The selection of principal components (PCs) is based on average fractional energy contribution of eigenvalue in a data matrix. Multichannel compression is implemented using uniform quantizer and entropy coding of PCA coefficients. The compressed signal quality is evaluated quantitatively using percentage root mean square difference (PRD), and wavelet energy-based diagnostic distortion (WEDD) measures. Using dataset from CSE multilead measurement library, multichannel compression ratio of 5.98:1 is found with PRD value 2.09% and the lowest WEDD value of 4.19%. Based on, gold standard subjective quality measure, the lowest mean opinion score error value of 5.56% is found.

Optimized intrusion detection mechanism using soft computing techniques

Intrusion detection is an important technique in computer and network security. A variety of intrusion detection approaches be present to resolve this severe issue but the main problem is performance. It is important to increase the detection rates and reduce false alarm rates in the area of intrusion detection. Therefore, in this research, an optimized intrusion detection mechanism using soft computing techniques is proposed to overcome performance issues. The KDD-cup dataset is used that is a benchmark for evaluating the security detection mechanisms. The Principal Component Analysis (PCA) is applied to transform the input samples into a new feature space. The selecting of an appropriate number of principal components is a critical problem. So, Genetic Algorithm (GA) is used in the optimum selection of principal components instead of using traditional method. The Support Vector Machine (SVM) is used for classification purpose. The performance of this approach is addresses. Further, a comparative analysis is made with existing approaches. Consequently, this method provides optimal intrusion detection mechanism which is capable to minimize amount of features and maximize the detection rates.

Face recognition using bacterial foraging strategy

This article presents an efficient algorithm for LDA-based face recognition with the selection of optimal principal components using E-coli Bacterial Foraging Optimization Technique. Different methods were suggested in the literature to select the largest eigenvalues and their corresponding eigenvectors for linear discriminant analysis (LDA). Some researchers have suggested eliminating the three largest eigenvalues to avoid the effect under varying illumination conditions. But, there is no unified approach for selecting optimal eigenvalues to enhance the performance of an algorithm. In this context, a GA–PCA algorithm has been proposed to select optimal eigenvalues and their corresponding eigenvectors in LDA. They proposed a fitness function to find the optimal eigenvectors using the Genetic Algorithm (GA). However, the crossover method used results in differences in offspring, and mutation never allowed them for a physical dispersal of the child in a chosen area. This prevents us in selecting optimal eigenvectors for improvising accuracy of the face recognition algorithm. This has motivated the authors to develop a new algorithm called BFO-Fisher which uses a nutrient concentration function (cost function) for optimization. In this work, the cost function is maximized through hill climbing via a type of biased random walk which is not possible in GA. Here the proposed BFO-Fisher algorithm offers us two distinct additional advantages—(i) the proposed algorithm can supplement the features of GA, and (ii) the random bias incorporated into the BFO algorithm guides us to move in the direction of increasingly favorable environment, which is desirable. In this experiment, both Yale and UMIST Databases are used for the performance evaluation. Experimental results presented in this article reveal the fact that about 3% (Rank 1) improvement can be achieved as compared to the GA-Fisher algorithm.

주성분회귀분석에서 주성분선정을 위한 새로운 방법

데이터마이닝 분야에서의 회귀모형에는 연관성이 높은 설명변수들이 포함되어 다중공선성을 유발하는 경우가 많은데, 다중공선성이 야기하는 문제를 해결하기 위하여 주성분회귀분석을 적용할 수 있다. 이 분석에서는 적절한 주성분을 선정하는 과정이 핵심인데, 기존의 선정방법들은 다중공선성을 잘 해결하지 못하거나 모형의 적합성을 저하시킨다는 지적을 받고 있다. 따라서 본 논문에서는 다중공선성 문제와 적합성 저하 현상을 동시에 해결할 수 있는 새로운 선정방법을 제안하였다. 다중공선성에 의해 최소제곱추정량의 분산이 팽창되는 문제를 주성분회귀에 의해 해결할 수 있지만, 주성분의 일부를 선정함에 따라 발생하는 편의도 동시에 통제해야 한다. 따라서 주성분회귀추정량의 평균제곱오차를 최소가 되게 하는 상태지수를 측정하고, 이 값에 영향을 미치는 주요 요인들을 컨조인트분석에 의해 파악하여 주성분 선정기준 모형을 구축하였다. 선정기준의 상한과 하한을 설정하고, 상태지수가 상한을 초과하면 해당 주성분을 제외시키고, 하한에 미달하면 해당 주성분을 포함시킨다. 그리고 상한과 하한 사이의 상태지수에 대응하는 주성분들에 대해서는 일반화선형검정을 순차적으로 적용하여 주성분을 선정하는 방법이다. Since the least squares estimation is not appropriate when multicollinearity exists among the regressors of the linear regression model, the principal components regression is used to deal with the multicollinearity problem. This article suggests a new procedure for the selection of suitable principal components. The procedure is based on the condition index instead of the eigenvalue. The principal components corresponding to the indices are removed from the model if any condition indices are larger than the upper limit of the cutoff value. On the other hand, the corresponding principal components are included if any condition indices are smaller than the lower limit. The forward inclusion method is employed to select proper principal components if any condition indices are between the upper limit and the lower limit. The limits are obtained from the linear model which is constructed on the basis of the conjoint analysis. The procedure is evaluated by Monte Carlo simulation in terms of the mean square error of estimator. The simulation results indicate that the proposed procedure is superior to the existing methods.

Signal-to-Noise Contribution of Principal Component Loads in Reconstructed Near-Infrared Raman Tissue Spectra

The overall quality of Raman spectra in the near-infrared region, where biological samples are often studied, has benefited from various improvements to optical instrumentation over the past decade. However, obtaining ample spectral quality for analysis is still challenging due to device requirements and short integration times required for (in vivo) clinical applications of Raman spectroscopy. Multivariate analytical methods, such as principal component analysis (PCA) and linear discriminant analysis (LDA), are routinely applied to Raman spectral datasets to develop classification models. Data compression is necessary prior to discriminant analysis to prevent or decrease the degree of over-fitting. The logical threshold for the selection of principal components (PCs) to be used in discriminant analysis is likely to be at a point before the PCs begin to introduce equivalent signal and noise and, hence, include no additional value. Assessment of the signal-to-noise ratio (SNR) at a certain peak or over a specific spectral region will depend on the sample measured. Therefore, the mean SNR over the whole spectral region (SNR(msr)) is determined in the original spectrum as well as for spectra reconstructed from an increasing number of principal components. This paper introduces a method of assessing the influence of signal and noise from individual PC loads and indicates a method of selection of PCs for LDA. To evaluate this method, two data sets with different SNRs were used. The sets were obtained with the same Raman system and the same measurement parameters on bladder tissue collected during white light cystoscopy (set A) and fluorescence-guided cystoscopy (set B). This method shows that the mean SNR over the spectral range in the original Raman spectra of these two data sets is related to the signal and noise contribution of principal component loads. The difference in mean SNR over the spectral range can also be appreciated since fewer principal components can reliably be used in the low SNR data set (set B) compared to the high SNR data set (set A). Despite the fact that no definitive threshold could be found, this method may help to determine the cutoff for the number of principal components used in discriminant analysis. Future analysis of a selection of spectral databases using this technique will allow optimum thresholds to be selected for different applications and spectral data quality levels.

Classification of toxigenic and atoxigenic strains of Aspergillus flavus with hyperspectral imaging

Aspergillus flavus ( A. flavus) produces secondary metabolites, aflatoxins, that are harmful to both humans and animals. Because of stringent federal regulation requirements as well as the limitations of available detection methods, there is an urgent need for rapid, non-invasive and effective techniques such as hyperspectral imaging, for the detection of the toxigenic strains of A. flavus. Hyperspectral images of toxigenic and atoxigenic strains of A. flavus were classified. Principal component analysis (PCA) was applied for data decorrelation and dimensionality reduction. A Genetic Algorithm (GA) was implemented for the selection of principal components (PCs) based on Bhattacharya Distance (B-Distance). A Support Vector Machine (SVM) was successfully applied for the classification. Under halogen light sources, in average 83% of the toxigenic fungus pixels and 74% of the atoxigenic fungus pixels were correctly classified; while under UV light sources, 67% of the toxigenic fungus pixels and 85% of the atoxigenic fungus pixels were correctly classified. The pair-wise classification accuracies between toxigenic AF13 and each atoxigenic fungus species (AF38, AF283 and AF2038) were 80%, 91% and 95% under halogen light sources, and 75%, 97% and 99% under UV lights, respectively.

로버스트추정에 바탕을 둔 주성분로지스틱회귀

로지스틱회귀분석은 고객관계관리를 위한 데이터마이닝 분야에서 많이 사용되는 기법인데, 이 분야의 모형설정 과정에서는 연관성이 매우 높은 설명변수들이 모형에 함께 포함되어 다중공선성의 문제를 유발하며, 더욱이 회귀자료에 이상점들이 포함되면 최우추정량은 심각한 결함을 갖게 된다. 두 가지 문제점을 동시에 해결하기 위하여 로버스트주성분로지스틱회귀를 적용할 수 있는데, 본 논문에서는 주성분의 선정기준을 결정하는 모형을 개발하고, 주성분모형에서의 추정치에 미치는 이상점의 영향을 축소하기 위한 로버스트추정법을 제안하였다. 제안된 추정법은 다중공선성과 이상점이 유발하는 문제들을 적절히 해결해 준다는 사실이 모의실험을 통하여 확인되었다. Logistic regression is widely used as a datamining technique for the customer relationship management. The maximum likelihood estimator has highly inflated variance when multicollinearity exists among the regressors, and it is not robust against outliers. Thus we propose the robust principal components logistic regression to deal with both multicollinearity and outlier problem. A procedure is suggested for the selection of principal components, which is based on the condition index. When a condition index is larger than the cutoff value obtained from the model constructed on the basis of the conjoint analysis, the corresponding principal component is removed from the logistic model. In addition, we employ an algorithm for the robust estimation, which strives to dampen the effect of outliers by applying the appropriate weights and factors to the leverage points and vertical outliers identified by the V-mask type criterion. The Monte Carlo simulation results indicate that the proposed procedure yields higher rate of correct classification than the existing method.

SubXPCA and a generalized feature partitioning approach to principal component analysis

In this paper we propose a general feature partitioning framework to PCA computation and raise issues of cross-sub-pattern correlation, feature ordering dependence, selection of sub-pattern size, overlap of sub-patterns and selection of principal components. These issues are critical to the design and performance of feature partitioning approaches to PCA computation. We show several open issues and present a novel algorithm, SubXPCA which proposes a solution to the cross-sub-pattern correlation issue in the feature partitioning framework. SubXPCA is shown to be a general technique since we derive PCA and SubPCA as special cases of SubXPCA. We show SubXPCA has theoretically better time complexity as compared to PCA. Comprehensive experimentation on UCI repository data and face data sets (ORL, CMU, Yale) confirms the superiority of SubXPCA with better classification accuracy. SubXPCA not only has better time performance but is also superior in its summarization of variance as compared to SubPCA. SubXPCA is shown to be robust in its performance with respect to feature ordering and overlapped sub-patterns.

GA-Fisher: A New LDA-Based Face Recognition Algorithm With Selection of Principal Components

This paper addresses the dimension reduction problem in Fisherface for face recognition. When the number of training samples is less than the image dimension (total number of pixels), the within-class scatter matrix (Sw) in Linear Discriminant Analysis (LDA) is singular, and Principal Component Analysis (PCA) is suggested to employ in Fisherface for dimension reduction of Sw so that it becomes nonsingular. The popular method is to select the largest nonzero eigenvalues and the corresponding eigenvectors for LDA. To attenuate the illumination effect, some researchers suggested removing the three eigenvectors with the largest eigenvalues and the performance is improved. However, as far as we know, there is no systematic way to determine which eigenvalues should be used. Along this line, this paper proposes a theorem to interpret why PCA can be used in LDA and an automatic and systematic method to select the eigenvectors to be used in LDA using a Genetic Algorithm (GA). A GA-PCA is then developed. It is found that some small eigenvectors should also be used as part of the basis for dimension reduction. Using the GA-PCA to reduce the dimension, a GA-Fisher method is designed and developed. Comparing with the traditional Fisherface method, the proposed GA-Fisher offers two additional advantages. First, optimal bases for dimensionality reduction are derived from GA-PCA. Second, the computational efficiency of LDA is improved by adding a whitening procedure after dimension reduction. The Face Recognition Technology (FERET) and Carnegie Mellon University Pose, Illumination, and Expression (CMU PIE) databases are used for evaluation. Experimental results show that almost 5 % improvement compared with Fisherface can be obtained, and the results are encouraging.

A Simple Rule for the Selection of Principal Components

A vast literature has been devoted to the assessment of the proper number of eigenvalues that have to be retained in Principal Components Analysis. Most of the publications are based on either distributional assumptions for the underlying populations or on empirical evident. In addition, techniques that are based on bootstrap or cross-validatory techniques have been proposed despite the computational effort implied. In this paper a simple technique based on a control chart approach is proposed for selecting the number of principal components to retain for the analysis. This approach accounts for the sampling variability which can lead to the selection of components that are not in fact statistically significant. The method is compared with other methods and is found to be superior regardless of the underlying distributional properties of the population as well as the existing structure. An illustrative example is provided.

On the optimal selection of principal components in QSPR studies

A heuristic method to sort principal components is analysed. The obtained arrangements are property dependent and it is demonstrated how the procedure is equivalent to the called Most Predictive Variable Method. As an application of the new algorithm, a Quantitative Structure–Property Relationships (QSPR) study is performed over the set of the 18 structural isomers of the octane molecule. The original molecular descriptors are obtained from a quantum similarity matrix related to the molecular family. The analysis is based on the use of linear models where distinct sets of principal components act as optimal descriptors for 6 physicochemical molecular properties. The proposed algorithm allows to determine sequences of the first Principal Components which are identified as forming the optimal descriptors set for each of the 6 studied properties. The benefits of the new approach are revealed when comparing the obtained results with classical ones arising from a standard principal component analysis study.

Alternative principal components regression procedures for dendrohydrologic reconstructions

Streamflow reconstruction using tree ring information (dendrohydrology) has traditionally used principal components analysis (PCA) and stepwise regression to form a transfer function. However, PCA has several procedural choices that may result in very different reconstructions. This study assesses the different procedures in PCA‐based regression and suggests alternative procedures for selection of variables and principal components. Cross‐validation statistics are presented as an alternative for independently testing and identifying the optimal model. The objective is to use these statistics as a measure of the model's performance to find a conceptually acceptable model with a low prediction error and the fewest number of variables. The results show that a parsimonious model with a low mean square error can be obtained by using strict rules for principal component selection and cross‐validation statistics. Additionally, the procedure suggested in this study results in a model that is physically consistent with the relationship between the predictand and the predictor. The alternative PCA‐based regression models presented here are applied to the reconstruction of the Upper Colorado River Basin streamflow and compared with results of a previous reconstruction using traditional procedures. The streamflow reconstruction proposed in this study shows more intense drought periods, which may influence the future allocation of water supply in the Colorado River Basin.

A decision rule for discarding principal components in regression

Criteria for the selection of principal components in regression are usually founded on one of two indicators of component effects: (i) Student's tests of the significance of the components or (ii) the magnitude of the eigenvalues of the explanatory variables correlation matrix. Deletion of a principal component can give a bias in the estimates (first criterion), but can also reduce unbiased residual variance (second criterion). In this article we propose a decision rule with respect to the partial correlation coefficient between the principal component and the explained variable whilst taking the previous criteria into account.

Comparison of Prediction- and Correlation-Based Methods to Select the Best Subset of Principal Components for Principal Component Regression and Detect Outlying Objects

In the present work the use of prediction and correlation criteria for the best subset selection of principal components for principal component regression is compared. Results for both methodologies are similar, and always equal to or better than those obtained by using top-down principal component regression. In this comparison, the prediction criterion is based on the use of leverage-corrected residuals. In addition, the plot of leave-one-out cross-validated residuals vs. leverage-corrected residuals for the selected model is also proposed as a new graphic tool to detect possible outliers. In a test of the different methodologies, three different data sets have been studied.

Genetic algorithm applied to the selection of principal components

The application of a genetic algorithm (GA) to the selection of principal components (PCs) is proposed as an efficient method to determine the optimal multivariate regression model. This stochastic method was compared with other deterministic methods such as: exhaustive search (here taken as a validation procedure), forward and backward-stepwise variable selection and correlation principal components regression (CPCR). It is shown that for the range of data sets used, the GA gives the same result as the those obtained by an exhaustive search and by CPCR whereas the stepwise procedures do not. These results also show that in order to build optimal predictive models using principal components regression (PCR) one needs to select the best subset of PCs rather than simply use those with the highest eigenvalues.

Comparison of multivariate methods based on latent vectors and methods based on wavelength selection for the analysis of near-infrared spectroscopic data

Comparison of several calibration methods (principal component regression (PCR), partial least-squares, multiple linear regression), with and without feature selection, applied on near-infrared spectroscopic data is presented for a pharmaceutical application. It is shown that PCR with selection of principal components instead of the usual top-down approach yields simpler and better models. As feature selection methods, selection of wavelengths correlated with concentration, with large covariance with concentration, with high loadings on the important principal components, and according to a method proposed by Brown, are considered. The presented results suggests that feature selection can improve multivariate calibration.

Chemometrics with undergraduates

Abstract Kalivas, J., 1992. Chemometrics with undergraduates. Chemometrics and Intelligent Laboratory Systems, 15:127–135. This paper reviews John Kalivas' work as an evolving chemometrician. Discussion concentrates on his most recent investigations with the generalized simulated annealing (GSA) optimization algorithm and library searches of spectrochromatograms. Generalized simulated annealing represents an optimization procedure for determining global optima of multidimensional response surfaces. Studies described encompass GSA selection of regression variables and principal components for principal component regression. Results are also presented using condition index evolving profiles for library searching spectrochromatograms.

A Significance Test for Principal Components Applied to a Cyclone Climatology

Abstract A technique is presented for selection of principal components for which the geophysical signal is greater than the level of noise. The level of noise is simulated by repeated sampling of principal components computed from a spatially and temporally uncorrected random process. By contrasting the application of principal components based upon the covariance matrix and correlation matrix for a given data set of cyclone frequencies, it is shown that the former is more suitable to fitting data and locating the individual variables that represent large variance in the record, while the latter is more suitable for resolving spatial oscillations such as the movement of primary storm tracks.