Principal Component Analysis Subspace Research Articles

Multivariate curve resolution-soft independent modelling of class analogy (MCR-SIMCA)

BackgroundVarious classification, class modeling, and clustering techniques operate within abstract spaces, utilizing Principal Components (e.g., Linear Discriminant Analysis (LDA), Principal Component Analysis (PCA)) or latent variable spaces (e.g., Partial Least Squares Discriminant Analysis (PLS-DA)). It's important to note that PCA, despite being a mathematical tool, defines its Principal Components under certain mathematical constraints, it has a wide range of applications in the analysis of real-world systems. In this research, we assess the viability of employing the Multivariate Curve Resolution (MCR) subspace within class modeling techniques, as an alternative to the PC subspace. (92). ResultsThis study evaluates the use of the MCR subspace in class modeling methods, specifically in tandem with soft independent modeling of class analogy (SIMCA), to investigate the advantages of employing the meaningful physico-chemical subspace of MCR over the mathematical subspace of PCA. In the MCR-SIMCA strategy, the model is constructed by applying MCR to training samples from a target class. The MCR model effectively partitions the data into two smaller sub-matrices: the contribution matrix and the corresponding response matrix. In the next step, the contribution matrix resulting from the decomposition of the training set develops a distance plot (DP). First, the theory of the MCR-SIMCA model is discussed in detail. Next, two real experimental datasets were analyzed, and their performance was compared with the DD-SIMCA model. In most cases, the results were as good as or even more satisfactory than those obtained with the DD-SIMCA model. (146). SignificanceThe suggested class modeling method presents a promising avenue for the analysis of real-world natural systems. The study's results emphasize the practical utility of the MCR approach, underscoring the significance of the MCR subspace advantages over the PCA subspace. (39).

Detecting Under-Resolved Flow Physics Using Supervised Machine Learning

The accuracy of flow simulations is a major concern in computational fluid dynamics (CFD) applications. A possible outcome of inaccuracy in CFD results is missing a major feature in the flowfield. Many methods have been proposed to reduce numerical errors and increase overall accuracy, but these are not always efficient or even feasible. In this study, the principal component analysis (PCA) has been performed on compressible flow simulations around an airfoil to map the high-dimensional CFD data to a lower-dimensional PCA subspace. A machine learning classifier based on the extracted principal components has been developed to detect the simulations that miss the separation bubble behind the airfoil. The evaluative measures indicate that the model is able to detect most of the simulations where the separation region is poorly resolved. Moreover, a single mode responsible for the missing flow separation was uncovered that could be the subject of future studies. The results demonstrate that a machine learning model based on the principal components of the dataset is a promising tool for detecting possible missing flow features in CFD.

Evaluation of complementary numerical and visual approaches for investigating pairwise comparisons after principal component analysis

We propose and evaluate numerical and visual methods for investigating paired comparisons after principal component analysis (PCA). PCA results can be visualized to facilitate an understanding of the relationships between the products and the sensory attributes. But identifying and visualizing significant product differences in multiple PCs simultaneously is not straightforward. A benefit of the proposed methods is that they provide a screening tool for evaluating PCA results rapidly. We begin with a real data set which is analyzed and submitted to the truncated total bootstrap (TTB) procedure. This TTB procedure simulates and analyzes results from virtual panels. The TTB-derived results form clouds of uncertainty around each product and paired comparison. Although these clouds can be visualized directly or by plotting the smallest contours that enclose 95% of their kernel-estimated densities, we propose that plotting TTB-derived 95% confidence ellipsoids provide a less cumbersome approach. We show that it is also possible to calculate P values that evaluate whether pairs of products are discriminated in the PCA subspace. The interpretation of these P values coincides with the visual interpretation of the confidence ellipsoids. The volumes of these confidence ellipsoids, which quantify uncertainty, are calculated easily. The confidence ellipsoids, the P values, and the volumes provide a simple and consistent approach for investigating paired comparisons after PCA. We illustrate the methods with two real data sets, one a sensory quantitative-descriptive data set from a trained panel, the other a check-all-that-apply (CATA) data set from a consumer panel. We also conduct a simulation study based on each of these data sets. The results from these simulation studies show that under repetition, the 95% confidence ellipsoids often have coverage of approximately 95%, but in some cases, coverage can be substantially lower. This indicates that the proposed ellipsoids have an approximately frequentist interpretation, but coverage varies. The complementary numerical and visual approaches can be applied to a wide range of data sets from sensory evaluation and to data from other domains.

CX-DaGAN: Domain Adaptation for Pneumonia Diagnosis on a Small Chest X-Ray Dataset.

Recent advances in deep learning led to several algorithms for the accurate diagnosis of pneumonia from chest X-rays. However, these models require large training medical datasets, which are sparse, isolated, and generally private. Furthermore, these models in medical imaging are known to over-fit to a particular data domain source, i.e., these algorithms do not conserve the same accuracy when tested on a dataset from another medical center, mainly due to image distribution discrepancies. In this work, a domain adaptation and classification technique is proposed to overcome the over-fit challenges on a small dataset. This method uses a private-small dataset (target domain), a public-large labeled dataset from another medical center (source domain), and consists of three steps. First, it performs a data selection of the source domain's most representative images based on similarity constraints through principal component analysis subspaces. Second, the selected samples from the source domain are fit to the target distribution through an image to image translation based on a cycle-generative adversarial network. Finally, the target train dataset and the adapted images from the source dataset are used within a convolutional neural network to explore different settings to adjust the layers and perform the classification of the target test dataset. It is shown that fine-tuning a few specific layers together with the selected-adapted images increases the sorting accuracy while reducing the trainable parameters. The proposed approach achieved a notable increase in the target dataset's overall classification accuracy, reaching up to 97.78 % compared to 90.03 % by standard transfer learning.

DeepMNN: Deep Learning-Based Single-Cell RNA Sequencing Data Batch Correction Using Mutual Nearest Neighbors.

It is well recognized that batch effect in single-cell RNA sequencing (scRNA-seq) data remains a big challenge when integrating different datasets. Here, we proposed deepMNN, a novel deep learning-based method to correct batch effect in scRNA-seq data. We first searched mutual nearest neighbor (MNN) pairs across different batches in a principal component analysis (PCA) subspace. Subsequently, a batch correction network was constructed by stacking two residual blocks and further applied for the removal of batch effects. The loss function of deepMNN was defined as the sum of a batch loss and a weighted regularization loss. The batch loss was used to compute the distance between cells in MNN pairs in the PCA subspace, while the regularization loss was to make the output of the network similar to the input. The experiment results showed that deepMNN can successfully remove batch effects across datasets with identical cell types, datasets with non-identical cell types, datasets with multiple batches, and large-scale datasets as well. We compared the performance of deepMNN with state-of-the-art batch correction methods, including the widely used methods of Harmony, Scanorama, and Seurat V4 as well as the recently developed deep learning-based methods of MMD-ResNet and scGen. The results demonstrated that deepMNN achieved a better or comparable performance in terms of both qualitative analysis using uniform manifold approximation and projection (UMAP) plots and quantitative metrics such as batch and cell entropies, ARI F1 score, and ASW F1 score under various scenarios. Additionally, deepMNN allowed for integrating scRNA-seq datasets with multiple batches in one step. Furthermore, deepMNN ran much faster than the other methods for large-scale datasets. These characteristics of deepMNN made it have the potential to be a new choice for large-scale single-cell gene expression data analysis.

Robust Subspace Tracking Algorithms in Signal Processing: A Brief Survey

Principal component analysis (PCA) and subspace estimation (SE) are popular data analysis tools and used in a wide range of applications. The main interest in PCA/SE is for dimensionality reduction and low-rank approximation purposes. The emergence of big data streams have led to several essential issues for performing PCA/SE. Among them are (i) the size of such data streams increases over time, (ii) the underlying models may be time-dependent, and (iii) problem of dealing with the uncertainty and incompleteness in data. A robust variant of PCA/SE for such data streams, namely robust online PCA or robust subspace tracking (RST), has been introduced as a good alternative. The main goal of this paper is to provide a brief survey on recent RST algorithms in signal processing. Particularly, we begin this survey by introducing the basic ideas of the RST problem. Then, different aspects of RST are reviewed with respect to different kinds of non-Gaussian noises and sparse constraints. Our own contributions on this topic are also highlighted.

Incremental visual tracking via sparse discriminative classifier

Currently, visual object tracking is a core research area as it can be applied in many applications of computer vision. However, tracking of a visual object is a difficult task as it can go through different varying conditions like occlusion of the target object, appearance variation, illumination variation, etc. during the tracking process. An efficient and robust visual object tracking based on sparse discriminative classier (SDC) and principal component analysis (PCA) subspace representation is presented in this work. The PCA subspace representation modelled the appearance model of the target object and SDC separates the target object and background object very efficiently. The computational complexity is much better than the other existing methods in the literature. Both quantitative and qualitative analyses of different video sequences are done to compare the proposed tracking algorithm with the other existing tracking algorithms. The experimental results show that the proposed method outperforms the other existing tracking algorithms.

Incremental Kernel Principal Components Subspace Inference With Nyström Approximation for Bayesian Deep Learning

As the state-of-the-art technology of Bayesian inference, based on low-dimensional principal components analysis (PCA) subspace inference methods can provide approximately accurate predictive distribution and well calibrated uncertainty. However, the main problem of PCA method is that it is a linear subspace feature extractor, and it cannot effectively represent the nonlinearly high-dimensional parameter space of deep neural networks (DNNs). Firstly, in this paper, in order to solve the main problem of the linear characteristics of PCA in high-dimensional space, we apply kernel PCA to extract higher-order statistical information in parameter space of DNNs. Secondly, to improve the efficiency of subsequent computation, we propose a strictly ordered incremental kernel PCA (InKPCA) subspace of parameter space within stochastic gradient descent (SGD) trajectories. In the proposed InKPCA subspace, we employ two approximation inference methods: elliptical slice sampling (ESS) and variational inference (VI). Finally, to further improve the memory efficiency of computing the kernel matrix, we apply Nyström approximation to determine the suitable size of subsets in the original datasets. The novelty of this paper is that it is the first time to apply the proposed InKPCA subspace with Nyström approximation for Bayesian inference in DNNs, and the results show that it can produce more accurate predictions and well-calibrated predictive uncertainty in regression and classification tasks of deep learning.

Nested AdaBoost procedure for classification and multi-class nonlinear discriminant analysis

AdaBoost methods find an accurate classifier by combining moderate learners that can be computed using traditional techniques based, for instance, on separating hyperplanes. Recently, we proposed a strategy to compute each moderate learner using a linear ensemble of weak classifiers that are built through the kernel support vector machine (KSVM) hypersurface geometry. In this way, we apply AdaBoost procedure in a nested loop: Each iteration of the inner loop boosts weak classifiers to a moderate one while the outer loop combines the moderate classifiers to build the global decision rule. In this paper, we explore this methodology in two ways: (a) For classification in principal component analysis (PCA) spaces; (b) For multi-class nonlinear discriminant PCA, named MNDPCA. Up to the best of our knowledge, the former is a new AdaBoost-based classification technique. Besides, in this paper we study the influence of kernel types for MNDPCA in order to set a near optimum configuration for feature selection and ranking in PCA subspaces. We compare the proposed methodologies with counterpart ones using facial expressions of the Radboud Faces database and Karolinska Directed Emotional Faces (KDEF) image database. Our experimental results have shown that MNDPCA outperforms counterpart techniques for selecting PCA features in the Radboud database while it performs close to the best technique for KDEF images. Moreover, the proposed classifier achieves outstanding recognition rates if compared with the literature techniques.

Discriminative object tracking with subspace representation

Visual object tracking is a core research area in the field of pattern recognition and computer vision. It becomes one of the most significant tasks in computer vision application. But tracking of a visual object is not an easy task as it is always restricted by appearance change, illumination, occlusion and so on. Object tracking based on principal component analysis (PCA) is one of the most effective tracking methods as it can handle the different challenging problems of the tracking algorithm. But in this PCA-based tracking method, the background pixels are also included in the subspace representation of the target object, and so this method cannot overcome all the problems of tracking. In this work, a robust visual object tracking method is proposed by introducing sparse discriminative classifier (SDC) feature selection in PCA subspace representation. The SDC method is utilized to extract the target object from the template image target by removing the background pixels which is unnecessary for tracking task without much computational complexity. The PCA algorithm adequately represents a presentation model of the target object and account of occlusion with trivial template. Qualitative and quantitative analysis of different diverse videos shows that the newly proposed method outperforms the other existing state-of-the-art tracking algorithm.

Exploring bigram character features for Arabic text clustering

The vector space model (VSM) is an algebraic model that is widely used for data representation in text mining applications. However, the VSM poses a critical challenge, as it requires a high-dimensional feature space. Therefore, many feature selection techniques, such as employing roots or stems (i.e. words without infixes and prefixes, and/or suffixes) instead of using complete word forms, are proposed to tackle this space challenge problem. Recently, the literature shows that one more basic unit feature can be used to handle the textual features, which is the twoneighboring character form that we call microword. To evaluate this feature type, we measure the accuracy of the Arabic text clustering using two feature types: the complete word form and the microword form. Hence, the microword is two consecutive characters which are also known as the Bigram character feature. In the experiment, the principal component analysis (PCA) is used to reduce the feature vector dimensions while the k-means algorithm is used for the clustering purposes. The testing set includes 250 documents of five categories. The entire corpus contains 54,472 words, whereas the vocabulary contains 13,356 unique words. The experimental results show that the complete word form score accuracy is 97.2% while the two-character form score is 96.8%. In conclusion, the accuracies are almost the same; however, the two-character form uses a smaller vocabulary as well as less PCA subspaces. The study experiments might be a significant indication of the necessity to consider the Bigram character feature in the future text processing and natural language processing applications.

Incremental online object tracking via superpixel dimension reduction

Visual object tracking based on the principal component analysis (PCA) has been widely used in the last few decades as it deals with appearance changes caused by in-plane rotation, out-of-plane rotation, illumination variation and motion blur. But this method fails to handle partial occlusion problem. Also, it has computational complexity and speed problem when the size of image resolution is large. In this paper, a new method is proposed based on superpixel, PCA and sparse prototypes. The size of the image is reduced with preserving much of the visual information by superpixel segmentation. Then, the incremental PCA subspace representation is applied to model target appearance and account of occlusion with trivial templates. Both qualitative and quantitative analyses are done in many challenging image sequences. The proposed method gives a better performance than that of the other state-of-the-art algorithms.

Large Scale Dual Tree Complex Wavelet Transform based robust features in PCA and SVD subspace for digital image watermarking

A new watermarking scheme based on robust feature extraction using dual tree complex wavelet transform (DTCWT) in low dimensional subspace for grayscale imagesis proposed in this work. First the decomposition of host image is performed using DTCWT where large scale features having low frequency coefficients are extracted. The features are then further analyzed using the score matrix obtained from principal component analysis (PCA). Singular values from the score matrix are then obtained using the singular value decomposition (SVD) in a lower dimensional sub space. The watermark is subjected to similar processing after being scrambled using Arnold transform. The resultant robust lower dimensional DTCWT – PCA – SVD features are combined with the features extracted from the watermark. The robustness of the technique is evaluated by calculating quality assessment parameters. Further the robustness is evaluated according to the StirMark benchmarks by subjecting the signed image to various image processing attacks. High PSNR and correlation values as compared to state of art techniques indicate the imperceptibility and robustness of the proposed scheme.

LWS based PCA subspace ensemble model for soft sensor development

Most regression approaches, such as principal component analysis (PCA), are based on an assumption that the process data follow a Gaussian distribution. However, the process data usually dissatisfy that assumption. Thus, the locally weighted standardization (LWS) method is employed for transforming data into an approximate Gaussian distribution. Furthermore, the LWS based subspace PCA ensemble modeling method is developed. The subspace PCA can select important variables in each subspace for ensemble modeling. As a result, the proposed method gives a weaker assumption constrain and a better regression performance. The effectiveness of this approach is testified by two study cases.

Enhancing WO3 gas sensor selectivity using a set of pollutant detection classifiers

PurposeIn this paper, a novel hybrid approach aimed at solving the problem of cross-selectivity of gases in electronic nose (E-nose) using the combination classifiers of support vector machine (SVM) and k-nearest neighbors (KNN) methods was proposed.Design/methodology/approachFirst, three WO3 sensors E-nose system was used for data acquisition to detect three gases, namely, ozone, ethanol and acetone. Then, two transient parameters, derivate and integral, were extracted for each gas response. Next, the principal component analysis (PCA) was been applied to extract the most relevant sensor data and dimensionality reduction. The new coordinates calculated by PCA were used as inputs for classification by the SVM method. Finally, the classification achieved by the KNN method was carried out to calculate only the support vectors (SVs), not all the data.FindingsThis work has proved that the proposed fusion method led to the highest classification rate (100 per cent) compared to the accuracy of the individual classifiers: KNN, SVM-linear, SVM-RBF, SVM-polynomial that present, respectively, 89, 75.2, 80 and 79.9 per cent as classification rate.Originality/valueThe authors propose a fusion classifier approach to improve the classification rate. In this method, the extracted features are projected into the PCA subspace to reduce the dimensionality. Then, the obtained principal components are introduced to the SVM classifier and calculated SVs which will be used in the KNN method.

Collaborative PCA/DCA Learning Methods for Compressive Privacy

In the Internet era, the data being collected on consumers like us are growing exponentially, and attacks on our privacy are becoming a real threat. To better ensure our privacy, it is safer to let the data owner control the data to be uploaded to the network as opposed to taking chance with data servers or third parties. To this end, we propose compressive privacy , a privacy-preserving technique to enable the data creator to compress data via collaborative learning so that the compressed data uploaded onto the Internet will be useful only for the intended utility and not be easily diverted to malicious applications. For data in a high-dimensional feature vector space, a common approach to data compression is dimension reduction or, equivalently, subspace projection. The most prominent tool is principal component analysis (PCA). For unsupervised learning, PCA can best recover the original data given a specific reduced dimensionality. However, for the supervised learning environment, it is more effective to adopt a supervised PCA, known as discriminant component analysis (DCA), to maximize the discriminant capability. The DCA subspace analysis embraces two different subspaces. The signal-subspace components of DCA are associated with the discriminant distance/power (related to the classification effectiveness), whereas the noise subspace components of DCA are tightly coupled with recoverability and/or privacy protection. This article presents three DCA-related data compression methods useful for privacy-preserving applications: — Utility-driven DCA : Because the rank of the signal subspace is limited by the number of classes, DCA can effectively support classification using a relatively small dimensionality (i.e., high compression). — Desensitized PCA : By incorporating a signal-subspace ridge into DCA, it leads to a variant especially effective for extracting privacy-preserving components. In this case, the eigenvalues of the noise-space are made to become insensitive to the privacy labels and are ordered according to their corresponding component powers. — Desensitized K-means/SOM : Since the revelation of the K-means or SOM cluster structure could leak sensitive information, it is safer to perform K-means or SOM clustering on a desensitized PCA subspace.

A Comparison of Three Procedures for Robust PCA in High Dimensions

In this paper we compare three procedures for robust Principal Components Analysis (PCA). The first method is called ROBPCA (see Hubert et al., 2005). It combines projection pursuit ideas with robust covariance estimation. The original algorithm for its computation is designed to construct an optimal PCA subspace of a fixed dimension k. If instead the optimal PCA subspace is searched within a whole range of dimensions k, this algorithm is not computationally efficient. Hence we present an adjusted algorithm that yields several PCA models in one single run. A different approach is the LTS-subspace estimator (see Wolbers, 2002; Maronna, 2005). It seeks for the subspace that minimizes an objective function based on the squared orthogonal distances of the observations to this subspace. It can be computed in analogy with the computation of the LTS regression estimator (see Rousseeuw and Van Driessen, 2000). The three approaches are compared by means of a simulation study.

On the Nyström and Column-Sampling Methods for the Approximate Principal Components Analysis of Large Datasets

In this article, we analyze approximate methods for undertaking a principal components analysis (PCA) on large datasets. PCA is a classical dimension reduction method that involves the projection of the data onto the subspace spanned by the leading eigenvectors of the covariance matrix. This projection can be used either for exploratory purposes or as an input for further analysis, for example, regression. If the data have billions of entries or more, the computational and storage requirements for saving and manipulating the design matrix in fast memory are prohibitive. Recently, the Nyström and column-sampling methods have appeared in the numerical linear algebra community for the randomized approximation of the singular value decomposition of large matrices. However, their utility for statistical applications remains unclear. We compare these approximations theoretically by bounding the distance between the induced subspaces and the desired, but computationally infeasible, PCA subspace. Additionally we show empirically, through simulations and a real data example involving a corpus of emails, the trade-off of approximation accuracy and computational complexity.

Memory Efficient PCA Methods for Large Group ICA.

Principal component analysis (PCA) is widely used for data reduction in group independent component analysis (ICA) of fMRI data. Commonly, group-level PCA of temporally concatenated datasets is computed prior to ICA of the group principal components. This work focuses on reducing very high dimensional temporally concatenated datasets into its group PCA space. Existing randomized PCA methods can determine the PCA subspace with minimal memory requirements and, thus, are ideal for solving large PCA problems. Since the number of dataloads is not typically optimized, we extend one of these methods to compute PCA of very large datasets with a minimal number of dataloads. This method is coined multi power iteration (MPOWIT). The key idea behind MPOWIT is to estimate a subspace larger than the desired one, while checking for convergence of only the smaller subset of interest. The number of iterations is reduced considerably (as well as the number of dataloads), accelerating convergence without loss of accuracy. More importantly, in the proposed implementation of MPOWIT, the memory required for successful recovery of the group principal components becomes independent of the number of subjects analyzed. Highly efficient subsampled eigenvalue decomposition techniques are also introduced, furnishing excellent PCA subspace approximations that can be used for intelligent initialization of randomized methods such as MPOWIT. Together, these developments enable efficient estimation of accurate principal components, as we illustrate by solving a 1600-subject group-level PCA of fMRI with standard acquisition parameters, on a regular desktop computer with only 4 GB RAM, in just a few hours. MPOWIT is also highly scalable and could realistically solve group-level PCA of fMRI on thousands of subjects, or more, using standard hardware, limited only by time, not memory. Also, the MPOWIT algorithm is highly parallelizable, which would enable fast, distributed implementations ideal for big data analysis. Implications to other methods such as expectation maximization PCA (EM PCA) are also presented. Based on our results, general recommendations for efficient application of PCA methods are given according to problem size and available computational resources. MPOWIT and all other methods discussed here are implemented and readily available in the open source GIFT software.

Systematic Exploration of an Efficient Amino Acid Substitution Matrix: MIQS.

Amino acid sequence comparisons to find similarities between proteins are fundamental sequence information analyses for inferring protein structure and function. In this study, we improve amino acid substitution matrices to identify distantly related proteins. We systematically sampled and benchmarked substitution matrices generated from the principal component analysis (PCA) subspace based on a set of typical existing matrices. Based on the benchmark results, we identified a region of highly sensitive matrices in the PCA subspace using kernel density estimation (KDE). Using the PCA subspace, we were able to deduce a novel sensitive matrix, called MIQS, which shows better detection performance for detecting distantly related proteins than those of existing matrices. This approach to derive an efficient amino acid substitution matrix might influence many fields of protein sequence analysis. MIQS is available at http://csas.cbrc.jp/Ssearch/ .