Extension Of Principal Component Analysis Research Articles

Extensions of Some Statistical Concepts to the Complex Domain

This paper deals with the extension of principal component analysis, canonical correlation analysis, the Cramer–Rao inequality, and a few other statistical concepts in the real domain to the corresponding complex domain. Optimizations of Hermitian forms under a linear constraint, a bilinear form under Hermitian-form constraints, and similar maxima/minima problems in the complex domain are discussed. Some vector/matrix differential operators are developed to handle the above types of problems. These operators in the complex domain and the optimization problems in the complex domain are believed to be new and novel. These operators will also be useful in maximum likelihood estimation problems, which will be illustrated in the concluding remarks. Detailed steps are given in the derivations so that the methods are easily accessible to everyone.

Solar Radiation and Weather Analysis of Meteorological Satellite Data by Tensor Decomposition

In this study, the data obtained from meteorological satellites were analyzed using tensor decomposition. The data used in this paper are meteorological image data observed by the Himawari-8 satellite and solar radiation data generated from Himawari Standard Data. First, we applied Higher-Order Singular Value Decomposition (HOSVD), a type of tensor decomposition, to the original image data and analyzed the features of the data, called the core tensor, obtained from the decomposition. As a result, it was found that the maximum value of the core tensor element is related to the cloud cover in the observed area. We then applied Multidimensional Principal Component Analysis (MPCA), an extension of principal component analysis computed using HOSVD, to the solar radiation data and analyzed the Principal Components (PC) obtained from MPCA. We also found that the PC with the highest contribution rate is related to the solar radiation in the entire observation area. The resulting PC score was compared to actual weather data. From the result, it was confirmed that the temporal transition of the amount of solar radiation in this area can be expressed almost correctly by using the PC score.

Investigating Machine Learning and Control Theory Approaches for Process Fault Detection: A Comparative Study of KPCA and the Observer-Based Method.

The paper focuses on the importance of prompt and efficient process fault detection in contemporary manufacturing industries, where product quality and safety protocols are critical. The study compares the efficiencies of two techniques for process fault detection: Kernel Principal Component Analysis (KPCA) and the observer method. Both techniques are applied to observe water volume variation within a hydraulic system comprising three tanks. PCA is an unsupervised learning technique used for dimensionality reduction and pattern recognition. It is an extension of Principal Component Analysis (PCA) that utilizes kernel functions to transform data into higher-dimensional spaces, where it becomes easier to separate classes or identify patterns. In this paper, KPCA is applied to detect faults in the hydraulic system by analyzing the variation in water volume. The observer method originates from control theory and is utilized to estimate the internal states of a system based on its output measurements. It is commonly used in control systems to estimate the unmeasurable or hidden states of a system, which is crucial for ensuring proper control and fault detection. In this study, the observer method is applied to the hydraulic system to estimate the water volume variations within the three tanks. The paper presents a comparative study of these two techniques applied to the hydraulic system. The results show that both KPCA and the observer method perform similarly in detecting faults within the system. This similarity in performance highlights the efficacy of these techniques and their potential adaptability in various fault diagnosis scenarios within modern manufacturing processes.

Domain Adaptation Principal Component Analysis: Base Linear Method for Learning with Out-of-Distribution Data

Domain adaptation is a popular paradigm in modern machine learning which aims at tackling the problem of divergence (or shift) between the labeled training and validation datasets (source domain) and a potentially large unlabeled dataset (target domain). The task is to embed both datasets into a common space in which the source dataset is informative for training while the divergence between source and target is minimized. The most popular domain adaptation solutions are based on training neural networks that combine classification and adversarial learning modules, frequently making them both data-hungry and difficult to train. We present a method called Domain Adaptation Principal Component Analysis (DAPCA) that identifies a linear reduced data representation useful for solving the domain adaptation task. DAPCA algorithm introduces positive and negative weights between pairs of data points, and generalizes the supervised extension of principal component analysis. DAPCA is an iterative algorithm that solves a simple quadratic optimization problem at each iteration. The convergence of the algorithm is guaranteed, and the number of iterations is small in practice. We validate the suggested algorithm on previously proposed benchmarks for solving the domain adaptation task. We also show the benefit of using DAPCA in analyzing single-cell omics datasets in biomedical applications. Overall, DAPCA can serve as a practical preprocessing step in many machine learning applications leading to reduced dataset representations, taking into account possible divergence between source and target domains.

Principal Component Analysis for Gaussian Process Posteriors.

This letter proposes an extension of principal component analysis for gaussian process (GP) posteriors, denoted by GP-PCA. Since GP-PCA estimates a low-dimensional space of GP posteriors, it can be used for metalearning, a framework for improving the performance of target tasks by estimating a structure of a set of tasks. The issue is how to define a structure of a set of GPs with an infinite-dimensional parameter, such as coordinate system and a divergence. In this study, we reduce the infiniteness of GP to the finite-dimensional case under the information geometrical framework by considering a space of GP posteriors that have the same prior. In addition, we propose an approximation method of GP-PCA based on variational inference and demonstrate the effectiveness of GP-PCA as meta-learning through experiments.

Affective Dynamics: Principal Motion Analysis of Temporal Dominance of Sensations and Emotions Data

Temporal dominance (TD) methods can be used to record temporal changes in multiple sensory and affective responses. TD methods have found wide applications in the analysis of eating experiences of humans. However, extant analyses performed on TD data do not fully utilize the time-series properties of such data. The present study validates the prospect of principal motion analysis (PMA) of TD data. PMA is an extension of principal component analysis, and can be used to resolve multivariate motion data into base principal motions. In this study, panelists were asked to evaluate the tastes of ten types of pickled plums using the temporal dominance of sensations (TDS) and emotions (TDE) methods. Additionally, the panelists were asked to rate the plums using the semantic differential method. Results obtained using both methods were observed to demonstrate good agreement with each other in terms of the structures of reduced variable spaces. As realized in this article, implementation of the combined TD–PMA approach can potentially facilitate statistical discrimination of all food products, whereas conventional methods, such as principal component analysis of data provided via use of the semantic differential method, can at best discriminate only 67 percent of product pairs. PMA can, therefore, be considered as a suitable technique to reveal the characteristics of TD data.

Moving dynamic principal component analysis for non-stationary multivariate time series

This paper proposes an extension of principal component analysis to non-stationary multivariate time series data. A criterion for determining the number of final retained components is proposed. An advance correlation matrix is developed to evaluate dynamic relationships among the chosen components. The theoretical properties of the proposed method are given. Many simulation experiments show our approach performs well on both stationary and non-stationary data. Real data examples are also presented as illustrations. We develop four packages using the statistical software R that contain the needed functions to obtain and assess the results of the proposed method.

Deep learning of multibody minimal coordinates

AbstractMultibody systems [1] are the state‐of‐the‐art tool to model complex mechanical mechanisms. However, they typically include redundant coordinates plus constraints, leading to differential algebraic equations for the dynamics which require dedicated integration schemes and control/estimation algorithms.In the deep learning field, autoencoders are a particular neural network class with a symmetric structure to perform a nonlinear dimensionality reduction. In fact, they can be seen as a non‐linear extension of principal component analysis [2].In this work, it is proposed to use an autoencoder to reduce a general multibody model to minimal coordinates, eliminating the constraints. The obtained decoder function is used to perform a non‐linear projection of the multibody coordinates: urn:x-wiley:16177061:media:PAMM202000348:pamm202000348-math-0001 where are respectively the encoder and decoder neural network functions, while are respectively the obtained minimal coordinates and reconstructed full multibody coordinates.The novelty is that the autoencoder training includes the multibody physics information [3]. In this way, the autoencoder does not only perform a dimensionality reduction of the original coordinates but can be used for model order reduction. The obtained reduced‐order model consists of ordinary differential equations (with respect to the original differential algebraic equations) and can be used to perform dynamic simulations and easily coupled with standard estimation algorithms for state‐estimation.A typical application example is a mechanism with closed‐loops, such as a car suspension, for which no analytical expression is available to reduce the full multibody model to minimal coordinates. The introduced physics‐informed neural network allows to reduce the model and combine it with an augmented extended Kalman filtering scheme [4], in order to estimate difficult‐to‐measure inputs or parameters in the system.In a first application, the model in combination with measured strut displacement or acceleration, allows to estimate the strut force, in addition to the body full coordinates in position, velocity and acceleration. The rationale is reported in the figure below.rationale.pngThe Flanders Innovation & Entrepreneurship Agency within the IMPROVED project and the AI impulse program of the Flemish Government are gratefully acknowledged for their support.

Principal Motion Ellipsoids: Gait Variability Index Invariant With Gait Speed

The walking motion of an individual involves considerable variability. We develop a gait variability index that determines how and to what degree repeated human gait motions vary, based on generalized principal motion analysis (GPMA). Principal motion analysis (PMA) is an extension of principal component analysis and decomposes multivariate time-series data, such as human joint angles during walking, into a linear combination of several principal motion bases. The developed gait variability index is defined by the size of an ellipsoid referred to as a PM ellipsoid and approximates the distribution of repeated gait motions on the motion base space. We expand the method to compute PM ellipsoids using GPMA, which enables us to mask the effects of a certain factor affecting gait motion observed under multiple factors. We compute the principal motion bases invariant with the gait speed from the gait motions of nine participants at two speed levels, 3.5 and 4.0 km/h. The sizes of the PM ellipsoids computed using GPMA do not depend on the gait speed and exhibit good agreement with the MeanSD, i.e., a typical gait variability index with correlation coefficients greater than 0.87.

Probabilistic Rank-One Tensor Analysis With Concurrent Regularizations.

Subspace learning for tensors attracts increasing interest in recent years, leading to the development of multilinear extensions of principal component analysis (PCA) and probabilistic PCA (PPCA). Existing multilinear PPCAs are based on the Tucker or CANDECOMP/PARAFAC (CP) models. Although both kinds of multilinear PPCAs have shown their effectiveness in dealing with tensors, they also have their own limitations. Tucker-based multilinear PPCAs have a restrictive subspace representation and suffer from rotational ambiguity, while CP-based ones are more prone to overfitting. To address these problems, we propose probabilistic rank-one tensor analysis (PROTA), a CP-based multilinear PPCA. PROTA has a more flexible subspace representation than Tucker-based PPCAs, and avoids rotational ambiguity. To alleviate overfitting for CP-based PPCAs, we propose two simple and effective regularization strategies, named as concurrent regularizations (CRs). By adjusting the noise variance or the moments of latent features, our strategies concurrently and coherently penalize the entire subspace. This relaxes unnecessary scale restrictions and gains more flexibility in regularizing CP-based PPCAs. To take full advantage of the probabilistic framework, we further propose a Bayesian treatment of PROTA, which achieves both automatic feature determination and robustness against overfitting. Experiments on synthetic and real-world datasets demonstrate the superiority of PROTA in subspace estimation and classification, as well as the effectiveness of CRs in alleviating overfitting.

2D Dimensionality Reduction Methods without Loss

In this paper, several two-dimensional extensions of principal component analysis (PCA) and linear discriminant analysis (LDA) techniques has been applied in a lossless dimensionality reduction framework, for face recognition application. In this framework, the benefits of dimensionality reduction were used to improve the performance of its predictive model, which was a support vector machine (SVM) classifier. At the same time, the loss of the useful information was minimized using the projection penalty idea. The well-known face databases were used to train and evaluate the proposed methods. The experimental results indicated that the proposed methods had a higher average classification accuracy in general compared to the classification based on Euclidean distance, and also compared to the methods which first extracted features based on dimensionality reduction technics, and then used SVM classifier as the predictive model.

Unsupervised classification of multichannel profile data using PCA: An application to an emission control system

Abstract Modern sensing technologies have facilitated real-time data collection for process monitoring and fault diagnosis in several research fields of industrial engineering. The challenges associated with diagnosis of multichannel (multiple) profiles are yet to be addressed in the literature. Motivated by an application of fault diagnosis of an emission control system, this paper proposes an approach for efficient and interpretable modeling of multichannel profile data in high-dimensional spaces. The method is based on unsupervised classification of multichannel profile data provided by several sensors related to a fault event. The final goal is to isolate fault events in a restricted number of clusters (scenarios), each one described by a reference pattern. This can provide practitioners with useful information to support the diagnosis and to find root cause. Two multilinear extensions of principal component analysis (PCA), which can analyze the multichannel profiles without unfolding the original data set, are investigated and compared to regular PCA applied to vectors generated by unfolding the original data set. The effectiveness of multilinear extensions of PCA is demonstrated using an experimental campaign carried out on an emission control system. Results of unsupervised classification show that the multilinear extension of PCA may lead to a classification with better compactness and separation of clusters.

Prediction of activation patterns preceding hallucinations in patients with schizophrenia using machine learning with structured sparsity.

Despite significant progress in the field, the detection of fMRI signal changes during hallucinatory events remains difficult and time-consuming. This article first proposes a machine-learning algorithm to automatically identify resting-state fMRI periods that precede hallucinations versus periods that do not. When applied to whole-brain fMRI data, state-of-the-art classification methods, such as support vector machines (SVM), yield dense solutions that are difficult to interpret. We proposed to extend the existing sparse classification methods by taking the spatial structure of brain images into account with structured sparsity using the total variation penalty. Based on this approach, we obtained reliable classifying performances associated with interpretable predictive patterns, composed of two clearly identifiable clusters in speech-related brain regions. The variation in transition-to-hallucination functional patterns not only from one patient to another but also from one occurrence to the next (e.g., also depending on the sensory modalities involved) appeared to be the major difficulty when developing effective classifiers. Consequently, second, this article aimed to characterize the variability within the prehallucination patterns using an extension of principal component analysis with spatial constraints. The principal components (PCs) and the associated basis patterns shed light on the intrinsic structures of the variability present in the dataset. Such results are promising in the scope of innovative fMRI-guided therapy for drug-resistant hallucinations, such as fMRI-based neurofeedback.

Online reduced kernel principal component analysis for process monitoring

Kernel principal component analysis (KPCA), which is a nonlinear extension of principal component analysis (PCA), has gained significant attention as a monitoring method for nonlinear processes. However, KPCA cannot perform well for dynamic systems and when the training data set is large. Therefore, in this paper, an online reduced KPCA algorithm for process monitoring is proposed. The process monitoring performances are studied using two examples: a numerical example and Tennessee Eastman Process (TEP). The simulation results demonstrate the effectiveness of the proposed method when compared to the online KPCA method.

How Germany benefits the most from its Eurozone membership

Post-institutionalization, the Eurozone has seen a marked increase in the integration of its financial markets. An important outcome of this has been the elimination of exchange rate risk because of the introduction of a common currency. We study the effect of this integration through a discrete-time affine term structure model with state variables obtained from the application of two extensions of principal components analysis to the multi-country setting by focusing on seven members of the Eurozone. Our results, derived from formulating and testing three hypotheses, show that Germany has gained disparately larger benefits from its membership in the Eurozone.

The utility of empirically assigning ancestry groups in cross-population genetic studies of addiction.

Given moderate heritability and significant heterogeneity among addiction phenotypes, successful genome-wide association studies (GWAS) are expected to need very large samples. As sample sizes grow, so can genetic diversity leading to challenges in analyzing these data. Methods for empirically assigning individuals to genetically informed ancestry groups are needed. We describe a strategy for empirically assigning ancestry groups in ethnically diverse GWAS data including extensions of principal component analysis (PCA) and population matching through minimum Mahalanobis distance. We apply these methods to data from Spit for Science (S4S): the University Student Survey, a study following college students longitudinally that includes genetic and environmental data on substance use and mental health (n = 7,603). The genetic-based population assignments for S4S were 48.7% European, 22.5% African, 10.4% Americas, 9.2% East Asian, and 9.2% South Asian descent. Self-reported census categories "More than one race" and "Unknown"as well as "Hawaiian/Pacific Islander" and "American-Indian/Native Alaskan" were empirically assigned representing a +9% sample retention over conventional methods. Although there was high concordance between self-reported race and empirical population-match (+.924), there was reduction in variance for most ancestry PCs for genetic-based population assignments. We were able to create more genetically homogenous groups and reduce sample and marker loss through cross-ancestry meta-analysis, potentially increasing power to detect etiologically relevant variation. Our approach provides a framework for empirically assigning genetic ancestry groups which can be applied to other ethnically diverse genetic studies. Given the important public health impact and demonstrable gains in statistical power from studying diverse populations, empirically sound practices for genetic studies are needed. (Am J Addict 2017;26:494-501).

Distinct Variation Pattern Discovery Using Alternating Nonlinear Principal Component Analysis.

Autoassociative neural networks (ANNs) have been proposed as a nonlinear extension of principal component analysis (PCA), which is commonly used to identify linear variation patterns in high-dimensional data. While principal component scores represent uncorrelated features, standard backpropagation methods for training ANNs provide no guarantee of producing distinct features, which is important for interpretability and for discovering the nature of the variation patterns in the data. Here, we present an alternating nonlinear PCA method, which encourages learning of distinct features in ANNs. A new measure motivated by the condition of orthogonal loadings in PCA is proposed for measuring the extent to which the nonlinear principal components represent distinct variation patterns. We demonstrate the effectiveness of our method using a simulated point cloud data set as well as a subset of the MNIST handwritten digits data. The results show that standard ANNs consistently mix the true variation sources in the low-dimensional representation learned by the model, whereas our alternating method produces solutions where the patterns are better separated in the low-dimensional space.

A new fault detection method for nonlinear process monitoring

Kernel Principal Component Analysis (KPCA) is a nonlinear extension of Principal Component Analysis (PCA). Recently, it is the most popular technique for monitoring nonlinear processes. However, the time-varying property of the industrial processes requires the adaptive ability of the KPCA. Therefore, in this paper, a Variable Moving Window Kernel PCA (VMWKPCA) method is proposed to update the KPCA model. The concept of this method consists of varying the size of the moving window according to the change of the normal process. To evaluate the performance of the proposed method, the VMWKPCA is applied for monitoring a Continuous Stirred Tank Reactor (CSTR) and a Tennessee Eastman process (TE). The results are satisfactory compared to the conventional Moving Window Kernel PCA (MWKPCA) and the Adaptive Kernel PCA (AKPCA).

Reconstructing the micrometeorological dynamics of the southern Amazonian transitional forest.

In this work, we reconstruct and analyze the micrometeorological dynamics of the transitional forest located south of the Amazon basin. For this, we use time series of micrometeorological variables collected over five years in the transitional forest of Mato Grosso (Brazil). We employ local feature analysis, a recently proposed extension of principal component analysis, to extract the most relevant physical variables from this set. We show in this way that temperature records contain most of the dynamical information in all seasons. Based on this result, the dimensionality of the space spanned by the system's dynamics and the properties of the so defined attractors are obtained. In the dry season, the system presents a robust oscillatory character described by a well-defined limit cycle. In the wet season, the dynamics becomes more irregular but can still be seen as a periodic behavior affected by external noise. These results can help to develop accurate models for the meteorology of the Amazonian transitional forest and can thus lead to a better understanding of this important ecosystem.

Principal Curves on Riemannian Manifolds.

Euclidean statistics are often generalized to Riemannian manifolds by replacing straight-line interpolations with geodesic ones. While these Riemannian models are familiar-looking, they are restricted by the inflexibility of geodesics, and they rely on constructions which are optimal only in Euclidean domains. We consider extensions of Principal Component Analysis (PCA) to Riemannian manifolds. Classic Riemannian approaches seek a geodesic curve passing through the mean that optimizes a criteria of interest. The requirements that the solution both is geodesic and must pass through the mean tend to imply that the methods only work well when the manifold is mostly flat within the support of the generating distribution. We argue that instead of generalizing linear Euclidean models, it is more fruitful to generalize non-linear Euclidean models. Specifically, we extend the classic Principal Curves from Hastie & Stuetzle to data residing on a complete Riemannian manifold. We show that for elliptical distributions in the tangent of spaces of constant curvature, the standard principal geodesic is a principal curve. The proposed model is simple to compute and avoids many of the pitfalls of traditional geodesic approaches. We empirically demonstrate the effectiveness of the Riemannian principal curves on several manifolds and datasets.