Probabilistic Canonical Correlation Analysis Research Articles

A Review of Statistical-Based Fault Detection and Diagnosis with Probabilistic Models

As industrial processes grow increasingly complex, fault identification becomes challenging, and even minor errors can significantly impact both productivity and system safety. Fault detection and diagnosis (FDD) has emerged as a crucial strategy for maintaining system reliability and safety through condition monitoring and abnormality recovery to manage this challenge. Statistical-based FDD methods that rely on large-scale process data and their features have been developed for detecting faults. This paper overviews recent investigations and developments in statistical-based FDD methods, focusing on probabilistic models. The theoretical background of these models is presented, including Bayesian learning and maximum likelihood. We then discuss various techniques and methodologies, e.g., probabilistic principal component analysis (PPCA), probabilistic partial least squares (PPLS), probabilistic independent component analysis (PICA), probabilistic canonical correlation analysis (PCCA), and probabilistic Fisher discriminant analysis (PFDA). Several test statistics are analyzed to evaluate the discussed methods. In industrial processes, these methods require complex matrix operation and cost computational load. Finally, we discuss the current challenges and future trends in FDD.

Kernel Probabilistic Dependent-Independent Canonical Correlation Analysis

There is growing interest in developing linear/nonlinear feature fusion methods that fuse the elicited features from two different sources of information for achieving a higher recognition rate. In this regard, canonical correlation analysis (CCA), cross-modal factor analysis, and probabilistic CCA (PCCA) have been introduced to better deal with data variability and uncertainty. In our previous research, we formerly developed the kernel version of PCCA (KPCCA) to capture both nonlinear and probabilistic relation between the features of two different source signals. However, KPCCA is only able to estimate latent variables, which are statistically correlated between the features of two independent modalities. To overcome this drawback, we propose a kernel version of the probabilistic dependent-independent CCA (PDICCA) method to capture the nonlinear relation between both dependent and independent latent variables. We have compared the proposed method to PDICCA, CCA, KCCA, cross-modal factor analysis (CFA), and kernel CFA methods over the eNTERFACE and RML datasets for audio-visual emotion recognition and the M2VTS dataset for audio-visual speech recognition. Empirical results on the three datasets indicate the superiority of both the PDICCA and Kernel PDICCA methods to their counterparts.

The Effect of Neuroepo on Cognition in Parkinson's Disease Patients Is Mediated by Electroencephalogram Source Activity.

We report on the quantitative electroencephalogram (qEEG) and cognitive effects of Neuroepo in Parkinson’s disease (PD) from a double-blind safety trial (https://clinicaltrials.gov/, number NCT04110678). Neuroepo is a new erythropoietin (EPO) formulation with a low sialic acid content with satisfactory results in animal models and tolerance in healthy participants and PD patients. In this study, 26 PD patients were assigned randomly to Neuroepo (n = 15) or placebo (n = 11) groups to test the tolerance of the drug. Outcome variables were neuropsychological tests and resting-state source qEEG at baseline and 6 months after administering the drug. Probabilistic Canonical Correlation Analysis was used to extract latent variables for the cognitive and for qEEG variables that shared a common source of variance. We obtained canonical variates for Cognition and qEEG with a correlation of 0.97. Linear Mixed Model analysis showed significant positive dependence of the canonical variate cognition on the dose and the confounder educational level (p = 0.003 and p = 0.02, respectively). Additionally, in the mediation equation, we found a positive dependence of Cognition with qEEG for (p = < 0.0001) and with dose (p = 0.006). Despite the small sample, both tests were powered over 89%. A combined mediation model showed that 66% of the total effect of the cognitive improvement was mediated by qEEG (p = 0.0001), with the remaining direct effect between dose and Cognition (p = 0.002), due to other causes. These results suggest that Neuroepo has a positive influence on Cognition in PD patients and that a large portion of this effect is mediated by brain mechanisms reflected in qEEG.

A Survey on Canonical Correlation Analysis

In recent years, the advances in data collection and statistical analysis promotes canonical correlation analysis (CCA) available for more advanced research. CCA is the main technique for two-set data dimensionality reduction such that the correlation between the pairwise variables in the common subspace is mutually maximized. Over 80-years of developments, a number of CCA models have been proposed according to different machine learning mechanisms. However, the field lacks an insightful review for the state-of-art developments. This survey targets to provide a well-organized overview for CCA and its extensions. Specifically, we first review the CCA theory from the perspective of both model formation and model optimization. The association between two popular solution methods, i.e., eigen value decomposition (EVD) and singular value decomposition (SVD), are discussed. Following that, we present a taxonomy of current progresses and classify them into seven groups: 1) multi-view CCA, 2) probabilistic CCA, 3) deep CCA, 4) kernel CCA, 5) discriminative CCA, 6) sparse CCA and 7) locality preserving CCA. For each group, we demonstrate two or three representative mathematical models, identifying their strengths and limitations. We summarize the representative applications and numerical results of these seven groups in real-world practices, collecting the data sets and open-sources for implementation. In the end, we provide several promising future research directions that can improve the current state of the art.

A whitening approach to probabilistic canonical correlation analysis for omics data integration

BackgroundCanonical correlation analysis (CCA) is a classic statistical tool for investigating complex multivariate data. Correspondingly, it has found many diverse applications, ranging from molecular biology and medicine to social science and finance. Intriguingly, despite the importance and pervasiveness of CCA, only recently a probabilistic understanding of CCA is developing, moving from an algorithmic to a model-based perspective and enabling its application to large-scale settings.ResultsHere, we revisit CCA from the perspective of statistical whitening of random variables and propose a simple yet flexible probabilistic model for CCA in the form of a two-layer latent variable generative model. The advantages of this variant of probabilistic CCA include non-ambiguity of the latent variables, provisions for negative canonical correlations, possibility of non-normal generative variables, as well as ease of interpretation on all levels of the model. In addition, we show that it lends itself to computationally efficient estimation in high-dimensional settings using regularized inference. We test our approach to CCA analysis in simulations and apply it to two omics data sets illustrating the integration of gene expression data, lipid concentrations and methylation levels.ConclusionsOur whitening approach to CCA provides a unifying perspective on CCA, linking together sphering procedures, multivariate regression and corresponding probabilistic generative models. Furthermore, we offer an efficient computer implementation in the “whitening” R package available at https://CRAN.R-project.org/package=whitening.

Bilinear Probabilistic Canonical Correlation Analysis via Hybrid Concatenations

Canonical Correlation Analysis (CCA) is a classical technique for two-view correlation analysis, while Probabilistic CCA (PCCA) provides a generative and more general viewpoint for this task. Recently, PCCA has been extended to bilinear cases for dealing with two-view matrices in order to preserve and exploit the matrix structures in PCCA. However, existing bilinear PCCAs impose restrictive model assumptions for matrix structure preservation, sacrificing generative correctness or model flexibility. To overcome these drawbacks, we propose BPCCA, a new bilinear extension of PCCA, by introducing a hybrid joint model. Our new model preserves matrix structures indirectly via hybrid vector-based and matrix-based concatenations. This enables BPCCA to gain more model flexibility in capturing two-view correlations and obtain close-form solutions in parameter estimation. Experimental results on two real-world applications demonstrate the superior performance of BPCCA over competing methods.

FF-SKPCCA: Kernel probabilistic canonical correlation analysis

Several information fusion methods are developed for increasing the recognition accuracy in multimodal systems. Canonical correlation analysis (CCA), cross-modal factor analysis (CFA) and their kernel versions are known as successful fusion techniques but they cannot digest the data variability. Probabilistic CCA (PCCA) is suggested as a linear fusion method to capture input variability. A new kernel PCCA (KPCCA) is proposed here to capture both the nonlinear correlations of sources and input variability. The functionality of KPCCA decreases when the number of samples, which determines the size of kernel matrix increases. In the conventional fusion methods the latent variables of different modalities are concatenated; consequently, a large-scale covariance matrix with just limited number of samples must be estimated To overcome this drawback, a sparse KPCCA (SKPCCA) is introduced which scarifies the covariance matrix elements at the cost of decreasing its rank. In the final stage of the gradual evolution of KPCCA, a new feature fusion manner is proposed for SKPCCA (FF-SKPCCA) as a second stage fusion. This proposed method unifies the latent variables of two modalities into a feature vector with an acceptable size. Audio-visual databases like M2VTS (for speech recognition) eNTERFACE and RML (for emotion recognition) are applied to assess FF-SKPCCA compared to state-of-the-art fusion methods. The comparative results indicate the superiority of the proposed method in most cases.

Mixture of Probabilistic Canonical Correlation Analysis

Mixture of Probabilistic Canonical Correlation Analysis

Dynamic Probabilistic CCA for Analysis of Affective Behavior and Fusion of Continuous Annotations.

Fusing multiple continuous expert annotations is a crucial problem in machine learning and computer vision, particularly when dealing with uncertain and subjective tasks related to affective behavior. Inspired by the concept of inferring shared and individual latent spaces in Probabilistic Canonical Correlation Analysis (PCCA), we propose a novel, generative model that discovers temporal dependencies on the shared/individual spaces (Dynamic Probabilistic CCA, DPCCA). In order to accommodate for temporal lags, which are prominent amongst continuous annotations, we further introduce a latent warping process, leading to the DPCCA with Time Warpings (DPCTW) model. Finally, we propose two supervised variants of DPCCA/DPCTW which incorporate inputs (i.e., visual or audio features), both in a generative (SG-DPCCA) and discriminative manner (SD-DPCCA). We show that the resulting family of models (i) can be used as a unifying framework for solving the problems of temporal alignment and fusion of multiple annotations in time, (ii) can automatically rank and filter annotations based on latent posteriors or other model statistics, and (iii) that by incorporating dynamics, modeling annotation-specific biases, noise estimation, time warping and supervision, DPCTW outperforms state-of-the-art methods for both the aggregation of multiple, yet imperfect expert annotations as well as the alignment of affective behavior.

Probabilistic canonical correlation analysis forecasts, with application to tropical Pacific sea‐surface temperatures

ABSTRACTCanonical correlation analysis (CCA) is a higher‐dimensional extension of univariate multiple regression that is often used to construct seasonal and other forecasts in a climatological context. Although its use is widespread, to date it has apparently been used only to produce nonprobabilistic forecasts. Here an analytic result for the prediction covariance matrix of vector CCA forecasts is presented, which is sufficient to define a full forecast probability distribution if a multivariate Gaussian distribution can reasonably be assumed for the forecast errors. The approach is illustrated by computing and verifying probabilistic seasonal forecasts for tropical Pacific sea‐surface temperatures.

Matrix-Variate Probabilistic Model for Canonical Correlation Analysis

Motivated by the fact that in computer vision data samples are matrices, in this paper, we propose a matrix-variate probabilistic model for canonical correlation analysis (CCA). Unlike probabilistic CCA which converts the image samples into the vectors, our method uses the original image matrices for data representation. We show that the maximum likelihood parameter estimation of the model leads to the two-dimensional canonical correlation directions. This model helps for better understanding of two-dimensional Canonical Correlation Analysis (2DCCA), and for further extending the method into more complex probabilistic model. In addition, we show that two-dimensional Linear Discriminant Analysis (2DLDA) can be obtained as a special case of 2DCCA.

Image Annotation and Retrieval for Weakly Labeled Images Using Conceptual Learning

One of the most promising new technologies for widespread application is image annotation and retrieval. Nevertheless, this task is very difficult to accomplish as target images differ significantly in appearance and belong to a wide variety of categories. In this paper, we propose a new image annotation and retrieval method for miscellaneous weakly labeled images, by combining higher-order local auto-correlation (HLAC) features and a framework of probabilistic canonical correlation analysis. The distance between images can be defined in the intrinsic space for annotation using conceptual learning of images and their labels. Because this intrinsic space is highly compressed compared to the image feature space, our method achieves both faster and more accurate image annotation and retrieval. The HLAC features are powerful global features with additive and position invariant properties. These properties work well with images, which have an arbitrary number of objects at arbitrary locations. The proposed method is shown to outperform existing methods using a standard benchmark dataset.

Probabilistic approach to detecting dependencies between data sets

We study data fusion under the assumption that data source-specific variation is irrelevant and only shared variation is relevant. Traditionally the shared variation has been sought by maximizing a dependency measure, such as correlation of linear projections in canonical correlation analysis (CCA). In this traditional framework it is hard to tackle overfitting and model order selection, and thus we turn to probabilistic generative modeling which makes all tools of Bayesian inference applicable. We introduce a family of probabilistic models for the same task, and present conditions under which they seek dependency. We show that probabilistic CCA is a special case of the model family, and derive a new dependency-seeking clustering algorithm as another example. The solution is computed with variational Bayes.

Variational Bayesian Approach to Canonical Correlation Analysis

As a dimension reduction algorithm, canonical correlation analysis (CCA) encounters the issue of selecting the number of canonical correlations. In this letter, we present a Bayesian model selection algorithm for CCA based on a probabilistic interpretation. A hierarchical Bayesian model is applied to probabilistic CCA and learned by variational approximation. This method not only estimates the model parameters, but also automatically determines the number of canonical correlations and avoids overfitting. Experiments show that it performs better compared with maximum likelihood and some other model selection methods.