Generalized Canonical Correlation Analysis Research Articles

Deep Multi-Modal Discriminative and Interpretability Network for Alzheimer's Disease Diagnosis.

Multi-modal fusion has become an important data analysis technology in Alzheimer's disease (AD) diagnosis, which is committed to effectively extract and utilize complementary information among different modalities. However, most of the existing fusion methods focus on pursuing common feature representation by transformation, and ignore discriminative structural information among samples. In addition, most fusion methods use high-order feature extraction, such as deep neural network, by which it is difficult to identify biomarkers. In this paper, we propose a novel method named deep multi-modal discriminative and interpretability network (DMDIN), which aligns samples in a discriminative common space and provides a new approach to identify significant brain regions (ROIs) in AD diagnosis. Specifically, we reconstruct each modality with a hierarchical representation through multilayer perceptron (MLP), and take advantage of the shared self-expression coefficients constrained by diagonal blocks to embed the structural information of inter-class and the intra-class. Further, the generalized canonical correlation analysis (GCCA) is adopted as a correlation constraint to generate a discriminative common space, in which samples of the same category gather while samples of different categories stay away. Finally, in order to enhance the interpretability of the deep learning model, we utilize knowledge distillation to reproduce coordinated representations and capture influence of brain regions in AD classification. Experiments show that the proposed method performs better than several state-of-the-art methods in AD diagnosis.

Communication-Efficient Federated Linear and Deep Generalized Canonical Correlation Analysis

Classic and deep learning-based generalized canonical correlation analysis (GCCA) algorithms seek low-dimensional common representations of data entities from multiple “views” (e.g., audio and image) using linear transformations and neural networks, respectively. When the views are acquired and stored at different computing agents (e.g., organizations and edge devices) and data sharing is undesired due to privacy or communication cost considerations, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">federated learning</i> -based GCCA is well-motivated. In federated learning, the views are kept locally at the agents and only derived, limited information exchange with a central server is allowed. However, applying existing GCCA algorithms onto such federated learning setting may incur prohibitively high communication overhead. This work puts forth a communication-efficient federated learning framework for both linear and deep GCCA under the maximum variance (MAX-VAR) formulation. The overhead issue is addressed by aggressively compressing (via quantization) the exchanging information between the computing agents and a central controller. Compared to the unquantized version, our empirical study shows that the proposed algorithm enjoys a substantial reduction of communication overheads with virtually no loss in accuracy and convergence speed. Rigorous convergence analyses are also presented, which is a nontrivial effort. Generic federated optimization results do not cover the special problem structure of GCCA, which is a manifold constrained multi-block nonconvex eigen problem. Our result shows that the proposed algorithms for both linear and deep GCCA converge to critical points in a sublinear rate, even under heavy quantization and stochastic approximations. In addition, in the linear MAX-VAR case, the quantized algorithm approaches a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">global optimum</i> in a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">geometric</i> rate under reasonable conditions. Synthetic and real-data experiments are used to showcase the effectiveness of the proposed approach.

The Generalized Multiple CO-inertia Analysis (GMCOA)

The Multiple CO-inertia Analysis (MCOA) is a method which makes it possible simultaneously to analyze multiple tables of a multiblock having the same lines (individuals). It stipulates the existence of the auxiliary variables as for the generalized canonical correlation analysis (GCCA). Its numerical stability, its easiness to use and its good properties, make of it a good alternative to the GCCA, which is numerically unstable or not easily interpretable. In this article, we propose a generalization of this method in several multiblocks having the same lines, named: Generalized Multiple CO-inertia Analysis (GMCOA). Properties of this method are given. This method is applied to the sensory analysis.

Dynamic guided metric representation learning for multi-view clustering.

Multi-view clustering (MVC) is a mainstream task that aims to divide objects into meaningful groups from different perspectives. The quality of data representation is the key issue in MVC. A comprehensive meaningful data representation should be with the discriminant characteristics in a single view and the correlation of multiple views. Considering this, a novel framework called Dynamic Guided Metric Representation Learning for Multi-View Clustering (DGMRL-MVC) is proposed in this paper, which can cluster multi-view data in a learned latent discriminated embedding space. Specifically, in the framework, the data representation can be enhanced by multi-steps. Firstly, the class separability is enforced with Fisher Discriminant Analysis (FDA) within each single view, while the consistence among different views is enhanced based on Hilbert-Schmidt independence criteria (HSIC). Then, the 1st enhanced representation is obtained. In the second step, a dynamic routing mechanism is introduced, in which the location or direction information is added to fulfil the expression. After that, a generalized canonical correlation analysis (GCCA) model is used to get the final ultimate common discriminated representation. The learned fusion representation can substantially improve multi-view clustering performance. Experiments validated the effectiveness of the proposed method for clustering tasks.

Learning Low-Dimensional Semantics for Music and Language via Multi-Subject fMRI.

Embodied Cognition (EC) states that semantics is encoded in the brain as firing patterns of neural circuits, which are learned according to the statistical structure of human multimodal experience. However, each human brain is idiosyncratically biased, according to its subjective experience, making this biological semantic machinery noisy with respect to semantics inherent to media, such as music and language. We propose to represent media semantics using low-dimensional vector embeddings by jointly modeling the functional Magnetic Resonance Imaging (fMRI) activity of several brains via Generalized Canonical Correlation Analysis (GCCA). We evaluate the semantic richness of the resulting latent space in appropriate semantic classification tasks: music genres and language topics. We show that the resulting unsupervised representations outperform the original high-dimensional fMRI voxel spaces in these downstream tasks while being more computationally efficient. Furthermore, we show that joint modeling of several subjects increases the semantic richness of the learned latent vector spaces as the number of subjects increases. Quantitative results and corresponding statistical significance testing demonstrate the instantiation of music and language semantics in the brain, thereby providing further evidence for multimodal embodied cognition as well as a method for extraction of media semantics from multi-subject brain dynamics.

An Efficient Newton-Based Method for Sparse Generalized Canonical Correlation Analysis

Generalized canonical correlation analysis (GCCA) that aims to deal with multi-view data has attracted extensive attention in signal processing. To improve the representation performance, this paper proposes a new sparsity constrained GCCA (SCGCCA). Technically, it integrates the L2,0-norm constrained optimization into GCCA, which has not been investigated in the literature. Compared with the existing L2,1-norm regularized GCCA, the proposed SCGCCA can not only exploit the similarity information belonging to the same features but also determine the number of extracted features. Although it is a nonconvex minimization problem, an efficient alternating minimization algorithm can be designed. Furthermore, a Newton hard thresholding pursuit technique is developed to accelerate the convergence tremendously. Empirical studies suggest both the effectiveness and efficiency of the proposed SCGCCA comparing with the existing GCCA and its variants. In particular, the speed can be increased by 150 times for the simulated dataset.

Cell-Edge Detection via Selective Cooperation and Generalized Canonical Correlation

Improving the uplink quality of service for users located around the boundaries between cells is a key challenge in cellular systems. Existing approaches relying on power control throttle the rates of cell-center users, while multi-user detection requires accurate channel estimates for the cell-edge users, which is another challenge due to their low received signal-to-noise ratio (SNR). Utilizing the fact that cell-edge user signals are weak but common (received at roughly equal power) at different base stations (BSs), this paper establishes a connection between cell-edge user detection and generalized canonical correlation analysis (GCCA). It puts forth a GCCA-based method that leverages selective BS cooperation to recover the cell-edge user signal subspace even at low SNR. The cell-edge user signals can then be extracted from the resulting mixture via algebraic signal processing techniques. The paper includes theoretical analysis showing why GCCA recovers the correct subspace containing the cell-edge user signals under mild conditions. The proposed method can also identify the number of cell-edge users in the system, i.e., the common subspace dimension. Simulations reveal significant performance improvement relative to various multiuser detection techniques. Cell-edge detection performance is further studied as a function of how many / which BSs are selected, and it is shown that using the closest three BS is always the best choice.

Generalized Canonical Correlation Analysis: A Subspace Intersection Approach

Generalized Canonical Correlation Analysis (GCCA) is an important tool that finds numerous applications in data mining, machine learning, and artificial intelligence. It aims at finding `common' random variables that are strongly correlated across multiple feature representations (views) of the same set of entities. CCA and to a lesser extent GCCA have been studied from the statistical and algorithmic points of view, but not as much from the standpoint of linear algebra. This paper offers a fresh algebraic perspective of GCCA based on a (bi-)linear generative model that naturally captures its essence. It is shown that from a linear algebra point of view, GCCA is tantamount to subspace intersection; and conditions under which the common subspace of the different views is identifiable are provided. A novel GCCA algorithm is proposed based on subspace intersection, which scales up to handle large GCCA tasks. Synthetic as well as real data experiments are provided to showcase the effectiveness of the proposed approach.

Generalized canonical correlation analysis for labeled data

Multi-view learning is a method that is used to extract standard features from different information sources in various fields such as medical data analysis, computer vision, and web data analysis. Canonical correlation analysis (CCA), a dimensionality reduction method used for the data acquired in multi-view learning, can extract a low-dimensional space where the correlation between two multivariate data is high. However, CCA has the following problems. First, it is difficult to perform dimensionality reduction while taking advantage of the label information attached to the data. Second, CCA is an analysis method for two sets of data; it cannot be directly applied when we have three or more datasets. Discriminative canonical correlation analysis (DCCA) can be used to solve the first problem. It enables the dimensionality reduction of two datasets while reflecting the label information. Further, generalized canonical correlation analysis (GCCA) can be used to solve the second problem. It calculates canonical correlation variables, which are the products of parameters and data, for three or more datasets, and assumes that all datasets aggregate to a piece of shared information so that the parameters aggregate the information of each data. However, DCCA and GCCA do not simultaneously solve the problems of CCA. Therefore, in this study, we extend DCCA and propose a dimensionality reduction method using the label information for three or more datasets. We validate the usefulness of the proposed method through a simulation study.

Cross-domain recommender system using generalized canonical correlation analysis

Recommender systems provide personalized recommendations to the users from a large number of possible options in online stores. Matrix factorization is a well-known and accurate collaborative filtering approach for recommender system, which suffers from cold-start problem for new users and items. When new users join the system, it will take some time before they enter some ratings in the system, until that time, there are not enough ratings to learn the matrix factorization model. Using auxiliary data such as user’s demographic, ratings and reviews in relevant domains, is an effective solution to reduce the new user problem. In this paper, we used the data of users activity from auxiliary domains to build domain-independent users representation that could be used to predict users ratings in the target domains. We proposed an iterative method which applied MAX-VAR generalized canonical correlation analysis (GCCA) on user’s latent factors learned from matrix factorization on each domain. Also, to improve the capability of GCCA to learn latent factors for new users, we propose a generalized canonical correlation analysis by inverse sum of selection matrices (GCCA-ISSM) approach, which provides better recommendations in cold-start scenarios. The proposed approach is extended using content-based features like topic models extracted from user’s reviews. We demonstrate the accuracy and effectiveness of the proposed approaches on cross-domain rating predictions using comprehensive experiments on Amazon and MovieLens datasets.

Unsupervised multi-view representation learning with proximity guided representation and generalized canonical correlation analysis

Multi-view data can collaborate with each other to provide more comprehensive information than single-view data. Although there exist a few unsupervised multi-view representation learning methods taking both the discrepancies and incorporating complementary information from different views into consideration, they always ignore the use of inner-view discriminant information. It remains challenging to learn a meaningful shared representation of multiple views. To overcome this difficulty, this paper proposes a novel unsupervised multi-view representation learning model, MRL. Unlike most state-of-art multi-view representation learning, which only can be used for clustering or classification task, our method explores the proximity guided representation from inner-view and complete the task of multi-label classification and clustering by the discrimination fusion representation simultaneously. MRL consists of three parts. The first part is a deep representation learning for each view and then aims to represent the latent specific discriminant characteristic of each view, the second part builds a proximity guided dynamic routing to preserve its inner features of direction,location and etc. At last, the third part, GCCA-based fusion, exploits the maximum correlations among multiple views based on Generalized Canonical Correlation Analysis (GCCA). To the best of our knowledge, the proposed MRL could be one of the first unsupervised multi-view representation learning models that work in proximity guided dynamic routing and GCCA modes. The proposed model MRL is tested on five multi-view datasets for two different tasks. In the task of multi-label classification, the results show that our model is superior to the state-of-the-art multi-view learning methods in precision, recall, F1 and accuracy. In clustering task, its performance is better than the latest related popular algorithms. And the performance varies w.r.t. the dimensionality of G is also made to explore the characteristics of MRL.

Generalized canonical correlation analysis for functional data

Summary There is a growing need to analyze data sets characterized by several sets of variables observed on the same set of individuals. Such complex data structures are known as multiblock (or multiple-set) data sets. Multi-block data sets are encountered in diverse fields including bioinformatics, chemometrics, food analysis, etc. Generalized Canonical Correlation Analysis (GCCA) is a very powerful method to study this kind of relationships between blocks. It can also be viewed as a method for the integration of information from K &gt; 2 distinct sources (Takane and Oshima-Takane 2002). In this paper, GCCA is considered in the context of multivariate functional data. Such data are treated as realizations of multivariate random processes. GCCA is a technique that allows the joint analysis of several sets of data through dimensionality reduction. The central problem of GCCA is to construct a series of components aiming to maximize the association among the multiple variable sets. This method will be presented for multivariate functional data. Finally, a practical example will be discussed.

Quantitative Diagnosis Method of Gearbox Under Varying Conditions Based on ARX Model and Generalized Canonical Correlation Analysis

Fault diagnosis of gearboxes under the condition of varying speed and varying load is a hotspot and difficulty in the research of gearboxes. The response signals of gearbox under varying conditions exhibit non-linear and non-stationary characteristics, which increase the complexity of quantitative diagnosis of gearbox faults. A quantitative diagnosis method of gearbox faults based on the improved autoregressive with exogenous (ARX) model and generalized canonical correlation analysis (GCCA) is proposed in this paper. The ARX model is improved based on incremental recursive identification of Kalman filter to build system transfer characteristic models using the excitation and response signals of gearboxes. ARX models of gearboxes are nonlinear and the GCCA is proposed to build the quantitatively relationship between models with faulty status and healthy status. Simulation and experiment results indicate that the proposed method can effectively identify the severity of the gearbox failures under varying conditions and provides a promising method for the quantitative diagnosis of rotating machinery.

Efficient and Distributed Generalized Canonical Correlation Analysis for Big Multiview Data

Generalized canonical correlation analysis (GCCA) integrates information from data samples that are acquired at multiple feature spaces (or ‘views’) to produce low-dimensional representations—which is an extension of classical two-view CCA. Since the 1960s, (G)CCA has attracted much attention in statistics, machine learning, and data mining because of its importance in data analytics. Despite these efforts, the existing GCCA algorithms have serious complexity issues. The memory and computational complexities of the existing algorithms usually grow as a quadratic and cubic function of the problem dimension (the number of samples / features), respectively—e.g., handling views with $\approx \!\!1,000$ ≈ 1 , 000 features using such algorithms already occupies $\approx \!\!10^6$ ≈ 10 6 memory and the per-iteration complexity is $\approx\!\!10^9$ ≈ 10 9 flops—which makes it hard to push these methods much further. To circumvent such difficulties, we first propose a GCCA algorithm whose memory and computational costs scale linearly in the problem dimension and the number of nonzero data elements, respectively. Consequently, the proposed algorithm can easily handle very large sparse views whose sample and feature dimensions both exceed $\approx\!\! 100,000$ ≈ 100 , 000 . Our second contribution lies in proposing two distributed algorithms for GCCA, which compute the canonical components of different views in parallel and thus can further reduce the runtime significantly if multiple computing agents are available. We provide detailed convergence analyses of the proposed algorithms and show that all the large-scale GCCA algorithms converge to a Karush-Kuhn-Tucker (KKT) point at least sublinearly. Judiciously designed synthetic and real-data experiments are employed to showcase the effectiveness of the proposed algorithms.

Unsupervised multiblock data analysis: A unified approach and extensions

For the analysis of multiblock data, a unified approach of several strategies such as Generalized Canonical Correlation Analysis (GCCA), Multiblock Principal Components Analysis (MB-PCA), Hierarchical Principal Components Analysis (H-PCA) and ComDim is outlined. These methods are based on the determination of global and block components. The unified approach postulates, on the one hand, two link functions that relate the block components to their associated global components and, on the other hand, two summing up expressions to compute the global components from their associated block components. Not only several well-known methods are retrieved but we also introduce a variant of GCCA. More generally, we hint to other possibilities of extensions thus emphasizing the fact that the unified approach, besides being simple, is versatile. We also show how this approach of analysis although basically unsupervised could be adapted to yield a supervised method to be used for a prediction purpose. Illustrations on the basis of simulated and real case studies are discussed.

Label guided correlation hashing for large-scale cross-modal retrieval

With the explosive growth of multimedia data such as text and image, large-scale cross-modal retrieval has attracted more attention from vision community. But it still confronts the problems of the so-called “media gap” and search efficiency. Looking into the literature, we find that one leading type of existing cross-modal retrieval methods has been broadly investigated to alleviate the above problems by capturing the correlations across modalities as well as learning hashing codes. However, supervised label information is usually independently considered in the process of either generating hashing codes or learning hashing function. To this, we propose a label guided correlation cross-modal hashing method (LGCH), which investigates an alternative way to exploit label information for effective cross-modal retrieval from two aspects: 1) LGCH learns the discriminative common latent representation across modalities through joint generalized canonical correlation analysis (GCCA) and a linear classifier; 2) to simultaneously generate binary codes and hashing function, LGCH introduces an adaptive parameter to effectively fuse the common latent representation and the label guided representation for effective cross-modal retrieval. Moreover, each subproblem of LGCH has the elegant analytical solution. Experiments of cross-modal retrieval on three multi-media datasets show LGCH performs favorably against many well-established baselines.

Speech Recognition via fNIRS Based Brain Signals

In this paper, we present the first evidence that perceived speech can be identified from the listeners' brain signals measured via functional-near infrared spectroscopy (fNIRS)—a non-invasive, portable, and wearable neuroimaging technique suitable for ecologically valid settings. In this study, participants listened audio clips containing English stories while prefrontal and parietal cortices were monitored with fNIRS. Machine learning was applied to train predictive models using fNIRS data from a subject pool to predict which part of a story was listened by a new subject not in the pool based on the brain's hemodynamic response as measured by fNIRS. fNIRS signals can vary considerably from subject to subject due to the different head size, head shape, and spatial locations of brain functional regions. To overcome this difficulty, a generalized canonical correlation analysis (GCCA) was adopted to extract latent variables that are shared among the listeners before applying principal component analysis (PCA) for dimension reduction and applying logistic regression for classification. A 74.7% average accuracy has been achieved for differentiating between two 50 s. long story segments and a 43.6% average accuracy has been achieved for differentiating four 25 s. long story segments. These results suggest the potential of an fNIRS based-approach for building a speech decoding brain-computer-interface for developing a new type of neural prosthetic system.

The adaptive block sparse PCA and its application to multi-subject FMRI data analysis using sparse mCCA

Abstract Motivated by the problem of multi-subject functional magnetic resonance imaging (fMRI) data sets analysis using multiple-set canonical correlation analysis (mCCA), in this paper we propose a new variant of the principal component analysis (PCA) method, namely the adaptive block sparse PCA. It has the advantage to produce modified principal components with block sparse loadings. It is derived using penalized rank one matrix approximation where the penalty is introduced in the minimization problem to promote block sparsity of the loading vectors. An efficient algorithm is proposed for its computation. The effectiveness of the proposed method is illustrated on the problem of multi-subject fMRI data sets analysis using mCCA which is a generalization of canonical correlation analysis (CCA) to three or more sets of variables. This application is obtained by deriving the connection between mCCA and the singular value decomposition (SVD).

Structured SUMCOR Multiview Canonical Correlation Analysis for Large-Scale Data

The sum-of-correlations (SUMCOR) formulation of generalized canonical correlation analysis (GCCA) seeks highly correlated low-dimensional representations of different views via maximizing pairwise latent similarity of the views. SUMCOR is considered arguably the most natural extension of classical two-view CCA to the multiview case, and thus has numerous applications in signal processing and data analytics. Recent work has proposed effective algorithms for handling the SUMCOR problem at very large scale. However, the existing scalable algorithms cannot incorporate structural regularization and prior information -- which are critical for good performance in real-world applications. In this work, we propose a new computational framework for large-scale SUMCOR GCCA that can easily incorporate a suite of structural regularizers which are frequently used in data analytics. The updates of the proposed algorithm are lightweight and the memory complexity is also low. In addition, the proposed algorithm can be readily implemented in a parallel fashion. We show that the proposed algorithm converges to a Karush-Kuhn-Tucker (KKT) point of the regularized SUMCOR problem. Judiciously designed simulations and real-data experiments are employed to demonstrate the effectiveness of the proposed algorithm.

Scalable and Flexible Multiview MAX-VAR Canonical Correlation Analysis

Generalized canonical correlation analysis (GCCA) aims at finding latent low-dimensional common structure from multiple views (feature vectors in different domains) of the same entities. Unlike principal component analysis (PCA) that handles a single view, (G)CCA is able to integrate information from different feature spaces. Here we focus on MAX-VAR GCCA, a popular formulation which has recently gained renewed interest in multilingual processing and speech modeling. The classic MAX-VAR GCCA problem can be solved optimally via eigen-decomposition of a matrix that compounds the (whitened) correlation matrices of the views; but this solution has serious scalability issues, and is not directly amenable to incorporating pertinent structural constraints such as non-negativity and sparsity on the canonical components. We posit regularized MAX-VAR GCCA as a non-convex optimization problem and propose an alternating optimization (AO)-based algorithm to handle it. Our algorithm alternates between {\em inexact} solutions of a regularized least squares subproblem and a manifold-constrained non-convex subproblem, thereby achieving substantial memory and computational savings. An important benefit of our design is that it can easily handle structure-promoting regularization. We show that the algorithm globally converges to a critical point at a sublinear rate, and approaches a global optimal solution at a linear rate when no regularization is considered. Judiciously designed simulations and large-scale word embedding tasks are employed to showcase the effectiveness of the proposed algorithm.