Canonical Correlation Analysis Problem Research Articles

TSSNALM: A fast two-stage semi-smooth Newton augmented Lagrangian method for sparse CCA

Abstract Canonical correlation analysis (CCA) is a very useful tool for measuring the linear relationship between two multidimensional variables. However, it often fails to extract meaningful features in high-dimensional settings. This motivates the sparse CCA problem, in which l1 constraints are applied to the canonical vectors. Although some sparse CCA solvers exist in the literature, we found that none of them is efficient. We propose a fast two-stage semi-smooth Newton augmented Lagrangian method ( tSSNALM ) to solve sparse CCA problems, and we provide convergence analysis. Numerical comparisons between our approach and a number of state-of-the-art solvers, on simulated data sets, are presented to demonstrate its efficiency. To the best of our knowledge, this is the first time that duality has been integrated with a semi-smooth Newton method for solving sparse CCA.

Robust Multi-View Representation Learning (Student Abstract)

Multi-view data has become ubiquitous, especially with multi-sensor systems like self-driving cars or medical patient-side monitors. We propose two methods to approach robust multi-view representation learning with the aim of leveraging local relationships between views.The first is an extension of Canonical Correlation Analysis (CCA) where we consider multiple one-vs-rest CCA problems, one for each view. We use a group-sparsity penalty to encourage finding local relationships. The second method is a straightforward extension of a multi-view AutoEncoder with view-level drop-out.We demonstrate the effectiveness of these methods in simple synthetic experiments. We also describe heuristics and extensions to improve and/or expand on these methods.

Nonlinear Canonical Correlation Analysis:A Compressed Representation Approach.

Canonical Correlation Analysis (CCA) is a linear representation learning method that seeks maximally correlated variables in multi-view data. Nonlinear CCA extends this notion to a broader family of transformations, which are more powerful in many real-world applications. Given the joint probability, the Alternating Conditional Expectation (ACE) algorithm provides an optimal solution to the nonlinear CCA problem. However, it suffers from limited performance and an increasing computational burden when only a finite number of samples is available. In this work, we introduce an information-theoretic compressed representation framework for the nonlinear CCA problem (CRCCA), which extends the classical ACE approach. Our suggested framework seeks compact representations of the data that allow a maximal level of correlation. This way, we control the trade-off between the flexibility and the complexity of the model. CRCCA provides theoretical bounds and optimality conditions, as we establish fundamental connections to rate-distortion theory, the information bottleneck and remote source coding. In addition, it allows a soft dimensionality reduction, as the compression level is determined by the mutual information between the original noisy data and the extracted signals. Finally, we introduce a simple implementation of the CRCCA framework, based on lattice quantization.

Using elastic net restricted kernel canonical correlation analysis for cross-language information retrieval

Kernel methods, which are a non-linear variant of linear methods, are used to increase the flexibility and allow to examine non-linear relationships by linear methods. The conventional solution of the restricted kernel canonical correlation analysis problem has a major drawback, it solves the problem in a reasonable time frame only for problems with few variables. We successfully overcame this limitation by implementing the method with the alternating least-squares algorithm. This allowed us to apply the method to cross-language information retrieval problem on a big dataset. We compared the results to other established methods and they were encouraging.

Minimax estimation in sparse canonical correlation analysis

Canonical correlation analysis is a widely used multivariate statistical technique for exploring the relation between two sets of variables. This paper considers the problem of estimating the leading canonical correlation directions in high-dimensional settings. Recently, under the assumption that the leading canonical correlation directions are sparse, various procedures have been proposed for many high-dimensional applications involving massive data sets. However, there has been few theoretical justification available in the literature. In this paper, we establish rate-optimal nonasymptotic minimax estimation with respect to an appropriate loss function for a wide range of model spaces. Two interesting phenomena are observed. First, the minimax rates are not affected by the presence of nuisance parameters, namely the covariance matrices of the two sets of random variables, though they need to be estimated in the canonical correlation analysis problem. Second, we allow the presence of the residual canonical correlation directions. However, they do not influence the minimax rates under a mild condition on eigengap. A generalized sin-theta theorem and an empirical process bound for Gaussian quadratic forms under rank constraint are used to establish the minimax upper bounds, which may be of independent interest.

Distributed Canonical Correlation Analysis in Wireless Sensor Networks With Application to Distributed Blind Source Separation

Canonical correlation analysis (CCA) is a widely used data analysis tool that allows to assess the correlation between two distinct sets of signals. It computes optimal linear combinations of the signals in both sets such that the resulting signals are maximally correlated. The weight vectors defining these optimal linear combinations are referred to as “principal CCA directions”. In addition to this particular type of data analysis, CCA is also often used as a blind source separation (BSS) technique, i.e., under certain assumptions, the principal CCA directions have certain demixing properties. In this paper, we propose a distributed CCA (DCCA) algorithm that can operate in wireless sensor networks (WSNs) with a fully connected or a tree topology. The algorithm estimates the $Q$ principal CCA directions from the sensor signal observations collected by the different nodes in the WSN and extracts the corresponding sources. These network-wide principal CCA directions are estimated in a time-recursive fashion without explicitly constructing the corresponding network-wide correlation matrices, i.e., without the need for data centralization. Instead, each node locally computes smaller CCA problems and only transmits compressed sensor signal observations (of dimension $Q$ ), which significantly reduces the bit rate over the wireless links of the WSN. We prove convergence and optimality of the DCCA algorithm, and we demonstrate its performance by means of numerical simulations in a blind source separation scenario.

Sparse canonical correlation analysis from a predictive point of view.

Canonical correlation analysis (CCA) describes the associations between two sets of variables by maximizing the correlation between linear combinations of the variables in each dataset. However, in high-dimensional settings where the number of variables exceeds the sample size or when the variables are highly correlated, traditional CCA is no longer appropriate. This paper proposes a method for sparse CCA. Sparse estimation produces linear combinations of only a subset of variables from each dataset, thereby increasing the interpretability of the canonical variates. We consider the CCA problem from a predictive point of view and recast it into a regression framework. By combining an alternating regression approach together with a lasso penalty, we induce sparsity in the canonical vectors. We compare the performance with other sparse CCA techniques in different simulation settings and illustrate its usefulness on a genomic dataset.

Robust Sparse Canonical Correlation Analysis

Canonical correlation analysis (CCA) is a multivariate statistical method which describes the associations between two sets of variables. The objective is to find linear combinations of the variables in each data set having maximal correlation. This paper discusses a method for Robust Sparse CCA. Sparse estimation produces canonical vectors with some of their elements estimated as exactly zero. As such, their interpretability is improved. We also robustify the method such that it can cope with outliers in the data. To estimate the canonical vectors, we convert the CCA problem into an alternating regression framework, and use the sparse Least Trimmed Squares estimator. We illustrate the good performance of the Robust Sparse CCA method in several simulation studies and two real data examples.

Sparse Canonical Correlation Analysis: New Formulation and Algorithm

In this paper, we study canonical correlation analysis (CCA), which is a powerful tool in multivariate data analysis for finding the correlation between two sets of multidimensional variables. The main contributions of the paper are: 1) to reveal the equivalent relationship between a recursive formula and a trace formula for the multiple CCA problem, 2) to obtain the explicit characterization for all solutions of the multiple CCA problem even when the corresponding covariance matrices are singular, 3) to develop a new sparse CCA algorithm, and 4) to establish the equivalent relationship between the uncorrelated linear discriminant analysis and the CCA problem. We test several simulated and real-world datasets in gene classification and cross-language document retrieval to demonstrate the effectiveness of the proposed algorithm. The performance of the proposed method is competitive with the state-of-the-art sparse CCA algorithms.

Sparse Canonical Correlation Analysis from a Predictive Point of View

Canonical correlation analysis (CCA) describes the associations between two sets of variables by maximizing the correlation between linear combinations of the variables in each data set. However, in high-dimensional settings where the number of variables exceeds the sample size or when the variables are highly correlated, traditional CCA is no longer appropriate. This paper proposes a method for sparse CCA. Sparse estimation produces linear combinations of only a subset of variables from each data set, thereby increasing the interpretability of the canonical variates. We consider the CCA problem from a predictive point of view and recast it into a multivariate regression framework. By combining a multi-variate alternating regression approach together with a lasso penalty, we induce sparsity in the canonical vectors. We compare the performance with other sparse CCA techniques in different simulation settings and illustrate its usefulness on a genomic data set.