Deep Canonical Correlation Analysis Research Articles

Cross-Project Software Defect Prediction Based on SMOTE and Deep Canonical Correlation Analysis

Cross-Project Software Defect Prediction Based on SMOTE and Deep Canonical Correlation Analysis

Real-Time Reconstruction of HIFU Focal Temperature Field Based on Deep Learning.

Objective and Impact Statement: High-intensity focused ultrasound (HIFU) therapy is a promising noninvasive method that induces coagulative necrosis in diseased tissues through thermal and cavitation effects, while avoiding surrounding damage to surrounding normal tissues. Introduction: Accurate and real-time acquisition of the focal region temperature field during HIFU treatment marked enhances therapeutic efficacy, holding paramount scientific and practical value in clinical cancer therapy. Methods: In this paper, we initially designed and assembled an integrated HIFU system incorporating diagnostic, therapeutic, and temperature measurement functionalities to collect ultrasound echo signals and temperature variations during HIFU therapy. Furthermore, we introduced a novel multimodal teacher-student model approach, which utilizes the shared self-expressive coefficients and the deep canonical correlation analysis layer to aggregate each modality data, then through knowledge distillation strategies, transfers the knowledge from the teacher model to the student model. Results: By investigating the relationship between the phantoms, invitro, and invivo ultrasound echo signals and temperatures, we successfully achieved real-time reconstruction of the HIFU focal 2D temperature field region with a maximum temperature error of less than 2.5 °C. Conclusion: Our method effectively monitored the distribution of the HIFU temperature field in real time, providing scientifically precise predictive schemes for HIFU therapy, laying a theoretical foundation for subsequent personalized treatment dose planning, and providing efficient guidance for noninvasive, nonionizing cancer treatment.

Discriminative Deep Canonical Correlation Analysis for Multi-View Data.

Over the past few years, multimodal data analysis has emerged as an inevitable method for identifying sample categories. In the multi-view data classification problem, it is expected that the joint representation should include the supervised information of sample categories so that the similarity in the latent space implies the similarity in the corresponding concepts. Since each view has different statistical properties, the joint representation should be able to encapsulate the underlying nonlinear data distribution of the given observations. Another important aspect is the coherent knowledge of the multiple views. It is required that the learning objective of the multi-view model efficiently captures the nonlinear correlated structures across different modalities. In this context, this article introduces a novel architecture, termed discriminative deep canonical correlation analysis (D2CCA), for classifying given observations into multiple categories. The learning objective of the proposed architecture includes the merits of generative models to identify the underlying probability distribution of the given observations. In order to improve the discriminative ability of the proposed architecture, the supervised information is incorporated into the learning objective of the proposed model. It also enables the architecture to serve as both a feature extractor as well as a classifier. The theory of CCA is integrated with the objective function so that the joint representation of the multi-view data is learned from maximally correlated subspaces. The proposed framework is consolidated with corresponding convergence analysis. The efficacy of the proposed architecture is studied on different domains of applications, namely, object recognition, document classification, multilingual categorization, face recognition, and cancer subtype identification with reference to several state-of-the-art methods.

Multimodal Deep Learning for Activity Detection from IoT Sensors

The advancement of various computing technologies such as Artificial Intelligence (AI), cloud computing, mobile computing, and Internet of Things (IoT) is widely applied in vast applications. The rapid evolvement of IoT is determined in advance with embedded systems, data analytics, and wireless connectivity. However, the Deep Learning (DL) method is highly applicable to evaluating human activity which is a significant element to determine the surveillance of human behavior. But the common issue obtained in the detection of human activity is the time series problem and failure to detect accurately in heterogeneous sensors. So a novel Multi-Modal Deep-Learning (MMDL) is proposed to detect human activity accurately with reduced time consumption. However, in real-time applications, huge amounts of data are monitored and the activity of every individual is detected independently by integrating the proposed method with 2D Deep Canonical Correlation Analysis (DCCA). Furthermore, the detection of human activity is employed through IoT devices like multi-modal sensors. The multi-head CNN is used to decrease the cross channels and spatial interactions while training with high-dimensional datasets. The MMDL and binary Support Vector Machine (SVM) architectures overcome the drawbacks of the existing techniques such as the need for increased width/depth for improved performance and the rare usage of hierarchical representations. Thus the effectiveness of the proposed method is validated by STISEN and GAIT datasets. The experimentation results revealed that the proposed method attained a better accuracy of 97% with reduced time in various scenarios related to existing methods.

Multichannel CNN-BLSTM Architecture for Speech Emotion Recognition System by Fusion of Magnitude and Phase Spectral Features Using DCCA for Consumer Applications

Conventional Speech Emotion Recognition (SER) approaches put more emphasis on extracting magnitude spectrum-based features, such as Mel Frequency Cepstral Coefficients (MFCCs), and Mel spectrogram. However, phase information is ignored due to signal processing difficulties such as the phase wrapping issue. This work develops a multichannel Convolution Neural Network-Bidirectional Long Short Term Memory (CNN-BLSTM) architectures with an attention mechanism for speaker-independent SER by considering phase and magnitude spectrum-based features. The phase-based features are extracted using the Modified Group Delay Function (MODGD). The obtained phase features are combined with MFCC features. The CNN-BLSTM network extract learned representation from magnitude and phase features. The learned representation from MFCCs and MODGD are combined and given as an input to the Support Vector Machine (SVM) for classification. The Deep Canonical Correlation Analysis (DCCA) is used to maximize the correlation between magnitude and phase information to improve the conventional SER system’s performance. Here the IEMOCAP database is used for performance analysis. The experimental results show improvement over MFCC features and existing approaches for unimodal SER. In this work, we also developed real-time web server application for the proposed architecture.

A Bimodal Emotion Recognition Approach through the Fusion of Electroencephalography and Facial Sequences.

In recent years, human-computer interaction (HCI) systems have become increasingly popular. Some of these systems demand particular approaches for discriminating actual emotions through the use of better multimodal methods. In this work, a deep canonical correlation analysis (DCCA) based multimodal emotion recognition method is presented through the fusion of electroencephalography (EEG) and facial video clips. A two-stage framework is implemented, where the first stage extracts relevant features for emotion recognition using a single modality, while the second stage merges the highly correlated features from the two modalities and performs classification. Convolutional neural network (CNN) based Resnet50 and 1D-CNN (1-Dimensional CNN) have been utilized to extract features from facial video clips and EEG modalities, respectively. A DCCA-based approach was used to fuse highly correlated features, and three basic human emotion categories (happy, neutral, and sad) were classified using the SoftMax classifier. The proposed approach was investigated based on the publicly available datasets called MAHNOB-HCI and DEAP. Experimental results revealed an average accuracy of 93.86% and 91.54% on the MAHNOB-HCI and DEAP datasets, respectively. The competitiveness of the proposed framework and the justification for exclusivity in achieving this accuracy were evaluated by comparison with existing work.

A Novel Lightweight Audio-visual Saliency Model for Videos

Audio information has not been considered an important factor in visual attention models regardless of many psychological studies that have shown the importance of audio information in the human visual perception system. Since existing visual attention models only utilize visual information, their performance is limited but also requires high-computational complexity due to the limited information available. To overcome these problems, we propose a lightweight audio-visual saliency (LAVS) model for video sequences. To the best of our knowledge, this article is the first trial to utilize audio cues for an efficient deep-learning model for the video saliency estimation. First, spatial-temporal visual features are extracted by the lightweight receptive field block (RFB) with the bidirectional ConvLSTM units. Then, audio features are extracted by using an improved lightweight environment sound classification model. Subsequently, deep canonical correlation analysis (DCCA) aims at capturing the correspondence between audio and spatial-temporal visual features, thus obtaining a spatial-temporal auditory saliency. Lastly, the spatial-temporal visual and auditory saliency are fused to obtain the audio-visual saliency map. Extensive comparative experiments and ablation studies validate the performance of the LAVS model in terms of effectiveness and complexity.

Deep Canonical Correlation Analysis Using Sparsity-Constrained Optimization for Nonlinear Process Monitoring

This article proposes an efficient nonlinear process monitoring method (DCCA-SCO) by integrating canonical correlation analysis (CCA), deep autoencoder neural networks (DAENNs), and sparsity-constrained optimization (SCO). Specifically, DAENNs are first used to learn a nonlinear function automatically, which characterizes intrinsic features of the original process data. Then, the CCA is performed in that low-dimensional representation space to extract the most correlated variables. In addition, the SCO is imposed to reduce the redundancy of the hidden representation. Unlike other deep CCA methods, the DCCA-SCO provides a new nonlinear method that is able to learn a nonlinear mapping with a sparse prior. The validity of the proposed DCCA-SCO is extensively demonstrated on the benchmark Tennessee Eastman (TE) process and the diesel generator process. In particular, compared with the classical CCA, the fault detection rate is increased by 8.00% for the fault IDV(11) in the TE process.

SDGCCA: Supervised Deep Generalized Canonical Correlation Analysis for Multi-Omics Integration.

Integration of multi-omics data provides opportunities for revealing biological mechanisms related to certain phenotypes. We propose a novel method of multi-omics integration called supervised deep generalized canonical correlation analysis (SDGCCA) for modeling correlation structures between nonlinear multi-omics manifolds that aims at improving the classification of phenotypes and revealing the biomarkers related to phenotypes. SDGCCA addresses the limitations of other canonical correlation analysis (CCA)-based models (such as deep CCA, deep generalized CCA) by considering complex/nonlinear cross-data correlations between multiple (2) modalities. Although there are a few methods to learn nonlinear CCA projections for classifying phenotypes, they only consider two views. Methods extended to multiple views either do not perform classification or do not provide feature ranking. In contrast, SDGCCA is a nonlinear multi-view CCA projection method that performs classification and ranks features. When we applied SDGCCA in predicting patients with Alzheimer's disease (AD) and discrimination of early- and late-stage cancers, it outperformed other CCA-based and other supervised methods. In addition, we demonstrate that SDGCCA can be applied for feature selection to identify important multi-omics biomarkers. On applying AD data, SDGCCA identified clusters of genes in multi-omics data, well known to be associated with AD.

Divide-and-Attention Network for HE-Stained Pathological Image Classification.

Simple SummaryWe propose a Divide-and-Attention network that can learn representative pathological image features with respect to different tissue structures and adaptively focus on the most important ones. In addition, we introduce deep canonical correlation analysis constraints in the feature fusion process of different branches, so as to maximize the correlation of different branches and ensure that the fused branches emphasize specific tissue structures. Extensive experiments on three different pathological image datasets show that the proposed method achieved competitive results.Since pathological images have some distinct characteristics that are different from natural images, the direct application of a general convolutional neural network cannot achieve good classification performance, especially for fine-grained classification problems (such as pathological image grading). Inspired by the clinical experience that decomposing a pathological image into different components is beneficial for diagnosis, in this paper, we propose a Divide-and-Attention Network (DANet) for Hematoxylin-and-Eosin (HE)-stained pathological image classification. The DANet utilizes a deep-learning method to decompose a pathological image into nuclei and non-nuclei parts. With such decomposed pathological images, the DANet first performs feature learning independently in each branch, and then focuses on the most important feature representation through the branch selection attention module. In this way, the DANet can learn representative features with respect to different tissue structures and adaptively focus on the most important ones, thereby improving classification performance. In addition, we introduce deep canonical correlation analysis (DCCA) constraints in the feature fusion process of different branches. The DCCA constraints play the role of branch fusion attention, so as to maximize the correlation of different branches and ensure that the fused branches emphasize specific tissue structures. The experimental results of three datasets demonstrate the superiority of the DANet, with an average classification accuracy of 92.5% on breast cancer classification, 95.33% on colorectal cancer grading, and 91.6% on breast cancer grading tasks.

Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition

Multimodal signals are powerful for emotion recognition since they can represent emotions comprehensively. In this article, we compare the recognition performance and robustness of two multimodal emotion recognition models: 1) deep canonical correlation analysis (DCCA) and 2) bimodal deep autoencoder (BDAE). The contributions of this article are threefold: 1) we propose two methods for extending the original DCCA model for multimodal fusion: a) weighted sum fusion and b) attention-based fusion; 2) we systemically compare the performance of DCCA, BDAE, and traditional approaches on five multimodal data sets; and 3) we investigate the robustness of DCCA, BDAE, and traditional approaches on SEED-V and DREAMER data sets under two conditions: 1) adding noises to multimodal features and 2) replacing electroencephalography features with noises. Our experimental results demonstrate that DCCA achieves state-of-the-art recognition results on all five data sets: 1) 94.6% on the SEED data set; 2) 87.5% on the SEED-IV data set; 3) 84.3% and 85.6% on the DEAP data set; 4) 85.3% on the SEED-V data set; and 5) 89.0%, 90.6%, and 90.7% on the DREAMER data set. Meanwhile, DCCA has greater robustness when adding various amounts of noises to the SEED-V and DREAMER data sets. By visualizing features before and after DCCA transformation on the SEED-V data set, we find that the transformed features are more homogeneous and discriminative across emotions.

A Robust and Accurate Indoor Localization Using Learning-Based Fusion of Wi-Fi RTT and RSSI.

Great attention has been paid to indoor localization due to its wide range of associated applications and services. Fingerprinting and time-based localization techniques are among the most popular approaches in the field due to their promising performance. However, fingerprinting techniques usually suffer from signal fluctuations and interference, which yields unstable localization performance. On the other hand, the accuracy of time-based techniques is highly affected by multipath propagation errors and non-line-of-sight transmissions. To combat these challenges, this paper presents a hybrid deep-learning-based indoor localization system called RRLoc which fuses fingerprinting and time-based techniques with a view of combining their advantages. RRLoc leverages a novel approach for fusing received signal strength indication (RSSI) and round-trip time (RTT) measurements and extracting high-level features using deep canonical correlation analysis. The extracted features are then used in training a localization model for facilitating the location estimation process. Different modules are incorporated to improve the deep model’s generalization against overtraining and noise. The experimental results obtained at two different indoor environments show that RRLoc improves localization accuracy by at least 267% and 496% compared to the state-of-the-art fingerprinting and ranging-based-multilateration techniques, respectively.

Investigating EEG-based functional connectivity patterns for multimodal emotion recognition

Objective. Previous studies on emotion recognition from electroencephalography (EEG) mainly rely on single-channel-based feature extraction methods, which ignore the functional connectivity between brain regions. Hence, in this paper, we propose a novel emotion-relevant critical subnetwork selection algorithm and investigate three EEG functional connectivity network features: strength, clustering coefficient, and eigenvector centrality. Approach. After constructing the brain networks by the correlations between pairs of EEG signals, we calculated critical subnetworks through the average of brain network matrices with the same emotion label to eliminate the weak associations. Then, three network features were conveyed to a multimodal emotion recognition model using deep canonical correlation analysis along with eye movement features. The discrimination ability of the EEG connectivity features in emotion recognition is evaluated on three public datasets: SEED, SEED-V, and DEAP. Main results. The experimental results reveal that the strength feature outperforms the state-of-the-art features based on single-channel analysis. The classification accuracies of multimodal emotion recognition are on the SEED dataset, on the SEED-V dataset, and and for arousal and valence on the DEAP dataset, respectively, which all achieved the best performance. In addition, the brain networks constructed with 18 channels achieve comparable performance with that of the 62-channel network and enable easier setups in real scenarios. Significance. The EEG functional connectivity networks combined with emotion-relevant critical subnetworks selection algorithm we proposed is a successful exploration to excavate the information between channels.

Identifying imaging genetics biomarkers in Alzheimer’s disease via integrating graph convolutional neural network and canonical correlation analysis

AbstractBackgroundBrain imaging genetics is an emerging research topic in the study of Alzheimer’s disease (AD). The conventional approach, such as canonical correlation analysis (CCA), has been widely used to identify imaging genetic associations. A deep learning model has recently been proposed to better understand the roots of the complex association between imaging and genetic measures. We propose a graph convolutional neural network (GCN) with CCA loss function to integrate and identify the complex imaging genetics associations in AD.MethodWe proposed a spectral GCN approach with CCA loss function (GCN‐CCA) to extract feature representations from imaging and genetics data. Briefly, the graph embeddings on the graph nodes were filtered in the Fourier domain. We used two hidden layers with 64 hidden units for extracting imaging and genetic data. ReLU activations were used after each graph convolution layer. A canonical correlation loss function was optimized based on Adam optimizer. We compared our model with the deep CCA model (DCCA) for AD classification.ResultWe downloaded data for 310 participants (103 AD and 207 Cognitive Normal [CN]) including neuroimaging and genetic data from the ADNI database. We used average structural connectivity based on the AAL atlas as the graph in our GCN model, three imaging measurements (VBM, FDG, FBR) as initial attributes on the graph nodes. We selected 2,644 candidate SNPs from the GWAS catalog (https://www.ebi.ac.uk/gwas/). The proposed model obtained 82.25 % test accuracy for the AD/CN classification, outperforming the DCCA model (77.41%). For interpretation, we generated the saliency maps using guided gradient backpropagation (Figs 1 and 2). We observed the imaging phenotypes from left middle temporal gyrus, superior temporal, frontal inferior triangularis, putamen, paracentral lobule, frontal medial orbital, and pallidum, and right posterior cingulum, and genetic markers from ABCA13 (rs2163935, rs6955132, rs4024044) and APOE (rs429358) contributed to the AD outcome prediction.ConclusionHere, we demonstrated the utility of GCN‐CCA model and its interpretability. The GCN‐CCA not only obtained higher prediction performance but also highlighted important regions for AD classification. We plan to apply our algorithm to other AD cohorts to see if our algorithm generalizes to independent data sets.

Deep-based Self-refined Face-top Coordination

Face-top coordination, which exists in most clothes-fitting scenarios, is challenging due to varieties of attributes, implicit correlations, and tradeoffs between general preferences and individual preferences. We present a Deep-Based Self-Refined (DBSR) system to simulate face-top coordination based on intuition evaluation. To this end, we first establish a well-coordinated face-top (WCFT) dataset from fashion databases and communities. Then, we use a jointly trained CNN Deep Canonical Correlation Analysis (DCCA) method to bridge the semantic face-top gap based on the WCFT dataset to deal with general preferences. Subsequently, an irrelevance-based Optimum-path Forest (OPF) method is developed to adapt the results to individual preferences iteratively. Experimental results and user study demonstrate the effectiveness of our method.

Multi-label text classification via joint learning from label embedding and label correlation

For the multi-label text classification problems with many classes, many existing multi-label classification algorithms become infeasible or suffer an unaffordable cost. Some researches hence perform the Label Space Dimension Reduction(LSDR) to solve this problem, but a number of methods ignore the sequence information of texts and the label correlation in the original label space, and treat the label as a meaningless multi-hot vector. In this paper, we put forward a multi-label text classification algorithm LELC(joint learning from Label Embedding and Label Correlation) based on the multi-layer attention and label correlation to solve the issue of multi-label text classification with a large number of class labels. Specifically, we firstly extract features through Bidirectional Gated Recurrent Unit Network(Bi-GRU), multi-layer attention and linear layers. Bi-GRU will capture the content information and sequence information of the text at the same time, and the attention mechanism can help us select the valid features related to labels. Then, we use matrix factorization to perform LSDR, and consider label correlation of the original label space in this process, which allows us to implicitly encode the latent space and simplify the model learning. Finally, Deep Canonical Correlation Analysis(CCA) technology is exploited to couple features and the latent space in an end-to-end pattern, so that these two can influence each other to learn the mapping of feature space to latent space. Experiments on 11 real-world datasets show the comparability between our proposed model and the state-of-the-art methods.

Multi-Label Learning Based Antenna Selection in Massive MIMO Systems

Antenna selection (AS) is a signal processing technology that can greatly reduce the hardware complexity of multi-antenna systems. Specifically, AS can decrease the number of required radio frequency chains by activating only a subset of the available antennas in each transmission slot. However, optimal AS suffers from a high computational complexity that increases exponentially with the scale of the antenna array. In this paper, we propose a low-complexity AS algorithm based on multi-label learning (MLL), where a deep neural network is employed to determine the set of selected antennas for a given channel matrix. Specifically, the MLL network combines deep canonical correlation analysis and an autoencoder in a unified network structure, which can extract the low dimensional features of channel matrix as well as the interdependency among selected antennas, so as to achieve an accurate prediction of the set of selected antennas with a relatively small-scale learning model. Simulation results show that, in comparison with the convex relaxation based method, our proposed MLL-based method can achieve comparable capacity with significantly reduced computation time.

Corticomuscular coupling analysis based on improved LSTM and transfer entropy

The study of functional corticomuscular coupling can reflect the interaction between the cerebral cortex and muscle tissue, thereby helping to understand how the brain controls muscle tissue and the effect of muscle movement on brain function. This study proposes a detection model of the coupling strength between the cortex and muscles. The detection model uses an adaptive selector to choose the optimal long short-term memory network, uses this network to extract the features of electroencephalography and electromyography, and finally transforms time characteristics into the frequency domain. The transfer entropy is used to represent the interaction intensity of signals in different frequency bands. Using this model, we analyze the coupling relationship between the cortex and muscles in the three movements of wrist flexion, wrist extension, and clench fist, and compare the model with traditional wavelet coherence analysis and deep canonical correlation analysis. The experimental results show that our model can not only express the bidirectional coupling relationship between different frequency bands but also suppress the possible false coupling that traditional methods may detect. Our research shows that the proposed model has great potential in medical rehabilitation, movement decoding, and other fields.

A Survey on Canonical Correlation Analysis

In recent years, the advances in data collection and statistical analysis promotes canonical correlation analysis (CCA) available for more advanced research. CCA is the main technique for two-set data dimensionality reduction such that the correlation between the pairwise variables in the common subspace is mutually maximized. Over 80-years of developments, a number of CCA models have been proposed according to different machine learning mechanisms. However, the field lacks an insightful review for the state-of-art developments. This survey targets to provide a well-organized overview for CCA and its extensions. Specifically, we first review the CCA theory from the perspective of both model formation and model optimization. The association between two popular solution methods, i.e., eigen value decomposition (EVD) and singular value decomposition (SVD), are discussed. Following that, we present a taxonomy of current progresses and classify them into seven groups: 1) multi-view CCA, 2) probabilistic CCA, 3) deep CCA, 4) kernel CCA, 5) discriminative CCA, 6) sparse CCA and 7) locality preserving CCA. For each group, we demonstrate two or three representative mathematical models, identifying their strengths and limitations. We summarize the representative applications and numerical results of these seven groups in real-world practices, collecting the data sets and open-sources for implementation. In the end, we provide several promising future research directions that can improve the current state of the art.

Data-Driven Fault Diagnosis Using Deep Canonical Variate Analysis and Fisher Discriminant Analysis

In this article, a novel data-driven fault diagnosis method by combining deep canonical variate analysis and Fisher discriminant analysis (DCVA-FDA) is proposed for complex industrial processes. Inspired by the recently developed deep canonical correlation analysis, a new nonlinear canonical variate analysis (CVA) called DCVA is first developed by incorporating deep neural networks into CVA. Based on DCVA, a residual generator is designed for the fault diagnosis process. FDA is applied in the feature space spanned by residual vectors. Then, a Bayesian inference classifier is performed in the reduced dimensional space of FDA to label the class of process data. A continuous stirred-tank reactor and an industrial benchmark of the Tennessee Eastman process are carried out to test the performance of DCVA-FDA fault diagnosis. The experimental results demonstrate that the proposed DCVA-FDA fault diagnosis is able to significantly improve the fault diagnosis performance when compared to other methods also examined in this article.