Common Subspace Research Articles

Adaptive Adversarial Learning based cross-modal retrieval

There exists a heterogeneity gap between multi-modal data, hence it is difficult to directly measure the similarity between them. A common way to solve the problem is representation learning. Due to the unique adversarial optimization principle and the efficient cross-modal correlation learning ability, the cross-modal retrieval based on Generative Adversarial Network (GAN) has received significant attention recently. However, the potential semantic information is not fully explored in most GAN-based cross-modal learning approaches. In this paper, we propose a novel Adaptive Adversarial Learning (AAL) based cross-modal retrieval method. The generator of a specific modality projects heterogeneous data into the potential common subspace, while the discriminator is against the generator to maintain discriminability. In addition, three task-specific loss functions are designed in the generators to comprehensively exploit the semantic and label information. One problem is that directly optimizing the generator network will lead to ignoring the assessment of contribution to multi-loss functions. To overcome the above challenge, we present an adaptive balance strategy to match the appropriate contribution for each loss according to the degree of dispersion. Comprehensive experimental results on three widely-used databases show that the proposed method is effective and superior to the existing cross-modal retrieval methods.

Adversarial discriminative domain adaptation for modulation classification based on Ulam stability

AbstractDomain adaptation modulation classification aims to improve the cross‐domain robustness of modulation classification, with the basic idea of mitigating the domain shift between the label‐rich source domain and label‐poor target domain in the latent common feature subspace. Due to the complex and heterogeneous wireless propagation conditions, how to construct the optimal latent common feature subspace is a hard problem. In this letter, the authors introduce Ulam stability to character the boundary of the latent common feature space. These theorems can provide guidance for cross‐domain feature extraction and eliminate the domain shift. Based on the proposed stable theorems, the authors define the Ulam loss and present a new method named Adversarial Discriminative Domain Adaptation with Ulam stability (U‐ADDA). Experiments on public datasets prove the effectiveness and generality of the method.

Fast Asymmetric and Discrete Cross-Modal Hashing With Semantic Consistency

Hashing has attracted widespread attention in the field of supervised cross-modal retrieval due to its advantages in search and storage. However, there are still some issues to be addressed, e.g.: 1) how to effectively combine sample and label semantics to learn hash codes; 2) how to reduce high computational requirements brought by computing a pairwise similarity matrix; and 3) how to effectively solve discrete optimization problems. To cope with them, a fast asymmetric and discrete cross-modal hashing (FADCH) method is proposed in this article. First, matrix factorization is leveraged to collaboratively construct a common semantic subspace between different modalities. Second, semantic consistency is preserved by aligning the common semantic subspace with the semantic representation constructed from labels, which effectively exploits the semantic complementarity of labels and samples. Third, we embed labels into hash codes and keep the correlation between different modal samples by using a pairwise similarity matrix. Fourth, we use an asymmetric strategy with relaxation to associate hash codes with semantic representation, which not only avoids the difficulty of symmetric frame optimization but also embeds more semantic information into the Hamming space. In addition, a strongly orthogonal constraint is introduced to optimize the hash codes. Finally, an effective optimization algorithm is developed to directly generate discrete hash codes while reducing the complexity from <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$O(n^{2})$ </tex-math></inline-formula> to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$O(n)$ </tex-math></inline-formula> . The experimental results on three benchmark datasets illustrate the superiority of the FADCH method.

Common canonical variate analysis (CCVA) based modeling and monitoring for multimode processes

Multiple operating modes are common in industrial processes due to feed stock alterations, product specifications, working environment changes and so on. Although different modes show different behaviors, some underlying process characteristics may stay invariable as mode changes, which reveal the essence information of the process. In this work, the issue of multimode process monitoring is studied with subspace separation, in which each mode is divided into the common subspace and specific subspace. A modified canonical variate analysis (CVA), termed as common CVA (CCVA), is put forward to extract the mode-common features based on joint approximate diagonalization method. The concatenation of all Hankel matrices is analyzed to find a common orthonormal set eigenvectors by minimization of joint diagonality criterion. Then, the remaining part of each mode is regarded as local specific subspace, which provides more representative information in each different mode. CVA algorithm is applied to build multiple local models based on mode-specific information. Two case studies, a numerical example and Tennessee Eastman (TE) process, are provided to validate the feasibility and effectiveness of the proposed method in monitoring abnormal operation for multimode processes.

Variational Autoencoder with CCA for Audio–Visual Cross-modal Retrieval

Cross-modal retrieval is to utilize one modality as a query to retrieve data from another modality, which has become a popular topic in information retrieval, machine learning, and databases. Finding a method to effectively measure the similarity between different modality data is the major challenge of cross-modal retrieval. Although several research works have calculated the correlation between different modality data via learning a common subspace representation, the encoder’s ability to extract features from multi-modal information is not satisfactory. In this article, we present a novel variational autoencoder architecture for audio–visual cross-modal retrieval by learning paired audio–visual correlation embedding and category correlation embedding as constraints to reinforce the mutuality of audio–visual information. On the one hand, audio encoder and visual encoder separately encode audio data and visual data into two different latent spaces. Further, two mutual latent spaces are respectively constructed by canonical correlation analysis. On the other hand, probabilistic modeling methods are used to deal with possible noise and missing information in the data. Additionally, in this way, the cross-modal discrepancies from intra-modal and inter-modal information are simultaneously eliminated in the joint embedding subspace. We conduct extensive experiments over two benchmark datasets. The experimental results confirm that the proposed architecture is effective in learning audio–visual correlation and is appreciably better than the existing cross-modal retrieval methods.

Learning Explicit and Implicit Dual Common Subspaces for Audio-visual Cross-modal Retrieval

Audio-visual tracks in video contain rich semantic information with potential in many applications and research. Since the audio-visual data have inconsistent distributions and because of the heterogeneous nature of representations, the heterogeneous gap between modalities makes them impossible to compare directly. To bridge the modality gap, a frequently adopted approach is to simultaneously project audio-visual data into a common subspace to capture the commonalities and characteristics of modalities for measurement, which has been extensively studied in relation to the issues of modality-common and modality-specific feature learning in previous research. However, it is difficult for existing methods to address the tradeoff between both issues; e.g., the modality-common feature is learned from the latent commonalities of audio-visual data or the correlated features as aligned projections, in which the modality-specific feature can be lost. To solve the tradeoff, we propose a novel end-to-end architecture, which synchronously projects audio-visual data into the explicit and the implicit dual common subspaces. The explicit subspace is used to learn modality-common features and reduce the modality gap of explicitly paired audio-visual data, where the representation-specific details are abandoned to retain the common underlying structure of audio-visual data. The implicit subspace is used to learn modality-specific features, where each modality privately pulls apart the feature distances between different categories to maintain the category-based distinctions, by minimizing the distance between audio-visual features and corresponding labels. The comprehensive experimental results on two audio-visual datasets, VEGAS and AVE, demonstrate that our proposed model for using two different common subspaces for audio-visual cross-modal learning is effective and significantly outperforms the state-of-the-art cross-modal models that learn features from a single common subspace by 4.30% and 2.30% in terms of average MAP on the VEGAS and AVE datasets, respectively.

Discovering Community Structure in Multiplex Networks via a Co-Regularized Robust Tensor-Based Spectral Approach

Complex networks arise in various fields, such as biology, sociology and communication, to model interactions among entities. Entities in many real-world systems exhibit different types of interactions, which requires modeling these type of systems properly. Multiplex networks are used to model these systems, as they can reflect the nodes’ pair-wise interactions as multiple distinct types of links across layers. Community detection is a widely studied application in network analysis as it provides insights into the structure and organization of the network. Even though multiple algorithms have been developed in the community detection field, many of them have a limited performance in the presence of noise. In this article, we develop a novel algorithm that combines tensor low-rank representation, spectral clustering and distance regularization to improve the accuracy in discovering communities in multiplex networks. The low-rank representation leads to reducing the noise and errors existing in the network and the optimization of an accurate consensus set of eigenvectors that reveals the communities in the network. Moreover, the proposed approach balances the agreement between the eigenvectors of each layer, i.e., individual subspaces, and the consensus set of eigenvectors, i.e., common subspaces, by minimizing the projection distance between them. The common and individual subspaces are computed efficiently through Tucker decomposition and modified spectral clustering, respectively. Finally, multiple experiments are conducted on real and simulated networks to evaluate the proposed approach and compare it to state-of-the-art algorithms. The proposed approach shows its robustness and efficiency in discovering the communities in multiplex networks.

Domain adaptation for few-sample nonlinear process monitoring with deep networks

Multiple modes are ubiquitous in current industrial processes, and the amount of historical data contained in different modes may vary considerably. Insufficient data can easily lead to cold start problems when building a fault detection model for a particular mode. To solve this problem, while considering the similarity and differences between multiple modes, a deep model using domain adaptation based on feature separation is proposed for nonlinear process monitoring with few samples. The model extracts common features from modes and the data deficiency is compensated by transferring the domain knowledge from the source to the common features. On the other hand, to avoid missing useful information by focusing only on common features, the model also extracts the specific features of the target domain. Thus, monitoring performance is improved with the help of domain adaptation while taking into account the specific characteristics of the target domain. Furthermore, three detection indices are designed to monitor the common feature subspace, the specific feature subspace, and the residual subspace, respectively. The benefit of this is allowing more diagnostic information to be obtained when a fault occurs. The proposed method was tested with a numerical example and a real industrial hydrocracking process to verify the detection effectiveness.

Multiview Jointly Sparse Discriminant Common Subspace Learning

Multiview data leads to the demand for classifying samples from various views, and the large gap between different views makes the classification task challenging. Recently, researchers have extended linear discriminant analysis (LDA) to multi-view scenarios. However, the extended methods are generally associated with the small-class problem, that is, the projection size is limited by the number of classes. In addition, they are sensitive to variations in images or outliers. To solve these problems, this study proposes a generalized robust multiview discriminant analysis (GRMDA) to obtain a linear transform for each view and for learning multiview jointly sparse discriminant common subspace. GRMDA aims to achieve both maximal between-class and minimal within-class variation for data from multiple views in a common space. Instead of formulating the ratio trace problem, we reformulate GRMDA inspired by maximum margin criterion (MMC) to address the small-class problem. Moreover, the proposed method achieves stronger robustness by reconstructing the within-class and between-class scatter terms from the definition of L2,1 norm. Furthermore, GRMDA ensures joint sparsity using the L2,1 norm-based regularization term. Additionally, we present an iterative algorithm, convergence proof, and complexity analysis. Experiments on six popular databases, that is, COIL100, USPS/MNIST, Extended Yale Face B, AR, BBCSport, and multiple feature datasets, were conducted to evaluate the performance of GRMDA against the state-of-the-art multiview methods. The experimental results demonstrate that the proposed method can achieve a significant performance with strong robustness and fast convergence.

CDFSL: Image Registration for Spaceborne Hyperspectral and Multispectral Data Having Large Spatial-Resolution Difference

Image registration aims to eliminate the geometric deviation between multi-source data with the same range, and to promote the collaborative application of data. In recent years, spaceborne hyperspectral (HS) and multispectral (MS) data have been widely used in Earth observation. However, the difference in the number of bands, spatial resolution, and spectral resolution puts forward higher requirements on the registration algorithm. The key to HS and MS image registration is to extract more common key points, weaken and eliminate the difference of radiation and spatial texture information to build superior descriptors, and achieve high-precision matching of key points. This paper introduces a new robust HS and MS registration method based on common deep feature subspaces. We first construct the common deep feature subspaces extraction network to extract consistent edge features and common subspace images of the image pair. Then, Harris algorithm is used to extract key points from consistent edge features between images, which reduces the impact of spatial resolution differences between images. Besides, the SIFT descriptor and subspace images are used to describe key points, which reduces the impact of radiation differences between images. Finally, Euclidean distance is used for the initial matching of key points, and the affine matrix is calculated after the outliers are eliminated, and image registration is performed. We perform experiments on spaceborne HS and MS datasets of different spatial resolutions and comparisons with state-of-the-art methods. Experimental results show that our method can obtain satisfactory registration results and is robust.

UCSL: Toward Unsupervised Common Subspace Learning for Cross-Modal Image Classification

The emerging research line of cross-modal learning focuses on the issue of transferring feature representation manner learned from limited multimodal data with labelings to the testing phase with partial modalities. This is essentially common and practical in the remote sensing community when only modal-incomplete data are in users’ hands due to inevitable imaging or access restrictions under large-scale observation scenarios. However, most of the existing cross-modal learning methods have been designed with exclusive reliance on labeling, which can be either limited or noisy due to their costly production. To address this issue, we explore in this paper the possibility to learn cross-modal feature representation in an unsupervised fashion. By integrating the multimodal data into a fully recombined matrix form, we propose 1) the use of common subspace representation as the regression target instead of conventionally adopted binary labels, and 2) the orthogonality and manifold alignment regularization terms to shrink the solution space whilst preserving the pairwise manifold correlations. Through this manner, the modality-specific and mutual latent representations in this common subspace as well as their corresponding projections can be learned simultaneously and their optimums can be efficiently reached through a nearly one-step computation with the help of Eigen decomposition. Finally, we show the superiority of our method through extensive image classification experiments on three multimodal datasets with four remotely sensed modalities involved (i.e., hyperspectral, multispectral, synthetic aperture radar, and light detection and ranging data). The code and dataset will be made freely available at https://github.com/jingyao16/UCSL after a possible publication to encourage the reproduction of our method and further use.

Algorithms for simultaneous block triangularization and block diagonalization of sets of matrices

&lt;abstract&gt;&lt;p&gt;In a recent paper, a new method was proposed to find the common invariant subspaces of a set of matrices. This paper investigates the more general problem of putting a set of matrices into block triangular or block-diagonal form simultaneously. Based on common invariant subspaces, two algorithms for simultaneous block triangularization and block diagonalization of sets of matrices are presented. As an alternate approach for simultaneous block diagonalization of sets of matrices by an invertible matrix, a new algorithm is developed based on the generalized eigenvectors of a commuting matrix. Moreover, a new characterization for the simultaneous block diagonalization by an invertible matrix is provided. The algorithms are applied to concrete examples using the symbolic manipulation system Maple.&lt;/p&gt;&lt;/abstract&gt;

CAMR: cross-aligned multimodal representation learning for cancer survival prediction.

Accurately predicting cancer survival is crucial for helping clinicians to plan appropriate treatments, which largely improves the life quality of cancer patients and spares the related medical costs. Recent advances in survival prediction methods suggest that integrating complementary information from different modalities, e.g. histopathological images and genomic data, plays a key role in enhancing predictive performance. Despite promising results obtained by existing multimodal methods, the disparate and heterogeneous characteristics of multimodal data cause the so-called modality gap problem, which brings in dramatically diverse modality representations in feature space. Consequently, detrimental modality gaps make it difficult for comprehensive integration of multimodal information via representation learning and therefore pose a great challenge to further improvements of cancer survival prediction. To solve the above problems, we propose a novel method called cross-aligned multimodal representation learning (CAMR), which generates both modality-invariant and -specific representations for more accurate cancer survival prediction. Specifically, a cross-modality representation alignment learning network is introduced to reduce modality gaps by effectively learning modality-invariant representations in a common subspace, which is achieved by aligning the distributions of different modality representations through adversarial training. Besides, we adopt a cross-modality fusion module to fuse modality-invariant representations into a unified cross-modality representation for each patient. Meanwhile, CAMR learns modality-specific representations which complement modality-invariant representations and therefore provides a holistic view of the multimodal data for cancer survival prediction. Comprehensive experiment results demonstrate that CAMR can successfully narrow modality gaps and consistently yields better performance than other survival prediction methods using multimodal data. CAMR is freely available at https://github.com/wxq-ustc/CAMR. Supplementary data are available at Bioinformatics online.

Multi-View Diffusion Process for Spectral Clustering and Image Retrieval.

This paper presents a novel approach to multi-view graph learning that combines weight learning and graph learning in an alternating optimization framework. Multi-view graph learning refers to the problem of constructing a unified affinity graph using heterogeneous sources of data representation, which is a popular technique in many learning systems where no prior knowledge of data distribution is available. Our approach is based on a fusion-and-diffusion strategy, in which multiple affinity graphs are fused together via a weight learning scheme based on the unsupervised graph smoothness and utilised as a consensus prior to the diffusion. We propose a novel multi-view diffusion process that learns a manifold-aware affinity graph by propagating affinities on tensor product graphs, leveraging high-order contextual information to enhance pairwise affinities. In contrast to existing multi-view graph learning approaches, our approach is not limited by the quality of initial graphs or the assumption of a latent common subspace among multiple views. Instead, our approach is able to identify the consistency among views and fuse multiple graphs adaptively. We formulate both weight learning and diffusion-based affinity learning in a unified framework and propose an alternating optimization solver that is guaranteed to converge. The proposed approach is applied to image retrieval and clustering tasks on 16 real-world datasets. Extensive experimental results demonstrate that our approach outperforms state-of-the-art methods for both retrieval and clustering on 13 out of 16 datasets.

Dynamic classifier approximation for unsupervised domain adaptation

In recent years, domain adaptation (DA) method has been proposed to solve the problem of domain shift between the training set and test set. However, most feature-based unsupervised domain adaptation methods have a weakness of strict requirement on the transformation matrix and they are not integrated with classifier. Therefore, this paper proposes a novel dynamic classifier approximation (DCA) method for unsupervised domain adaptation. Specifically, the proposed DCA, which combines classifier method to relax the requirements of transformation matrix, learns domain invariant features from structure and data level. 1) At the structure level, the low-rank representation, sparse representation and manifold learning are combined to preserve the structure information of data in the original feature space. 2) At the data level, distribution alignments are adopted to minimize the distribution difference of the source and target domains in the common subspace, which is helpful to reduce the negative impact caused by domain shift. In addition, for learning the common subspace more flexibly, a dynamic classifier is introduced to reduce the strict requirements on projection matrix. This paper carries out a large number of experiments on four data sets, and the experimental results show that the proposed method is superior to many of the latest domain adaptation methods, which can indicate its effectiveness.

An Adaptive EEG Classification Algorithm Based on CSSD and ELM_Kernel for Small Training Samples.

Rehabilitation technologies based on brain-computer interface (BCI) have become a promising approach for patients with dyskinesia to regain movement. In BCI experiment, there is often a necessary stage of calibration measurement before the feedback applications. To reduce the time required for initial training, it is of great importance to have a method which can learn to classify electroencephalogram (EEG) signals with a little amount of training data. In this paper, the novel combination of feature extraction and classification algorithm is proposed for classification of EEG signals with a small number of training samples. For feature extraction, the motor imagery EEG signals are pre-processed, and a relative distance criterion is defined to select the optimal combination of channels. Subsequently, common spatial subspace decomposition (CSSD) algorithm and extreme learning machine with kernel (ELM_Kernel) algorithm are used to perform the types of tasks classification of motor imagery EEG signals. Simulation results demonstrate that the proposed method produces a high average classification accuracy of 99.1% on BCI Competition III dataset IVa and 76.92% on BCI Competition IV dataset IIa outperforming state-of-the-art algorithms and obtains a good classification accuracy.

Learning specific and conserved features of multi-layer networks

Complex systems are composed of multiple types of interactions, where each type of interaction is encoded in a layer, resulting in multi-layer networks. Detecting layer-specific modules in multi-layer networks are for revealing the functions and structure of systems. However, current algorithms are criticized for failing to quantify and balance the specificity and connectivity of communities in multi-layer networks, resulting in undesirable performance. To address these problems, we propose a joint Learning Specific and Conserved features for Clustering in multi-layer networks (called LSCC), where features of vertices simultaneously characterize the shared and layer-specific structure of networks. Specifically, LSCC jointly factorizes multi-layer networks by projecting all layers into a common subspace with nonnegative matrix factorization, where the structure of various layers is represented. Then, LSCC decomposes features of vertices into the conserved and specific parts, where the specificity of vertices of each layer is explicitly quantified. To balance the specificity and connectivity of modules, LSCC joint learns feature extraction and subspace clustering, which is formulated as an optimization problem. The experimental results on 8 datasets demonstrate that the proposed algorithm significantly outperforms the baselines on various measurements.

WAD-CMSN: Wasserstein distance-based cross-modal semantic network for zero-shot sketch-based image retrieval

Zero-shot sketch-based image retrieval (ZSSBIR) aims at retrieving natural images given free hand-drawn sketches that may not appear during training. Previous approaches used semantic aligned sketch-image pairs or utilized memory expensive fusion layer for projecting the visual information to a low-dimensional subspace, which ignores the significant heterogeneous cross-domain discrepancy between highly abstract sketch and relevant image. This may yield poor performance in the training phase. To tackle this issue and overcome this drawback, we propose a Wasserstein distance-based cross-modal semantic network (WAD-CMSN) for ZSSBIR. Specifically, it first projects the visual information of each branch (sketch, image) to a common low-dimensional semantic subspace via Wasserstein distance in an adversarial training manner. Furthermore, a novel identity matching loss is employed to select useful features, which can not only capture complete semantic knowledge, but also alleviate the over-fitting phenomenon caused by the WAD-CMSN model. Experimental results on the challenging Sketchy (Extended) and TU-Berlin (Extended) datasets indicate the effectiveness of the proposed WAD-CMSN model over several competitors.

Parallel learned generative adversarial network with multi-path subspaces for cross-modal retrieval

Cross-modal retrieval aims to narrow the heterogeneity gap between different modalities, such as retrieving images through texts or vice versa. One of the key challenges of cross-modal retrieval is the inconsistent distribution across diverse modalities. Most existing methods tend to construct a common representation subspace to overcome the challenge. However, the supervision information is not fully explored in most single-path cross-modal learning approaches. In this paper, we present a novel Parallel Learned generative adversarial network with Multi-path Subspaces (PLMS) for cross-modal retrieval. PLMS is a parallel learned architecture that aims to capture more effective information in an end-to-end trained cross-modal retrieval model. To be specific, a dual-branch network is constructed in the modality-specific generator, thereby the overall framework learns two common subspaces to emphasize discrepant supervision information and preserve more effective transformed features. We further design two objective functions for the training of the dual branches in generators. Through joint training, the feature representations generated by dual branches in a specific modality are fused for similarity measurement between modalities. To avoid redundancy and overlap during fusion, a Multi-source Domain Balancing (MDB) mechanism is presented to explore the contribution of the two specific-task branches. Extensive experiments show that our proposed method is effective and achieves state-of-the-art results on four widely-used databases.

A converse statement to Hutchinson’s theorem and a dimension gap for self-affine measures

A well-known theorem of J. E. Hutchinson states that if an iterated function system consists of similarity transformations and satisfies the open set condition then its attractor supports a self-similar measure with Hausdorff dimension equal to the similarity dimension. In this article we prove the following result which may be regarded as a form of partial converse: if an iterated function system consists of invertible affine transformations whose linear parts do not preserve a common invariant subspace, and its attractor supports a self-affine measure with Hausdorff dimension equal to the affinity dimension, then the system necessarily consists of similarity transformations. We obtain this result by showing that the equilibrium measures of an affine iterated function system are never Bernoulli measures unless the system either is reducible or consists of similarity transformations. The proof builds on earlier results in the thermodynamic formalism of affine iterated function systems due to Bochi, Feng, Käenmäki, Shmerkin and the first named author and also relies on the work of Benoist on the spectral properties of Zariski-dense subsemigroups of reductive linear groups.