Common Embeddings Research Articles

Towards Adaptive Information Fusion in Graph Convolutional Networks

Graph Convolutional Networks (GCNs) have gained great popularity in tackling various analytic tasks on graph and network data. However, some recent studies raise concerns about whether GCNs can optimally integrate node features and topological structures in a complex graph. In this paper, we first present an experimental investigation. Surprisingly, our experimental results clearly show that the capability of the state-of-the-art GCNs in fusing node features and topological structures is distant from optimal or even satisfactory. The weakness may severely hinder the capability of GCNs in some classification tasks, since GCNs may not be able to adaptively learn some deep correlation information between topological structures and node features. Can we remedy the weakness and design a new type of GCNs that can retain the advantages of the state-of-the-art GCNs and, at the same time, enhance the capability of fusing topological structures and node features substantially? We tackle the challenge and propose an <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</b> daptive <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</b> ulti-channel <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G</b> raph <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</b> onvolutional <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</b> etwork for semi-supervised classification ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AM-GCN</b> ). The central idea is that we extract the specific and common embeddings from node features, topological structures, and their combinations simultaneously, and use the attention mechanism to learn adaptive importance weights of the embeddings. However, considering that the input topology and feature structure in AM-GCN are still predefined and fixed, once the properties of graph structures are not consistent with tasks, the fusion performance of AM-GCN will be hindered from the beginning. Therefore, we need to adjust the structure and further propose the <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</b> abel <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">P</b> ropagation guided <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</b> ulti-channel <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G</b> raph <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</b> onvolutional <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</b> etwork ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LPM-GCN</b> ). LPM-GCN introduces edge weights learning on both topology and feature spaces to improve structural homophily, which can better promote the fusion process of graph convolutional networks. Our extensive experiments on benchmark data sets clearly show that our proposed models extract the most correlated information from both node features and topological structures substantially, and improves the classification accuracy with a clear margin.

MSTAD: A masked subspace-like transformer for multi-class anomaly detection

Unsupervised anomaly detection techniques, which do not rely on prior knowledge of anomalies, have attracted considerable attention in the field of industrial surface inspection. However, existing approaches commonly employ separate models for each product class, resulting in substantial storage requirements and inefficiency during the training phase Accordingly, we propose a masked subspace-like transformer for multi-class anomaly detection (MSTAD), which employs an encoder–decoder architecture to reconstruct the pre-trained image features by recognizing the greater resilience of high-level semantic features compared with low-level pixel features To address the issue of identity mapping, which refers to the tendency of a model to overgeneralize when reconstructing abnormal samples, MSTAD integrates two essential components: the multi-layer subspace-like embedding (MLSE) module and random block mask (RBM) method The MLSE module incorporates an attention mechanism to selectively emphasize the common embeddings associated with each class, thereby enhancing both the ability of the model to reconstruct anomalies as normal and its capacity for training RBM applies a random mask block mechanism to the pre-trained feature map to enhance the comprehension ability of the model and improve the reconstruction of normal features We conducted extensive experiments on the MVTec AD and BTAD datasets, and the results demonstrated that MSTAD outperformed previous state-of-the-art methods in terms of anomaly detection and localization performance for multi-class anomaly detection tasks.

Multichannel Domain Adaptation Graph Convolutional Networks-Based Fault Diagnosis Method and With Its Application

Intelligent fault diagnosis of the complex systems has made great progress based on the availability of massive labeled data. However, due to the diversity of working conditions and the lack of sufficient fault samples in practice, the generalization of the existing fault diagnosis methods are weak. To handle this issue, a multi-channel domain adaptation graph convolutional network method is proposed. In the proposed network, a feature mapping layer based on convolutional neural network is used firstly to extract features from input data, which then are transmitted to the graph generator to construct two association graphs. After that, three distributed graph convolutional networks are used to extract the specific and common embeddings from two association graphs and their combination. Meanwhile, to fuse these embeddings adaptively, an attention mechanism is used to learn importance weights. Besides, a domain discriminator is leveraged to reduce the distribution discrepancy of different data domains. Finally, a label classifier is used to output fault diagnosis results. Two experimental studies with different signal types show that the proposed method not only presents better diagnosis performance than existing methods with few samples, but also can extract domain-invariant features for cross-domain under varying working conditions.

Joint variational autoencoders for multimodal imputation and embedding.

Single-cell multimodal datasets have measured various characteristics of individual cells, enabling a deep understanding of cellular and molecular mechanisms. However, multimodal data generation remains costly and challenging, and missing modalities happen frequently. Recently, machine learning approaches have been developed for data imputation but typically require fully matched multimodalities to learn common latent embeddings that potentially lack modality specificity. To address these issues, we developed an open-source machine learning model, Joint Variational Autoencoders for multimodal Imputation and Embedding (JAMIE). JAMIE takes single-cell multimodal data that can have partially matched samples across modalities. Variational autoencoders learn the latent embeddings of each modality. Then, embeddings from matched samples across modalities are aggregated to identify joint cross-modal latent embeddings before reconstruction. To perform cross-modal imputation, the latent embeddings of one modality can be used with the decoder of the other modality. For interpretability, Shapley values are used to prioritize input features for cross-modal imputation and known sample labels. We applied JAMIE to both simulation data and emerging single-cell multimodal data including gene expression, chromatin accessibility, and electrophysiology in human and mouse brains. JAMIE significantly outperforms existing state-of-the-art methods in general and prioritized multimodal features for imputation, providing potentially novel mechanistic insights at cellular resolution.

MGAE-DC: Predicting the synergistic effects of drug combinations through multi-channel graph autoencoders.

Accurate prediction of synergistic effects of drug combinations can reduce the experimental costs for drug development and facilitate the discovery of novel efficacious combination therapies for clinical studies. The drug combinations with high synergy scores are regarded as synergistic ones, while those with moderate or low synergy scores are additive or antagonistic ones. The existing methods usually exploit the synergy data from the aspect of synergistic drug combinations, paying little attention to the additive or antagonistic ones. Also, they usually do not leverage the common patterns of drug combinations across different cell lines. In this paper, we propose a multi-channel graph autoencoder (MGAE)-based method for predicting the synergistic effects of drug combinations (DC), and shortly denote it as MGAE-DC. A MGAE model is built to learn the drug embeddings by considering not only synergistic combinations but also additive and antagonistic ones as three input channels. The later two channels guide the model to explicitly characterize the features of non-synergistic combinations through an encoder-decoder learning process, and thus the drug embeddings become more discriminative between synergistic and non-synergistic combinations. In addition, an attention mechanism is incorporated to fuse each cell-line's drug embeddings across various cell lines, and a common drug embedding is extracted to capture the invariant patterns by developing a set of cell-line shared decoders. The generalization performance of our model is further improved with the invariant patterns. With the cell-line specific and common drug embeddings, our method is extended to predict the synergy scores of drug combinations by a neural network module. Experiments on four benchmark datasets demonstrate that MGAE-DC consistently outperforms the state-of-the-art methods. In-depth literature survey is conducted to find that many drug combinations predicted by MGAE-DC are supported by previous experimental studies. The source code and data are available at https://github.com/yushenshashen/MGAE-DC.

Text Semantic Fusion Relation Graph Reasoning for Few-Shot Object Detection on Remote Sensing Images

Most object detection methods based on remote sensing images are generally dependent on a large amount of high-quality labeled training data. However, due to the slow acquisition cycle of remote sensing images and the difficulty in labeling, many types of data samples are scarce. This makes few-shot object detection an urgent and necessary research problem. In this paper, we introduce a remote sensing few-shot object detection method based on text semantic fusion relation graph reasoning (TSF-RGR), which learns various types of relationships from common sense knowledge in an end-to-end manner, thereby empowering the detector to reason over all classes. Specifically, based on the region proposals provided by the basic detection network, we first build a corpus containing a large number of text language descriptions, such as object attributes and relations, which are used to encode the corresponding common sense embeddings for each region. Then, graph structures are constructed between regions to propagate and learn key spatial and semantic relationships. Finally, a joint relation reasoning module is proposed to actively enhance the reliability and robustness of few-shot object feature representation by focusing on the degree of influence of different relations. Our TSF-RGR is lightweight and easy to expand, and it can incorporate any form of common sense information. Sufficient experiments show that the text information is introduced to deliver excellent performance gains for the baseline model. Compared with other few-shot detectors, the proposed method achieves state-of-the-art performance for different shot settings and obtains highly competitive results on two benchmark datasets (NWPU VHR-10 and DIOR).

MAMF-GCN: Multi-scale adaptive multi-channel fusion deep graph convolutional network for predicting mental disorder

PurposeExisting diagnoses of mental disorders rely on symptoms, patient descriptions, and scales, which are not objective enough. We attempt to explore an objective diagnostic method on fMRI data. Graph neural networks (GNN) have been paid more attention recently because of their advantages in processing unstructured relational data, especially for fMRI data. However, how to deeply embed and well-integrate with different modalities and scales on GNN is still a challenge. Instead of reaching a high degree of fusion, existing GCN methods simply combine image and non-image data. Most graph convolutional network (GCN) models use shallow structures, making it challenging to learn about potential information. Furthermore, current graph construction approaches usually use a single specific brain atlas, limiting the analysis and results. MethodIn this paper, a multi-scale adaptive multi-channel fusion deep graph convolutional network based on an attention mechanism (MAMF-GCN) is proposed to better integrate features of modalities and different atlas by exploiting multi-channel correlation. An encoder automatically combines one channel with non-imaging data to generate similarity weights between subjects using a similarity perception mechanism. Other channels generate multi-scale imaging features of fMRI data after processing in the different atlas. Multi-modal information is fused using an adaptive convolution module that applies a deep graph convolutional network (GCN) to extract information from richer hidden layers. ResultsTo demonstrate the effectiveness of our approach, we evaluate the performance of the proposed method on the Autism Brain Imaging Data Exchange (ABIDE) dataset and the Major Depressive Disorder (MDD) dataset. The experimental result shows that the proposed method outperforms many state-of-the-art methods in node classification performance. An extensive group of experiments on two disease prediction tasks demonstrates that the performance of the proposed MAMF-GCN on MDD/ABIDE dataset is improved by 3.37%–39.83% and 12.59%–32.92%, respectively. Moreover, our proposed method has also shown very effective performance in real-life clinical diagnosis. The comprehensive experiments demonstrate that our method is effective for node classification with brain disorders diagnosis. ConclusionThe proposed MAMF-GCN method simultaneously extracts specific and common embeddings from the topology composed of multi-scale imaging features, phenotypic information, and their combinations, then learning adaptive embedding weights by attention mechanism, which can capture and fuse the multi-scale essential embeddings to improve the classification performance of brain disorder diagnosis.

One-step multi-view spectral clustering by learning common and specific nonnegative embeddings

Multi-view spectral clustering is a hot research area which has attracted increasing attention. Most existing multi-view spectral clustering methods utilize a two-step strategy. The first step obtains a common embedding by fusing spectral embeddings of different views, and the second step conducts hard clustering, such as K-means or spectral rotation, on the common embedding. Because the goal of the first step is not obtaining optimal clustering result, and the requirement to post-processing makes the final clustering result uncertain. In this paper, we propose a novel one-step multi-view spectral clustering method, in which the spectral embedding and nonnegative embedding are unified into one framework. Therefore, our method can avoid the uncertainty brought by post-processing and obtain optimal clustering result. Moreover, the nonnegative embedding is divided into two parts. The common nonnegative embedding indicates the shared cluster structure, and the specific nonnegative embedding indicates the exclusive cluster structure of each view. Hence, our method can well tackle with noises and outliers of different views. Furthermore, an alternating iterative algorithm is used to solve the joint optimization problem. Extensive experimental results on four real-world datasets have demonstrated the effectiveness of the proposed method.

Group-Wise Hub Identification by Learning Common Graph Embeddings on Grassmannian Manifold.

Human brain is a complex yet economically organized system, where a small portion of critical hub regions support the majority of brain functions. The identification of common hub nodes in a population of networks is often simplified as a voting procedure on the set of identified hub nodes across individual brain networks, which ignores the intrinsic data geometry and partially lacks the reproducible findings in neuroscience. Hence, we propose a first-ever group-wise hub identification method to identify hub nodes that are common across a population of individual brain networks. Specifically, the backbone of our method is to learn common graph embedding that can represent the majority of local topological profiles. By requiring orthogonality among the graph embedding vectors, each graph embedding as a data element is residing on the Grassmannian manifold. We present a novel Grassmannian manifold optimization scheme that allows us to find the common graph embeddings, which not only identify the most reliable hub nodes in each network but also yield a population-based common hub node map. Results of the accuracy and replicability on both synthetic and real network data show that the proposed manifold learning approach outperforms all hub identification methods employed in this evaluation.

Sentiment aware word embeddings using refinement and senti-contextualized learning approach

Most pre-trained word embeddings are achieved from context-based learning algorithms trained over a large text corpus. This leads to learning similar vectors for words that share most of their contexts, while expressing different meanings. Therefore, the complex characteristics of words cannot be fully learned by using such models. One of the natural language processing applications that suffers from this problem is sentiment analysis. In this task, two words with opposite sentiments are not distinguished well by using common pre-trained word embeddings. This paper addresses this problem and proposes two simple, but empirically effective, approaches to learn word embeddings for sentiment analysis. The both approaches exploit sentiment lexicons and take into account the polarity of words in learning word embeddings. While the first approach encodes the sentiment information of words into existing pre-trained word embeddings, the second one builds synthetic sentimental contexts for embedding models along with other semantic contexts. The word embeddings obtained from the both approaches are evaluated on several sentiment classification tasks using Skip-gram and GloVe models. Results show that both approaches improve state-of-the-art results using basic deep learning models over sentiment analysis benchmarks.

Aligned attention for common multimodal embeddings

Deep learning has been attributed to incredible advances in computer vision, natural language processing, and general pattern understanding. Recent discoveries have enabled efficient vector representations of both visual and written stimuli. Robustly transferring between the two modalities remains a challenge that could yield benefits for search, retrieval, and storage applications. We introduce a simple, yet highly effective approach for building a connection space where natural language sentences are tightly coupled with visual data. In this connection space, similar concepts lie close, whereas dissimilar concepts lie far apart, irrespective of their modality. We introduce an attention mechanism to align multimodal embeddings that are learned through a multimodal metric loss function. We evaluate the learned common vector space on multiple image–text datasets—Pascal Sentences, NUS-WIDE-10k, XMediaNet, Flowers, and Caltech-UCSD Birds. We extend our method to five different modalities of image, sentence, audio, video, and 3D models to demonstrate cross-modal retrieval on the XMedia dataset. We obtain state-of-the-art retrieval and zero-shot retrieval across all datasets.

Adversarial task-specific learning

In this paper, we investigate a principle way to learn a common feature space for data of different modalities (e.g. image and text), so that the similarity between different modal items can be directly measured for benefiting cross-modal retrieval task. To effectively keep semantic/distribution consistent for common feature embeddings, we propose a new Adversarial Task-Specific Learning (ATSL) approach to learn distinct embeddings for different retrieval tasks, i.e. images retrieve texts (I2T) or texts retrieve images (T2I). In particular, the proposed ATSL is with the following advantages: (a) semantic attributes are leveraged to encourage the learned common feature embeddings of couples to be semantic consistent; (b) adversarial learning is applied to relieve the inconsistent distribution of common feature embeddings for different modalities; (c) triplet optimization is employed to guarantee that similar items from different modalities are with smaller distances in the learned common space compared with the dissimilar ones; (d) task-specific learning produces better optimized common feature embeddings for different retrieval tasks. Our ATSL is embedded in a deep neural network, which can be learned in an end-to-end manner. We conduct extensive experiments on two popular benchmark datasets, e.g. Flickr30K and MS COCO. We achieve R@1 accuracy of 57.1% and 38.4% for I2T and 56.5% and 38.6% T2I on MS COCO and Flickr30K respectively, which are the new state-of-the-arts.

Image-to-video person re-identification with cross-modal embeddings

Despite the great progress achieved, image-to-video person re-identification is still challenging in the cross-modal scenario. Currently, state-of-the-art approaches mainly concentrate on the task-specific data, neglecting the extra information from the different but related tasks. In this paper, we propose an end-to-end neural network framework for image-to-video person re-identification with cross-modal embeddings learned from extra information. Concretely speaking, cross-modal embedding layers from image captioning and video captioning models, are incorporated to learn common latent embeddings for multiple modalities. The learned multimodal embeddings are expected to focus on person’s prominent distinctions, due to textual descriptive information generally paying close attention to person’s explicit characteristics. Apart from that, our proposed framework resorts to CNNs and LSTMs for extracting visual and spatiotemporal features, and combines the strengths of identification and verification model to improve the discriminative ability of the learned features. The experimental results demonstrate the effectiveness of our framework on narrowing down the gap between heterogeneous data and obtaining observable improvement in the image-to-video person re-identification task.

Zero-Shot Cross-Media Embedding Learning With Dual Adversarial Distribution Network

Existing cross-media retrieval methods are mainly based on the condition where the training set covers all the categories in the testing set, which lack extensibility to retrieve data of new categories. Thus, zero-shot cross-media retrieval has been a promising direction in practical application, aiming to retrieve data of new categories (unseen categories), only with data of limited known categories (seen categories) for training. It is challenging for not only the heterogeneous distributions across different media types, but also the inconsistent semantics across seen and unseen categories need to be handled. To address the above issues, we propose dual adversarial distribution network (DADN) , to learn common embeddings and explore the knowledge from word-embeddings of different categories. The main contributions are as follows. First, zero-shot cross-media dual generative adversarial networks architecture is proposed, in which two kinds of generative adversarial networks (GANs) for common embedding generation and representation reconstruction form dual processes. The dual GANs mutually promote to model semantic and underlying structure information, which generalizes across different categories on heterogeneous distributions and boosts correlation learning. Second, distribution matching with maximum mean discrepancy criterion is proposed to combine with dual GANs, which enhances distribution matching between common embeddings and category word-embeddings. Finally, adversarial inter-media metric constraint is proposed with an inter-media loss and a quadruplet loss, which further model the inter-media correlation information and improve semantic ranking ability. The experiments on four widely used cross-media datasets demonstrate the effectiveness of our DADN approach.

On the embedding of the attractor generated by Navier-Stokes equations into finite dimensional spaces

For 2-D Navier-Stokes equations on a C 2 C^2 bounded domain Ω \Omega , a class of nonlinear homeomorphisms is constructed from the attractor of Navier-Stokes to curves in R N \mathbb {R}^N for sufficiently large N N . The construction uses an ε \varepsilon -net on Ω \Omega (so does not use the information “near” the boundary) and is more physically perceivable compared to abstract common embeddings.

Complex networks on hyperbolic surfaces

We explore a novel method to generate and characterize complex networks by means of their embedding on hyperbolic surfaces. Evolution through local elementary moves allows the exploration of the ensemble of networks which share common embeddings and consequently share similar hierarchical properties. This method provides a new perspective to classify network-complexity both on local and global scale. We demonstrate by means of several examples that there is a strong relation between the network properties and the embedding surface.

Graded state machines: The representation of temporal contingencies in simple recurrent networks

We explore a network architecture introduced by Elman (1990) for predicting successive elements of a sequence. The network uses the pattern of activation over a set of hidden units from time-step t-1, together with element t, to predict element t + 1. When the network is trained with strings from a particular finite-state grammar, it can learn to be a perfect finite-state recognizer for the grammar. When the net has a minimal number of hidden units, patterns on the hidden units come to correspond to the nodes of the grammars however, this correspondence is not necessary for the network to act as a perfect finite-state recognizer. Next, we provide a detailed analysis of how the network acquires its internal representations. We show that the network progressively encodes more and more temporal context by means of a probability analysis. Finally, we explore the conditions under which the network can carry information about distant sequential contingencies across intervening elements to distant elements. Such information is maintained with relative ease if it is relevant at each intermediate steps it tends to be lost when intervening elements do not depend on it. At first glance this may suggest that such networks are not relevant to natural language, in which dependencies may span indefinite distances. However, embeddings in natural language are not completely independent of earlier information. The final simulation shows that long distance sequential contingencies can be encoded by the network even if only subtle statistical properties of embedded strings depend on the early information. The network encodes long-distance dependencies by shading internal representations that are responsible for processing common embeddings in otherwise different sequences. This ability to represent simultaneously similarities and differences between several sequences relies on the graded nature of representations used by the network, which contrast with the finite states of traditional automata. For this reason, the network and other similar architectures may be called Graded State Machines.