Multimodal Graph Research Articles

Multi-modal Graph Fusion for Named Entity Recognition with Targeted Visual Guidance

Multi-modal named entity recognition (MNER) aims to discover named entities in free text and classify them into pre-defined types with images. However, dominant MNER models do not fully exploit fine-grained semantic correspondences between semantic units of different modalities, which have the potential to refine multi-modal representation learning. To deal with this issue, we propose a unified multi-modal graph fusion (UMGF) approach for MNER. Specifically, we first represent the input sentence and image using a unified multi-modal graph, which captures various semantic relationships between multi-modal semantic units (words and visual objects). Then, we stack multiple graph-based multi-modal fusion layers that iteratively perform semantic interactions to learn node representations. Finally, we achieve an attention-based multi-modal representation for each word and perform entity labeling with a CRF decoder. Experimentation on the two benchmark datasets demonstrates the superiority of our MNER model.

Knowledge-Enhanced Hierarchical Graph Transformer Network for Multi-Behavior Recommendation

Accurate user and item embedding learning is crucial for modern recommender systems. However, most existing recommendation techniques have thus far focused on modeling users' preferences over singular type of user-item interactions. Many practical recommendation scenarios involve multi-typed user interactive behaviors (e.g., page view, add-to-favorite and purchase), which presents unique challenges that cannot be handled by current recommendation solutions. In particular: i) complex inter-dependencies across different types of user behaviors; ii) the incorporation of knowledge-aware item relations into the multi-behavior recommendation framework; iii) dynamic characteristics of multi-typed user-item interactions. To tackle these challenges, this work proposes a Knowledge-Enhanced Hierarchical Graph Transformer Network (KHGT), to investigate multi-typed interactive patterns between users and items in recommender systems. Specifically, KHGT is build upon a graph-structured neural architecture to i) capture type-specific behavior semantics; ii) explicitly discriminate which types of user-item interactions are more important in assisting the forecasting task on the target behavior. Additionally, we further integrate the multi-modal graph attention layer with temporal encoding strategy, to empower the learned embeddings be reflective of both dedicated multiplex user-item and item-item collaborative relations, as well as the underlying interaction dynamics. Extensive experiments conducted on three real-world datasets show that KHGT consistently outperforms many state-of-the-art recommendation methods across various evaluation settings. Our implementation is available in https://github.com/akaxlh/KHGT.

Ensemble Manifold Regularized Multi-Modal Graph Convolutional Network for Cognitive Ability Prediction.

Multi-modal functional magnetic resonance imaging (fMRI) can be used to make predictions about individual behavioral and cognitive traits based on brain connectivity networks. To take advantage of complementary information from multi-modal fMRI, we propose an interpretable multi-modal graph convolutional network (MGCN) model, incorporating both fMRI time series and functional connectivity (FC) between each pair of brain regions. Specifically, our model learns a graph embedding from individual brain networks derived from multi-modal data. A manifold-based regularization term is enforced to consider the relationships of subjects both within and between modalities. Furthermore, we propose the gradient-weighted regression activation mapping (Grad-RAM) and the edge mask learning to interpret the model, which is then used to identify significant cognition-related biomarkers. We validate our MGCN model on the Philadelphia Neurodevelopmental Cohort to predict individual wide range achievement test (WRAT) score. Our model obtains superior predictive performance over GCN with a single modality and other competing approaches. The identified biomarkers are cross-validated from different approaches. This paper develops a new interpretable graph deep learning framework for cognition prediction, with the potential to overcome the limitations of several current data-fusion models. The results demonstrate the power of MGCN in analyzing multi-modal fMRI and discovering significant biomarkers for human brain studies.

Multimodal graph inference network for scene graph generation

A scene graph can describe images concisely and structurally. However, existing methods of scene graph generation have low capabilities of inferring certain relationships, because of the lack of semantic information and their heavy dependence on the statistical distribution of the training set. To alleviate the above problems, a Multimodal Graph Inference Network (MGIN), which includes two modules; Multimodal Information Extraction (MIE) and Target with Multimodal Feature Inference (TMFI), is proposed in this study. MGIN can increase the inference capability of triplets, especially for uncommon samples. In the proposed MIE module, the prior statistical knowledge of the training set is incorporated into the network in a reprocess to relieve the problem of overfitting to the training set. Visual and semantic features are extracted in the MIE module and fused as unified multimodal features in the TMFI module. These features are efficient for the inference module to increase the prediction capability of MGIN, especially for some uncommon samples. The proposed method achieves 27.0% average mean recall and 55.9% average recall, with improvements of 0.48% and 0.50%, respectively, compared with state-of-the-art methods. It also increases the average recall of 20 relationships with the lowest probability by 4.91%.

Longitudinal Speech Biomarkers for Automated Alzheimer's Detection

We introduce a novel audio processing architecture, the Open Voice Brain Model (OVBM), improving detection accuracy for Alzheimer's (AD) longitudinal discrimination from spontaneous speech. We also outline the OVBM design methodology leading us to such architecture, which in general can incorporate multimodal biomarkers and target simultaneously several diseases and other AI tasks. Key in our methodology is the use of multiple biomarkers complementing each other, and when two of them uniquely identify different subjects in a target disease we say they are orthogonal. We illustrate the OBVM design methodology by introducing sixteen biomarkers, three of which are orthogonal, demonstrating simultaneous above state-of-the-art discrimination for two apparently unrelated diseases such as AD and COVID-19. Depending on the context, throughout the paper we use OVBM indistinctly to refer to the specific architecture or to the broader design methodology. Inspired by research conducted at the MIT Center for Brain Minds and Machines (CBMM), OVBM combines biomarker implementations of the four modules of intelligence: The brain OS chunks and overlaps audio samples and aggregates biomarker features from the sensory stream and cognitive core creating a multi-modal graph neural network of symbolic compositional models for the target task. In this paper we apply the OVBM design methodology to the automated diagnostic of Alzheimer's Dementia (AD) patients, achieving above state-of-the-art accuracy of 93.8% using only raw audio, while extracting a personalized subject saliency map designed to longitudinally track relative disease progression using multiple biomarkers, 16 in the reported AD task. The ultimate aim is to help medical practice by detecting onset and treatment impact so that intervention options can be longitudinally tested. Using the OBVM design methodology, we introduce a novel lung and respiratory tract biomarker created using 200,000+ cough samples to pre-train a model discriminating cough cultural origin. Transfer Learning is subsequently used to incorporate features from this model into various other biomarker-based OVBM architectures. This biomarker yields consistent improvements in AD detection in all the starting OBVM biomarker architecture combinations we tried. This cough dataset sets a new benchmark as the largest audio health dataset with 30,000+ subjects participating in April 2020, demonstrating for the first time cough cultural bias.

Unsupervised Multi-modal Hashing for Cross-Modal Retrieval

The explosive growth of multimedia data on the Internet has magnified the challenge of information retrieval. Multimedia data usually emerges in different modalities, such as image, text, video, and audio. Unsupervised cross-modal hashing techniques that support searching among multi-modal data have gained importance in large-scale retrieval tasks because of the advantage of low storage cost and high efficiency. Current methods learn the hash functions by transforming high-dimensional data into discrete hash codes. However, the original manifold structure and semantic correlation are not preserved well in compact hash codes. We propose a novel unsupervised cross-modal hashing method to cope with this problem from two perspectives. On the one hand, the semantic correlation in textual space and the locally geometric structure in the visual space are reconstructed by unified hashing features seamlessly and simultaneously. On the other hand, the \(\ell _{2,1}\)-norm penalties are imposed on the projection matrices separately to learn the relevant and discriminative hash codes. The experimental results indicate that our proposed method achieves an improvement of 1%, 6%, 9%, and 2% over the best comparison method on the four publicly available datasets (WiKi, PASCAL-VOC, UCI Handwritten Digit, and NUS-WIDE), respectively. In conclusion, the proposed framework which combines hash functions learning and multimodal graph embedding is effective in learning hash codes and achieves superior retrieval performance compared to state-of-the-art methods.

Multisource Remote Sensing Data Classification With Graph Fusion Network

The land cover classification has been an important task in remote sensing. With the development of various sensors technologies, carrying out classification work with multisource remote sensing (MSRS) data has shown an advantage over using a single type of data. Hyperspectral images (HSIs) are able to represent the spectral properties of land cover, which is quite common for land cover understanding. Light detection and ranging (LiDAR) images contain altitude information of the ground, which is greatly helpful with urban scene analysis. Current HSI and LiDAR fusion methods perform feature extraction and feature fusion separately, which cannot well exploit the correlation of data sources. In order to make full use of the correlation of multisource data, an unsupervised feature extraction-fusion network for HSI and LiDAR, which utilizes feature fusion to guide the feature extraction procedure, is proposed in this article. More specifically, the network takes multisource data as input and directly output the unified fused feature. A multimodal graph is constructed for feature fusion, and graph-based loss functions including Laplacian loss and t-distributed stochastic neighbor embedding (t-SNE) loss are utilized to constrain the feature extraction network. Experimental results on several data sets demonstrate the proposed network can achieve more excellent classification performance than some state-of-the-art methods.

A diagnostic unified classification model for classifying multi-sized and multi-modal brain graphs using graph alignment

BackgroundPresence of multimodal brain graphs derived from different neuroimaging modalities is inarguably one of the most critical challenges in building unified classification models that can be trained and tested on any brain graph regardless of its size and the modality it was derived from. Existing methodsOne solution is to learn a model for each modality independently, which is cumbersome and becomes more time-consuming as the number of modalities increases. Another traditional solution is to build a model inputting multimodal brain graphs for the target prediction task; however, this is only applicable to datasets where all samples have joint neuro-modalities. New methodIn this paper, we propose to build a unified brain graph classification model trained on unpaired multimodal brain graphs, which can classify any brain graph of any size. This is enabled by incorporating a graph alignment step where all multi-modal graphs of different sizes and heterogeneous distributions are mapped to a common template graph. Next, we design a graph alignment strategy to the target fixed-size template and further apply linear discriminant analysis (LDA) to the aligned graphs as a supervised dimensionality reduction technique for the target classification task. ResultsWe tested our method on unpaired autistic and healthy brain connectomes derived from functional and morphological MRI datasets (two modalities). ConclusionOur results showed that our unified model method not only has great promise in solving such a challenging problem but achieves comparable performance to models trained on each modality independently.

Hierarchical multi-label propagation using speaking face graphs for multimodal person discovery

TV archives are growing in size so fast that manually indexing becomes unfeasible. Automatic indexing techniques can be applied to overcome this issue, and this work proposes an unsupervised technique for multimodal person discovery. To achieve this goal, we propose a hierarchical label propagation technique based on quasi-flat zones theory, that learns from labeled and unlabeled data and propagates names through a multimodal graph representation. In this representation, we combine audio, video, and text processing techniques to model the data as a graph of speaking faces. In the proposed mod-eling, we extract names via optical character recognition and propagate them through the graph using audiovisual relationships between speaking faces. We also use a random walk label propagation and two graph clustering strategies to serve as baselines. The proposed label propagation techniques always outper-form the clustering baselines on the quantitative assessments. Our approach also outperforms all literature methods tested on the same dataset except for one, which uses a different preprocessing step. The proposed hierarchical label propagation and the random walk baseline produce highly equivalent results according to the Kappa coefficient, but the hierarchical propagation is parameter-free and over 9 times faster than the random walk under the same configurations.

Multimodal graph convolutional networks for high quality content recognition

With the development of the Internet, more and more creators publish articles on social media. How to automatically filter high quality content from a large number of multimedia articles is one of the core functions of information recommendation, search engine, and other systems. However, existing approaches typically suffer from two limitations: (1) They usually model content as word sequences, which ignores the semantics provided by non-consecutive phrases, long-distance word dependency, and visual information. (2) They rely on a large amount of manually annotated data to train a quality assessment model while users may only provide labels of interest in a single class for a small number of samples in reality. To address these limitations, we propose a Multimodal Graph Convolutional Networks (MGCN) to model the semantic representations in a unified framework for High Quality Content Recognition. Instead of viewing text content as word sequences, we convert them into graphs, which can model non-consecutive phrases and long-distance word dependency for better obtaining the composition of semantics. Besides, visual content is also modeled into the graphs to provide complementary semantics. A well-designed graph convolutional network is proposed to capture the semantic representations based on these graphs. Furthermore, we employ a non-negative risk estimator for high quality content recognition and the loss is back-propagated for model learning. Experiments on real datasets validate the effectiveness of our approach.

MGAT: Multimodal Graph Attention Network for Recommendation

Graph neural networks (GNNs) have shown great potential for personalized recommendation. At the core is to reorganize interaction data as a user-item bipartite graph and exploit high-order connectivity among user and item nodes to enrich their representations. While achieving great success, most existing works consider interaction graph based only on ID information, foregoing item contents from multiple modalities (e.g., visual, acoustic, and textual features of micro-video items). Distinguishing personal interests on different modalities at a granular level was not explored until recently proposed MMGCN (Wei et al., 2019). However, it simply employs GNNs on parallel interaction graphs and treats information propagated from all neighbors equally, failing to capture user preference adaptively. Hence, the obtained representations might preserve redundant, even noisy information, leading to non-robustness and suboptimal performance. In this work, we aim to investigate how to adopt GNNs on multimodal interaction graphs, to adaptively capture user preference on different modalities and offer in-depth analysis on why an item is suitable to a user. Towards this end, we propose a new Multimodal Graph Attention Network, short for MGAT, which disentangles personal interests at the granularity of modality. In particular, built upon multimodal interaction graphs, MGAT conducts information propagation within individual graphs, while leveraging the gated attention mechanism to identify varying importance scores of different modalities to user preference. As such, it is able to capture more complex interaction patterns hidden in user behaviors and provide a more accurate recommendation. Empirical results on two micro-video recommendation datasets, Tiktok and MovieLens, show that MGAT exhibits substantial improvements over the state-of-the-art baselines like NGCF (Wang, He, et al., 2019) and MMGCN (Wei et al., 2019). Further analysis on a case study illustrates how MGAT generates attentive information flow over multimodal interaction graphs.

Deep Graph-Based Multimodal Feature Embedding for Endomicroscopy Image Retrieval.

Representation learning is a critical task for medical image analysis in computer-aided diagnosis. However, it is challenging to learn discriminative features due to the limited size of the data set and the lack of labels. In this article, we propose a deep graph-based multimodal feature embedding (DGMFE) framework for medical image retrieval with application to breast tissue classification by learning discriminative features of probe-based confocal laser endomicroscopy (pCLE). We first build a multimodality graph model based on the visual similarity between pCLE data and reference histology images. The latent similar pCLE-histology pairs are extracted by walking with the cyclic path on the graph while the dissimilar pairs are extracted based on the geodesic distance. Given the similar and dissimilar pairs, the latent feature space is discovered by reconstructing the similarity between pCLE and histology images via deep Siamese neural networks. The proposed method is evaluated on a clinical database with 700 pCLE mosaics. The accuracy of image retrieval demonstrates that DGMFE can outperform previous works on feature learning. Especially, the top-1 accuracy in an eight-class retrieval task is 0.739, thus demonstrating a 10% improvement compared to the state-of-the-art method.

Deep graph embedding for prioritizing synergistic anticancer drug combinations

Drug combinations are frequently used for the treatment of cancer patients in order to increase efficacy, decrease adverse side effects, or overcome drug resistance. Given the enormous number of drug combinations, it is cost- and time-consuming to screen all possible drug pairs experimentally. Currently, it has not been fully explored to integrate multiple networks to predict synergistic drug combinations using recently developed deep learning technologies. In this study, we proposed a Graph Convolutional Network (GCN) model to predict synergistic drug combinations in particular cancer cell lines. Specifically, the GCN method used a convolutional neural network model to do heterogeneous graphembedding, and thus solved a link prediction task. The graph in this study was a multimodal graph, which was constructed by integrating the drug-drug combination, drug-protein interaction, and protein–protein interaction networks. We found that the GCN model was able to correctly predict cell line-specific synergistic drug combinations from a large heterogonous network. The majority (30) of the 39 cell line-specific models show an area under the receiver operational characteristic curve (AUC) larger than 0.80, resulting in a mean AUC of 0.84. Moreover, we conducted an in-depth literature survey to investigate the top predicted drug combinations in specific cancer cell lines and found that many of them have been found to show synergistic antitumor activity against the same or other cancers in vitro or in vivo. Taken together, the results indicate that our study provides a promising way to better predict and optimize synergistic drug pairs in silico.

Distributed Multimodal Path Queries

Multimodal path queries over transportation networks are receiving increasing attention due to their widespread applications. A multimodal path query consists of finding multimodal journeys from source to destination in transportation networks, including unrestricted walking, driving, cycling, and schedule-based public transportation. Transportation networks are generally continent-sized. This characteristic highlights the need for parallel computing to accelerate multimodal path queries. Meanwhile, transportation networks are often fragmented and distributively stored on different machines. This situation calls for exploiting parallel computing power for these distributed systems. Therefore, in this paper, we study distributed multimodal path (DMP) queries over large transportation networks. We develop algorithms to explore parallel computation. When evaluating a DMP query Q on a distributed multimodal graph Gmult, we show that the algorithms possess the following performance guarantees, irrespective of how Gmult is fragmented and distributed: (1) each machine is visited only once; (2) the total network traffic is determined by the size of Q and the fragmentation of Gmult, independent of the size of Gmult; (3) the response time is decided by the largest fragment of Gmult; and (4) the algorithm is parallel scalable. Using real-life and synthetic data, we experimentally verify that the algorithms are scalable on large graphs.

Designing a Multimodal Graph System to Support Non-Visual Interpretation of Graphical Information

While researchers have performed numerous studies to understand the human interpretation of visual graphs in reading, comprehending and interpreting displayed data; visually impaired (VI) users still face many challenges that prevent them from fully benefiting from these graphs. Thus, it influences their understanding of data visualization and in turn reduces their role in collaborating with their sighted colleagues in educational and working environments. We intend to develop a mobile application where visually impaired users can work together to build a collaborative graph that supported by data sonification in the mobile environment. The system properties were all tested by the task of identifying line trends in time series, which resulted in an accuracy of more than 80% for notes below 20 points. The usability testing has given result of 6.7 out 10 based on users’ perception on the effectivity of the features.

Phylogeographic analysis of Iranian wildcats (Felis lybica / Felis silvestris) as revealed by mitochondrial cytochrome b gene

The wildcat (Felis lybica/Felis silvestris) is widely distributed in Africa and Eurasia. Iran is situated at the intersection of the distributions of the European Wildcat F. silvestris with the African Wildcat F. lybica. The genetic relationships of the Iranian wildcat population with either group remain unclear. We examined sequence variation of 1,140 bp of mitochondrial DNA cytochrome b gene from 23 wildcat samples collected across Iran. The analyses revealed two subclades, of which one is related to the Asian Wildcat (F. lybica ornata) and the other to the African Wildcat (F. lybica lybica) and the Domestic Cat (F. catus). Sixteen closely related haplotypes were identified for the entire country, with two geographically distinct subclades for western and eastern Zagros Mountain Range and overlapping in some localities. The analyses of fixation index (FST) and molecular variance (AMOVA) demonstrated significant genetic structuring among the two subclades(FST=0.63). Analysis of mismatch distribution and multimodal graph indicated that the Iranian wildcat population is in a condition of demographic equilibrium.

Multi-modal graph regularization based class center discriminant analysis for cross modal retrieval

On the network, a large amount of multi-modal data has emerged. Efficiently utilizing such data to conduct cross modal retrieval has become a hot topic of research. Some solutions have been proposed for this problem. However, many of these methods only considered the local structural information, thus losing sight of the global structural information of data. To overcome this problem and enhance retrieval accuracy, we propose a multi-modal graph regularization based class center discriminant analysis for cross modal retrieval. The core of our method is to maximize the intra-modality distance and minimize the inter-modality distance of class center samples to strengthen the discriminant ability of the model. Meanwhile, a multi-modal graph, which consists of the inter-modality similarity graph, the class center intra-modality graph and the inter-modality graph, is fused into the method to further reinforce the semantic similarity between different modalities. The method considers the local structural information of data together with the global structural information of data. Experimental results on three benchmark datasets demonstrate the superiority of this proposed scheme over several state-of-the-art methods.

F131. Multimodal Graph Theoretical Brain Networks and the 9-Year Cumulative Disease Load of Depression and Anxiety

F131. Multimodal Graph Theoretical Brain Networks and the 9-Year Cumulative Disease Load of Depression and Anxiety

Unsupervised cross-modal retrieval via Multi-modal graph regularized Smooth Matrix Factorization Hashing

The existing cross-modal hashing methods often encounter quantization loss which is caused by relaxing discrete hash codes in the process of cross-modal retrieval. To counter this problem, a Multi-modal graph regularized Smooth matrix Factorization Hashing (MSFH) approach is represented for unsupervised cross-modal retrieval. In the proposal framework, a smooth matrix generated by a control parameter is introduced into the matrix decomposition model, which can guarantee the sparsity of the dictionaries learned and the extracted common features at the same time, thus reducing the quantization loss in the hashing process. Furthermore, to preserve the topology of the original data, a multi-modal graph regularization term is drawn into the model, which consists of two parts. One is the intra-modal similarity graph which is used to preserve the geometric structure of each modality. The other is the inter-modal similarity graph reconstructed by the symmetric nonnegative matrix factorization, which is employed to soften the structure difference between modalities. The goal of MSFH is to learn unified hash-codes for multi-modal data in a shared latent semantic space in which the similarity of different modalities can be estimated effectively. And the corresponding experimental results on three benchmark data sets demonstrate the superiority of the proposed approach over several state-of-the-art cross-modality hashing approaches.

Integrating Content Analysis Into Urban Research: Compatibility With Sociotope Method and Multimodal Graph

The content analysis approach is well-established and acknowledged sociological research technique, although it is constantly evolving and its field of application is expanding. The aim of the article is to demonstrate the possibilities to use the content analysis method in urban studies. It includes the analysis of literature and the example of methodology design in the frame of case of the study of modernization of Lithuanian cities during the Soviet period. It can be concluded that the content analysis method is a flexible tool that can be integrated both with sociospatial (sociotope methodology) and spatial (Space Syntax, multimodal graph) research methods and reinforce social dimension in spatial analysis of cities.