Graph Reasoning Research Articles

Dual convolutional network based on hypergraph and multilevel feature fusion for road extraction from high-resolution remote sensing images

ABSTRACT Road extraction from high-resolution remote sensing images (HRSI) is confronted with the challenge that roads are occluded by other objects, including opaque obstructions and similarly colored areas. This paper proposes a dual convolutional network based on hypergraph and multilevel feature fusion (DHM) for road extraction to address these challenges. The DHM consists of two branch networks (HGNN branch and CNN branch) and a bimodal feature fusion module (BFFM). In the HGNN branch, an algorithm is developed to exploit the shape features of roads and construct hypergraphs on the HRSI. Then, hypergraph neural networks are employed for the first time to capture the long-range context of roads to enhance road connectivity. In the CNN branch, a bottleneck fusion module integrated with an encoder-decoder network structure is built to aggregate multiscale local features. In BFFM, the long-range context from the HGNN branch and the local features from the CNN branch are fused through the designed position converter and enhanced graph reasoning module to achieve the complementary advantages of the dual-branch network. Extensive experiments on three datasets show that DHM outperforms other state-of-the-art methods, especially on the GS-Mountain road dataset. Furthermore, DHM significantly improves road extraction in occluded and similar road regions.

Entity recognition based on heterogeneous graph reasoning of visual region and text candidate

Entity recognition based on heterogeneous graph reasoning of visual region and text candidate

Learning dual disentangled representation with self-supervision for temporal knowledge graph reasoning

Temporal knowledge graph (TKG) reasoning aims to infer the missing links from the massive historical facts. One of the big issues is that how to model the entity evolution from both the local and especially global perspectives. The primary temporal dependency models often fail to disentangle both perspectives due to the lack explicit annotations to distinguish the boundary of these two representations. To address these limitations, we propose a contrastive learning framework to Disentangle Local and Global perspectives for TKG Reasoning with self-supervision framework (DLGR). Our proposed DLGR can jointly utilize the local and global perspectives on two separate graphs and disentangle them in a self-supervised manner. Firstly, we construct a temporal subgraph and a temporal unified graph to effectively learn the local and global perspective representations, respectively. Second, we extract proxies regarding the different neighbors as pseudo labels to supervise the local and global disentanglement in a contrastive manner. Finally, we adaptively fuse the learned two perspective representations for TKG reasoning. The empirical results show that our DLGR significantly outperforms other baselines (e.g., compared to the strong baseline HGLS, our DLGR achieves 4.3%, 3.4%, 1.6% and 1.1% improvements on ICEWS14, ICEWS18, YAGO and WIKI using MRR).

Graph Reasoning With Supervised Contrastive Learning for Legal Judgment Prediction.

Given the fact descriptions of legal cases, the legal judgment prediction (LJP) problem aims to determine three judgment tasks of law articles, charges, and the term of penalty. Most existing studies have considered task dependencies while neglecting the prior dependencies of labels among different tasks. Therefore, how to make better use of the information on the relation dependencies among tasks and labels becomes a crucial issue. To this end, we transform the text classification problem into a node classification framework based on graph reasoning and supervised contrastive learning (SCL) techniques, named GraSCL. Specifically, we first design a graph reasoning network to model the potential dependency structures and facilitate relational learning under various graph topologies. Then, we introduce the SCL method for the LJP task to further leverage the label relation on the graph. To accommodate the node classification settings, we extend the traditional SCL method to novel variants for SCL at the node level, which allows the GraSCL framework to be trained efficiently even with small batches. Furthermore, to recognize the importance of hard negative samples in contrastive learning, we introduce a simple yet effective technique called online hard negative mining (OHNM) to enhance our SCL approach. This technique complements our SCL method and enables us to control the number and complexity of negative samples, leading to further improvements in the model's performance. Finally, extensive experiments are conducted on two well-known benchmarks, demonstrating the effectiveness and rationality of our proposed SCL approach as compared to the state-of-the-art competitors.

Dynamic Spatio-Temporal Graph Reasoning for VideoQA With Self-Supervised Event Recognition.

Video question answering (VideoQA) requires the ability of comprehensively understanding visual contents in videos. Existing VideoQA models mainly focus on scenarios involving a single event with simple object interactions and leave event-centric scenarios involving multiple events with dynamically complex object interactions largely unexplored. These conventional VideoQA models are usually based on features extracted from the global visual signals, making it difficult to capture the object-level and event-level semantics. Although there exists a recent work utilizing a static spatio-temporal graph to explicitly model object interactions in videos, it ignores the dynamic impact of questions for graph construction and fails to exploit the implicit event-level semantic clues in questions. To overcome these limitations, we propose a Self-supervised Dynamic Graph Reasoning (SDGraphR) model for video question answering (VideoQA). Our SDGraphR model learns a question-guided spatio-temporal graph that dynamically encodes intra-frame spatial correlations and inter-frame correspondences between objects in the videos. Furthermore, the proposed SDGraphR model discovers event-level cues from questions to conduct self-supervised learning with an auxiliary event recognition task, which in turn helps to improve its VideoQA performances without using any extra annotations. We carry out extensive experiments to validate the substantial improvements of our proposed SDGraphR model over existing baselines.

Mining Semantic Information With Dual Relation Graph Network for Multi-Label Image Classification

The purpose of multi-label image classification is to assign multiple labels for multiple objects presented in one image. Recent research efforts exploit graph convolution network (GCN) to learn the label co-occurrence dependencies for enhancing the semantic representation. Although these methods have achieved promising results, they can not capture the intrinsic correlation between objects in images and do not consider the inter-channel relationship. In addition, the previous methods treat each single image independently and fail to explore the relationship between different images. To address the above challenges, we propose a novel Dual Relation Graph Network (DRGN) model, which adopts a double branch structure to excavate rich semantic information from intra-image and cross-image simultaneously. Specifically, we first develop an intra-image channel-relation mining (ICM) module to mine the inter-channel relationship in features while learning the importance of different channels. Secondly, we design a new GCN-based intra-image spatial-relation exploring (ISE) module to capture the correlation between objects in individual image. Notably, ISE module and ICM module can complement and promote each other from the spatial and channel dimensions of images to improve the correlation between objects in individual image. Thirdly, we propose a novel GCN-based cross-image semantic learning (CSL) module to learn the semantic relationship between different images in the mini-batch. Through graph reasoning, our CSL module can iteratively refine input image features by acquiring common semantic information from other images in the mini-batch. Extensive experiments on the MS-COCO 2014, PASCAL VOC 2007, and VG-500 datasets demonstrate that the proposed DRGN model outperforms current state-of-the-art methods.

NL2SQL with partial missing metadata based on multi-view metadata graph compensation and reasoning

NL2SQL with partial missing metadata based on multi-view metadata graph compensation and reasoning

Mind Reasoning Manners: Enhancing Type Perception for Generalized Zero-Shot Logical Reasoning Over Text.

Logical reasoning task involves diverse types of complex reasoning over text, based on the form of multiple-choice question answering (MCQA). Given the context, question and a set of options as the input, previous methods achieve superior performances on the full-data setting. However, the current benchmark dataset has the ideal assumption that the reasoning type distribution on the train split is close to the test split, which is inconsistent with many real application scenarios. To address it, there remain two problems to be studied: 1) how is the zero-shot capability of the models (train on seen types and test on unseen types)? and 2) how to enhance the perception of reasoning types for the models? For problem 1, we propose a new benchmark for generalized zero-shot logical reasoning, named ZsLR. It includes six splits based on the three type sampling strategies. For problem 2, a type-aware model TaCo is proposed. It utilizes the heuristic input reconstruction and builds a text graph with a global node. Incorporating graph reasoning and contrastive learning, TaCo can improve the type perception in the global representation. Extensive experiments on both the zero-shot and full-data settings prove the superiority of TaCo over the state-of-the-art (SOTA) methods. Also, we experiment and verify the generalization capability of TaCo on other logical reasoning dataset.

Hybrid Graph Reasoning With Dynamic Interaction for Visual Dialog

Hybrid Graph Reasoning With Dynamic Interaction for Visual Dialog

From Concept to Instance: Hierarchical Reinforced Knowledge Graph Reasoning

From Concept to Instance: Hierarchical Reinforced Knowledge Graph Reasoning

Leveraging Knowledge Graph Reasoning in a Multihop Question Answering System for Hot Rolling Line Fault Diagnosis

Leveraging Knowledge Graph Reasoning in a Multihop Question Answering System for Hot Rolling Line Fault Diagnosis

Learning Dynamic and Static Representations for Extrapolation-based Temporal Knowledge Graph Reasoning

Learning Dynamic and Static Representations for Extrapolation-based Temporal Knowledge Graph Reasoning

TLNet: Temporal Span Localization Network With Collaborative Graph Reasoning for Video Question Answering

TLNet: Temporal Span Localization Network With Collaborative Graph Reasoning for Video Question Answering

RGR-Net: Refined Graph Reasoning Network for multi-height hotspot defect detection in photovoltaic farms

Unmanned aerial vehicle (UAV) detection of hotspot defects at multi-height is crucial for the reliable operation of photovoltaic (PV) farms. However, there are two major challenges in PV power plant data: defect information loss and noise information interference. In this paper, we design a Refined Graph Reasoning Network (RGR-Net) mainly composed of the backbone network, Cross-embedding Refined Graph Reasoning (CRGR) module, and prediction head. The proposed RGR-Net learns rich discriminative feature representations of hotspots by fully using the complementary information between the network layers and establishing global long-range dependencies. Specifically, CRGR learns to project input features into a global vector of a channel and performs inference on the representation of each defect with the global vector. CRGR utilizes two different transformation matrices to convert nodes and their connected nodes into a new feature space, thereby refining feature expression. Furthermore, we introduce the multi-height hotspot dataset (MHHS), marking the first comprehensive database tailored for defect detection across various flying altitudes, aiming to propel advancements in ensuring the safety and efficiency of PV farms. It should be noted that the current study focuses on data collected under favorable weather conditions, and further research is needed to assess the robustness of our model in extreme weather scenarios such as fog, heavy rain, or snow. Experimental results demonstrate that RGR-Net achieves an AP of 84.0% on the MHHS on average, surpassing the current state-of-the-art detectors.11https://github.com/zss313/MHHS-Dataset.

LASFormer: Light Transformer for Action Segmentation with Receptive Field-Guided Distillation and Action Relation Encoding

Transformer-based models for action segmentation have achieved high frame-wise accuracy against challenging benchmarks. However, they rely on multiple decoders and self-attention blocks for informative representations, whose huge computing and memory costs remain an obstacle to handling long video sequences and practical deployment. To address these issues, we design a light transformer model for the action segmentation task, named LASFormer, with a novel encoder–decoder structure based on three key designs. First, we propose a receptive field-guided distillation to realize mode reduction, which can overcome more generally the gap in semantic feature structure between the intermediate features by aggregated temporal dilation convolution (ATDC). Second, we propose a simplified implicit attention to replace self-attention to avoid its quadratic complexity. Third, we design an efficient action relation encoding module embedded after the decoder, where the temporal graph reasoning introduces an inductive bias that adjacent frames are more likely to belong to the same class of model global temporal relations, and the cross-model fusion structure integrates frame-level and segment-level temporal clues, which can avoid over-segmentation independent of multiple decoders, thus reducing further computational complexity. Extensive experiments have verified the effectiveness and efficiency of the framework. Against the challenging 50Salads, GTEA, and Breakfast benchmarks, LASFormer significantly outperforms the current state-of-the-art methods in accuracy, edit score, and F1 score.

DegreEmbed: Incorporating entity embedding into logic rule learning for knowledge graph reasoning

Knowledge graphs (KGs), as structured representations of real world facts, are intelligent databases incorporating human knowledge that can help machine imitate the way of human problem solving. However, KGs are usually huge and there are inevitably missing facts in KGs, thus undermining applications such as question answering and recommender systems that are based on knowledge graph reasoning. Link prediction for knowledge graphs is the task aiming to complete missing facts by reasoning based on the existing knowledge. Two main streams of research are widely studied: one learns low-dimensional embeddings for entities and relations that can explore latent patterns, and the other gains good interpretability by mining logical rules. Unfortunately, the heterogeneity of modern KGs that involve entities and relations of various types is not well considered in the previous studies. In this paper, we propose DegreEmbed, a model that combines embedding-based learning and logic rule mining for inferring on KGs. Specifically, we study the problem of predicting missing links in heterogeneous KGs from the perspective of the degree of nodes. Experimentally, we demonstrate that our DegreEmbed model outperforms the state-of-the-art methods on real world datasets and the rules mined by our model are of high quality and interpretability.

Multi-hop path reasoning over sparse temporal knowledge graphs based on path completion and reward shaping

Multi-hop path reasoning plays a crucial role in temporal knowledge graph reasoning, aiming to infer deep complex relationships and obtain interpretable reasoning results. Previous reasoning models primarily focus on designing temporal knowledge graphs with rich paths between entities but struggle to handle path sparsity, known as sparse temporal knowledge graphs. Sparse temporal knowledge graphs store only essential knowledge information, resulting in missing connection paths between certain entities and encountering issues of sparse rewards and information scarcity. To tackle these challenges, this paper proposes a multi-hop path reasoning model over sparse temporal knowledge graph based on path completion and reward shaping (STKGR-PR). STKGR-PR dynamically completes missing paths using a temporal embedding model to alleviate path sparsity. To tackle the issue of sparse rewards caused by reduced hit rates, we propose semantic composition-based reasoning path embeddings derived from relation embeddings and timestamp embeddings for reward shaping. Considering the limited information contained in sparse temporal knowledge graphs, we incorporate time vectors into the embedding and multi-hop path reasoning models to enhance the accuracy of reasoning paths and results. Experimental results conducted on six sparse temporal datasets sampled from ICEWS14 and ICEWS515–05 demonstrate that STKGR-PR outperforms state-of-the-art multi-hop path reasoning models over temporal knowledge graphs across all evaluation metrics, while ensuring interpretability.

MPTN: A message-passing transformer network for drug repurposing from knowledge graph

Drug repurposing (DR) based on knowledge graphs (KGs) is challenging, which uses knowledge graph reasoning models to predict new therapeutic pathways for existing drugs. With the rapid development of computing technology and the growing availability of validated biomedical data, various knowledge graph-based methods have been widely used to analyze and process complex and novel data to discover new indications for given drugs. However, existing methods need to be improved in extracting semantic information from contextual triples of biomedical entities. In this study, we propose a message-passing transformer network named MPTN based on knowledge graph for drug repurposing. Firstly, CompGCN is used as precoder to jointly aggregate entity and relation embeddings. Then, to fully capture the semantic information of entity context triples, the message propagating transformer module is designed. The module integrates the transformer into the message passing mechanism and incorporates the attention weight information of computing entity context triples into the entity embedding to update the entity embedding. Next, the residual connection is introduced to retain information as much as possible and improve prediction accuracy. Finally, MPTN utilizes the InteractE module as the decoder to obtain heterogeneous feature interactions in entity and relation representations and predict new pathways for drug treatment. Experiments on two datasets show that the model is superior to the existing knowledge graph embedding (KGE) learning methods.

Inferring Complementary and Substitutable Products Based on Knowledge Graph Reasoning

Complementarity and substitutability between products are essential concepts in retail and marketing. To achieve this, existing approaches take advantage of knowledge graphs to learn more evidence for inference. However, they often omit the knowledge that lies in the unstructured data. In this research, we concentrate on inferring complementary and substitutable products in e-commerce from mass structured and unstructured data. An improved knowledge-graph-based reasoning model has been proposed which cannot only derive related products but also provide interpretable paths to explain the relationship. The methodology employed in our study unfolds through several stages. First, a knowledge graph refining entities and relationships from data was constructed. Second, we developed a two-stage knowledge representation learning method to better represent the structured and unstructured knowledge based on TransE and SBERT. Then, the relationship inferring problem was converted into a path reasoning problem under the Markov decision process environment by learning a dynamic policy network. We also applied a soft pruning strategy and a modified reward function to improve the effectiveness of the policy network training. We demonstrate the effectiveness of the proposed method on standard Amazon datasets, and it gives about 5–15% relative improvement over the state-of-the-art models in terms of NDCG@10, Recall@10, Precision @10, and HR@10.

Dual-graph hierarchical interaction network for referring image segmentation

Referring Image Segmentation (RIS) aims to extract the object or stuff from an image according to the given natural language expression. As a representative multi-modal reasoning task, the main challenge of RIS lies in accurately understanding and aligning two types of heterogeneous data (i.e. image and text). Existing methods typically conduct this task via inexplicit cross-modal fusion toward the visual and linguistic features, which are separately extracted from different encoders and hard to capture accurate image–text alignment information due to their distinct latent representation structures. In this paper, we propose a Dual-Graph Hierarchical Interaction Network (DGHIN) to facilitate the explicit and comprehensive alignment between the image and text data. Firstly, two graphs are separately built for the initial visual and linguistic features extracted with different pre-trained encoders. By means of graph reasoning, we obtain a unified representation structure for different modalities to capture the intra-modal entities and their contexts, where each projected node incorporates the long-range dependencies into the latent representation. Then, the Hierarchical Interaction Module (HIM) is applied to the visual and linguistic graphs to extract comprehensive inter-modal correlations from the entity level and graph level, which not only capture the corresponding keywords and visual patches but also draws the whole sentence closer to the image region with the consistent context in the latent space. Extensive experiments on RefCOCO, RefCOCO+, G-Ref, and ReferIt demonstrate that the proposed DGHIN outperforms many state-of-the-art methods. Code is available at https://github.com/ZhaofengSHI/referring-DGHIN.