Graph Matching Research Articles

Multimodal Data Mining Based on Self Attention Feature Alignment

This study explores the application of multimodal data mining in text and image vector processing, aiming to improve the depth and breadth of data analysis by integrating information from different data types. We use the FairFace dataset combined with the CLIP model encoding layer to obtain text and image vectors, and use the K-Means clustering algorithm to achieve vector dimensionality reduction. Subsequently, we introduced the bipartite graph matching algorithm to achieve maximum matching between text vectors and image vectors, and calculated the contrastive learning loss and similarity loss. The entire process covers steps such as data preparation, feature extraction, vector dimensionality reduction, matching algorithms, and loss assessment, constructing a complete text and image matching task process. Our research contributions include using K-Means clustering algorithm to achieve vector dimensionality reduction, as well as introducing bipartite graph matching algorithm and calculating two types of losses in text and image vector matching, further improving matching quality.

From communities to interpretable network and word embedding: an unified approach

Abstract Modelling information from complex systems such as humans social interaction or words co-occurrences in our languages can help to understand how these systems are organized and function. Such systems can be modelled by networks, and network theory provides a useful set of methods to analyze them. Among these methods, graph embedding is a powerful tool to summarize the interactions and topology of a network in a vectorized feature space. When used in input of machine learning algorithms, embedding vectors help with common graph problems such as link prediction, graph matching, etc In Natural Language Processing (NLP), such a vectorization process is also employed. Word embedding has the goal of representing the sense of words, extracting it from large text corpora. Despite differences in the structure of information in input of embedding algorithms, many graph embedding approaches are adapted and inspired from methods in NLP. Limits of these methods are observed in both domains. Most of these methods require long and resource greedy training. Another downside to most methods is that they are black-box, from which understanding how the information is structured is rather complex. Interpretability of a model allows understanding how the vector space is structured without the need for external information, and thus can be audited more easily. With both these limitations in mind, we propose a novel framework to efficiently embed network vertices in an interpretable vector space. Our Lower Dimension Bipartite Framework (LDBGF) leverages the bipartite projection of a network using cliques to reduce dimensionality. Along with LDBGF, we introduce two implementations of this framework that rely on communities instead of cliques: SINr-NR and SINr-MF. We show that SINr-MF can perform well on classical graphs and SINr-NR can produce high-quality graph and word embeddings that are interpretable and stable across runs.

HAGMN-UQ: Hyper association graph matching network with uncertainty quantification for coronary artery semantic labeling

Coronary artery disease (CAD) is one of the leading causes of death worldwide. Accurate extraction of individual arterial branches from invasive coronary angiograms (ICA) is critical for CAD diagnosis and detection of stenosis. Generating semantic segmentation for coronary arteries through deep learning-based models presents challenges due to the morphological similarity among different types of coronary arteries, making it difficult to maintain high accuracy while keeping low computational complexity. To address this challenge, we propose an innovative approach using the hyper association graph-matching neural network with uncertainty quantification (HAGMN-UQ) for coronary artery semantic labeling on ICAs. The graph-matching procedure maps the arterial branches between two individual graphs, so that the unlabeled arterial segments are classified by the labeled segments, and the coronary artery semantic labeling is achieved. Leveraging hypergraphs not only extends representation capabilities beyond pairwise relationships, but also improves the robustness and accuracy of the graph matching by enabling the modeling of higher-order associations. In addition, employing the uncertainty quantification to determine the trustworthiness of graph matching reduces the required number of comparisons, so as to accelerate the inference speed. Consequently, our model achieved an accuracy of 0.9211 for coronary artery semantic labeling with a fast inference speed, leading to an effective and efficient prediction in real-time clinical decision-making scenarios.

New Extracted Features to Recognize Faces Effected by Occlusions and Common Variations

Face recognition from non-identical face photos is a prominent area of research in pattern recognition and computer vision. Existing face recognition systems struggle with diverse changes like lighting conditions, expressions, and facial occlusions. This paper proposes a new Face Recognition (FR) approach that combines the Elastic Bunch Graph Matching (EBGM) approach with the greedy algorithm to automatically identify face landmarks. The proposed approach independently selects each optimal landmark of face image from different corresponding face images where the corresponding landmark of corresponding face image which achieves the best similarity is used rather than using one or at most two corresponding face images and computing the average between both. The locations of corresponding landmarks can be displaced to achieve maximum similarity with optimal landmarks. This proposed approach demonstrates improved recognition performance compared to contemporary face recognition methods. It effectively handles changing ratios of face parts and can recognize faces even with increasing occlusion sizes.

Bi-VLGM: Bi-Level Class-Severity-Aware Vision-Language Graph Matching for Text Guided Medical Image Segmentation

AbstractMedical reports containing specific diagnostic results and additional information not present in medical images can be effectively employed to assist image understanding tasks, and the modality gap between vision and language can be bridged by vision-language matching (VLM). However, current vision-language models distort the intra-model relation and only include class information in reports that is insufficient for segmentation task. In this paper, we introduce a novel Bi-level class-severity-aware Vision-Language Graph Matching (Bi-VLGM) for text guided medical image segmentation, composed of a word-level VLGM module and a sentence-level VLGM module, to exploit the class-severity-aware relation among visual-textual features. In word-level VLGM, to mitigate the distorted intra-modal relation during VLM, we reformulate VLM as graph matching problem and introduce a vision-language graph matching (VLGM) to exploit the high-order relation among visual-textual features. Then, we perform VLGM between the local features for each class region and class-aware prompts to bridge their gap. In sentence-level VLGM, to provide disease severity information for segmentation task, we introduce a severity-aware prompting to quantify the severity level of disease lesion, and perform VLGM between the global features and the severity-aware prompts. By exploiting the relation between the local (global) and class (severity) features, the segmentation model can include the class-aware and severity-aware information to promote segmentation performance. Extensive experiments proved the effectiveness of our method and its superiority to existing methods. The source code will be released.

ROPGMN: Effective ROP and variants discovery using dynamic feature and graph matching network

Return Oriented Programming (ROP) is one of the most challenging threats to operating systems. Traditional detection and defense techniques for ROP such as stack protection, address randomization, compiler optimization, control flow integrity, and basic block thresholds have certain limitations in accuracy or efficiency. At the same time, they cannot effectively detect ROP variant attacks, such as COP, COOP, JOP. In this paper, we propose a novel ROP and its variants detection approach that first filters the normal execution flow according to four strategies provided and then adopts Graph Matching Network (GMN) to determine whether there is ROP or its variant attack. Moreover, we developed a prototype named ROPGMN with shared memory to solve cross-language and cross-process problems. Using real-world vulnerable programs and constructed programs with dangerous function calls, we conduct extensive experiments with 6 ROP detectors to evaluate ROPGMN. The experimental results demonstrate the effectiveness of ROPGMN in discovering ROPs and their variant attacks with low performance overhead.

Further results on enumeration of perfect matchings of Cartesian product graphs

Abstract Counting perfect matchings is an interesting and challenging combinatorial task. It has important applications in statistical physics and chemistry. As the general problem is #P-complete, it is usually tackled by randomized heuristics and approximation schemes. Let G G and H H be two graphs. Denote by G × H G\times H the Cartesian product of graphs G G and H H . The number of perfect matchings of some types of Cartesian product G × H G\times H is investigated by the Pfaffian method, where G ≅ P 2 G\cong {P}_{2} , P 3 {P}_{3} , P 4 {P}_{4} or C 4 {C}_{4} , and H ≅ T H\cong T or H H is a non-bipartite graph with a unique cycle. In this article, we construct a Pfaffian orientation of the graph G × P 2 G\times {P}_{2} , where G G is a non-bipartite graph with k k odd cycles. We obtain the counting formula on the number of perfect matchings of G × P 2 G\times {P}_{2} .

Maximum Cardinality \(f\) -Matching in Time \(O(n^{2/3}m)\)

We present an algorithm that finds a maximum cardinality \(f\) -matching of a simple graph in time \(O(n^{2/3}m)\) . Here \(f:V\to\mathbb{N}\) is a given function and an \(f\) -matching is a subgraph wherein each vertex \(v\in V\) has degree \(\leq f(v)\) . This result generalizes a string of algorithms that concentrate on simple bipartite graphs. The bipartite case is based on the notion of level graph, introduced by Dinic for network flow. In general graphs this notion breaks down: Vertices no longer have unique levels, and there are too many levels to analyze the corresponding level graph like bipartite graphs ( \(\Theta(n^{2})\) vs. \(n\) ). We use “natural” levels to prove properties of shortest augmenting trails (e.g., formulas for trail length). We use “shortened” levels to derive the algorithm's time bound. The algorithm, unmodified, is also efficient on multigraphs, achieving time \(O(\min\{\sqrt{f(V)},n\} m)\) for \(f(V)=\sum_{v}f(v)\) . The special case \(f\equiv 1\) shows the algorithm duplicates the classic time bound for maximum cardinality matching, \(O(\sqrt{n} m)\) .

DETR-ORD: An Improved DETR Detector for Oriented Remote Sensing Object Detection with Feature Reconstruction and Dynamic Query

Optical remote sensing images often feature high resolution, dense target distribution, and uneven target sizes, while transformer-based detectors like DETR reduce manually designed components, DETR does not support arbitrary-oriented object detection and suffers from high computational costs and slow convergence when handling large sequences of images. Additionally, bipartite graph matching and the limit on the number of queries result in transformer-based detectors performing poorly in scenarios with multiple objects and small object sizes. We propose an improved DETR detector for Oriented remote sensing object detection with Feature Reconstruction and Dynamic Query, termed DETR-ORD. It introduces rotation into the transformer architecture for oriented object detection, reduces computational cost with a hybrid encoder, and includes an IFR (image feature reconstruction) module to address the loss of positional information due to the flattening operation. It also uses ATSS to select auxiliary dynamic training queries for the decoder. This improved DETR-based detector enhances detection performance in challenging oriented optical remote sensing scenarios with similar backbone network parameters. Our approach achieves superior results on most optical remote sensing datasets, such as DOTA-v1.5 (72.07% mAP) and DIOR-R (66.60% mAP), surpassing the baseline detector.

License Plate Recognition Using Three-Dimensional Rotated Character Recognition and Instance Segmentation by Deep Learning

License plate recognition is currently used in various situations, such as parking lot vehicle management and the tracking of wanted vehicles. These applications require recognition from all possible camera angles. Therefore, we propose a method for license plate recognition in images captured from various camera angles. First, the license plate area is detected from the input image using the YOLOv5 object detection method. The subsequent process can be performed in two manners: by rotating the license plate to the front using projective transformation and trapezoidal correction, or by using graph matching. The first method performs Hough transforms or linear approximations as preprocessing, and then calculates the rectangle surrounding the license plate from the YOLOv5 output. After performing trapezoidal correction, the maximally stable extremal regions (MSER) are used to detect character candidates, and characters are recognized using 3D rotated character recognition. The second method performs character candidate detection and recognition without trapezoidal correction, followed by graph matching. These methods are versatile because they do not depend on the layout of license plates, which varies among countries and regions. Two types of datasets were used: a set of images containing Japanese license plates collected by ourselves, and a set of publicly available images containing Taiwanese license plates. Two recognition rates were evaluated: the license plate character recognition rate and license plate recognition rate in which all characters in one plate are recognized. The license plate character recognition rates using graph matching were 91.9% for the Japanese license plates and 89.3% for the Taiwanese license plates. The license plate recognition rates were 84.0% and 84.5%, respectively.

Accurate Sampling-Based Cardinality Estimation for Complex Graph Queries

Accurately estimating the cardinality (i.e., the number of answers) of complex queries plays a central role in database systems. This problem is particularly difficult in graph databases, where queries often involve a large number of joins and self-joins. Recently, Park et al. [ 55 ] surveyed seven state-of-the-art cardinality estimation approaches for graph queries. The results of their extensive empirical evaluation show that a sampling method based on the WanderJoin online aggregation algorithm [ 47 ] consistently offers superior accuracy. We extended the framework by Park et al. [ 55 ] with three additional datasets and repeated their experiments. Our results showed that WanderJoin is indeed very accurate, but it can often take a large number of samples and thus be very slow. Moreover, when queries are complex and data distributions are skewed, it often fails to find valid samples and estimates the cardinality as zero. Finally, complex graph queries often go beyond simple graph matching and involve arbitrary nesting of relational operators such as disjunction, difference, and duplicate elimination. Neither of the methods considered by Park et al. [ 55 ] is applicable to such queries. In this article, we present a novel approach for estimating the cardinality of complex graph queries. Our approach is inspired by WanderJoin, but, unlike all approaches known to us, it can process complex queries with arbitrary operator nesting. Our estimator is strongly consistent, meaning that the average of repeated estimates converges with probability one to the actual cardinality. We present optimisations of the basic algorithm that aim to reduce the chance of producing zero estimates and improve accuracy. We show empirically that our approach is both accurate and quick on complex queries and large datasets. Finally, we discuss how to integrate our approach into a simple dynamic programming query planner, and we confirm empirically that our planner produces high-quality plans that can significantly reduce end-to-end query evaluation times.

Interpretable Representation and Customizable Retrieval of Traffic Congestion Patterns Using Causal Graph-Based Feature Associations

The substantial increase in traffic data offers new opportunities to inspect traffic congestion dynamics from different perspectives. This paper presents a novel framework for the interpretable representation and customizable retrieval of traffic congestion patterns using causal relation graphs, which harnesses many of these opportunities. By integrating domain knowledge with innovative data management techniques, we address the challenges of effectively handling and retrieving the growing volume of traffic data for diverse analytical purposes. The framework leverages causal graphs to encode traffic congestion patterns, capturing fundamental phenomena and their spatiotemporal relationships, thus facilitating an interpretable high-level view of traffic dynamics. Moreover, a customizable similarity measurement function is introduced based on inexact graph matching, allowing users to tailor the retrieval process to specific requirements. This framework’s capability to retrieve customizable and interpretable congestion patterns is demonstrated through extensive experiments with real-world highway traffic data in the Netherlands, highlighting its value in supporting diverse data-driven studies and applications.

DGM-Flow: Appearance flow estimation for virtual try-on via dynamic graph matching

A virtual-image try-on is aimed at warping the target garment image to align with structure of the human body. In recent studies, flexible flow field estimation has been employed via feature matching between the garment and the human body to simulate garment deformation. However, currently available methods for estimating appearance flow rely on a direct comparison of feature descriptors between the source garment and the target pose, while neglecting the surrounding information. This limitation is prone to noisy feature matching, resulting in low accuracy for complex deformations. To overcome this challenge, herein we propose a novel framework named DGM-Flow that exploits the neighbor structure invariance of key features for feature matching, thus improving the estimation accuracy of the appearance flow. Specifically, DGM-Flow models higher-order geometric similarity information between the source garment and the target pose via dynamic graph matching. Moreover, considering the incomplete mapping of garment features before and after deformation, DGM-Flow is used to estimate a hard attention mask matrix to filter irrelevant features using cross-graph matching. Experiments conducted on both the VITON and VITON-HD datasets revealed that DGM-Flow affords better performance than the current leading algorithms, especially when dealing with intricate clothing deformations.

Temporal subgraph matching method for multi-connected temporal graph

Temporal graph matching is used to find a matching subgraph set that satisfies temporal and topological information on the temporal graph. It is widely used in real scenarios and provides important support for big data mining and analysis. Most existing matching methods take the single-connected graph as the matching graph, which has a high computational cost. Simultaneously, the construction and matching complexity of its index on multi-connected temporal subgraph matching is high. To solve the above problems, we propose a new temporal subgraph matching problem for multi-connected temporal graphs and proposes the corresponding efficient filtering-selecting-matching three-stage multi-connected temporal subgraph matching method. In the filtering stage, a vertex-based filter is proposed to generate a candidate temporal subgraph set. In the select phase, a set of temporal subgraphs satisfying the vertex mapping is found. In the matching stage, the single-filter temporal subgraph matching method and temporal subgraph matching based on the temporal information tree index method are proposed to solve the problems that the edges of multi-connected temporal subgraphs cannot be mapped and the single graph matching efficiency is low, respectively. The experimental results show that the proposed method improves the matching accuracy, and the corresponding index size is reduced.

Lightning graph matching

The spectral matching algorithm is a classic method for finding correspondences between two graphs, a fundamental task in pattern recognition. It has a time complexity of O(n4) and a space complexity of O(n4), where n is the number of nodes. However, such complexity limits its applicability to large-scale graph matching tasks. This paper proposes a redesign of the algorithm by transforming the graph matching problem into a one-dimensional linear assignment problem. This transformation enables efficient solving by sorting two n×1 vectors. The resulting algorithm is named the Lightning Spectral Assignment Method (LiSA), which enjoys a complexity of O(n2). Numerical experiments demonstrate the efficiency of LiSA, supported by theoretical analysis.

Meta-collaborative comparison for effective cross-domain few-shot learning

Recent advancements in cross-domain few-shot learning (CD-FSL) primarily focus on learning to compare global representations between query and support images for classification. However, due to the notorious cross-domain semantic gap, the ideal global representations can be totally different across domains, thereby solely learning to compare global representations is not sufficient to achieve effective generalization in challenging cases. To mitigate this problem, we present a Meta-collaborative Comparison Network (MeCo-Net) for CD-FSL, which imitates humans to recognize unfamiliar objects through collaborative comparison on both global and local representations. Following this idea, paralleling with a conventional global comparison branch, we additionally feed random crops of both query and support images into a feature encoder to separately extract their local representations. Subsequently, we associate these local representations across images through bipartite graph matching for local comparison. Thanks to the complementary global and local comparisons, we can obtain a more generalizable classifier for CD-FSL by meta-integrating them for final prediction. Experimental results on eight benchmarks demonstrate that the proposed model generalizes to multiple target domains with state-of-the-art performance without the need for fine-tuning.

Intelligent Conceptual Design of Railway Bridge Based on Graph Neural Networks

In the conceptual design stage of railway bridge, the beam type of the bridge at the main control point must be modified repeatedly to satisfy varying requirements. Thus, the demand for design efficiency is high. However, railway bridge design relies heavily on professional knowledge and experience and is typically completed manually by senior designers, thereby requiring considerable time. An intelligent beam type recommendation algorithm named AutoDis Graph Ontology Attention Matching (AGOAM) is proposed to rapidly generate bridge design plans for railway route main control points. This method acquires the node embeddings of the main control point and beam type attribute graphs through graph neural networks (GNNs) and predicts the score of each beam type through graph matching technology. The beam type with the highest prediction score is recommended. In addition, the accuracy of the recommendation results is improved through ontology-enhanced attribute interaction and attention mechanism-based graph pooling. The efficiency of the proposed method is demonstrated with a real-world railway bridge design dataset, and ablation study is conducted to evaluate the effectiveness of the ontology-enhanced attribute interaction and attention mechanism-based graph pooling.

Learnable Graph Matching: A Practical Paradigm for Data Association.

Data association is at the core of many computer vision tasks, e.g., multiple object tracking, image matching, and point cloud registration. however, current data association solutions have some defects: they mostly ignore the intra-view context information; besides, they either train deep association models in an end-to-end way and hardly utilize the advantage of optimization-based assignment methods, or only use an off-the-shelf neural network to extract features. In this paper, we propose a general learnable graph matching method to address these issues. Especially, we model the intra-view relationships as an undirected graph. Then data association turns into a general graph matching problem between graphs. Furthermore, to make optimization end-to-end differentiable, we relax the original graph matching problem into continuous quadratic programming and then incorporate training into a deep graph neural network with KKT conditions and implicit function theorem. In MOT task, our method achieves state-of-the-art performance on several MOT datasets. For image matching, our method outperforms state-of-the-art methods on a popular indoor dataset, ScanNet. For point cloud registration, we also achieve competitive results.

Deep graph matching meets mixed-integer linear programming: Relax or not ?

Graph matching is an important problem that has received widespread attention, especially in the field of computer vision. Recently, state-of-the-art methods seek to incorporate graph matching with deep learning. However, there is no research to explain what role the graph matching algorithm plays in the model. Therefore, we propose an approach integrating a MILP formulation of the graph matching problem. This formulation is solved to optimal and it provides inherent baseline. Meanwhile, similar approaches are derived by releasing the optimal guarantee of the graph matching solver and by introducing a quality level. This quality level controls the quality of the solutions provided by the graph matching solver. In addition, several relaxations of the graph matching problem are put to the test. Our experimental evaluation gives several theoretical insights and guides the direction of deep graph matching methods.

MFAE: Multimodal Fusion and Alignment for Entity-level Disinformation Detection

Nowadays, the dissemination of disinformation on social media has evolved from a purely textual form to multiple modalities consisting of both text and images. This further amplifies the misleading and deceptive nature of disinformation. Overcoming the misleading and confusing noise to achieve accurate disinformation detection presents a significant challenge. To address this challenge, we propose a method named Multimodal Fusion and Alignment for Entity-level Disinformation Detection (MFAE). MFAE first uses an improved dynamic routing algorithm to extract more comprehensive semantic visual entity features. Then, a graph matching network is used to capture the correspondences between entities within modalities. The experiment shows that MFAE is capable of capturing textual and visual semantic information more comprehensively. On the TWITTER and WEIBO datasets, MFAE achieves accuracy improvements of approximately 2.0% and 7.5%, respectively, compared to the state-of-the-art methods, resulting in accuracy of 89.5% and 96.7%.