Graph Matching Problem Research Articles

Bi-VLGM: Bi-Level Class-Severity-Aware Vision-Language Graph Matching for Text Guided Medical Image Segmentation

AbstractMedical reports containing specific diagnostic results and additional information not present in medical images can be effectively employed to assist image understanding tasks, and the modality gap between vision and language can be bridged by vision-language matching (VLM). However, current vision-language models distort the intra-model relation and only include class information in reports that is insufficient for segmentation task. In this paper, we introduce a novel Bi-level class-severity-aware Vision-Language Graph Matching (Bi-VLGM) for text guided medical image segmentation, composed of a word-level VLGM module and a sentence-level VLGM module, to exploit the class-severity-aware relation among visual-textual features. In word-level VLGM, to mitigate the distorted intra-modal relation during VLM, we reformulate VLM as graph matching problem and introduce a vision-language graph matching (VLGM) to exploit the high-order relation among visual-textual features. Then, we perform VLGM between the local features for each class region and class-aware prompts to bridge their gap. In sentence-level VLGM, to provide disease severity information for segmentation task, we introduce a severity-aware prompting to quantify the severity level of disease lesion, and perform VLGM between the global features and the severity-aware prompts. By exploiting the relation between the local (global) and class (severity) features, the segmentation model can include the class-aware and severity-aware information to promote segmentation performance. Extensive experiments proved the effectiveness of our method and its superiority to existing methods. The source code will be released.

Temporal subgraph matching method for multi-connected temporal graph

Temporal graph matching is used to find a matching subgraph set that satisfies temporal and topological information on the temporal graph. It is widely used in real scenarios and provides important support for big data mining and analysis. Most existing matching methods take the single-connected graph as the matching graph, which has a high computational cost. Simultaneously, the construction and matching complexity of its index on multi-connected temporal subgraph matching is high. To solve the above problems, we propose a new temporal subgraph matching problem for multi-connected temporal graphs and proposes the corresponding efficient filtering-selecting-matching three-stage multi-connected temporal subgraph matching method. In the filtering stage, a vertex-based filter is proposed to generate a candidate temporal subgraph set. In the select phase, a set of temporal subgraphs satisfying the vertex mapping is found. In the matching stage, the single-filter temporal subgraph matching method and temporal subgraph matching based on the temporal information tree index method are proposed to solve the problems that the edges of multi-connected temporal subgraphs cannot be mapped and the single graph matching efficiency is low, respectively. The experimental results show that the proposed method improves the matching accuracy, and the corresponding index size is reduced.

Lightning graph matching

The spectral matching algorithm is a classic method for finding correspondences between two graphs, a fundamental task in pattern recognition. It has a time complexity of O(n4) and a space complexity of O(n4), where n is the number of nodes. However, such complexity limits its applicability to large-scale graph matching tasks. This paper proposes a redesign of the algorithm by transforming the graph matching problem into a one-dimensional linear assignment problem. This transformation enables efficient solving by sorting two n×1 vectors. The resulting algorithm is named the Lightning Spectral Assignment Method (LiSA), which enjoys a complexity of O(n2). Numerical experiments demonstrate the efficiency of LiSA, supported by theoretical analysis.

Max-flow min-cut theorem in quantum computing

The max-flow min-cut theorem in graph theory states that the maximum flow of a network is equal to the minimum cut of edges of the flow network. Max-flow and min-cut are related as two primal–dual linear programs. The theorem is applicable in applications like network connectivity, graph matching problems, transportation and logistics, and scheduling problems. The aim of the paper is to model the classical max-flow and min-cut theorem in quantum computing. Two well-known methods for quantum optimization are quantum annealing and quantum approximate optimization algorithm. The max-flow min-cut is converted to its equivalent model of the above two methods for execution in polynomial time. This paper shows detailed classical to quantum conversion, analysis, implementation, and result of the theorem.

A many-to-many matching with externalities solution for parallel task offloading in IoT networks

The efficient and timely execution of tasks is a fundamental challenge in the realm of future Internet of Things (IoT) networks. To address this challenge, fog devices are often deployed close to end devices to facilitate task processing on behalf of IoT nodes. One strategy for improving task computational delay is to employ parallel task offloading, in which tasks are subdivided into subtasks and sent to different fog devices for execution in parallel. However, allocating computational resources to fog nodes and mapping these resources to IoT subtasks is a key challenge in this area. This work models the parallel task offloading problem as a graph-matching problem and utilizes a many-to-many matching technique to achieve a stable mapping of IoT subtasks to fog node resources. Unfortunately, the proposed solution is subject to the problem of externalities due to the dynamic preference profiling of fog nodes. To address this issue, we employ an iterative algorithm to resolve any blocking pairs that may arise. Our results demonstrate that the proposed technique reduces the task latency by 29% as compared to other matching-based techniques available in the literature.

Learnable Graph Matching: A Practical Paradigm for Data Association.

Data association is at the core of many computer vision tasks, e.g., multiple object tracking, image matching, and point cloud registration. however, current data association solutions have some defects: they mostly ignore the intra-view context information; besides, they either train deep association models in an end-to-end way and hardly utilize the advantage of optimization-based assignment methods, or only use an off-the-shelf neural network to extract features. In this paper, we propose a general learnable graph matching method to address these issues. Especially, we model the intra-view relationships as an undirected graph. Then data association turns into a general graph matching problem between graphs. Furthermore, to make optimization end-to-end differentiable, we relax the original graph matching problem into continuous quadratic programming and then incorporate training into a deep graph neural network with KKT conditions and implicit function theorem. In MOT task, our method achieves state-of-the-art performance on several MOT datasets. For image matching, our method outperforms state-of-the-art methods on a popular indoor dataset, ScanNet. For point cloud registration, we also achieve competitive results.

Deep graph matching meets mixed-integer linear programming: Relax or not ?

Graph matching is an important problem that has received widespread attention, especially in the field of computer vision. Recently, state-of-the-art methods seek to incorporate graph matching with deep learning. However, there is no research to explain what role the graph matching algorithm plays in the model. Therefore, we propose an approach integrating a MILP formulation of the graph matching problem. This formulation is solved to optimal and it provides inherent baseline. Meanwhile, similar approaches are derived by releasing the optimal guarantee of the graph matching solver and by introducing a quality level. This quality level controls the quality of the solutions provided by the graph matching solver. In addition, several relaxations of the graph matching problem are put to the test. Our experimental evaluation gives several theoretical insights and guides the direction of deep graph matching methods.

Optimal bipartite graph matching-based goal selection for policy-based hindsight learning

The sparse reward problem stands as a significant challenge in the field of reinforcement learning. Hindsight Experience Replay (HER) addresses this by goal relabeling, allowing the agent to learn from unsuccessful experiences. Some studies combine policy gradient methods with HER, resulting in policy-based hindsight learning algorithms. However, Policy-based hindsight learning involves the use of importance sampling, where the distribution of hindsight goals and the distribution of desired goals contribute to the computation of importance weights. When there is a significant difference between the two distributions, importance weights may become skewed, thereby impacting the evaluation of the policy and leading to suboptimal policies. To address this, we propose modeling the goal selection as an optimization problem for distribution matching. After we augment the original desired goals using Kernel Density Estimation (KDE), we further convert the optimization problem for distribution matching into a bipartite graph matching problem that minimizes the sum of weights. Our optimal bipartite graph matching-based hindsight goal selection method can select hindsight goals that are the most closely aligned with the original goals. Experimental results show that algorithms combined with the optimal bipartite graph matching-based hindsight goal selection outperform the original algorithms. Visualizations also demonstrate the superiority of our method in selecting hindsight goals.

SAT-Based Algorithms for Regular Graph Pattern Matching

Graph matching is a fundamental problem in pattern recognition, with many applications such as software analysis and computational biology. One well-known type of graph matching problem is graph isomorphism, which consists of deciding if two graphs are identical. Despite its usefulness, the properties that one may check using graph isomorphism are rather limited, since it only allows strict equality checks between two graphs. For example, it does not allow one to check complex structural properties such as if the target graph is an arbitrary length sequence followed by an arbitrary size loop. We propose a generalization of graph isomorphism that allows one to check such properties through a declarative specification. This specification is given in the form of a Regular Graph Pattern (ReGaP), a special type of graph, inspired by regular expressions, that may contain wildcard nodes that represent arbitrary structures such as variable-sized sequences or subgraphs. We propose a SAT-based algorithm for checking if a target graph matches a given ReGaP. We also propose a preprocessing technique for improving the performance of the algorithm and evaluate it through an extensive experimental evaluation on benchmarks from the CodeSearchNet dataset.

Load-aware task migration algorithm toward adaptive load balancing in Edge Computing

The rapid advancement of the Internet of Things (IoT) is leading to more and more devices joining the network to interact with information, which requires improving the performance of IoT applications to accommodate more data, faster response times, and more complex tasks. Edge computing, as a new computing paradigm, brings resource contention and load imbalance while reducing communication overhead and task latency. This paper addresses the workload distribution challenge in edge networks, aiming to optimize resource utilization and thereby enhance IoT application performance. To achieve this goal, we present the Load-aware Task Migration (LATM) algorithm. Firstly, we present a load state detection model that captures edge nodes’ workloads and dynamically classifies them according to their resource requirements. We then propose an innovative optimization problem, transforming task migration into a weighted tripartite graph matching problem. This problem leverages the Kuhn–Munkres(KM) task migration algorithm to attain the optimal matching between tasks and nodes. Finally, we assess the algorithm’s performance through experimental simulation. The experimental results underscore the algorithm’s substantial potential in reducing task response times, task execution times, load balance, and enhancing resource utilization.

Kartograf: A Geometrically Accurate Atom Mapper for Hybrid-Topology Relative Free Energy Calculations.

Relative binding free energy (RBFE) calculations have emerged as a powerful tool that supports ligand optimization in drug discovery. Despite many successes, the use of RBFEs can often be limited by automation problems, in particular, the setup of such calculations. Atom mapping algorithms are an essential component in setting up automatic large-scale hybrid-topology RBFE calculation campaigns. Traditional algorithms typically employ a 2D subgraph isomorphism solver (SIS) in order to estimate the maximum common substructure. SIS-based approaches can be limited by time-intensive operations and issues with capturing geometry-linked chemical properties, potentially leading to suboptimal solutions. To overcome these limitations, we have developed Kartograf, a geometric-graph-based algorithm that uses primarily the 3D coordinates of atoms to find a mapping between two ligands. In free energy approaches, the ligand conformations are usually derived from docking or other previous modeling approaches, giving the coordinates a certain importance. By considering the spatial relationships between atoms related to the molecule coordinates, our algorithm bypasses the computationally complex subgraph matching of SIS-based approaches and reduces the problem to a much simpler bipartite graph matching problem. Moreover, Kartograf effectively circumvents typical mapping issues induced by molecule symmetry and stereoisomerism, making it a more robust approach for atom mapping from a geometric perspective. To validate our method, we calculated mappings with our novel approach using a diverse set of small molecules and used the mappings in relative hydration and binding free energy calculations. The comparison with two SIS-based algorithms showed that Kartograf offers a fast alternative approach. The code for Kartograf is freely available on GitHub (https://github.com/OpenFreeEnergy/kartograf). While developed for the OpenFE ecosystem, Kartograf can also be utilized as a standalone Python package.

FusionQuery: On-demand Fusion Queries over Multi-source Heterogeneous Data

Centralised data management systems (e.g., data lakes) support queries over multi-source heterogeneous data. However, the query results from multiple sources commonly involve between-source conflicts, which makes query results unreliable and confusing and degrades the usability of centralised data management systems. Therefore, resolving the between-sourced conflicts is one of the most important problems for centralised data management systems. To solve it, many batch data fusion-based methods have been proposed, which require traversing all the data in the centralised data management systems and cause scalability and flexibility issues. To address these issues, this paper explores the problem of on-demand fusion queries, where the between-sourced conflicts are solved with only the query-related data; moreover, we propose an efficient on-demand fusion query framework, FusionQuery, which consists of a query stage and a fusion stage. In the query stage, we frame the heterogeneous data query problem as a knowledge graph matching problem and present a line graph-based method to accelerate it. In the fusion stage, we develop an Expectation Maximization-style algorithm to iteratively updates data veracity and source trustworthiness. Furthermore, we design an incremental estimation method of source trustworthiness to address the lack of sufficient observations. Extensive experiments on two real-world datasets demonstrate that FusionQuery outperforms state-of-the-art data fusion methods in terms of both effectiveness and efficiency.

SCGTracker: Spatio-temporal correlation and graph neural networks for multiple object tracking

Multiple Object Tracking is an important vitally important fundamental task in computer vision. Visual tracking becomes challenging when objects move in groups and are obscured from each other. There are two mainstream solution strategies for these group models. One is to transform the data association problem into a graph matching problem for solving, while the other is to apply the social power model as an advanced constraint for group tracking. In the former case, the solving difficulty geometric growth as the number of tracked objects increases, and the computing efficiency for real-time tracking demand cannot be met. The latter strategy tends to set up fixed-size groups or offline training rules, resulting in a lack of flexibility that limits their scenario generalization. According to the shortcomings of existing methods, this paper proposes a novel multiple object tracking method with spatio-temporal correlation and graph neural networks. Firstly, the relational features of the historical trajectories are extracted through the spatio-temporal relationship learning module, which models the spatio-temporal correlations of the objects and dynamically constructs the group structure online. Then, the graph neural network is combined with appearance and motion information, and the similarity between each detection and tracklet is used as a weight in node feature aggregation to make powerful distinctions between node features. Meanwhile, the spatio-temporal correlation method is also used to solve target loss issues caused by occlusion. Even collocated with linearly assigned data association method, good tracking results are still achieved, with a low computational complexity. Experiments on three challenging public datasets, namely MOT16, MOT17, and MOT20, validated the accuracy and efficiency of the proposed tracking method.

A Robust Framework for One-Shot Key Information Extraction via Deep Partial Graph Matching.

Text field labelling plays a key role in Key Information Extraction (KIE) from structured document images. However, existing methods ignore the field drift and outlier problems, which limit their performance and make them less robust. This paper casts the text field labelling problem into a partial graph matching problem and proposes an end-to-end trainable framework called Deep Partial Graph Matching (dPGM) for the one-shot KIE task. It represents each document as a graph and estimates the correspondence between text fields from different documents by maximizing the graph similarity of different documents. Our framework obtains a strict one-to-one correspondence by adopting a combinatorial solver module with an extra one-to-(at most)-one mapping constraint to do the exact graph matching, which leads to the robustness of the field drift problem and the outlier problem. Finally, a large one-shot KIE dataset named DKIE is collected and annotated to promote research of the KIE task. This dataset will be released to the research and industry communities. Extensive experiments on both the public and our new DKIE datasets show that our method can achieve state-of-the-art performance and is more robust than existing methods.

Population-wise labeling of sulcal graphs using multi-graph matching.

Population-wise matching of the cortical folds is necessary to compute statistics, a required step for e.g. identifying biomarkers of neurological or psychiatric disorders. The difficulty arises from the massive inter-individual variations in the morphology and spatial organization of the folds. The task is challenging both methodologically and conceptually. In the widely used registration-based techniques, these variations are considered as noise and the matching of folds is only implicit. Alternative approaches are based on the extraction and explicit identification of the cortical folds. In particular, representing cortical folding patterns as graphs of sulcal basins-termed sulcal graphs-enables to formalize the task as a graph-matching problem. In this paper, we propose to address the problem of sulcal graph matching directly at the population level using multi-graph matching techniques. First, we motivate the relevance of the multi-graph matching framework in this context. We then present a procedure for generating populations of artificial sulcal graphs, which allows us to benchmark several state-of-the-art multi-graph matching methods. Our results on both artificial and real data demonstrate the effectiveness of multi-graph matching techniques in obtaining a population-wise consistent labeling of cortical folds at the sulcal basin level.

Optimization approach in quadratic assignment and graph isomorphism

Graph matching problem, also known as quadratic assignment problem (QAP) in a practical context, aims at finding the best correspondence between two given graphs. If two nodes are connected in one graph, it would be ideal if their corresponding images are connected in the other. The optimal solution should maximize the number of matches between the two graphs, while minimizing the weight of mismatches. Graph isomorphism is a particular case where an exact correspondence with no mismatches could be found. The quadratic assignment problem itself is hard to solve due to its high computational complexity. In this paper, the researcher analyzes two optimization approaches in detail: convex relaxation and the Fast Approximate QAP algorithm, both trying to solve the problem with lower computational effort. The performances of these two methods are further compared through numerical experiments. This paper focuses on pairs of equal-sized undirected graphs. ρ correlated graphs are constructed to simulate the noise in inexact graph matching problems. After comparing two existing methods, we propose an even faster optimization approach with comparable accuracy. This new approach provides a particular way to approximate an ideal initialization for the traditional Fast Approximate QAP algorithm. The significant reduction in the runtime makes obtaining good correspondence between large graphs feasible.

The difficulty of beating the Taxman

The Taxman game has proven to be hard to solve optimally, so efforts have been made to find heuristic strategies that do well in practice. We present results on the NP-hardness of a variant of the game via an equivalence to a particular kind of graph matching problem. Furthermore this equivalence is used to derive a winning strategy for all n along with efficiently computable lower and upper bounds on the optimal achievable score.

Bisected graph matching improves automated pairing of bilaterally homologous neurons from connectomes.

Graph matching algorithms attempt to find the best correspondence between the nodes of two networks. These techniques have been used to match individual neurons in nanoscale connectomes-in particular, to find pairings of neurons across hemispheres. However, since graph matching techniques deal with two isolated networks, they have only utilized the ipsilateral (same hemisphere) subgraphs when performing the matching. Here, we present a modification to a state-of-the-art graph matching algorithm that allows it to solve what we call the bisected graph matching problem. This modification allows us to leverage the connections between the brain hemispheres when predicting neuron pairs. Via simulations and experiments on real connectome datasets, we show that this approach improves matching accuracy when sufficient edge correlation is present between the contralateral (between hemisphere) subgraphs. We also show how matching accuracy can be further improved by combining our approach with previously proposed extensions to graph matching, which utilize edge types and previously known neuron pairings. We expect that our proposed method will improve future endeavors to accurately match neurons across hemispheres in connectomes, and be useful in other applications where the bisected graph matching problem arises.

Universe Points Representation Learning for Partial Multi-Graph Matching

Many challenges from natural world can be formulated as a graph matching problem. Previous deep learning-based methods mainly consider a full two-graph matching setting. In this work, we study the more general partial matching problem with multi-graph cycle consistency guarantees. Building on a recent progress in deep learning on graphs, we propose a novel data-driven method (URL) for partial multi-graph matching, which uses an object-to-universe formulation and learns latent representations of abstract universe points. The proposed approach advances the state of the art in semantic keypoint matching problem, evaluated on Pascal VOC, CUB, and Willow datasets. Moreover, the set of controlled experiments on a synthetic graph matching dataset demonstrates the scalability of our method to graphs with large number of nodes and its robustness to high partiality.

Self-Supervised Bidirectional Learning for Graph Matching

Deep learning methods have demonstrated promising performance on the NP-hard Graph Matching (GM) problems. However, the state-of-the-art methods usually require the ground-truth labels, which may take extensive human efforts or be impractical to collect. In this paper, we present a robust self-supervised bidirectional learning method (IA-SSGM) to tackle GM in an unsupervised manner. It involves an affinity learning component and a classic GM solver. Specifically, we adopt the Hungarian solver to generate pseudo correspondence labels for the simple probabilistic relaxation of the affinity matrix. In addition, a bidirectional recycling consistency module is proposed to generate pseudo samples by recycling the pseudo correspondence back to permute the input. It imposes a consistency constraint between the pseudo affinity and the original one, which is theoretically supported to help reduce the matching error. Our method further develops a graph contrastive learning jointly with the affinity learning to enhance its robustness against the noise and outliers in real applications. Experiments deliver superior performance over the previous state-of-the-arts on five real-world benchmarks, especially under the more difficult outlier scenarios, demon- strating the effectiveness of our method.