Graph Partitioning Algorithm Research Articles

GCphase: an SNP phasing method using a graph partition and error correction algorithm

BackgroundThe utilization of long reads for single nucleotide polymorphism (SNP) phasing has become popular, providing substantial support for research on human diseases and genetic studies in animals and plants. However, due to the complexity of the linkage relationships between SNP loci and sequencing errors in the reads, the recent methods still cannot yield satisfactory results.ResultsIn this study, we present a graph-based algorithm, GCphase, which utilizes the minimum cut algorithm to perform phasing. First, based on alignment between long reads and the reference genome, GCphase filters out ambiguous SNP sites and useless read information. Second, GCphase constructs a graph in which a vertex represents alleles of an SNP locus and each edge represents the presence of read support; moreover, GCphase adopts a graph minimum-cut algorithm to phase the SNPs. Next, GCpahse uses two error correction steps to refine the phasing results obtained from the previous step, effectively reducing the error rate. Finally, GCphase obtains the phase block. GCphase was compared to three other methods, WhatsHap, HapCUT2, and LongPhase, on the Nanopore and PacBio long-read datasets. The code is available from https://github.com/baimawjy/GCphase.ConclusionsExperimental results show that GCphase under different sequencing depths of different data has the least number of switch errors and the highest accuracy compared with other methods.

Property Graph Partition Algorithm Based on Improved Barnacle Mating Optimization

Abstract Distributed graph processing systems have been used more frequently in various fields, and graph partitioning is the basis of these systems. Graph partitioning algorithms generally aim to minimize the communication cost while the number of vertices or edges reach the load balance. But the vertices and/or edges of property graphs require some storage volume, so the traditional graph partition algorithms will lead to an unbalanced storage volume load. This paper proposes an edge-cut graph partitioning algorithm to generate partitions with equal size and storage volume as well as low cut-edge ratio. Initially, it partitions the graph into k partitions with equal size and storage volume. It then migrates vertices based on the improved Barnacles mating optimizer. The experiments on real-world graphs show that the proposed algorithm can achieve the partition size of volume balance, and the cut-edge ratio is also very low.

Synergizing a Deep Learning and Enhanced Graph-Partitioning Algorithm for Accurate Individual Rubber Tree-Crown Segmentation from Unmanned Aerial Vehicle Light-Detection and Ranging Data

The precise acquisition of phenotypic parameters for individual trees in plantation forests is important for forest management and resource exploration. The use of Light-Detection and Ranging (LiDAR) technology mounted on Unmanned Aerial Vehicles (UAVs) has become a critical method for forest resource monitoring. Achieving the accurate segmentation of individual tree crowns (ITCs) from UAV LiDAR data remains a significant technical challenge, especially in broad-leaved plantations such as rubber plantations. In this study, we designed an individual tree segmentation framework applicable to dense rubber plantations with complex canopy structures. First, the feature extraction module of PointNet++ was enhanced to precisely extract understory branches. Then, a graph-based segmentation algorithm focusing on the extracted branch and trunk points was designed to segment the point cloud of the rubber plantation. During the segmentation process, a directed acyclic graph is constructed using components generated through grey image clustering in the forest. The edge weights in this graph are determined according to scores calculated using the topologies and heights of the components. Subsequently, ITC segmentation is performed by trimming the edges of the graph to obtain multiple subgraphs representing individual trees. Four different plots were selected to validate the effectiveness of our method, and the widths obtained from our segmented ITCs were compared with the field measurement. As results, the improved PointNet++ achieved an average recall of 94.6% for tree trunk detection, along with an average precision of 96.2%. The accuracy of tree-crown segmentation in the four plots achieved maximal and minimal R2 values of 98.2% and 92.5%, respectively. Further comparative analysis revealed that our method outperforms traditional methods in terms of segmentation accuracy, even in rubber plantations characterized by dense canopies with indistinct boundaries. Thus, our algorithm exhibits great potential for the accurate segmentation of rubber trees, facilitating the acquisition of structural information critical to rubber plantation management.

An Improved Graph Partitioning Algorithm Based Approach for Workflow Offloading in a Fog Environment

An Improved Graph Partitioning Algorithm Based Approach for Workflow Offloading in a Fog Environment

An efficient approach for discovering Graph Entity Dependencies (GEDs)

Graph entity dependencies (GEDs) are novel graph constraints, unifying keys and functional dependencies, for property graphs. They have been found useful in many real-world data quality and data management tasks, including fact checking on social media networks and entity resolution. In this paper, we study the discovery problem of GEDs—finding a minimal cover of valid GEDs in a given graph data. We formalise the problem, and propose an effective and efficient approach to overcome major bottlenecks in GED discovery. In particular, we leverage existing graph partitioning algorithms to enable fast GED-scope discovery, and employ effective pruning strategies over the prohibitively large space of candidate dependencies. Furthermore, we define an interestingness measure for GEDs based on the minimum description length principle, to score and rank the mined cover set of GEDs. Finally, we demonstrate the scalability and effectiveness of our GED discovery approach through extensive experiments on real-world benchmark graph data sets; and present the usefulness of the discovered rules in different downstream data quality management applications.

Load Balanced PIM-Based Graph Processing

Graph processing is widely used for many modern applications, such as social networks, recommendation systems, and knowledge graphs. However, processing large-scale graphs on traditional Von Neumann architectures is challenging due to the irregular graph data and memory-bound graph algorithms. Processing-in-memory (PIM) architecture has emerged as a promising approach for accelerating graph processing by enabling computation to be performed directly on memory. Despite having many processing units and high local memory bandwidth, PIM often suffers from insufficient global communication bandwidth and high synchronization overhead due to load imbalance. This article proposes GraphB, a novel PIM-based graph processing system, to address all these issues. From the algorithm perspective, we propose a degree-aware graph partitioning algorithm that can generate balanced partitioning at a low cost. From the architecture perspective, we introduce tile buffers incorporated with an on-chip 2D-Mesh, which provides high bandwidth for inter-node data transfer. Dataflow in GraphB is designed to enable computation–communication overlap and dynamic load balancing. In a PyMTL3-based cycle-accurate simulator with five real-world graphs and three common algorithms, GraphB achieves an average 2.2× and maximum 2.8× speedup compared to the SOTA PIM-based graph processing system GraphQ.

FragQC: An efficient quantum error reduction technique using quantum circuit fragmentation

Quantum computers must meet extremely stringent qualitative and quantitative requirements on their qubits in order to solve real-life problems. Quantum circuit fragmentation techniques divide a large quantum circuit into a number of sub-circuits that can be executed on the smaller noisy quantum hardware available. However, the process of quantum circuit fragmentation involves finding an ideal cut that has exponential time complexity and also the classical post-processing required to reconstruct the output. In this paper, we represent a quantum circuit using a weighted graph and propose a novel classical graph partitioning algorithm for selecting an efficient fragmentation that reduces the entanglement between the sub-circuits along with balancing the estimated error in each sub-circuit. We also demonstrate a comparative study of different classical and quantum approaches to graph partitioning for finding such a cut. We present FragQC, a software tool that cuts a quantum circuit into sub-circuits when its error probability exceeds a certain threshold. With this proposed approach, we achieve an increase in fidelity of 13.38% compared to direct execution without cutting the circuit, and 7.88% over the state-of-the-art ILP-based method for the benchmark circuits.The code for FragQC is available at https://github.com/arnavdas88/FragQC.

PICO: Pipeline Inference Framework for Versatile CNNs on Diverse Mobile Devices

Distributing the inference of convolutional neural network (CNN) to multiple mobile devices has been studied in recent years to achieve real-time inference without losing accuracy. However, how to map CNN to devices remains a challenge. On the one hand, scheduling the workload of state-of-the-art CNNs with multiple devices is NP-Hard because the structures of CNNs are directed acyclic graphs (DAG) rather than simple chains. On the other hand, distributing the inference workload suffers from expensive communication and unbalanced computation due to the wireless environment and heterogeneous devices. This paper presents PICO, a pipeline cooperation framework to accelerate the inference of versatile CNNs on diverse mobile devices. At its core, PICO features: (1) a generic graph partition algorithm that considers the characteristics of any given CNN and orchestrates it into a list of model pieces with suitable granularity, and (2) a many-to-many mapping algorithm that produces the best pipeline configuration for heterogeneous devices. In our experiment with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$2 \sim 8$</tex-math></inline-formula> Raspberry-Pi devices, the throughput can be improved by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$1.8 \sim 6.8 \times$</tex-math></inline-formula> under different CPU frequencies.

A large-scale graph partition algorithm with redundant multi-order neighbor vertex storage

Recently, graph data become increasingly important and larger in many fields, and many distributed graph computing systems have been proposed to deal with large-scale graphs. The graph partition algorithm is the basis of these systems. In many graph applications, a vertex updates its feature by aggregating information of its multi-order neighboring vertices, and the existing graph partition algorithms often lead to heavy communication cost. This paper proposes a graph partition algorithm to store the multi-order neighbor vertices redundantly to avoid the communication requirement. It first formulates the problem of graph partitioning with redundant multi-order neighbor vertices storage as an optimization problem, and then proposes PARN (partition algorithm with redundant multi-order neighbor) algorithm to tackle it. The PARN algorithm consists of initial partition and vertex migration based on genetic algorithm, and it classifies the vertices of a partition into primary and auxiliary vertices. The auxiliary vertices required for the primary vertices in a partition are stored in the same partition, so the primary vertices don't need to fetch the information from the other partitions. The experiment results show that the proposed algorithm is comparable to Hash, GAP, GNP, and QCLP algorithms in terms of load balance and produces fewer auxiliary vertices than these algorithms.

PaSTG: A Parallel Spatio-Temporal GCN Framework for Traffic Forecasting in Smart City

Predicting future traffic conditions from urban sensor data is crucial for smart city applications. Recent traffic forecasting methods are derived from Spatio-Temporal Graph Convolution Networks (STGCNs). Despite their remarkable achievements, these spatio-temporal models have mainly been evaluated on small-scale datasets. In light of the rapid growth of the Internet of Things and urbanization, cities are witnessing an increased deployment of sensors, resulting in the collection of extensive sensor data to provide more accurate insights into citywide traffic dynamics. Spatio-temporal graph modeling on large-scale traffic data is challenging due to the memory constraint of the computing device. For traffic forecasting, subgraph sampling from road networks onto multiple devices is feasible. Many GCN sampling methods have been proposed recently. However, combining these with STGCNs degrades performance. This is primarily due to prediction biases introduced by each sampled subgraph, which analyze traffic states from a regional perspective. Addressing these challenges, we introduce a parallel STGCN framework called PaSTG. PaSTG divides the road network into regions, each processed by an individual STGCN in a device. To mitigate regional biases, Aggregation Blocks in PaSTG merge spatial-temporal features from each STBlock. This collaboration enhances traffic forecasting. Furthermore, PaSTG implements pipeline parallelism and employs a graph partition algorithm for optimized pipeline efficiency. We evaluate PaSTG on various STGCNs using three traffic datasets on multiple GPUs. Results demonstrate that our parallel approach applies widely to diverse STGCN models, surpassing existing GCN samplers by up to 57.4% in prediction accuracy. Additionally, the parallel framework achieves speedups of up to 2.87x and 4.70x in training and inference compared to GCN samplers.

Template-Guided Hierarchical Multi-View Registration Framework of Unordered Bridge Terrestrial Laser Scanning Data.

The registration of bridge point cloud data (PCD) is an important preprocessing step for tasks such as bridge modeling, deformation detection, and bridge health monitoring. However, most existing research on bridge PCD registration only focused on pairwise registration, and payed insufficient attention to multi-view registration. In addition, to recover the overlaps of unordered multiple scans and obtain the merging order, extensive pairwise matching and the creation of a fully connected graph of all scans are often required, resulting in low efficiency. To address these issues, this paper proposes a marker-free template-guided method to align multiple unordered bridge PCD to a global coordinate system. Firstly, by aligning each scan to a given registration template, the overlaps between all the scans are recovered. Secondly, a fully connected graph is created based on the overlaps and scanning locations, and then a graph-partition algorithm is utilized to construct the scan-blocks. Then, the coarse-to-fine registration is performed within each scan-block, and the transformation matrix of coarse registration is obtained using an intelligent optimization algorithm. Finally, global block-to-block registration is performed to align all scans to a unified coordinate reference system. We tested our framework on different bridge point cloud datasets, including a suspension bridge and a continuous rigid frame bridge, to evaluate its accuracy. Experimental results demonstrate that our method has high accuracy.

Benchmarking Artificial Neural Network Architectures for High-Performance Spiking Neural Networks.

Organizations managing high-performance computing systems face a multitude of challenges, including overarching concerns such as overall energy consumption, microprocessor clock frequency limitations, and the escalating costs associated with chip production. Evidently, processor speeds have plateaued over the last decade, persisting within the range of 2 GHz to 5 GHz. Scholars assert that brain-inspired computing holds substantial promise for mitigating these challenges. The spiking neural network (SNN) particularly stands out for its commendable power efficiency when juxtaposed with conventional design paradigms. Nevertheless, our scrutiny has brought to light several pivotal challenges impeding the seamless implementation of large-scale neural networks (NNs) on silicon. These challenges encompass the absence of automated tools, the need for multifaceted domain expertise, and the inadequacy of existing algorithms to efficiently partition and place extensive SNN computations onto hardware infrastructure. In this paper, we posit the development of an automated tool flow capable of transmuting any NN into an SNN. This undertaking involves the creation of a novel graph-partitioning algorithm designed to strategically place SNNs on a network-on-chip (NoC), thereby paving the way for future energy-efficient and high-performance computing paradigms. The presented methodology showcases its effectiveness by successfully transforming ANN architectures into SNNs with a marginal average error penalty of merely 2.65%. The proposed graph-partitioning algorithm enables a 14.22% decrease in inter-synaptic communication and an 87.58% reduction in intra-synaptic communication, on average, underscoring the effectiveness of the proposed algorithm in optimizing NN communication pathways. Compared to a baseline graph-partitioning algorithm, the proposed approach exhibits an average decrease of 79.74% in latency and a 14.67% reduction in energy consumption. Using existing NoC tools, the energy-latency product of SNN architectures is, on average, 82.71% lower than that of the baseline architectures.

An effective algorithm for genealogical graph partitioning

An effective algorithm for genealogical graph partitioning

GpDB: A Graph-partition Based Storage Strategy for DAG-Blockchain in Edge-cloud IIoT

The industrial Internet-of-things (IIoT) has attracted extensive attention due to its real-time and automation characteristics. Edge computing and blockchain technologies facilitate the IIoT in terms of low latency services and data security respectively. However, with the continuous expansion of industrial data and the growth of industrial nodes, traditional blockchain technology has some critical limitations on low transaction throughput and high data storage costs. Directed Acyclic Graph (DAG)-blockchain adopts a graph structure of a single transaction as the basic unit, and it has the characteristics of asynchronous consensus. Some existing studies use DAGblockchain to replace the traditional blockchain to alleviate its low throughput problems like IOTA. However, with the rapid data generation in the IIoT environment, the topology scale of DAG-blockchain will increase sharply, which will aggravate the data storage cost of blockchain nodes. In this article, to reduce the data storage cost of edge servers, we design a Graphpartition based storage strategy for DAG-Blockchain (GpDB), equipped with a graph partition algorithm based on transaction freshness, which can partition DAG-blockchain topology in edge servers into two parts, which will be retained and removed respectively. Simulation shows that, in terms of storage cost, GpDB outperforms LDV and Layerchain by 62% and 74% respectively, and with the increasing number of transactions, GpDB has good scalability in reducing the storage cost, and better transaction throughput than IOTA.

A Sharding Scheme Based on Graph Partitioning Algorithm for Public Blockchain

A Sharding Scheme Based on Graph Partitioning Algorithm for Public Blockchain

Agglomeration of polygonal grids using graph neural networks with applications to multigrid solvers

Agglomeration-based strategies are important both within adaptive refinement algorithms and to construct scalable multilevel algebraic solvers. In order to automatically perform agglomeration of polygonal grids, we propose the use of Machine Learning (ML) strategies, that can naturally exploit geometrical information about the mesh in order to preserve the grid quality, enhancing performance of numerical methods and reducing the overall computational cost. In particular, we employ the k-means clustering algorithm and Graph Neural Networks (GNNs) to partition the connectivity graph of a computational mesh. Moreover, GNNs have high online inference speed and the advantage to process naturally and simultaneously both the graph structure of mesh and the geometrical information, such as the areas of the elements or their barycentric coordinates. These techniques are compared with METIS, a standard algorithm for graph partitioning, which is meant to process only the graph information of the mesh. We demonstrate that performance in terms of quality metrics is enhanced for ML strategies. Such models also show a good degree of generalization when applied to more complex geometries, such as brain MRI scans, and the capability of preserving the quality of the grid. The effectiveness of these strategies is demonstrated also when applied to MultiGrid (MG) solvers in a Polygonal Discontinuous Galerkin (PolyDG) framework. In the considered experiments, GNNs show overall the best performance in terms of inference speed, accuracy and flexibility of the approach.

Object Affinity Learning: Towards Annotation-Free Instance Segmentation.

We address the problem of annotation-free instance segmentation in the wild, aiming to relieve the expensive cost of manual mask annotations. Existing approaches utilize appearance cues, such as color, edge, and texture information, to generate pseudo masks for instance segmentation. However, due to the ambiguity of defining an object by visual appearance alone, these methods fail to distinguish objects from the background under complex scenes. Beyond visual cues, objects are one-piece in space and move together over time, which indicates that geometry cues, such as spatial continuity and motion consistency, are also exploitable for this problem. To directly utilize geometry cues, we propose an affinity-based paradigm for annotation-free instance segmentation. The new paradigm is called object affinity learning, a proxy task of annotation-free instance segmentation, which aims to tell whether two pixels come from the same object by learning feature representation from geometry cues. During inference, the learned object affinity could be further converted into instance segmentation masks by some graph partition algorithms. The proposed object affinity learning achieves much better instance segmentation performance than existing pseudo-mask-based methods on the large-scale Waymo Open Dataset and KITTI dataset.

A Multi-edge Collaborative Computational Offloading Scheme Based on Game Theory

&lt;p&gt;A common approach in existing collaborative edge computing offloading schemes is to partition tasks into independent sub-tasks and offload them to participating servers. However, in practice, these sub-tasks often have dependencies, resulting in waiting time. To address this problem, we propose a collaborative computation offloading scheme based on Stackelberg game theory and graph theory (CCOSGG). First, we introduce a task clustering method based on graph theory, which uses task reconstruction and graph partition algorithm to cluster strongly related sub-tasks into appropriately sized clusters. Second, we use Stackelberg game theory to introduce an incentive mechanism that encourages remote edges to participate in the collaborative offloading. Finally, simulation results demonstrate that the proposed scheme can minimize latency and energy consumption at different network scales.&lt;/p&gt; &lt;p&gt;&amp;nbsp;&lt;/p&gt;

A massively parallel implicit 3D unstructured grid solver for computing turbulent flows on latest distributed memory computational architectures

An implicit unstructured grid density-based solver–PRAVAHA based on a parallel variant of the lower upper symmetric Gauss-Seidel (LUSGS) method is developed to compute large-scale engineering problems. A four-layered parallel algorithm is designed to efficiently compute three-dimensional turbulent flows on massively parallel modern multiple instruction multiple data-stream (MIMD) computational hardware. This data parallel approach achieves multiple layers of parallelism including continuity of flow solution, transfer of solution gradients, and calculation of drag/lift/solution residuals, right up to the innermost implicit LUSGS solver sub-routine, which is relatively less explored in the literature. Domain decomposition is performed using the METIS software based on multi-level graph partitioning algorithms. Non-blocking message-passing interface library functions are used to manage inter-processor communication through explicit message passing, efficiently. Super-linear scalability of the parallel solver is established on the current state-of-the-art supercomputing facility, the 838 teraflops PARAM seva on up to 6144 cores. Linear or even super-linear speedup on problems of significant size is observed even on ad-hoc parallel computing platforms like workstations and multi-node clusters, for turbulent flow simulations.

Hypergraph partitioning using tensor eigenvalue decomposition.

Hypergraphs have gained increasing attention in the machine learning community lately due to their superiority over graphs in capturing super-dyadic interactions among entities. In this work, we propose a novel approach for the partitioning of k-uniform hypergraphs. Most of the existing methods work by reducing the hypergraph to a graph followed by applying standard graph partitioning algorithms. The reduction step restricts the algorithms to capturing only some weighted pairwise interactions and hence loses essential information about the original hypergraph. We overcome this issue by utilizing tensor-based representation of hypergraphs, which enables us to capture actual super-dyadic interactions. We extend the notion of minimum ratio-cut and normalized-cut from graphs to hypergraphs and show that the relaxed optimization problem can be solved using eigenvalue decomposition of the Laplacian tensor. This novel formulation also enables us to remove a hyperedge completely by using the "hyperedge score" metric proposed by us, unlike the existing reduction approaches. We propose a hypergraph partitioning algorithm inspired from spectral graph theory and also derive a tighter upper bound on the minimum positive eigenvalue of even-order hypergraph Laplacian tensor in terms of its conductance, which is utilized in the partitioning algorithm to approximate the normalized cut. The efficacy of the proposed method is demonstrated numerically on synthetic hypergraphs generated by stochastic block model. We also show improvement for the min-cut solution on 2-uniform hypergraphs (graphs) over the standard spectral partitioning algorithm.