Distributed Graph Research Articles

Onboard cooperative relative positioning system for Micro-UAV swarm based on UWB/Vision/INS fusion through distributed graph optimization

In recent years, the swarm of micro Unmanned Aerial Vehicles (micro-UAVs) is becoming a research hotspot. As the basis of swarm autonomous flight, traditional relative positioning methods typically rely on external base stations or onboard sensors of high power consumption, the vast majority of which are not feasible on resource-constrained micro-UAVs flying in a complex environment. This article proposes a lightweight relative positioning system implemented on a micro-UAV platform weighing 150 g. We have developed a distributed graph optimization (DGO) scheme fusing the results of the onboard Ultra-Wideband (UWB) module, camera, and inertial sensors. A Delay Extended Kalman Filter (DEKF) is utilized to mitigate communication delays and enhance accuracy. This scheme enables micro-UAVs to collaboratively compute the relative position graph in real-time. Through real-world experiments, our system has achieved a relative positioning accuracy of 0.167 m, exhibiting significant potential for future swarm applications.

FSM: A Fine-Grained Splitting and Merging Framework for Dual-Balanced Graph Partition

Partitioning a large graph into smaller subgraphs by minimizing the number of cutting vertices and edges, namely cut size or replication factor, plays a crucial role in distributed graph processing tasks. However, many prior works have primarily focused on optimizing the cut size by considering only vertex balance or edge balance, leading to significant workload imbalance and consequently hindering the performance of downstream tasks. Therefore, in this paper, we address the dual-balanced graph partition problem that minimizes the cut size while simultaneously guaranteeing both vertex and edge balance. We propose a lightweight effective two-phase framework, namely fine-grained splitting and merging (FSM), which decomposes the graph into more and smaller partitions and then merges them. FSM offers the flexibility of integrating with various state-of-the-art single-balanced techniques. We develop two efficient algorithms Fast Merging and Precise Merging to enable trade-offs between computational efficiency and partitioning quality. Experimental results on large real-world graphs demonstrate that FSM achieves state-of-the-art cut size while maintaining dual balance. The runtime for downstream tasks PageRank, connected component, and diameter estimation, can be reduced by a large proportion, up to 9.43%, 11.35%, and 17.94%, respectively.

GraphSER: Distance-Aware Stream-Based Edge Repartition for Many-Core Systems

With the explosive growth of graph data, distributed graph processing becomes popular and many graph hardware accelerators use distributed frameworks. Graph partitioning is foundation in distributed graph processing. However, dynamic changes in graph make existing partitioning shifted from its optimized points and cause system performance degraded. Therefore, more efficient dynamic graph partition methods are needed. In this work, we propose GraphSER, a dynamic graph partition method for many-core systems. In order to improve the cross-node spatial locality and reduce the overhead of repartition, we propose a stream-based edge repartition, in which each computing node sequentially traverses its local edge list in parallel, then migrating edges based on distance and replica degree. GraphSER does not need costly searching and prioritizes nodes so it can avoid poor cross-node spatial locality. Our evaluation shows that compared to state-of-the-art edge repartition software methods, GraphSER has an average speedup 1.52x, with the maximum up to 2x. Compared to the previous many-core hardware repartition method, GraphSER performance has an average of 40% improvement, with the maximum to 117%.

FSM-BC-BSP: Frequent Subgraph Mining Algorithm Based on BC-BSP

As graph models become increasingly prevalent in the processing of scientific data, the exploration of effective methods for the mining of meaningful patterns from large-scale graphs has garnered significant research attention. This paper delves into the complexity of frequent subgraph mining and proposes a frequent subgraph mining (FSM) algorithm. This FSM algorithm is developed within a distributed graph iterative system, designed for the Big Cloud (BC) environment of the China Mobile Corp., and is based on the bulk synchronous parallel (BSP) model, named FSM-BC-BSP. Its aim is to address the challenge of mining frequent subgraphs within a single, large graph. This study advocates for the incorporation of a message sending and receiving mechanism to facilitate data sharing across various stages of the frequent subgraph mining algorithm. Additionally, it suggests employing a standard coded subgraph and sending it to the same node for global support calculation on the large graph. The adoption of the rightmost path expansion strategy in generating candidate subgraphs helps to mitigate the occurrence of redundant subgraphs. The use of standard coding ensures the unique identification of subgraphs, thus eliminating the need for isomorphism calculations. Support calculation is executed using the Minimum Image (MNI) measurement method, aligning with the downward closure attribute. The experimental results demonstrate the robust performance of the FSM-BC-BSP algorithm across diverse input datasets and parameter configurations. Notably, the algorithm exhibits exceptional efficacy, particularly in scenarios with low support requirements, showcasing its superior performance under such conditions.

Distributed Graph Processing in Cloud Computing: A Review of Large-Scale Graph Analytics

The rapid growth of graph data in various domains has propelled the need for efficient distributed graph processing techniques in cloud computing environments. This paper presents a comprehensive review of distributed graph processing for graph analytics of massive size in the context of cloud computing. The paper begins by highlighting the challenges associated with distributed graph processing, including load balancing, communication overhead, scalability, and partitioning strategies. It provides an overview of existing frameworks and tools specifically designed for distributed graph processing in cloud environments. Furthermore, the review encompasses various techniques and algorithms employed in distributed graph processing. The paper also reviews recent research advancements in optimizing distributed graph processing in cloud computing. To provide practical insights, the paper presents a comparative analysis of representative large-scale graph analytics applications implemented on different cloud computing platforms. Performance, scalability, and efficiency metrics are evaluated under varying workload sizes and graph characteristics. Overall, this comprehensive review paper serves as a highly prized asset for researchers and large-scale graph analytics professionals who are practitioners in the field. It provides a holistic understanding of the state-of-the-art distributed graph processing techniques in cloud computing and guides future research efforts towards more efficient and scalable graph processing in cloud environments.

Application of Distributed Graphs for Facilitation of Scalable Botnet Detection and Response

Botnets pose a significant threat to modern network environments, exploiting compromised devices to carry out malicious activities such as distributed denial-of-service attacks, spam campaigns, and data theft. Traditional centralized detection systems often struggle to handle the scale and complexity of botnet attacks, leading to delays in detection and response. In response to these challenges, this paper explores the application of distributed graphs for facilitating scalable botnet detection and response. Distributed graphs offer a promising approach for modelling and analyzing complex network structures, enabling efficient detection of botnet propagation patterns and anomalous behaviour across distributed computing environments. The paper presents an overview of distributed graph-based botnet detection systems, discussing their architecture, design considerations, and key concepts such as graph partitioning, vertex-centric computation, and message passing in distributed graph algorithms. Case studies illustrate the practical application of distributed graph-based botnet detection in diverse network environments, highlighting success stories, challenges encountered, and lessons learned from deploying distributed graph systems in production cybersecurity operations. Finally, the paper discusses challenges and open research questions in the field of distributed graph-based botnet detection, addressing issues such as graph partitioning strategies, fault tolerance, privacy-preserving techniques, and integration with other security tools. It proposes potential avenues for future research and development in scalable botnet detection using distributed graphs, emphasizing the importance of adaptive threat response, collaboration with industry partners, and continuous improvement in detection algorithms for enhancing cybersecurity resilience against botnet attacks.

A large-scale graph partition algorithm with redundant multi-order neighbor vertex storage

Recently, graph data become increasingly important and larger in many fields, and many distributed graph computing systems have been proposed to deal with large-scale graphs. The graph partition algorithm is the basis of these systems. In many graph applications, a vertex updates its feature by aggregating information of its multi-order neighboring vertices, and the existing graph partition algorithms often lead to heavy communication cost. This paper proposes a graph partition algorithm to store the multi-order neighbor vertices redundantly to avoid the communication requirement. It first formulates the problem of graph partitioning with redundant multi-order neighbor vertices storage as an optimization problem, and then proposes PARN (partition algorithm with redundant multi-order neighbor) algorithm to tackle it. The PARN algorithm consists of initial partition and vertex migration based on genetic algorithm, and it classifies the vertices of a partition into primary and auxiliary vertices. The auxiliary vertices required for the primary vertices in a partition are stored in the same partition, so the primary vertices don't need to fetch the information from the other partitions. The experiment results show that the proposed algorithm is comparable to Hash, GAP, GNP, and QCLP algorithms in terms of load balance and produces fewer auxiliary vertices than these algorithms.

Multi-node knowledge graph assisted distributed fault detection for large-scale industrial processes based on graph attention network and bidirectional LSTMs

Modern industrial processes are characterized by extensive, multiple operation units, and strong coupled correlation of subsystems. Fault detection of large-scale processes is still a challenging problem, especially for tandem plant-wide processes in multiple fields such as water treatment process. In this paper, a novel distributed graph attention network-bidirectional long short-term memory (D-GATBLSTM) fault detection model is proposed for large-scale industrial processes. Firstly, a multi-node knowledge graph (MNKG) is constructed using a joint data and knowledge driven strategy. Secondly, for large-scale industrial process, a global feature extractor of graph attention networks (GATs) is constructed, on the basis of which, sub-blocks are decomposed based on MNKG. Then, local feature extractors of bidirectional long short-term memory (Bi-LSTM) for each sub-block are constructed, in which correlations among multiple sub-blocks are considered. Finally, a multi-subblock fusion collaborative prediction model is constructed and the comprehensive fault detection results are given by the grid search method. The effectiveness of our D-GATBLSTM is exemplified in a secure water treatment process case, where it outperforms baseline models compared, with 27% improvement in precision, 15% increase in recall, and overall F-score enhancement of 0.22.

Distributed Graph Algorithms: From Local Data to Global Solutions

As data scales increase, traditional centralized graph algorithms struggle to meet modern computational demands. Distributed graph algorithms, which parallelize data processing across multiple computing nodes, have significantly improved the efficiency of handling large-scale graph data. This report explores the principles, application scenarios, key technologies, and challenges of distributed graph algorithms, aiming to provide a comprehensive perspective from local data to global solutions. With the rapid development of computer networks and big data technologies, solving large-scale graph data problems has become a hot research topic. Distributed graph algorithms can solve problems without global information and offer new solutions for processing massive graph structures. This report introduces the basic concepts, key technologies, and challenges of distributed graph algorithms and discusses methods for achieving global solutions starting from local data through case analyses.

Knowledge-Enhanced Distributed Graph Autoencoder for Multiunit Industrial Plant-Wide Process Monitoring

In multiunit industrial plant-wide processes (MIPPs), data-driven distributed process monitoring methods have played an important role in ensuring process safety and reliability. However, the existing studies fail to leverage the relational knowledge about the underlying process structure of MIPPs, which may lead to inaccurate process modeling and degradation of the monitoring performance. To tackle these issues, this work proposes a novel knowledge-enhanced distributed graph autoencoder (KDGAE) for plant-wide process monitoring. Firstly, a posterior graph structure learning module (PGSL) is designed for relational knowledge discovery, which explicitly characterizes the dependencies between operation units in MIPPs into a directed graph. Prior knowledge is introduced into PGSL as a learning bias that steers the posterior graph to adhere to the underlying physics. Then, a novel distributed graph autoencoder (DGAE) is developed to encode both the local information within each unit and the global information between units for distributed process monitoring. The discovered posterior knowledge is embedded as a relational inductive bias in DGAE to enhance the capability of unsupervised representation learning, thereby improving the monitoring performance. Finally, the effectiveness of the proposed method is demonstrated through the Tennessee Eastman process and a real-world air separation process.

Integrating Knowledge Graphs into Distribution Grid Decision Support Systems

Distribution grids are complex networks containing multiple pieces of equipment. These components are interconnected, and each of them is described by various attributes. A knowledge graph is an interesting data format that represents pieces of information as nodes and relations between the pieces as edges. In this paper, we describe the proposed vocabulary used to build a distribution system knowledge graph. We identify the concepts used in such graphs and a set of relations to represent links between concepts. Both provide a semantically rich representation of a system. Additionally, we offer a few illustrative examples of how a distributed system knowledge graph can be utilized to gain more insight into the operations of the grid. We show a simplified analysis of how outages can influence customers based on their locations and how adding DERs can influence/change it. These demonstrative use cases show that the graph-based representation of a distribution grid allows for integrating information of different types and how such a repository can be efficiently utilized. Based on the experiments with distribution system knowledge graphs presented in this article, we postulate that graph-based representation enables a novel way of storing information about power grids and facilitates interactive methods for their visualization and analysis.

Distributed Graph Coloring Made Easy

In this article, we present a deterministic \(\mathsf {CONGEST}\) algorithm to compute an O ( k Δ)-vertex coloring in O (Δ / k )+log * n rounds, where Δ is the maximum degree of the network graph and k ≥ 1 can be freely chosen. The algorithm is extremely simple: each node locally computes a sequence of colors and then it tries colors from the sequence in batches of size k . Our algorithm subsumes many important results in the history of distributed graph coloring as special cases, including Linial’s color reduction [Linial, FOCS’87], the celebrated locally iterative algorithm from [Barenboim, Elkin, Goldenberg, PODC’18], and various algorithms to compute defective and arbdefective colorings. Our algorithm can smoothly scale between several of these previous results and also simplifies the state-of-the-art (Δ +1)-coloring algorithm. At the cost of losing some of the algorithm’s simplicity we also provide a O ( k Δ)-coloring algorithm in \(O(\sqrt {\Delta /k})+\log ^{*} n\) rounds. We also provide improved deterministic algorithms for ruling sets, and, additionally, we provide a tight characterization for one-round color reduction algorithms.

A Self-Adapting IoT Network Configuration Supported by Distributed Graph Transformations

The research described in this article aims to propose the creation of a framework that would enable the self-optimization of IoT device networks. The work is based on two foundations: distributed graph transformations and a flexible IoT network supported by the several standards and definitions proposed by The Open Group, such as the definition of microservices architecture (MSA) as well as IoT and semantic interoperability standards, providing a broader context for the research. It introduces the concept of capabilities, both at the individual device and network levels, which are used to describe the desired functions that will be performed by the given system. The network of distributed IoT devices is visualized as a graph, and graph transformations are used to specify and optimize the network in response to events like degraded performance, failures, or configuration changes. These actions are automatically performed in order to restore the original set of capabilities defined for the system. Validation of these capabilities is used to assess the success rate of the performed actions. The document describes a practical implementation of an IoT network for managing and monitoring an indoor greenhouse. Thanks to the introduction of formal representation of capabilities, the programming effort required to build the system was significantly reduced. Furthermore, automation related to the validation of capabilities and the performance of automated actions reduces the effort by a factor of a hundred compared with a manual action.

A Client-Cloud-Chain Data Annotation System of Internet of Things for Semi-Supervised Missing Data

With continuous progress in science and technology, a large amount of data are produced in all fields of the world at anytime and anywhere. These data are unmarked and lack marking information, while manual marking is time-consuming and laborious. Herein, this paper introduces a distributed semi-supervised labeling framework. This framework addresses the issue of missing data by proposing an attribute-filling method based on subspace learning. Furthermore, this paper presents a distributed semi-supervised learning strategy that trains sub-models (private models) within each sub-system. Finally, this paper develops a distributed graph convolutional neural network fusion technique with enhanced interpretability grounded on the attention mechanism. This paper assigns weights of importance to the edges of each layer in the graph neural network based on sub-models and public data, thereby enabling distributed and interpretable graph convolutional attention. Extensive experimentation using public datasets demonstrates the superiority of the proposed scheme over other state-of-the-art baselines, achieving a reduction in loss of 50% compared to the original approach.

RAGraph: A Region-Aware Framework for Geo-Distributed Graph Processing

In many global businesses of multinational enterprises, graph-structure data is usually geographically distributed in different regions to support low-latency services. Geo-distributed graph processing suffers from the Wide Area Networks (WANs) with scarce and heterogeneous bandwidth, thus essentially differs from traditional distributed graph processing. In this paper, we propose RAGraph, a Region-Aware framework for geo-distributed graph processing. At the core of RAGraph, we design a region-aware graph processing framework that allows advancing inefficient global updates locally and enables sensible coordination-free message interactions. RAGraph also contains an adaptive hierarchical message interaction engine to switch interaction modes adaptively based on network heterogeneity and fluctuation, and a discrepancy-aware message filtering strategy to filter important messages. Finally, the experiments show that RAGraph can achieve 1.69X - 40.53X speedup and 20.9% - 97% WAN cost reduction compared with state-of-the-art systems.

Distributed half-integral matching and beyond

By prior work, it is known that any distributed graph algorithm that finds a maximal matching requires Ω(log⁎⁡n) communication rounds, while it is possible to find a maximal fractional matching in O(1) rounds in bounded-degree graphs. However, all prior O(1)-round algorithms for maximal fractional matching use arbitrarily fine-grained fractional values. In particular, none of them is able to find a half-integral solution, using only values from {0,12,1}. We show that the use of fine-grained fractional values is necessary, and moreover we give a complete characterization on exactly how small values are needed: if we consider maximal fractional matching in graphs of maximum degree Δ=2d, and any distributed graph algorithm with round complexity T(Δ) that only depends on Δ and is independent of n, we show that the algorithm has to use fractional values with a denominator at least 2d. We give a new algorithm that shows that this is also sufficient.

The energy complexity of diameter and minimum cut computation in bounded-genus networks

This paper investigates the energy complexity of distributed graph problems in multi-hop radio networks, where the energy cost of an algorithm is measured by the maximum number of awake rounds of a vertex. Recent works revealed that some problems, such as broadcast, breadth-first search, and maximal matching, can be solved with energy-efficient algorithms that consume only polylog⁡n energy. However, there exist some problems, such as computing the diameter of the graph, that require Ω(n) energy to solve. To improve energy efficiency for these problems, we focus on a special graph class: bounded-genus graphs. We present algorithms for computing the exact diameter, the exact global minimum cut size, and a (1±ϵ)-approximate s-t minimum cut size with O˜(n) energy for bounded-genus graphs. Our approach is based on a generic framework that divides the vertex set into high-degree and low-degree parts and leverages the structural properties of bounded-genus graphs to control the number of certain connected components in the subgraph induced by the low-degree part.

Distributed Subgraph Query Processing Using Filtering Scores on Spark

As various services have been generating large-scale graphs to represent multiple relationships between objects, studies have been conducted to obtain subgraphs with particular patterns. In this paper, we propose a distributed query processing method to efficiently search a subgraph for a large graph on Spark. To reduce unnecessary processing costs, the search order is determined by filtering scores using the probability distribution. The partitioned queries are searched in parallel in the distributed graph of each slave node according to the search order, and the local search results obtained from each slave node are combined and returned. The query is partitioned in triplets based on the determined search order. The performance of the proposed method is compared with the performance of existing methods to demonstrate its superiority.

Federated Clique Percolation for Privacy-preserving Overlapping Community Detection

Community structure is a typical characteristic of complex networks. Finding communities in complex networks has many important applications, such as the advertisement and recommendation based on social networks and the discovery of new protein molecules in biological networks, which make it a hot topic in the field of complex network analysis. With the increasing concerns about the leakage of personal privacy, discovering communities spread across the local networks owned by multiple participants accurately while preserving each participant’s privacy has become an emerging challenge in distributed community detection. In this article, we propose a general federated graph learning model for privacy-preserving distributed graph learning and develop two federated clique percolation algorithms (CPAs) based on it to discover overlapping communities distributed across multiple participants’ local networks without disclosing any participant’s network privacy. Homomorphic encryption and hash operation are used in combination to protect the privacy of the vertices and edges of each local network. Furthermore, vertex attributes are involved in the calculation of clique similarity and clique percolation when dealing with attributed networks. The experimental results on real-world and artificial datasets demonstrate that the proposed algorithms achieve identical results to those of their stand-alone counterparts and more than 200% higher accuracy than the simple distributed CPAs without federating learning.

GraphTune: An Efficient Dependency-Aware Substrate to Alleviate Irregularity in Concurrent Graph Processing

With the increasing need for graph analysis, massive Concurrent iterative Graph Processing (CGP) jobs are usually performed on the common large-scale real-world graph. Although several solutions have been proposed, these CGP jobs are not coordinated with the consideration of the inherent dependencies in graph data driven by graph topology. As a result, they suffer from redundant and fragmented accesses of the same underlying graph dispersed over distributed platform, because the same graph is typically irregularly traversed by these jobs along different paths at the same time. In this work, we develop GraphTune , which can be integrated into existing distributed graph processing systems, such as D-Galois, Gemini, PowerGraph, and Chaos, to efficiently perform CGP jobs and enhance system throughput. The key component of GraphTune is a dependency-aware synchronous execution engine in conjunction with several optimization strategies based on the constructed cross-iteration dependency graph of chunks. Specifically, GraphTune transparently regularizes the processing behavior of the CGP jobs in a novel synchronous way and assigns the chunks of graph data to be handled by them based on the topological order of the dependency graph so as to maximize the performance. In this way, it can transform the irregular accesses of the chunks into more regular ones so that as many CGP jobs as possible can fully share the data accesses to the common graph. Meanwhile, it also efficiently synchronizes the communications launched by different CGP jobs based on the dependency graph to minimize the communication cost. We integrate it into four cutting-edge distributed graph processing systems and a popular out-of-core graph processing system to demonstrate the efficiency of GraphTune. Experimental results show that GraphTune improves the throughput of CGP jobs by 3.1∼6.2, 3.8∼8.5, 3.5∼10.8, 4.3∼12.4, and 3.8∼6.9 times over D-Galois, Gemini, PowerGraph, Chaos, and GraphChi, respectively.