Big Graph Research Articles

Approach to reduce the communication cost when partitioning a big graph

While several graph mining systems have been developed to run different graph mining algorithms on such large networks, they have difficulty processing Web-scale graphs owing to the significant communication and I/O expenses incurred when workers connect with one another and frequently read subgraphs. In this study, we provide a cost-effective, scalable, distributed method for mining real-world graphs by exploiting graph properties. Our technique drastically decreases the communication cost, the main bottleneck of distributed systems, by using alternate edge placement criteria depending on the types of vertices.

Distributed Partial Simulation for Graph Pattern Matching

Abstract Pattern matching in big graphs is important for different modern applications. Recently, this problem was defined in terms of multiple extensions of graph simulation, to reduce complexity and capture more meaningful results. These results were achieved through the relaxation of commonly used constraint in subgraph isomorphism pattern matching. Nevertheless, these graph simulation variant models are still too strict to provide results in many cases, especially when analyzed graphs contain anomalies and incomplete information. To deal with this issue, we introduce a new graph pattern matching (GPM) method, called partial simulation, capable of retrieving matches despite missing parts of the pattern graph, such as vertices and/or edges. Furthermore, considering the number and inequality of the outputs, we define a relevance function to compute a value expressing how each match vertex respects the pattern graph. Similarly, we define partial dual simulation GPM that returns vertices that satisfy a part of the dual simulation constraints and assigns a relevance value to them. Additionally, we provide distributed scalable algorithms to evaluate the proposed partial simulation methods based on the distributed vertex-centric programming paradigm. Finally, our experiments on real-world data graphs demonstrate the effectiveness of the proposed models and the efficiency of their associated algorithms.

Interval-Valued Intuitionistic Fuzzy Decision With Graph Pattern in Big Graph

Graph pattern matching (GPM) in big graph has been widely used in decision making, such as expert finding, social group discovery, etc. However, these existing works consider neither the preference of the decision maker (DM), nor the subjectivity of constraints during the process of GPM. Therefore, this paper proposes an interval-valued intuitionistic fuzzy decision (IVIFD) with graph pattern in big graph. As traditionally IVIFD can maximally reduce the uncertainty of decision making and it is only valid for small datasets, which makes it impossible to be applied in big graph. In this paper, GPM is adopted to prune the searching space, which makes it possible to process IVIFD under the preference of the DM later. Technically, firstly, each DM selects the preferred vertices and/or edges in big graph, and the interval-valued intuitionistic fuzzy preference (IVIFP) is calculated and used to form the contextual constraints to conduct GPM. Secondly, a probability-certainty density function is introduced to capture the subjective probability of the contextual preference of subgraphs via the bijection from rating space of the context to preference space of the context, which leads to an interval-valued intuitionistic fuzzy set (IVIFS). In addition to the IVIFS, the IVIFD is made through interval-valued intuitionistic fuzzy cross entropy and grey relation degree. Moreover, the weight problem between different contexts is taken into account and handled respectively as three cases. Finally, numerical experiments and perturbation analysis validate the effectiveness and stability of our proposed method, and verify its necessity and efficiency through ablation experiments.

Guest Editorial Special Issue on Computational Intelligence in Big Graph Data Management

The papers in this special section focus on emerging computational intelligence (CI) based theories and methodologies for big graph data management. The emerging CI techniques cover a broad range of nature-inspired, multidisciplinary computational methodologies, such as fuzzy logic, graph pattern matching, graph neural networks, graph embedding, graph attention, evolutionary computing, cognitive computing, learning theory, and probabilistic methods. The objective of this special issue is to explore how CI models and their variants can be adapted, augmented and extended to deal with applications involving very large scale graph data.

Big graphs

Big data is typically characterized with 4V's: Volume, Velocity, Variety and Veracity. When it comes to big graphs, these challenges become even more staggering. Each and every of the 4V's raises new questions, from theory to systems and practice. Is it possible to parallelize sequential graph algorithms and guarantee the correctness of the parallelized computations? Given a computational problem, does there exist a parallel algorithm for it that guarantees to reduce parallel runtime when more machines are used? Is there a systematic method for developing incremental algorithms with effectiveness guarantees in response to frequent updates? Is it possible to write queries across relational databases and semistructured graphs in SQL? Can we unify logic rules and machine learning, to improve the quality of graph-structured data, and deduce associations between entities? This paper aims to incite interest and curiosity in these topics. It raises as many questions as it answers.

BubbleGun: enumerating bubbles and superbubbles in genome graphs.

MotivationWith the fast development of sequencing technology, accurate de novo genome assembly is now possible even for larger genomes. Graph-based representations of genomes arise both as part of the assembly process, but also in the context of pangenomes representing a population. In both cases, polymorphic loci lead to bubble structures in such graphs. Detecting bubbles is hence an important task when working with genomic variants in the context of genome graphs.ResultsHere, we present a fast general-purpose tool, called BubbleGun, for detecting bubbles and superbubbles in genome graphs. Furthermore, BubbleGun detects and outputs runs of linearly connected bubbles and superbubbles, which we call bubble chains. We showcase its utility on de Bruijn graphs and compare our results to vg’s snarl detection. We show that BubbleGun is considerably faster than vg especially in bigger graphs, where it reports all bubbles in less than 30 min on a human sample de Bruijn graph of around 2 million nodes.Availability and implementation BubbleGun is available and documented as a Python3 package at https://github.com/fawaz-dabbaghieh/bubble_gun under MIT license.Supplementary information Supplementary data are available at Bioinformatics online.

Finding locally densest subgraphs

Finding the densest subgraph (DS) from a graph is a fundamental problem in graph databases. The DS obtained, which reveals closely related entities, has been found to be useful in various application domains such as e-commerce, social science, and biology. However, in a big graph that contains billions of edges, it is desirable to find more than one subgraph cluster that are not necessarily the densest, yet they reveal closely-related vertices. In this paper, we study the locally densest subgraph (LDS), a recently-proposed variant of DS. An LDS is a subgraph which is the densest among the "local neighbors". Given a graph G , a number of LDS's can be returned, which reflect different dense regions of G and thus give more information than DS. The existing LDS solution suffers from low efficiency. We thus develop a convex-programming-based solution that enables powerful pruning. Extensive experiments on seven real large graph datasets show that our proposed algorithm is up to four orders of magnitude faster than the state-of-the-art.

Big Data and Knowledge Graph Based Fault Diagnosis for Electric Power Systems

Fault detection plays an important role in the daily maintenance of power electric system. Big data and knowledge graph (KG) have been proposed by researchers to solve many problems in industrial Internet of Things, which also give lots of potentials in improving the performance of fault detection for electric power systems. In particular, this paper analyzes a distributed knowledge graph framework for fault detection in the electric power systems, where multiple devices train their local detection models used for fault detection assisted with a central server. Each device owns its local data set composed of historical fault information and current device state, which can be used to train a local model for fault detection. To enhance the detection performance, the distributed devices interact with each other in the KG framework, where the devices ought to achieve the regional computation in addition to the model aggregation within a specified latency threshold. Through searching for the vibrant qualities together with determined ability at the devices, we enhance the knowledge graph framework by the optimum variety of energetic devices together with the restriction of latency as well as data transmission. Particularly, two data transmission bandwidth allocation (BA) schemes are developed for the distributed knowledge graph framework, through which scheme I is actually bared after the instantaneous device state information (DSI), and scheme II utilizes particle swarm optimization (PSO) technique along with the statistical DSI. The results of simulation on the examination as well as convergence are lastly demonstrated to show the advantages of the proposed distributed KG framework in the fault detection for the electric power systems.

A Personal Growth System Supporting the Sustainable Development of Students Based on Intelligent Graph Element Technology

In recent years, the sustainable development of education has become an increasing concern, and new technology characterized by intelligence has played an important role in promoting it. However, facing the endless stream of teaching platforms, learning platforms, student management platforms and learning APPs, teachers and students are tired of coping. Meanwhile, there has been a serious lack of real information about student growth, especially in the fields of electronic graphics, and audio and video materials. At present, there is no continuous student growth system that can be used over the course of their life, which is very unfavorable to their individual development. Graphic code is a technology with the potential to solve these problems; however, the existing graphic code technology suffers from clear deficiencies in the realization of a personalized student growth system that incorporates intelligence, security and sustainability. In response to this, this paper proposes a new generation of graphic code technology, namely intelligent graph element technology (IGET). Further, a new sustainable personalized student growth system model is designed based on artificial intelligence, big data analysis and intelligent graph element technologies, and the architecture and implementation of this system platform are completed. Finally, a student growth system based on intelligent code is verified through by an analysis of the results of a questionnaire survey. The research results show that, compared with the traditional student management system, the student growth system based on an intelligent graph element code has obvious advantages in convenience, intelligence, precision, security, and sustainability.

Frequent Pattern Mining in Big Social Graphs

With the popularity of graph applications, frequent pattern mining (FPM) has been playing a significant role in many domains, such as social networks and bioinformatics. However, due to the exponential time complexity of FPM, it is a challenge for most existing techniques in big dense graphs, such as social graphs. In this paper, with the defined concept of social pattern, a corresponding linear time computable support calculation measurement, called Minimum Independent Individual based support, is proposed. Then, we adopt the concept of pathgraph to store the appearance of social patterns and propose a novel approach ( <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SocMi</small> ) to solve the problem of frequent social pattern mining. Additionally, in order to reduce the exponential time consumption to explore big graphs, an approximate approach ( <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ASocMi</small> ) with a quick exploration strategy is proposed. Moreover, the proposed approaches have been further optimized by using cache during processing. Finally, an extensive empirical study in real-world social graphs has demonstrated the effectiveness and efficiency of the proposed approaches compared with the state-of-the-art approaches.

Does the traffic volume of a port determine connectivity? Revisiting port connectivity measures with high-frequency satellite data

Seaports play an important role in the global shipping network. Shipping participants often attach great importance to the measurement of container port connectivity, as it reflects countries' access to world markets. As a result, various port connectivity index systems have been proposed by members of the shipping industry and scholars. In recent years, technological developments especially the advancement of high coverage and real-time Automatic Identification System (AIS) data, have provided a chance to improve the scope and frequency of the existing index systems. An improved system is expected to reflect the dynamic changes in a port's connectivity which may be induced by either local disruptions or shocks in the wider economy. This study builds a monthly container port connectivity index system by applying big data mining techniques, graph theory, and principal component analysis (PCA) to AIS data, taking both port factors and shipping network factors into consideration. AIS records from 2020 are used to calculate the connectivity score of 25 major container ports. We also compare our system with the connectivity index commonly used in the shipping industry, the Liner Shipping Connectivity Index (LSCI). Our results show that the measurement of connectivity can be improved over indices that depend primarily on indicators of traffic volume. Ports like Antwerp and Tanjung Pelepas rank high in the proposed system due to their sound performance on their accessibility and strategic position in the local region instead of their traffic volume. The monthly index system is also proven to reflect timely changes in the shipping industry through its accurate portrayal of changes in port connectivity during the COVID-19 outbreak.

Incremental Graph Computations: Doable and Undoable

The incremental problem for a class \( {\mathcal {Q}} \) of graph queries aims to compute, given a query \( Q \in {\mathcal {Q}} \) , graph G , answers Q ( G ) to Q in G and updates ΔG to G as input, changes ΔO to output Q ( G ) such that Q ( G ⊕ ΔG ) = Q ( G )⊕ ΔO . It is called bounded if its cost can be expressed as a polynomial function in the sizes of Q , ΔG and ΔO , which reduces the computations on possibly big G to small ΔG and ΔO . No matter how desirable, however, our first results are negative: For common graph queries such as traversal, connectivity, keyword search, pattern matching, and maximum cardinality matching, their incremental problems are unbounded. In light of the negative results, we propose two characterizations for the effectiveness of incremental graph computation: (a) localizable , if its cost is decided by small neighbors of nodes in ΔG instead of the entire G ; and (b) bounded relative to a batch graph algorithm \( {\mathcal {T}} \) , if the cost is determined by the sizes of ΔG and changes to the affected area that is necessarily checked by any algorithms that incrementalize \( {\mathcal {T}} \) . We show that the incremental computations above are either localizable or relatively bounded by providing corresponding incremental algorithms. That is, we can either reduce the incremental computations on big graphs to small data, or incrementalize existing batch graph algorithms by minimizing unnecessary recomputation. Using real-life and synthetic data, we experimentally verify the effectiveness of our incremental algorithms.

A random walk sampling on knowledge graphs for semantic-oriented statistical tasks

A knowledge graph (KG) manages large-scale and real-world facts as a big graph in a schema-flexible manner, which has recently attracted considerable attention . It is very common that users deploy some statistical tasks on a KG to achieve the latent information of interest. There are two types of the statistical tasks, that are, topology-oriented and semantic-oriented statistical tasks. Many efforts have been made for the former one (e.g., finding the average degree of a KG). The basic idea is concluded as: estimating an approximate statistical result based on a random sample collected through a topology-aware KG sampling approach. Unfortunately, this method cannot be directly deployed to support semantic-oriented statistical tasks (e.g., achieving the average fuel economy of cars produced in Germany), because the topology-aware sampling does not consider the semantics of a KG (or we say the sample is collected only based on the topological information of a KG, while excluding the semantics of a KG), hence leading to a low-quality random sample and would significantly affect the accuracy. In this paper, we propose a semantic-aware random walk sampling on KGs to quickly and accurately collect samples that match the semantic constraint of the semantic-oriented statistical task, and obtain an approximate statistical result by well-designed unbiased estimators. Moreover, we propose an optimization on our semantic-aware sampling to improve the sampling efficiency. Finally, extensive experiments were conducted on our method, which confirmed the effectiveness and efficiency of our approach.

HyGraph: a subgraph isomorphism algorithm for efficiently querying big graph databases

The big graph database provides strong modeling capabilities and efficient querying for complex applications. Subgraph isomorphism which finds exact matches of a query graph in the database efficiently, is a challenging problem. Current subgraph isomorphism approaches mostly are based on the pruning strategy proposed by Ullmann. These techniques have two significant drawbacks- first, they are unable to efficiently handle complex queries, and second, their implementations need the large indexes that require large memory resources. In this paper, we describe a new subgraph isomorphism approach, the HyGraph algorithm, that is efficient both in querying and with memory requirements for index creation. We compare the HyGraph algorithm with two popular existing approaches, GraphQL and Cypher using complexity measures and experimentally using three big graph data sets—(1) a country-level population database, (2) a simulated bank database, and (3) a publicly available World Cup big graph database. It is shown that the HyGraph solution performs significantly better (or equally) than competing algorithms for the query operations on these big databases, making it an excellent candidate for subgraph isomorphism queries in real scenarios.

Discovering association rules from big graphs

This paper tackles two challenges to discovery of graph rules. Existing discovery methods often (a) return an excessive number of rules, and (b) do not scale with large graphs given the intractability of the discovery problem. We propose an application-driven strategy to cut back rules and data that are irrelevant to users' interests, by training a machine learning (ML) model to identify data pertaining to a given application. Moreover, we introduce a sampling method to reduce a big graph G to a set H of small sample graphs. Given expected support and recall bounds, the method is able to deduce samples in H and mine rules from H to satisfy the bounds in the entire G . As proof of concept, we develop an algorithm to discover Graph Association Rules (GARs), which are a combination of graph patterns and attribute dependencies, and may embed ML classifiers as predicates. We show that the algorithm is parallelly scalable, i.e. , it guarantees to reduce runtime when more machines are used. We experimentally verify that the method is able to discover rules with recall above 91% when using sample ratio 10%, with speedup of 61 times.

Special issue on big graph data management and processing

Special issue on big graph data management and processing

Making graphs compact by lossless contraction

This paper proposes a scheme to reduce big graphs to small graphs. It contracts obsolete parts and regular structures into supernodes. The supernodes carry a synopsis S_mathcal {Q} for each query class mathcal {Q} in use, to abstract key features of the contracted parts for answering queries of mathcal {Q}. Moreover, for various types of graphs, we identify regular structures to contract. The contraction scheme provides a compact graph representation and prioritizes up-to-date data. Better still, it is generic and lossless. We show that the same contracted graph is able to support multiple query classes at the same time, no matter whether their queries are label based or not, local or non-local. Moreover, existing algorithms for these queries can be readily adapted to compute exact answers by using the synopses when possible and decontracting the supernodes only when necessary. As a proof of concept, we show how to adapt existing algorithms for subgraph isomorphism, triangle counting, shortest distance, connected component and clique decision to contracted graphs. We also provide a bounded incremental contraction algorithm in response to updates, such that its cost is determined by the size of areas affected by the updates alone, not by the entire graphs. We experimentally verify that on average, the contraction scheme reduces graphs by 71.9% and improves the evaluation of these queries by 1.69, 1.44, 1.47, 2.24 and 1.37 times, respectively.

Fast datalog evaluation for batch and stream graph processing

Implementing complex algorithms for big data, artificial intelligence, and graph processing requires enormous effort. Succinct, declarative programs to solve complex problems that can be efficiently executed for batching and streaming data are in demand. This paper presents Nexus, a distributed Datalog evaluation system. It evaluates Datalog programs using the semi-naive algorithm for batch and streaming data using incremental and asynchronous iteration. Furthermore, we evaluate Datalog programs with aggregates to determine the advantages of implementing the semi-naive algorithm using incremental iteration on its performance. Our experimental results show that Nexus significantly outperforms acyclic dataflow-based systems.

GDLL: A Scalable and Share Nothing Architecture Based Distributed Graph Neural Networks Framework

Deep learning has recently been shown to be effective in uncovering hidden patterns in non-Euclidean space, where data is represented as graphs with complex object relationships and interdependencies. Because of the implicit data dependence in the big graphs with millions of nodes and billions of edges, it is hard for industrial communities to exploit these methods to address real-world challenges at scale. The skewness property of big graphs, distributed file system performance penalty on small k-hop neighborhood subgraphs, and varying size of subgraph makes Graph Neural Networks (GNNs) training further challenging in a distributed environment using parameter servers. To address such issues, we propose a scalable, layered, fault-tolerance, and in-memory distributed computing-based graph neural network framework called Graph Distributed Learning Library (GDLL). The base layer utilizes an optimized distributed file system and a scalable graph data store to reduce the performance penalty. The second layer provides distributed graph processing using in-memory graph programming models while optimizing and hiding the underlying complexity of information complete subgraph computation. In the third layer, GNN modules are deployed on top of the first two layers for efficient distributed training using parameter servers. Finally, we evaluate and compare GDLL with the state-of-the-art solutions and outperform it significantly in terms of efficiency while maintaining similar GNN convergence.

Big graph partitioning

At many levels, from storage infrastructures to programming paradigms, billion-node graphs pose considerable obstacles. It is necessary to provide a general-purpose graph-processing platform. A distributed memory system is thought to be a viable platform for both on-line and offline query processing and graph analytics. In this research, we look at how to partition a billion-node graph on such a platform, which is crucial because it affects load balancing and communication overhead directly. It's difficult not only because the graph is vast, but also because we can't assume that the data can be structured in any way to maximize the partitioning algorithm's speed. Instead, the algorithm must follow the system's and other applications' data and programming models. We present a multi-level label propagation (MLP) method for graph partitioning in this paper.