Graph Computation Research Articles

Distributed Graph Processing System and Processing-in-memory Architecture with Precise Loop-carried Dependency Guarantee

To hide the complexity of the underlying system, graph processing frameworks ask programmers to specify graph computations in user-defined functions (UDFs) of graph-oriented programming model. Due to the nature of distributed execution, current frameworks cannot precisely enforce the semantics of UDFs, leading to unnecessary computation and communication. It exemplifies a gap between programming model and runtime execution. This article proposes novel graph processing frameworks for distributed system and Processing-in-memory (PIM) architecture that precisely enforces loop-carried dependency; i.e., when a condition is satisfied by a neighbor, all following neighbors can be skipped. Our approach instruments the UDFs to express the loop-carried dependency, then the distributed execution framework enforces the precise semantics by performing dependency propagation dynamically. Enforcing loop-carried dependency requires the sequential processing of the neighbors of each vertex distributed in different nodes. We propose to circulant scheduling in the framework to allow different nodes to process disjoint sets of edges/vertices in parallel while satisfying the sequential requirement. The technique achieves an excellent trade-off between precise semantics and parallelism—the benefits of eliminating unnecessary computation and communication offset the reduced parallelism. We implement a new distributed graph processing framework SympleGraph, and two variants of runtime systems— GraphS and GraphSR —for PIM-based graph processing architecture, which significantly outperform the state-of-the-art.

G raph O ne

There is a growing need to perform a diverse set of real-time analytics (batch and stream analytics) on evolving graphs to deliver the values of big data to users. The key requirement from such applications is to have a data store to support their diverse data access efficiently, while concurrently ingesting fine-grained updates at a high velocity. Unfortunately, current graph systems, either graph databases or analytics engines, are not designed to achieve high performance for both operations; rather, they excel in one area that keeps a private data store in a specialized way to favor their operations only. To address this challenge, we have designed and developed G raph O ne , a graph data store that abstracts the graph data store away from the specialized systems to solve the fundamental research problems associated with the data store design. It combines two complementary graph storage formats (edge list and adjacency list) and uses dual versioning to decouple graph computations from updates. Importantly, it presents a new data abstraction, GraphView , to enable data access at two different granularities of data ingestions (called data visibility ) for concurrent execution of diverse classes of real-time graph analytics with only a small data duplication. Experimental results show that G raph O ne is able to deliver 11.40× and 5.36× average speedup in ingestion rate against LLAMA and Stinger, the two state-of-the-art dynamic graph systems, respectively. Further, they achieve an average speedup of 8.75× and 4.14× against LLAMA and 12.80× and 3.18× against Stinger for BFS and PageRank analytics (batch version), respectively. G raph O ne also gains over 2,000× speedup against Kickstarter, a state-of-the-art stream analytics engine in ingesting the streaming edges and performing streaming BFS when treating first half as a base snapshot and rest as streaming edge in a synthetic graph. G raph O ne also achieves an ingestion rate of two to three orders of magnitude higher than graph databases. Finally, we demonstrate that it is possible to run concurrent stream analytics from the same data store.

CSMqGraph: Coarse-Grained and Multi-external-storage Multi-queue I/O Management for Graph Computing

As graphs continue growing, external storage graph processing systems serve as a promising alternative to distributed in-memory solutions for low cost and high scalability. To obtain high I/O throughput, these systems usually use multiple external storage devices. They adopt the operating system I/O management method based on striped volume, resulting in unsatisfactory performance, such as low sequential bandwidth utilization of each external storage device, limited I/O parallelism and expensive management overhead. In this paper, we analyzed the problems of the operating system I/O management method based on striped volume. Then we designed CSMqGraph, a graph processing system adopts coarse-grained striping method matching sequential large I/O to fully utilize the maximum sequential bandwidth of each external storage device and an I/O management strategy based on multi-external-storage multi-queue making I/O threads dedicated to each external storage device to further improve I/O throughput and fully exploit the parallelism of multiple external storage devices. For different graph algorithms and datasets, our evaluation shows that CSMqGraph consistently outperforms state-of-the-art engines GridGraph by up to 40%, and has better I/O scalability.

Computation Graphs for AAD and Machine Learning Part I: Introduction to Computation Graphs and Automatic Differentiation

First in a series of three articles with code, exploring the notion of computation graph, with words mathematics and code, and application in Machine Learning and finance to compute a vast number of derivative sensitivities with spectacular speed and accuracy.

A Novel Graph-Based Energy Management System

The typical five-minute computational latency of current Energy Management Systems (EMSs) can leave and has left system operators blind to the system state created by events that lead to cascading outages, system separation and/or system collapse. In this paper, we present a graph-computing based EMS that allows the EMS computational cycle time to be reduced to the SCADA sampling cycle, about five seconds. This is achieved in part by taking advantage of both nodal and hierarchical parallelism, to achieve high-speed graph-based state estimation, power flow, and contingency analysis applications. A production-mode case study on an existing large-scale system verifies the advantages and performance of graph computing, demonstrating that the graph-based EMS is more than 20 times faster than a contemporary commercial EMS.

Academic Team Formulation Based on Liebig’s Barrel: Discovery of Anticask Effect

Academic team formulation is one of the most critical and fundamental issues in team research. Inspired by the Liebig’s metaphor, we formulate an academic team by a barrel composed of planks, namely, Liebig’s barrel. We study the capacity of Liebig’s barrel from several aspects. Then, we abstract the features in academic teams as those in Liebig’s barrel due to the correspondences between team size and plank number, individual member ability and plank height, individual contribution and plank width, allocation of work and plank combination, teamwork attitude and plank tilt angle, team output property, and barrel’s content granularity. The investigated model can also be extended to formulate various multifactor objective optimization issues for team science research. In order to verify the rationality of the proposed model, we implement case studies of this model in both the American Physics Society and the Microsoft Academic Graph Computer Science data sets. The experimental results show that the investigated model is effective in academic team formulation. Moreover, an interesting phenomenon is discovered in the proposed model, which we name as the anticask effect that a team’s output quality mainly lies on the team member having the maximal research ability rather than those having minimal research ability. We then analyze the underlying reasons for the anticask effect and multiple-factor interaction on academic teams.

On Computing Entity Relatedness in Wikipedia, with Applications

Many text mining tasks, such as clustering, classification, retrieval, and named entity linking, benefit from a measure of relatedness between entities in a knowledge graph. We present a thorough study of all entity relatedness measures in recent literature based on Wikipedia as the knowledge graph. To facilitate this study, we introduce a new dataset with human judgments of entity relatedness. No clear dominance is seen between measures based on textual similarity and graph proximity. Some of the better measures involve expensive global graph computations. We propose a new, space-efficient, computationally lightweight, two-stage framework for relatedness computation. In the first stage, a small weighted subgraph is dynamically grown around the two query entities; in the second stage, relatedness is derived based on computations on this subgraph. Our system shows better agreement with human judgment than existing proposals both on the new dataset and on an established one. Our framework also shows improvements with respect to the state-of-the-art on three different extrinsic evaluations in the domains of ranking entity pairs, entity linking, and synonym extraction.

Automatic Identification of Overpass Structures: A Method of Deep Learning

The identification of overpass structures in road networks has great significance for multi-scale modeling of roads, congestion analysis, and vehicle navigation. The traditional vector-based methods identify overpasses by the methodologies coming from computational geometry and graph theory, and they overly rely on the artificially designed features and have poor adaptability to complex scenes. This paper presents a novel method of identifying overpasses based on a target detection model (Faster-RCNN). This method utilizes raster representation of vector data and convolutional neural networks (CNNs) to learn task adaptive features from raster data, then identifies the location of an overpass by a Region Proposal network (RPN). The contribution of this paper is: (1) An overpass labelling geodatabase (OLGDB) for the OpenStreetMap (OSM) road network data of six typical cities in China is established; (2) Three different CNNs (ZF-net, VGG-16, Inception-ResNet V2) are integrated into Faster-RCNN and evaluated by accuracy performance; (3) The optimal combination of learning rate and batchsize is determined by fine-tuning; and (4) Five geometric metrics (perimeter, area, squareness, circularity, and W/L) are synthetized into image bands to enhance the training data, and their contribution to the overpass identification task is determined. The experimental results have shown that the proposed method has good accuracy performance (around 90%), and could be improved with the expansion of OLGDB and switching to more sophisticated target detection models. The deep learning target detection model has great application potential in large-scale road network pattern recognition, it can task-adaptively learn road structure features and easily extend to other road network patterns.

H2Pregel : A partition-based hybrid hierarchical graph computation approach

A partition-based hybrid hierarchical graph computation approach, called H2Pregel is proposed to address the redundant supersteps and inefficient computation problems due to low access locality. The H2Pregel preprocesses the input graph through a distributed recode algorithm to ensure the continuity and sequence of vertex ids, then employs a hybrid approach to combine the advantages of both synchronous and asynchronous models, and hierarchically computes the high proportion of interior messages generated by high quality partition algorithms. Moreover, H2Pregel leverages configurable parallel threads to accelerate local computation by “sub-supersteps”, and employs an exterior messages stealing optimization to avoid extra communication overheads between tasks. We implemented H2Pregel on Giraph, a classic open source system based on Pregel. The evaluation results on large-scale graphs show that, compared with Pregel in three partition algorithms, H2Pregel can achieve average speedups by 1.12–4.52 times and decrease average communication messages by 23.5%-55.5%, and average supersteps by 15.8%-82.0%.

Exemplar based regular texture synthesis using LSTM

Exemplar based texture synthesis is an important technique for image processing and computer graph in texture mapping. So far, great achievements have been made in this field. However, both traditional and modern methods based on deep learning are making errors in synthesizing patterned texture due to the failure for catching the regularity of texture. To obtain a better synthesized result, a new framework for regular texture synthesis is proposed in this paper. Besides, we use recurrent neural network (RNN) of long-shot term memory (LSTM) to produce a regular texture based on exemplar. Our method can generate at any size texture without errors, which is an improvement for texture synthesis with deep learning techniques.Compared with traditional method as well as deep learning method, our method is obviously better in synthesizing regular texture.

EpiGraph: an open-source platform to quantify epithelial organization.

SummaryHere we present EpiGraph, an image analysis tool that quantifies epithelial organization. Our method combines computational geometry and graph theory to measure the degree of order of any packed tissue. EpiGraph goes beyond the traditional polygon distribution analysis, capturing other organizational traits that improve the characterization of epithelia. EpiGraph can objectively compare the rearrangements of epithelial cells during development and homeostasis to quantify how the global ensemble is affected. Importantly, it has been implemented in the open-access platform Fiji. This makes EpiGraph very user friendly, with no programming skills required.Availability and implementationEpiGraph is available at https://imagej.net/EpiGraph and the code is accessible (https://github.com/ComplexOrganizationOfLivingMatter/Epigraph) under GPLv3 license.Supplementary information Supplementary data are available at Bioinformatics online.

State complexity characterizations of parameterized degree-bounded graph connectivity, sub-linear space computation, and the linear space hypothesis

The linear space hypothesis is a practical working hypothesis, which originally states the insolvability of a restricted 2CNF Boolean formula satisfiability problem parameterized by the number of Boolean variables. From this hypothesis, it naturally follows that the degree-3 directed graph connectivity problem (3DSTCON) parameterized by the number of vertices in a given graph cannot belong to PsubLIN, composed of all parameterized decision problems computable by polynomial-time, sub-linear-space deterministic Turing machines. This hypothesis immediately implies L≠NL and it was used as a solid foundation to obtain new lower bounds on the computational complexity of various NL search and NL optimization problems. The state complexity of transformation refers to the cost of converting one type of finite automata to another type, where the cost is measured in terms of the increase of the number of inner states of the converted automata from that of the original automata. We relate the linear space hypothesis to the state complexity of transforming restricted 2-way nondeterministic finite automata to computationally equivalent 2-way alternating finite automata having narrow computation graphs. For this purpose, we present state complexity characterizations of 3DSTCON and PsubLIN. We further characterize a nonuniform version of the linear space hypothesis in terms of the state complexity of transformation.

Differentiable Programming Tensor Networks

Differentiable programming is a fresh programming paradigm which composes parameterized algorithmic components and trains them using automatic differentiation (AD). The concept emerges from deep learning but is not only limited to training neural networks. We present theory and practice of programming tensor network algorithms in a fully differentiable way. By formulating the tensor network algorithm as a computation graph, one can compute higher order derivatives of the program accurately and efficiently using AD. We present essential techniques to differentiate through the tensor networks contractions, including stable AD for tensor decomposition and efficient backpropagation through fixed point iterations. As a demonstration, we compute the specific heat of the Ising model directly by taking the second order derivative of the free energy obtained in the tensor renormalization group calculation. Next, we perform gradient based variational optimization of infinite projected entangled pair states for quantum antiferromagnetic Heisenberg model and obtain start-of-the-art variational energy and magnetization with moderate efforts. Differentiable programming removes laborious human efforts in deriving and implementing analytical gradients for tensor network programs, which opens the door to more innovations in tensor network algorithms and applications.

Demystifying differentiable programming: shift/reset the penultimate backpropagator

Deep learning has seen tremendous success over the past decade in computer vision, machine translation, and gameplay. This success rests crucially on gradient-descent optimization and the ability to “learn” parameters of a neural network by backpropagating observed errors. However, neural network architectures are growing increasingly sophisticated and diverse, which motivates an emerging quest for even more general forms of differentiable programming, where arbitrary parameterized computations can be trained by gradient descent. In this paper, we take a fresh look at automatic differentiation (AD) techniques, and especially aim to demystify the reverse-mode form of AD that generalizes backpropagation in neural networks. We uncover a tight connection between reverse-mode AD and delimited continuations, which permits implementing reverse-mode AD purely via operator overloading and without managing any auxiliary data structures. We further show how this formulation of AD can be fruitfully combined with multi-stage programming (staging), leading to an efficient implementation that combines the performance benefits of deep learning frameworks based on explicit reified computation graphs (e.g., TensorFlow) with the expressiveness of pure library approaches (e.g., PyTorch).

Special Issue: Graph Computing

Special Issue: Graph Computing

Finite element discretizations of nonlocal minimal graphs: Convergence

In this paper, we propose and analyze a finite element discretization for the computation of fractional minimal graphs of order s∈(0,1∕2) on a bounded domain Ω. Such a Plateau problem of order s can be reinterpreted as a Dirichlet problem for a nonlocal, nonlinear, degenerate operator of order s+1∕2. We prove that our numerical scheme converges in W12r(Ω) for all r<s, where W12s(Ω) is closely related to the natural energy space. Moreover, we introduce a geometric notion of error that, for any pair of H1 functions, in the limit s→1∕2 recovers a weighted L2-discrepancy between the normal vectors to their graphs. We derive error bounds with respect to this novel geometric quantity as well. In spite of performing approximations with continuous, piecewise linear, Lagrangian finite elements, the so-called stickiness phenomenon becomes apparent in the numerical experiments we present.

A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting

This paper presents the winning submission of the M4 forecasting competition. The submission utilizes a dynamic computational graph neural network system that enables a standard exponential smoothing model to be mixed with advanced long short term memory networks into a common framework. The result is a hybrid and hierarchical forecasting method.

Deep Reactive Policies for Planning in Stochastic Nonlinear Domains

Recent advances in applying deep learning to planning have shown that Deep Reactive Policies (DRPs) can be powerful for fast decision-making in complex environments. However, an important limitation of current DRP-based approaches is either the need of optimal planners to be used as ground truth in a supervised learning setting or the sample complexity of high-variance policy gradient estimators, which are particularly troublesome in continuous state-action domains. In order to overcome those limitations, we introduce a framework for training DRPs in continuous stochastic spaces via gradient-based policy search. The general approach is to explicitly encode a parametric policy as a deep neural network, and to formulate the probabilistic planning problem as an optimization task in a stochastic computation graph by exploiting the re-parameterization of the transition probability densities; the optimization is then solved by leveraging gradient descent algorithms that are able to handle non-convex objective functions. We benchmark our approach against stochastic planning domains exhibiting arbitrary differentiable nonlinear transition and cost functions (e.g., Reservoir Control, HVAC and Navigation). Results show that DRPs with more than 125,000 continuous action parameters can be optimized by our approach for problems with 30 state fluents and 30 action fluents on inexpensive hardware under 6 minutes. Also, we observed a speedup of 5 orders of magnitude in the average inference time per decision step of DRPs when compared to other state-of-the-art online gradient-based planners when the same level of solution quality is required.

Efficient Derivative Evaluation for Rigid-Body Dynamics Based on Recursive Algorithms Subject to Kinematic and Loop Constraints

Simulation and control of robotic and bio-mechanical systems depend on a mathematical model description, typically a rigid-body system connected by joints, for which efficient algorithms to compute the forward or inverse dynamics exist. Gradient-based optimization and control methods require derivatives of the dynamics, often approximated by numerical differentiation (FD). However, they benefit from accurate gradients, which promote faster convergence, less iterations, and improved handling of nonlinearities or ill-conditioning of the problem formulations, which are particularly observed when kinematic constraints are involved. In this letter, we apply algorithmic differentiation (AD) to propagate sensitivities through dynamics algorithms. To this end, we augment the computational graph of these algorithms with derivative information. We provide analytic derivatives for elementary operations, in particular matrix factorizations of the descriptor form of the equation of motions, which yields a very efficient derivative evaluation for constrained dynamics. The proposed approach is implemented within the free software package rigid body dynamics library (RBDL), which heavily employs so-called spatial transformations in its implementation of the dynamics algorithms. Thus, manipulations of spatial transformations are treated as elementary operations. The efficiency is improved further by sparsity exploitation. We validate and benchmark the implementation against its FD counterpart for a lifting motion of a human model.

FiniteFlow: multivariate functional reconstruction using finite fields and dataflow graphs

Complex algebraic calculations can be performed by reconstructing analytic results from numerical evaluations over finite fields. We describe FiniteFlow, a framework for defining and executing numerical algorithms over finite fields and reconstructing multivariate rational functions. The framework employs computational graphs, known as dataflow graphs, to combine basic building blocks into complex algorithms. This allows to easily implement a wide range of methods over finite fields in high-level languages and computer algebra systems, without being concerned with the low-level details of the numerical implementation. This approach sidesteps the appearance of large intermediate expressions and can be massively parallelized. We present applications to the calculation of multi-loop scattering amplitudes, including the reduction via integration-by-parts identities to master integrals or special functions, the computation of differential equations for Feynman integrals, multi-loop integrand reduction, the decomposition of amplitudes into form factors, and the derivation of integrable symbols from a known alphabet. We also release a proof-of-concept C++ implementation of this framework, with a high-level interface in Mathematica.