Graph Streams Research Articles

Road Detection via a Dual-Task Network Based on Cross-Layer Graph Fusion Modules

Road detection based on remote sensing images is of great significance to intelligent traffic management. The performances of the mainstream road detection methods are mainly determined by their extracted features, whose richness and robustness can be enhanced by fusing features of different types and cross-layer connections. However, the features in the existing mainstream model frameworks are often similar in the same layer by the single-task training, and the traditional cross-layer fusion ways are too simple to obtain an efficient effect, so more complex fusion ways besides concatenation and addition deserve to be explored. Aiming at the above defects, we propose a dual-task network (DTnet) for road detection and cross-layer graph fusion module (CGM): the DTnet consists of two parallel branches for road area and edge detection, respectively, while enhancing the feature diversity by fusing features between two branches through our designed feature bridge modules (FBM). The CGM improves the cross-layer fusion effect by a complex feature stream graph, and four graph patterns are evaluated. Experimental results on three public datasets demonstrate that our method effectively improves the final detection result.

BSR-TC: Adaptively Sampling for Accurate Triangle Counting over Evolving Graph Streams

Triangle counting is a fundamental graph mining problem, widely employed in various real-world application scenarios. Given the large scale of graph streams and limited memory space, it is feasible to achieve the estimation of global and local triangles by sampling. Existing streaming algorithms for triangle counting can be generalized into two categories: Reservoir-based methods and Bernoulli-based methods. The former use a fixed memory budget, whose size is difficult to set for accurate estimation without any prior knowledge about graph streams. The latter sample edges by a specified probability, but memory budget is uncontrollable for following a binomial distribution. In this work, we propose a novel and bounded-sampling-ratio algorithm for both global and local triangle counting, called BSR-TC, by adaptively resizing memory budget upwards over evolving graph streams. Specifically, our proposed single-pass BSR-TC can gain more advantage than the state-of-the-art algorithms over the continuous growth of graph streams. Experimental results show that BSR-TC achieves accuracy of at least 99.8% for global triangles, when the ratio of initial memory budget against whole graph streams [Formula: see text] and given [Formula: see text], respectively.

The power of summarization in graph mining and learning

Our ability to generate, collect, and archive data related to everyday activities, such as interacting on social media, browsing the web, and monitoring well-being, is rapidly increasing. Getting the most benefit from this large-scale data requires analysis of patterns it contains, which is computationally intensive or even intractable. Summarization techniques produce compact data representations (summaries) that enable faster processing by complex algorithms and queries. This talk will cover summarization of interconnected data (graphs) [3], which can represent a variety of natural processes (e.g., friendships, communication). I will present an overview of my group's work on bridging the gap between research on summarized network representations and real-world problems. Examples include summarization of massive knowledge graphs for refinement [2] and on-device querying [4], summarization of graph streams for persistent activity detection [1], and summarization within graph neural networks for fast, interpretable classification [5]. I will conclude with open challenges and opportunities for future research.

DMatrix: Toward fast and accurate queries in graph stream

The graph stream has recently arisen in many interactive scenarios. Characterizing with large volume and high dynamic, graph streams are known to be difficult for high-speed summary and analysis, especially provided with limited resource availability. Existing solutions mainly use sketch-based methods to estimate the weight of items (e.g., Count-Min Sketch) and preserve the underlying graph structure information (e.g., TCM). Unfortunately, these solutions neither support complex graph-based queries nor achieve efficient real-time queries. In view of these limitations, we design DMatrix, a novel 3-dimensional graph sketch to facilitate fast and accurate queries in graph stream. Both structural query and weight-based estimation are supported with DMatrix. Through the integration of representative key reservation and majority voting, DMatrix can effectively narrow the error bounds of queries with real-time response efficiency. Both theoretical analysis and experimental results confirm that our solution is superior in accuracy and efficiency comparing with the state-of-the-art.

Hammer lightweight graph partitioner based on graph data volumes

The graph partitioning challenge is well known and ongoing classical problem. Many heuristic methods tried to propose solutions focusing mainly on load processing and cost-efficiency. With the emergency of big data technology, the graph partitioning challenge became even more demanding, as an imminent need to handle big volume of data in real time. This reveals a new challenge as most of the existing studies does not consider the volume metric with their streaming graph algorithms causing imbalanced workloads and graph storage. With this article, we propose a specific lightweight algorithm which we called “Hammer” Algorithm. Our proposed Hammer algorithm is a streaming graph based on volume metric to ensure optimal load processing and communication cost efficiency. Our proof of concept was done on real world dataset and the Hammer algorithm showed considerable performance against some existing graph partitioning algorithms.

Interactive Graph Stream Analytics in Arkouda

Data from emerging applications, such as cybersecurity and social networking, can be abstracted as graphs whose edges are updated sequentially in the form of a stream. The challenging problem of interactive graph stream analytics is the quick response of the queries on terabyte and beyond graph stream data from end users. In this paper, a succinct and efficient double index data structure is designed to build the sketch of a graph stream to meet general queries. A single pass stream model, which includes general sketch building, distributed sketch based analysis algorithms and regression based approximation solution generation, is developed, and a typical graph algorithm—triangle counting—is implemented to evaluate the proposed method. Experimental results on power law and normal distribution graph streams show that our method can generate accurate results (mean relative error less than 4%) with a high performance. All our methods and code have been implemented in an open source framework, Arkouda, and are available from our GitHub repository, Bader-Research. This work provides the large and rapidly growing Python community with a powerful way to handle terabyte and beyond graph stream data using their laptops.

Query-driven video event processing for the internet of multimedia things

Advances in Deep Neural Network (DNN) techniques have revolutionized video analytics and unlocked the potential for querying and mining video event patterns. This paper details GNOSIS, an event processing platform to perform near-real-time video event detection in a distributed setting. GNOSIS follows a serverless approach where its component acts as independent microservices and can be deployed at multiple nodes. GNOSIS uses a declarative query-driven approach where users can write customize queries for spatiotemporal video event reasoning. The system converts the incoming video streams into a continuous evolving graph stream using machine learning (ML) and DNN models pipeline and applies graph matching for video event pattern detection. GNOSIS can perform both stateful and stateless video event matching. To improve Quality of Service (QoS), recent work in GNOSIS incorporates optimization techniques like adaptive scheduling, energy efficiency, and content-driven windows. This paper demonstrates the Occupational Health and Safety query use cases to show the GNOSIS efficacy.

Maximizing Influence Over Streaming Graphs with Query Sequence

Now, with the prevalence of social media, such as Facebook, Weibo, how to maximize influence of individuals, products, actions in new media is of practical significance. Generally, maximizing influence first needs to identify the most influential individuals since they can spread their influence to most of others in the social media. Many studies on influence maximization aimed to select a subset of nodes in static graphs once. Actually, real graphs are evolving. So, influential individuals are also changing. In these scenarios, people tend to select influential individuals multiple times instead of once. Namely, selections are raised sequentially, forming a sequence (query sequence). It raises several new challenges due to changing influential individuals. In this paper, we explore the problem of Influence Maximization over Streaming Graph (SGIM). Then, we design a compact solution for storing and indexing streaming graphs and influential nodes that eliminates the redundant computation. The solution includes Influence-Increment-Index along with two sketch-centralized indices called Influence-Index and Reverse-Influence-Index. Computing influence set of nodes will incur a large number of redundant computations. So, these indices are designed to keep track of the nodes’ influence in sketches. Finally, with the indexing scheme, we present the algorithm to answer SGIM queries. Extensive experiments on several real-world datasets demonstrate that our method is competitive in terms of both efficiency and effectiveness owing to the design of index.

CoCoS: Fast and Accurate Distributed Triangle Counting in Graph Streams

Given a graph stream, how can we estimate the number of triangles in it using multiple machines with limited storage? Specifically, how should edges be processed and sampled across the machines for rapid and accurate estimation? The count of triangles (i.e., cliques of size three) has proven useful in numerous applications, including anomaly detection, community detection, and link recommendation. For triangle counting in large and dynamic graphs, recent work has focused largely on streaming algorithms and distributed algorithms but little on their combinations for “the best of both worlds.” In this work, we propose CoCoS , a fast and accurate distributed streaming algorithm for estimating the counts of global triangles (i.e., all triangles) and local triangles incident to each node. Making one pass over the input stream, CoCoS carefully processes and stores the edges across multiple machines so that the redundant use of computational and storage resources is minimized. Compared to baselines, CoCoS is: (a) accurate: giving up to smaller estimation error; (b) fast : up to faster, scaling linearly with the size of the input stream; and (c) theoretically sound : yielding unbiased estimates.

A Workload-Adaptive Streaming Partitioner for Distributed Graph Stores

Streaming graph partitioning methods have recently gained attention due to their ability to scale to very large graphs with limited resources. However, many such methods do not consider workload and graph characteristics. This may degrade the performance of queries by increasing inter-node communication and computational load imbalance. Moreover, existing workload-aware methods cannot consistently provide good performance as they do not consider dynamic workloads that keep emerging in graph applications. We address these issues by proposing a novel workload-adaptive streaming partitioner named WASP, that aims to achieve low-latency and high-throughput online graph queries. As each workload typically contains frequent query patterns, WASP exploits the existing workload to capture active vertices and edges which are frequently visited and traversed, respectively. This information is used to heuristically improve the quality of partitions either by avoiding the concentration of active vertices in a few partitions proportional to their visit frequencies or by reducing the probability of the cut of active edges proportional to their traversal frequencies. In order to assess the impact of WASP on a graph store and to show how easily the approach can be plugged on top of the system, we exploit it in a distributed graph-based RDF store. Our experiments over three synthetic and real-world graph datasets and the corresponding static and dynamic query workloads show that WASP achieves a better query performance against state-of-the-art graph partitioners, especially in dynamic query workloads.

Effective partitioning mechanisms for time-evolving graphs in the Flink system

Graphs are suitable data structures for expressing the relationship between different types of data. With a continuous increase in the graph size, using suitable methods to divide graphs and parallelize the processing load becomes crucial. Balanced graph partitioning has been extensively studied for static and streaming graphs. However, for a time-evolving graph (TEG), whose size and structure are periodically updated, related partitioning methods are lacking. A straightforward approach is to capture snapshots of a TEG and adopt the partitioning methods designed for static or streaming graphs. Although feasible partitioning quality can be expected, the time overhead is high due to frequent repartitioning. This paper proposes two TEG partitioning methods, namely seed and similarity, to decrease the partitioning time. According to the experimental results, on average, seed and similarity require 29–39% of the partitioning time required by snapshot. Moreover, the proposed methods maintain reasonable partitioning quality.

Tracking triadic cardinality distributions for burst detection in high-speed graph streams

In everyday life, we often observe unusually frequent interactions among people before or during important events, e.g., people send/receive more greetings to/from their friends on holidays than regular days. We also observe that some videos or hashtags suddenly go viral through people’s sharing on online social networks (OSNs). Do these seemingly different phenomena share a common structure? All these phenomena are associated with the sudden surges of node interactions in networks, which we call “bursts” in this work. We uncover that, in many scenarios, the emergence of a burst is accompanied with the formation of triangles in networks. This finding motivates us to propose a new and robust method for burst detection on an OSN. We first introduce a new measure, i.e., “triadic cardinality distribution,” corresponding to the fractions of nodes with different numbers of triangles, i.e., triadic cardinalities, in a network. We show that this distribution not only changes when a burst occurs, but it also has a robustness property that it is immunized against common spamming social-bot attacks. Hence, by tracking triadic cardinality distributions, we can more reliably detect bursts than simply counting node interactions on an OSN. To avoid handling massive activity data generated by OSN users during the triadic tracking, we design an efficient “sample-estimate” framework to provide maximum likelihood estimate of the triadic cardinality distribution. We propose several sampling methods and provide insights into their performance difference through both theoretical analysis and empirical experiments on real-world networks.

Analyzing a large and unobtainable relationship graph using a streaming activity graph

The availability of large-scale data about interactions of social media users allows the study of complex human behavior. Graphs are typically employed to represent user interactions, but several algorithms become impractical for analyzing large graphs. Hence, it can be useful to analyze a small sub-graph instead in a practice known as graph sampling. However, if the graph is unobtainable, for example, due to privacy limitations, graph sampling is impossible. We introduce an innovative algorithm for representing a large unobtainable graph of user relationships such as Facebook friendships, using a streaming graph of user activity that can include, for example, wall posts on Facebook. We applied different methods of the proposed algorithm to two large datasets. The results show that averages and distribution statistics of nodes in a large, unobtainable relationship graph are well represented by a graph of about 20% of the size of the unobtainable graph. Finally, we apply the proposed algorithm to identify influencers in an unobtainable graph by analyzing a representative graph. We find that 63% to 76% of identified influencers in the representative graph act as influencers in the unobtainable graph, suggesting that the developed algorithm can effectively capture properties of the unobtainable graph.

An Approach For Concept Drift Detection in a Graph Stream Using Discriminative Subgraphs

The emergence of mining complex networks like social media, sensor networks, and the world-wide-web has attracted considerable research interest. In a streaming scenario, the concept to be learned can change over time. However, while there has been some research done for detecting concept drift in traditional data streams, little work has been done on addressing concept drift in data represented as a graph . We propose a novel unsupervised concept-drift detection method on graph streams called Discriminative Subgraph-based Drift Detector (DSDD). The methodology starts by discovering discriminative subgraphs for each graph in the stream. We then compute the entropy of the window based on the distribution of discriminative subgraphs with respect to the graphs and then use the direct density-ratio estimation approach for detecting concept drift in the series of entropy values obtained by moving one step forward in the sliding window. The effectiveness of the proposed method is demonstrated through experiments using artificial and real-world datasets and its performance is evaluated by comparing against related baseline methods. Similarly, the usefulness of the proposed concept drift detection approach is studied by incorporating it in a popular graph stream classification algorithm and studying the impact of drift detection in classification accuracy.

Fast shared-memory streaming multilevel graph partitioning

A fast parallel graph partitioner can benefit many applications by reducing data transfers. The online methods for partitioning graphs have to be fast and they often rely on simple one-pass streaming algorithms, while the offline methods for partitioning graphs contain more involved algorithms and the most successful methods in this category belong to the multilevel approaches. In this work, we assess the feasibility of using streaming graph partitioning algorithms within the multilevel framework. Our end goal is to come up with a fast parallel offline multilevel partitioner that can produce competitive cutsize quality. We rely on a simple but fast and flexible streaming algorithm throughout the entire multilevel framework. This streaming algorithm serves multiple purposes in the partitioning process: a clustering algorithm in the coarsening, an effective algorithm for the initial partitioning, and a fast refinement algorithm in the uncoarsening. Its simple nature also lends itself easily for parallelization. The experiments on various graphs show that our approach is on the average up to 5.1x faster than the multi-threaded MeTiS, which comes at the expense of only 2x worse cutsize.

Approximate Top-k Subgraph Matching Scheme Considering Data Reuse in Large Graph Stream Environments

Approximate Top-k Subgraph Matching Scheme Considering Data Reuse in Large Graph Stream Environments

Distributed Computation Approach for Link Prediction in Graph Stream using DCS Features and Gradient Boosted Trees

Distributed Computation Approach for Link Prediction in Graph Stream using DCS Features and Gradient Boosted Trees

Temporal locality-aware sampling for accurate triangle counting in real graph streams

If we cannot store all edges in a dynamic graph, which edges should we store to estimate the triangle count accurately? Counting triangles (i.e., cliques of size three) is a fundamental graph problem with many applications in social network analysis, web mining, anomaly detection, etc. Recently, much effort has been made to accurately estimate the counts of global triangles (i.e., all triangles) and local triangles (i.e., all triangle incident to each node) in large dynamic graphs, especially with limited space. Although existing algorithms use sampling techniques without considering temporal dependencies in edges, we observe temporal locality in the formation of triangles in real dynamic graphs. That is, future edges are more likely to form triangles with recent edges than with older edges. In this work, we propose a family of single-pass streaming algorithms called Waiting-Room Sampling (WRS) for estimating the counts of global and local triangles in a fully dynamic graph, where edges are inserted and deleted over time, within a fixed memory budget. WRS exploits the temporal locality by always storing the most recent edges, which future edges are more likely to form triangles with, in the waiting room, while it uses reservoir sampling and its variant for the remaining edges. Our theoretical and empirical analyses show that WRS is: (a) Fast and ‘any time’: runs in linear time, always maintaining and updating estimates, while the input graph evolves, (b) Effective: yields up to 47% smaller estimation error than its best competitors, and (c) Theoretically sound: gives unbiased estimates with small variances under the temporal locality.

Rapid Detection of Local Communities in Graph Streams

We examine the problem of uncovering communities in complex networks whose elements and their respective associations manifest as streams of data. Community detection is applied in emerging computational environments and concerns critical applications in diverse areas. Despite the already expended related research efforts, the task of revealing the community structure of massive and rapidly-evolving networks remains very challenging. There is an emerging need for online approaches that ingest graph data as a stream. We propose a streaming-graph community-detection algorithm that expands seed-sets of nodes to communities. We consider an online setting and process a stream of edges while aiming to form communities using partial knowledge of the graph. We maintain very limited information regarding the nodes of the graph and the communities we seek, so that we effectively process large scale networks. In addition, we develop a technique that increases the accuracy of our algorithm considerably and propose a new clustering algorithm that allows for automatically deriving the size of the communities sought. Our experimental evaluation using ground-truth communities for a wide range of large real-word and synthetic networks shows that our approach achieves accuracy comparable or even better to the state-of-the-art non-streaming community detection algorithms. More importantly, we attain significant improvements in both execution time and memory requirements.

GS4: Graph stream summarization based on both the structure and semantics

Nowadays internet-based applications collect and distribute large datasets, which are mostly modeled by pertinent massive graphs. One solution to process such massive graphs is summarization. There are two kinds of graphs, stationary and stream. There are several algorithms to summarize stationary graphs; however, no comprehensive method has been devised to summarize stream graphs. This is because of the challenges of the graph stream, which are the high data volume and the continuous changes of data over time. To tackle such challenges, we propose a novel method based on the sliding window model that performs summarization using both the structure and vertex attributes of the input graph stream. We devise a new structure for a summary graph by considering the structural and semantical attributes that can better elucidate every heterogeneous summary graph. Moreover, our framework comprises innovative components for comparing hybrid summary graphs. To the best of our knowledge, this is the first method that summarizes a graph stream using both the structure and vertex attributes with varying contributions. Our approach also takes user directions and ontology into account. Aiming to study the efficiency and effectiveness of our proposed method, we conduct extensive experiments on two real-life datasets: American political web-logs and Amazon co-purchasing products. The experimental results confirm that compared to the existing approaches the proposed method generates graph summaries with better quality. The expected time of our proposed method in this paper ( $$O(n^3)$$ ) has significantly enhanced the efficiency compared to the current best complexity which is $$O(n^5)$$ .