Distributed Stream Processing Research Articles

Online Nonstop Task Management for Storm-Based Distributed Stream Processing Engines

Online Nonstop Task Management for Storm-Based Distributed Stream Processing Engines

Adaptive key partitioning in distributed stream processing

In stream processing systems, Key Grouping is a commonly employed partitioning scheme for distributing input tuples among parallel instances of stateful operators. With key grouping, tuples shared public keys in the stream are designated to the specific instance responsible for that key. Typically, the implementation of key grouping involves the use of a hash function. While it is convenient and deterministic, it is also known to cause load imbalance between parallel instances, especially in the presence of skewed data streams. Key-Splitting is an effective technique that distributes tasks associated with keys to downstream operators, facilitating load balancing at a relatively low cost. However, overly increasing parallel instances can lead to excessive aggregation costs, becoming a system bottleneck. In this paper, we show the high aggregation cost brought by the Key-Splitting partitioner at different levels of key separation. To address this challenge, we introduce an adaptive Key-Splitting method which controlling the degree of key separation. We propose a partitioner named FlexD, which aims to achieve dynamic adaptation of key separation limits for streaming data. The partitioner employs key grouping to distribute rare keys and dynamic expansion of processing instances to distribute hot keys. We implemented our method on Apache Storm and evaluated it by using real-world and synthetic datasets. Experimental results show that our method achieves a good balance between load balancing and aggregation cost. Moreover, it outperforms existing methods, achieving higher throughput.

To Migrate or Not to Migrate: An Analysis of Operator Migration in Distributed Stream Processing

To Migrate or Not to Migrate: An Analysis of Operator Migration in Distributed Stream Processing

A generic node identification and routing algorithm in a distributed data acquisition platform: D-Matrix

Data acquisition (DAQ) systems are vital components in large-scale physics experiments. For communication, control and tracking purposes, all system components must be unambiguously identified. D-Matrix, as a generic distributed stream processing DAQ platform, accommodates various device connection methods, including networking, PCIe bus, point-to-point optical fiber links, etc. Although there are mature node identification solutions for each connection method individually, the uniform solution applicable to all connection methods remains crucial in a generic DAQ platform. D-Matrix abstracts a unified Multiple Point-to-Point transmission model (MPP model), supporting various physical connection methods and enabling multiple communication channels on a single physical link. Referring to the tree universal address system, a generic automatic node identification algorithm is proposed based on the MPP model. With simple configuration, this algorithm enables automatic node traversal, yielding a routing-based identification result that supports clustering and hierarchical node management requirements common in large-scale physics experiments. This paper explains the details of the algorithm and presents an example of the DAQ system based on the algorithm.

Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud

Context:The combination of distributed stream processing with microservice architectures is an emerging pattern for building data-intensive software systems. In such systems, stream processing frameworks such as Apache Flink, Apache Kafka Streams, Apache Samza, Hazelcast Jet, or the Apache Beam SDK are used inside microservices to continuously process massive amounts of data in a distributed fashion. While all of these frameworks promote scalability as a core feature, there is only little empirical research evaluating and comparing their scalability. Objective:The goal of this study to obtain evidence about the scalability of state-of-the-art stream processing framework in different execution environments and regarding different scalability dimensions. Method:We benchmark five modern stream processing frameworks regarding their scalability using a systematic method. We conduct over 740 h of experiments on Kubernetes clusters in the Google cloud and in a private cloud, where we deploy up to 110 simultaneously running microservice instances, which process up to one million messages per second. Results:All benchmarked frameworks exhibit approximately linear scalability as long as sufficient cloud resources are provisioned. However, the frameworks show considerable differences in the rate at which resources have to be added to cope with increasing load. There is no clear superior framework, but the ranking of the frameworks depends on the use case. Using Apache Beam as an abstraction layer still comes at the cost of significantly higher resource requirements regardless of the use case. We observe our results regardless of scaling load on a microservice, scaling the computational work performed inside the microservice, and the selected cloud environment. Moreover, vertical scaling can be a complementary measure to achieve scalability of stream processing frameworks. Conclusion:While scalable microservices can be designed with all evaluated frameworks, the choice of a framework and its deployment has a considerable impact on the cost of operating it.

Retracted: Probabilistic Hesitant Fuzzy Methods for Prioritizing Distributed Stream Processing Frameworks for IoT Applications

Retracted: Probabilistic Hesitant Fuzzy Methods for Prioritizing Distributed Stream Processing Frameworks for IoT Applications

Mathematical modeling for further improving task scheduling on Big Data systems

In the big data era which we have entered, the development of smart scheduler has become a necessity. A Distributed Stream Processing System (DSPS) has the role of assigning processing tasks to the available resources (dynamically or not) and route streaming data between them. Smart and efficient task scheduling can reduce latencies and eliminate network congestions. The most commonly used scheduler is the default Storm scheduler, which has proven to have certain disadvantages, like the inability to handle system changes in a dynamic environment. In such cases, rescheduling is necessary. This paper is an extension of a previous work on dynamic task scheduling. In such a scenario, some type of rescheduling is necessary to have the system working in the most efficient way. In this paper, we extend our previous works Souravlas and Anastasiadou (Appl Sci 10(14):4796, 2020); Souravlas et al. (Appl Sci 11(1):61, 2021) and present a mathematical model that offers better balance and produces fewer communication steps. The scheduler is based on the idea of generating larger sets of communication steps among the system nodes, which we call superclasses. Our experiments have shown that this scheme achieves better balancing and reduces the overall latency.

Bounding substreams in distributed stream processing

A common problem in distributed stream processing is to split a stream into finite chunks of messages (substreams) and to determine their boundaries: stateful streaming operators should clear outdated state; time window operators should release output after all elements within the specified time range have arrived. Most state-of-the-art SPEs use punctuations to divide a stream into bounded substreams of messages. The punctuation approach is powerful but has limitations: it does not support cyclic dataflows, is poorly scalable in some cases due to intensive use of broadcasts, and becomes inefficient when the number of chunks or cluster size becomes significant. We introduce a new substream tracking technique called trAcker that overcomes the limits of punctuations. We experimentally evaluate the properties of trAcker in both synthetic and real-world environments. Experiments show that our technique is scalable, outperforms punctuations for a large number of substreams, and efficiently handles real-world cyclic dataflows.

A scheduling algorithm to maximize storm throughput in heterogeneous cluster

In the most popular distributed stream processing frameworks (DSPFs), programs are modeled as a directed acyclic graph. Using this model, a DSPF can benefit from the parallelism capabilities of distributed clusters. Choosing a reasonable number of vertices for each operator and mapping the vertices to the appropriate processing resources significantly affect the overall system performance. Due to the simplicity of the current DSPF schedulers, these frameworks perform poorly on large-scale clusters. In this paper, we present a heterogeneity-aware scheduling algorithm that finds the proper number of the vertices of an application graph and maps them to the most suitable cluster node. We begin with a pre-processing step which allocates the vertices to the given cluster nodes using profiling data. Then, we gradually increase the topology input rate in order to scale up the application graph. Finally, using a CPU utilization model which predicts the CPU workload based on the input rate to vertices and the processing node’s CPU characteristics, we identify the bottlenecked vertices and allocate new instances derived from them to the least utilized processing resource. Our experimental results on Storm Micro-Benchmark show that (1) the prediction model estimate CPU utilization with 92% accuracy. (2) Compared to the default scheduler of Storm, our scheduler provides 7 to 44% throughput enhancement. (3) The proposed method can find the solution within 4% (worst case) of the optimal scheduler, which obtains the best scheduling scenario using an exhaustive search over problem design space.

Ghost: A General Framework for High-Performance Online Similarity Queries over Distributed Trajectory Streams

Trajectory similarity queries, including similarity search and similarity join, offer a foundation for many geo-spatial applications. With the rapid increase of streaming trajectory data volumes, e.g., data from mobile phones, vessel monitoring, or traffic systems, many location-based services benefit from online similarity analytics over trajectory data streams, where moving objects continually emit real-time position data. However, most existing studies focus on offline settings, and thus several major challenges remain unanswered in an online setting. To this end, we describe Ghost, a distributed stream processing framework that enables generic, efficient, and scalable online trajectory similarity search and join. We propose a novel incremental online similarity computation (IOSC) mechanism to accelerate pair-wise streaming trajectory distance calculation, which supports a broad range of trajectory distance metrics. Compared with previous studies, IOSC reduces the complexity from quadratic to linear in terms of trajectory length. Building on this foundation, we propose histogram-based algorithms that exploit histogram indexes and a series of pruning bounds to enable streaming trajectory similarity search and join. Finally, we extend our methods to the distributed platform Flink for scalability, where a CostPartitioner is developed to ensure parallel processing and workload balancing. An experimental study using two real-life and one synthetic datasets shows that Ghost (i) acquires 6-20× efficiency/throughput gains and one order of magnitude memory overhead savings over state-of-the-art baselines, (ii) achieves 3--8× workload balancing gains on Flink, and (iii) exhibits low parameter sensitivity and high robustness.

Stream-DBSCAN: A Streaming Distributed Clustering Model for Water Quality Monitoring

With the increasing use of wireless sensor networks in water quality monitoring, an enormous amount of streaming data is generated by widely deployed sensors. However, the current batch mode used for data analysis can no longer meet the diverse combination of monitoring indicators and the requirement for timely analysis results on an all-weather basis. To overcome these challenges and analyze a large amount of water quality data quickly and accurately, we propose a stream-DBSCAN distributed stream processing clustering model. First, real-time data streams are processed using the distributed stream computing framework Flink. Then, the DBSCAN clustering algorithm is applied to cluster each dataset as a different dimension of the cluster. Finally, the time distribution characteristics of the data in the same cluster are analyzed to identify the water quality variation rules. The system can extract data noise points and identify sudden deterioration of water quality. We tested the model using datasets on three water quality indices, pH, ammonia nitrogen (NH4N), and turbidity, in the Yantai Menlou Reservoir from May to August 2019. The results demonstrate that the system can efficiently and quickly perform cluster analysis on streaming data. By analyzing the clustering results, we found that the daily variation of water quality and sudden pollution events in the Menlou Reservoir are consistent with the actual situation.

Evaluating Task-Level CPU Efficiency for Distributed Stream Processing Systems

Big Data and primarily distributed stream processing systems (DSPSs) are growing in complexity and scale. As a result, effective performance management to ensure that these systems meet the required service level objectives (SLOs) is becoming increasingly difficult. A key factor to consider when evaluating the performance of a DSPS is CPU efficiency, which is the ratio of the workload processed by the system to the CPU resources invested. In this paper, we argue that developing new performance tools for creating DSPSs that can fulfill SLOs while using minimal resources is crucial. This is especially significant in edge computing situations where resources are limited and in large cloud deployments where conserving power and reducing computing expenses are essential. To address this challenge, we present a novel task-level approach for measuring CPU efficiency in DSPSs. Our approach supports various streaming frameworks, is adaptable, and comes with minimal overheads. This enables developers to understand the efficiency of different DSPSs at a granular level and provides insights that were not previously possible.

An empirical analysis of stateful operator migration for online scheduling in distributed stream processing systems

Streaming data is the data from sensors as well as other real-time surveillance systems. Distributed stream processing systems are the software that manages such data. Such frameworks have to deliver outcomes on the go instantly. They are susceptible to delay and malfunction or system failures. The system must be tolerant of faults and always accessible. Many variables, such as improved network arrival rates, node failures, and so on, disrupt the system's reliability. Some operators need to be relocated online from one physical resource to another to manage or reimburse a slow or failing node. In this study, we propose a co-location based systematic migration heuristic for live operator migration between physical resources using a migration map revised with costs for each migration. The suggested method evaluates continuous operator performance patterns and makes online scheduling decisions based on the same. The decisions include migrating operators during a node failure or straggling.

SDN-enabled Resource Provisioning Framework for Geo-Distributed Streaming Analytics

Geographically distributed (geo-distributed) datacenters for stream data processing typically comprise multiple edges and core datacenters connected through Wide-Area Network (WAN) with a master node responsible for allocating tasks to worker nodes. Since WAN links significantly impact the performance of distributed task execution, the existing task assignment approach is unsuitable for distributed stream data processing with low latency and high throughput demand. In this paper, we propose SAFA, a resource provisioning framework using the Software-Defined Networking (SDN) concept with an SDN controller responsible for monitoring the WAN, selecting an appropriate subset of worker nodes, and assigning tasks to the designated worker nodes. We implemented the data plane of the framework in P4 and the control plane components in Python. We tested the performance of the proposed system on Apache Spark, Apache Storm, and Apache Flink using the Yahoo! streaming benchmark on a set of custom topologies. The results of the experiments validate that the proposed approach is viable for distributed stream processing and confirm that it can improve at least 1.64× the processing time of incoming events of the current stream processing systems.

Geostatistics on Real-Time Geodata Streams—An Extended Spatiotemporal Moran’s I Index with Distributed Stream Processing Technologies

The availability of geodata with high spatial and temporal resolution is increasing steadily. Often, these data are continuously generated by distributed sensor networks and provided as geodata streams. Geostatistical analysis methods, such as spatiotemporal autocorrelation, have thus far been applied primarily to historized data. As such, the advantages of continuous and up-to-date acquisition of geodata have not yet been transferred to the analysis phase. At the same time, open-source frameworks for distributed stream processing have been developed into powerful real-time data processing tools. In this paper a methodology is developed to apply analyses of spatiotemporal autocorrelation directly to geodata streams through a distributed streaming process using open-source software frameworks. For this purpose, we adapt the extended Moran’s I index for continuous and up-to-date computation, then apply it to simulated geospatial data streams of recorded taxi trip data. Various application scenarios for the developed methodology are tested and compared on a distributed computing cluster. The results show that the developed methodology can provide geostatistical analysis results in real time. This research demonstrates how modern datastream processing technologies have the potential to significantly advance the way geostatistical analysis can be performed and used in the future.

Event Building Algorithm in a Distributed Stream Processing Data Acquisition Platform: D-Matrix

Event building is an important task for data acquisition (DAQ) systems in high-energy physics experiments. D-Matrix is designed as a generic distributed stream processing DAQ platform, containing hardware description language (HDL)-based modules and software modules. In the D-Matrix system, the data transfer is abstracted into the form of a stream, and the event building is implemented through a cascade of various reusable stream processing modules. These stream processing modules will determine their work modes automatically based on a set of attributes, which describe the abstracted data stream characteristics, such as spatiotemporal organization and granularity. This article introduces the event building architecture, the physics data structure abstraction, and the algorithmic implementation of specific event building modules in the D-Matrix system. An application of the D-Matrix event building architecture in the CSR external-target experiment is presented in the end.

T-PORP: A Trusted Parallel Route Planning Model on Dynamic Road Networks

Route planning over dynamic road networks is an increasingly fundamental problem of modern transportation systems for human society, especially in the field of Intelligent Supply Chain (ISC). Due to the high degree of urbanization and the high number of vehicles, longer response time caused by massive concurrent queries, as well as more attacks caused by malicious vehicles, results in low efficiency of the transportation system and huge waste of computation resources. Thus, it is necessary to provide an efficient and safe transportation service for intelligent transportation planning. To achieve it, we utilize and improve the trust model to prevent the waste of computation resources. Meanwhile, we introduce a Trusted Parallel Optimization on Route Planning (T-PORP) based on Dual-level Grid (DLG) index to continuously handle the process of route planning in parallel. Considering the evolving traffic condition, we employ an LSTM (Long Short-Term Memory) neural network to periodically predict the weights of roads. Experimental results indicate that T-PORP is effective to sorts of trust model attacks and reduces the response time by an average of about 46.7% and saves the processing time by an average of about 27.6% compared with CANDS (Continuous Optimal Navigation via Distributed Stream Processing) algorithm.

Network-aware worker placement for wide-area streaming analytics

Many organizations leverage Distributed Stream processing systems (DPSs) to get insights from the data generated by different users/devices, e.g., the Internet of Things (IoT) devices or user clicks on a website, on geographically distributed datacenters. The worker nodes in such environments are connected through Wide Area Network (WAN) links with various delays and bandwidth. Therefore, minimizing the execution latency of a task on the worker nodes while using the links with enough bandwidth and lower cost to steer the traffic of the applications is a challenging task. In this paper, we formulate the worker node placement for a geo-distributed DSPs network as a multi-criteria decision-making problem. Then, we propose an additive weighting-based approach to solve it. The users can prioritize the worker node placement according to the network-relevant parameters. We also propose a framework that can be integrated with the current DPSs to execute the tasks. We test our placement approach on three widely used stream processing systems, i.e., Apache Spark, Apache Storm, and Apache Flink, on three custom graphs adopted from the real cloud providers. We run the streaming query of the Yahoo! streaming benchmark on these three DPSs. The experimental results show that our approach improves the performance of Spark up to 2.2x–7.2x, Storm up to 1.2x–3.4x, and Flink up to 1.4x–3.3x compared with other placement approaches, which makes our framework useful for use in practical environments.

Algorithms for Windowed Aggregations and Joins on Distributed Stream Processing Systems

Window aggregations and windowed joins are central operators of modern real-time analytic workloads and significantly impact the performance of stream processing systems.This paper gives an overview of state-of-the-art research in this area conducted by the Berlin Institute for the Foundations of Learning and Data (BIFOLD) and the Technische Universität Berlin. To this end, we present different algorithms for efficiently processing windowed operators and discuss techniques for distributed stream processing. Recently, several approaches have leveraged modern hardware for windowed stream processing, which we will also include in this overview. Additionally, we describe the integration of windowed operators into various stream processing systems and diverse applications that use specialized window operations.

Targeting a light-weight and multi-channel approach for distributed stream processing

Processing high-throughput data-streams has become a major challenge in areas such as real-time event monitoring, complex dataflow processing, and big data analytics. While there has been tremendous progress in distributed stream processing systems in the past few years, the high-throughput and low-latency (a.k.a. high sustainable-throughput) requirement of modern applications is pushing the limits of traditional data processing infrastructures. This paper introduces a new distributed stream processing engine (DSPE), called Asynchronous Iterative Routing (or simply “AIR”), which implements a light-weight, dynamic sharding protocol. AIR expedites direct and asynchronous communication among all the worker nodes via a channel-like communication protocol on top of the Message Passing Interface (MPI), thereby completely avoiding the need for a dedicated driver node. The system adopts a new progress-tracking protocol, called hew-meld, which has been experimentally observed to show a low processing latency on our asynchronous master-less architecture when compared to the conventional low-watermark technique. The current version of AIR is also equipped with two fault tolerance and recovery strategies namely checkpointing & rollback and replication. With its unique design, AIR scales out particularly well to multi-core HPC architectures; specifically, we deployed it on clusters with up to 16 nodes and 448 cores (thus reaching a peak of 435.3 million events and 55.14 GB of data processed per second), which we found to significantly outperform existing DSPEs.