Stream Processing Engines Research Articles

A holistic view over ontologies for Streaming Linked Data

Streaming Linked Data represents a domain within the Semantic Web dedicated to incorporating Stream Reasoning capabilities into the Semantic Web stack to address dynamic data challenges. Such applied endeavours typically necessitate a robust data modelling process. To this end, RDF Stream Processing (RSP) engines frequently utilize OWL 2 ontologies to facilitate this requirement. Despite the rich body of research on Knowledge Representation (KR), even concerning time-sensitive data, a notable gap exists in the literature regarding a comprehensive survey on KR techniques tailored for Streaming Linked Data. This paper critically overviews the key ontologies employed in RSP applications, evaluating their data modelling and KR abilities specifically for Streaming Linked Data contexts. We analyze these ontologies through three distinct KR perspectives: the conceptualization of streams as Web resources, the structural organization of data streams, and the event modelling within the streams. An analytical framework is introduced for each perspective to ensure a thorough and equitable comparison and deepen the understanding of the surveyed ontologies.

Multi-Resolution Odd Sketch for Mining Extended Jaccard Similarity of Dynamic Streaming sets

Estimating the Jaccard similarity between two or more streaming sets is a fundamental problem with many applications, such as mining the co-occurrences of host communication behaviors in IP networks and the co-purchasing behaviors of online users in E-commence systems. For a “streaming” data set, its elements arrive sequentially at the stream processing engine, whose working memory is limited in size. To meet the resource constraint, the data set must be stored as a compressed format called a “sketch”, so that millions or even billions of such sets can be held in the memory of the engine. To strike a balance between memory cost and similarity estimation accuracy, many sketching algorithms were proposed, such as MinHash, HyperLogLog and theta sketch. As far as we know, most of previous solutions fail to handle the fully dynamic streaming sets that allow both the insertion and the deletion of elements. Although PCSA <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\pm$</tex-math></inline-formula> , and virtual odd sketch (VOS) partially solves the deletion problem, their memory efficiency and estimation accuracy can be fundamentally improved. In this paper, we propose a multi-resolution odd sketch (MROS), which allows more accurate similarity estimation with less memory consumption. Its design is to encode a streaming set into multiple layers of odd sketches with exponentially reducing sampling probabilities. No matter a set is small or large, we can pick a suitable sampling probability to accurately estimate its cardinality. Next, we present an algorithm to estimate the extended Jaccard similarity of multiple streaming sets. Their compressed MROS summaries are merged by bitwise XOR, which in turn helps estimate the cardinality of symmetric difference of the multiple sets. Then, using these estimated cardinalities as observations, we estimate the size of an arbitrary set expression, which is connected by union <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\bigcup$</tex-math></inline-formula> , intersection <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\bigcap$</tex-math></inline-formula> , relative complement <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\setminus$</tex-math></inline-formula> , and symmetric difference <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\Delta$</tex-math></inline-formula> . Our evaluation results show that its accuracy outperforms MinHash, KMV, PCSA <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\pm$</tex-math></inline-formula> and VOS, and it can support element deletion.

Online Nonstop Task Management for Storm-Based Distributed Stream Processing Engines

Online Nonstop Task Management for Storm-Based Distributed Stream Processing Engines

SPinDP: A High-Speed Distributed Processing Platform for Sampling and Filtering Data Streams

Recently, there has been an explosive generation of streaming data in various fields such as IoT and network attack detection, medical data monitoring, and financial trend analysis. These domains require precise and rapid analysis capabilities by minimizing noise from continuously generated raw data. In this paper, we propose SPinDP (Stream Purifier in Distributed Platform), an open source-based high-speed stream purification platform, to support real-time stream purification. SPinDP consists of four major components, Data Stream Processing Engine, Purification Library, Plan Manager, and Shared Storage, and operates based on open-source systems including Apache Storm and Apache Kafka. In these components, stream processing throughput and latency are critical performance metrics, and SPinDP significantly enhances distributed processing performance by utilizing the ultra-high-speed network RDMA (Remote Direct Memory Access). For the performance evaluation, we use a distributed cluster environment consisting of nine nodes, and we show that SPinDP’s stream processing throughput is more than 28 times higher than that of the existing Ethernet environment. SPinDP also significantly reduces the processing latency by more than 2473 times on average. These results indicate that the proposed SPinDP is an excellent integrated platform that can efficiently purify high-speed and large-scale streams through RDMA-based distributed processing.

Neural adaptive IoT streaming analytics with RL-Adapt

IoT stream processing is an emerging technology that plays a crucial role in enabling time-critical IoT applications, which often demand high accuracy and low latency. For example, Augmented Reality (AR) applications require high object detection precision and object localization precision, as well as low latency. Note that the exact definition of accuracy varies depending on the specific application Existing stream processing engines are insufficient to meet these requirements, since they could not integrate and respond timely to variable network conditions in the dynamic wireless environment. Recent efforts focusing on adaptive streaming support user-specified policies to adapt to the variable network conditions. However, existing works suffer from manual policies or simplified models of the deployment environment which limit their ability to achieve optimal performance across a broad set of network conditions and quality of experience (QoE) objectives. In this paper, we present a Reinforcement Learning-based Adaptive streaming system (RL-Adapt) that is capable of generating adaption policies using RL-strategy and providing declarative APIs for efficient development. RL-Adapt trains a neural network model that can automatically select the optimal policy based on the observed network conditions. RL-Adapt does not rely on pre-defined models or assumptions about the environment. Instead, it learns to make decisions solely through observations of the resulting performance of past decisions. We implemented RL-Adapt and evaluated its performance extensively in three representative real-world IoT applications. Our results show that RL-Adapt outperforms the state-of-the-art schemes in terms of QoE metrics.

MorphStream: Adaptive Scheduling for Scalable Transactional Stream Processing on Multicores

Transactional stream processing engines (TSPEs) differ significantly in their designs, but all rely on non- adaptive scheduling strategies for processing concurrent state transactions. Subsequently, none exploit multicore parallelism to its full potential due to complex workload dependencies. This paper introduces MorphStream, which adopts a novel approach by decomposing scheduling strategies into three dimensions and then strives to make the right decision along each dimension, based on analyzing the decision trade-offs under varying workload characteristics. Compared to the state-of-the-art, MorphStream achieves up to 3.4 times higher throughput and 69.1% lower processing latency for handling real-world use cases with complex and dynamically changing workload dependencies.

STRETCH: Virtual Shared-Nothing Parallelism for Scalable and Elastic Stream Processing

Stream processing applications extract value from raw data through Directed Acyclic Graphs of data analysis tasks. Shared-nothing (SN) parallelism is the de-facto standard to scale stream processing applications. Given an application, SN parallelism ins9tantiates several copies of each analysis task, making each instance responsible for a dedicated portion of the overall analysis, and relies on dedicated queues to exchange data among connected instances. On the one hand, SN parallelism can scale the execution of applications both up and out since threads can run task instances within and across processes/nodes. On the other hand, its lack of sharing can cause unnecessary overheads and hinder the scaling up when threads operate on data that could be jointly accessed in shared memory. This trade-off motivated us in studying a way for stream processing applications to leverage shared memory and boost the scale up (before the scale out) while adhering to the widely-adopted and SN-based APIs for stream processing applications. We introduce <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">STRETCH</i> , a framework that maximizes the scale up and offers instantaneous elastic reconfigurations (without state transfer) for stream processing applications. We propose the concept of Virtual Shared-Nothing (VSN) parallelism and elasticity and provide formal definitions and correctness proofs for the semantics of the analysis tasks supported by <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">STRETCH</i> , showing they extend the ones found in common Stream Processing Engines. We also provide a fully implemented prototype and show that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">STRETCH</i> 's performance exceeds that of state-of-the-art frameworks such as Apache Flink and offers, to the best of our knowledge, unprecedented ultra-fast reconfigurations, taking less than 40 ms even when provisioning tens of new task instances.

MF-Storm: a maximum flow-based job scheduler for stream processing engines on computational clusters to increase throughput.

BackgroundA scheduling algorithm tries to schedule multiple computational tasks on a cluster of multiple computing nodes to maximize throughput with optimal utilization of computational and communicational resources. A Stream Processing Engine (SPE) is deployed to run streaming applications (computational tasks) on a computational cluster which helps execution and coordination of these applications. It is observed that there is a gap in the optimal mapping of a computational and communicational load of a streaming application on the underlying computational and communication power of the resources (cluster). Frequently communicated tasks are scheduled at different processing nodes with relatively slow communicating links. This increases network latency with a decrease in resource utilization. Hence, reduces the achieved throughput of the cluster significantly.MethodsMF-Storm, a max-flow min-cut based job scheduler is presented to achieve a near-optimum schedule to maximize throughput. It schedules a streaming application by considering the processing, communication demands, available computational and communicational resources in a heterogeneous cluster, dynamically with minimized scheduling cost. To keep the scheduling cost minimum, the scheduler is built in a pipeline with two major stages: in the first stage, the application’s tasks graph is partitioned using the max-flow min-cut algorithm to minimize inter-partition traffic, and in the second stage, these partitions are assigned to computing nodes according to the computational power of the cluster’s nodes.ResultsExtensive experiments were done to evaluate the performance of MF-Storm using different topologies with multiple scenarios on a physical cluster implementation. Results showed on average 148% improvement in throughput with 30% less computational resources as compared to different state-of-the-art schedulers.

Targeting a light-weight and multi-channel approach for distributed stream processing

Processing high-throughput data-streams has become a major challenge in areas such as real-time event monitoring, complex dataflow processing, and big data analytics. While there has been tremendous progress in distributed stream processing systems in the past few years, the high-throughput and low-latency (a.k.a. high sustainable-throughput) requirement of modern applications is pushing the limits of traditional data processing infrastructures. This paper introduces a new distributed stream processing engine (DSPE), called Asynchronous Iterative Routing (or simply “AIR”), which implements a light-weight, dynamic sharding protocol. AIR expedites direct and asynchronous communication among all the worker nodes via a channel-like communication protocol on top of the Message Passing Interface (MPI), thereby completely avoiding the need for a dedicated driver node. The system adopts a new progress-tracking protocol, called hew-meld, which has been experimentally observed to show a low processing latency on our asynchronous master-less architecture when compared to the conventional low-watermark technique. The current version of AIR is also equipped with two fault tolerance and recovery strategies namely checkpointing & rollback and replication. With its unique design, AIR scales out particularly well to multi-core HPC architectures; specifically, we deployed it on clusters with up to 16 nodes and 448 cores (thus reaching a peak of 435.3 million events and 55.14 GB of data processed per second), which we found to significantly outperform existing DSPEs.

Real Time Traffic Prediction Based On Social Media Text Data Using Deep Learning

Due to urbanization Traffic management is one of the major issues in contemporary civic management, considering this circumstance traffic analysis is turning into the need of the present world. Text data generated by Twitter, Facebook and other social media platforms can be used for traffic management. Big data helps in traffic prediction and traffic analysis of advancing metropolitan zones. Constant traffic investigation requires preparing of information streams that are produced persistently to increase fast experiences. To measures stream information at a fast rate advancements on high figuring limit is required. Social media text data can be processed by using batch processing and stream processing with big data architecture through Spark and Hadoop framework. In this paper big data architecture is proposed for real time traffic text data analysis. In architecture Spark and Kafka are used in combination. Kafka helps in pipelines text data used in conjunction with spark stream processing engine. Big data architecture using Spark, Kafka with ability for processing and preparing huge measure of information, have settled the serious issue of handling and putting away constantly streaming data. The traffic information from Twitter API is streamed. In The proposed model pointed toward ensemble neural network model to reduce the variance in results for better prediction foreseeing traffic stream text data by incorporating Spark and Kafka that will be of an extraordinary incentive to the public authority for traffic management and analysis.

Scabbard

Single-node multi-core stream processing engines (SPEs) can process hundreds of millions of tuples per second. Yet making them fault-tolerant with exactly-once semantics while retaining this performance is an open challenge: due to the limited I/O bandwidth of a single-node, it becomes infeasible to persist all stream data and operator state during execution. Instead, single-node SPEs rely on upstream distributed systems, such as Apache Kafka, to recover stream data after failure, necessitating complex cluster-based deployments. This lack of built-in fault-tolerance features has hindered the adoption of single-node SPEs. We describe Scabbard, the first single-node SPE that supports exactly-once fault-tolerance semantics despite limited local I/O bandwidth. Scabbard achieves this by integrating persistence operations with the query workload. Within the operator graph, Scabbard determines when to persist streams based on the selectivity of operators: by persisting streams after operators that discard data, it can substantially reduce the required I/O bandwidth. As part of the operator graph, Scabbard supports parallel persistence operations and uses markers to decide when to discard persisted data. The persisted data volume is further reduced using workload-specific compression: Scabbard monitors stream statistics and dynamically generates computationally efficient compression operators. Our experiments show that Scabbard can execute stream queries that process over 200 million tuples per second while recovering from failures with sub-second latencies.

Fine-Grained Multi-Query Stream Processing on Integrated Architectures

Exploring the sharing opportunities among multiple stream queries is crucial for high-performance stream processing. Modern stream processing necessitates accelerating multiple queries by utilizing heterogeneous coprocessors, such as GPUs, and this has shown to be an effective method. Emerging CPU-GPU integrated architectures 6integrate CPU and GPU on the same chip and eliminate PCI-e bandwidth bottleneck. Such a novel architecture provides new opportunities for improving multi-query performance in stream processing but has not been fully explored by existing systems. We introduce a stream processing engine, called FineStream, for efficient multi-query window-based stream processing on CPU-GPU integrated architectures. FineStream's key contribution is a novel fine-grained workload scheduling mechanism between CPU and GPU to take advantage of both architectures. Particularly, FineStream is able to efficiently handle multiple queries in both static and dynamic streams. Our experimental results show that 1) on integrated architectures, FineStream achieves an average 52 percent throughput improvement and 36 percent lower latency over the state-of-the-art stream processing engine; 2) compared to the coarse-grained strategy of applying different devices for multiple queries, FineStream achieves 32 percent throughput improvement; 3) compared to the stream processing engine on the discrete architecture, FineStream on the integrated architecture achieves 10.4× price-throughput ratio, 1.8× energy efficiency, and can enjoy lower latency benefits.

Towards autoscaling of Apache Flink jobs

Abstract Data stream processing has been gaining attention in the past decade. Apache Flink is an open-source distributed stream processing engine that is able to process a large amount of data in real time with low latency. Computations are distributed among a cluster of nodes. Currently, provisioning the appropriate amount of cloud resources must be done manually ahead of time. A dynamically varying workload may exceed the capacity of the cluster, or leave resources underutilized. In our paper, we describe an architecture that enables the automatic scaling of Flink jobs on Kubernetes based on custom metrics, and describe a simple scaling policy. We also measure the e ects of state size and target parallelism on the duration of the scaling operation, which must be considered when designing an autoscaling policy, so that the Flink job respects a Service Level Agreement.

Stochastic distributed data stream partitioning using task locality: design, implementation, and optimization

Distributed stream processing engines (DSPEs) provide stream partitioning methods for distributing messages to tasks deployed in the distributed environment for real-time stream processing. Among these methods, the original locality-aware stream partitioning (LSP) is a binary LSP that sends messages only to downstreams on the same node as upstreams. The binary LSP degrades performance at general configurations because it focuses only on task locality and does not consider downstream status like distributed batch processing engines. In this paper, we propose a Stochastic LSP (SLSP) method that considers not only task locality but also downstream status by computing stream partitioning probability based on the round-trip time to downstreams. We also present coarse-grained and fine-grained methods for probing downstreams at node-level and process-level, respectively. Then, we optimize our SLSP using a weighted closeness to prioritize the partitioning probabilities and a parallel thread model to process each stage of the SLSP in parallel. Finally, we implement the SLSP in Apache Storm, a representative DSPE, and empirically evaluate it with the binary LSP. Experimental results show that our SLSP greatly reduces latency by up to 208% while maintaining a similar throughput compared to the binary LSP at general configurations. These results indicate that our SLSP performs the optimized stream partitioning by reflecting downstream status as well as task locality.

Heterogeneity-aware elastic scaling of streaming applications on cloud platforms

Rise of Big Data techniques has led to the requirement for low latency analysis of high-velocity continuous data streams in real time. Several solutions, including Stream Processing Systems (SPSs), have been developed to enable real-time distributed stream processing. However, emerging application scenarios such as smart cities and wearable assistance that involve highly variable data rates keep on posing new challenges to the established stream processing engines for maintaining cost-effective executions. To cater to such scenarios, many modern SPSs have been proposed that leverage Cloud environment. The run-time scalability incorporated in these SPSs is in their early adaptations and are based on fixed scale sizes. Moreover, these scaling approaches do not adequately consider both the structure of the hosted streaming applications and the characteristic features of the underlying Cloud environment. Achieving true cost benefits of orchestrating streaming applications on Cloud-based pay-as-you-go model while maintaining the desired QoS, necessitates that both these issues are accounted in making the scaling decisions. This work presents a heterogeneity-aware, efficient auto-scaling strategy StreamScale-H which addresses both these issues. Simulation experiments, on representative stream applications, indicate that the proposed StreamScale-H auto-scaling algorithm exhibits much better performance in comparison with the state-of-the-art algorithms.

Metadata-based measurements transmission verified by a Merkle Tree

The Data Stream Processing Strategy (DSPS) is focused on the automatization of measurement projects based on a measurement framework. The measurement adapter (MA) is an architecture component located on mobile devices aims to integrate heterogeneous data sources (i.e., sensors). The Gathering Function (GF) is the component responsible for interacting and receiving measures from the MAs, and it resides on the Stream Processing Engine (SPE). MA and GF share the project definition based on a measurement framework to foster data interoperability, while MA regulates the frequency, size, and route related to data transmission. As contributions (i) The brief data message is introduced to optimize the data transmission keeping immutable the hierarchical data organization based on the project definition, and (ii) The integrity record for mobile and SPE environments is described based on a Merkle Tree. This allows optimizing each data transaction, incorporating a historical integrity record both MA and SPE. The proposals and simulations have been implemented on the cincamimis, cincamipd, mair, and pabmmcommons libraries, which are freely available on GitHub under the terms of the Apache 2.0 licence. Four simulations are explained to detail how to measures were obtained. Interesting results show that the brief data message consumes 17.50 KB to transmit 1000 measures (2.4 times smaller than JSON), while a message with 200 measures could be generated and compressed using GZIP in 25.12 ms (2.43 times faster than JSON). 196 KB is required to keep 17 min of the integrity history in a MA, being created in 4.85 ms.

Theodolite: Scalability Benchmarking of Distributed Stream Processing Engines in Microservice Architectures

Distributed stream processing engines are designed with a focus on scalability to process big data volumes in a continuous manner. We present the Theodolite method for benchmarking the scalability of distributed stream processing engines. Core of this method is the definition of use cases that microservices implementing stream processing have to fulfill. For each use case, our method identifies relevant workload dimensions that might affect the scalability of a use case. We propose to design one benchmark per use case and relevant workload dimension. We present a general benchmarking framework, which can be applied to execute the individual benchmarks for a given use case and workload dimension. Our framework executes an implementation of the use case's dataflow architecture for different workloads of the given dimension and various numbers of processing instances. This way, it identifies how resources demand evolves with increasing workloads. Within the scope of this paper, we present 4 identified use cases, derived from processing Industrial Internet of Things data, and 7 corresponding workload dimensions. We provide implementations of 4 benchmarks with Kafka Streams and Apache Flink as well as an implementation of our benchmarking framework to execute scalability benchmarks in cloud environments. We use both for evaluating the Theodolite method and for benchmarking Kafka Streams' and Flink's scalability for different deployment options.

A Flexible IoT Stream Processing Architecture Based on Microservices

The Internet of Things (IoT) has created new and challenging opportunities for data analytics. The IoT represents an infinitive source of massive and heterogeneous data, whose real-time processing is an increasingly important issue. IoT applications usually consist of multiple technological layers connecting ‘things’ to a remote cloud core. These layers are generally grouped into two macro levels: the edge level (consisting of the devices at the boundary of the network near the devices that produce the data) and the core level (consisting of the remote cloud components of the application). The aim of this work is to propose an adaptive microservices architecture for IoT platforms which provides real-time stream processing functionalities that can seamlessly both at the edge-level and cloud-level. More in detail, we introduce the notion of μ-service, a stream processing unit that can be indifferently allocated on the edge and core level, and a Reference Architecture that provides all necessary services (namely Proxy, Adapter and Data Processing μ-services) for dealing with real-time stream processing in a very flexible way. Furthermore, in order to abstract away from the underlying stream processing engine and IoT layers (edge/cloud), we propose: (1) a service definition language consisting of a configuration language based on JSON objects (interoperability), (2) a rule-based query language with basic filter operations that can be compiled to most of the existing stream processing engines (portability), and (3) a combinator language to build pipelines of filter definitions (compositionality). Although our proposal has been designed to extend the Senseioty platform, a proprietary IoT platform developed by FlairBit, it could be adapted to every platform based on similar technologies. As a proof of concept, we provide details of a preliminary prototype based on the Java OSGi framework.

D2IA: User-defined interval analytics on distributed streams

Nowadays, modern Big Stream Processing Solutions (e.g. Spark, Flink) are working towards being the ultimate framework for streaming analytics. In order to achieve this goal, they started to offer extensions of SQL that incorporate stream-oriented primitives such as windowing and Complex Event Processing (CEP). The former enables stateful computation on infinite sequences of data items while the latter focuses on the detection of events pattern. In most of the cases, data items and events are considered instantaneous, i.e., they are single time points in a discrete temporal domain. Nevertheless, a point-based time semantics does not satisfy the requirements of a number of use-cases. For instance, it is not possible to detect the interval during which the temperature increases until the temperature begins to decrease, nor for all the relations this interval subsumes. To tackle this challenge, we present D2IA; a set of novel abstract operators to define analytics on user-defined event intervals based on raw events and to efficiently reason about temporal relationships between intervals and/or point events. We realize the implementation of the concepts of D2IA on top of Flink, a distributed stream processing engine for big data.

Locality/Fairness-Aware Job Scheduling in Distributed Stream Processing Engines

Distributed stream processing engines (DSPEs) deploy multiple tasks on distributed servers to process data streams in real time. Many DSPEs have provided locality-aware stream partitioning (LSP) methods to reduce network communication costs. However, an even job scheduler provided by DSPEs deploys tasks far away from each other on the distributed servers, which cannot use the LSP properly. In this paper, we propose a Locality/Fairness-aware job scheduler (L/F job scheduler) that considers locality together to solve problems of the even job scheduler that only considers fairness. First, the L/F job scheduler increases cohesion of contiguous tasks that require message transmissions for the locality. At the same time, it reduces coupling of parallel tasks that do not require message transmissions for the fairness. Next, we connect the contiguous tasks into a stream pipeline and evenly deploy stream pipelines to the distributed servers so that the L/F job scheduler achieves high cohesion and low coupling. Finally, we implement the proposed L/F job scheduler in Apache Storm, a representative DSPE, and evaluate it in both synthetic and real-world workloads. Experimental results show that the L/F job scheduler is similar in throughput compared to the even job scheduler, but latency is significantly improved by up to 139.2% for the LSP applications and by up to 140.7% even for the non-LSP applications. The L/F job scheduler also improves latency by 19.58% and 12.13%, respectively, in two real-world workloads. These results indicate that our L/F job scheduler provides superior processing performance for the DSPE applications.