Stream Processing Engines Research Articles

A Metadata and Z Score-based Load-Shedding Technique in IoT-based Data Collection Systems

The Internet-of-Things (IoT) has emerged as an alternative to communicate different pieces of technology to foster the distributed data collection. The measurement projects and the Real-time data processing are articulated to take advantage of this environment, fostering a sustainable data-driven decision making. The Data Stream Processing Strategy (DSPS) is a Stream Processing Engine focused on measurement projects, where each concept is previously agreed through a measurement framework. The Measurement Adapter (MA) is a component whose responsibility is to pair each metric’s definition from the measurement project with data sensors to transmit data (i.e., measures) with metadata (i.e., tags indicating the data meaning) together. The Gathering Function (GF) receives and derivates data for its processing from each MA, while it implements load-shedding (LS) techniques based on Metadata to avoid a processing collapse when all MAs informs jointly and frequently. Here, a Metadata and Z-score based load-shedding technique implemented locally in the MA is proposed. Thus, the load-shedding is located at the same data source to avoid data transmission and saving resources. Also, an incremental estimation of average, deviations, covariance, and correlations are implemented and employed to calculate the Z-scores and to data retain/discard selectively. Four simulations discrete were designed and performed to analyze the proposal. Results indicate that the local LS required only 24% of the original data transmissions, a minimum of 18.61 ms as the data lifespan, while it consumes 890.26 KB. As future work, other kinds of dependencies analysis will be analyzed to provide local alternatives to LS.

IPC: Resource and network cost-aware distributed stream scheduling on skewed streams

The performance of distributed stream processing engines is significantly compromised when processing stream data with skewed distribution. Current stream partitioning schemes are not able to meet the rigorous requirements of distributed stream processing. We show that network cost is an essential factor for partitioning data, and this factor should be considered when designing a stream partitioning scheme. Additionally, we should efficiently utilize resources in the data partitioning process. Current stream partitioning schemes either use a shuffle grouping approach that efficiently manages workload but faces scalability issues in terms of memory or uses hash-based key grouping schemes that suffer from load balancing issues.We argue that network cost and resource utilization are two crucial factors for stream partitioning schemes. We propose and implement a distributed stream partitioning scheme call IPC that minimizes the network cost and efficiently utilizes resources by leveraging two techniques: process near source and process at local. It also utilizes key splitting and local load estimation techniques to achieve load balancing. We implement the IPC on top of Apache Storm. Experiment results using large scale real-time datasets show that IPC achieves an up to 4.2x improvement in throughput and reduces processing latency by 97% compared to state-of-the-art designs.

Meet me halfway

From Industry 4.0-driven factories to real-time trading algorithms, businesses depend on analytics on high-velocity real-time data. Often these analytics are performed not in dedicated stream processing engines but on views within a general-purpose database to combine current with historical data. However, traditional view maintenance algorithms are not designed with both the volume and velocity of data streams in mind. In this paper, we propose a new type of view specialized for queries involving high-velocity inputs, called continuous view. The key component of continuous views is a novel maintenance strategy, splitting the work between inserts and queries. By performing initial parts of the view's query for each insert and the remainder at query time, we achieve both high input rates and low query latency. Further, we keep the memory overhead of our views small, independent of input velocity. To demonstrate the practicality of this strategy, we integrate continuous views into our Umbra database system. We show that split maintenance can outperform even dedicated stream processing engines on analytical workloads, all while still offering similar insert rates. Compared to modern materialized view maintenance approaches, such as deferred and incremental view maintenance, that often need to materialize expensive deltas, we achieve up to an order of magnitude higher insert throughput.

A Distributed Stream Processing Middleware Framework for Real-Time Analysis of Heterogeneous Data on Big Data Platform: Case of Environmental Monitoring.

In recent years, the application and wide adoption of Internet of Things (IoT)-based technologies have increased the proliferation of monitoring systems, which has consequently exponentially increased the amounts of heterogeneous data generated. Processing and analysing the massive amount of data produced is cumbersome and gradually moving from classical ‘batch’ processing—extract, transform, load (ETL) technique to real-time processing. For instance, in environmental monitoring and management domain, time-series data and historical dataset are crucial for prediction models. However, the environmental monitoring domain still utilises legacy systems, which complicates the real-time analysis of the essential data, integration with big data platforms and reliance on batch processing. Herein, as a solution, a distributed stream processing middleware framework for real-time analysis of heterogeneous environmental monitoring and management data is presented and tested on a cluster using open source technologies in a big data environment. The system ingests datasets from legacy systems and sensor data from heterogeneous automated weather systems irrespective of the data types to Apache Kafka topics using Kafka Connect APIs for processing by the Kafka streaming processing engine. The stream processing engine executes the predictive numerical models and algorithms represented in event processing (EP) languages for real-time analysis of the data streams. To prove the feasibility of the proposed framework, we implemented the system using a case study scenario of drought prediction and forecasting based on the Effective Drought Index (EDI) model. Firstly, we transform the predictive model into a form that could be executed by the streaming engine for real-time computing. Secondly, the model is applied to the ingested data streams and datasets to predict drought through persistent querying of the infinite streams to detect anomalies. As a conclusion of this study, a performance evaluation of the distributed stream processing middleware infrastructure is calculated to determine the real-time effectiveness of the framework.

A3-Storm: topology-, traffic-, and resource-aware storm scheduler for heterogeneous clusters

Like other emerging fields, Stream Processing Engines (SPEs) pose several challenges to the researchers such as resource awareness, dynamic configurations, heterogeneous clusters, and load balancing. All of these aspects play a major role in the job scheduling process. Inefficiency in any of them causes problems for achieving the maximum throughput. SPEs must contemplate other aspects like resource provisioning, job’s computation requirement, physical distance between communicating nodes, etc. Currently, SPEs ignore topology’s structure as well as inter-executor traffic while scheduling. Due to this, frequently communicating tasks may end up at different computing nodes which increases network latency. In this paper, A3-Storm, a scheduler, based on topology and traffic is proposed that optimizes resource usage for heterogeneous clusters. The aim is to improve efficiency using resource-aware task assignments that results in enhanced throughput and resource utilization. A3-Storm schedules topology using inter-executor traffic and supervisor node’s computing power. A3-Storm is divided into two phases: in the first phase, executors are logically grouped to minimize inter-group communication traffic according to the topology structure or inter-executor traffic. In the second phase, these groups are assigned to physical nodes starting from the most powerful node. Apache Storm (a popular open-source SPE) is used for the implementation of A3-Storm. Results are generated with the help of 2 benchmark topologies, and results are compared with 3 state-of-the-art algorithms. Extensive experiment results show up to 25% and 12% improvement in throughput as compared to the default Storm scheduler and resource-aware scheduler, respectively, with a significant amount of resource savings through consolidation.

TOP-Storm: A topology-based resource-aware scheduler for Stream Processing Engine

TOP-Storm: A topology-based resource-aware scheduler for Stream Processing Engine

A novel temporal and spatial panorama stream processing engine on IoT applications

Abstract Nowadays, more and more streaming data are generated with the development of Internet of Things. Although, streaming data show great application values in practical scenarios, raw streams from terminal sensors are quite massive, heterogeneous and complex, and those features make it difficult for applications to deal with them. In order to simplify streaming data processing for applications, a novel temporal and spatial panorama stream processing engine is proposed. This stream processing engine forms an effective link between bottom sensors and upper stream-based applications, and provides configurable, flexible, available and usable stream services for various upper applications. With support of Internet plus and domain data, raw streaming data are formatted thorough data fusion and are represented in temporal, spatial, logic and storage views. Fusion data are encapsulated with multiple strategies and methods of clustering models on every dimension. According to configurable strategies, encapsulated data are extracted, partitioned and distributed as the form of dynamical variable gratitude data blocks into stream channels. Our engine is applied to a practical application scenario. Case study of dynamic power distribution application based on people crowd streaming data proves the customized service capabilities of our engine. And the outstanding performance of the engine is shown further by experiments evaluation in this case.

BackStreamDB: A stream processing engine for backbone traffic monitoring with anomaly detection

AbstractBackStreamDB is distributed traffic monitoring system based on a stream processing engine (SPE) designed to monitor the traffic of wide area backbones. BackStreamDB provides arbitrary metrics about the traffic in real time, taking into account the backbone as a whole. The system was developed for and successfully deployed on the Brazilian National Academic Network (RNP). In this work, we describe the functionality for the detection of traffic anomalies. A large number of Internet attacks are continuously reported, and several types of attacks result in anomalous traffic. In the proposed strategy for anomaly detection, the traffic is sampled by monitors that are distributed across the backbone, which are accessed and processed by the SPE. BackStreamDB was extended with stream processing modules for computing traffic entropy and principal component analysis, which are the employed to detect traffic anomalies. Experimental results are reported which were obtained to validate the effectiveness of the proposed strategy for different types of attacks.

Real-Time Compliant Stream Processing Agents for Physical Rehabilitation.

Digital rehabilitation is a novel concept that integrates state-of-the-art technologies for motion sensing and monitoring, with personalized patient-centric methodologies emerging from the field of physiotherapy. Thanks to the advances in wearable and portable sensing technologies, it is possible to provide patients with accurate monitoring devices, which simplifies the tracking of performance and effectiveness of physical exercises and treatments. Employing these approaches in everyday practice has enormous potential. Besides facilitating and improving the quality of care provided by physiotherapists, the usage of these technologies also promotes the personalization of treatments, thanks to data analytics and patient profiling (e.g., performance and behavior). However, achieving such goals implies tackling both technical and methodological challenges. In particular, (i) the capability of undertaking autonomous behaviors must comply with strict real-time constraints (e.g., scheduling, communication, and negotiation), (ii) plug-and-play sensors must seamlessly manage data and functional heterogeneity, and finally (iii) multi-device coordination must enable flexible and scalable sensor interactions. Beyond traditional top-down and best-effort solutions, unsuitable for safety-critical scenarios, we propose a novel approach for decentralized real-time compliant semantic agents. In particular, these agents can autonomously coordinate with each other, schedule sensing and data delivery tasks (complying with strict real-time constraints), while relying on ontology-based models to cope with data heterogeneity. Moreover, we present a model that represents sensors as autonomous agents able to schedule tasks and ensure interactions and negotiations compliant with strict timing constraints. Furthermore, to show the feasibility of the proposal, we present a practical study on upper and lower-limb digital rehabilitation scenarios, simulated on the MAXIM-GPRT environment for real-time compliance. Finally, we conduct an extensive evaluation of the implementation of the stream processing multi-agent architecture, which relies on existing RDF stream processing engines.

StreamingCube: Seamless Integration of Stream Processing and OLAP Analysis

Many organizations require real-time analysis of their business data streams to make instantaneous decisions. Data streams are often multidimensional and at the lowest level of abstraction, yet analysts are interested in their multilevel interactive analyses across several dimensions. Online analytical processing (OLAP) is a technique whose utility has been proven for such an analysis. Traditionally, OLAP over streams has been achieved by coupling a stream processing engine (SPE) with an OLAP engine. However, for many organizations, this is not an effective solution as it results in lower performance, resource wastage, and increased complexity and maintenance costs. Hence, we present StreamingCube, which is a unified framework for stream processing and OLAP analysis. The idea of a unified framework is supported by the observation that the incremental computation in stream processing and the maintenance of the materialized view of OLAP have many similarities. To seamlessly integrate the SPE and OLAP, as well as to maintain the OLAP lattice vertices incrementally, a cubify operator is introduced. StreamingCube supports two types of queries, namely, continuous queries (CQs) and OLAP queries. To demonstrate the effectiveness of the proposed framework, a detailed experimental evaluation is presented herein.

Semi-Stream Similarity Join Processing in a Distributed Environment

Similarity join has become very important for semi- or un-structured data processing and analysis. Although several studies have been conducted on the similarity join, little attention has been paid to a semi-stream similarity join, which is a similarity join between stream data and a large disk-based relation. In this study, we propose the first distributed solution called DSim-Join for semi-stream similarity join problem. DSim-Join minimizes the data transmission, reduces database accesses using a cache in a distributed stream processing engine, parallelizes join processing, and balances the load between parallel join threads. Experimental results obtained using real-world datasets show that DSim-Join yields significantly improved throughput compared with state-of-the-art methods, especially for large datasets. The results also show that DSim-Join is scalable and stable; it is not very sensitive to the parameters such as the micro-batch interval, checkpoint interval, and similarity threshold.

AJoin

The processing model of state-of-the-art stream processing engines is designed to execute long-running queries one at a time. However, with the advance of cloud technologies and multi-tenant systems, multiple users share the same cloud for stream query processing. This results in many ad-hoc stream queries sharing common stream sources. Many of these queries include joins. There are two main limitations that hinder performing ad-hoc stream join processing. The first limitation is missed optimization potential both in stream data processing and query optimization layers. The second limitation is the lack of dynamicity in query execution plans. We present AJoin, a dynamic and incremental ad-hoc stream join framework. AJoin consists of an optimization layer and a stream data processing layer. The optimization layer periodically reoptimizes the query execution plan, performing join reordering and vertical and horizontal scaling at run-time without stopping the execution. The data processing layer implements pipelineparallel join architecture. This layer enables incremental and consistent query processing supporting all the actions triggered by the optimizer. We implement AJoin on top of Apache Flink, an open-source data processing framework. AJoin outperforms Flink not only at ad-hoc multi-query workloads but also at single-query workloads.

Distributed and parallel processing for real-time and dynamic spatio-temporal graph

As a non-linear data structure consisting of nodes and edges, the graph data span many different domains. In the real world, applications based on such data structures are always time-sensitive, that is, the value of graph data tends to decrease with time. Furthermore, the application based on spatio-temporal graph is one of the typical representatives of time-sensitive, since the time dimension is an inherent feature of spatio-temporal data. The Distributed Stream Processing Engine (DSPE) seems an excellent choice for the above requirement, which is commonly partitioned and concurrently processed by a number of threads to maximize the throughput. However, it is not feasible to do such mission directly using the traditional DSPE. In this paper, we propose a computational model suitable for handling the spatio-temporal graph in DSPE, by reconstructing the DSPE’s parallel processing slots. Specifically, our proposal includes a general processing framework to deal with the data structure of the spatio-temporal graph, a state information compensation mechanism to ensure the correctness of processing such stateful operation in DSPE, a lightweight summary information calculation method to ensure the performance of the system. Empirical studies on real-world stream applications validate the usefulness of our proposals and prove the considerable advantage of our approaches over state-of-the-art solutions in the literature.

Lock-free Data Structures for Data Stream Processing

Processing data in real-time instead of storing and reading from tables has led to a specialization of DBMS into the so-called data stream processing paradigm. While high throughput and low latency are key requirements to keep up with varying stream behavior and to allow fast reaction to incoming events, there are many possibilities how to achieve them. In combination with modern hardware, like server CPUs with tens of cores, the parallelization of stream queries for multithreading and vectorization is a common schema. High degrees of parallelism, however, need efficient synchronization mechanisms to allow good scaling with threads for shared memory access.In this work, we identify the most time-consuming operations for stream processing exemplarily for our own stream processing engine PipeFabric. In addition, we present different design principles of lock-free data structures which are suited to overcome those bottlenecks. We will finally demonstrate how lock-freedom greatly improves performance for join processing and tuple exchange between operators under different workloads. Nevertheless, the efficient usage of lock-free data structures comes with additional efforts and pitfalls, which we also discuss in this paper.

GeneaLog: Fine-grained data streaming provenance in cyber-physical systems

Streaming applications continuously process data to deliver streams of up-to-date results. Their growing adoption for data analysis in many distributed systems is motivated by their performance (in terms of processing throughput and latency) and their support for easy-to-program distributed and parallel analysis. When streaming applications are designed to detect unusual or critical events (e.g., security- or safety-related), it can be beneficial to maintain the associated source data for further analysis. This can be achieved by fine-grained data provenance, which links each detected event back to the source data that contributed to it, allowing to distinguish and isolate the source data that generated such unusual or critical events.Fine-grained data provenance can be especially useful in cyber-physical systems, such as vehicular networks and smart grids. By enabling the extraction of valuable information from raw sensor data, it could, for instance, reduce data transmission and storage requirements. Since cyber-physical systems can have heterogeneous multi-core architectures, ranging from inexpensive single-board computers to high-end servers, there is a demand for efficient provenance techniques that can take advantage of such parallel architectures with minimal overhead. Motivated by this challenge, we present GeneaLog, a novel fine-grained data provenance technique for data streaming applications. Leveraging the logical dependencies of the data, GeneaLog takes advantage of cross-layer properties of the software stack and incurs a minimal, constant size per-tuple overhead. Furthermore, it allows for a modular and efficient algorithmic implementation using only standard (instrumented) data streaming operators. This is particularly useful to distribute the provenance overheads to operators that can be run in parallel, thus leveraging multi-core architectures. We evaluate two implementations of GeneaLog, one based on Apache Flink, a widely-adopted state-of-the-art Stream Processing Engine, and one based on Liebre, an edge-tailored lightweight Stream Processing Engine. We test them both on vehicular and smart grid applications with single-board embedded devices and a high-end server, also studying how GeneaLog affects their scalability and confirming that it efficiently captures fine-grained provenance data with minimal overhead.

Online template induction for machine-generated emails

In emails, information abounds. Whether it be a bill reminder, a hotel confirmation, or a shipping notification, our emails contain useful bits of information that enable a number of applications. Most of this email traffic is machine-generated, sent from a business to a human. These business-to-consumer emails are typically instantiated from a set of email templates, and discovering these templates is a key step in enabling a variety of intelligent experiences. Existing email information extraction systems typically separate information extraction into two steps: an offline template discovery process (called template induction) that is periodically run on a sample of emails, and an online email annotation process that applies discovered templates to emails as they arrive. Since information extraction requires an email's template to be known, any delay in discovering a newly created template causes missed extractions, lowering the overall extraction coverage. In this paper, we present a novel system called Crusher that discovers templates completely online, reducing template discovery delay from a week (for the existing MapReduce-based batch system) to minutes. Furthermore, Crusher has a resource consumption footprint that is significantly smaller than the existing batch system. We also report on the surprising lesson we learned that conventional stream processing systems do not present a good framework on which to build Crusher. Crusher delivers an order of magnitude more throughput than a prototype built using a stream processing engine. We hope that these lessons help designers of stream processing systems accommodate a broader range of applications like online template induction in the future.

SLA-Based Adaptation Schemes in Distributed Stream Processing Engines

With the upswing in the volume of data, information online, and magnanimous cloud applications, big data analytics becomes mainstream in the research communities in the industry as well as in the scholarly world. This prompted the emergence and development of real-time distributed stream processing frameworks, such as Flink, Storm, Spark, and Samza. These frameworks endorse complex queries on streaming data to be distributed across multiple worker nodes in a cluster. Few of these stream processing frameworks provides fundamental support for controlling the latency and throughput of the system as well as the correctness of the results. However, none has the ability to handle them on the fly at runtime. We present a well-informed and efficient adaptive watermarking and dynamic buffering timeout mechanism for the distributed streaming frameworks. It is designed to increase the overall throughput of the system by making the watermarks adaptive towards the stream of incoming workload, and scale the buffering timeout dynamically for each task tracker on the fly while maintaining the Service Level Agreement (SLA)-based end-to-end latency of the system. This work focuses on tuning the parameters of the system (such as window correctness, buffering timeout, and so on) based on the prediction of incoming workloads and assesses whether a given workload will breach an SLA using output metrics including latency, throughput, and correctness of both intermediate and final results. We used Apache Flink as our testbed distributed processing engine for this work. However, the proposed mechanism can be applied to other streaming frameworks as well. Our results on the testbed model indicate that the proposed system outperforms the status quo of stream processing. With the inclusion of learning models like naïve Bayes, multilayer perceptron (MLP), and sequential minimal optimization (SMO)., the system shows more progress in terms of keeping the SLA intact as well as quality of service (QoS).

Integrating workload balancing and fault tolerance in distributed stream processing system

Distributed Stream Processing Engine (DSPE) is designed for processing continuous streams so as to achieve the real-time performance with low latency guaranteed. To satisfy such requirement, the availability and efficiency are the main concern of the DSPE system, which can be achieved by a proper design of the fault tolerance module and the workload balancing module, respectively. However, the inherent characteristics of data streams, including persistence, dynamic and unpredictability, pose great challenges in satisfying both properties. As far as we know, most of the state-of-the-art DSPE systems take either fault tolerance or workload balancing as its single optimization goal, which in turn receives a higher resource overhead or longer recovery time. In this paper, we combine the fault tolerance and workload balancing mechanisms in the DSPE to reduce the overall resource consumption while keeping the system interactive, high-throughput, scalable and highly available. Based on our data-level replication strategy, our method can handle the dynamic data skewness and node failure scenario: during the distribution fluctuation of the incoming stream, we rebalance the workload by selectively inactivate the data in high-load nodes and activate their replicas on low-load nodes to minimize the migration overhead within the stateful operator; when a fault occurs in the process, the system activates the replicas of the data affected to ensure the correctness while keeping the workload balanced. Extensive experiments on various join workloads on both benchmark data and real data show our superior performance compared with baseline systems.

Distributed Join Processing Between Streaming and Stored Big Data Under the Micro-Batch Model

In order to interpret, enrich, and analyze the streaming data, stream applications often access the data stored in an external database. Although there has been a lot of studies on stream processing, little attention has been paid so far to the join between streaming data and stored data. In this paper, we propose a comprehensive solution called DS-join for distributed processing of the join under the micro-batch model of recently distributed stream processing engines (SPEs), such as spark streaming. The micro-batch model performs stream processing as a series of very small batch jobs and is more fault-tolerant in a distributed environment compared with the record-at-a-time model. The DS-join reduces the number of database accesses by using micro-batching. Furthermore, the DS-join optimizes the join operation by minimizing the data shuffling, managing a cache in a distributed SPE, parallelizing the join processing, and balancing the load between the SPE and the external database system. The experimental results using real and synthetic datasets show that, compared with the state-of-the-art methods, the DS-join significantly improves throughput, especially for large databases.

Passive and Partially Active Fault Tolerance for Massively Parallel Stream Processing Engines

Fault-tolerance techniques for stream processing engines can be categorized into passive and active approaches. However, both approaches have their own inadequacies in Massively Parallel Stream Processing Engines (MPSPE). The passive approach incurs a long recovery latency especially when a number of correlated nodes fail simultaneously, while the active approach requires extra replication resources. In this paper, we propose a new fault-tolerance framework, which is Passive and Partially Active (PPA). In a PPA scheme, the passive approach is applied to all tasks while only a selected set of tasks will be actively replicated. The number of actively replicated tasks depends on the available resources. If tasks without active replicas fail, tentative outputs will be generated before the completion of the recovery process. We also propose effective and efficient algorithms to optimize a partially active replication plan to maximize the quality of tentative outputs. We implemented PPA on top of Storm, an open-source MPSPE and conducted extensive experiments using both real and synthetic datasets to verify the effectiveness of our approach.