Data Stream Management Systems Research Articles

Monitoring Big Data Streams Using Data Stream Management Systems: Industrial Needs, Challenges, and Improvements

Real-time monitoring systems are important for industry since they allow for avoiding unplanned system stops and keeping system availability high. The technical requirements for such systems include being both scalable and online, as the amount of generated data is increasing with time. Therefore, monitoring systems must integrate tools that can manage and analyze the data streams. The data stream management system is a stream processing tool that has the ability to manage and support operations on data streams in real-time. Several researchers have proposed and tested real-time monitoring systems which have the ability to search big data streams. In this paper, the research works that discuss the analysis of online data streams for fault detection in industry are reviewed. Based on the literature analysis, the industrial needs and challenges of monitoring big data streams are presented. Furthermore, feasible suggestions for improving the real-time monitoring system are proposed.

Big Data Streaming Platforms: A Review

Yesterday’s “Big Data” is today’s “data.” As technology advances, new difficulties and new solutions emerge. In recent years, as a result of the development of Internet of Things (IoT) applications, the area of Data Mining has been confronted with the difficulty of analyzing and interpreting data streams in real time and at a high data throughput. This situation is known as the velocity element of big data. The rapid advancement of technology has come with an increased use of social media, computer networks, cloud computing, and the IoT. Experiments in the laboratory also generate a large quantity of data, which must be gathered, handled, and evaluated. This massive amount of data is referred to as “Big Data.” Analysts have seen an upsurge in data including valuable and worthless elements. In extracting usable information, data warehouses struggle to keep up with the rising volume of data collected. This article provides an overview of big data architecture and platforms, tools for data stream processing, and examples of implementations. Streaming computing is the focus of our project, which is building a data stream management system to deliver large-scale, cost-effective big data services. Owing to this study, the feasibility of large-scale data processing for distributed, real-time computing is improved even when the systems are overwhelmed.

GeoFlink: An Efficient and Scalable Spatial Data Stream Management System

This era is witnessing an exponential growth in spatial data due to the increase in GPS-enabled devices. Spatial data can be of extreme use to commercial businesses, governments and NGOs if processed timely. Spatial data is voluminous and is usually generated as a continuous data stream, for instance, vehicles tracking data, mobile location data, etc. To process such a huge data streams, highly scalable systems are needed. Apache Spark Streaming, Apache Flink, and Apache Samza are among the state-of-the-art scalable stream processing platforms; however, they lack spatial objects, indexes, and queries support. Besides them, other scalable spatial data processing platforms including GeoSpark, Spatial Hadoop do not support streaming workloads and can only handle static or batch data. To fill this gap, we present GeoFlink which extends Apache Flink to support spatial objects, indexes and continuous queries over spatial data streams. A grid-based index is introduced to support efficient spatial query processing and effective data distribution across distributed cluster nodes. GeoFlink supports spatial range, spatial <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>NN and spatial join queries on Point, LineString, Polygon, MultiPoint, MultiLineString, and MultiPolygon spatial objects. Besides, GeoFlink supports data streams in GeoJSON, WKT, and CSV data formats. A detailed experimental study on real and synthetic spatial data streams proves that GeoFlink achieves significantly higher query throughput than the existing state-of-the-art streaming platforms.

SPEAR: Dynamic Spatio-Temporal Query Processing over High Velocity Data Streams.

With the advent of IoT and emerging 5G technology, real-time streaming data are being generated at unprecedented speed and volume, and coming with both temporal and spatial dimensions. Effective analysis at such scale and speed requires support for dynamically adjusting querying capabilities in real-time. In spatio-temporal domain, this warrants for data as well as query optimization strategies especially for objects with changing motion states. Contemporary spatio-temporal data stream management systems in distributed domain are mostly dominated by specified-once-applied-continuously query model. Any modification in query state requires query restart limiting system responsiveness and producing outdated or in worst case erroneous results. In this paper, we propose adaptations of principles from streaming databases, spatial data management and distributed computing to support dynamic spatio-temporal query processing over high velocity big data streams. We first formulate a set of spatio-temporal data types and functions to seamlessly handle changes in distributed query states. We develop a comprehensive set of streaming spatio-temporal querying methods, and propose geohash based dynamic spatial partitioning for effective parallel processing. We implement a prototype on top of Apache Flink, where the in-memory stream processing fits nicely with our spatio-temporal models. Comparative evaluation of our prototype demonstrates the effectiveness our strategy by maintaining high consistent processing rates for both stationary as well as moving queries over high velocity spatio-temporal big data streams.

Optimization of threshold functions over streams

A common stream processing application is alerting, where the data stream management system (DSMS) continuously evaluates a threshold function over incoming streams. If the threshold is crossed, the DSMS raises an alarm. The threshold function is often calculated over two or more streams, such as combining temperature and humidity readings to determine if moisture will form on a machine and therefore cause it to malfunction. This requires taking a temporal join across the input streams. We show that for the broad class of functions called quasiconvex functions, the DSMS needs to retain very few tuples per-data-stream for any given time interval and still never miss an alarm. This surprising result yields a large memory savings during normal operation. That savings is also important if one stream fails, since the DSMS would otherwise have to cache all tuples in other streams until the failed stream recovers. We prove our algorithm is optimal and provide experimental evidence that validates its substantial memory savings.

On-board Clutch Slippage Detection and Diagnosis in Heavy Duty Machine

In order to reduce unnecessary stops and expensive downtime originating from clutch failure of construction equipment machines; adequate real time sensor data measured on the machine in combination with feature extraction and classification methods may be utilized.This paper presents a framework with feature extraction methods and an anomaly detection module combined with Case-Based Reasoning (CBR) for on-board clutch slippage detection and diagnosis in heavy duty equipment. The feature extraction methods used are Moving Average Square Value Filtering (MASVF) and a measure of the fourth order statistical properties of the signals implemented as continuous queries over data streams. The anomaly detection module has two components, the Gaussian Mixture Model (GMM) and the Logistics Regression classifier. CBR is a learning approach that classifies faults by creating a new solution for a new fault case from the solution of the previous fault cases. Through use of a data stream management system and continuous queries (CQs), the anomaly detection module continuously waits for a clutch slippage event detected by the feature extraction methods, the query returns a set of features, which activates the anomaly detection module. The first component of the anomaly detection module trains a GMM to extracted features while the second component uses a Logistic Regression classifier for classifying normal and anomalous data. When an anomaly is detected, the Case-Based diagnosis module is activated for fault severity estimation.

SSTD

Streaming spatio-textual data that contains geolocations and textual contents, e.g., geo-tagged tweets, is becoming increasingly available. Users can register continuous queries to receive up-to-date results continuously, or pose snapshot queries to receive results instantly. The large scale of spatio-textual data streams and huge amounts of queries pose great challenges to the current location-based services, and call for more efficient data management systems. In this paper, we present SSTD (Streaming Spatio-Textual Data), a distributed in-memory system supporting both continuous and snapshot queries with spatial, textual, and temporal constraints over data streams. Compared with existing distributed data stream management systems, SSTD has at least three novelty: (1) It supports more types of queries over streamed spatio-textual data ; (2)SSTD adopts a novel workload partitioning method termed QT (Quad-Text) tree, that utilizes the joint distribution of queries and spatio-textual data to reduce query latency and enhance system throughput. (3) To achieve load balance and robustness, we develop three new workload adjustment methods for SSTD to fit the changes in the distributions of data or queries. Extensive experiments on real-life datasets demonstrate the superior performance of SSTD.

Window Operators for Processing Spatio-Temporal Data Streams on Unmanned Vehicles

Abstract. Unmanned aerial and submersible vehicles are used in an increasing number of applications especially for data collection in misanthropic environments. During a mission, such vehicles generate multiple spatio-temporal data streams suitable to be processed by data stream management systems (DSMS). The main approach of a DSMS is limiting the elements of a stream by using sliding and tilting windows with time intervals as temporal condition. However, due to varying vehicle speed and limited on-board resources, such temporal windows do not provide adequate support for spatio-temporal problems. For solving this problem, we propose a set of six new spatio-temporal window operators in this paper. This set comprises of sliding distance, tilting distance, tilting waypoint, session distance, jumping distance and an area window to limit stream elements based on spatial conditions. Each of the listed operators provides an individual behaviour to support sophisticated applications like spatial interpolation and forecasting. An evaluation based on an example trajectory shows the benefit of the presented operators for spatio-temporal applications.

RCD+: A Partitioning Method for Data Streams Based on Multiple Queries

Big data stream management systems often must transform a query application into multiple query tasks, simultaneously and dynamically partitioning data streams based on attribute values or partitioning keys. However, due to different partitioning orders or strategies of partitioning keys, the redundant and repetitive transmission of data streams at different nodes leads to system performance degradation. In addition, with the change of data skewness, the problem of unbalanced data stream partitioning still exists between different processing units within the same node. This paper presents the partitioning framework RCD+ (Runtime Correlation Discovery) according to runtime correlation discovery. RCD+ implements the full granularity partitioning strategy, which includes runtime positive correlation partitioning (RPC-partitioning) and clustering partitioning (Clu-partitioning). First, in the process of RPC-partitioning, we introduce the mini-batch scheme to reduce the number of output stream caches and partition data streams using the relevance of partitioning keys. Furthermore, in the process of Clu-partitioning, we re-partition data streams by clustering of skewed data streams between the inter-node and the intra-node. Then, we construct the routing table to manage partition states in order to ensure the correctness of multiple query tasks. Finally, we have implemented this framework on Apache Storm. Experiments with synthetic data and real data show that our proposed framework exhibits better query performance.

Microblaze-Based Coprocessor for Data Stream Management System

Data generation speed and volume have increased exponentially with the boom in Internet usage and with the advent of Internet of Things (IoT). Consequently, the need for processing these data faster and with higher efficiency has also grown in-over time. Many previous works tried to address this need and among them is Symbiote Coprocessor Unit (SCU), an accelerator capable of providing speedup of up to 150x compared with traditional data stream processors. The proposed architecture aims to reduce the complexity of SCU, making it flexible and still retaining its performance. The new design is more software driven and thus is very easy to be altered in the future if needed. We have also changed the older interface to industrial standard PCIe interface and AMBA AXI4 bus interconnect in order to make the design simple and open for future expansions.

A Hybrid Deep Learning-Based Model for Anomaly Detection in Cloud Datacenter Networks

With the emergence of the Internet-of-Things (IoT) and seamless Internet connectivity, the need to process streaming data on real-time basis has become essential. However, the existing data stream management systems are not efficient in analyzing the network log big data for real-time anomaly detection. Further, the existing anomaly detection approaches are not proficient because they cannot be applied to networks, are computationally complex, and suffer from high false positives. Thus, in this paper a hybrid data processing model for network anomaly detection is proposed that leverages grey wolf optimization (GWO) and convolutional neural network (CNN). To enhance the capabilities of the proposed model, GWO and CNN learning approaches were enhanced with: 1) improved exploration, exploitation, and initial population generation abilities and 2) revamped dropout functionality, respectively. These extended variants are referred to as Improved-GWO (ImGWO) and Improved-CNN (ImCNN). The proposed model works in two phases for efficient network anomaly detection. In the first phase, ImGWO is used for feature selection in order to obtain an optimal trade-off between two objectives, i.e., reduced error rate and feature-set minimization. In the second phase, ImCNN is used for network anomaly classification. The efficacy of the proposed model is validated on benchmark (DARPA’98 and KDD’99) and synthetic datasets. The results obtained demonstrate that the proposed cloud-based anomaly detection model is superior in comparison to the other state-of-the-art models (used for network anomaly detection), in terms of accuracy, detection rate, false positive rate, and F-score. In average, the proposed model exhibits an overall improvement of 8.25%, 4.08%, and 3.62% in terms of detection rate, false positives, and accuracy, respectively; relative to standard GWO with CNN.

Data stream management system for video on demand hybrid storage server

Video on demand (VOD) server requires a large storage capacity and rapid video retrieval from this storage to quickly stream these videos to users. The hybrid storage system, which combines HDDs and SSD in the server, has become popular because of such requirements. The combination of both storage modes is relatively weak in terms of optimising fast access prior and in supporting the production of a high number of simultaneous streams. This paper presents an enhanced hybrid storage system (EHSS)-based VOD server to improve the performance of the VOD server. The design of the EHSS and its streaming management scheme produce high performance and satisfy the performance requirements of a VOD server in terms of I/O throughput and access latency. The experimental results show that the proposed work provides better performance than the VOD server-based FADM because it enhances the average response times for the various scales of intensive workload by 69.89%.

Data stream management system for video on demand hybrid storage server

Data stream management system for video on demand hybrid storage server

A need for an integrative security model for semantic stream reasoning systems

State-of-the-art security frameworks have been extensively addressing security issues for web resources, agents and services in the Semantic Web. The provision of Stream Reasoning as a new area spanning Semantic Web and Data Stream Management Systems has eventually opened up new challenges. Namely, their decentralized nature, the metadata descriptions, the number of users, agents, and services, make securing Stream Reasoning systems difficult to handle. Thus, there is an inherent need of developing new security models which will handle security and automate security mechanisms to a more autonomous system that supports complex and dynamic relationships between data, clients and service providers. We plan to validate our approach on a typical application of stream data, on Wireless Sensor Networks (WSNs). In particular, WSNs for water quality monitoring will serve as a case study. The paper describes the initial findings and research plan for building a consistent security model for stream reasoning systems.

Automating Characterization Deployment in Distributed Data Stream Management Systems

Distributed data stream management systems (DDSMS) are usually composed of upper layer relational query systems (RQS) and lower layer stream processing systems (SPS). When users submit new queries to RQS, a query planner needs to be converted into a directed acyclic graph (DAG) consisting of tasks which are running on SPS. Based on different query requests and data stream properties, SPS need to configure different deployments strategies. However, how to dynamically predict deployment configurations of SPS to ensure the processing throughput and low resource usage is a great challenge. This article presents OrientStream, a framework for automating characterization deployment in DDSMS using incremental machine learning techniques. By introducing the data-level, query plan-level, operator-level, and cluster-level’s four-level feature extraction mechanism, we first use the different query workloads as training sets to predict the resource usage by DDSMS, and select the optimal resource configuration from candidate settings based on the current query requests and stream properties, then migrate the operator state by introducing dynamic reconfiguration. Finally, we validate our approach on the open source SPS–Storm. In view of the application scenarios with long monitoring cycle and non-frequent data fluctuation, experiments show that OrientStream can reduce CPU usage of 8-15 percent and memory usage of 38-48 percent, respectively.

A Stepwise Auto-Profiling Method for Performance Optimization of Streaming Applications

Data stream management systems (DSMSs) are scalable, highly available, and fault-tolerant systems that aggregate and analyze real-time data in motion. To continuously perform analytics on the fly within the stream, state-of-the-art DSMSs host streaming applications as a set of interconnected operators, with each operator encapsulating the semantic of a specific operation. For parallel execution on a particular platform, these operators need to be appropriately replicated in multiple instances that split and process the workload simultaneously. Because the way operators are partitioned affects the resulting performance of streaming applications, it is essential for DSMSs to have a method to compare different operators and make holistic replication decisions to avoid performance bottlenecks and resource wastage. To this end, we propose a stepwise profiling approach to optimize application performance on a given execution platform. It automatically scales distributed computations over streams based on application features and processing power of provisioned resources and builds the relationship between provisioned resources and application performance metrics to evaluate the efficiency of the resulting configuration. Experimental results confirm that the proposed approach successfully fulfills its goals with minimal profiling overhead.

Streaming Tables: Native Support to Streaming Data in DBMSs

Data stream management systems (DSMSs) are conceived for running continuous queries (CQs) on the most recently streamed data. This model does not completely fit the needs of several modern data-intensive applications that require to manage recent/historical/static data and execute both CQs and OTQs joining such data. In order to cope with these new needs, some DSMSs have moved toward the integration of database management systems (DBMSs) functionalities to augment their capabilities. In this paper we adopt the opposite perspective and we lay the groundwork for extending DBMSs to natively support streaming facilities. To this end, we introduce a new kind of table, the streaming table , as a persistent structure where streaming data enters and remains stored for a long period, ideally forever. Streaming tables feature a novel access paradigm: continuous writes and one-time as well as continuous reads. We present a streaming table implementation and two novel types of indices that efficiently support both update and scan high rates. A detailed experimental evaluation shows the effectiveness of the proposed technology.

Stream-Based Reasoning for IoT Applications — Proposal of Architecture and Analysis of Challenges

As distributed IoT applications become larger and more complex, the pure processing of raw sensor and actuation data streams becomes impractical. Instead, data streams must be fused into tangible facts and these pieces of information must be combined with a background knowledge to infer new pieces of knowledge. And since many IoT applications require almost real-time reactivity to stimulus of the environment, such information inference process has to be performed in a continuous, on-line manner. This paper proposes a new semantic model for data stream processing and real-time reasoning based on the concepts of Semantic Stream and Fact Stream, as a natural extension of Complex Event Processing (CEP) and RDF (graph-based knowledge model). The main advantages of our approach are that: (a) it considers time as a key relation between pieces of information; (b) the processing of streams can be implemented using CEP; (c) it is general enough to be applied to any Data Stream Management System (DSMS). We describe a scenario about patients flux monitoring in a hospital as an example of prospective application. Last, we present challenges and prospects on using machine learning and induction algorithms to learn abstractions and reasoning rules from a continuous data stream.

C-SWRL: A Unique Semantic Web Framework for Reasoning Over Stream Data

The synergy of Data Stream Management Systems and Semantic Web applications has steered towards a new paradigm known as Stream Reasoning. The Semantic Web standards for knowledge base modeling and querying, namely RDF, OWL and SPARQL, has extensively been used by the Stream Reasoning community. However, the Semantic Web rule languages, such as SWRL and RIF, have never been used in stream data applications. Instead, different non-Semantic Web rule systems have been approached. Since RIF is primarily intended for exchanging rules among systems, we focused on SWRL applications with stream data. This proves difficult following the SWRL’s open world semantics. To overcome SWRL’s expressivity issues we propose an infrastructure extension, which will enable SWRL reasoning with stream data. Namely, a query processing system, such as C-SPARQL, was layered under SWRL to support closed-world and time-aware reasoning. Moreover, OWLAPI constructs were utilized to enable non-monotonicity, while SPARQL constructs were used to enable negation as failure. Water quality monitoring was used as a validation domain of the proposed system.

Scheduling parallel and distributed processing for automotive data stream management system

In this paper, to analyze end-to-end timing behavior in heterogeneous processor and network environments accurately, we adopt and modify a heterogeneous selection value on communication contention (HSV_CC) algorithm, which can synchronize tasks and messages simultaneously, for stream processing distribution. In order to adapt the concepts of a static algorithm like HSV_CC to automotive data stream management system (DSMSs), one must first address three issues: (i) previous task and message schedules might lead to less efficient resource usages in this scenario; (ii) the conventional method to determine the task scheduling order may not be best suited to deal with stream processing graphs, and; (iii) there is a need to be able to schedule tasks with time-varying computational requirements efficiently. To address (i), we propose the heterogeneous value with load balancing and communication contention (HVLB_CC) (A) algorithm, which considers load balancing in addition to the parameters considered by the HSV_CC algorithm. We propose HVLB_CC (B) to address issue (ii). HVLB_CC (B) can deal with stream processing task graphs and more various directed acyclic graphs to prevent assigning a higher priority to successor tasks. In addition, to address issue (iii), we propose HVLB_CC_IC. To schedule tasks more efficiently with various computation times, HVLB_CC_IC utilizes schedule holes left in processors. These idle time slots can be used for the execution of an optional part to generate more precise data results by applying imprecise computation models. Experimental results demonstrate that the proposed algorithms improve minimum schedule length, accuracy, and load balancing significantly compared to the HSV_CC algorithm. In addition, the proposed HVLB_CC (B) algorithm can schedule more varied task graphs without reducing performance, and, using imprecise computation models, HVLB_CC_IC yields higher precision data than HVLB_CC without imprecise computation models.