Real-time Data Stream Processing Research Articles

A Distance-Window Approach for the Continuous Processing of Spatial Data Streams

Real-time and continuous processing of citywide spatial data is an essential requirement of smart cities to guarantee the delivery of basic life necessities to its residents and to maintain law and order. To support real-time continuous processing of data streams, continuous queries (CQs) are used. CQs utilize windows to split the unbounded data streams into finite sets or windows. Existing stream processing engines either support time-based or count-based windows. However, these are not much useful for the spatial streams containing the trajectories of moving objects. Hence, this paper presents a distance-window based approach for the processing of spatial data streams, where the unbounded streams can be split with respect to the trajectory length. Since the window operation involves repeated computation, this work presents two incremental distance-based window approaches to avoid the repetition. A detailed experimental evaluation is presented to prove the effectiveness of the proposed incremental distance-based windows.

SBASH Stack Based Allocation of Sheer Window Architecture for Real Time Stream Data Processing

The processing of real-time data streams is complex with large number of volume and variety. The volume and variety of data streams enhances a number of processing units to run in real time. The required number of processing units used for processing data streams are lowered by using a windowing mechanism. Therefore, the appropriate size of window selection is vital for stream data processing. The coarse size window will directly affect the overall processing time. On the other hand, a finely sized window has to deal with an increased number of management costs. In order to manage such streams of data, we have proposed a SBASH architecture, which can be helpful for determining a unipartite size of a sheer window. The sheer window reduces the overall latency of data stream processing by a certain extent. The time complexity to process such sheer window is equivalent to w log n w. These windows are allocated and retrieved in a stack-based manner, where stacks ≥ n, which is helpful in reducing the number of comparisons made during retrieval.

An efficient algorithm for reducing the flow of real-time data stream with least sampling error

Nature of data stream is determined after complete scanning of whole data sets during real-time data processing. However, it becomes inconvenient to process entire data stream at once in real-time data stream processing. Thus, a sheer sized fixed window of data streams is processed at a particular time. The intensification of sheer sized fixed window at processing node is mitigated by reducing the flowing rate of data stream. Heuristic clustering windowing (HCW) approach and partial blind window (PBW) algorithms are proposed for reducing the flow of data stream with least sampling error. These approaches consist of the combination of systematic sampling and clustering mechanism. A clustering approach is applied on one fraction of data streams whereas systematic sampling handles other portion of streams. These approaches are helpful in reducing flow of data streams in minimum latency.

Real-Time Healthcare Monitoring System using Online Machine Learning and Spark Streaming

The real-time monitoring and tracking systems play a critical role in the healthcare field. Wearable medical devices with sensors, mobile applications, and health cloud have continuously generated an enormous amount of data, often called streaming big data. Due to the higher speed of the streaming data, it is difficult to ingest, process, and analyze such huge data in real-time to make real-time actions in case of emergencies. Using traditional methods that are inadequate and time-consuming. Therefore, there is a significant need for real-time big data stream processing to guarantee an effective and scalable solution. So, we proposed a new system for online prediction to predict health status using Spark streaming framework. The proposed system focuses on applying streaming machine learning models (i.e. streaming linear regression with SGD) on streaming health data events ingested to spark streaming through Kafka topics. The experimental results are done on the historical medical datasets (i.e. diabetes dataset, heart disease dataset, and breast cancer dataset) and generated dataset which is simulated to wearable medical sensors. The historical datasets have shown that the accuracy improvement ratio obtained using the diabetes disease dataset is the highest one with respect to the other two datasets with an accuracy of 81%. For generated datasets, the online prediction system has achieved accuracy with 98% at 5 seconds window size. Beyond this, the experimental results have proofed that the online prediction system can online learn and update the model according to the new data arrival and window size.

A New Big Data Architecture for Real-Time Student Attention Detection and Analysis

Big Data technologies and their analytical methods can help improve the quality of education. They can be used to process and analyze classroom video streams to predict student attention, this would greatly improve the learning-teaching experience. With the increasing number of students and the expansion of educational institutions, processing and analyzing video streams in real-time become a complicated issue. In this paper, we have reviewed the existing systems of student attention detection, open-source real-time data stream processing technologies, and the two major data stream processing architectures. We also proposed a new Big Data architecture for real-time student attention detection.

Big Data Processing-Beyond Batch Processing

This paper mainly focus on analysis of large sets of students data with one of the batch processing analysis techniques Beyond batch process, analysis of data streaming is done based on program of word counting program which executes data with HDFS along with dynamic created data. To compute similar coherent strategies one can implement a schema named batch and streaming process which dynamically creates data. The architecture is reduced to serve as X-Platform which uses ample number of tools for batch and stream analysis on this proposed frame work. Here we use spark-sql, a query language which acts as interface for interactive process to have iterative processes. Real time streaming data processing involves spark streaming works. Here we focus on preliminary evaluation of results and analysis report which compares data sets performance and also achieve low latency rate with usage of RDD.

A new Internet of Things architecture for real-time prediction of various diseases using machine learning on big data environment

A number of technologies enabled by Internet of Thing (IoT) have been used for the prevention of various chronic diseases, continuous and real-time tracking system is a particularly important one. Wearable medical devices with sensor, health cloud and mobile applications have continuously generating a huge amount of data which is often called as streaming big data. Due to the higher speed of the data generation, it is difficult to collect, process and analyze such massive data in real-time in order to perform real-time actions in case of emergencies and extracting hidden value. using traditional methods which are limited and time-consuming. Therefore, there is a significant need to real-time big data stream processing to ensure an effective and scalable solution. In order to overcome this issue, this work proposes a new architecture for real-time health status prediction and analytics system using big data technologies. The system focus on applying distributed machine learning model on streaming health data events ingested to Spark streaming through Kafka topics. Firstly, we transform the standard decision tree (DT) (C4.5) algorithm into a parallel, distributed, scalable and fast DT using Spark instead of Hadoop MapReduce which becomes limited for real-time computing. Secondly, this model is applied to streaming data coming from distributed sources of various diseases to predict health status. Based on several input attributes, the system predicts health status, send an alert message to care providers and store the details in a distributed database to perform health data analytics and stream reporting. We measure the performance of Spark DT against traditional machine learning tools including Weka. Finally, performance evaluation parameters such as throughput and execution time are calculated to show the effectiveness of the proposed architecture. The experimental results show that the proposed system is able to effectively process and predict real-time and massive amount of medical data enabled by IoT from distributed and various diseases.

The identifying hidden data features problem solution

In the article, we considered recommender models based on matrix factorization demonstrate excellent performance in collaborative filtering. The standard Matrix Factorization approach in MLlib deals with clear ratings. To work with implicit data, we used the trainImplicit method. To simulate the processing of real-time data streams, we used the Spark Streaming library, which is responsible for receiving data from the input source and converting the raw data into a discretized stream discretized stream (DStream) consisting of Spark RDD. The rank parameter determines the number of hidden features in the low rank approximation matrices. As a rule, the greater the number of factors, the better, but for a large number of users or elements, it will directly affect the memory usage of the computing system and the amount of data required for training. Therefore, in our problem it was a compromise solution.

Hydrological stream data pipeline framework based on IoTDB

With the increasing amount of hydrological data in Chuhe river basin, the traditional relational database has been unable to meet the needs of users, which not only makes it difficult to achieve low latency and high throughput in the real-time transmission of hydrological data, but also causes the phenomenon of long time or even system crash when querying large amount of annual water-level data. To solve this problem, this paper proposes a stream data pipeline framework based on timeseries databases IoTDB and Kafka, which can provide services for hydrological early warning and anomaly detection researchers. Based on the hydrological sensor data of Chuhe river, the processing scenarios of sensor stream data are set and compared with other NoSQL (HBase, MongoDB, RiakTS and Redis) in different scenarios. The performance and workload of different NoSQL in this data pipeline are tested. Finally, it is docked with Flink real-time stream data processing platform and compared with other data pipelines. The experimental results show that the stream data pipeline composed of IoTDB, Kafka and Flink is outstanding in data acquisition, transmission, incremental query and data analysis.

Action recognition using optimized deep autoencoder and CNN for surveillance data streams of non-stationary environments

Action recognition is a challenging research area in which several convolutional neural networks (CNN) based action recognition methods are recently presented. However, such methods are inefficient for real-time online data stream processing with satisfied accuracy. Therefore, in this paper we propose an efficient and optimized CNN based system to process data streams in real-time, acquired from visual sensor of non-stationary surveillance environment. Firstly, frame level deep features are extracted using a pre-trained CNN model. Next, an optimized deep autoencoder (DAE) is introduced to learn temporal changes of the actions in the surveillance stream. Furthermore, a non-linear learning approach, quadratic SVM is trained for the classification of human actions. Finally, an iterative fine-tuning process is added in the testing phase that can update the parameters of trained model using the newly accumulated data of non-stationary environment. Experiments are conducted on benchmark datasets and results reveal the better performance of our system in terms of accuracy and running time compared to state-of-the-art methods. We believe that our proposed system is a suitable candidate for action recognition in surveillance data stream of non-stationary environments.

A Survey of Distributed Data Stream Processing Frameworks

Big data processing systems are evolving to be more stream oriented where each data record is processed as it arrives by distributed and low-latency computational frameworks on a continuous basis. As the stream processing technology matures and more organizations invest in digital transformations, new applications of stream analytics will be identified and implemented across a wide spectrum of industries. One of the challenges in developing a streaming analytics infrastructure is the difficulty in selecting the right stream processing framework for the different use cases. With a view to addressing this issue, in this paper we present a taxonomy, a comparative study of distributed data stream processing and analytics frameworks, and a critical review of representative open source (Storm, Spark Streaming, Flink, Kafka Streams) and commercial (IBM Streams) distributed data stream processing frameworks. The study also reports our ongoing study on a multilevel streaming analytics architecture that can serve as a guide for organizations and individuals planning to implement a real-time data stream processing and analytics framework.

FGCH: a fast and grid based clustering algorithm for hybrid data stream

Streaming large volumes of data has a wide range of real-world applications, e.g., video flows, internet calls, and online games etc. Thus, fast and real-time data stream processing is important. Traditionally, data clustering algorithms are efficient and effective to mine information from large data. However, they are mostly not suitable for online data stream clustering. Therefore, in this work, we propose a novel fast and grid based clustering algorithm for hybrid data stream (FGCH). Specifically, we have made the following main contributions: 1), we develop a non-uniform attenuation model to enhance the resistance to noise; 2), we propose a similarity calculation method for hybrid data, which can calculate the similarity more efficiently and accurately; and 3), we present a novel clustering center fast determination algorithm (CCFD), which can automatically determine the number, center, and radius of clusters. Our technique is compared with several state-of-art clustering algorithms. The experimental results show that our technique can achieve more than better clustering accuracy on average. Meanwhile, the running time is shorter compared with the closest algorithm.

Constant-Time Sliding Window Framework with Reduced Memory Footprint and Efficient Bulk Evictions

The fast evolution of data analytics platforms has resulted in an increasing demand for real-time data stream processing. From Internet of Things applications to the monitoring of telemetry generated in large data centers, a common demand for currently emerging scenarios is the need to process vast amounts of data with low latencies, generally performing the analysis process as close to the data source as possible. Stream processing platforms are required to be malleable and absorb spikes generated by fluctuations of data generation rates. Data is usually produced as time series that have to be aggregated using multiple operators, being sliding windows one of the most common abstractions used to process data in real-time. To satisfy the above-mentioned demands, efficient stream processing techniques that aggregate data with minimal computational cost need to be developed. In this paper we present the Monoid Tree Aggregator general sliding window aggregation framework, which seamlessly combines the following features: amortized $O(1)$ time complexity and a worst-case of $O(\log {n})$ between insertions; it provides both a window aggregation mechanism and a window slide policy that are user programmable; the enforcement of the window sliding policy exhibits amortized $O(1)$ computational cost for single evictions and supports bulk evictions with cost $O(\log {n})$ ; and it requires a local memory space of $O(\log {n})$ . The framework can compute aggregations over multiple data dimensions, and has been designed to support decoupling computation and data storage through the use of distributed Key-Value Stores to keep window elements and partial aggregations.

Design of Real-time Disease Warning System Based on Storm Technology

With the improvement of the quality of clinical diagnosis and treatment, the traditional scheduled "ward round" mode cannot meet the demands for real-time monitoring of acute and critically ill patients. This paper introduces the Storm, a real-time data stream processing technology and its application in the real time disease early warning system. By collecting the clinical data flow and calculating the MEWS scores in real time, the system can identify the potential deterioration of the disease, and promptly notify the medical staff. Score calculation results can be stored for further analysis and presentation as well.

Online Real-Time Analysis of Data Streams Based on an Incremental High-Order Deep Learning Model

As the core part of the new generation of information technology, the Internet of Things has accumulated a large number of real-time data streams of various types and structures. The data stream is generated at an extremely fast speed, and its content and distribution characteristics are all in high-speed dynamic changes, which must be processed in real time. Therefore, the feature learning algorithm is required to support incremental updates and learn the characteristics of high-speed dynamic change data in real time. Most of the current machine learning models for processing big data belong to the static learning model. The batch learning method makes it impossible to analyze data streams in real time, and the learning ability of dynamic data streams is poor. Therefore, this paper proposes an incremental high-order deep learning model to extend the data from the vector space to the tensor space and update the parameters and structure of the network model in the high-order tensor space. In the process of parameter updating, the first-order approximation concept is introduced to avoid incrementing parameters by the iterative method and to improve the parameter update efficiency, so that the updated model can quickly learn the characteristics of dynamically changing big data and satisfy the real-time requirements of big data feature learning while maintaining the original knowledge of the neural network model as much as possible. To evaluate the performance of the proposed model, experiments were performed on real image data sets-MNIST, and the model was evaluated for stability, plasticity, and run time. The experimental results show that the model not only has the ability to incrementally learn the characteristics of new data online but also retains the ability to learn the original data features, improve the model update efficiency, and maximize the online analysis and real-time processing of dynamic data streams.

A Real Time Stream Data Processing and Analysis Model and Catchments over Twitter Stream Data

A Real Time Stream Data Processing and Analysis Model and Catchments over Twitter Stream Data

ScaleJoin: A Deterministic, Disjoint-Parallel and Skew-Resilient Stream Join

The inherently large and varying volumes of information generated in large scale systems demand near real-time processing of data streams. In this context, data streaming is imperative for data-intensive processing infrastructures. Stream joins, the streaming counterpart of database joins, compare tuples coming from different streams and constitute one of the most important and expensive data streaming operators. Algorithmic implementations of stream joins have to be capable of efficiently processing bursty and rate-varying data streams in a deterministic and skew-resilient fashion. To leverage the design of modern multicore architectures, scalability and parallelism need to be addressed also in the algorithmic design. In this paper we present ScaleJoin, an algorithmic construction for deterministic and parallel stream joins that guarantees all the above properties, thus filling in a gap in the existing state-of-the-art. Key to the novelty of ScaleJoin is the ScaleGate data structure and its lock-free implementation. ScaleGate facilitates concurrent data exchange and balances independent actions among processing threads; enabling fine-grain parallelism and deterministic processing. It allows ScaleJoin to run on an arbitrary number of processing threads, evenly sharing the overall comparisons run in parallel and achieving disjoint and skew-resilient high processing throughput and low processing latency.

Real-time Data Stream Processing - Challenges and Perspectives

Real-time Data Stream Processing - Challenges and Perspectives

On dynamic memory allocation in sliding-window parallel patterns for streaming analytics

This work studies the issues related to dynamic memory management in Data Stream Processing, an emerging paradigm enabling the real-time processing of live data streams. In this paper, we consider two streaming parallel patterns and we discuss different implementation variants related to how dynamic memory is managed. The results show that the standard mechanisms provided by modern C++ are not entirely adequate for maximizing the performance. Instead, the combined use of an efficient general purpose memory allocator, a custom allocator optimized for the pattern considered and a custom variant of the C++ shared pointer mechanism, provides a performance improvement up to 16% on the best case.

Strider

Real-time processing of data streams emanating from sensors is becoming a common task in industrial scenarios. An increasing number of processing jobs executed over such platforms are requiring reasoning mechanisms. The key implementation goal is thus to efficiently handle massive incoming data streams and support reasoning, data analytic services. Moreover, in an on-going industrial project on anomaly detection in large potable water networks, we are facing the effect of dynamically changing data and work characteristics in stream processing. The Strider system addresses these research and implementation challenges by considering scalability, fault-tolerance, high throughput and acceptable latency properties. We will demonstrate the benefits of Strider on an Internet of Things-based real world and industrial setting.