Real-time Data Stream Processing Research Articles

From CountMin to Super kJoin Sketches for Flow Spread Estimation

Real-time data stream processing is key to many Internet applications ranging from e-commerce, social networks, to network monitoring. Sketches (compact data structures) are often used to track large, high-rate streams and estimate their statistics. However, the traditional model for measuring flow size, e.g., number of packets in each flow, is unsuitable for more sophisticated applications that require flow spreads, i.e., number of distinct elements in each flow. When there are numerous flows, most existing work modifies CountMin sketches — which were designed to measure flow size — for measuring flow spread. They inherit a similar design and a commonly-used min operation from the CountMin sketches. This paper casts doubt on such a solution path, arguing that the inherited sketch design and the min operation were both intended for a different purpose and are not the best for flow spread measurement. Replacing the min operation with new position-aware operations, we propose the kJoin sketches that achieve much better average accuracy. We mathematically derive the estimation error bound. We also apply our sketches for network-wide measurement. Trace-driven experiments on software/GPU/FPGA platforms demonstrate our work significantly improves estimation accuracy, streaming rate (recording throughput), and query throughput, compared to the state-of-the-art.

Quantum-Accelerated Hyperparameter Tuning for Dynamic NLP Models

In the rapidly evolving field of natural language processing (NLP), performance optimization of large-scale NLP models is crucial. Through the application of Quantum-Accelerated Hyperparameter Tuning (QAHT), this abstract introduces a novel approach to addressing this issue. Our proposed framework leverages quantum computing capabilities to dynamically optimize NLP model hyperparameters in real-time, catering to the ever-changing character of textual data streams. Traditional hyperparameter optimisation methods usually rely on laborious grid searches or random exploration, which may not be suitable for dynamic NLP jobs. Contrarily, QAHT uses Quantum Neural Network (QNN) architectures that have been specially designed for hyperparameter optimisation. These QNNs improve performance and efficacy by dynamically modifying and improving model configurations. This abstract discusses the key elements of the QAHT architecture, including real-time model deployment, adaptive learning, and continuous data stream processing. In addition to speeding up the hyperparameter optimisation process, QAHT makes sure that NLP models are still flexible and responsive to shifts in the sentiment of the data and its distribution. This method has applications beyond NLP since it provides a foundation for effectively optimising machine learning models in complex, real-time situations. As quantum computing develops, QAHT represents a promising future in machine learning, where quantum-enhanced capabilities satisfy the needs of contemporary data-driven applications.

Multi-Source Information Fusion Technology and Its Application in Smart Distribution Power System

Compared to traditional measurement devices, the micro-synchrophasor measurement unit (D-PMU or μPMU) in the distribution power system has great differences in data acquisition frequency, data format, data dimension, time-stamped information, etc. Hence, it is imperative to research the integration mechanism of heterogeneous data from multiple sources. Based on the analysis of the current technology of multi-source information fusion, this paper proposes a novel approach, which considers two aspects: the interoperability of multi-source data and the real-time processing of large-scale streaming data. To solve the problem of data interoperability, we have modified the model of D-PMU data and established a unified information model. Meanwhile, an advanced distributed processing technology has been deployed to solve the problem of real-time processing of streaming data. Based on this approach, a smart distribution power system wide-area measurement and control station can be established, and the correctness and practicality of the proposed method are verified by an on-field project.

Real-Time Big Data Analytics for Data Stream Challenges: An Overview

The conventional approach of evaluating massive data is inappropriate for real-time analysis; therefore, analysing big data in a data stream remains a critical issue for numerous applications. It is critical in real-time big data analytics to process data at the point where they are arriving at a quick reaction and good decision making, necessitating the development of a novel architecture that allows for real-time processing at high speed and low latency. Processing and anlayzing a data stream in real-time is critical for a variety of applications; however, handling a large amount of data from a variety of sources, such as sensor networks, web traffic, social media, video streams, and other sources, is a considerable difficulty. The main goal of this paper is to give an overview of the current architecture for real time big data analytics, real-time data stream processing methods available, including their system architectures Lambda, kappa, and delta large data stream processing.

Remote Run-Time Failure Detection and Recovery Control For Quadcopters

We propose an adaptive run-time failure recovery control system for quadcopter drones, based on remote real-time processing of measurement data streams. Particularly, the measured RPM values of the quadcopter motors are transmitted to a remote machine which hosts failure detection algorithms and performs recovery procedure. The proposed control system consists of three distinct parts: (1) A set of computationally simple PID controllers locally onboard the drone, (2) a set of computationally more demanding remotely hosted algorithms for real-time drone state detection, and (3) a digital twin co-execution software platform — the ModelConductor-eXtended — for two-way signal data exchange between the former two. The local on-board control system is responsible for maneuvering the drone in all conditions: path tracking under normal operation and safe landing in a failure state. The remote control system, on the other hand, is responsible for detecting the state of the drone and communicating the corresponding control commands and controller parameters to the drone in real time. The proposed control system concept is demonstrated via simulations in which a drone is represented by the widely studied Quad-Sim six degrees-of-freedom Simulink model. Results show that the trained failure detection binary classifier achieves a high level of performance with F1-score of 96.03%. Additionally, time analysis shows that the proposed remote control system, with average execution time of 0.49 milliseconds and total latency of 6.92 milliseconds in two-way data communication link, meets the real-time constraints of the problem. The potential practical applications for the presented approach are in drone operation in complex environments such as factories (indoor) or forests (outdoor).

Enhancing dynamic energy network management using a multiagent cloud-fog structure

Smart Grid benefits from information and communications technology (ICT) to integrate data from different sources across the network. The growing market for electric vehicles (EV) and electric devices is increasing energy demand. To manage a large amount of complex intelligent equipment, EV charging and discharging, and data-intensive devices, a reliable modern smart grid with a service-oriented, secure, efficient, and cost-effective system is compelling. Cloud computing is a promising approach to achieve that objective as monitoring power grid data flow and data center management are the key. This paper proposes an optimization model for energy management within the power cloud. Our energy system model enables accessible services to computing resources and real-time data stream processing within an integrated environment. Using digital technology, the proper implementation of our model reduces costs and energy consumption and improves the smart grid reliability for customers and energy providers in a distributed manner. This paper presents the implementation of a three-procedure optimization algorithm within a novel Multi-Agent Cloud-fog Structure (MACS) to meet the requirements raised by smart grid communication and distribution. Our model promotes reliable energy consumption adjustments by end-users who can choose power supplied by solar, wind, geothermal, biomass, or other renewable sources or from non-renewable sources in ways that reveal opportunities for demand-supply balance and energy saving.

Steam++ An Extensible End-to-end Framework for Developing IoT Data Processing Applications in the Fog

IoT applications usually rely on cloud computing services to perform data analysis such as filtering, aggregation, classification, pattern detection, and prediction. When applied to specific domains, the IoT needs to deal with unique constraints. Besides the hostile environment such as vibration and electricmagnetic interference, resulting in malfunction, noise, and data loss, industrial plants often have Internet access restricted or unavailable, forcing us to design stand-alone fog and edge computing solutions. In this context, we present STEAM++, a lightweight and extensible framework for real-time data stream processing and decision-making in the network edge, targeting hardware-limited devices, besides proposing a micro-benchmark methodology for assessing embedded IoT applications. In real-case experiments in a semiconductor industry, we processed an entire data flow, from values sensing, processing and analysing data, detecting relevant events, and finally, publishing results to a dashboard. On average, the application consumed less than 500kb RAM and 1.0% of CPU usage, processing up to 239 data packets per second and reducing the output data size to 14% of the input raw data size when notifying events.

Design of Computer Multimedia Intelligent Platform Using Big Data Analysis

This exploration aims to transfer, process and store multimedia information timely, accurately and comprehensively through computer comprehensive technology processing, and organically combine various elements under the background of big data analysis, so as to form a complete intelligent platform design for multimedia information processing and application. In this exploration, the intelligent vehicle monitoring system is taken as an example. Data acquisition, data transmission, real-time data processing, data storage and data application are realized through the real-time data stream processing framework of [Formula: see text] of big data technology. Data interaction is realized through Spring, Spring MVC, VUE front-end framework, and Ajax asynchronous communication local update technology. Data storage is achieved through Red is cache database, and intelligent vehicle operation supervision system is achieved through multimedia information technology processing. Its purpose is to manage the vehicle information, real-time monitor the running state of the vehicle and give an alarm when there are some problems. The basic functions of vehicle operation monitoring and management system based on big data analysis are realized. The research on the design of vehicle operation monitoring and management system based on big data analysis shows that big data technology can be applied to the design of computer multimedia intelligent platform, and provides a reference case for the development of computer multimedia intelligent platform based on big data analysis.

A NOVEL TRUE REAL-TIME SPATIOTEMPORAL DATA STREAM PROCESSING FRAMEWORK

The ability to interpret spatiotemporal data streams in real-time is critical for a range of systems. However, processing vast amounts of spatiotemporal data out of several sources, such as online traffic, social platforms, sensor networks, and other sources, is a considerable challenge. The major goal of this study is to create a framework for processing and analyzing spatiotemporal data from multiple sources with irregular shapes so that researchers can focus on data analysis instead of worrying about the data sources' structure. We introduced a novel spatiotemporal data paradigm for true-real-time stream processing, which enables high-speed and low-latency real-time data processing, with these considerations in mind. A comparison of two state-of-the-art real-time process architectures was offered, as well as a full review of the various open-source technologies for real-time data stream processing, and their system topologies were also presented. Hence, this study proposed a brand-new framework that integrates Apache Kafka for spatiotemporal data ingestion, Apache flink for true real-time processing of spatiotemporal stream data, as well as machine learning for real-time predictions, and Apache Cassandra at the storage layer for distributed storage in real-time. The proposed framework was compared with others from the literature using the following features: Scalability (Sc), prediction tools (PT), data analytics (DA), multiple event types (MET), data storage (DS), Real-time (Rt), and performance evaluation (PE) stream processing (SP), and our proposed framework provided the ability to handle all of this task.

Remote Run-Time Failure Detection and Recovery Control For Quadcopters

We propose an adaptive run-time failure recovery control system for quadcopter drones, based on remote real-time processing of measurement data streams. Particularly, the measured RPM values of the quadcopter motors are transmitted to a remote machine which hosts failure detection algorithms and performs recovery procedure. The proposed control system consists of three distinct parts: (1) A set of computationally simple PID controllers locally onboard the drone, (2) a set of computationally more demanding remotely hosted algorithms for real-time drone state detection, and (3) a digital twin co-execution software platform — the ModelConductor-eXtended — for two-way signal data exchange between the former two. The local on-board control system is responsible for maneuvering the drone in all conditions: path tracking under normal operation and safe landing in a failure state. The remote control system, on the other hand, is responsible for detecting the state of the drone and communicating the corresponding control commands and controller parameters to the drone in real time. The proposed control system concept is demonstrated via simulations in which a drone is represented by the widely studied Quad-Sim six degrees-of-freedom Simulink model. Results show that the trained failure detection binary classifier achieves a high level of performance with F1-score of 96.03%. Additionally, time analysis shows that the proposed remote control system, with average execution time of 0.49 milliseconds and total latency of 6.92 milliseconds in two-way data communication link, meets the real-time constraints of the problem. The potential practical applications for the presented approach are in drone operation in complex environments such as factories (indoor) or forests (outdoor).

Incremental semi-supervised Extreme Learning Machine for Mixed data stream classification

With an explosive growth of data generated in the Internet and other fields, the data stream classification has sparked broad interest recently. Nowadays, some of the challenges in data streams, such as concept drift detection and supervised data stream classification, have been well-developed. However, when confronted with mixed data streams (containing categorical and numerical values) or limited available labeled samples, many data stream methods cannot achieve a satisfying performance or even cannot work. To tackle these two problems, we proposed an Incremental Semi-supervised Extreme Learning Machine for Mixed data stream classification (MIS-ELM). To be specific, for the issue of mixed data in data streams, we designed a novel soft one-hot encoding method by combining the coupling object similarity method and the one-hot encoding method, which can embed categorical data into high-quality numerical data and is used in the data preprocessing phase of MIS-ELM; for the issue of limited labeled samples, we introduced an incremental learning method based on unlabeled data, which is employed in the training classifier phase of MIS-ELM. When no concept drift occurs in the data stream, MIS-ELM uses only unlabeled data for incremental learning to fine-tune the classifier trained in the previous sliding window. Also, MIS-ELM instinctively inherits the fast computability of ELM, so it is very suitable for the real-time processing of data streams. Finally, we evaluated the representation performance of the soft one-hot encoding and the classification performance of MIS-ELM, within real data streams. The experimental results demonstrate the superiority of the proposed methods over the state-of-the-art techniques in their areas, respectively.

MEdit4CEP-SP: A model-driven solution to improve decision-making through user-friendly management and real-time processing of heterogeneous data streams

Organisations today are constantly consuming and processing huge amounts of data. Such datasets are often heterogeneous, making it difficult to work with them quickly and easily due to their format constraints or their disparate data structures. Therefore, being able to efficiently and intuitively work with such data to analyse them in real time to detect situations of interest as quickly as possible is a great competitive advantage for companies. Existing approaches have tried to address this issue by providing users with analytics or modelling tools in an isolated way, but not combining them as a one-in-all solution. In order to fill this gap, we present MEdit4CEP-SP, a model-driven system that integrates Stream Processing (SP) and Complex Event Processing (CEP) technologies for consuming, processing and analysing heterogeneous data in real time. It provides domain experts with a graphical editor that allows them to infer and define heterogeneous data domains, while also modelling, in a user-friendly way, the situations of interest to be detected in such domains. These graphical definitions are then automatically transformed into code, which is deployed in the processing system at runtime. The alerts detected by the system, in real-time, allow users to react as quickly as possible, thus improving the decision-making process. Additionally, MEdit4CEP-SP provides persistence, storing these definitions in a NoSQL database to permit their reuse by other instances of the system. Further benefits of this system are evaluated and compared with other existing approaches in this paper.

Real-Time Vehicle Traffic Prediction in Apache Spark Using Ensemble Learning for Deep Neural Networks

Escalating traffic congestion in large and rapidly evolving metropolitan areas all around the world is one of the inescapable problems in our daily lives. In light of this situation, traffic monitoring and analytics is becoming the need of the hour in today's world. Real-time traffic analysis requires processing of data streams that are being generated continuously in real time to gain quick insights. The challenge of analyzing streaming data for real-time prediction can be overcome by exploiting deep learning techniques. Taking this as a motivation, this work aims to integrate big data technologies and deep learning techniques to develop a real-time data stream processing model for vehicle traffic forecast using ensemble learning approach. Real-time traffic data from an API is streamed using a distributed streaming platform called Kafka into Apache Spark where it is processed, and the traffic flow is predicted by a neural network ensemble model. This will reduce the travel time, cost, and energy through efficient decision making, thus having a positive impact on the environment.

Data stream clustering: a review

Number of connected devices is steadily increasing and these devices continuously generate data streams. Real-time processing of data streams is arousing interest despite many challenges. Clustering is one of the most suitable methods for real-time data stream processing, because it can be applied with less prior information about the data and it does not need labeled instances. However, data stream clustering differs from traditional clustering in many aspects and it has several challenging issues. Here, we provide information regarding the concepts and common characteristics of data streams, such as concept drift, data structures for data streams, time window models and outlier detection. We comprehensively review recent data stream clustering algorithms and analyze them in terms of the base clustering technique, computational complexity and clustering accuracy. A comparison of these algorithms is given along with still open problems. We indicate popular data stream repositories and datasets, stream processing tools and platforms. Open problems about data stream clustering are also discussed.

Reаl-time heаrt diseаse detection аnd monitoring system bаsed on fаst mаchine leаrning using Spаrk

Heart disease is one of the most prominent and dangerous diseases that threaten public health around the world, often leading to heart attacks and strokes. More recently, the Internet of Things (IoT) based health monitoring system and many others continuously generate enormous amounts of data. With the heart disease patients under supervision have been increasing significantly, data generation speed is very high. In the scope of application of machine learning to the health data, there is a significant need to real-time streaming data processing to ensure an effective and scalable solution to effectively find and prevent the heart disease within a short and very specific timeline. As the amount of available data becomes highly big, the process of building and testing the traditional machine learning can be quite time consuming. The combination of streaming big data and machine learning is a breakthrough technology that can have a significant impact in the healthcare field especially real-time detection of heart disease. In this paper, we propose a new heart disease monitoring system based on a new classification approach consisting of the real-time distributed machine learning which uses the real-time predictive analysis algorithm in the Spark environment to predict heart disease. Firstly, we transform the traditional decision tree (C4.5) algorithm into parallel, distributed, scalable and fast decision tree. Secondly, this model is applied to streaming data coming from distributed sources to predict heart disease in real-time. The result as well as data streams will be stored in a distributed database for real-time reporting and monitoring. The results show better performance and have the potential to open new opportunities in distributed machine learning.

Storm Scheduling Based on Non-cooperative Game

With the continuous development of Internet technology, a large amount of data with rich value has been generated, and lagging data analysis will affect the timeliness of data, so real-time streaming data processing is becoming increasingly important. Storm is a pure streaming data processing framework, but it uses the polling scheduling algorithm by default. This algorithm ignores the network communication overhead between workers and cluster load balancing. Aiming at Storm’s default scheduling problem, a non-cooperative game-based Storm scheduling algorithm (G-Storm) was proposed. Storm extracts the source data in real time through the component “Spout”, passes it to the logical processing component “Bolt”, and finally loads it into the target warehouse. The experimental results show that the game scheduling algorithm proposed in this paper reduces the system processing delay by 28.6% compared with the default scheduling algorithm.

A Distance-Window Approach for the Continuous Processing of Spatial Data Streams

Real-time and continuous processing of citywide spatial data is an essential requirement of smart cities to guarantee the delivery of basic life necessities to its residents and to maintain law and order. To support real-time continuous processing of data streams, continuous queries (CQs) are used. CQs utilize windows to split the unbounded data streams into finite sets or windows. Existing stream processing engines either support time-based or count-based windows. However, these are not much useful for the spatial streams containing the trajectories of moving objects. Hence, this paper presents a distance-window based approach for the processing of spatial data streams, where the unbounded streams can be split with respect to the trajectory length. Since the window operation involves repeated computation, this work presents two incremental distance-based window approaches to avoid the repetition. A detailed experimental evaluation is presented to prove the effectiveness of the proposed incremental distance-based windows.

SBASH Stack Based Allocation of Sheer Window Architecture for Real Time Stream Data Processing

The processing of real-time data streams is complex with large number of volume and variety. The volume and variety of data streams enhances a number of processing units to run in real time. The required number of processing units used for processing data streams are lowered by using a windowing mechanism. Therefore, the appropriate size of window selection is vital for stream data processing. The coarse size window will directly affect the overall processing time. On the other hand, a finely sized window has to deal with an increased number of management costs. In order to manage such streams of data, we have proposed a SBASH architecture, which can be helpful for determining a unipartite size of a sheer window. The sheer window reduces the overall latency of data stream processing by a certain extent. The time complexity to process such sheer window is equivalent to w log n w. These windows are allocated and retrieved in a stack-based manner, where stacks ≥ n, which is helpful in reducing the number of comparisons made during retrieval.

An efficient algorithm for reducing the flow of real-time data stream with least sampling error

Nature of data stream is determined after complete scanning of whole data sets during real-time data processing. However, it becomes inconvenient to process entire data stream at once in real-time data stream processing. Thus, a sheer sized fixed window of data streams is processed at a particular time. The intensification of sheer sized fixed window at processing node is mitigated by reducing the flowing rate of data stream. Heuristic clustering windowing (HCW) approach and partial blind window (PBW) algorithms are proposed for reducing the flow of data stream with least sampling error. These approaches consist of the combination of systematic sampling and clustering mechanism. A clustering approach is applied on one fraction of data streams whereas systematic sampling handles other portion of streams. These approaches are helpful in reducing flow of data streams in minimum latency.

Real-Time Healthcare Monitoring System using Online Machine Learning and Spark Streaming

The real-time monitoring and tracking systems play a critical role in the healthcare field. Wearable medical devices with sensors, mobile applications, and health cloud have continuously generated an enormous amount of data, often called streaming big data. Due to the higher speed of the streaming data, it is difficult to ingest, process, and analyze such huge data in real-time to make real-time actions in case of emergencies. Using traditional methods that are inadequate and time-consuming. Therefore, there is a significant need for real-time big data stream processing to guarantee an effective and scalable solution. So, we proposed a new system for online prediction to predict health status using Spark streaming framework. The proposed system focuses on applying streaming machine learning models (i.e. streaming linear regression with SGD) on streaming health data events ingested to spark streaming through Kafka topics. The experimental results are done on the historical medical datasets (i.e. diabetes dataset, heart disease dataset, and breast cancer dataset) and generated dataset which is simulated to wearable medical sensors. The historical datasets have shown that the accuracy improvement ratio obtained using the diabetes disease dataset is the highest one with respect to the other two datasets with an accuracy of 81%. For generated datasets, the online prediction system has achieved accuracy with 98% at 5 seconds window size. Beyond this, the experimental results have proofed that the online prediction system can online learn and update the model according to the new data arrival and window size.