Real-time Stream Processing Research Articles

Using Online Algorithms to Solve NP-Hard Problems

This paper explores the application of online algorithms to tackle NP-hard problems, a class of computational challenges characterized by their intractability and wide-ranging real-world implications. Unlike traditional offline algorithms that have access to complete input data, online algorithms make decisions sequentially, often under constraints of incomplete information. We investigate various strategies, including greedy approaches, randomization, and competitive analysis, to assess their effectiveness in solving NP-hard problems such as the Traveling Salesman Problem, the Knapsack Problem, and the Set Cover Problem. Our analysis highlights the trade-offs between solution quality and computational efficiency, emphasizing the significance of the competitive ratio in evaluating algorithm performance. Additionally, we discuss the practical applications of online algorithms in dynamic environments, such as real-time systems and streaming data processing. Through a comprehensive review of existing literature and novel algorithmic designs, we aim to provide insights into the viability of online algorithms as a robust framework for addressing NP-hard problems in scenarios where immediate decision-making is crucial. The findings underscore the potential for future research to enhance these algorithms, making them increasingly applicable in complex, real-world contexts.

From CountMin to Super kJoin Sketches for Flow Spread Estimation

Real-time data stream processing is key to many Internet applications ranging from e-commerce, social networks, to network monitoring. Sketches (compact data structures) are often used to track large, high-rate streams and estimate their statistics. However, the traditional model for measuring flow size, e.g., number of packets in each flow, is unsuitable for more sophisticated applications that require flow spreads, i.e., number of distinct elements in each flow. When there are numerous flows, most existing work modifies CountMin sketches — which were designed to measure flow size — for measuring flow spread. They inherit a similar design and a commonly-used min operation from the CountMin sketches. This paper casts doubt on such a solution path, arguing that the inherited sketch design and the min operation were both intended for a different purpose and are not the best for flow spread measurement. Replacing the min operation with new position-aware operations, we propose the kJoin sketches that achieve much better average accuracy. We mathematically derive the estimation error bound. We also apply our sketches for network-wide measurement. Trace-driven experiments on software/GPU/FPGA platforms demonstrate our work significantly improves estimation accuracy, streaming rate (recording throughput), and query throughput, compared to the state-of-the-art.

Quantum-Accelerated Hyperparameter Tuning for Dynamic NLP Models

In the rapidly evolving field of natural language processing (NLP), performance optimization of large-scale NLP models is crucial. Through the application of Quantum-Accelerated Hyperparameter Tuning (QAHT), this abstract introduces a novel approach to addressing this issue. Our proposed framework leverages quantum computing capabilities to dynamically optimize NLP model hyperparameters in real-time, catering to the ever-changing character of textual data streams. Traditional hyperparameter optimisation methods usually rely on laborious grid searches or random exploration, which may not be suitable for dynamic NLP jobs. Contrarily, QAHT uses Quantum Neural Network (QNN) architectures that have been specially designed for hyperparameter optimisation. These QNNs improve performance and efficacy by dynamically modifying and improving model configurations. This abstract discusses the key elements of the QAHT architecture, including real-time model deployment, adaptive learning, and continuous data stream processing. In addition to speeding up the hyperparameter optimisation process, QAHT makes sure that NLP models are still flexible and responsive to shifts in the sentiment of the data and its distribution. This method has applications beyond NLP since it provides a foundation for effectively optimising machine learning models in complex, real-time situations. As quantum computing develops, QAHT represents a promising future in machine learning, where quantum-enhanced capabilities satisfy the needs of contemporary data-driven applications.

Analyzing and Applying Big Data Technologies Based on Industrial Internet Data

With the development of the industrial internet, data has gradually become the key engine driving the digital transformation of enterprises. The progress of the industrial internet is inseparable from the application of industrial data. To more effectively utilize data to empower businesses, many enterprises have put forward higher requirements for the real-time and reliability of industrial data collection. In the actual process of industrial data collection, issues such as large data volume, high real-time data processing requirements, and inconsistent data protocol standards are often encountered. In response to these problems, this paper, combined with practical engineering applications, proposes a big data construction technology solution based on industrial internet data processing. This solution aims to meet the high-concurrency data access needs of industrial equipment, using distributed messaging systems, high-throughput real-time stream processing frameworks, and distributed storage databases as the technical support for industrial data processing. This will help to complete the processing of massive industrial equipment data, thereby meeting the application requirements of various industrial big data processing business systems.

A Contemporary and Comprehensive Survey on Streaming Tensor Decomposition

Tensor decomposition has been demonstrated to be successful in a wide range of applications, from neuroscience and wireless communications to social networks. In an online setting, factorizing tensors derived from multidimensional data streams is however nontrivial due to several inherent problems of real-time stream processing. In recent years, many research efforts have been dedicated to developing online techniques for decomposing such tensors, resulting in significant advances in streaming tensor decomposition or tensor tracking. This topic is emerging and enriches the literature on tensor decomposition, particularly from the data stream analystics perspective. Thus, it is imperative to carry out an overview of tensor tracking to help researchers and practitioners understand its developments and achievements, summarise the current trends and advances, and identify challenging problems. In this article, we provide a contemporary and comprehensive survey on different types of tensor tracking techniques. We particularly categorize the state-of-the-art methods into three main groups: streaming CP decompositions, streaming Tucker decompositions, and streaming decompositions under other tensor formats (i.e., tensor-train, t-SVD, and BTD). In each group, we further divide the existing algorithms into sub-categories based on their main optimization framework and model architectures. Finally, we present several applications, research challenges, open problems, and potential directions of tensor tracking in the future.

Gigantic Data in Interruption Recognition Structures and Interruption Anticipation Structures

This document presents network attacks, intrusion detection systems, intrusion prevention systems, and intrusion detection methods, including signature-based detection and anomaly-based detection. Intrusion detection/prevention (ID/PS) system methods are compared. Some data mining and machine learning methods and their applications in intrusion detection are presented. Big data is introduced into intrusion detection systems, big data analysis for large volumes of data, heterogeneous resources, and real-time stream processing. It also discusses the challenges of intrusion detection systems and the challenges posed by processing huge streams of data in the systems.

Fog Computing-Based 5G LPWAN Anomaly Detection for Smart Cities

Known for excellent convenience and abundant facilities, smart cities offer CCTV, delivery robots, security robots, and so on to its residents. Along with the collaboration of IoT (Internet Of Things), the innovation of smart city has gained immense attraction at present. Besides, the risks and challenging in the field of telecommunication still persists as the implemented wireless networks results in traffic and anomaly behaviour. Such issues become critical in case of large-scale infrastructure networks like WSN’s. As such circumstances, to perform efficient health and environment monitoring, the need for a next generation networked system raises. As the traditional anomaly detection schemes doesn’t work out for delay-sensitive environments due to increased latency, we propose a scalable, hybrid spatiotemporal anomaly detection approach that can effectively detect potential anomalies in the network. With the use of real-time stream processing, and other methodologies like Software-Defined Networking (SDN), a Fog Computing-based 5G low-power Wide Area Network (LPWAN) solution is developed and tested on a Antwerp’s City of Things testbed. The proposed approach is found to be beneficial when deployed in a real network environment with nearly 1800 sensor nodes.

Real-Time Big Data Analytics for Data Stream Challenges: An Overview

The conventional approach of evaluating massive data is inappropriate for real-time analysis; therefore, analysing big data in a data stream remains a critical issue for numerous applications. It is critical in real-time big data analytics to process data at the point where they are arriving at a quick reaction and good decision making, necessitating the development of a novel architecture that allows for real-time processing at high speed and low latency. Processing and anlayzing a data stream in real-time is critical for a variety of applications; however, handling a large amount of data from a variety of sources, such as sensor networks, web traffic, social media, video streams, and other sources, is a considerable difficulty. The main goal of this paper is to give an overview of the current architecture for real time big data analytics, real-time data stream processing methods available, including their system architectures Lambda, kappa, and delta large data stream processing.

Sixth-Generation (6G) Mobile Cloud Security and Privacy Risks for AI System Using High-Performance Computing Implementation

The exchange of information from one person to another is called communication. Telecommunication makes it possible with electronic devices and their tools. The scientist Alexander Graham Bell has invented the basic telephone in 1876 in the USA. Telephones now have the new format in the form of mobile phones, which are the primary media for communicating and transmitting data. We are using 5th-generation mobile network standards. Still, there are some requirements for the users that are believed to be solved in the 6th-generation mobile network standards. By 2030, all of the people would be using 6G. The computing model in the cloud is not dependent on either the location or any specific device that would provide the service. It is an on-demand computational service-oriented mechanism. Combining these two technologies as mobile cloud computing provides customized options with more flexible implementations. Artificial intelligence is being used in devices in many fields. AI can be used in mobile network services (MNS) to provide more reliable and customized services to the users, such as network operation monitoring, network operation management, fraud detection, and reduction in mobile transactions and security to the cyber devices. Combining cloud with AI in mobile network services in the 6th generation would improve human beings’ lives, such as zero road accidents, advanced level special health care, and zero crime rates in society. However, the most vital needs for sixth-generation standards are the capability to manage large volumes of records and excessive-statistics-fee connectivity in step with gadgets. The sixth-generation mobile network is under development. This generation has many exciting features. Security is the central issue that needs to be sorted out using appropriate forensic mechanisms. There is a need to approach high-performance computing for improved services to the end-user. Considering three-dimensional research methodologies (technical dimension, organizational dimension, and applications hosted on the cloud) in a high-performance computing environment leads to two different cases such as real-time stream processing and remote desktop connection and performance test. By ‘narrowing the targeted worldwide audience with a wide range of experiential opportunities,’ this paper is aimed at delivering dynamic and varied resource allocation for reliable and justified on-demand services.

Enhancing dynamic energy network management using a multiagent cloud-fog structure

Smart Grid benefits from information and communications technology (ICT) to integrate data from different sources across the network. The growing market for electric vehicles (EV) and electric devices is increasing energy demand. To manage a large amount of complex intelligent equipment, EV charging and discharging, and data-intensive devices, a reliable modern smart grid with a service-oriented, secure, efficient, and cost-effective system is compelling. Cloud computing is a promising approach to achieve that objective as monitoring power grid data flow and data center management are the key. This paper proposes an optimization model for energy management within the power cloud. Our energy system model enables accessible services to computing resources and real-time data stream processing within an integrated environment. Using digital technology, the proper implementation of our model reduces costs and energy consumption and improves the smart grid reliability for customers and energy providers in a distributed manner. This paper presents the implementation of a three-procedure optimization algorithm within a novel Multi-Agent Cloud-fog Structure (MACS) to meet the requirements raised by smart grid communication and distribution. Our model promotes reliable energy consumption adjustments by end-users who can choose power supplied by solar, wind, geothermal, biomass, or other renewable sources or from non-renewable sources in ways that reveal opportunities for demand-supply balance and energy saving.

Steam++ An Extensible End-to-end Framework for Developing IoT Data Processing Applications in the Fog

IoT applications usually rely on cloud computing services to perform data analysis such as filtering, aggregation, classification, pattern detection, and prediction. When applied to specific domains, the IoT needs to deal with unique constraints. Besides the hostile environment such as vibration and electricmagnetic interference, resulting in malfunction, noise, and data loss, industrial plants often have Internet access restricted or unavailable, forcing us to design stand-alone fog and edge computing solutions. In this context, we present STEAM++, a lightweight and extensible framework for real-time data stream processing and decision-making in the network edge, targeting hardware-limited devices, besides proposing a micro-benchmark methodology for assessing embedded IoT applications. In real-case experiments in a semiconductor industry, we processed an entire data flow, from values sensing, processing and analysing data, detecting relevant events, and finally, publishing results to a dashboard. On average, the application consumed less than 500kb RAM and 1.0% of CPU usage, processing up to 239 data packets per second and reducing the output data size to 14% of the input raw data size when notifying events.

FJoin: an FPGA-based parallel accelerator for stream join

Stream join is widely used to extract key information between multi-source stream data and is an important supporting technology for big data processing. Join is easy to become a performance bottleneck because of the large-scale join predicate calculation when joining two big data streams. To improve performance, stream join systems often adopt parallel or distributed expansion methods. However, the multi-core parallel stream join system cannot cope with large-scale data streams because scalability is limited by the number of CPU cores. And the distributed extended stream join system introduces the overhead of distributed framework, resulting in a serious drop in hardware processing efficiency. To achieve efficient and large-scale expansion, this paper proposes a stream join system FJoin that uses the FPGA accelerator to scale up. FJoin can do High-Parallel Flow Join, in which data of the join window can flow through once to complete all join calculations after loading multiple stream tuples. For join predicates whose logic is easy to implement in FPGA, a large number of basic join units are connected in series to form a deep join pipeline to achieve large-scale parallelism. The host CPU and FPGA device coordinate control, divide the continuous stream join calculation into independent small-batch tasks and efficiently ensure completeness of parallel stream join. FJoin is implemented on a platform equipped with an FPGA accelerator card. The test results based on large-scale real data sets show that FJoin can increase the join calculation speed by 16 times using a single FPGA accelerator card and reach 5 times system throughput compared with the current best stream join system deployed on a 40-node cluster, and latency meets the real-time stream processing requirements.

Design of Computer Multimedia Intelligent Platform Using Big Data Analysis

This exploration aims to transfer, process and store multimedia information timely, accurately and comprehensively through computer comprehensive technology processing, and organically combine various elements under the background of big data analysis, so as to form a complete intelligent platform design for multimedia information processing and application. In this exploration, the intelligent vehicle monitoring system is taken as an example. Data acquisition, data transmission, real-time data processing, data storage and data application are realized through the real-time data stream processing framework of [Formula: see text] of big data technology. Data interaction is realized through Spring, Spring MVC, VUE front-end framework, and Ajax asynchronous communication local update technology. Data storage is achieved through Red is cache database, and intelligent vehicle operation supervision system is achieved through multimedia information technology processing. Its purpose is to manage the vehicle information, real-time monitor the running state of the vehicle and give an alarm when there are some problems. The basic functions of vehicle operation monitoring and management system based on big data analysis are realized. The research on the design of vehicle operation monitoring and management system based on big data analysis shows that big data technology can be applied to the design of computer multimedia intelligent platform, and provides a reference case for the development of computer multimedia intelligent platform based on big data analysis.

A NOVEL TRUE REAL-TIME SPATIOTEMPORAL DATA STREAM PROCESSING FRAMEWORK

The ability to interpret spatiotemporal data streams in real-time is critical for a range of systems. However, processing vast amounts of spatiotemporal data out of several sources, such as online traffic, social platforms, sensor networks, and other sources, is a considerable challenge. The major goal of this study is to create a framework for processing and analyzing spatiotemporal data from multiple sources with irregular shapes so that researchers can focus on data analysis instead of worrying about the data sources' structure. We introduced a novel spatiotemporal data paradigm for true-real-time stream processing, which enables high-speed and low-latency real-time data processing, with these considerations in mind. A comparison of two state-of-the-art real-time process architectures was offered, as well as a full review of the various open-source technologies for real-time data stream processing, and their system topologies were also presented. Hence, this study proposed a brand-new framework that integrates Apache Kafka for spatiotemporal data ingestion, Apache flink for true real-time processing of spatiotemporal stream data, as well as machine learning for real-time predictions, and Apache Cassandra at the storage layer for distributed storage in real-time. The proposed framework was compared with others from the literature using the following features: Scalability (Sc), prediction tools (PT), data analytics (DA), multiple event types (MET), data storage (DS), Real-time (Rt), and performance evaluation (PE) stream processing (SP), and our proposed framework provided the ability to handle all of this task.

A Dynamic Source Tracing Method for Food Supply Chain Quality and Safety Based on Big Data

The data of food quality tracing information have a few features, such as wide coverage range, many circulation links, complex data sources, low authenticity, and difficult information sharing. The continuous development of big data technology provides infinite possibilities for the construction of food quality source tracing systems. Currently, there are many studies on the application of food quality source tracing systems; however, most of them are in the field of food quality databases, and few have concerned about its application in the field of big data. Therefore, to fill in this research gap, this paper aimed to study a dynamic source tracing method for food supply chain quality and safety based on big data. At first, this paper summarized the variables of food supply chain quality and safety, constructed a Petri net model and a Bayesian network model for food quality prediction and source tracing, and realized the prediction of food quality features. Then, this paper applied two data analysis and processing methods—the density‐based clustering algorithm and the cosine similarity algorithm—to preliminarily process the collected quality tracing information of each link in the food supply chain and analyzed the influencing factors of food quality. Finally, experimental results proved the effectiveness of the constructed model. Relying on the real‐timeliness and authenticity of big data, this paper guarantees the credibility of the traceable information in the tracking process and improves the accuracy through real‐time stream processing of the updated data, providing unlimited possibilities for the comprehensive tracking of food sources.

A Task Execution Scheme for Dew Computing with State-of-the-Art Smartphones

The computing resources of today’s smartphones are underutilized most of the time. Using these resources could be highly beneficial in edge computing and fog computing contexts, for example, to support urban services for citizens. However, new challenges, especially regarding job scheduling, arise. Smartphones may form ad hoc networks, but individual devices highly differ in computational capabilities and (tolerable) energy usage. We take into account these particularities to validate a task execution scheme that relies on the computing power that clusters of mobile devices could provide. In this paper, we expand the study of several practical heuristics for job scheduling including execution scenarios with state-of-the-art smartphones. With the results of new simulated scenarios, we confirm previous findings and better comprehend the baseline approaches already proposed for the problem. This study also sheds some light on the capabilities of small-sized clusters comprising mid-range and low-end smartphones when the objective is to achieve real-time stream processing using Tensorflow object recognition models as edge jobs. Ultimately, we strive for industry applications to improve task scheduling for dew computing contexts. Heuristics such as ours plus supporting dew middleware could improve citizen participation by allowing a much wider use of dew computing resources, especially in urban contexts in order to help build smart cities.

Design and implementation of a cloud-based event-driven architecture for real-time data processing in wireless sensor networks

The growth of the Internet of Things (IoTs) and the number of connected devices is driven by emerging applications and business models. One common aim is to provide systems able to synchronize these devices, handle the big amount of daily generated data and meet business demands. This paper proposes a cost-effective cloud-based architecture using an event-driven backbone to process many applications’ data in real-time, called REDA. It supports the Amazon Web Service (AWS) IoT core, and it opens the door as a free software-based implementation. Measured data from several wireless sensor nodes are transmitted to the cloud running application through the lightweight publisher/subscriber messaging transport protocol, MQTT. The real-time stream processing platform, Apache Kafka, is used as a message broker to receive data from the producer and forward it to the correspondent consumer. Micro-services design patterns, as an event consumer, are implemented with Java spring and managed with Apache Maven to avoid the monolithic applications’ problem. The Apache Kafka cluster co-located with Zookeeper is deployed over three availability zones and optimized for high throughput and low latency. To guarantee no message loss and to simulate the system performances, different load tests are carried out. The proposed architecture is reliable in stress cases and can handle records goes to 8000 messages in a second with low latency in a cheap hosted and configured architecture.

Stochastic distributed data stream partitioning using task locality: design, implementation, and optimization

Distributed stream processing engines (DSPEs) provide stream partitioning methods for distributing messages to tasks deployed in the distributed environment for real-time stream processing. Among these methods, the original locality-aware stream partitioning (LSP) is a binary LSP that sends messages only to downstreams on the same node as upstreams. The binary LSP degrades performance at general configurations because it focuses only on task locality and does not consider downstream status like distributed batch processing engines. In this paper, we propose a Stochastic LSP (SLSP) method that considers not only task locality but also downstream status by computing stream partitioning probability based on the round-trip time to downstreams. We also present coarse-grained and fine-grained methods for probing downstreams at node-level and process-level, respectively. Then, we optimize our SLSP using a weighted closeness to prioritize the partitioning probabilities and a parallel thread model to process each stage of the SLSP in parallel. Finally, we implement the SLSP in Apache Storm, a representative DSPE, and empirically evaluate it with the binary LSP. Experimental results show that our SLSP greatly reduces latency by up to 208% while maintaining a similar throughput compared to the binary LSP at general configurations. These results indicate that our SLSP performs the optimized stream partitioning by reflecting downstream status as well as task locality.

An efficient approach for low latency processing in stream data.

Stream data is the data that is generated continuously from the different data sources and ideally defined as the data that has no discrete beginning or end. Processing the stream data is a part of big data analytics that aims at querying the continuously arriving data and extracting meaningful information from the stream. Although earlier processing of such stream was using batch analytics, nowadays there are applications like the stock market, patient monitoring, and traffic analysis which can cause a drastic difference in processing, if the output is generated in levels of hours and minutes. The primary goal of any real-time stream processing system is to process the stream data as soon as it arrives. Correspondingly, analytics of the stream data also needs consideration of surrounding dependent data. For example, stock market analytics results are often useless if we do not consider their associated or dependent parameters which affect the result. In a real-world application, these dependent stream data usually arrive from the distributed environment. Hence, the stream processing system has to be designed, which can deal with the delay in the arrival of such data from distributed sources. We have designed the stream processing model which can deal with all the possible latency and provide an end-to-end low latency system. We have performed the stock market prediction by considering affecting parameters, such as USD, OIL Price, and Gold Price with an equal arrival rate. We have calculated the Normalized Root Mean Square Error (NRMSE) which simplifies the comparison among models with different scales. A comparative analysis of the experiment presented in the report shows a significant improvement in the result when considering the affecting parameters. In this work, we have used the statistical approach to forecast the probability of possible data latency arrives from distributed sources. Moreover, we have performed preprocessing of stream data to ensure at-least-once delivery semantics. In the direction towards providing low latency in processing, we have also implemented exactly-once processing semantics. Extensive experiments have been performed with varying sizes of the window and data arrival rate. We have concluded that system latency can be reduced when the window size is equal to the data arrival rate.

Heterogeneity-aware elastic scaling of streaming applications on cloud platforms

Rise of Big Data techniques has led to the requirement for low latency analysis of high-velocity continuous data streams in real time. Several solutions, including Stream Processing Systems (SPSs), have been developed to enable real-time distributed stream processing. However, emerging application scenarios such as smart cities and wearable assistance that involve highly variable data rates keep on posing new challenges to the established stream processing engines for maintaining cost-effective executions. To cater to such scenarios, many modern SPSs have been proposed that leverage Cloud environment. The run-time scalability incorporated in these SPSs is in their early adaptations and are based on fixed scale sizes. Moreover, these scaling approaches do not adequately consider both the structure of the hosted streaming applications and the characteristic features of the underlying Cloud environment. Achieving true cost benefits of orchestrating streaming applications on Cloud-based pay-as-you-go model while maintaining the desired QoS, necessitates that both these issues are accounted in making the scaling decisions. This work presents a heterogeneity-aware, efficient auto-scaling strategy StreamScale-H which addresses both these issues. Simulation experiments, on representative stream applications, indicate that the proposed StreamScale-H auto-scaling algorithm exhibits much better performance in comparison with the state-of-the-art algorithms.