Data Stream Processing Research Articles

Stacking Ensemble Deep Learning for Real-Time Intrusion Detection in IoMT Environments

The Internet of Medical Things (IoMT) is revolutionizing healthcare by enabling advanced patient care through interconnected medical devices and systems. However, its critical role and sensitive data make it a prime target for cyber threats, requiring the implementation of effective security solutions. This paper presents a novel intrusion detection system (IDS) specifically designed for IoMT networks. The proposed IDS leverages machine learning (ML) and deep learning (DL) techniques, employing a stacking ensemble method to enhance detection accuracy by integrating the strengths of multiple classifiers. To ensure real-time performance, the IDS is implemented within a Kappa Architecture framework, enabling continuous processing of IoMT data streams. The system effectively detects and classifies a wide range of cyberattacks, including ARP spoofing, DoS, Smurf, and Port Scan, achieving an outstanding detection accuracy of 0.991 in binary classification and 0.993 in multi-class classification. This research highlights the potential of combining advanced ML and DL methods with ensemble learning to address the unique cybersecurity challenges of IoMT systems, providing a reliable and scalable solution for safeguarding healthcare services.

Stream Data Analysis and Processing Frameworks for Detecting Outliers in Human Activities: A Review

Stream Data Analysis and Processing Frameworks for Detecting Outliers in Human Activities: A Review

SYNCHRONIZATION OF EVENT-DRIVEN MANAGEMENT DURING DATA COLLECTION

The article considers an approach to implementing the architecture of a microservice system for processing large volumes of data based on the event-oriented approach to managing the sequence of using individual microservices. This becomes especially important when processing large volumes of data from information sources with different performance levels when the task is to minimize the total time for processing data streams. In this case, as a rule, the task is to minimize the number of requests for information sources to obtain a sufficient amount of data relevant to the request. The efficiency of the entire software system as a whole depends on how the microservices that provide extraction and primary processing of the received data are managed. To obtain the required amount of relevant data from diverse information sources, the software system must adapt to the request during its operation so that the maximum number of requests are directed to sources that have the maximum probability of finding the data necessary for the request in them. An approach is proposed that allows adaptively managing the choice of microservices during data collection and by emerging events and, thus, forming a choice of information sources based on an assessment of the efficiency of obtaining relevant information from these sources. Events are generated as a result of data extraction and primary processing from certain sources in terms of assessing the availability of data relevant to the request in each of the sources considered within the framework of the selected search scenario. Event-oriented microservice architecture adapts the system operation to the current loads on individual microservices and the overall performance by analyse the relevant events. The use of an adaptive event-oriented microservice architecture can be especially effective in the development of various information and analytical systems constructed by real-time data collection and design scenarios of analytical activity. The article considers the features of synchronous and asynchronous options in the implementation of event-oriented architecture, which can be used in various software systems depending on their purpose. An analysis of the features of synchronous and asynchronous options in the implementation of event-oriented architecture, their quantitative parameters, and features of their use depending on the type of tasks is carried out.

New Scalable and Efficient Online Pairwise Learning Algorithm.

Pairwise learning is an important machine-learning topic with many practical applications. An online algorithm is the first choice for processing streaming data and is preferred for handling large-scale pairwise learning problems. However, existing online pairwise learning algorithms are not scalable and efficient enough for large-scale high-dimensional data, because they were designed based on singly stochastic gradients. To address this challenging problem, in this article, we propose a dynamic doubly stochastic gradient algorithm (D2SG) for online pairwise learning. Especially, only the time and space complexities of O (d) are needed for incorporating a new sample, where d is the dimensionality of data. This means that our D2SG is much faster and more scalable than the existing online pairwise learning algorithms while the statistical accuracy can be guaranteed through our rigorous theoretical analysis under standard assumptions. The experimental results on a variety of real-world datasets not only confirm the theoretical result of our new D2SG algorithm, but also show that D2SG has better efficiency and scalability than the existing online pairwise learning algorithms.

A lightweight approach to real-time speaker diarization: from audio toward audio-visual data streams

This manuscript deals with the task of real-time speaker diarization (SD) for stream-wise data processing. Therefore, in contrast to most of the existing papers, it considers not only the accuracy but also the computational demands of individual investigated methods. We first propose a new lightweight scheme allowing us to perform speaker diarization of streamed audio data. Our approach utilizes a modified residual network with squeeze-and-excitation blocks (SE-ResNet-34) to extract speaker embeddings in an optimized way using cached buffers. These embeddings are subsequently used for voice activity detection (VAD) and block-online k-means clustering with a look-ahead mechanism. The described scheme yields results similar to the reference offline system while operating solely on a CPU with a low real-time factor (RTF) below 0.1 and a constant latency of around 5.5 s. In the next part of the work, our research moves toward much more demanding and complex real-time processing of audio-visual data streams. For this purpose, we extend the above-mentioned scheme for audio data processing by adding an audio-video module. This module utilizes SyncNet combined with visual embeddings for identity tracking. Our resulting multi-modal SD framework then combines the outputs from audio and audio-video modules by using a new overlap-based fusion strategy. It yields diarization error rates that are competitive with the existing state-of-the-art offline audio-visual methods while allowing us to process various audio-video streams, e.g., from Internet or TV broadcasts, in real-time using GPU and with the same latency as for audio data processing.

Data Processing of Used Car Export Service Flow Based on Zookeeper + Kafka

The used car export industry has shown a trend of rapid growth in recent years. The export of used cars involves data pertaining to vehicle information, market demand, logistics, customs, and numerous other aspects. These data are constantly generated in the form of streams, posing challenges to the data processing capabilities of service platforms. Traditional data processing methods struggle to meet the requirements of real-time and large-scale data processing, thus necessitating an efficient streaming data processing architecture. Based on this, this paper proposes a streaming data processing scheme for a used car export service platform utilizing Zookeeper and Kafka. It describes the architecture design and data processing flow, analyzes the advantages of the scheme in terms of reliability, scalability, and real-time performance, and demonstrates its effectiveness through experiments and case studies.

Real-Time Disease Prediction Using Big Data and Human-Computer Interaction

Big data streaming involves managing the vast volumes of data generated continuously by wearable medical devices with sensors, healthcare cloud platforms, and mobile applications. Traditional methods for processing this data are often time- and resource-intensive. To address this challenge, there is a need for efficient and scalable real-time big data stream processing. This study introduces a novel architecture for a big data-driven real-time health status prediction and analytics system. In this architecture, we replace Hadoop MapReduce with Spark to enable a parallel, distributed, and scalable decision tree algorithm capable of handling real-time computations. This model is then applied to streaming data from various sources, supporting the prediction of health statuses across multiple diseases. Using distributed streaming data, the system predicts health conditions associated with different disorders. To evaluate the performance, we compare Spark's decision tree (Spark DT) with traditional machine learning tools such as Weka. Key performance metrics, including execution time and throughput, are analyzed to assess the effectiveness of the proposed architecture. Experimental results demonstrate that the proposed system can effectively manage and predict vast amounts of real-time IoT-enabled medical data related to various disorders, showcasing its potential for real-time healthcare applications.

Novelty-aware concept drift detection for neural networks

Neural network models are widely adopted in real-world applications for processing streaming data. However, these applications often face challenges in terms of accuracy degradation, caused by changes in the data distribution of the stream data compared to the training data. Two underlying reasons contribute to these changes. The first, known as the concept drift problem, occurs when there is a change in the correlation between the input data and the prediction output, making the models trained on the training data no longer suitable for the new data. The second reason, known as the novelty problem, arises when real-world data contains unexpected data categories that were not present in the training data, resulting in incorrect predictions. The research community has divided into different groups and each developed various methods to detect either concept drift or novelty distribution changes. However, these methods only address one aspect of the problem and are unable to distinguish between them. This leads to an inappropriate allocation of model maintenance resources, including the high cost of model retraining and the acquisition of true label data. In this study, we aim to address this gap by proposing a novel concept drift detection method that is capable of distinguishing between known labeled concept drift and novelty. Our method is also more efficient than existing drift detection methods, making it suitable for applications on neural networks.

An Entropy-Based Clustering Algorithm for Real-Time High-Dimensional IoT Data Streams.

The rapid growth of data streams, propelled by the proliferation of sensors and Internet of Things (IoT) devices, presents significant challenges for real-time clustering of high-dimensional data. Traditional clustering algorithms struggle with high dimensionality, memory and time constraints, and adapting to dynamically evolving data. Existing dimensionality reduction methods often neglect feature ranking, leading to suboptimal clustering performance. To address these issues, we introduce E-Stream, a novel entropy-based clustering algorithm for high-dimensional data streams. E-Stream performs real-time feature ranking based on entropy within a sliding time window to identify the most informative features, which are then utilized with the DenStream algorithm for efficient clustering. We evaluated E-Stream using the NSL-KDD dataset, comparing it against DenStream, CluStream, and MR-Stream. The evaluation metrics included the average F-Measure, Jaccard Index, Fowlkes-Mallows Index, Purity, and Rand Index. The results show that E-Stream outperformed the baseline algorithms in both clustering accuracy and computational efficiency while effectively reducing dimensionality. E-Stream also demonstrated significantly less memory consumption and fewer computational requirements, highlighting its suitability for real-time processing of high-dimensional data streams. Despite its strengths, E-Stream requires manual parameter adjustment and assumes a consistent number of active features, which may limit its adaptability to diverse datasets. Future work will focus on developing a fully autonomous, parameter-free version of the algorithm, incorporating mechanisms to handle missing features and improving the management of evolving clusters to enhance robustness and adaptability in dynamic IoT environments.

Балансування ефективності та точності: поступове навчання як ключ до обробки великих даних

The article provides a comprehensive overview of incremental learning in the context of big data processing. The basic concepts, modern approaches, and key aspects of incremental learning are considered. The advantages of this approach for processing large amounts of data are analyzed, including the efficient use of computing resources, the ability to process streaming data in real time, and adaptability to changes in data. The main limitations and challenges, such as the problem of "catastrophic forgetting", the difficulty of balancing new and old knowledge, dependence on the order of data arrival, and potential loss of accuracy, are investigated. An analysis of specific problems is presented, including the handling of conceptual drift, unbalanced classes, and missing features. Applications of incremental learning in various fields, including data analytics, robotics, autonomous driving, and activity recognition, are discussed. We suggest directions for future research to address the identified problems and improve the effectiveness of incremental learning in the context of big data.

Benchmarking Big Data Systems: Performance and Decision-Making Implications in Emerging Technologies

Systems for graph processing are a key enabler for insights from large-scale graphs that are critical to many new advanced technologies such as Artificial Intelligence, Internet of Things, and blockchain. In this study, we benchmark another two widely utilized graph processing systems, Apache Spark GraphX and Apache Fink, concerning the key performance criterion by means of response time, scalability, and computational complexity. We demonstrate our results which show the capability of each system for real-world graph applications, and hence, providing a quantitative understanding to select the system for our purpose. GraphX’s strength was in processing batch in-memory workloads typical of blockchain and machine learning model optimization, while Flink excelled in processing stream data, which is timely and important to the IoT world. These performance characteristics emphasize how the capabilities of graph processing systems can match the requirements for the performance of different emerging technology applications. Our findings ultimately inform practitioners about system efficiencies and limitations, but also the recent advances in hardware accelerators and algorithmic improvements aimed at shaping the new graph processing frontier in diverse technology domains.

Stream ETL framework for twitter-based sentiment analysis: Leveraging big data technologies

Twitter has emerged as a rich source of real-time data, providing valuable insights across various domains such as healthcare and business. Sentiment analysis is crucial in understanding public reactions and sentiments expressed on Twitter, empowering organizations to make informed decisions. However, efficiently analyzing sentiment from social media data, presents a challenge for real-time streaming. Conventional Extract-Transform-Load (ETL) methods, inadequate for this challenge, limit their applicability in processing vast volumes of Twitter data. Big data technologies like Kafka, Spark, Hadoop HDFS, Hive, and HBase have become indispensable in addressing this challenge. To address this, we propose a stream ETL framework for Twitter-based sentiment analysis, leveraging Kafka, Spark, Cassandra, HBase, Hive, and HDFS. Our framework enables data stream processing, bias detection and correction, sentiment-based analysis, and visualization of tweets’ geospatial distribution. We present a set of use case studies to illustrate the applicability of the proposed framework comprises of sentiment classification, and bias detection and correction capability. We also present a comparative study to demonstrate the performance of data streaming processing, analysis, and visualization that has been implemented using multiple big data technologies under different parameter settings. Experimental results demonstrate the framework scalability and trade-off factors of the data stream processing pipeline in the execution of big data processing tasks.

Using Online Algorithms to Solve NP-Hard Problems

This paper explores the application of online algorithms to tackle NP-hard problems, a class of computational challenges characterized by their intractability and wide-ranging real-world implications. Unlike traditional offline algorithms that have access to complete input data, online algorithms make decisions sequentially, often under constraints of incomplete information. We investigate various strategies, including greedy approaches, randomization, and competitive analysis, to assess their effectiveness in solving NP-hard problems such as the Traveling Salesman Problem, the Knapsack Problem, and the Set Cover Problem. Our analysis highlights the trade-offs between solution quality and computational efficiency, emphasizing the significance of the competitive ratio in evaluating algorithm performance. Additionally, we discuss the practical applications of online algorithms in dynamic environments, such as real-time systems and streaming data processing. Through a comprehensive review of existing literature and novel algorithmic designs, we aim to provide insights into the viability of online algorithms as a robust framework for addressing NP-hard problems in scenarios where immediate decision-making is crucial. The findings underscore the potential for future research to enhance these algorithms, making them increasingly applicable in complex, real-world contexts.

Online parallel multi-task relationship learning via alternating direction method of multipliers

Online multi-task learning (OMTL) enhances streaming data processing by leveraging the inherent relations among multiple tasks. It can be described as an optimization problem in which a single loss function is defined for multiple tasks. Existing gradient-descent-based methods for this problem might suffer from gradient vanishing and poor conditioning issues. Furthermore, the centralized setting hinders their application to online parallel optimization, which is vital to big data analytics. Therefore, this study proposes a novel OMTL framework based on the alternating direction multiplier method (ADMM), a recent breakthrough in optimization suitable for the distributed computing environment because of its decomposable and easy-to-implement nature. The relations among multiple tasks are modeled dynamically to fit the constant changes in an online scenario. In a classical distributed computing architecture with a central server, the proposed OMTL algorithm with the ADMM optimizer outperforms SGD-based approaches in terms of accuracy and efficiency. Because the central server might become a bottleneck when the data scale grows, we further tailor the algorithm to a decentralized setting, so that each node can work by only exchanging information with local neighbors. Experimental results on a synthetic and several real-world datasets demonstrate the efficiency of our methods.

Send the important stuff back: An innovative approach to on-board stream processing for remotely deployed passive acoustic monitoring systems

Real-time monitoring of underwater soundscapes is necessary for rapid assessment of oceanic ecosystem health, controlling anthropogenic acoustic pollution, and establishing multi-scale spatiotemporal variability of the ocean acoustic environment. Such an assessment requires progression from existing regional and specialized passive acoustic monitoring (PAM) observational systems to a large-scale systematic passive acoustic monitoring of the World Ocean, complementary to the ARGO float Integrated Marine Observing System. This research develops an innovative approach to on-board stream processing of acoustic data on underwater autonomous mobile platforms using PyPAM, a Python-based package for calculating hybrid millidecade spectra and statistics. The design optimizes limited computational resources while preserving sufficient information to characterize the sampled soundscape’s noise levels and assemblage of sound sources. The association between frequency-temporal acoustic signatures of biophonic, geophonic, and anthropophonic sounds and corresponding features in percentiles of Spectral Probability Density have been established from PAM data collected at stationary receivers and test deployments of PAM-enabled underwater profilers. Financial support to Joseph Ross was provided by ASA SURIEA. Hardware development was supported by NPS CRUSER program. The PyPAM software is developed by Clea Parcerisas (LifeWatch). The research is conducted at NPS in collaboration with John Ryan (MBARI) and Josh Laney (Seatrec Inc.).

On metafeatures’ ability of implicit concept identification

Concept drift in data stream processing remains an intriguing challenge and states a popular research topic. Methods that actively process data streams usually employ drift detectors, whose performance is often based on monitoring the variability of different stream properties. This publication provides an overview and analysis of metafeatures variability describing data streams with concept drifts. Five experiments conducted on synthetic, semi-synthetic, and real-world data streams examine the ability of over 160 metafeatures from 9 categories to recognize concepts in non-stationary data streams. The work reveals the distinctions in the considered sources of streams and specifies 17 metafeatures with a high ability of concept identification.

Data processing in internet of things networks

One of the distinctive features of cloud applications in the Internet of Things (IoT) network is the real-time collection and processing of data about the objects of the physical environment. To solve this problem, with the help of sensors, data about the state of physical objects is collected to form the processing of data streams. The performance of such systems is critically dependent on the ability to securely and efficiently collect, transmit, and analyze data streams between environmental (real-world) objects and information systems. In such a case, it is relevant to study the data stream in Internet of Things networks and the operations that may occur on them. The issue of which platforms to process the operations to be carried out at the same time is the main subject of discussion.

Anomaly detection of power grid data based on storm cloud platform sliding window

Abstract To realize the monitoring and diagnosis of multi-source data in the power industry and meet the real-time processing requirements of the power system, the storm-distributed real-time computing platform is introduced to process the data. On this platform, the streaming data processing model based on a sliding window is deployed to realize the anomaly diagnosis of data flow. By calculating the predicted value of a point and the confidence interval of the predicted value of the point, we can judge whether the actual point is an abnormal value. By testing the data processing capacity of a cluster and a single machine, the experiment shows that in the cluster environment, reasonably setting the component parallelism can improve the throughput of flow computing.

Application of Event Cameras and Neuromorphic Computing to VSLAM: A Survey.

Simultaneous Localization and Mapping (SLAM) is a crucial function for most autonomous systems, allowing them to both navigate through and create maps of unfamiliar surroundings. Traditional Visual SLAM, also commonly known as VSLAM, relies on frame-based cameras and structured processing pipelines, which face challenges in dynamic or low-light environments. However, recent advancements in event camera technology and neuromorphic processing offer promising opportunities to overcome these limitations. Event cameras inspired by biological vision systems capture the scenes asynchronously, consuming minimal power but with higher temporal resolution. Neuromorphic processors, which are designed to mimic the parallel processing capabilities of the human brain, offer efficient computation for real-time data processing of event-based data streams. This paper provides a comprehensive overview of recent research efforts in integrating event cameras and neuromorphic processors into VSLAM systems. It discusses the principles behind event cameras and neuromorphic processors, highlighting their advantages over traditional sensing and processing methods. Furthermore, an in-depth survey was conducted on state-of-the-art approaches in event-based SLAM, including feature extraction, motion estimation, and map reconstruction techniques. Additionally, the integration of event cameras with neuromorphic processors, focusing on their synergistic benefits in terms of energy efficiency, robustness, and real-time performance, was explored. The paper also discusses the challenges and open research questions in this emerging field, such as sensor calibration, data fusion, and algorithmic development. Finally, the potential applications and future directions for event-based SLAM systems are outlined, ranging from robotics and autonomous vehicles to augmented reality.

Challenges and strategies of software engineering in big data environment

Software engineering in a big data environment involves developing and maintaining large-scale, scalable, data-intensive applications in a rapidly evolving technology domain. As data volumes grow exponentially, so do the requirements for software systems to process and analyze this data. These systems need to be able to efficiently store, retrieve, process and analyze data collected from sources such as social media, IoT devices, enterprises and scientific research. Software engineers must address challenges including data heterogeneity and quality, scalability of data storage and computation, and real-time processing of high-speed data streams. With the evolution of technology, new architecture and database technologies have emerged, such as NoSQL and distributed computing frameworks. These technologies have challenged traditional methods of software engineering practice and also brought new design and implementation strategies.