Data Streams Research Articles

A systematic review of machine learning and deep learning techniques for anomaly detection in data mining

The growing use of the internet has increased the threat of cyberattacks. Anomaly detection systems are vital for protecting networks by spotting irregular activities. Various studies investigated anomaly detection techniques without a systematic approach. So far, the existing reviews mainly concerned time series and data streaming methods almost neglected the growing interest in graph-based data mining techniques which are vital in social networks, finance, and IoT domains. Following PRISMA guidelines, this systematic review examines anomaly detection methods applied to time series, data streaming, and graph-based data from 2018 to 2023. A total of 34 papers were selected from four key databases ScienceDirect, Scopus, Web of Science, and IEEE. In addition, this review addressed several issues with existing techniques including low scalability, explainability, and interpretability for real-time anomaly detection systems. In modern applications where data structures are complex, and processing requirements are high these traditional techniques are insufficient for real-time data processing. Finally, our study demanded more advanced, complex methods to address these evolving challenges.

MEVDT: Multi-modal event-based vehicle detection and tracking dataset.

In this data article, we introduce the Multi-Modal Event-based Vehicle Detection and Tracking (MEVDT) dataset. This dataset provides a synchronized stream of event data and grayscale images of traffic scenes, captured using the Dynamic and Active-Pixel Vision Sensor (DAVIS) 240c hybrid event-based camera. MEVDT comprises 63 multi-modal sequences with approximately 13k images, 5M events, 10k object labels, and 85 unique object tracking trajectories. Additionally, MEVDT includes manually annotated ground truth labels - consisting of object classifications, pixel-precise bounding boxes, and unique object IDs - which are provided at a labeling frequency of 24 Hz. Designed to advance the research in the domain of event-based vision, MEVDT aims to address the critical need for high-quality, real-world annotated datasets that enable the development and evaluation of object detection and tracking algorithms in automotive environments.

A Semi-Supervised Active Learning Neural Network for Data Streams With Concept Drift

A Semi-Supervised Active Learning Neural Network for Data Streams With Concept Drift

Comparative Analysis of Machine Learning and Deep Learning Approaches in IoT-Enabled Enterprise Resource Planning Systems

The integration of Internet of Things (IoT) technologies into Enterprise Resource Planning (ERP) systems has revolutionized real-time decision-making, process automation, and operational efficiency. Machine Learning (ML) and Deep Learning (DL) provide two distinct approaches to leveraging IoT data within ERP systems, each with its own strengths and limitations. ML is well-suited for structured and semi-structured data, making it effective for tasks where the data format is consistent and manageable. However, it struggles with large-scale unstructured data typical of IoT environments, such as images and videos, and often requires manual, domain-specific feature extraction. In contrast, DL excels in processing unstructured and high-dimensional data streams, such as those from IoT sensors, by automatically extracting relevant features without human intervention. While ML is advantageous for simpler datasets and straightforward IoT applications, DL offers superior performance for complex, high-dimensional, and unstructured data, making it particularly beneficial in advanced IoTenabled ERP systems

Advancing public safety and housing solutions: A comprehensive framework for machine learning and predictive analytics in urban policy optimization

Urban areas worldwide face mounting challenges, including public safety concerns, housing shortages, and efficient resource allocation. Addressing these complexities requires innovative solutions that integrate technology and policy frameworks. This paper proposes an integrated framework leveraging machine learning (ML) and predictive analytics to optimize urban policy-making and provide actionable insights for sustainable urban development. The framework combines real-time data streams with advanced predictive modeling to enhance decision-making in urban governance. Key features include the ability to assess and respond to emerging trends in public safety by analyzing crime patterns, resource deployment, and community risk factors. Similarly, the framework addresses housing shortages by evaluating demand-supply dynamics, forecasting population growth, and identifying areas requiring immediate intervention. A critical review of existing models underscores the limitations of traditional urban policy approaches in managing socioeconomic disparities and environmental influences. Building upon these insights, the proposed framework integrates diverse data sources, such as demographic information, environmental metrics, and public sentiment analysis, to create a holistic view of urban systems. This multidimensional approach ensures that policy recommendations are context-sensitive, equitable, and inclusive. Moreover, the framework emphasizes stakeholder engagement, fostering collaboration among policymakers, urban planners, community leaders, and data scientists. It prioritizes scalability and adaptability, allowing its application across diverse urban contexts and geographic locations. This adaptability ensures that the framework remains relevant as urban environments evolve and new challenges arise. By addressing the interplay between data-driven insights and policy optimization, this research aims to bridge the gap between theoretical models and practical implementation. The framework holds the potential to redefine urban governance by enabling proactive decision-making, equitable resource allocation, and sustainable development. Future research should explore real-world applications of this framework to validate its efficacy and refine its components further.

Utilizing Raspberry Pi and Internet of Things (IoT) Frameworks for Comprehensive Monitoring of Urban Pollutants and Climate Variables

Global warming is the effect of a rise in Earth's climate temperature due to air pollution. Urban areas have higher levels of air pollution than rural areas due to increased traffic and rapid development. In addition to brief health problems like headaches, eye infections, and throat in-factions, pollution also has lengthy health repercussions like lung cancer and heart disease. As a result, it's important to keep an eye on the many climatic and pollution indicators, including light intensity, temperature, humidity, air pressure, oxygen and carbon dioxide level and camera. Using suitable equipment and Internet of Things (IoT) technologies, a pollution and climate monitoring system is constructed in the proposed paper that can measure the parameters above at regular intervals. The system then uploads the data to a webpage in the Thing Speak. The IoT analytics platform service Thing Speak enables the collection, visualization, and analysis of real-time data streams. The sensors are used to feed data to Thing Speak, which instantly visualizes the data in a graph. The main tool for gathering data from the sensors and sending parameters to the website is the Raspberry Pi3 computer. The software code is developed using Python. By logging on to the website from any location in the world, one can view the submitted pollution and climate parameters of a specific location.

Real-time human-AI collaboration through scalable cloud platforms for emergency response

Real-time human-AI collaboration is revolutionizing emergency response, yet challenges remain in achieving seamless interaction at scale. This article explores an innovative approach leveraging scalable cloud platforms to enable effective collaboration between human responders and AI systems during critical incidents. Integrating cloud-native solutions ensures real-time data processing, rapid decision support, and dynamic adaptation to evolving scenarios. Key features include adaptive load balancing to accommodate fluctuating data streams, AI-driven predictive analytics for preemptive action, and intelligent communication channels to enhance coordination among responders. The proposed architecture minimizes latency, optimizes resource allocation, and maintains service continuity, even under extreme conditions. The framework addresses data security, scalability, and compliance challenges to offer a robust, reliable solution for time-sensitive operations. Implementation results demonstrate significant improvements in response times, incident handling capacity, and resource utilization across multiple real-world emergency scenarios.

Redefining Event Detection and Information Dissemination: Lessons from X (Twitter) Data Streams and Beyond

Redefining Event Detection and Information Dissemination: Lessons from X (Twitter) Data Streams and Beyond

ISOUP-SymRF: Symbolic feature ranking with random forests in online multi-target regression and multi-label classification

The task of feature ranking has received considerable attention across various predictive modelling tasks in the batch learning scenario, but not in the online learning setting. Available methods that estimate feature importances on data streams have so far predominantly focused on ranking the features for the tasks of classification and occasionally multi-label classification. We propose a novel online feature ranking method for online multi-target regression iSOUP-SymRF, which estimates feature importance scores based on the positions at which a feature appears in the trees of a random forest of iSOUP-Trees, and additionally extend it to task of online feature ranking for multi-label classification. By utilizing iSOUP-Trees, which can address multiple structured output prediction tasks on data streams, iSOUP-SymRF promises feature ranking across a variety of online structured output prediction tasks. We examine the ranking convergence of iSOUP-SymRF in terms of the methods’ parameters, the size of the ensemble and the number of selected features, as well as their stability under different random seeds. Furthermore, to show the utility of iSOUP-SymRF and its rankings we use them in conjunction with two state-of-the-art online multi-target regression and multi-label classification methods, iSOUP-Tree and AMRules, and analyze the impact of adding features according to the rankings obtained from iSOUP-SymRF.

Pitfalls and practical suggestions for using local field potential recordings in DBS clinical practice and research.


Local field potential (LFP) recordings using chronically implanted sensing-enabled stimulators are a powerful tool for indexing symptom presence and severity in neurological and neuropsychiatric disorders, and for enhancing our neurophysiological understanding of brain processes. LFPs have gained interest as input signals for closed-loop deep brain stimulation (DBS) and can be used to inform DBS parameter selection. LFP recordings using chronically implanted sensing-enabled stimulators have various implementational challenges.
Approach: Here we describe our collective experience using BrainSense (Medtronic®) for clinical and research work. We aim to provide insightful tips and practical advice to empower readers with the knowledge needed to navigate the intricacies of the device and make the most out of its features.
Main results:
The central issues that apply to several BrainSense features encompass restricted compatibility of stimulation configuration with sensing, differences in electrophysiological signal properties between 'stimulation OFF' and 'stimulation ON at 0.0 mA', and challenges associated with the internal clock of the neurostimulator. In addition, since recordings are obtained from bipolar and not monopolar channels, spatial certainty regarding the distribution of LFPs around the DBS electrode is limited. Several options exist to synchronize LFP time series with external data streams, but standardization and generalization are lacking. The use of at-home chronic LFP recording is limited by a low temporal and spectral resolution. Regarding at-home LFP snapshots, LFP time series are not stored, parts of the power spectrum are censored when stimulating at high or low frequencies, and the stimulation amplitude is not readily available.
Significance: We discussed practical applications, implementation, system limitations, and pitfalls with the aim that sensing can be better applied for clinical practice and research.&#xD.

Angus: efficient active learning strategies for provenance based intrusion detection

As modern attack methods become more concealed and complex, obtaining many labeled samples in big data streams is difficult. Active learning has long been used to achieve better intrusion detection performance by using only a small number of training samples. Intrusion behaviors can be described by provenance graphs that record the dependency relationships between intrusion processes and the infected files. It is a challenge to develop active learning strategies that consider defining and selecting the most valuable provenance and ensure that the strategy for querying provenance is efficient. We present Angus, an active learning framework for provenance-based intrusion detection. We propose two novel active learning strategies: the most similar graph query strategy and the maximum difference query strategy. They either select samples to update the training set according to similarities of provenance graphs or preferentially select samples with low redundancy and large differences from the current training set. Besides, we also improve the above query strategies by using the parallel query to reduce detection time overheads. The experiments on various real-world applications demonstrate their performance and efficiency.

Multi-Label Learning with Distribution Matching Ensemble: An Adaptive and Just-In-Time Weighted Ensemble Learning Algorithm for Classifying a Nonstationary Online Multi-Label Data Stream

Learning from a nonstationary data stream is challenging, as a data stream is generally considered to be endless, and the learning model is required to be constantly amended for adapting the shifting data distributions. When it meets multi-label data, the challenge would be further intensified. In this study, an adaptive online weighted multi-label ensemble learning algorithm called MLDME (multi-label learning with distribution matching ensemble) is proposed. It simultaneously calculates both the feature matching level and label matching level between any one reserved data block and the new received data block, further providing an adaptive decision weight assignment for ensemble classifiers based on their distribution similarities. Specifically, MLDME abandons the most commonly used but not totally correct underlying hypothesis that in a data stream, each data block always has the most approximate distribution with that emerging after it; thus, MLDME could provide a just-in-time decision for the new received data block. In addition, to avoid an infinite extension of ensemble classifiers, we use a fixed-size buffer to store them and design three different dynamic classifier updating rules. Experimental results for nine synthetic and three real-world multi-label nonstationary data streams indicate that the proposed MLDME algorithm is superior to some popular and state-of-the-art online learning paradigms and algorithms, including two specifically designed ones for classifying a nonstationary multi-label data stream.

Efficient GPU-accelerated multisource global fit pipeline for LISA data analysis

The large-scale analysis task of deciphering gravitational-wave signals in the LISA data stream will be difficult, requiring a large amount of computational resources and extensive development of computational methods. Its high dimensionality, multiple model types, and complicated noise profile require a global fit to all parameters and input models simultaneously. In this work, we detail our global fit algorithm, called “Erebor,” designed to accomplish this challenging task. It is capable of analyzing current state-of-the-art datasets and then growing into the future as more pieces of the pipeline are completed and added. We describe our pipeline strategy, the algorithmic setup, and the results from our analysis of the LDC2A Sangria dataset, which contains massive black hole binaries, compact galactic binaries, and a parametrized noise spectrum whose parameters are unknown to the user. The Erebor algorithm includes three unique and very useful contributions: GPU acceleration for enhanced computational efficiency; ensemble Markov Chain Monte Carlo (MCMC) sampling with multiple MCMC walkers per temperature for better mixing and parallelized sample creation; and special online updates to reversible-jump (or transdimensional) sampling distributions to ensure sampler mixing and accurate initial estimates for detectable sources in the data. We recover posterior distributions for all 15 (6) of the injected massive black hole binaries (MBHB) in the LDC2A training (hidden) dataset. We catalog ∼12000 galactic binaries (∼8000 as high confidence detections) for both the training and hidden datasets. All of the sources and their posterior distributions are provided in publicly available catalogs. Published by the American Physical Society 2025

Exploring the Effectiveness of Continuous Glucose Monitoring in Improving Quality of Life and Reducing Hypoglycemic Episodes in Children with Type 1 Diabetes: A Narrative Review

Type 1 diabetes mellitus (T1DM) poses unique challenges in pediatric populations, requiring meticulous glycemic management to mitigate risks such as hypoglycemia and long-term complications. Continuous glucose monitoring (CGM) has emerged as a transformative tool, providing real-time data to support proactive diabetes management. This narrative review evaluated the effectiveness of CGM in improving quality of life (QoL) and reducing hypoglycemic episodes among children with T1DM. A comprehensive literature review methodology was employed to synthesize evidence on CGM’s impact, challenges, and future potential. CGM enhances glycemic control by providing continuous data streams and predictive alarms, addressing the limitations of traditional self-monitoring methods. Psychosocial benefits include reduced anxiety, greater independence, and improved communication between families and healthcare providers. However, barriers such as cost, adherence, and accessibility impede its widespread adoption. Emerging technologies, including artificial intelligence integration and advanced hybrid closed-loop systems, promise to overcome these challenges, further enhancing CGM’s usability and accuracy. Future research should explore CGM’s long-term outcomes, including its impact on psychological well-being and academic performance, while prioritizing equitable access. Addressing these factors will enable CGM to fulfill its potential as a cornerstone of pediatric diabetes management, significantly improving the lives of children with T1DM and their families. Keywords: Continuous Glucose Monitoring (CGM), Type 1 Diabetes Mellitus (T1DM), Pediatric Diabetes Management, Quality of Life (QoL), Hypoglycemic Episodes.

Stacking Ensemble Deep Learning for Real-Time Intrusion Detection in IoMT Environments

The Internet of Medical Things (IoMT) is revolutionizing healthcare by enabling advanced patient care through interconnected medical devices and systems. However, its critical role and sensitive data make it a prime target for cyber threats, requiring the implementation of effective security solutions. This paper presents a novel intrusion detection system (IDS) specifically designed for IoMT networks. The proposed IDS leverages machine learning (ML) and deep learning (DL) techniques, employing a stacking ensemble method to enhance detection accuracy by integrating the strengths of multiple classifiers. To ensure real-time performance, the IDS is implemented within a Kappa Architecture framework, enabling continuous processing of IoMT data streams. The system effectively detects and classifies a wide range of cyberattacks, including ARP spoofing, DoS, Smurf, and Port Scan, achieving an outstanding detection accuracy of 0.991 in binary classification and 0.993 in multi-class classification. This research highlights the potential of combining advanced ML and DL methods with ensemble learning to address the unique cybersecurity challenges of IoMT systems, providing a reliable and scalable solution for safeguarding healthcare services.

Cost-effective dynamic sampling in high dimensional online monitoring with advantage actor-critic

Industry 5.0 puts a spotlight on the team effort between people and machines, making the fast-paced development of IoT technology and sensor systems crucial. However, limitations such as the high cost of data acquisition, constrained bandwidth, and computational capacity can often hinder the real-time processing needed for human machine collaboration. To address this issue, we propose an integrated framework using Advantage Actor-Critic (A2C) and statistical process control (SPC) for economically monitoring high-dimensional data streams online. Our method leverages deep reinforcement learning with SPC charts to quickly identify system issues by smartly watching a few essential data streams. This approach not only saves resources but also boosts both the sustainability and efficiency of monitoring systems. Specifically, we construct a nonparametric monitoring statistic for each stream and develop a reinforcement learning framework to automatically identify the most informative and minimal number of data streams for observation at each time epoch. Our strategy is proven effective through simulations and a practical example in smart manufacturing, showcasing its superiority in reducing detection delay and minimising resources use compared to state-of-the-art algorithms.

Building Scalable Data Processing Systems with Kafka on Kubernetes

Real-time data processing has become a critical requirement for modern enterprises, presenting unique challenges in scalability, reliability, and performance. This technical article explores the synergy between Apache Kafka and Kubernetes in building robust, production-grade streaming architectures. This article presents a detailed examination of how Kafka's distributed streaming capabilities, combined with Kubernetes' container orchestration, create a resilient foundation for handling high-velocity data streams. Through practical examples from insurance claims processing and manufacturing IoT systems, we demonstrate proven patterns for deployment, scaling, and monitoring. The article provides deep technical insights into topic partitioning strategies, StatefulSet configurations, and operational best practices while addressing critical concerns around security, cost optimization, and disaster recovery. This article shows that this architectural approach can significantly reduce processing latency, handle unpredictable workload spikes, and maintain system reliability at scale, making it particularly valuable for enterprises dealing with real-time analytics and event-driven applications.

Moving from monitoring to real-time interventions for air quality: are low-cost sensor networks ready to support urban digital twins?

Modern cities now have an increasing multitude of Internet-of-Things data streams on urban phenomena, including transport, mobility, and meteorology. One area of development has been the use of low-cost sensors to complement (or in some cases, substitute for) regulatory monitoring of ambient air pollution. As part of a bigger integrated approach to monitoring cities, such as Urban Observatories, disparate live data streams can now readily be collated and disseminated via a platform to facilitate the use of hyperlocal data for real-time decision making whilst supporting longer term sustainable development goals. Urban digital twins are the next logical step on this journey and these are becoming increasingly popular as a tool, at least conceptually, to better interpret this data as well as better understand the consequences of management interventions. To date, there are few examples of true digital twins of environmental challenges with many limited to the ‘digital shadow’ stage of development, characterized by lack of bi-directional feedback between the digital model and physical world. Urban Observatories present an opportunity to change this by providing the often overlooked, but crucial, underpinning foundations of urban digital twins. This paper focuses on the utilization of live stream data and demonstrates that air quality applications can provide a realistic target given the density of observations available, which can routinely be combined with other urban datasets to provide the added value and insights needed for urban air pollution management. However, the availability and standardization of live streams of big data is a major challenge and there are issues with interoperability, metadata management, communicating uncertainty, network longevity, data ownership and transparency. This paper contributes insights concerning how to overcome these challenges and calls for common practice in generating and managing live streams of big data.

An enhanced Transformer framework with incremental learning for online stock price prediction.

To address the limitations of existing stock price prediction models in handling real-time data streams-such as poor scalability, declining predictive performance due to dynamic changes in data distribution, and difficulties in accurately forecasting non-stationary stock prices-this paper proposes an incremental learning-based enhanced Transformer framework (IL-ETransformer) for online stock price prediction. This method leverages a multi-head self-attention mechanism to deeply explore the complex temporal dependencies between stock prices and feature factors. Additionally, a continual normalization mechanism is employed to stabilize the data stream, enhancing the model's adaptability to dynamic changes. To ensure that the model retains prior knowledge while integrating new information, a time series elastic weight consolidation (TSEWC) algorithm is introduced to enable efficient incremental training with incoming data. Experiments conducted on five publicly available datasets demonstrate that the proposed method not only effectively captures the temporal information in the data but also fully exploits the correlations among multi-dimensional features, significantly improving stock price prediction accuracy. Notably, the method shows robust performance in coping with non-stationary and frequently changing financial market data.

Enhanced sensor web services by incorporating IoT interface protocols and spatio-temporal data streams for edge computing-based sensing

ABSTRACT The Geospatial Sensor Web (GSW) integrates heterogeneous aerial and ground sensors via cloud-edge linkages and GIS-based approaches, forming a multi-dimensional observation network. However, existing systems struggle to support edge-side collaborative observation due to fragmented physical standards, incompatible protocols, and limited self-configuration. This study proposes an enhanced Sensor Web, integrating IoT protocols and spatio-temporal models for unified access, collaborative management, and dynamic planning. Validated through the City Sensing Base Station (CSBS), a pilot experiment demonstrated the framework integrates diverse sensing resources across over eight protocols, achieving autonomous alignment of more than five platforms with rapid aerial-ground network formation during emergencies. It also validated autonomous collaboration and coordination of aerial-ground resources, enabling dynamic task allocation and execution across heterogeneous systems. Compared with cloud-based architectures, this approach significantly improves resource accessibility and real-time processing. By extending SensorML and Sensor Observation Service (SOS), the framework bridges the gap between conventional Sensor Webs and edge computing demands. Results confirm its effectiveness in coordinating heterogeneous resources and managing dynamic spatio-temporal data. These findings show how Internet of Things (IoT) protocols advance earth observation, modeling and improve GSW efficiency.