Data Streams Research Articles

Online Bayesian Variable Selection and Bayesian Model Averaging for Streaming Data

ABSTRACTThere is an increasing prevalence of streaming data generation in diverse fields like healthcare, finance, social media, and weather forecasting. In order to acquire helpful insights from these massive datasets, timely analysis is essential. In this article, we assume that the streaming data are analysed in batches. Traditional offline methods, which involve storing and analysing all individual records, can be repeatedly applied to the cumulative data, but encounter significant challenges in storage and computing costs. Existing online methods offer faster approximations but most methods neglect model uncertainty, causing overconfidence and instability. To bridge this gap, we propose novel online Bayesian approaches that incorporate model uncertainty within a Bayesian model averaging (BMA) framework, for generalized linear models (GLMs). We propose computationally efficient methods to update the posterior, with individual records from the latest batch of data and summary statistics from previous batches. We demonstrate using simulation studies and real data that our methods can offer much faster analysis compared to traditional methods, with no substantial drop in accuracy.

Scalable Anomaly Detection with Machine Learning: Techniques for Managing High-Dimensional Data Streams

The increase in big data values from industries, especially in Analytics, Risk, and Management Information systems, offers a great catchment and, at the same time, books with a lot of potential hurdles. This paper provides the conceptual basis for combining unsupervised and deep learning for real-time anomaly detection in high feature-space trajectories and offers practical applications for several industries. Autoencoders, Isolation Forests, and RNNs are tested for their performance and compared with the PCA in finance, manufacturing, healthcare, and cybersecurity applications. Other benefits are measured in specifics, such as cutting fraud detection mistakes by 25% and increasing the effectiveness of predictive maintenance. Further, the examined study focuses on problems like big data, real-time streaming data, distributed computing, edge computing, and incremental learning. Some key contributions are the scalable methods for IoT system-driven scenarios and the possibility of quickly recognizing anomalies with low latency. The paper also outlines emerging aspects of data quality, model interpretability, data privacy, and possible strategies such as Explainable AI and data masking. Examples from fraud detection and equipment failure prediction further explain the various opportunities these models offer for making decisions more effectively in complex and evolving environments.

Identification and explanation of disinformation in wiki data streams

Social media platforms, increasingly used as news sources for varied data analytics, have transformed how information is generated and disseminated. However, the unverified nature of this content raises concerns about trustworthiness and accuracy, potentially negatively impacting readers’ critical judgment due to disinformation. This work aims to contribute to the automatic data quality validation field, addressing the rapid growth of online content on wiki pages. Our scalable solution includes stream-based data processing with feature engineering, feature analysis and selection, stream-based classification, and real-time explanation of prediction outcomes. The explainability dashboard is designed for the general public, who may need more specialized knowledge to interpret the model’s prediction. Experimental results on two datasets attain approximately 90% values across all evaluation metrics, demonstrating robust and competitive performance compared to works in the literature. In summary, the system assists editors by reducing their effort and time in detecting disinformation.

A systematic review of machine learning and deep learning techniques for anomaly detection in data mining

The growing use of the internet has increased the threat of cyberattacks. Anomaly detection systems are vital for protecting networks by spotting irregular activities. Various studies investigated anomaly detection techniques without a systematic approach. So far, the existing reviews mainly concerned time series and data streaming methods almost neglected the growing interest in graph-based data mining techniques which are vital in social networks, finance, and IoT domains. Following PRISMA guidelines, this systematic review examines anomaly detection methods applied to time series, data streaming, and graph-based data from 2018 to 2023. A total of 34 papers were selected from four key databases ScienceDirect, Scopus, Web of Science, and IEEE. In addition, this review addressed several issues with existing techniques including low scalability, explainability, and interpretability for real-time anomaly detection systems. In modern applications where data structures are complex, and processing requirements are high these traditional techniques are insufficient for real-time data processing. Finally, our study demanded more advanced, complex methods to address these evolving challenges.

MEVDT: Multi-modal event-based vehicle detection and tracking dataset.

In this data article, we introduce the Multi-Modal Event-based Vehicle Detection and Tracking (MEVDT) dataset. This dataset provides a synchronized stream of event data and grayscale images of traffic scenes, captured using the Dynamic and Active-Pixel Vision Sensor (DAVIS) 240c hybrid event-based camera. MEVDT comprises 63 multi-modal sequences with approximately 13k images, 5M events, 10k object labels, and 85 unique object tracking trajectories. Additionally, MEVDT includes manually annotated ground truth labels - consisting of object classifications, pixel-precise bounding boxes, and unique object IDs - which are provided at a labeling frequency of 24 Hz. Designed to advance the research in the domain of event-based vision, MEVDT aims to address the critical need for high-quality, real-world annotated datasets that enable the development and evaluation of object detection and tracking algorithms in automotive environments.

Functional verification for Endcap Concentrator ASICs in the High-Granularity Calorimeter upgrade of CMS

The High-Granularity Calorimeter (HGCAL) will replace the current CMS Endcap Calorimeter during Long-Shutdown 3. The Endcap Concentrator (ECON) ASICs represent key elements in the readout chain, processing trigger (ECON-T) and data (ECON-D) streams from the HGCROC to the lpGBT. The ECONs will operate in a radiation environment with a High-Energy Hadron (E≥20 MeV) flux up to 2·107 cm-2s-1.This contribution describes the Universal Verification Methodology (UVM)-based functional verification of the ECON ASICs focusing on the re-use of existing components to manage the complexity of the verification environment.

An Iterative Shifting Disaggregation Algorithm for Multi-Source, Irregularly Sampled, and Overlapped Time Series

Accurate time series forecasting often requires higher temporal resolution than that provided by available data, such as when daily forecasts are needed from monthly data. Existing temporal disaggregation techniques, which typically handle only single, uniformly sampled time series, have limited applicability in real-world, multi-source scenarios. This paper introduces the Iterative Shifting Disaggregation (ISD) algorithm, designed to process and disaggregate time series derived from sensor-sourced low-frequency measurements, transforming multiple, nonuniformly sampled sensor data streams into a single, coherent high-frequency signal. ISD operates in an iterative, two-phase process: a prediction phase that uses multiple linear regression to generate high-frequency series from low-frequency data and correlated variables, followed by an update phase that redistributes low-frequency observations across high-frequency periods. This process repeats, refining estimates with each iteration cycle. The ISD algorithm’s key contribution is its ability to disaggregate multiple, nonuniformly spaced time series with overlapping intervals into a single daily representation. In two case studies using natural gas data, ISD successfully disaggregates billing cycle and grouped residential customer data into daily time series, achieving a 1.4–4.3% WMAPE improvement for billing cycle data and a 4.6–10.4% improvement for residential data over existing methods.

A Semi-Supervised Active Learning Neural Network for Data Streams With Concept Drift

A Semi-Supervised Active Learning Neural Network for Data Streams With Concept Drift

Comparative Analysis of Machine Learning and Deep Learning Approaches in IoT-Enabled Enterprise Resource Planning Systems

The integration of Internet of Things (IoT) technologies into Enterprise Resource Planning (ERP) systems has revolutionized real-time decision-making, process automation, and operational efficiency. Machine Learning (ML) and Deep Learning (DL) provide two distinct approaches to leveraging IoT data within ERP systems, each with its own strengths and limitations. ML is well-suited for structured and semi-structured data, making it effective for tasks where the data format is consistent and manageable. However, it struggles with large-scale unstructured data typical of IoT environments, such as images and videos, and often requires manual, domain-specific feature extraction. In contrast, DL excels in processing unstructured and high-dimensional data streams, such as those from IoT sensors, by automatically extracting relevant features without human intervention. While ML is advantageous for simpler datasets and straightforward IoT applications, DL offers superior performance for complex, high-dimensional, and unstructured data, making it particularly beneficial in advanced IoTenabled ERP systems

Stochastic feature selection with annealing and its applications to streaming data

Feature selection is an important topic in high-dimensional statistics and machine learning, for prediction and understanding the underlying phenomena. It has many applications in computer vision, natural language processing, bioinformatics, etc. However, most feature selection methods in the literature have been proposed for offline learning, and the existing online feature selection methods have theoretical and practical limitations in true support recovery. This paper proposes two novel online feature selection methods by stochastic gradient descent with a hard thresholding operator. The proposed methods can simultaneously select the relevant features and build linear regression or classification models based on the selected variables. The theoretical justification is provided for the consistency of the proposed methods. Numerical experiments on simulated and real sparse datasets show that the proposed methods compare favourably with state-of-the-art online methods from the literature.

Advancing public safety and housing solutions: A comprehensive framework for machine learning and predictive analytics in urban policy optimization

Urban areas worldwide face mounting challenges, including public safety concerns, housing shortages, and efficient resource allocation. Addressing these complexities requires innovative solutions that integrate technology and policy frameworks. This paper proposes an integrated framework leveraging machine learning (ML) and predictive analytics to optimize urban policy-making and provide actionable insights for sustainable urban development. The framework combines real-time data streams with advanced predictive modeling to enhance decision-making in urban governance. Key features include the ability to assess and respond to emerging trends in public safety by analyzing crime patterns, resource deployment, and community risk factors. Similarly, the framework addresses housing shortages by evaluating demand-supply dynamics, forecasting population growth, and identifying areas requiring immediate intervention. A critical review of existing models underscores the limitations of traditional urban policy approaches in managing socioeconomic disparities and environmental influences. Building upon these insights, the proposed framework integrates diverse data sources, such as demographic information, environmental metrics, and public sentiment analysis, to create a holistic view of urban systems. This multidimensional approach ensures that policy recommendations are context-sensitive, equitable, and inclusive. Moreover, the framework emphasizes stakeholder engagement, fostering collaboration among policymakers, urban planners, community leaders, and data scientists. It prioritizes scalability and adaptability, allowing its application across diverse urban contexts and geographic locations. This adaptability ensures that the framework remains relevant as urban environments evolve and new challenges arise. By addressing the interplay between data-driven insights and policy optimization, this research aims to bridge the gap between theoretical models and practical implementation. The framework holds the potential to redefine urban governance by enabling proactive decision-making, equitable resource allocation, and sustainable development. Future research should explore real-world applications of this framework to validate its efficacy and refine its components further.

Tools and techniques for real-time data processing: A review

Real-time data processing is an essential component in the modern data landscape, where vast amounts of data are generated continuously from various sources such as Internet of Things devices, social media, financial transactions, and manufacturing systems. Unlike traditional batch processing methods that analyse data in intervals, real-time data processing enables the continuous intake, manipulation, and analysis of data within milliseconds of generation. This capability is critical for applications requiring instant insights and rapid decision-making, including fraud detection, predictive maintenance, real-time analytics, and autonomous operations. This paper reviews the tools and techniques that have revolutionized real-time data processing, with a focus on cutting-edge platforms such as Apache Kafka and Apache Flink, as well as cloud-native solutions. These technologies offer scalable and fault-tolerant systems capable of managing high-volume data streams while ensuring low latency and data consistency. Apache Kafka provides a highly scalable distributed messaging system, while Apache Flink combines stateful and stateless processing to support complex event-driven applications. This review highlights the. This paper reviews key techniques and tools used in real-time data processing, including stream processing, complex event processing, in-memory computing, micro-batching, and real-time dashboards. In addition, it highlights advancements in real-time data processing frameworks, their capabilities, and their impact on modern business applications. Additionally, the paper explores various tools used in real-time data processing, including Apache Kafka for data ingestion, Apache Flink and Spark Streaming for stream processing, Redis and Apache Druid for real-time storage, and Grafana and Kibana for data visualization. By examining these techniques and tools, this paper highlights the importance of real-time data processing in enabling businesses to make data-driven decisions with minimal latency, ultimately gaining a competitive edge in the rapidly evolving digital world.

Utilizing Raspberry Pi and Internet of Things (IoT) Frameworks for Comprehensive Monitoring of Urban Pollutants and Climate Variables

Global warming is the effect of a rise in Earth's climate temperature due to air pollution. Urban areas have higher levels of air pollution than rural areas due to increased traffic and rapid development. In addition to brief health problems like headaches, eye infections, and throat in-factions, pollution also has lengthy health repercussions like lung cancer and heart disease. As a result, it's important to keep an eye on the many climatic and pollution indicators, including light intensity, temperature, humidity, air pressure, oxygen and carbon dioxide level and camera. Using suitable equipment and Internet of Things (IoT) technologies, a pollution and climate monitoring system is constructed in the proposed paper that can measure the parameters above at regular intervals. The system then uploads the data to a webpage in the Thing Speak. The IoT analytics platform service Thing Speak enables the collection, visualization, and analysis of real-time data streams. The sensors are used to feed data to Thing Speak, which instantly visualizes the data in a graph. The main tool for gathering data from the sensors and sending parameters to the website is the Raspberry Pi3 computer. The software code is developed using Python. By logging on to the website from any location in the world, one can view the submitted pollution and climate parameters of a specific location.

Handling high data loads in Kafka: Protecting hardware resources and ensuring system resilience

The exponential growth of data streams has made platforms like Apache Kafka indispensable for real-time data processing. However, managing high data loads while protecting hardware resources remains a challenge, especially in cloud environments. This paper presents a dynamic, cloud-native Kafka architecture that integrates rate-limiting, dynamic scaling, and data prioritization to prevent resource exhaustion and optimize system performance. This approach demonstrates significant improvements in message latency, lag clearance, and cost efficiency, offering a practical solution for modern data-intensive applications.

Real-time human-AI collaboration through scalable cloud platforms for emergency response

Real-time human-AI collaboration is revolutionizing emergency response, yet challenges remain in achieving seamless interaction at scale. This article explores an innovative approach leveraging scalable cloud platforms to enable effective collaboration between human responders and AI systems during critical incidents. Integrating cloud-native solutions ensures real-time data processing, rapid decision support, and dynamic adaptation to evolving scenarios. Key features include adaptive load balancing to accommodate fluctuating data streams, AI-driven predictive analytics for preemptive action, and intelligent communication channels to enhance coordination among responders. The proposed architecture minimizes latency, optimizes resource allocation, and maintains service continuity, even under extreme conditions. The framework addresses data security, scalability, and compliance challenges to offer a robust, reliable solution for time-sensitive operations. Implementation results demonstrate significant improvements in response times, incident handling capacity, and resource utilization across multiple real-world emergency scenarios.

Integrating Python and MATLAB for Digital Twin Applications in Aeroservoelasticity

The development of Digital Twin (DT) applications in intricate engineering areas has been greatly accelerated by the incorporation of sophisticated computational tools like Python and MATLAB. The goal of this research is to model, simulate, and analyze aeroservoelastic systems for Digital Twin implementations by utilizing the combined powers of Python and MATLAB. In order to achieve the real-time predictive skills needed for Digital Twins, aeroservoelasticity—which involves the coupled interplay of aerodynamic, structural, and control forces—presents special obstacles. This work successfully tackles these issues by fusing MATLAB's strong numerical and control system capabilities with Python's adaptability and library ecosystem. Using real-time sensor data streaming, and feedback control system implementation is used in this work. High-fidelity state-space modeling, sophisticated control design (such as PID and state feedback controllers), and system dynamics visualization are all done concurrently with MATLAB. In order to simulate and stabilize a highly flexible aeroelastic system, a continuous feedback loop was modeled in both environments, highlighting the platforms' complimentary advantages. Through the MATLAB Engine API for Python, the integrated method facilitates smooth data transmission between Python and MATLAB, guaranteeing real-time communication and model parameter synchronization. The efficiency of this dual-platform system is demonstrated by case studies on wing flutter suppression, limit cycle oscillation (LCO) mitigation, and control input optimization. The method is well suited for the dynamic needs of aeroservoelastic Digital Twins because of the results, which demonstrate a shorter error convergence time, improved stability, and computational economy. By utilizing MATLAB and Python's interoperability, this work establishes the groundwork for scalable, real-time Digital Twin solutions in aeronautical engineering. It highlights how crucial hybrid computational ecosystems are for bridging the gap between high- fidelity simulations and real-time predictive capabilities, opening the door for improvements in fault prediction, control optimization, and aeronautical system monitoring.

Partially Observable Online Nonparametric Monitoring of Spatiotemporally Correlated Data Streams

Internet of Things sensor networks collect real-time data, characterized by spatial and temporal correlations, for process monitoring, significantly altering daily life and enabling automation. Considering sensor resource constraints due to limited budget of sensor operation and the complexity of capturing spatiotemporal correlation structure among data streams, sensor networks face challenges in monitoring such data streams via a parametric model and distribution, particularly when only subsets of data are available at each acquisition time. This paper develops a nonparametric scheme for monitoring such complex spatiotemporally correlated and partially observed data streams. It employs decorrelated rank-based statistics combined with data augmentation over multiple subdata streams, which are derived from the original high-dimensional data using ensemble random projections for dimensionality reduction. Monitoring and sampling decisions are informed by aggregated local statistics of all subdata streams. This method is distribution-free that eschews parametric spatiotemporal models and distributions for real-time monitoring, enhancing its practical applicability to various complex spatiotemporal engineering cases that cannot be accurately characterized by parametric models. The efficacy of the decorrelated rank-based statistics and sampling strategy is substantiated through theoretical analysis. Numerical experiments and case studies focusing on thermal data monitoring in grain storage and solar flare detection affirm robust performance of the proposed method across various scenarios.

Using Hybrid LSTM Neural Networks to Detect Anomalies in the Fiber Tube Manufacturing Process

The production process of tubes for fiber optic cables is a complex process, where proper execution is crucial to the quality of the final product. This process has a complex state vector whose structure and coordinates dynamically change during the tube extrusion process. Small fluctuations in process parameters, such as temperature, extrusion pressure, production speed, and optical fiber tension, affect the optical attenuation of the final product. Such defects necessitate the withdrawal of the product. Due to the high number of process coordinates and the technological inability to automatically label those segments of the production process that cause anomalies in the final product, the authors used data clustering methods to create a training set that enabled the use of neural tools for anomaly detection. The system proposed in the main part of the paper includes a hybrid Long short-term memory (LSTM) network model, which is fed with data streams recorded on the tube extrusion production line. The input module, which performs preprocessing of input data, conducts multiresolution analysis of recorded process parameters, and recommends the process state’s belonging to a set of classes describing individual production anomalies to appropriate LSTM network modules. The learning process of the three–channel network allowed effective recognition of five classes of the monitored tube production process. The fit level of the proposed network model reached R2 values of ≥0.85.

Redefining Event Detection and Information Dissemination: Lessons from X (Twitter) Data Streams and Beyond

X (formerly known as Twitter), Reddit, and other social media forums have dramatically changed the way society interacts with live events in this day and age. The huge amount of data generated by these platforms presents challenges, especially in terms of processing speed and the complexity of finding meaningful patterns and events. These data streams are generated in multiple formats, with constant updating, and are real-time in nature; thus, they require sophisticated algorithms capable of dynamic event detection in this dynamic environment. Event detection techniques have recently achieved substantial development, but most research carried out so far evaluates only single methods, not comparing the overall performance of these methods across multiple platforms and types of data. With that view, this paper represents a deep investigation of complex state-of-the-art event detection algorithms specifically customized for streams of data from X. We review various current techniques based on a thorough comparative performance test and point to problems inherently related to the detection of patterns in high-velocity streams with noise. We introduce some novelty to this research area, supported by appropriate robust experimental frameworks, to performed comparisons quantitatively and qualitatively. We provide insight into how those algorithms perform under varying conditions by defining a set of clear, measurable metrics. Our findings contribute new knowledge that will help inform future research into the improvement of event detection systems for dynamic data streams and enhance their capabilities for real-time and actionable insights. This paper will go a step further than the present knowledge of event detection and discuss how algorithms can be adapted and refined in view of the emerging demands imposed by data streams.

Data Streaming with Apache Spark

In today’s fast-paced world, real-time data streaming is crucial for many businesses and industries. With tools like Apache Spark and Hadoop, organizations can process vast amounts of data as it’s created, gaining insights in seconds rather than hours. This article provides a comprehensive look at real-time streaming with Spark and Hadoop, including use cases, architecture, and step-by-step integration.