Real-time Data Streams Research Articles

ESARS: A Situation-Aware Multi-Agent System for Real-Time Emergency Response Management

This paper describes ESARS, a real-time situation-aware social media- enabled emergency situation alert and reporting system, as a decision support system built on multi-agent software design architecture for emergency situation management. The impact of an incident or disruption due to the incident could be minimized by implementing real-time intervention strategies that involve event monitoring, detection and situation identification via classification and prediction, notification, visualization and reporting that culminate in providing emergency support within time. The nature of agent behavior, which is autonomous, proactive and cooperative, makes them a suitable method for the design and deployment of a dynamic system of this nature. The system relies on historical and streamed real-time geolocation-enabled Twitter data stream for the target emergency events to provide decision-makers with dynamic, comprehensive, and timely information specific to the emergency situation.

Efficient topic partitioning of Apache Kafka for high-reliability real-time data streaming applications

Apache Kafka is a widely-used event streaming platform for reliable high-volume real-time data exchange following a producer–consumer pattern. Despite its popularity, Apache Kafka requires expertise and attention to detail, and there are no default guidelines that can be applied to all use cases without careful consideration. In this paper, we propose a novel approach to optimise the number of partitions and brokers in Apache Kafka, which are two key configuration parameters, under the given characteristics and constraints of the target applications. In particular, we consider the distribution of data-intensive real-time flows exchanged between a set of producers and consumers, which is representative of fog computing environments for ML/AI analytics. We introduce a methodology for modelling the topic partitioning process in Apache Kafka and formulate an optimisation problem to determine the optimal number of partitions to satisfy the application requirements and constraints. We propose two efficient heuristics to solve the optimisation problem, considering the trade-off between resource utilisation and application performance. We evaluate the performance of our approach through numerical simulations, and we demonstrate its practicality by implementing a prototype on an Apache Kafka cluster and conducting experiments in three different scenarios focused on mass consumption vs. production and real-time data streaming. To carry out repeatable experiments in controlled conditions, we developed a reusable framework that fully automatises cluster setup and performance assessment, and we make it available to the community as open-source software.

Streaming machine learning algorithms with streaming big data systems

As the era of big data unfolds, the need for real-time analytics and decision-making becomes increasingly crucial. Streaming big data systems, designed to process and analyse data in motion, have emerged as a pivotal solution for handling vast streams of continuously arriving information. This research delves into the synergy between streaming big data systems and machine learning algorithms, aiming to harness the power of real-time insights. We explore the challenges and opportunities presented by the dynamic nature of streaming data, emphasizing the importance of adapting traditional machine learning methodologies to suit the evolving requirements of streaming environments.The research begins with an overview of streaming big data systems, laying the foundation for understanding the unique characteristics of data in motion. We then delve into the selection and adaptation of machine learning algorithms that are well-suited for continuous learning and updating. Key aspects of the research include the preprocessing and feature extraction techniques tailored for real-time data streams, ensuring the effective utilization of streaming machine learning algorithms. The paper provides insights into the challenges of model training and updating in a dynamic environment, emphasizing the importance of accuracy and efficiency.

Smart Sensors for Real-time Monitoring in a Sustainable Logistics

The new logistics landscape needs openness, efficiency, and agility. Conventional tracking techniques are unable to keep up with the complex dance of items traveling across countries. Intelligent and communicative smart sensors have changed the game by allowing for real-time process monitoring and revolutionizing the logistics process from start to finish. This study explores how smart sensors can revolutionize the logistics industry. First, smart devices were study in-depth and their various features were equally examined, such as GPS tracking, RFID scanning, temperature and humidity sensors, package integrity assessment, vibration and acoustic sensors, and environmental monitoring. By conducting a comprehensive investigation, this research paper aims to scrutinize the data collecting ability of these sensors. The study will focus on their potential for gathering an abundance of information regarding cargo whereabouts, surrounding environment conditions, physical soundness, and equipment efficiency. This stream of real-time current data helps in active problem-solving, and imparts all the relevant insights one needs to know about the complete sustainable logistics operation. Their critical role toward improving productivity, transparency, and environmental sustainability in the dynamic logistics environment. For this reason, they can offer all perspectives needed (which are the most complete) to foster a proactive analysis and enhance any aspect over time.

Quantum-Accelerated Hyperparameter Tuning for Dynamic NLP Models

In the rapidly evolving field of natural language processing (NLP), performance optimization of large-scale NLP models is crucial. Through the application of Quantum-Accelerated Hyperparameter Tuning (QAHT), this abstract introduces a novel approach to addressing this issue. Our proposed framework leverages quantum computing capabilities to dynamically optimize NLP model hyperparameters in real-time, catering to the ever-changing character of textual data streams. Traditional hyperparameter optimisation methods usually rely on laborious grid searches or random exploration, which may not be suitable for dynamic NLP jobs. Contrarily, QAHT uses Quantum Neural Network (QNN) architectures that have been specially designed for hyperparameter optimisation. These QNNs improve performance and efficacy by dynamically modifying and improving model configurations. This abstract discusses the key elements of the QAHT architecture, including real-time model deployment, adaptive learning, and continuous data stream processing. In addition to speeding up the hyperparameter optimisation process, QAHT makes sure that NLP models are still flexible and responsive to shifts in the sentiment of the data and its distribution. This method has applications beyond NLP since it provides a foundation for effectively optimising machine learning models in complex, real-time situations. As quantum computing develops, QAHT represents a promising future in machine learning, where quantum-enhanced capabilities satisfy the needs of contemporary data-driven applications.

Local-global methods for generalised solar irradiance forecasting

For efficient operation, solar power operators often require generation forecasts for multiple sites with varying data availability. Many proposed methods for forecasting solar irradiance / solar power production formulate the problem as a time-series, using current observations to generate forecasts. This necessitates a real-time data stream and enough historical observations at every location for these methods to be deployed. In this paper, we propose the use of Global methods to train generalised models. Using data from 20 locations distributed throughout the UK, we show that it is possible to learn models without access to data for all locations, enabling them to generate forecasts for unseen locations. We show a single Global model trained on multiple locations can produce more consistent and accurate results across locations. Furthermore, by leveraging weather observations and measurements from other locations we show it is possible to create models capable of accurately forecasting irradiance at locations without any real-time data. We apply our approaches to both classical and state-of-the-art Machine Learning methods, including a Transformer architecture. We compare models using satellite imagery or point observations (temperature, pressure, etc.) as weather data. These methods could facilitate planning and optimisation for both newly deployed solar farms and domestic installations from the moment they come online.

Edge-Based Real-Time Sensor Data Processing for Anomaly Detection in Industrial IoT Applications

The Industrial Internet of Things (IIoT), which uses devices with sensors to provide real-time insights into crucial processes, has completely changed how industries function. However, there are many problems associated with the sheer volume and speed of data created in industrial environments, particularly when it comes to anomaly detection. The development of edge-based real-time sensor data processing techniques was required because traditional cloud-based solutions frequently experience latency problems and privacy issues. This study suggests a novel method for IIoT applications that focuses on processing sensor data at the edge, close to the data source, for anomaly identification. We offer real-time analysis of sensor data without the need for continuous data transfer to the cloud by utilising the processing capabilities of edge devices, such as industrial gateways and embedded systems. To find anomalies in streams of real-time sensor data, our methodology integrates data pre-processing, feature engineering, and machine learning algorithms. This strategy not only lessens the strain on the network's bandwidth but also ensures quick reaction to urgent situations, cutting downtime and boosting operational effectiveness. Proposed system has adaptive learning features that enable it to continuously adjust to altering ambient factors and sensor properties, enhancing the precision of anomaly detection over time. We provide experimental findings that show how our edge-based anomaly detection system performs well in diverse industrial situations. The results show that, while protecting data privacy and minimising latency, our methodology outperforms conventional cloud-based methods in terms of anomaly detection performance.

Proposal of a New Anonymization Algorithm for Real-Time Stream Data

Proposal of a New Anonymization Algorithm for Real-Time Stream Data

Publish/Subscribe Method for Real-Time Data Processing in Massive IoT Leveraging Blockchain for Secured Storage.

In the Internet of Things (IoT) era, the surge in Machine-Type Devices (MTDs) has introduced Massive IoT (MIoT), opening new horizons in the world of connected devices. However, such proliferation presents challenges, especially in storing and analyzing massive, heterogeneous data streams in real time. In order to manage Massive IoT data streams, we utilize analytical database software such as Apache Druid version 28.0.0 that excels in real-time data processing. Our approach relies on a publish/subscribe mechanism, where device-generated data are relayed to a dedicated broker, effectively functioning as a separate server. This broker enables any application to subscribe to the dataset, promoting a dynamic and responsive data ecosystem. At the core of our data transmission infrastructure lies Apache Kafka version 3.6.1, renowned for its exceptional data flow management performance. Kafka efficiently bridges the gap between MIoT sensors and brokers, enabling parallel clusters of brokers that lead to more scalability. In our pursuit of uninterrupted connectivity, we incorporate a fail-safe mechanism with two Software-Defined Radios (SDR) called Nutaq PicoLTE Release 1.5 within our model. This strategic redundancy enhances data transmission availability, safeguarding against connectivity disruptions. Furthermore, to enhance the data repository security, we utilize blockchain technology, specifically Hyperledger Fabric, known for its high-performance attributes, ensuring data integrity, immutability, and security. Our latency results demonstrate that our platform effectively reduces latency for 100,000 devices, qualifying as an MIoT, to less than 25 milliseconds. Furthermore, our findings on blockchain performance underscore our model as a secure platform, achieving over 800 Transactions Per Second in a dataset comprising 14,000 transactions, thereby demonstrating its high efficiency.

Real-time data stream learning for emergency decision-making under uncertainty

As one factor resulting in health emergency, climate change has received high attention in recent years. Real-time identifying and tracking the potential climate-related risk becomes more challenging due to the changeable and evolving data under uncertainty. To enhance risk early warning and decision-making, this paper proposes a self-adaptive machine learning framework based on real-time data stream learning, called C-SA-Stacking, for risk prediction. First, a stacking-based ensemble model for data stream learning is built with a meta-model embedded, which can self-update selectively and dynamically. Second, an online learning scheme is added to this framework for real-time adaptation. Finally, a mapping of correlations between data streams is constructed to further enhance the learning performance. By testing the framework on six synthetic datasets and six real-world datasets with different risk change scenarios and conducting comparative experimental analysis, the experimental results show that the prediction error is reduced as respected, indicating the effectiveness and real-time of the proposed framework for risk prediction under uncertainty. The results discussion reflects the framework can be applied to real-world scenarios to help identify and track the risk of climate change-caused health emergencies for decision-making.

Creating Serverless Web Applications with AWS

Abstract: Serverless computing has emerged as a groundbreaking innovation, allowing organizations to alleviate the complexities associated with provisioning, scaling, and managing infrastructure. The surge in serverless adoption poses challenges in choosing optimal solutions amid a myriad of proposed designs by experts. This paper categorizes serverless patterns using a multivocal literature review, highlighting benefits and challenges in coordination, aggregation, event management, availability, communication, and validation. To illustrate the practical application of serverless computing, we present a case study involving the use of AWS Lambda, Amazon DynamoDB, and other services to process real-time data streams for a fictional ride-sharing company. The study showcases the advantages of serverless computing, emphasizing its ability to free developers from server-related concerns, enabling them to focus on building scalable and reliable applications. This paper contributes to the understanding of serverless computing's role in the broader context of cloud technologies and its application in real-world scenarios

A Study on The Improvement of Information Loss Metrics in Real-Time Stream Data Anonymization

A Study on The Improvement of Information Loss Metrics in Real-Time Stream Data Anonymization

A Big Data System Architecture to Support the Monitoring of Paved Roads

Today, everything is connected, including the exchange of data and the generation of new information. As a result, large amounts of data are being collected at an ever-increasing rate and in a variety of forms, a phenomenon now known as Big Data. Recent developments in information and communication technologies are driving the generation of significant amounts of data from multiple sources, namely sensors. In response to these technological advances and data challenges, this paper proposes a Big Data system architecture for paved road monitoring and implements part of this architecture on a section of road in Portugal as a case study. The challenge in the case study architecture is to collect and process sensor data in real time, at a rate of 500 records per second, producing 15 GBytes of data per day, using a real-time data stream for real-time monitoring and a batch data stream for deeper analysis. This allows users to obtain instant updates on road conditions such as the number of vehicles, loads, weather, and pavement temperatures on the road. They can monitor what is happening on the road in real time, receive alerts, and even gain insight into historical data, such as analysing the condition of structures or identifying traffic patterns.

Active learning for data streams: a survey

Online active learning is a paradigm in machine learning that aims to select the most informative data points to label from a data stream. The problem of minimizing the cost associated with collecting labeled observations has gained a lot of attention in recent years, particularly in real-world applications where data is only available in an unlabeled form. Annotating each observation can be time-consuming and costly, making it difficult to obtain large amounts of labeled data. To overcome this issue, many active learning strategies have been proposed in the last decades, aiming to select the most informative observations for labeling in order to improve the performance of machine learning models. These approaches can be broadly divided into two categories: static pool-based and stream-based active learning. Pool-based active learning involves selecting a subset of observations from a closed pool of unlabeled data, and it has been the focus of many surveys and literature reviews. However, the growing availability of data streams has led to an increase in the number of approaches that focus on online active learning, which involves continuously selecting and labeling observations as they arrive in a stream. This work aims to provide an overview of the most recently proposed approaches for selecting the most informative observations from data streams in real time. We review the various techniques that have been proposed and discuss their strengths and limitations, as well as the challenges and opportunities that exist in this area of research.

Enhancing Cyber Security through Machine Learning-Based Anomaly Detection in IoT Networks

The rapid proliferation of IOT (Internet of Things) networks has brought transformative benefits to industries and everyday life. However, it has also introduced unprecedented cyber security challenges, necessitating advanced techniques for anomaly detection. This research focuses on enhancing cyber security through the application of machine learning-based anomaly detection methods, specifically One-Class Support Vector Machine (SVM) and Isolation Forest, in the context of IOT networks. While Isolation Forest effectively isolates anomalies by building isolation trees, One-Class SVM models the normal data distribution, effectively separating anomalies. To provide a strong security framework for IoT networks, we suggest a comprehensive strategy that combines both algorithms. Our method enables the detection of anomalies in real-time IOT data streams, facilitating prompt responses to new threats. Data collection, preprocessing, and model training are key components. This study helps protect IOT ecosystems and maintain data integrity and privacy in an increasingly connected world by utilizing the benefits of One-Class SVM and Isolation Forest.

Real-time motion-including dose estimation of simulated multi-leaf collimator-tracked magnetic resonance-guided radiotherapy.

Real-time dose estimation is a key-prerequisite to enable online intra-fraction treatment adaptation in magnetic resonance (MR)-guided radiotherapy (MRgRT). It is an essential component for the assessment of the dosimetric benefits and risks of online adaptive treatments, such as multi-leaf collimator (MLC)-tracking. We present a proof-of-concept for a software workflow for real-time dose estimation of MR-guided adaptive radiotherapy based on real-time data-streams of the linac delivery parameters and target positions. A software workflow, combining our in-house motion management software DynaTrack, a real-time dose calculation engine that connects to a research version of the treatment planning software (TPS) Monaco (v.6.09.00, Elekta AB, Stockholm, Sweden) was developed and evaluated. MR-guided treatment delivery on the Elekta Unity MR-linac was simulated with and without MLC-tracking for three prostate patients, previously treated on the Elekta Unity MR-linac (36.25Gy/five fractions). Three motion scenarios were used: no motion, regular motion, and erratic prostate motion. Accumulated monitor units (MUs), centre of mass target position and MLC-leaf positions, were forwarded from DynaTrack at a rate of 25Hz to a Monte Carlo (MC) based dose calculation engine which utilises the research GPUMCD-library (Elekta AB, Stockholm, Sweden). A rigid isocentre shift derived from the selected motion scenarios was applied to a bulk density-assigned session MR-image. The respective electron density used for treatment planning was accessed through the research Monaco TPS. The software workflow including the online dose reconstruction was validated against offline dose reconstructions. Our investigation showed that MC-based real-time dose calculations that account for all linac states (including MUs, MLC positions and target position) were infeasible, hence states were randomly sampled and used for calculation as follows; Once a new linac state was received, a dose calculation with 106 photons was started. Linac states that arrived during the time of the ongoing calculation were put into a queue. After completion of the ongoing calculation, one new linac state was randomly picked from the queue and assigned the MU accumulated from the previous state until the last sample in the queue. The queue was emptied, and the process repeated throughout treatment simulation. On average 27% (23%-30%) of received samples were used in the real-time calculation, corresponding to a calculation time for one linac state of 148ms. Median gamma pass rate (2%/3mm local) was 100.0% (99.9%-100%) within the PTV volume and 99.1% (90.1%-99.4.0%) with a 15% dose cut off. Differences in PTVDmean , CTVDmean , RectumD2% , and BladderD2% (offline-online, % of prescribed dose) were below 0.64%. Beam-by-beam comparisons showed deviations below 0.07Gy. Repeated simulations resulted in standard deviations below 0.31% and 0.12Gy for the investigated volume and dose criteria respectively. Real-time dose estimation was successfully performed using the developed software workflow for different prostate motion traces with and without MLC-tracking. Negligible dosimetric differences were seen when comparing online and offline reconstructed dose, enabling online intra-fraction treatment decisions based on estimates of the delivered dose.

Secure And Adaptive Data Stream Mining For New Generation Big Data

As the era of new-generation big data applications unfolds, the need for secure and adaptive data stream mining has become increasingly paramount. Evolving databases, characterized by ever-changing data streams and dynamic data distributions, present unique challenges and opportunities. This paper addresses the crucial intersection of security and adaptability in the context of data stream mining for new-generation big data. First, we delve into the evolving landscape of big data, where real-time data streams from diverse sources drive decision-making processes. Ensuring the privacy and security of sensitive information within these data streams is a fundamental concern. We explore cryptographic techniques, anonymization methods, and access control mechanisms that safeguard data while allowing for meaningful analysis. We present novel adaptive algorithms and model update strategies that can continuously learn and adjust to changing data distributions. These approaches enable data stream mining to remain effective and accurate over time. This paper offers insights into the fusion of security and adaptability in data stream mining, providing a foundation for the development of robust and privacy-conscious solutions for the evolving landscape of new-generation big data applications.

Enhancing Data Engineering Frameworks for Scalable Real-Time Marketing Solutions

This report discusses improvements to the data engineering frameworks for cost-efficient real-time marketing environment, with the applications in mobile advertising and smart city analytics. The proposed framework combines the processing of time-series data with map-reduce (M-R) systems in order to compensate for inefficiencies in the execution of temporal queries and the processing of real-time data streams. It enables effective behavioral targeting in mobile marketing and helps integrate various domains’ real-time data for smart city utilization. Specific examples that illustrate further refinements in the algorithms are showcased through case study and experimental evidence where gains in throughput, data reliability, and system effectiveness are quantified in real-time Marketing and Smart City applications.

A Dynamic Drilling Sampling Method and Evaluation Model for Big Streaming Data

The big data sampling method for real-time and high-speed streaming data is prone to lose the value and information of a large amount of discrete data, and it is not easy to make an efficient and accurate evaluation of the value characteristics of streaming data. The SDSLA sampling method based on mineral drilling exploration can evaluate the valuable information of streaming data containing many discrete data in real-time, but when the range of discrete data is irregular, it has low sampling accuracy for discrete data. Based on the SDSLA algorithm, we propose a dynamic drilling sampling method SDDS, which takes well as the analysis unit, dynamically changes the size and position of the well, and accurately locates the position and range of discrete data. A new model SDVEM is further proposed for data valuation, which evaluates the sample set from discrete, centralized, and overall dimensions. Experiments show that compared with the SDSLA algorithm, the sample sampled by the SDDS algorithm has higher evaluation accuracy, and the probability distribution of the sample is closer to the original streaming data, with the AOCV indicator being nearly 10% higher. In addition, the SDDS algorithm can achieve over 90% accuracy, recall, and F1 score for training and testing neural networks with small sampling rates, all of which are higher than the SDSLA algorithm. In summary, the SDDS algorithm not only accurately evaluates the value characteristics of streaming data but also facilitates the training of neural network models, which has important research significance in big data estimation.

Learning Ship Activity Patterns in Maritime Data Streams: Enhancing CEP Rule Learning by Temporal and Spatial Relations and Domain-Specific Functions

Maritime surveillance systems are of particular importance for the smooth and safe operation of maritime traffic. Such systems must efficiently analyze the continuous stream of incoming ship movement data to provide an up-to-date picture of the situation at sea. Complex Event Processing (CEP) is a software technology dedicated to the analysis of data streams in real time and seems to be promising for maritime surveillance as well. CEP is based on rules that usually have to be formulated manually by domain experts. Recently, several approaches of learning CEP rules have been proposed. However, all these approaches are limited to standard CEP rule languages which are not well suited for describing maritime situations. In this paper, we extend Bat4CEP, an innovative bat-inspired approach to automatic CEP rule learning, to meet the specific needs of the maritime domain. In particular, we extend the rule language of Bat4CEP to include temporal-spatial operators and domain-specific functions. This extended rule language makes it possible to express complex maritime facts in a simple and understandable form. Furthermore, it is shown how the proposed rule language extensions can be integrated into the Bat4CEP learning approach. The effectiveness of the approach is demonstrated through extensive experiments with real maritime data streams.