Data Stream Processing Research Articles

Benchmarking Big Data Systems: Performance and Decision-Making Implications in Emerging Technologies

Systems for graph processing are a key enabler for insights from large-scale graphs that are critical to many new advanced technologies such as Artificial Intelligence, Internet of Things, and blockchain. In this study, we benchmark another two widely utilized graph processing systems, Apache Spark GraphX and Apache Fink, concerning the key performance criterion by means of response time, scalability, and computational complexity. We demonstrate our results which show the capability of each system for real-world graph applications, and hence, providing a quantitative understanding to select the system for our purpose. GraphX’s strength was in processing batch in-memory workloads typical of blockchain and machine learning model optimization, while Flink excelled in processing stream data, which is timely and important to the IoT world. These performance characteristics emphasize how the capabilities of graph processing systems can match the requirements for the performance of different emerging technology applications. Our findings ultimately inform practitioners about system efficiencies and limitations, but also the recent advances in hardware accelerators and algorithmic improvements aimed at shaping the new graph processing frontier in diverse technology domains.

Using Online Algorithms to Solve NP-Hard Problems

This paper explores the application of online algorithms to tackle NP-hard problems, a class of computational challenges characterized by their intractability and wide-ranging real-world implications. Unlike traditional offline algorithms that have access to complete input data, online algorithms make decisions sequentially, often under constraints of incomplete information. We investigate various strategies, including greedy approaches, randomization, and competitive analysis, to assess their effectiveness in solving NP-hard problems such as the Traveling Salesman Problem, the Knapsack Problem, and the Set Cover Problem. Our analysis highlights the trade-offs between solution quality and computational efficiency, emphasizing the significance of the competitive ratio in evaluating algorithm performance. Additionally, we discuss the practical applications of online algorithms in dynamic environments, such as real-time systems and streaming data processing. Through a comprehensive review of existing literature and novel algorithmic designs, we aim to provide insights into the viability of online algorithms as a robust framework for addressing NP-hard problems in scenarios where immediate decision-making is crucial. The findings underscore the potential for future research to enhance these algorithms, making them increasingly applicable in complex, real-world contexts.

Online parallel multi-task relationship learning via alternating direction method of multipliers

Online multi-task learning (OMTL) enhances streaming data processing by leveraging the inherent relations among multiple tasks. It can be described as an optimization problem in which a single loss function is defined for multiple tasks. Existing gradient-descent-based methods for this problem might suffer from gradient vanishing and poor conditioning issues. Furthermore, the centralized setting hinders their application to online parallel optimization, which is vital to big data analytics. Therefore, this study proposes a novel OMTL framework based on the alternating direction multiplier method (ADMM), a recent breakthrough in optimization suitable for the distributed computing environment because of its decomposable and easy-to-implement nature. The relations among multiple tasks are modeled dynamically to fit the constant changes in an online scenario. In a classical distributed computing architecture with a central server, the proposed OMTL algorithm with the ADMM optimizer outperforms SGD-based approaches in terms of accuracy and efficiency. Because the central server might become a bottleneck when the data scale grows, we further tailor the algorithm to a decentralized setting, so that each node can work by only exchanging information with local neighbors. Experimental results on a synthetic and several real-world datasets demonstrate the efficiency of our methods.

Stream ETL framework for twitter-based sentiment analysis: Leveraging big data technologies

Twitter has emerged as a rich source of real-time data, providing valuable insights across various domains such as healthcare and business. Sentiment analysis is crucial in understanding public reactions and sentiments expressed on Twitter, empowering organizations to make informed decisions. However, efficiently analyzing sentiment from social media data, presents a challenge for real-time streaming. Conventional Extract-Transform-Load (ETL) methods, inadequate for this challenge, limit their applicability in processing vast volumes of Twitter data. Big data technologies like Kafka, Spark, Hadoop HDFS, Hive, and HBase have become indispensable in addressing this challenge. To address this, we propose a stream ETL framework for Twitter-based sentiment analysis, leveraging Kafka, Spark, Cassandra, HBase, Hive, and HDFS. Our framework enables data stream processing, bias detection and correction, sentiment-based analysis, and visualization of tweets’ geospatial distribution. We present a set of use case studies to illustrate the applicability of the proposed framework comprises of sentiment classification, and bias detection and correction capability. We also present a comparative study to demonstrate the performance of data streaming processing, analysis, and visualization that has been implemented using multiple big data technologies under different parameter settings. Experimental results demonstrate the framework scalability and trade-off factors of the data stream processing pipeline in the execution of big data processing tasks.

On metafeatures’ ability of implicit concept identification

Concept drift in data stream processing remains an intriguing challenge and states a popular research topic. Methods that actively process data streams usually employ drift detectors, whose performance is often based on monitoring the variability of different stream properties. This publication provides an overview and analysis of metafeatures variability describing data streams with concept drifts. Five experiments conducted on synthetic, semi-synthetic, and real-world data streams examine the ability of over 160 metafeatures from 9 categories to recognize concepts in non-stationary data streams. The work reveals the distinctions in the considered sources of streams and specifies 17 metafeatures with a high ability of concept identification.

Data processing in internet of things networks

One of the distinctive features of cloud applications in the Internet of Things (IoT) network is the real-time collection and processing of data about the objects of the physical environment. To solve this problem, with the help of sensors, data about the state of physical objects is collected to form the processing of data streams. The performance of such systems is critically dependent on the ability to securely and efficiently collect, transmit, and analyze data streams between environmental (real-world) objects and information systems. In such a case, it is relevant to study the data stream in Internet of Things networks and the operations that may occur on them. The issue of which platforms to process the operations to be carried out at the same time is the main subject of discussion.

Anomaly detection of power grid data based on storm cloud platform sliding window

Abstract To realize the monitoring and diagnosis of multi-source data in the power industry and meet the real-time processing requirements of the power system, the storm-distributed real-time computing platform is introduced to process the data. On this platform, the streaming data processing model based on a sliding window is deployed to realize the anomaly diagnosis of data flow. By calculating the predicted value of a point and the confidence interval of the predicted value of the point, we can judge whether the actual point is an abnormal value. By testing the data processing capacity of a cluster and a single machine, the experiment shows that in the cluster environment, reasonably setting the component parallelism can improve the throughput of flow computing.

Application of Event Cameras and Neuromorphic Computing to VSLAM: A Survey.

Simultaneous Localization and Mapping (SLAM) is a crucial function for most autonomous systems, allowing them to both navigate through and create maps of unfamiliar surroundings. Traditional Visual SLAM, also commonly known as VSLAM, relies on frame-based cameras and structured processing pipelines, which face challenges in dynamic or low-light environments. However, recent advancements in event camera technology and neuromorphic processing offer promising opportunities to overcome these limitations. Event cameras inspired by biological vision systems capture the scenes asynchronously, consuming minimal power but with higher temporal resolution. Neuromorphic processors, which are designed to mimic the parallel processing capabilities of the human brain, offer efficient computation for real-time data processing of event-based data streams. This paper provides a comprehensive overview of recent research efforts in integrating event cameras and neuromorphic processors into VSLAM systems. It discusses the principles behind event cameras and neuromorphic processors, highlighting their advantages over traditional sensing and processing methods. Furthermore, an in-depth survey was conducted on state-of-the-art approaches in event-based SLAM, including feature extraction, motion estimation, and map reconstruction techniques. Additionally, the integration of event cameras with neuromorphic processors, focusing on their synergistic benefits in terms of energy efficiency, robustness, and real-time performance, was explored. The paper also discusses the challenges and open research questions in this emerging field, such as sensor calibration, data fusion, and algorithmic development. Finally, the potential applications and future directions for event-based SLAM systems are outlined, ranging from robotics and autonomous vehicles to augmented reality.

Challenges and strategies of software engineering in big data environment

Software engineering in a big data environment involves developing and maintaining large-scale, scalable, data-intensive applications in a rapidly evolving technology domain. As data volumes grow exponentially, so do the requirements for software systems to process and analyze this data. These systems need to be able to efficiently store, retrieve, process and analyze data collected from sources such as social media, IoT devices, enterprises and scientific research. Software engineers must address challenges including data heterogeneity and quality, scalability of data storage and computation, and real-time processing of high-speed data streams. With the evolution of technology, new architecture and database technologies have emerged, such as NoSQL and distributed computing frameworks. These technologies have challenged traditional methods of software engineering practice and also brought new design and implementation strategies.

Incremental minimum distortion embedding for online structural damage classification.

The damage detection problem in structures diminishes the maintenance cost and extends the life of the structures. Online structural damage classification is a relevant topic in structural health monitoring since monitoring the structure using sensors can help decision-makers can rely on empirical evidence for making informed choices regarding the maintenance, repair, or replacement of structural components. Sensors generate high-dimensional data. To address this challenge, employing dimensionality reduction methods becomes essential. These methods help obtain a low-dimensional representation of the data. Traditional dimensionality reduction methods are trained on a fixed data set, and incorporating new data may require retraining the entire model. This study employed the Minimum Distortion Embedding (MDE) method incrementally, ensuring the maintenance of a consistent embedding as new data becomes available. This incremental embedding took advantage of an anchor constraint to pin the existing embedding in place, then embed the new points. An online structural classification methodology was developed using the incremental MDE and a stream based hoeffding tree classifier. Stream data processing differs from batch processing, as it involves the real-time consumption of each incoming data. The developed methodology was tested with data obtained by accelerometers in a laboratory scaled jacket-type wind turbine foundation. The structural damage classification problems had 5 different classes including the undamaged class and 4 different damage classes. The damage corresponded to a 5mm crack in four different structural elements of the wind turbine foundation. The results indicate the good behavior of the damage classification methodology supported by the high values of the classification metrics obtained.

Elevating Smart Manufacturing with a Unified Predictive Maintenance Platform: The Synergy between Data Warehousing, Apache Spark, and Machine Learning.

The transition to smart manufacturing introduces heightened complexity in regard to the machinery and equipment used within modern collaborative manufacturing landscapes, presenting significant risks associated with equipment failures. The core ambition of smart manufacturing is to elevate automation through the integration of state-of-the-art technologies, including artificial intelligence (AI), the Internet of Things (IoT), machine-to-machine (M2M) communication, cloud technology, and expansive big data analytics. This technological evolution underscores the necessity for advanced predictive maintenance strategies that proactively detect equipment anomalies before they escalate into costly downtime. Addressing this need, our research presents an end-to-end platform that merges the organizational capabilities of data warehousing with the computational efficiency of Apache Spark. This system adeptly manages voluminous time-series sensor data, leverages big data analytics for the seamless creation of machine learning models, and utilizes an Apache Spark-powered engine for the instantaneous processing of streaming data for fault detection. This comprehensive platform exemplifies a significant leap forward in smart manufacturing, offering a proactive maintenance model that enhances operational reliability and sustainability in the digital manufacturing era.

Model‐driven neural network based for HPO‐MIMO channel estimation

AbstractIntegrated Sensing and Communications (ISAC) need to process data streams in high‐speed sensor data acquisition or high‐speed wireless communications. To process the data can require more computing and communication resources, resulting in higher power consumption. Halved‐Phase Only Multiple Input Multiple Output (HPO‐MIMO) communication technology can solve this problem by using low‐power nonlinear detection devices. In ISAC, Channel Estimation (CE) technology can provide key channel characteristics and state information for sensing and collaborative work of perception and communication tasks. However, HPO‐MIMO system cannot realize CE using traditional receiver schemes because of the missing amplitude. In order to solve this problem, two HPO‐MIMO CE schemes based on model‐driven deep learning are proposed in this paper. The proposed schemes include a Densely Residual Network (DRN) and a Inception‐Resnet (IR), which is suitable for the case of sufficient data and insufficient data, respectively. The simulation results show that the performance of DRN based scheme is better than that of IR based scheme when the data amount is sufficient, and the performance of IR based scheme is better when the dataset is small. In addition, the proposed CE schemes work well with a range of antenna sizes and distances.

Online Concept Drift Detector: Optimally Balancing Delay Detection, Runtime, Memory, and Accuracy.

Online learning and real-time data processing are becoming increasingly vital across various domains such as sensor networks, banking, and telecommunications. A significant challenge in this context is concept drift, wherein the statistical properties of the data change over time. Traditional drift detectors often grapple with high memory usage, extended delay in detection, prolonged runtime, and accuracy inconsistencies. This paper introduces a novel Online Drift Detector that meticulously balances these four aspects. By processing data instance-by-instance, our proposed detector optimizes the trade-offs between delay detection, runtime, memory consumption, and accuracy. We incorporate a unique diversity calculation tailored for multi-label problems, ensuring swift drift detection with minimized memory usage and enhanced runtime efficiency. Comparative analyses reveal the dominance of our approach over contemporary drift detection techniques, particularly in the realms of memory efficiency, detection speed, and accuracy. This work substantially augments the field of online data stream processing by offering a refined strategy for timely and efficient concept drift detection across a myriad of applications.

Online Processing of Vehicular Data on the Edge Through an Unsupervised TinyML Regression Technique

The Internet of Things (IoT) has made it possible to include everyday objects in a connected network, allowing them to intelligently process data and respond to their environment. Thus, it is expected that those objects will gain an intelligent understanding of their environment and be able to process data more efficiently than before. Particularly, such edge computing paradigm has allowed the execution of inference methods on resource-constrained devices such as microcontrollers, significantly changing the way IoT applications have evolved in recent years. However, although this scenario has supported the development of Tiny Machine Learning (TinyML) approaches on such devices, there are still some challenges that require further investigation when optimizing data streaming on the edge. Therefore, this article proposes a new unsupervised TinyML regression technique based on the typicality and eccentricity of the samples to be processed. Moreover, the proposed technique also exploits a Recursive Least Squares (RLS) filter approach. Combining all these features, the proposed method uses similarities between samples to identify patterns when processing data streams, predicting outcomes based on these patterns. The results obtained through the extensive experimentation utilizing vehicular data streams were highly encouraging. The proposed algorithm was meticulously compared with the RLS algorithm and Convolutional Neural Networks (CNN). It exhibited significantly superior performance, with mean squared errors that were 4.68 and 12.02 times lower, respectively, compared to the aforementioned techniques.

From CountMin to Super kJoin Sketches for Flow Spread Estimation

Real-time data stream processing is key to many Internet applications ranging from e-commerce, social networks, to network monitoring. Sketches (compact data structures) are often used to track large, high-rate streams and estimate their statistics. However, the traditional model for measuring flow size, e.g., number of packets in each flow, is unsuitable for more sophisticated applications that require flow spreads, i.e., number of distinct elements in each flow. When there are numerous flows, most existing work modifies CountMin sketches — which were designed to measure flow size — for measuring flow spread. They inherit a similar design and a commonly-used min operation from the CountMin sketches. This paper casts doubt on such a solution path, arguing that the inherited sketch design and the min operation were both intended for a different purpose and are not the best for flow spread measurement. Replacing the min operation with new position-aware operations, we propose the kJoin sketches that achieve much better average accuracy. We mathematically derive the estimation error bound. We also apply our sketches for network-wide measurement. Trace-driven experiments on software/GPU/FPGA platforms demonstrate our work significantly improves estimation accuracy, streaming rate (recording throughput), and query throughput, compared to the state-of-the-art.

Method of aggregating temporal multimodal streaming data from iot devices based on reactive programming

Recent advancements in Internet of Things (IoT) technologies have led to an exponential increase in the generation of temporal multimodal streaming data from various IoT devices. Aggregating and processing this data efficiently poses significant challenges, including scalability, reliability, and synchronization issues. This paper proposes a novel method for aggregating temporal multimodal streaming data from IoT devices based on reactive programming principles. We begin by analyzing recent research and publications in the field to identify existing challenges and limitations in aggregating temporal data from IoT devices. The key problem addressed in this study is the efficient aggregation of data streams while ensuring scalability, reliability, and synchronization across different devices. The primary objective of this research is to develop a method that can hierarchically organize and aggregate temporal data streams from IoT devices while addressing the challenges of scalability and reliability. To achieve this, we leverage reactive programming techniques, specifically the GroupJoin operation, which allows for the seamless integration of data streams from multiple sources. Our method involves initializing the aggregation process by passing relevant data streams from two devices to the GroupJoin operation and configuring the settings for operation. Additionally, we introduce flexibility in the aggregation process by allowing for the hierarchical structuring of data streams based on physical or logical criteria. Furthermore, we discuss the implementation of our proposed method using the Rx library for .NET, which provides implementations for various programming languages, ensuring its adaptability across different platforms. We benchmark the performance of our method against a naive implementation, demonstrating its efficiency and scalability in aggregating temporal data streams from IoT devices. In conclusion, our proposed method offers a scalable and reliable solution for aggregating temporal multimodal streaming data from IoT devices. By leveraging reactive programming principles, we address key challenges in data aggregation and provide a flexible method for organizing and processing data streams efficiently.

Adaptive Scheduling of Continuous Operators for IoT Edge Analytics

In this paper, we address the problem of adaptive scheduling of data stream processing and analytics (DSPA) applications in a shared edge fog cloud (EFC) continuum with response time constraints. The focus is on handling the dynamic workload of DSPA applications caused by the variability of their input data stream rates generated by mobile IoT devices, and the dynamically available resource capacity in the EFC continuum. To address these challenges, we characterise the different types of resources in the EFC continuum, as well as the operators that make up a DSPA application. Based on this characterisation, we propose models to evaluate the response time and the cost of using the resources in the always dynamic EFC continuum. We then formulate the problem of adaptive scheduling of a DSPA application in the EFC continuum with the objective of minimising the cost of using the shared resources subject to the constraints of the response time and the available capacity of the EFC resources. We propose a heuristic algorithm that dynamically computes a new scheduling of the DSPA application, taking into account its current deployment state and the current state of the shared resources in the EFC continuum. Experimental results, using simulation, show the effectiveness of our proposed algorithm against algorithms of related work.

Exposing and explaining fake news on-the-fly

Social media platforms enable the rapid dissemination and consumption of information. However, users instantly consume such content regardless of the reliability of the shared data. Consequently, the latter crowdsourcing model is exposed to manipulation. This work contributes with an explainable and online classification method to recognize fake news in real-time. The proposed method combines both unsupervised and supervised Machine Learning approaches with online created lexica. The profiling is built using creator-, content- and context-based features using Natural Language Processing techniques. The explainable classification mechanism displays in a dashboard the features selected for classification and the prediction confidence. The performance of the proposed solution has been validated with real data sets from Twitter and the results attain 80% accuracy and macro F-measure. This proposal is the first to jointly provide data stream processing, profiling, classification and explainability. Ultimately, the proposed early detection, isolation and explanation of fake news contribute to increase the quality and trustworthiness of social media contents.

Fairness optimisation with multi-objective swarms for explainable classifiers on data streams

Recently, advanced AI systems equipped with sophisticated learning algorithms have emerged, enabling the processing of extensive streaming data for online decision-making in diverse domains. However, the widespread deployment of these systems has prompted concerns regarding potential ethical issues, particularly the risk of discrimination that can adversely impact certain community groups. This issue has been proven to be challenging to address in the context of streaming data, where data distribution can change over time, including changes in the level of discrimination within the data. In addition, transparent models like decision trees are favoured in such applications because they illustrate the decision-making process. However, it is essential to keep the models compact because the explainability of large models can diminish. Existing methods usually mitigate discrimination at the cost of accuracy. Accuracy and discrimination, therefore, can be considered conflicting objectives. Current methods are still limited in controlling the trade-off between these conflicting objectives. This paper proposes a method that can incrementally learn classification models from streaming data and automatically adjust the learnt models to balance multi-objectives simultaneously. The novelty of this research is to propose a multi-objective algorithm to maximise accuracy, minimise discrimination and model size simultaneously based on swarm intelligence. Experimental results using six real-world datasets show that the proposed algorithm can evolve fairer and simpler classifiers while maintaining competitive accuracy compared to existing state-of-the-art methods tailored for streaming data.

Online group streaming feature selection based on fuzzy neighborhood granular ball rough sets

Online group streaming feature selection holds significant research value in large-scale streaming data processing scenarios, and the related work based on rough set theory has attracted academic interest. Nevertheless, most relevant algorithms come with parameters, and performing a grid search for the optimal parameters reduces the efficiency. Moreover, the existing online group streaming feature selection frameworks cannot effectively fit the situation in practice. Focusing on these issues, this paper investigates an online group streaming feature selection method based on fuzzy neighborhood granular ball rough sets. First, Canopy clustering is introduced to granular ball computing, and the adaptive neighborhood of samples is generated based on the granular ball distribution. Second, we construct a fuzzy neighborhood granular ball rough set (FNGBRS) model and propose the integrated dependence degree to achieve maximal dependency and minimum classification error. Then, the purity of granular balls is considered as the weight of features, and some uncertainty measures based on FNGBRS are presented. Finally, we define a random factor to control the size of streaming groups and design an online group streaming feature selection algorithm. Comparative experimental results on sixteen public datasets demonstrate that the proposed algorithm exhibits superior and stable classification performance, coupled with increased efficiency from its parameter-free design.