Big Data Stream Processing Research Articles

The role of classifiers and data complexity in learned Bloom filters: insights and recommendations

Bloom filters, since their introduction over 50 years ago, have become a pillar to handle membership queries in small space, with relevant application in Big Data Mining and Stream Processing. Further improvements have been recently proposed with the use of Machine Learning techniques: learned Bloom filters. Those latter make considerably more complicated the proper parameter setting of this multi-criteria data structure, in particular in regard to the choice of one of its key components (the classifier) and accounting for the classification complexity of the input dataset. Given this State of the Art, our contributions are as follows. (1) A novel methodology, supported by software, for designing, analyzing and implementing learned Bloom filters that account for their own multi-criteria nature, in particular concerning classifier type choice and data classification complexity. Extensive experiments show the validity of the proposed methodology and, being our software public, we offer a valid tool to the practitioners interested in using learned Bloom filters. (2) Further contributions to the advancement of the State of the Art that are of great practical relevance are the following: (a) the classifier inference time should not be taken as a proxy for the filter reject time; (b) of the many classifiers we have considered, only two offer good performance; this result is in agreement with and further strengthens early findings in the literature; (c) Sandwiched Bloom filter, which is already known as being one of the references of this area, is further shown here to have the remarkable property of robustness to data complexity and classifier performance variability.

A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment

A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment

A novel family of online censoring based complex-valued least mean kurtosis algorithms

In this paper, we propose a novel family of online censoring (OC) based complex-valued least mean kurtosis (CLMK) algorithms by inspiring the advantages of the online censoring strategy and kurtosis-based cost function. We first develop important members of this family of algorithms such as OC based CLMK (OC-CLMK) and augmented CLMK (OC-ACLMK). These algorithms censor less informative complex-valued data streams in an online manner and keep only the most informative ones for performing their weight vector updates. Thus, they reduce the costs of data processing without markedly affecting performance accuracy. However, they do not take into account possible outliers that disturb their performances considerably. Therefore, we also develop robust members such as robust OC-CLMK (ROC-CLMK) and OC-ACLMK (ROC-CLMK) algorithms, which censor less informative complex-valued data streams as well as possible outliers. The convergence analyses of the proposed algorithms in the sense of Lyapunov are also presented to derive the bounds of their step sizes in a compact form. The simulation results on large-scale system identification and regression scenarios affirm the mentioned attractive features of the proposed algorithms. This study also shows that the noteworthy properties of the proposed algorithms will provide important contributions to the processing of complex-valued big data streams.

I/O-efficient GPU-based acceleration of coherent dedispersion for pulsar observation

Coherent dedispersion is a frequency domain filtering method extensively used in big data stream processing to achieve high time resolution and pulse-profile precision in pulsar observation. However, the computational complexity and challenges associated with processing large-scale data streams have hindered its widespread adoption in modern telescopes. Overcoming these hurdles is crucial to realizing wide-band real-time coherent dedispersion. In this paper, we propose a highly parallel and high-I/O throughput implementation of coherent dedispersion on GPU, specifically designed to support consecutive dispersion measure trials. Our primary objective is to achieve real-time coherent dedispersion, enabling precise high-time resolution and pulse-profile analysis in modern telescopes. To accomplish this, we leverage the power of GPU parallelism and employ advanced multi-threading techniques to optimize both computation and I/O throughput. Experiments show that our method achieves performance gains of 3.96x and 4.65x in processing time and IO throughput, respectively, when compared with the baseline work, the Coherent Dispersion Measure Trials (CDMT). Finally, the proposed techniques for improving computation parallelism and I/O efficiency are general that can provide valuable insights and serve as a reference for optimizing related algorithms in the field of big data stream processing.

SMERAS -State Management with Efficient Resource Allocation and Scheduling in Big Data Stream Processing Systems

SMERAS -State Management with Efficient Resource Allocation and Scheduling in Big Data Stream Processing Systems

Development of infrastructure for anomalies detectionin big data

The work describes the conducted analysis of models, methods, and technologies for detecting anomalies in data. It concludes that, based on the analysis, the solution to the problem of anomaly detection in data should be considered as a complex technology, which consists of the formation and application of mathematical models in combination with the research of data processing approaches. The article analyses the current state of big data stream processing technologies and reflects the peculiarities of the most commonly used and advanced of them, e.g. Apache Hadoop, Apache Spark, Apache Cassandra, Apache Kafka, Apache Storm, and Apache Beam. On top of these, it pays attention to the infrastructure, in which the created software models can be deployed and used, taking into account the high-load real-time nature of the data. The article proposes to form an infrastructure for anomaly detection in data as an applied example of big data processing cloud infrastructure. The paper demonstrates the developed infrastructure model for anomaly detection in real-time stream data, which is based on an expert method of forming requirements for a software component, choosing an algorithm for detecting anomalies, selecting tools, and improving the algorithm. The highlighted anomaly detection tools allow us to create a secure real-time anomaly detection solution using Dataflow, BigQuery ML, and Cloud DLP. The paper presents the applied implementation of anomaly detection in real-time using GCP and Apache Beam -data stream analysis of software logs in the information system and detection of fraudulent ones among them, which will help improve the cyber security of the system. In the end, the work demonstrates possible improvements to the basic model that could help to speed it up

Microservice architecture of the big data stream processing system

The issue of using microservice architecture in big data stream processing systems is considered. Advantages and disadvantages of existing big data analysis architectures are studied. A version of the microservice system for big data processing using the Kafka distributed event streaming platform, the Docker program development and launch platform, the Postgres object-relational database management system, and the FastAPI web platform for creating applications is proposed. Architectural patterns have been developed that can simplify the development of large data applications using microservices, and application concepts using the resulting patterns. Simulation results are presented, which show that a microservice architecture with distributed load on several replicas can provide better scalability and speed compared to a monolithic architecture.

Data stream fusion for accurate quantile tracking and analysis

UDDSketch is a recent algorithm for accurate tracking of quantiles in data streams, derived from the DDSketch algorithm. UDDSketch provides accuracy guarantees covering the full range of quantiles independently of the input distribution and greatly improves the accuracy with regard to DDSketch. In this paper we show how to compress and fuse two or more data streams (or datasets) by leveraging the mergeability of the UDDSketch data summaries. In general, two summaries on two data streams are said to be mergeable if there exists an algorithm that allows combining the two summaries into a single one related to the union of the two datasets, simultaneously preserving the error and size guarantees. The property of mergeability of a sketch enables the parallel and distributed processing of big volume data streams that can be compressed and fused by means of such mergeable data structures. Among the applications strictly related to accurate tracking of quantiles, requiring parallel and/or distributed processing we recall here estimating the latency of a web site, database query optimizers and the need of succinctly summarizing the distribution of values occurring over a sensor network. We prove that UDDSketch is fully mergeable and introduce PUDDSketch, a parallel version of UDDSketch suitable for message-passing based architectures. We formally prove its correctness and compare it to a parallel version of DDSketch, showing through extensive experimental results that our parallel algorithm almost always outperforms the parallel DDSketch algorithm with regard to the overall accuracy in determining the quantiles. Moreover, we also design and implement parallel versions of both the state of the art KLL and REQ sequential algorithms in order to compare and contrast PUDDSketch versus the corresponding parallel algorithms. Our experiments clearly show that PUDDSketch is faster or on par with regard to parallel running time, whilst providing simultaneously greater accuracy.

A Machine Learning and Multi-Agent Model to Automate Big Data Analytics in Smart Cities

The objective of this paper is to present an architecture to improve the process of automating big data analytics using a multi-agent system and machine learning techniques, to support the processing of real time big data streams and to enhance the process of decision-making for urban planning and management. With the rapidly evolving information technologies, and their utilization in many areas such as smart cities, social networks, urban management and planning, massive data streams are generated and need an efficient approach to deal with. The proposition in this paper adopts the concept of smart data which focuses on the value aspect from big data. The proposed architecture is composed of three layers: data acquisition and storage, data management and processing and the service layer, based on a multi-agent system to automate the big data analytics; the proposed model describe the functionalities of the system and the collaboration between agents, these autonomous entities receive data streams in real time, they perform operations of preprocessing, big data analytics and storage into a Hadoop cluster. The techniques of machine learning are also used to enhance the process of decision making, such the use of classification algorithms to predict habitat type based on the characteristics of a population to help making efficient urban planning decisions. The proposed system can serve as a platform to support data management and to conduct effective decision-making in smart cities.

A Review on Big Data Stream Processing Applications: Contributions, Benefits, and Limitations

The amount of data in our world has been rapidly keep growing from time to time. Â In the era of big data, the efficient processing and analysis of big data using machine learning algorithm is highly required, especially when the data comes in form of streams. There is no doubt that big data has become an important source of information and knowledge in making decision process. Nevertheless, dealing with this kind of data comes with great difficulties; thus, several techniques have been used in analyzing the data in the form of streams. Many techniques have been proposed and studied to handle big data and give decisions based on off-line batch analysis. Today, we need to make a constructive decision based on online streaming data analysis. Many researchers in recent years proposed some different kind of frameworks for processing the big data streaming. In this work, we explore and present in detail some of the recent achievements in big data streaming in term of contributions, benefits, and limitations. As well as some of recent platforms suitable to be used for big data streaming analytics. Moreover, we also highlight several issues that will be faced in big data stream processing. In conclusion, it is hoped that this study will assist the researchers in choosing the best and suitable framework for big data streaming projects.

Mille Cheval: a GPU-based in-memory high-performance computing framework for accelerated processing of big-data streams

Streams are temporally ordered, rapid changing, ample in volume, and infinite in nature. It is nearly impossible to store the entire data stream due to its large volume and high velocity. In this work, the principle of parallelism is employed to accelerate stream data computing. GPU-based high-performance computing (HPC) framework is proposed for accelerated processing of big-data streams using the in-memory data structure. We have implemented three parallel algorithms to prove the viability of the framework. The contributions of Mille Cheval are: (1) the viability of streaming on accelerators to increase throughput, (2) carefully chosen hash algorithms to achieve low collision rate and high randomness, and (3) memory sketches for approximation. The objective is to leverage the power of a single node using in-memory computing and hybrid computing. HPC does not always require high-end hardware but well-designed algorithms. Achievements of Mille Cheval are: (1) relative error is 1.32 when error rate and overestimate rate are chosen as 0.001 and (2) the host memory space requirement is just 63 MB for 1 terabyte of data. The proposed algorithms are pragmatic. It is evident from experimental results that the framework demonstrates 10X speed-up as compared with CPU implementations and 3X speed-up as compared with GPU implementations.

A survey on data analysis on large-Scale wireless networks: online stream processing, trends, and challenges

In this paper we focus on knowledge extraction from large-scale wireless networks through stream processing. We present the primary methods for sampling, data collection, and monitoring of wireless networks and we characterize knowledge extraction as a machine learning problem on big data stream processing. We show the main trends in big data stream processing frameworks. Additionally, we explore the data preprocessing, feature engineering, and the machine learning algorithms applied to the scenario of wireless network analytics. We address challenges and present research projects in wireless network monitoring and stream processing. Finally, future perspectives, such as deep learning and reinforcement learning in stream processing, are anticipated.

MOBDA: Microservice-Oriented Big Data Architecture for Smart City Transport Systems

Highly populated cities depend highly on intelligent transportation systems (ITSs) for reliable and efficient resource utilization and traffic management. Current transportation systems struggle to meet different stakeholder expectations while trying their best to optimize resources in providing various transport services. This paper proposes a Microservice-Oriented Big Data Architecture (MOBDA) incorporating data processing techniques, such as predictive modelling for achieving smart transportation and analytics microservices required towards smart cities of the future. We postulate key transportation metrics applied on various sources of transportation data to serve this objective. A novel hybrid architecture is proposed to combine stream processing and batch processing of big data for a smart computation of microservice-oriented transportation metrics that can serve the different needs of stakeholders. Development of such an architecture for smart transportation and analytics will improve the predictability of transport supply for transport providers and transport authority as well as enhance consumer satisfaction during peak periods.

Flink-ER: An Elastic Resource-Scheduling Strategy for Processing Fluctuating Mobile Stream Data on Flink

As real-time and immediate feedback becomes increasingly important in tasks related to mobile information, big data stream processing systems are increasingly applied to process massive amounts of mobile data. However, when processing a drastically fluctuating mobile data stream, the lack of an elastic resource-scheduling strategy limits the elasticity and scalability of data stream processing systems. To address this problem, this paper builds a flow-network model, a resource allocation model, and a data redistribution model as the foundation for proposing Flink with an elastic resource-scheduling strategy (Flink-ER), which consists of a capacity detection algorithm, an elastic resource reallocation algorithm, and a data redistribution algorithm. The strategy improves the performance of the platform by dynamically rescaling the cluster and increasing the parallelism of operators based on the processing load. The experimental results show that the throughput of a cluster was promoted under the premise of meeting latency constraints, which verifies the efficiency of the strategy.

Stream of Unbalanced Medical Big Data Using Convolutional Neural Network

In order to address the problem that the traditional algorithm can not predict the network link load effectively, which leads to high packet loss and energy loss, long turnaround time, slow stream rate and poor anti-attack ability, the paper proposes the stream algorithm of unbalanced medical big data based on convolutional neural network (CNN). The proposed algorithm included two stages:In the first stage, the decomposition-prediction model was constructed, the combined wavelet analysis and neural network analysis were used to complete the network link load prediction; In the second stage, based on the network link load situation, we analyzed the structure of each layer of convolution neural network, constructed the medical big data stream optimization model, introduced the ReLu function to calculate the convolution neural network, solved the optimization model, and completed the stream processing of unbalanced medical big data. The experimental results show that the network link load prediction accuracy of the proposed stream algorithm is as high as 93%, the lowest packet loss rate is only 2.0%, the energy loss of the stream process is low, the rate is fast, and the anti-attack efficiency is high, which is more conducive to the realization of data stream.

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

Contribution: Recently, real-time data warehousing (DWH) and big data streaming have become ubiquitous due to the fact that a number of business organizations are gearing up to gain competitive advantage. The capability of organizing big data in efficient manner to reach a business decision empowers data warehousing in terms of real-time stream processing. A systematic literature review for real-time stream processing systems is presented in this paper which rigorously look at the recent developments and challenges of real-time stream processing systems and can serve as a guide for the implementation of real-time stream processing framework for all shapes of data streams. Background: Published surveys and reviews either cover papers focusing on stream analysis in applications other than real-time DWH or focusing on extraction, transformation, loading (ETL) challenges for traditional DWH. This systematic review attempts to answer four specific research questions. Research Questions: 1)Which are the relevant publication channels for real-time stream processing research? 2) Which challenges have been faced during implementation of real-time stream processing? 3) Which approaches/tools have been reported to address challenges introduced at ETL stage while processing real-time stream for real-time DWH? 4) What evidence have been reported while addressing different challenges for processing real-time stream? Methodology: A systematic literature was conducted to compile studies related to publication channels targeting real-time stream processing/joins challenges and developments. Following a formal protocol, semi-automatic and manual searches were performed for work from 2011 to 2020 excluding research in traditional data warehousing. Of 679,547 papers selected for data extraction, 74 were retained after quality assessment. Findings: This systematic literature highlights implementation challenges along with developed approaches for real-time DWH and big data stream processing systems and provides their comparisons. This study found that there exists various algorithms for implementing real-time join processing at ETL stage for structured data whereas less work for un-structured data is found in this subject matter.

Boosting Big Data Streaming Applications in Clouds With BurstFlow

The rapid growth of stream applications in financial markets, health care, education, social media, and sensor networks represents a remarkable milestone for data processing and analytic in recent years, leading to new challenges to handle Big Data in real-time. Traditionally, a single cloud infrastructure often holds the deployment of Stream Processing applications because it has extensive and adaptative virtual computing resources. Hence, data sources send data from distant and different locations of the cloud infrastructure, increasing the application latency. The cloud infrastructure may be geographically distributed and it requires to run a set of frameworks to handle communication. These frameworks often comprise a Message Queue System and a Stream Processing Framework. The frameworks explore Multi-Cloud deploying each service in a different cloud and communication via high latency network links. This creates challenges to meet real-time application requirements because the data streams have different and unpredictable latencies forcing cloud providers’ communication systems to adjust to the environment changes continually. Previous works explore static micro-batch demonstrating its potential to overcome communication issues. This paper introduces BurstFlow, a tool for enhancing communication across data sources located at the edges of the Internet and Big Data Stream Processing applications located in cloud infrastructures. BurstFlow introduces a strategy for adjusting the micro-batch sizes dynamically according to the time required for communication and computation. BurstFlow also presents an adaptive data partition policy for distributing incoming streams across available machines by considering memory and CPU capacities. The experiments use a real-world multi-cloud deployment showing that BurstFlow can reduce the execution time up to 77% when compared to the state-of-the-art solutions, improving CPU efficiency by up to 49%.

Real-Time Healthcare Monitoring System using Online Machine Learning and Spark Streaming

The real-time monitoring and tracking systems play a critical role in the healthcare field. Wearable medical devices with sensors, mobile applications, and health cloud have continuously generated an enormous amount of data, often called streaming big data. Due to the higher speed of the streaming data, it is difficult to ingest, process, and analyze such huge data in real-time to make real-time actions in case of emergencies. Using traditional methods that are inadequate and time-consuming. Therefore, there is a significant need for real-time big data stream processing to guarantee an effective and scalable solution. So, we proposed a new system for online prediction to predict health status using Spark streaming framework. The proposed system focuses on applying streaming machine learning models (i.e. streaming linear regression with SGD) on streaming health data events ingested to spark streaming through Kafka topics. The experimental results are done on the historical medical datasets (i.e. diabetes dataset, heart disease dataset, and breast cancer dataset) and generated dataset which is simulated to wearable medical sensors. The historical datasets have shown that the accuracy improvement ratio obtained using the diabetes disease dataset is the highest one with respect to the other two datasets with an accuracy of 81%. For generated datasets, the online prediction system has achieved accuracy with 98% at 5 seconds window size. Beyond this, the experimental results have proofed that the online prediction system can online learn and update the model according to the new data arrival and window size.

A new Internet of Things architecture for real-time prediction of various diseases using machine learning on big data environment

A number of technologies enabled by Internet of Thing (IoT) have been used for the prevention of various chronic diseases, continuous and real-time tracking system is a particularly important one. Wearable medical devices with sensor, health cloud and mobile applications have continuously generating a huge amount of data which is often called as streaming big data. Due to the higher speed of the data generation, it is difficult to collect, process and analyze such massive data in real-time in order to perform real-time actions in case of emergencies and extracting hidden value. using traditional methods which are limited and time-consuming. Therefore, there is a significant need to real-time big data stream processing to ensure an effective and scalable solution. In order to overcome this issue, this work proposes a new architecture for real-time health status prediction and analytics system using big data technologies. The system focus on applying distributed machine learning model on streaming health data events ingested to Spark streaming through Kafka topics. Firstly, we transform the standard decision tree (DT) (C4.5) algorithm into a parallel, distributed, scalable and fast DT using Spark instead of Hadoop MapReduce which becomes limited for real-time computing. Secondly, this model is applied to streaming data coming from distributed sources of various diseases to predict health status. Based on several input attributes, the system predicts health status, send an alert message to care providers and store the details in a distributed database to perform health data analytics and stream reporting. We measure the performance of Spark DT against traditional machine learning tools including Weka. Finally, performance evaluation parameters such as throughput and execution time are calculated to show the effectiveness of the proposed architecture. The experimental results show that the proposed system is able to effectively process and predict real-time and massive amount of medical data enabled by IoT from distributed and various diseases.

DTA-HOC: Online HTTPS traffic service identification using DNS in large-scale networks

An increasing number of websites are making use of HTTPS encryption to enhance security and privacy for their users. However, HTTPS encryption makes it very difficult to identify the service over HTTPS flows, which poses challenges to network security management. In this paper we present DTA-HOC, a novel DNS-based two-level association HTTPS traffic online service identification method for large-scale networks, which correlates HTTPS flows with DNS flows using big data stream processing and association technologies to label the service in an HTTPS flow with a specific associated domain name. DTA-HOC has been specifically designed to address three practical challenges in the service identification process: domain name ambiguity, domain name query invisibility, and data association time window size contradictions. Several experiments on datasets collected from a 10-Gbps campus network are conducted alongside offline and online testing. Results show that DTA-HOC can achieve an average online association rate on HTTPS traffic of 83% and a generic accuracy of 86.16%. Its processing time for one minute of data is less than 20 seconds. These results indicate that DTA-HOC is an efficient method for online identification of services in HTTPS flows for large-scale networks. Moreover, our proposed method can contribute to the identification of other applications which make a Domain Name System (DNS) communication before establishing a connection.