Data Stream Processing Research Articles

Enhancing Data Engineering Frameworks for Scalable Real-Time Marketing Solutions

This report discusses improvements to the data engineering frameworks for cost-efficient real-time marketing environment, with the applications in mobile advertising and smart city analytics. The proposed framework combines the processing of time-series data with map-reduce (M-R) systems in order to compensate for inefficiencies in the execution of temporal queries and the processing of real-time data streams. It enables effective behavioral targeting in mobile marketing and helps integrate various domains’ real-time data for smart city utilization. Specific examples that illustrate further refinements in the algorithms are showcased through case study and experimental evidence where gains in throughput, data reliability, and system effectiveness are quantified in real-time Marketing and Smart City applications.

Streaming process based on Spark and Kafka and its case application

E-commerce platforms gradually integrate into our lives. Online shopping has become a daily habit for people. People who shop online will generate online shopping records in real time. These online shopping records are often massive. E-commerce platforms use relevant big data technology to conduct statistics on real-time user data. In response to the current demand for e-commerce platforms for big data real-time computing technology, this article provides an idea of big data streaming computing, a framework for combining two big data technologies, Spark and Kafka, to process streaming data. This framework uses Kafka to send real-time data to SparkStreaming for calculation. Then the obtained data is visualized. This article applies these two big data technologies to simulate and implement an application case of an e-commerce platform. This case design and implementation enables real-time statistics of the click volume of a mobile phone brand on an e-commerce platform and dynamic display of streaming data on the Web.

Hierarchical Auto-scaling Policies for Data Stream Processing on Heterogeneous Resources

Data Stream Processing (DSP) applications analyze data flows in near real-time by means of operators, which process and transform incoming data. Operators handle high data rates running parallel replicas across multiple processors and hosts. To guarantee consistent performance without wasting resources in the face of variable workloads, auto-scaling techniques have been studied to adapt operator parallelism at run-time. However, most of the effort has been spent under the assumption of homogeneous computing infrastructures, neglecting the complexity of modern environments.We consider the problem of deciding both how many operator replicas should be executed and which types of computing nodes should be acquired. We devise heterogeneity-aware policies by means of a two-layered hierarchy of controllers. While application-level components steer the adaptation process for whole applications, aiming to guarantee user-specified requirements, lower-layer components control auto-scaling of single operators. We tackle the fundamental challenge of performance and workload uncertainty, exploiting Bayesian optimization (BO) and reinforcement learning (RL) to devise policies. The evaluation shows that our approach is able to meet users’ requirements in terms of response time and adaptation overhead, while minimizing the cost due to resource usage, outperforming state-of-the-art baselines. We also demonstrate how partial model information is exploited to reduce training time for learning-based controllers.

Adaptive local Principal Component Analysis improves the clustering of high-dimensional data

In local Principal Component Analysis (PCA), a distribution is approximated by multiple units, each representing a local region by a hyper-ellipsoid obtained through PCA. We present an extension for local PCA which adaptively adjusts both the learning rate of each unit and the potential function which guides the competition between the local units. Our local PCA method is an online neural network method where unit centers and shapes are modified after the presentation of each data point. For several benchmark distributions, we demonstrate that our method improves the overall quality of clustering, especially for high-dimensional distributions where many conventional methods do not perform satisfactorily. Our online method is also well suited for the processing of streaming data: The two adaptive mechanisms lead to a quick reorganization of the clustering when the underlying distribution changes.

General-purpose data stream processing on heterogeneous architectures with WindFlow

Many emerging applications analyze data streams by running graphs of communicating tasks called operators. To develop and deploy such applications, Stream Processing Systems (SPSs) like Apache Storm and Flink have been made available to researchers and practitioners. They exhibit imperative or declarative programming interfaces to develop operators running arbitrary algorithms working on structured or unstructured data streams. In this context, the interest in leveraging hardware acceleration with GPUs has become more pronounced in high-throughput use cases. Unfortunately, GPU acceleration has been studied for relational operators working on structured streams only, while non-relational operators have often been overlooked. This paper presents WindFlow, a library supporting the seamless GPU offloading of general partitioned-stateful operators, extending the range of operators that benefit from hardware acceleration. Its design provides high throughput still exposing a high-level API to users compared with the raw utilization of GPUs in Apache Flink.

A new RFID Middleware architecture based on a hybrid security technique using data encryption and RBAC for modern real-time tracking applications

Radio Frequency Identification (RFID) is a contactless technology that has developed over the 90s and 20th centuries. It employs electromagnetic or electrostatic coupling in the radio frequency part of the electromagnetic spectrum to uniquely identify traceable objects, and is widely used in various sectors (e.g., medical, Supply Chain Management, transportation, and IoT applications.). Through the supply of real-world monitoring and context information about things, the integration of this technology in such areas delivers various benefits in the future of ubiquitous computing. However, one of the primary challenges will be the capacity to manage data since RFID events have specific characteristics and requires special treatment, such as the large volume of data flow, inaccuracy, temporal and spatial data, are typical examples of RFID event data. The goal of this research is to first highlight the concerns and limitations of existing middleware architectures before introducing and implementing a new Middleware architecture to address the identified issues, specifically real-time processing of massive volumes of data coming from physical RFID infrastructure. This middleware combines role-based access control with an encryption algorithm to increase security, a NoSQL database for storing large amounts of data, complex event processing (CEP) to provide high-volume data stream processing, and improved interoperability via the Data Transformation Module. Finally, our architecture is evaluated and compared to several middleware architectures based on standard ISO/IEC 9126 metrics.

Building Fake Review Detection Model Based on Sentiment Intensity and PU Learning.

Fake review detection has the characteristics of huge stream data processing scale, unlimited data increment, dynamic change, and so on. However, the existing fake review detection methods mainly target limited and static review data. In addition, deceptive fake reviews have always been a difficult point in fake review detection due to their hidden and diverse characteristics. To solve the above problems, this article proposes a fake review detection model based on sentiment intensity and PU learning (SIPUL), which can continuously learn the prediction model from the constantly arriving streaming data. First, when the streaming data arrive, the sentiment intensity is introduced to divide the reviews into different subsets (i.e., strong sentiment set and weak sentiment set). Then, the initial positive and negative samples are extracted from the subset using the marking mechanism of selection completely at random (SCAR) and Spy technology. Second, building a semi-supervised positive-unlabeled (PU) learning detector based on the initial sample to detect fake reviews in the data stream iteratively. According to the detection results, the data of initial samples and the PU learning detector are continuously updated. Finally, the old data are continually deleted according to the historical record points, so that the training sample data are within a manageable size and prevent overfitting. Experimental results show that the model can effectively detect fake reviews, especially deceptive reviews.

A real-time discovery method for vehicle companion via service collaboration

PurposeVehicle companion is one of the most common companion patterns in daily life, which has great value to accident investigation, group tracking, carpooling recommendation and road planning. Due to the complexity and large scale of vehicle sensor streaming data, existing work were difficult to ensure the efficiency and effectiveness of real-time vehicle companion discovery (VCD). This paper aims to provide a high-quality and low-cost method to discover vehicle companions in real time.Design/methodology/approachThis paper provides a real-time VCD method based on pro-active data service collaboration. This study makes use of dynamic service collaboration to selectively process data produced by relative sensors, and relax the temporal and spatial constraints of vehicle companion pattern for discovering more potential companion vehicles.FindingsExperiments based on real and simulated data show that the method can discover 67% more companion vehicles, with 62% less response time comparing with centralized method.Originality/valueTo reduce the amount of processing streaming data, this study provides a Service Collaboration-based Vehicle Companion Discovery method based on proactive data service model. And this study provides a new definition of vehicle companion through relaxing the temporal and spatial constraints for discover companion vehicles as many as possible.

ContTune: Continuous Tuning by Conservative Bayesian Optimization for Distributed Stream Data Processing Systems

The past decade has seen rapid growth of distributed stream data processing systems. Under these systems, a stream application is realized as a Directed Acyclic Graph (DAG) of operators, where the level of parallelism of each operator has a substantial impact on its overall performance. However, finding optimal levels of parallelism remains challenging. Most existing methods are heavily coupled with the topological graph of operators, unable to efficiently tune under-provisioned jobs. They either insufficiently use previous tuning experience by treating successively tuning independently, or explore the configuration space aggressively, violating the Service Level Agreements (SLA). To address the above problems, we propose ContTune, a continuous tuning system for stream applications. It is equipped with a novel Big-small algorithm, in which the Big phase decouples the tuning from the topological graph by decomposing the job tuning problem into sub-problems that can be solved concurrently. We propose a conservative Bayesian Optimization (CBO) technique in the Small phase to speed up the tuning process by utilizing the previous observations. It leverages the state-of-the-art (SOTA) tuning method as conservative exploration to avoid SLA violations. Experimental results show that ContTune reduces up to 60.75% number of reconfigurations under synthetic workloads and up to 57.5% number of reconfigurations under real workloads, compared to the SOTA method DS2.

HoppingSketch: More Accurate Temporal Membership Query and Frequency Query

Nowadays, research on temporal membership queries is indispensable. Generally, temporal membership queries exist in two modalities: fixed windows and sliding windows, the latter having obvious advantages. The first sketch that implements temporal membership queries is the persistent Bloom filter (PBF). PBF has two shortcomings: it does not support sliding windows nor frequency queries. Here, we propose HoppingSketch to promote the original PBF. It is the first sketch that implements temporal membership queries for sliding windows. HoppingSketch is a general and efficient data stream processing framework, able to implement different tasks thanks to different atomic sketches. When the atomic sketches are Bloom filters and we apply them to PBF, HoppingSketch can achieve significantly higher temporal membership query accuracy than the original PBF. When the atomic sketches are sketches of Count-Min, Conservative Update, and Count, HoppingSketch can achieve more accurate frequency query than by applying PBF on the corresponding sketches. Our experimental results demonstrate the advantages of HoppingSketch compared with the state-of-the-art.

Performance Analysis of Lambda Architecture-Based Big-Data Systems on Air/Ground Surveillance Application with ADS-B Data.

This study introduces a novel methodology designed to assess the accuracy of data processing in the Lambda Architecture (LA), an advanced big-data framework qualified for processing streaming (data in motion) and batch (data at rest) data. Distinct from prior studies that have focused on hardware performance and scalability evaluations, our research uniquely targets the intricate aspects of data-processing accuracy within the various layers of LA. The salient contribution of this study lies in its empirical approach. For the first time, we provide empirical evidence that validates previously theoretical assertions about LA, which have remained largely unexamined due to LA's intricate design. Our methodology encompasses the evaluation of prospective technologies across all levels of LA, the examination of layer-specific design limitations, and the implementation of a uniform software development framework across multiple layers. Specifically, our methodology employs a unique set of metrics, including data latency and processing accuracy under various conditions, which serve as critical indicators of LA's accurate data-processing performance. Our findings compellingly illustrate LA's "eventual consistency". Despite potential transient inconsistencies during real-time processing in the Speed Layer (SL), the system ultimately converges to deliver precise and reliable results, as informed by the comprehensive computations of the Batch Layer (BL). This empirical validation not only confirms but also quantifies the claims posited by previous theoretical discourse, with our results indicating a 100% accuracy rate under various severe data-ingestion scenarios. We applied this methodology in a practical case study involving air/ground surveillance, a domain where data accuracy is paramount. This application demonstrates the effectiveness of the methodology using real-world data-intake scenarios, therefore distinguishing this study from hardware-centric evaluations. This study not only contributes to the existing body of knowledge on LA but also addresses a significant literature gap. By offering a novel, empirically supported methodology for testing LA, a methodology with potential applicability to other big-data architectures, this study sets a precedent for future research in this area, advancing beyond previous work that lacked empirical validation.

Automated Seizure Detection in Neonatal EEG using Signal Processing Algorithms

The crucial challenge of identifying seizures in newborn electroencephalogram (EEG) data is the subject of this research. (ElectroEncephaloGram) EEG analysis has traditionally been labor-intensive and subjective due to the heavy reliance on human inspection by skilled electrophysiologists and neurologists. The authors suggest a novel online automatic detection approach to get over these restrictions. This approach intends to open the door for effective and reliable early seizure detection, delivering a potential development in neonatal care and neurological research. It does this by utilizing the capabilities of signal processing and brain-inspired modeling’s suggested machine-learning method makes effective use of real-time EEG signal processing methods. Its processing of incoming EEG data streams makes it appropriate for neonates to be closely tracked in critical care situations. Automation of the detection method minimizes the need for manual inspection by highly specialized experts, allowing for more rapid responses and possibly increasing seizure management's overall results. Using metal electrodes, the electroencephalogram (EEG) analyses the electrical activity of the brain. European Data Format (EDF) BROWSER and EEG STUDIO are used to analyses EEG signals. Root Mean Square (RMS) and signal samples are performed by EDF BROWSER, as well as mean, frequency, and frequency from filtered output using a band-pass filter. The standard deviation and average frequency are examined by EEG STUDIO. Spike detection, frequency domain analysis, and nonlinear techniques are used in seizure prediction. EOG, EKG, and EMG artifacts produced by the eyes, the heart, and the muscles, respectively may be present in EEG readings.

Anomaly detection with a container-based stream processing framework for Industrial Internet of Things

Online anomaly detection is a key challenge for industrial internet of things (IIoT) applications, as anomalies may occur in data streams from sensors and cause losses or damages. However, most existing methods for online anomaly detection have limitations in efficiency, effectiveness and timeliness, especially with the massive and distributed data streams from IIoT devices. Therefore, developing a data stream processing framework to discover anomalies in time and ensure the proper operation of the system is an urgent issue for IIoT. In this paper, we propose a flexible stream processing framework that enables online anomaly detection for IIoT applications. The framework exploits a distributed computing architecture based on docker containers to improve flexibility, migration capability and customization. The framework also uses a central mediator to coordinate data stream processing tasks running on different docker nodes. Moreover, we develop a prediction-based online anomaly detection model that consists of batch model training and data stream anomaly detection processes. The model uses long short-term memory (LSTM) neural networks to predict data stream values and a dynamic sliding window method to model prediction errors and detect anomalies. We implement a case study to detect abnormal heating temperatures from an industrial heating plant and evaluate the performance of the proposed framework and anomaly detection model. The results show that our framework and model can achieve high accuracy and low latency in detecting anomalies, and they outperform existing methods in terms of scalability, efficiency and adaptability for IIoT applications.

I/O-efficient GPU-based acceleration of coherent dedispersion for pulsar observation

Coherent dedispersion is a frequency domain filtering method extensively used in big data stream processing to achieve high time resolution and pulse-profile precision in pulsar observation. However, the computational complexity and challenges associated with processing large-scale data streams have hindered its widespread adoption in modern telescopes. Overcoming these hurdles is crucial to realizing wide-band real-time coherent dedispersion. In this paper, we propose a highly parallel and high-I/O throughput implementation of coherent dedispersion on GPU, specifically designed to support consecutive dispersion measure trials. Our primary objective is to achieve real-time coherent dedispersion, enabling precise high-time resolution and pulse-profile analysis in modern telescopes. To accomplish this, we leverage the power of GPU parallelism and employ advanced multi-threading techniques to optimize both computation and I/O throughput. Experiments show that our method achieves performance gains of 3.96x and 4.65x in processing time and IO throughput, respectively, when compared with the baseline work, the Coherent Dispersion Measure Trials (CDMT). Finally, the proposed techniques for improving computation parallelism and I/O efficiency are general that can provide valuable insights and serve as a reference for optimizing related algorithms in the field of big data stream processing.

Operator placement for data stream processing based on publisher/subscriber in hybrid cloud-fog-edge infrastructure

Operator placement for data stream processing based on publisher/subscriber in hybrid cloud-fog-edge infrastructure

Online event recognition over noisy data streams

Composite event recognition (CER) systems process streams of sensor data and infer composite events of interest by means of pattern matching. Data uncertainty is frequent in CER applications and results in erroneous detection. To support streaming applications, we present oPIECbd, an extension of oPIEC with a bounded memory, leveraging interval duration statistics to resolve memory conflicts. oPIECbd may achieve comparable predictive accuracy to batch reasoning, avoiding the prohibitive cost of such reasoning. Furthermore, the use of interval duration statistics allows oPIECbd to outperform significantly earlier versions of bounded oPIEC. The empirical evaluation demonstrates the efficacy of oPIECbd on a benchmark activity recognition dataset, as well as real data streams from the field of maritime situational awareness.

Research on Urban Monitoring Network Management Platform based on AI Data Stream Processing

In recent years, with the upsurge of the continuous development of high-tech information industry, the security industry has also made a new leap in technology, products, industry and so on. The traditional single small-scale monitoring and management software obviously cannot meet the actual needs of users in terms of system scale and functional requirements. This paper puts forward a network management platform for urban monitoring based on AI data stream processing. With the help of comprehensive processing and analysis of urban complex information, through network remote monitoring means, information supervision and public opinion trends are used to control social environmental security, timely warning and response to unexpected accidents, so as to ensure that governance departments can fully control and prevent possible problems such as road traffic, population distribution and disaster emergency.

Enabling Real-Time Remote Monitoring of Ships by Lossless Protocol Transformations

This paper uses data processing techniques to reduce the required transmission bandwidth in ship-to-shore communications. The proposed framework (ONline Efficient Sources Transmission Optimizer -) leverages state-of-the-art technologies and novel algorithms to automatically optimize transmissions under structural (e.g., available bandwidth, fixed packet overhead) and user-defined (e.g., maximum latency) constraints. In addition, authenticates and encrypts the communication between the ship and the shore via mainstream free and open-source software components. Initially, we present the abstract mathematical formulation of the problem, with its assumptions, goal function, constraints, and significant quantities. Then, we introduce the architecture of a system capable of continuously estimating the compressibility, processing and transmission time of streaming data. Such estimations allow to calculate and apply optimal parameters for achieving the best compression ratio. Lastly, using a prototypical implementation, we evaluate the system performance with a Class B ship simulator on two realistic use cases. Our experiments show an excellent compression ratio with maritime protocols (more than 40:1) and a limited latency impact, demonstrating the approach’s viability.

Проблемы масштабирования компонентов в системах потоковой обработки данных

The article examines the existing problems of scaling components in streaming data processing systems. The algorithm proposed by the authors significantly reduces the number of operations for creating and removing components when scaling the system. The algorithm is based on linear regression. For the proposed algorithm, the practical load on the system is simulated to confirm the theoretical results obtained.

Polarimetric airborne scientific instrument, mark 2, an ice‐sounding airborne synthetic aperture radar for subglacial 3D imagery

AbstractPolarimetric Airborne Scientific INstrument, mark 2 (PASIN2) is a 150 MHz coherent pulsed radar with the purpose of deep ice sounding for bedrock, subglacial channels and ice‐water interface detection in Antarctica. It is designed and operated by the British Antarctic Survey from 2014. With multiple antennas, oriented along and across‐track, for transmission and reception, it enables polarimetric 3D estimation of the ice base with a single pass, reducing the gridding density of the survey paths. The off‐line data processing stream consists of channel calibration; 2D synthetic aperture radar (SAR) imaging based on back‐projection, for along‐track and range dimensions; and finally, a direction of arrival estimation (DoA) of the remaining across‐track angle, by modifying the non‐linear MUSIC algorithm. Calibration flights, during the Antarctic Summer campaigns in 16/17 and 19/20 seasons, assessed and validated the instrument and processing performances. Imaging flights over ice streams and ice shelves close to grounding lines demonstrate the 3D sensing capabilities. By resolving directional ambiguities and accounting for reflector across‐track location, the true ice thickness and bed elevation are obtained, thereby removing the error of the usual assumption of vertical DoA, that greatly influence the output of flow models of ice dynamics.