Spark Streaming Research Articles

Differentially Private Stream Processing at Scale

We design, to the best of our knowledge, the first differentially private (DP) stream aggregation processing system at scale. Our system - Differential Privacy SQL Pipelines (DP-SQLP) - is built using a streaming framework similar to Spark streaming, and is built on top of the Spanner database and the F1 query engine from Google. Towards designing DP-SQLP we make both algorithmic and systemic advances, namely, we (i) design a novel (user-level) DP key selection algorithm that can operate on an unbounded set of possible keys, and can scale to one billion keys that users have contributed, (ii) design a preemptive execution scheme for DP key selection that avoids enumerating all the keys at each triggering time, and (iii) use algorithmic techniques from DP continual observation to release a continual DP histogram of user contributions to different keys over the stream length. We empirically demonstrate the efficacy by obtaining at least 16× reduction in error over meaningful baselines we consider. We implemented a streaming differentially private user impressions for Google Shopping with DP-SQLP. The streaming DP algorithms are further applied to Google Trends.

Event Driven Data Architecture: Design and Implementation with Kinesis and Spark Streaming

Event Driven Data Architecture: Design and Implementation with Kinesis and Spark Streaming

Digital banking fortification: a real-time isolation forest architecture for detecting online transaction fraud

Since the use of the Internet has increased exponentially, numerous organizations, including the financial industry, offer services online. As a result, financial scams are expanding in quantity and complexity worldwide, resulting in massive revenue losses and making digital fraudulent transactions a severe issue. Abnormal attempts and illegal access are instances of these dangers that fraudulent activity detection systems must identify. Machine learning and data mining approaches have been extensively used to address this issue in recent years. However, these approaches must be enhanced regarding real-time detection speed, tackling enormous amounts of data, and finding undiscovered attack patterns. Consequently, the present study provides a real-time architecture for averting and identifying digital transaction fraud, which relies on the Isolation Forest (IForest) approach and big data analytic tools, including Spark Streaming, sparkling water, Kafka, and PostgreSQL. This architecture seeks to improve present detection strategies by increasing accuracy for detection when considering enormous amounts of data. Two real datasets of online transactional fraud are used to assess the proposed architecture, and the findings are compared to relevant studies. The investigation results showed that IForest performed flawlessly, achieving an accuracy of 0.99 in two datasets.

Streamlining trajectory map-matching: a framework leveraging spark and GPU-based stream processing

Real-time online trajectory map-matching has emerged as a critical component in the era of location-based services (LBS) and intelligent transportation systems (ITS). It refers to the process of aligning a user’s GPS trajectory data with the corresponding road network in real-time. This technology has significant implications for various industries and applications. As our reliance on LBS and ITS continues to grow, the demand for faster, more accurate, and more reliable trajectory map-matching methods becomes increasingly important. Contemporary online map-matching predominantly employs stream processing techniques. Based on stream processing frameworks, we propose a heterogeneous hybrid architecture for map-matching. The architecture integrates Spark Streaming and graphics processing unit (GPU) heterogeneous computing for the first time. The hidden Markov model is employed as the map-matching algorithm, and Spark Streaming serves as the distributed processing platform. We conduct map-matching experiments using a GPS taxi trajectory dataset in Beijing’s Haidian District. The results demonstrate that in comparison to other analogous research, our framework’s performance has increased by over ten times, possessing a superior data processing capability and lower latency. This research provides a novel approach of stream-based heterogeneous computation for processing large-scale geographic data.

Real-Time Online Banking Fraud Detection Model by Unsupervised Learning Fusion

Digital trades and payments are becoming increasingly popular, as they typically entail monetary transactions. This not only makes electronic transactions more convenient for the end customer, but it also raises the likelihood of fraud. An adequate fraud detection system with a cutting-edge model is critical to minimizing fraud costs. Identifying fraud at the ideal time entails establishing and setting up ubiquitous systems to consume and analyze massive amounts of streaming data. Recent advances in data analytics methods and introducing open-source technology for big data storage and processing opened new options for detecting fraud. This study aims to tackle this critical issue by providing a newly real-time e-transaction fraud detection schema that consolidates the advantages of both unsupervised learners, including autoencoder and extended isolation forests, with cutting-edge big data gadgets such as Spark streaming and sparkling water. It addresses the shortage of non-fraudulent instances and handles the excessive dimension of the set of features. On two real-world transactional datasets, we assess our suggested technique. Compared with other current fraud identification systems, our methodology delivers an elevated accuracy yield of 99%. Furthermore, it outperforms state-of-the-art approaches in reliably identifying fraudulent samples. Doi: 10.28991/HIJ-2024-05-01-014 Full Text: PDF

Advancements in Data Ingestion: Building High-Throughput Pipelines with Kafka and Spark Streaming

Advancements in Data Ingestion: Building High-Throughput Pipelines with Kafka and Spark Streaming

Simulating real-time tweet sentiment analysis by different machine learning methods based on spark

Sentiment analysis is essential since it benefits many fields, such as politics and economics. Because much data is generated every moment, a real-time processing system can efficiently analyze sentiment. This paper uses Spark to simulate real-time tweet sentiment analysis, and compares the performances of three machine learning methods, Logistic Regression, Naive Bayes, and Decision Tree. The idea of the real-time tweet sentiment analysis system is using Spark Streaming to send a batch of tweets every fixed period to a machine learning pipeline to predict the emotions of tweets. In the pipeline, tweets will be tokenized first, then the stop words in tweets will be removed. After that, the author uses TF-IDF to extract features, transferring data from unstructured to structured. The last stage is using the machine learning method to predict the sentiments of tweets. By comparing, Logistic Regression has the best performance, and the second one is Naive Bayes, Decision Tree performs not as well as the other two methods.

An approach towards removal of data heterogeneity in SDN-based IoT framework

IoT is an emerging technology that provides innovative solutions to our day-to-day life challenges. With billions of diverse smart objects, IoT produces gigantic amount of heterogeneous data that mostly need to be processed in real time and shared across systems and applications. For IoT data to be truly accessable and interpretable, the data heterogeneity problem needs to be resolved first. Semantic technologies is one of the most used solutions for data integration across systems. Data represented as knowledge graphs enables easier querying and matching. To address this, we propose an SDN based IoT architectural framework utilizing semantics for data integration and sharing using Kafka and Spark Streaming for real-time data handling. One main contribution of this work is that any consistent IoT ontology, defined using Web Ontology Language(OWL) which explicitly specify its classes, objects and data properties, can be used to convert the stream data into RDF format automatically using Natural Language Processing endeavor. SDN helps in dynamically locating the data producing devices such as sensors for providing flexible data routing. We have presented a testbed experimentation framework and evaluated the results with simulated virtual sensors. The homogeneous results from diverse datasets was generated in acceptable minimal time. A use-case for Air Quality Monitoring is given and SPARQL queries executed have shown satisfactory response time with over thousands of triples.

A Spark Streaming-Based Early Warning Model for Gas Concentration Prediction

The prediction and early warning efficiency of mine gas concentrations are important for intelligent monitoring of daily gas concentrations in coal mines. It is used as an important means for ensuring the safe and stable operation of coal mines. This study proposes an early warning model for gas concentration prediction involving the Spark Streaming framework (SSF). The model incorporates a particle swarm optimisation algorithm (PSO) and a gated recurrent unit (GRU) model in the SSF, and further experimental analysis is carried out on the basis of optimising the model parameters. The operational efficiency of the model is validated using a control variable approach, and the prediction and warning errors is verified using MAE, RMSE and R2. The results show that the model is able to predict and warn of the gas concentration with high efficiency and high accuracy. It also features fast data processing and fault tolerance, which provides a new idea to continue improving the gas concentration prediction and warning efficiency and some theoretical and technical support for intelligent gas monitoring in coal mines.

An innovative parameter optimization of Spark Streaming based on D3QN with Gaussian process regression.

Nowadays, Spark Streaming, a computing framework based on Spark, is widely used to process streaming data such as social media data, IoT sensor data or web logs. Due to the extensive utilization of streaming media data analysis, performance optimization for Spark Streaming has gradually developed into a popular research topic. Several methods for enhancing Spark Streaming's performance include task scheduling, resource allocation and data skew optimization, which primarily focus on how to manually tune the parameter configuration. However, it is indeed very challenging and inefficient to adjust more than 200 parameters by means of continuous debugging. In this paper, we propose an improved dueling double deep Q-network (DQN) technique for parameter tuning, which can significantly improve the performance of Spark Streaming. This approach fuses reinforcement learning and Gaussian process regression to cut down on the number of iterations and speed convergence dramatically. The experimental results demonstrate that the performance of the dueling double DQN method with Gaussian process regression can be enhanced by up to 30.24%.

Coalbed methane concentration prediction and early-warning in fully mechanized mining face based on deep learning

Coalbed methane (CBM) disasters are a major safety problem in coal mining. CBM concentration prediction and early-warning technology play a vital role in the prevention and control of CBM disasters. Traditional prediction methods have some shortcomings such as unreasonable data analysis, inability to predict in real time, long prediction time, and overfitting. This study uses a large amount of coal mine CBM data and develops a fast and high-precision CBM concentration prediction method based on deep learning theory, which can be used for CBM disaster early-warning. The proposed method has the following advantages: (1) it combines three exponential smoothing methods, the autoregressive model, wavelet domain denoising method, and principal component analysis. This method optimizes data with outliers, missing values, and noise. (2) Particle swarm optimization and genetic algorithm are used to optimize the network parameters of the gated recurrent unit. The application and verification of the optimized model show that the running time and accuracy are significantly improved. (3) By combining the optimized prediction model with Spark streaming, an early-warning system is developed, which can complete the efficient early-warning of CBM concentration within 8 s. The proposed method provides decision-making support for mine safety and CBM disaster prevention and control.

De privacy encryption and extraction model of smart grid data based on Spark Streaming

In order to ensure the safe and efficient application of smart grid data, this paper studies the de privacy encryption and extraction of smart grid data based on Spark Streaming, and accurately completes the de privacy encryption and extraction of smart grid data. The construction of the model mainly includes two parts, which are de privacy decryption processing and data extraction. After data privacy is processed by using collaborative cognitive model, the data is processed by Spark Streaming, including data cleaning, data reduction, data standardization, etc. Then the data clustering center is extracted by using genetic neural algorithm. Finally, the similarity between the data set and the clustering center is calculated, and the data with the greatest similarity is selected to realize data extraction. The test results show that: the model can quickly complete data cleaning, effectively identify the abnormal data information in the data, the recognition rate is 99.72%, and complete data clustering in a few iterations, so as to realize the de privacy encryption and extraction of smart grid data.

Empirical analysis of chronic disease dataset for multiclass classification using optimal feature selection based hybrid model with spark streaming

Recent advancement in the field of pervasive healthcare monitoring systems causes the generation of a huge amount of lifelog data in real-time. Chronic diseases are one of the most serious health challenges in developing and developed countries. According to WHO, this accounts for 73% of all deaths and 60% of the global burden of diseases. Chronic disease classification models are now harnessing the potential of lifelog data to explore better healthcare practices. This paper basically constructs an optimal feature selection-based hybrid model which is designed by integrating k-means clustering for handling unlabeled data, applying Synthetic Minority Oversampling Technique (SMOTE), selecting optimal features using PCA, and applying Random Forest (RF) classifier to classify chronic diseases. Since lifelog data analysis is crucial due to its sensitive nature; thus the conventional classification models show limited performance. Therefore, designing new classifiers for the classification of chronic diseases using lifelog data is the need of the age. The vital part of building a good model depends on pre-processing of the dataset, identifying important features, and then training a learning algorithm with suitable hyper parameters for better performance. The proposed approach improves the performance of existing methods using a series of steps such as (i) removing redundant or invalid instances, (ii) making the data labeled using clustering and partitioning the data into classes, (iii) making the classes balance to avoid biased result across majority class (iv) identifying the suitable subset of features by applying either some domain knowledge or selection algorithm, (v) hyper parameter tuning for models to get best results (vi) developing a new hybrid model i.e. optimal feature selection based unsupervised random forest classifier (OFS-URFC), which gives best results in binary classification as well as multiclass classification and (vii) performance of this newly designed classifier is also evaluated using Spark streaming environment for processing the data in real time fashion. For this purpose, two-time series datasets are used in the experiment to compute the accuracy, recall, precision, and f1-score etc. The experimental analysis proves the suitability of the proposed approach as compared to the conventional classifiers and our newly constructed model achieved highest accuracy and reduced training complexity among all.

QuakeFlow: a scalable machine-learning-based earthquake monitoring workflow with cloud computing

SUMMARY Earthquake monitoring workflows are designed to detect earthquake signals and to determine source characteristics from continuous waveform data. Recent developments in deep learning seismology have been used to improve tasks within earthquake monitoring workflows that allow the fast and accurate detection of up to orders of magnitude more small events than are present in conventional catalogues. To facilitate the application of machine-learning algorithms to large-volume seismic records at scale, we developed a cloud-based earthquake monitoring workflow, QuakeFlow, which applies multiple processing steps to generate earthquake catalogues from raw seismic data. QuakeFlow uses a deep learning model, PhaseNet, for picking P/S phases and a machine learning model, GaMMA, for phase association with approximate earthquake location and magnitude. Each component in QuakeFlow is containerized, allowing straightforward updates to the pipeline with new deep learning/machine learning models, as well as the ability to add new components, such as earthquake relocation algorithms. We built QuakeFlow in Kubernetes to make it auto-scale for large data sets and to make it easy to deploy on cloud platforms, which enables large-scale parallel processing. We used QuakeFlow to process three years of continuous archived data from Puerto Rico within a few hours, and found more than a factor of ten more events that occurred on much the same structures as previously known seismicity. We applied Quakeflow to monitoring earthquakes in Hawaii and found over an order of magnitude more events than are in the standard catalogue, including many events that illuminate the deep structure of the magmatic system. We also added Kafka and Spark streaming to deliver real-time earthquake monitoring results. QuakeFlow is an effective and efficient approach both for improving real-time earthquake monitoring and for mining archived seismic data sets.

A Gas Concentration Prediction Method Driven by a Spark Streaming Framework

In the traditional coal-mine gas-concentration prediction process, problems such as low timeliness of data and low efficiency of the prediction model in learning data features result in low accuracy of the final prediction. To solve these problems, a gas-concentration prediction method driven by the Spark Streaming framework is proposed. In this research study, the Spark Streaming framework, autoregressive integrated moving average (ARIMA) model and support vector machine (SVM) model are used to construct a new prediction model called the SPARS model. The Spark Streaming framework is used to process large batches of real-time streaming data in a short period of time, and the model can be used to intermittently update and optimize the prediction model so that the model can fully learn the characteristics of the data. At the same time, the advantages of the ARIMA model and SVM model for processing linear data and nonlinear data are combined to improve the model’s prediction efficiency and fully reflect the timeliness of gas prediction. Finally, the proposed prediction model is verified using gas data collected on site. The optimal learning time for the SPARS model in predicting this set of data is determined, and a comparative analysis of the prediction results obtained from the ARIMA, SVM and other models fully confirms that high-precision prediction results can be obtained using the SPARS model. The proposed model can be used to realize scientific and accurate real-time prediction and analyses of coal-mine gas concentrations and provides a new idea for realizing real-time and accurate gas prediction in coal mines.

Prediction of Cardiovascular Disease Using PySpark Techniques

Abstract: On a day after day, human life is affected by differing kinds of diseases that is why their life is in distress. cardiovascular disease may be a generic class of disease that's effective in spreading infections and notably, it affects the heart and veins. it's determined that vessel diseases have become modest in old individuals besides in children too. it's terribly requisite to portend this sort of illness within the starting phases; many varieties of tests square measure used for diagnosticating these ailments. This implementation has been done by employing a big data tool that's Apache Spark and victimization spark's MLlib and PySpark libraries that square measure integrated with it. Apache Spark is among the foremost wide used big data technologies, and it's a stack of some libraries that are Spark SQL, Spark MLlib, Spark Streaming, etc. This analysis work aims to create a prediction model to predict whether or not people have cardiovascular disease or not, using machine learning classification techniques that embrace logistic regression, decision tree, random forest to enhance the performance of models. They compared the analysis of all applied machine learning models. The results obtained are compared with the results of existing models within the same domain and located to be improved. Keywords: heart, blood vessels, Xampp server, data analytics, cardiovascular diseases.

Performance Evaluation Analysis of Spark Streaming Backpressure for Data-Intensive Pipelines.

A significant rise in the adoption of streaming applications has changed the decision-making processes in the last decade. This movement has led to the emergence of several Big Data technologies for in-memory processing, such as the systems Apache Storm, Spark, Heron, Samza, Flink, and others. Spark Streaming, a widespread open-source implementation, processes data-intensive applications that often require large amounts of memory. However, Spark Unified Memory Manager cannot properly manage sudden or intensive data surges and their related in-memory caching needs, resulting in performance and throughput degradation, high latency, a large number of garbage collection operations, out-of-memory issues, and data loss. This work presents a comprehensive performance evaluation of Spark Streaming backpressure to investigate the hypothesis that it could support data-intensive pipelines under specific pressure requirements. The results reveal that backpressure is suitable only for small and medium pipelines for stateless and stateful applications. Furthermore, it points out the Spark Streaming limitations that lead to in-memory-based issues for data-intensive pipelines and stateful applications. In addition, the work indicates potential solutions.

A Framework for Similarity Search in Streaming Time Series based on Spark Streaming

A Framework for Similarity Search in Streaming Time Series based on Spark Streaming

A Fast Cross-Layer Dynamic Power Estimation Method by Tracking Cycle-Accurate Activity Factors With Spark Streaming

The advent of autonomous power-limited systems poses a new challenge for early design space exploration. The existing architecture-level power evaluation tools lose accuracy due to ignoring features of circuit-level behaviors and influences of process, voltage, and temperature variations. Although power estimations based on SPICE or PrimeTime PX (PTPX) are accurate enough, they come at the cost of long simulation time and are available only in very late phases of design flow. In this article, a fast and accurate dynamic power evaluation method is proposed, which estimates activity factors at the circuit level. The impact of process variation at the gate level is considered through the proposed effective capacitance model. Activity factors are then estimated by the model and input vectors of the circuit. Input vectors are generated by architecture-level simulations in the form of streaming. For real-time and high-speed power evaluation, a data streaming framework is proposed for massive parallelism. The cross-layer estimation is verified based on the functional units of PULPino processor running SPEC CPU2006 benchmarks. Compared with the SPICE results using SMIC 28-nm PDK, our cycle-by-cycle dynamic power analysis shows an average error of 5.4%. Meanwhile, our approach realizes 65.2% faster than the traditional PTPX simulation and 48.8% faster compared with the state-of-art cross-level evaluation method.

Real-Time intelligent Elevator Monitoring and Diagnosis: Case Studies and Solutions with applications using Artificial Intelligence

Under "Industry 4.0", the implementation and application of big data and Artificial Intelligence (AI) technology in the elevator industry has become more and more common. With the surge of the elevator operation data and higher requirements for its real-time performance, the traditional elevator fault monitoring is inaccurate, which needs to be solved urgently. In this paper, a fault monitoring and diagnosis method of elevator based on AI and big data is proposed. Firstly, the elevator system and its fault types and causes are analyzed. Then in order to select the best big data processing tools, the performance of Flink and Spark Streaming is compared. The results show that Flink features faster computing speed and is more suitable for handling big data. Thirdly, a pattern recognition algorithm based on finite state machine (FSM) is proposed to monitor the running state for whole elevator control system. Finally, simulation experiment has been made.