Spark Machine Learning Research Articles

The investigation of traffic flow prediction and optimization based on Spark

Spark and flink have made great strides in big data processing in recent years. Although the current spark can process a large amount of data at the same time, it still has a lot of shortcomings in the transparency, credibility, and practicality of the model. This paper provides a comprehensive overview of how to tackle the performance bottlenecks, insufficient model interpretability, and lack of regional adaptability faced by Spark and Flink in big data processing. It discusses the introduction of interpretable algorithms such as SHAP and LIME to enhance the transparency and user trust of neural network models. Then it discusses how to combine time-aware transfer learning and Geographic Information Systems (GIS) technology to enhance the technology generalization and adaptability of Spark machine learning models. Time-aware transfer learning uses historical datas temporal evolution to ensure models perform well in new time periods or scenarios, while GIS technology enables more precise predictions and analyses based on geographical data, enhancing spatial adaptability. Lastly, the study explores hybrid processing strategies by integrating Apache Flink, Kafka Streams, and Spark batch processing frameworks. This approach not only facilitates efficient real-time data processing and detailed analysis but also enhances the models flexibility and processing capabilities in complex data scenarios. By integrating these techniques, it is possible to improve the efficiency and effectiveness of big data processing frameworks in addressing complex real-world challenges, thereby advancing technology and application development in related fields.

Terrorist Activities Detection via Social Media Using Machine Learning

Abstract: Perhaps the richest source of human generated text input is social media. Internet users' opinions, feedback, and criticisms represent their attitudes and feelings towards specific topics and concerns. The sheer volume of such information makes reading it difficult for any group of people. As a result, social media has become a major instrument for propagating their views and influencing or enticing people to join their terrorist actions in general. Social media is the most frequent and straightforward approach to contact a large number of individuals in a short period of time. This research focuses on the construction of a system that can detect terrorism-supporting tweets automatically through real-time analytics using the Apache Spark machine learning framework. The proposed approach is completely reliant on training data and attempts to enhance accuracy

DETECTION OF CREDIT CARD FRAUD IN REAL TIME USING SPARK ML

Assume you have access to a credit card. Your prior spending habits will be investigated. For example, how much money you spend, where it is spent, how frequently it is spent, and what you buy. If your current credit card transaction differs from your historical spending patterns, it is suspected of fraud; otherwise, it is considered as a genuine transaction, and fraud transactions are warned in the dashboard. The predictions are expected to be based on millions of transactions. As result, distributed frameworks that can expand as the number of transactions grows are used. Spark this system for real-time identification of credit card theft is built with Kafka and Cassandra. Preprocessing is accomplished through the use of Spark Machine Learning Pipeline Stages such as String Indexer and Vector.

Analyzing SQL payloads using logistic regression in a big data environment

Abstract Protecting big data from attacks on large organizations is essential because of how vital such data are to organizations and individuals. Moreover, such data can be put at risk when attackers gain unauthorized access to information and use it in illegal ways. One of the most common such attacks is the structured query language injection attack (SQLIA). This attack is a vulnerability attack that allows attackers to illegally access a database quickly and easily by manipulating structured query language (SQL) queries, especially when dealing with a big data environment. To address these risks, this study aims to build an approach that acts as a middle protection layer between the client and database server layers and reduces the time consumed to classify the SQL payload sent from the user layer. The proposed method involves training a model by using a machine learning (ML) technique for logistic regression with the Spark ML library that handles big data. An experiment was conducted using the SQLI dataset. Results show that the proposed approach achieved an accuracy of 99.04, a precision of 98.87, a recall of 99.89, and an F-score of 99.04. The time taken to identify and prevent SQLIA is 0.05 s. Our approach can protect the data by using the middle layer. Moreover, using the Spark ML library with ML algorithms gives better accuracy and shortens the time required to determine the type of request sent from the user layer.

Retracted] Storage Method for Medical and Health Big Data Based on Distributed Sensor Network

Monitoring and collecting medical data using embedded medical diagnostic devices with multiple sensors and sending these actual measured data to the corresponding health monitoring centers using multipurpose wireless networks to take necessary measures to coordinate with family medical service centers and regional medical service departments is a popular medical big data architecture. However, healthcare big data is characterized by large data volume, fast growth, multimodality, high value and privacy, etc. How to organize and manage it in a unified and efficient way is an important research direction at present. In response to the problems of low balance and poor security in the storage of data collected by distributed sensor networks in healthcare systems, we propose a distributed storage algorithm for big data in healthcare systems. The platform adopts Hadoop distributed file system and distributed file storage framework as the healthcare big data storage solution, and implements data integration, multidimensional data query and analysis mining components based on Spark‐SQL data query tool, Spark machine learning algorithm library and its mining and analysis pipeline development, respectively. The distributed storage model of big data and three data storage levels are constructed using cloud storage architecture, and the data storage intensity as well as levels are calculated by high data access in the upper level, data connection in the middle level, and data archiving in the lower level according to the set known data granularity, odds, and elasticity to realize big data storage. It is experimentally verified that the above algorithm has high distribution balance and low load balance in the storage process.

Performance Evaluation of Data-driven Intelligent Algorithms for Big data Ecosystem.

Artificial intelligence, specifically machine learning, has been applied in a variety of methods by the research group to transform several data sources into valuable facts and understanding, allowing for superior pattern identification skills. Machine learning algorithms on huge and complicated data sets, computationally expensive on the other hand, processing requires hardware and logical resources, such as space, CPU, and memory. As the amount of data created daily reaches quintillion bytes, A complex big data infrastructure becomes more and more relevant. Apache Spark Machine learning library (ML-lib) is a famous platform used for big data analysis, it includes several useful features for machine learning applications, involving regression, classification, and dimension reduction, as well as clustering and features extraction. In this contribution, we consider Apache Spark ML-lib as a computationally independent machine learning library, which is open-source, distributed, scalable, and platform. We have evaluated and compared several ML algorithms to analyze the platform’s qualities, compared Apache Spark ML-lib against Rapid Miner and Sklearn, which are two additional Big data and machine learning processing platforms. Logistic Classifier (LC), Decision Tree Classifier (DTc), Random Forest Classifier (RFC), and Gradient Boosted Tree Classifier (GBTC) are four machine learning algorithms that are compared across platforms. In addition, we have tested general regression methods such as Linear Regressor (LR), Decision Tree Regressor (DTR), Random Forest Regressor (RFR), and Gradient Boosted Tree Regressor (GBTR) on SUSY and Higgs datasets. Moreover, We have evaluated the unsupervised learning methods like K-means and Gaussian Mixer Models on the data set SUSY and Hepmass to determine the robustness of PySpark, in comparison with the classification and regression models. We used ”SUSY,” ”HIGGS,” ”BANK,” and ”HEPMASS” dataset from the UCI data repository. We also talk about recent developments in the research into Big Data machines and provide future research directions.

Distributed big data analysis using spark parallel data processing

Nowadays, the big data marketplace is rising rapidly. The big challenge is finding a system that can store and handle a huge size of data and then processing that huge data for mining the hidden knowledge. This paper proposed a comprehensive system that is used for improving big data analysis performance. It contains a fast big data processing engine using Apache Spark and a big data storage environment using Apache Hadoop. The system tests about 11 Gigabytes of text data which are collected from multiple sources for sentiment analysis. Three different machine learning (ML) algorithms are used in this system which is already supported by the Spark ML package. The system programs were written in Java and Scala programming languages and the constructed model consists of the classification algorithms as well as the pre-processing steps in a figure of ML pipeline. The proposed system was implemented in both central and distributed data processing. Moreover, some datasets manipulation manners have been applied in the system tests to check which manner provides the best accuracy and time performance. The results showed that the system works efficiently for treating big data, it gains excellent accuracy with fast execution time especially in the distributed data nodes.

Performance analysis of disease diagnostic system using IoMT and real‐time data analytics

AbstractIn this article, the Internet of Medical Things (IoMT) framework based on Apache Spark big data processing technology is proposed for real‐time analysis of health data obtained from wireless body area networks (WBANs), which is one of the most important components of IoMT. The proposed framework consists of four layers: data source, data collection, data analytics and visualization. In addition, the proposed IoMT framework is presented with two different disease prediction scenarios, diabetes and heart disease. Diabetes and heart disease prediction processes are carried out using the random forest (RF), logistic regression (LR) and support vector machine (SVM) algorithms belonging to the Apache Spark machine learning library (MLlib). The analysis of health data generated in WBANs takes place in real‐time in the Apache Spark‐based data analytics layer. In this study, the performances of MLlib algorithms in the real‐time model developed for heart and diabetes disease are examined. The SVM algorithm with an accuracy rate of 93.33% for heart disease and the LR algorithm with an accuracy rate of 78.89% for diabetes are found to provide the best performances.

Flow-Based Programming for Machine Learning

Machine Learning (ML) has gained prominence and has tremendous applications in fields like medicine, biology, geography and astrophysics, to name a few. Arguably, in such areas, it is used by domain experts, who are not necessarily skilled-programmers. Thus, it presents a steep learning curve for such domain experts in programming ML applications. To overcome this and foster widespread adoption of ML techniques, we propose to equip them with domain-specific graphical tools. Such tools, based on the principles of flow-based programming paradigm, would support the graphical composition of ML applications at a higher level of abstraction and auto-generation of target code. Accordingly, (i) we have modelled ML algorithms as composable components; (ii) described an approach to parse a flow created by connecting several such composable components and use an API-based code generation technique to generate the ML application. To demonstrate the feasibility of our conceptual approach, we have modelled the APIs of Apache Spark ML as composable components and validated it in three use-cases. The use-cases are designed to capture the ease of program specification at a higher abstraction level, easy parametrisation of ML APIs, auto-generation of the ML application and auto-validation of the generated model for better prediction accuracy.

Assessing naive Bayes and support vector machine performance in sentiment classification on a big data platform

<p><span lang="EN-US">Nowadays, mining user reviews becomes a very useful mean for decision making in several areas. Traditionally, machine learning algorithms have been widely and effectively used to analyze user’s opinions on a limited volume of data. In the case of massive data, powerful hardware resources (CPU, memory, and storage) are essential for dealing with the whole data processing phases including, collection, pre-processing, and learning in an optimal time. Several big data technologies have emerged to efficiently process massive data, like Apache Spark, which is a distributed framework for data processing that provides libraries implementing several machine learning algorithms. In order to evaluate the performance of Apache Spark's machine learning library (MLlib) on a large volume of data, classification accuracies and processing time of two machine learning algorithms implemented in spark: naive </span><span>B</span><span lang="EN-US">ayes and support vector machine (SVM) are compared to the performance achieved by the standard implementation of these two algorithms on large different size datasets built from movie reviews. The results of our experiment show that the performance of classifiers running under spark is higher than traditional ones and reaches F-measure greater than 84%. At the same time, we found that under spark framework, the learning time is relatively low.</span></p>

Large scale data analysis using MLlib

Recent advancements in the internet, social media, and internet of things (IoT) devices have significantly increased the amount of data generated in a variety of formats. The data must be converted into formats that is easily handled by the data analysis techniques. It is mathematically and physically expensive to apply machine learning algorithms to big and complicated data sets. It is a resource-intensive process that necessitates a huge amount of logical and physical resources. Machine learning is a sophisticated data analytics technology that has gained in importance as a result of the massive amount of data generated daily that needs to be examined. Apache Spark machine learning library (MLlib) is one of the big data analysis platforms that provides a variety of outstanding functions for various machine learning tasks, spanning from classification to regression and dimension reduction. From a computational standpoint, this research investigated Apache Spark MLlib 2.0 as an open source, autonomous, scalable, and distributed learning library. Several real-world machine learning experiments are carried out in order to evaluate the properties of the platform on a qualitative and quantitative level. Some of the fundamental concepts and approaches for developing a scalable data model in a distributed environment are also discussed.

An approach to on-stream DDoS blitz detection using machine learning algorithms

Distributed Denial of service (DDoS) attacks is an enormous threat to today’s cyber world, cyber networks are compromised by the attackers to distribute attacks in a large volume by denying the service to legitimate users. The toughest and challenging task in today’s network and network security engineers is to identify compromised traffic (attacked) and legitimate (normal) traffic. The main goal of the paper is to detect DDos attacks using classification algorithms. To achieve the goal the proposed system uses attacking tool to initiate attacks using Loic attacking tool with the data set extracted from open source tool Wireshark and transferring the dataset to apache Spark for detection analysis. The system also uses Apache spark machine learning algorithms (MLib), classification algorithms to classify the dataset. We use Naive Bayes, KNN and Random forest classification algorithms to classify normal traffic and attacked traffic. Our system is capable of detecting attacks with respect to any traffic protocols ICMP, TCP, or UDP. The accuracy of detection is compared on three classification algorithms and noted that random forest gives the accuracy of 96.75%.

Behavior anomaly detection based on big data analysis of Internet of Things

The technical requirements of behavior anomaly detection are higher and higher. Using the Internet of things technology combined with a variety of big data analysis algorithms, we can achieve accurate behavior anomaly detection by classifying behavior data sets to a large extent. In this paper, PLA - PRF (parallel random forest) algorithm is used to realize the behavior anomaly detection model of Internet of things integrating big data analysis. In behavior detection, the PRF algorithm and DFS algorithm are compared in the case of a different number of decision trees. The results show that, compared with DRF algorithm, PLA-PRF, SPARK MLRF(Spark Machine Learning Random Forests) and PRF algorithm perform better on the four datasets, with kappa values increased by about 3.13%, 2.56% and 1.98% respectively. In contrast, PLA-PRF algorithm has higher accuracy in the case of a small sample size. With the increase of sample size, the accuracy of behavior anomaly detection gradually decreases; because the algorithm is in subspace in the process of construction, some high pheromone features are abandoned, which makes the new spatial information of features insufficient, resulting in the decision tree training process does not learn the inherent laws of abandoned data. Compared with spark MLRF and DRF, PLA-PRF has a faster execution speed in large data sets, and with the increase of data volume, the advantage is more prominent. This is because PLA-PRF uses data reuse strategy "DRS" in the process of parallelization, which reduces the data communication overhead in a distributed environment and improves the parallelization efficiency of the algorithm.

Iowa Liquor Sales Data Predictive Analysis Using Spark

The paper aims to analyze and predict sales of liquor in the state of Iowa by applying machine learning algorithms to models built for prediction. We have taken recourse of Azure ML and Spark ML for our predictive analysis, which is legacy machine learning (ML) systems and Big Data ML, respectively. We have worked on the Iowa liquor sales dataset comprising of records from 2012 to 2019 in 24 columns and approximately 1.8 million rows. We have concluded by comparing the models with different algorithms applied and their accuracy in predicting the sales using both Azure ML and Spark ML. We find that the Linear Regression model has the highest precision and Decision Forest Regression has the fastest computing time with the sample data set using the legacy Azure ML systems. Decision Tree Regression model in Spark ML has the highest accuracy with the quickest computing time for the entire data set using the Big Data Spark systems.

Exploratory and Predictive Analytics of User Preferences from Kaggle LEGO-Toys Datasets Using Spark ML

Apache Spark is an open-source distributed data processing framework. The paper presents a processing architecture for exploring and predicting user preferences using Apache Spark. The architecture is evaluated on LEGO-toys datasets of period 1949-2019 using the Spark Machine Learning (ML) algorithms. The large datasets analyzed consist of LEGO-toys parts, categories, themes and colour features. Spark ML algorithms are applied as (i) k-means analyses of clusters to identify commonalities in LEGO-toys themes and colours, (ii) classifications using the Support Vector Machines (SVMs), Naïve Bayes (NB) and Random Forest (RF) algorithms for theme-preference identification, and (iii) linear regression, decision tree regression, RF, and Gradient Boost for regression analyses to identify the colour-shift in user preferences. The paper elucidates the steps for analytics based on Spark. The results for exploratory and predictive analytics are presented. The evaluation metrics shows that the ensemble regression prediction is better when compared to other algorithms. The analytics give many interesting results. For example, LEGO company’s products have become more colourful (children preferences exhibiting colours spectral-shift and width), diversified and multifaceted over-the-time. The architecture helps in discovering future directions for the new designs in future LEGO products. The proposed architecture can be successfully employed in the related domain to predict product and user’s preferences.

A Big Data Approach to Black Friday Sales

Retail companies recognize the need to analyze and predict their sales and customer behavior against their products and product categories. Our study aims to help retail companies create personalized deals and promotions for their customers, even during the COVID-19 pandemic, through a big data framework that allows them to handle massive sales volumes with more efficient models. In this paper, we used Black Friday sales data taken from a dataset on the Kaggle website, which contains nearly 550,000 observations analyzed with 10 features: qualitative and quantitative. The class label is purchases and sales (in U.S. dollars). Because the predictor label is continuous, regression models are suited in this case. Using the Apache Spark big data framework, which uses the MLlib machine learning library, we trained two machine learning models: linear regression and random forest. These machine learning algorithms were used to predict future pricing and sales. We first implemented a linear regression model and a random forest model without using the Spark framework and achieved accuracies of 68% and 74%, respectively. Then, we trained these models on the Spark machine learning big data framework where we achieved an accuracy of 72% for the linear regression model and 81% for the random forest model.

Predicting the ratings of Amazon products using Big Data

AbstractThis paper aims to apply several machine learning (ML) models to the massive dataset present in the area of e‐commerce from Amazon to analyze and predict ratings and to recommend products. For this purpose, we have used both traditional and Big Data algorithms. As the Amazon product review dataset is large, we present Big Data architecture suitable massive dataset for storing and computation, which is not possible with the traditional architecture. Furthermore, the dataset contains 15 attributes and has about 7 million records. With the dataset, we develop several models in Oracle Big Data and Azure Cloud Computing services to predict the review rating and recommendation for the items at Amazon. We present a comparative conclusion in terms of the accuracy as well as the efficiency with Spark ML—the Big Data architecture, and Azure ML—the traditional architecture.This article is categorized under: Fundamental Concepts of Data and Knowledge > Big Data Mining Technologies > Machine Learning Technologies > Prediction

Java Web Cloud Data Analysis and Application Based on Spark Machine Learning Algorithm

The rise of information technology has promoted the rapid production of all kinds of information data, with the rapid growth of information data, the problem of data scalability began to appear, the system of data system time and space complexity will be more and more large, which brings great difficulties to the processing of information data. Java web cloud data based on Spark machine learning algorithm provides a new way for information data processing. The purpose of this paper is to analyze Java web cloud data, explore its application, and realize deep data mining and fast processing. This paper gives an overview and analysis of Java web cloud data and Spark machine learning algorithm. It attempts to build a Java web cloud data mining system with the help of Spark machine learning algorithm. First, it explains the basic design principle of the system, then points out the construction framework of the system, and finally carries out a simulation experiment on the system. Experimental results show that the application of Java web cloud data in data mining system is feasible and scientific, which not only changes the way of data storage, but also improves the overall efficiency of data processing.

Fast texture classification of denoised SAR image patches using GLCM on Spark

Classification of a synthetic aperture radar (SAR) image is an essential process for SAR image analysis and interpretation. Recent advances in imaging technologies have allowed data sizes to grow, and a large number of applications in many areas have been generated. However, analysis of high-resolution SAR images, such as classification, is a time-consuming process and high-speed algorithms are needed. In this study, classification of high-speed denoised SAR image patches by using Apache Spark clustering framework is presented. Spark is preferred due to its powerful open-source cluster-computing framework with fast, easy-to-use, and in-memory analytics. Classification of SAR images is realized on patch level by using the supervised learning algorithms embedded in the Spark machine learning library. The feature vectors used as the classifier input are obtained using gray-level cooccurrence matrix which is chosen to quantitatively evaluate textural parameters and representations. SAR image patches used to construct the feature vectors are first applied to the noise reduction algorithm to obtain a more accurate classification accuracy. Experimental studies were carried out using naive Bayes, decision tree, and random forest algorithms to provide comparative results, and significant accuracies were achieved. The results were also compared with a state-of-the-art deep learning method. TerraSAR-X images of high-resolution real-world SAR images were used as data.

A Lockable Abnormal Electromagnetic Signal Joint Detection Algorithm

With the development of computers and network technologies, network security has gradually become a global problem. Network security defenses need to be carried out not only on the Internet, but also on other communication media, such as electromagnetic signals. Existing electromagnetic signal communication is easily intercepted or infiltrated. In order to effectively detect the abnormal electromagnetic signal to find out the specific location, then classify it, it is necessary to study the way of communication. The existing electromagnetic signal detection accuracy is low and cannot be located. Considering the characteristics of different power sources in different locations, combined with spark streaming technology and machine learning classification technology, a joint platform for electromagnetic signal anomaly detection based on big data analysis is proposed. The electromagnetic signal is abnormally detected by feature comparison and small signal analysis, and the position and number between the signal sources are determined by three-point positioning and signal attenuation. The experimental results show that the method can detect abnormal electromagnetic signals and classify abnormal electromagnetic signals well, the accuracy rate can reach 95%, and the positioning accuracy can reach 89%.