Apache Spark Research Articles

Big Data Analytic Tools Usage among Academic Libraries in Tanzania

This study examines the implementation of Big Data analytics in Tanzanian University libraries. Employing a qualitative approach within the interpretive paradigm, the study conducted interviews to collect data from 12 library staff across various University libraries in Tanzania. The qualitative data collected was analyzed using Atlasi.ti 7, focusing on emerging themes. The findings established that most University libraries in Tanzania utilize social networking sites for marketing their products and services. However, only a few academic libraries employ analytics tools such as Apache Hadoop, MongoDB, Apache Spark, Tableau, Zoho Analytics, Apache Storm, and others due to insufficient skills. The study concludes that library administrators could greatly benefit from integrating Big Data analytics into their daily operations. Recommendations include developing Big Data analytics policies to ensure proper handling of user data and offering training on the use of such analytic tools to librarians to equip them with necessary skills and techniques.

Live Supplier Catalog and Pricing for Omnichannel

This paper proposes a technical architecture for an Omni-channel Supplier Catalog that provides real-time price updates. This system leverages a combination of open-source technologies, including Apache Camel, Spark Connector, Kafka Topics, Cassandra database, and Kubernetes, deployed on a resilient and elastic cloud platform. This paper details the data pipeline design, outlining how each component contributes to efficient data ingestion, processing, and dissemination. Additionally, the paper highlights the benefits of using this architecture, including improved decision-making, enhanced operational efficiency, and a seamless omnichannel experience for customers.

Big Data and Java are integrated with machine learning

This in-depth investigation explores the revolutionary nexus of Java, Big Data, and Machine Learning (ML), clarifying the innovations and synergies that result from their integration. The voyage commences with a summary of the past, following the growth from Java's fundamental function in high-level development to the revolutionary influence of Big Data frameworks such as Apache Hadoop and Apache Spark. The story then delves into the fundamentals of machine learning in Java, highlighting its adaptability, rich library support, and crucial role in building strong ML pipelines. The investigation delves into the transformative power of Big Data frameworks, highlighting the distributed file system of Hadoop and the in-memory processing capabilities of Spark. We observe the significant effects of this convergence on a variety of industries through real-world case studies, from e-commerce personalized suggestions to fraud detection in banking. The insights gained from these implementations highlight how crucial it is to use ML models with ethical considerations, interdisciplinary cooperation, and ongoing learning. The following sections cover the nuances of data preprocessing, including the use of Java in ETL workflows, scalable feature engineering using Big Data frameworks, and data quality assurance via transformation and cleansing. ML model deployment is the major focus, along with an exploration of the Java runtime environment, micro services architecture, and crucial aspects of model robustness monitoring and maintenance. The investigation concludes with a focus on case studies and success stories that demonstrate the real-world effects of this convergence in sectors like e-commerce, finance, and healthcare. These real-world examples highlight the accomplishments of companies like Netflix, Uber, Airbnb, and others and provide insightful information about how well integration works to accomplish a range of business objectives.

A distributed attribute reduction based on neighborhood evidential conflict with Apache Spark

Attribute reduction is widely employed to improve the efficiency and accuracy of data analysis by eliminating redundant and irrelevant attributes from datasets. However, with the emergence of growing big data, the sequential execution of such algorithms becomes time-consuming and requires distributed computing capabilities to achieve scalable parallelization. This study proposes a novel attribute reduction algorithm for neighborhood decision systems. We introduce two novel metrics—the neighborhood evidential conflict degree (NECD) and neighborhood evidential conflict rate (NECR)—to compute heterogeneity between samples in the neighborhood and assess the significance of attributes in the feature space, respectively. These metrics assess the quality and selection of attribute subsets in attribute reduction, improving classification accuracy and computational efficiency. We also develop a sequentially forward selection attribute reduction method to select a feature subset through the defined NECR. Finally, we develop a distributed attribute reduction algorithm implemented in Apache Spark. Our approach involves a two-phase Map-Reduce process for K-Nearest Neighbors search, evidence combination, and NECR computation. NECR, as a measure of feature subset quality, enhances the feature subset's decision approximation capability of the data. Experimental results on small and large datasets demonstrate that the proposed algorithm outperforms benchmarking algorithms regarding classification accuracy and computational efficiency.

Three-dimensional Geospatial Interlinking with JedAI-spatial

Geospatial data constitutes a considerable part of Semantic Web data, but so far, its sources are inadequately interlinked in the Linked Open Data cloud. Geospatial Interlinking aims to cover this gap by associating geometries with topological relations like those of the Dimensionally Extended 9-Intersection Model. Due to its quadratic time complexity, various algorithms aim to carry out Geospatial Interlinking efficiently. We present JedAI-spatial, a novel, open-source system that organizes these algorithms according to three dimensions: (i) Space Tiling, which determines the approach that reduces the search space, (ii) Budget-awareness, which distinguishes interlinking algorithms into batch and progressive ones, and (iii) Execution mode, which discerns between serial algorithms, running on a single CPU-core, and parallel ones, running on top of Apache Spark. We analytically describe JedAI-spatial’s architecture and capabilities and perform thorough experiments to provide interesting insights about the relative performance of its algorithms.

IoT-Based Intrusion Detection System Using New Hybrid Deep Learning Algorithm

The most significant threat that networks established in IoT may encounter is cyber attacks. The most commonly encountered attacks among these threats are DDoS attacks. After attacks, the communication traffic of the network can be disrupted, and the energy of sensor nodes can quickly deplete. Therefore, the detection of occurring attacks is of great importance. Considering numerous sensor nodes in the established network, analyzing the network traffic data through traditional methods can become impossible. Analyzing this network traffic in a big data environment is necessary. This study aims to analyze the obtained network traffic dataset in a big data environment and detect attacks in the network using a deep learning algorithm. This study is conducted using PySpark with Apache Spark in the Google Colaboratory (Colab) environment. Keras and Scikit-Learn libraries are utilized in the study. ‘CICIoT2023’ and ‘TON_IoT’ datasets are used for training and testing the model. The features in the datasets are reduced using the correlation method, ensuring the inclusion of significant features in the tests. A hybrid deep learning algorithm is designed using one-dimensional CNN and LSTM. The developed method was compared with ten machine learning and deep learning algorithms. The model’s performance was evaluated using accuracy, precision, recall, and F1 parameters. Following the study, an accuracy rate of 99.995% for binary classification and 99.96% for multiclassification is achieved in the ‘CICIoT2023’ dataset. In the ‘TON_IoT’ dataset, a binary classification success rate of 98.75% is reached.

LST-Bench: Benchmarking Log-Structured Tables in the Cloud

Data processing engines increasingly leverage distributed file systems for scalable, cost-effective storage. While the Apache Parquet columnar format has become a popular choice for data storage and retrieval, the immutability of Parquet files renders it impractical to meet the demands of frequent updates in contemporary analytical workloads. Log-Structured Tables (LSTs), such as Delta Lake, Apache Iceberg, and Apache Hudi, offer an alternative for scenarios requiring data mutability, providing a balance between efficient updates and the benefits of columnar storage. They provide features like transactions, time-travel, and schema evolution, enhancing usability and enabling access from multiple engines. Moreover, engines like Apache Spark and Trino can be configured to leverage the optimizations and controls offered by LSTs to meet specific business needs. Conventional benchmarks and tools are inadequate for evaluating the transformative changes in the storage layer resulting from these advancements, as they do not allow us to measure the impact of design and optimization choices in this new setting. In this paper, we propose a novel benchmarking approach and metrics that build upon existing benchmarks, aiming to systematically assess LSTs. We develop a framework, LST-Bench, which facilitates effective exploration and evaluation of the collaborative functioning of LSTs and data processing engines through tailored benchmark packages. A package is a mix of use patterns reflecting a target workload; LST-Bench makes it easy to define a wide range of use patterns and combine them into a package, and we include a baseline package for completeness. Our assessment demonstrates the effectiveness of our framework and benchmark packages in extracting valuable insights across diverse environments. The code for LST-Bench is open source and is available at https://github.com/microsoft/lst-bench/.

Pheromone-guided parallel rough hypercuboid attribute reduction algorithm

In knowledge discovery and data mining, removing redundant and irrelevant data attributes is crucial. Traditional algorithms, however, struggle with efficiency in high-dimensional big data contexts. To solve this problem, this paper proposes a novel global search attribute reduction method namely RHC-IGWO (Rough Hypercuboid Structure via Improved Grey Wolf Optimizer) by integrating the rough hypercuboid method and Improved Grey Wolf Optimizer (IGWO) with the pheromone mechanism. The algorithm is embedded into the Apache Spark parallel computing framework (Parallel computing Rough Hypercuboid Structure via Improved Grey Wolf Optimizer, PcRHC-IGWO) to accelerate and simplify the attribute reduction process. The algorithm divides the decision table into several independent blocks, introduces the pheromone mechanism to simulate the wolf pack behavior, and uses the IGWO for global search, which is conducive to efficient local search and global information sharing between individuals. The position of the individual is initialized by calculating the relevance between the attributes, and the pheromone value is dynamically updated according to the reduction quality. This allows automatically giving more search focus to more promising attribute regions. Experiments with public and real datasets demonstrate the RHC-IGWO algorithm's significant speedup and its efficacy in maintaining or enhancing classification accuracy. Particularly noteworthy is its performance on schizophrenia datasets, where the proposed method achieves outstanding classification accuracies of 86.2%, 88.89%, and 92.86% across various classifiers. These results not only demonstrate its effectiveness but also underline its potential in advanced data analysis scenarios. Additionally, on some large-scale datasets, the time required for processing has been reduced by 85.71%, showcasing the algorithm's efficiency in handling complex and voluminous data.

Distributed Computing with Dask and Apache Spark: A Comparative Study

Distributed Computing with Dask and Apache Spark: A Comparative Study

A Comparative Study on Hadoop MapReduce and Apache Spark Framework for Big Data Analytics

A Comparative Study on Hadoop MapReduce and Apache Spark Framework for Big Data Analytics

MFRLMO: Model-free reinforcement learning for multi-objective optimization of apache spark

Hyperparameter optimization (HO) is a must to figure out to what extent can a specific configuration of hyperparameters contribute to the performance of a machine learning task. The hardware and MLlib library of Apache Spark have the potential to improve big data processing performance when a tuning operation is combined with the exploitation of hyperparameters. To the best of our knowledge, the most of existing studies employ a black-box approach that results in misleading results due to ignoring the interior dynamics of big data processing. They suffer from one or more drawbacks including high computational cost, large search space, and sensitivity to the dimension of multi-objective functions. To address the issues above, this work proposes a new model-free reinforcement learning for multi-objective optimization of Apache Spark, thereby leveraging reinforcement learning (RL) agents to uncover the internal dynamics of Apache Spark in HO. To bridge the gap between multi-objective optimization and interior constraints of Apache Spark, our method runs a lot of iterations to update each cell of the RL grid. The proposed model-free learning mechanism achieves a tradeoff between three objective functions comprising time, memory, and accuracy. To this end, optimal values of the hyperparameters are obtained via an ensemble technique that analyzes the individual results yielded by each objective function. The results of the experiments show that the number of cores has not a direct effect on $speedup$. Further, although grid size has an impact on the time passed between two adjoining iterations, it is negligible in the computational burden. Dispersion and risk values of model-free RL differ when the size of the data is small. On average, MFRLMO produced $speedup$ that is 37% better than those of the competitors. Last, our approach is very competitive in terms of converging to a high accuracy when optimizing Convolutional Neural networks (CNN).

Prediction of Agricultural Commodity Prices using Big Data Framework

The agriculture sector plays a crucial role in the economy of Pakistan, contributing significantly to the Gross Domestic Product (GDP) and the employment rate. However, this sector faces challenges such as climate change, water scarcity, and low productivity, which have a direct impact on agricultural commodity prices. Accurate forecasting of commodity prices is essential for farmers, traders, and policymakers to make informed decisions and improve economic outcomes. This paper explores the use of a big data framework for agricultural commodity price forecasting in Pakistan, using a historical dataset on commodity prices in various Pakistani cities from 2007 to 2022 and Apache Spark to preprocess and clean the data. Based on historical spinach prices in Vehari City, the machine learning models Auto-Regressive Moving Average (ARIMA), Random Forest, and Long-Short-Term Memory (LSTM) were applied to price trends, and their performance was compared using Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE) and squared correlation coefficient (R2). LSTM outperformed ARIMA and Random Forest with a higher R2 value of 0.8 and the lowest MAE of 125.29. Such predictions can help farmers to effectively plan crop cultivation and traders to make well-informed decisions.

Boosting HPC data analysis performance with the ParSoDA-Py library

Developing and executing large-scale data analysis applications in parallel and distributed environments can be a complex and time-consuming task. Developers often find themselves diverted from their application logic to handle technical details about the underlying runtime and related issues. To simplify this process, ParSoDA, a Java library, has been proposed to facilitate the development of parallel data mining applications executed on HPC systems. It simplifies the process by providing built-in scalability mechanisms relying on the Hadoop and Spark frameworks. This paper presents ParSoDA-Py, the Python version of the ParSoDA library, which allows for further support of commonly used runtimes and libraries for big data analysis. After a complete library redesign, ParSoDA can be now easily integrated with other Python-based distributed runtimes for HPC systems, such as COMPSs and Apache Spark, and with the large ecosystem of Python-based data processing libraries. The paper discusses the adaptation process, which takes into consideration the new technical requirements, and evaluates both usability and scalability through some case study applications.

Elastic distributed dataset RDD (Resilient Distributed Datasets)

In recent years, the amount of data to be processed has become larger and larger, and it is difficult for a single processor to handle data, so we need to work hard to develop the expansion mode of cluster mode. Mapreduce, Storm, etc. are a solution for clusters to process data. However, we still have parallel processing, user resource sharing and other issues to be addressed. RDD data model is the key to solve these problems. In this paper, RDD is applied to Spark for detailed expansion, and project experiments are carried out to study the RDD data model. The programming language used in this article is python. The advantage of studying distributed iterative computing is that during the calculation process, the task calculation fails and can be recovered. The default is to allow four failures recoverable. If the stage fails, one can recover, and if the partition fails, one can also recover. Multiple partitions can improve parallelism and efficiency. After a shuffle, a partition failed because the many-to-one relationship needs to be recalculated.

Distributed and explainable GHSOM for anomaly detection in sensor networks

The identification of anomalous activities is a challenging and crucially important task in sensor networks. This task is becoming increasingly complex with the increasing volume of data generated in real-world domains, and greatly benefits from the use of predictive models to identify anomalies in real time. A key use case for this task is the identification of misbehavior that may be caused by involuntary faults or deliberate actions. However, currently adopted anomaly detection methods are often affected by limitations such as the inability to analyze large-scale data, a reduced effectiveness when data presents multiple densities, a strong dependence on user-defined threshold configurations, and a lack of explainability in the extracted predictions. In this paper, we propose a distributed deep learning method that extends growing hierarchical self-organizing maps, originally designed for clustering tasks, to address anomaly detection tasks. The SOM-based modeling capabilities of the method enable the analysis of data with multiple densities, by exploiting multiple SOMs organized as a hierarchy. Our map-reduce implementation under Apache Spark allows the method to process and analyze large-scale sensor network data. An automatic threshold-tuning strategy reduces user efforts and increases the robustness of the method with respect to noisy instances. Moreover, an explainability component resorting to instance-based feature ranking emphasizes the most salient features influencing the decisions of the anomaly detection model, supporting users in their understanding of raised alerts. Experiments are conducted on five real-world sensor network datasets, including wind and photovoltaic energy production, vehicular traffic, and pedestrian flows. Our results show that the proposed method outperforms state-of-the-art anomaly detection competitors. Furthermore, a scalability analysis reveals that the method is able to scale linearly as the data volume presented increases, leveraging multiple worker nodes in a distributed computing setting. Qualitative analyses on the level of anomalous pollen in the air further emphasize the effectiveness of our proposed method, and its potential in determining the level of danger in raised alerts.

A predictive analytics framework for sensor data using time series and deep learning techniques

IoT devices convert billions of objects into data-generating entities, enabling them to report status and interact with their surroundings. This data comes in various formats, like structured, semi-structured, or unstructured. In addition, it can be collected in batches or in real time. The problem now is how to benefit from all of this data gathered by sensing and monitoring changes like temperature, light, and position. In this paper, we propose a predictive analytics framework constructed on top of open-source technologies such as Apache Spark and Kafka. The framework focuses on forecasting temperature time series data using traditional and deep learning predictive analytics methods. The analysis and prediction tasks were performed using Autoregressive Integrated Moving Average (ARIMA), Seasonal Autoregressive Integrated Moving Average (SARIMA), Long Short-Term Memory (LSTM), and a novel hybrid model based on Convolution Neural Network (CNN) and LSTM. The purpose of this paper is to determine whether and how recently developed deep learning-based models outperform traditional algorithms in the prediction of time series data. The empirical studies conducted and reported in this paper demonstrate that deep learning-based models, specifically LSTM and CNN-LSTM, exhibit superior performance compared to traditional-based algorithms, ARIMA and SARIMA. More specifically, the average reduction in error rates obtained by LSTM and CNN-LSTM models were substantial when compared to other models indicating the superiority of deep learning. Moreover, the CNN-LSTM-based deep learning model exhibits a higher degree of closeness to the actual values when compared to the LSTM-based model.

Physical Activities Recommender System Based on Sequential Data Use K-Mean Clustering

Physical activities such as Exercise are essential in maintaining health and fitness, especially for those who adopt a healthy lifestyle. Irregularity in doing Exercise can hurt the body and health, especially if it is not done according to one's physical capacity. In the framework of this research, we developed a Recommender System that aims to provide exercise suggestions according to the user's preferences, especially in the categories of cycling, running, walking, and horse riding. The primary considerations of the variables include heart rate (Average Heart Rate) and pace (Speed Rate). This research approach uses the FitRec Dataset and applies the K-Mean Clustering Algorithm, with the support of APACHE SPARK, for large-scale data processing, given the large data size in the FitRec dataset. Grouping is done using the FitRec dataset and K-Mean. Users are grouped according to heart rate and pace information; this provides appropriate Exercise for users. The test results show that the proposed system performs well, as indicated by the silhouette score = 0.596, calinzski-harabaz score = 2133.09, and davies bouldin score = 0.480. These test metrics reflect the system's ability to cluster. Indirectly, the accuracy performance of the system is assessed through these metrics, showing good accuracy test results.

Cloud-Based Infrastructure and DevOps for Energy Fault Detection in Smart Buildings

We have designed a real-world smart building energy fault detection (SBFD) system on a cloud-based Databricks workspace, a high-performance computing (HPC) environment for big-data-intensive applications powered by Apache Spark. By avoiding a Smart Building Diagnostics as a Service approach and keeping a tightly centralized design, the rapid development and deployment of the cloud-based SBFD system was achieved within one calendar year. Thanks to Databricks’ built-in scheduling interface, a continuous pipeline of real-time ingestion, integration, cleaning, and analytics workflows capable of energy consumption prediction and anomaly detection was implemented and deployed in the cloud. The system currently provides fault detection in the form of predictions and anomaly detection for 96 buildings on an active military installation. The system’s various jobs all converge within 14 min on average. It facilitates the seamless interaction between our workspace and a cloud data lake storage provided for secure and automated initial ingestion of raw data provided by a third party via the Secure File Transfer Protocol (SFTP) and BLOB (Binary Large Objects) file system secure protocol drivers. With a powerful Python binding to the Apache Spark distributed computing framework, PySpark, these actions were coded into collaborative notebooks and chained into the aforementioned pipeline. The pipeline was successfully managed and configured throughout the lifetime of the project and is continuing to meet our needs in deployment. In this paper, we outline the general architecture and how it differs from previous smart building diagnostics initiatives, present details surrounding the underlying technology stack of our data pipeline, and enumerate some of the necessary configuration steps required to maintain and develop this big data analytics application in the cloud.

Solving Traveling Salesman Problem Using Parallel River Formation Dynamics Optimization Algorithm on Multi-core Architecture Using Apache Spark

According to Moore’s law, computer processing hardware technology performance is doubled every year. To make effective use of this technological development, the algorithmic solutions have to be developed at the same speed. Consequently, it is necessary to design parallel algorithms to be implemented on parallel machines. This helps to exploit the multi-core environment by executing multiple instructions simultaneously on multiple processors. Traveling Salesman (TSP) is a challenging non-deterministic-hard optimization problem that has exponential running time using brute-force methods. TSP is concerned with finding the shortest path starting with a point and returning to that point after visiting the list of points, provided that these points are visited only once. Meta-heuristic optimization algorithms have been used to tackle TSP and find near-optimal solutions in a reasonable time. This paper proposes a parallel River Formation Dynamics Optimization Algorithm (RFD) to solve the TSP problem. The parallelization technique depends on dividing the population into different processors using the Map-Reduce framework in Apache Spark. The experiments are accomplished in three phases. The first phase compares the speedup, running time, and efficiency of RFD on 1 (sequential RFD), 4, 8, and 16 cores. The second phase compares the proposed parallel RFD with three parallel water-based algorithms, namely the Water Flow algorithm, Intelligent Water Drops, and the Water Cycle Algorithm. To achieve fairness, all algorithms are implemented using the same system specifications and the same values for shared parameters. The third phase compares the proposed parallel RFD with the reported results of metaheuristic algorithms that were used to solve TSP in the literature. The results demonstrate that the RFD algorithm has the best performance for the majority of problem instances, achieving the lowest running times across different core counts. Our findings highlight the importance of selecting the most suitable algorithm and core count based on the problem characteristics to achieve optimal performance in parallel optimization.

Instance segmentation on distributed deep learning big data cluster

AbstractDistributed deep learning is a promising approach for training and deploying large and complex deep learning models. This paper presents a comprehensive workflow for deploying and optimizing the YOLACT instance segmentation model as on big data clusters. OpenVINO, a toolkit known for its high-speed data processing and ability to optimize deep learning models for deployment on a variety of devices, was used to optimize the YOLACT model. The model is then run on a big data cluster using BigDL, a distributed deep learning library for Apache Spark. BigDL provides a high-level programming interface for defining and training deep neural networks, making it suitable for large-scale deep learning applications. In distributed deep learning, input data is divided and distributed across multiple machines for parallel processing. This approach offers several advantages, including the ability to handle very large data that can be stored in a distributed manner, scalability to decrease processing time by increasing the number of workers, and fault tolerance. The proposed workflow was evaluated on virtual machines and Azure Databricks, a cloud-based platform for big data analytics. The results indicated that the workflow can scale to large datasets and deliver high performance on Azure Databricks. This study explores the benefits and challenges of using distributed deep learning on big data clusters for instance segmentation. Popular distributed deep learning frameworks are discussed, and BigDL is chosen. Overall, this study highlights the practicality of distributed deep learning for deploying and scaling sophisticated deep learning models on big data clusters.