MapReduce Framework Research Articles

Reflecting on a Decade of Evolution: MapReduce‐Based Advances in Partitioning‐Based, Hierarchical‐Based, and Density‐Based Clustering (2013–2023)

ABSTRACTThe traditional clustering algorithms are not appropriate for large real‐world datasets or big data, which is attributable to computational expensiveness and scalability issues. As a solution, the last decade's research headed towards distributed clustering using the MapReduce framework. This study conducts a bibliometric review to assess, establish, and measure the patterns and trends of the MapReduce‐based partitioning, hierarchical, and density clustering algorithms over the past decade (2013–2023). A digital text‐mining‐based comprehensive search technique with multiple field‐specific keywords, inclusion measures, and exclusion criteria is employed to obtain the research landscape from the Scopus database. The Scopus‐obtained data is analyzed using the VOSViewer software tool and coded using the R statistical analysis tool. The analysis identifies the numbers of scholarly articles, diversities of article sources, their impact and growth patterns, details of most influential authors and co‐authors, most cited articles, most contributing affiliations and countries, and their collaborations, use of different keywords and their impact, and so forth. The study further explores the articles and reports the methodologies employed for designing MapReduce‐based counterparts of traditional partitioning, hierarchical, and density clustering algorithms and their optimizations and hybridizations. Finally, the study lists the main research challenges encountered in the past decade for MapReduce‐based partitioning, hierarchical, and density clustering. It suggests possible areas for future research to contribute further in this field.

Cyber-attack Detection and Mitigation Process under Big Data Consideration: Improved Recursive Feature Elimination-based Feature Selection

Due to the rapid growth of network technology, huge volume and distinct data sent via networks is expanding constantly. The situation shows how complex and dense cyber attacks and hazards are developing. Due to the rapid advancement in network density, cyber security specialists find it difficult to monitor all network activity. Due to frequent and sophisticated cyber attacks, it is becoming more challenging to detect and identify abnormalities in network events. The use of deep learning provides a variety of tools and strategies for automated cyber-attack detection as well as quick attack-type prediction as well as evaluation. This work introduces a novel cyber-attack detection and mitigation process under the following phases including preprocessing, feature extraction via the Map Reduce framework that handles the big data, feature selection, attack detection and mitigation. The Improved Normalisation process is achieved on the preprocessing phase. The work is examined from a big data perspective; hence Map Reduce framework is utilised for this. As a result, the framework will manage the feature extraction process, where features including statistical features, raw features, improved correlation-based features, and info gain-based features will be extracted. Following feature extraction, the Improved Recursive Feature Elimination procedure is processed that selects the relevant features. The hybrid detection model, which combines Recurrent Neural Networks (RNN) Deep and Belief Networks (DBN) is used to detect the attacks. Once an attack has been detected, the attacker must be mitigated. To accomplish this, an improved BAIT-based mitigation procedure is used. The two datasets used in this work are, namely, Intrusion Detection Systems (IDS) 2018 Intrusion CSVs (CSE-CIC-IDS2018) and UNSW_NB15. Finally, the suggested model and the alternative methods are contrasted using a variety of measures such as accuracy, sensitivity, specificity, precision, FDR, FNR and FPR.

STORM: A MapReduce framework for Symbolic Time Intervals series classification

STORM: A MapReduce framework for Symbolic Time Intervals series classification

Input/output optimization scheduler for cloud-based map reduce framework

Hadoop MapReduce (HMR) provides the most common MapReduce (MR) framework, and it is available as open source. MR is a famous computational framework for evaluating unstructured, and semi-structured big data and executing applications in the past ten years. Memory and input/output (I/O) overhead are just two of the many problems affecting the current HMR scheduler system. This study aims to improve systems resource use including the processing of data in real-time by creating a memory I/O optimized scheduler (MIOOS) for HMR. The disk I/O seek can be reduced by using MIOOS, which analyzes the entire memory management. Additionally, the MIOOS makespan approach is used to reduce the occurrence of problems in intermediary tasks. Both the MIOOS approach and the current approach are assessed by using complex scientific workflow applications with extreme task inter-dependencies. Further, the comparison study demonstrates that the MIOOS framework outdoes the current approach regarding makespan and overall memory usage.

Analysis of Aliyun-based serverless on MapReduce efficiency

In the context of the current era of big data, traditional Hadoop and cluster-based MapReduce frameworks are unable to meet the demands of modern research. This paper presents a MapReduce framework based on the AliCloud Serverless platform, which has been developed with the objective of optimizing word frequency counting in large-scale English texts. Leveraging AliCloud's dynamic resource allocation and elastic scaling, we have created an efficient and flexible text data processing system. This paper details the design and implementation of the Map and Reduce phases and analyses the impact of vCPU and memory specifications, as well as parallel resource allocation on system performance. Experimental results show that increasing vCPU specifications significantly improves processing capacity and execution efficiency. While the impact of memory specifications is relatively minor, it can positively influence performance in specific scenarios. Parallel processing markedly enhances system performance. Experiments on "Harry Potter and the Sorcerer's Stone" validate the framework's performance across various configurations. This study offers valuable insights for the design and optimization of serverless-based MapReduce frameworks, as well as suggesting future enhancements. These include the implementation of advanced parallel computing strategies, improved error handling, and refined data preprocessing, which collectively aim to boost system performance and stability.

Mapreduce framework based sentiment analysis of twitter data using hierarchical attention network with chronological leader algorithm

Mapreduce framework based sentiment analysis of twitter data using hierarchical attention network with chronological leader algorithm

Word frequency statistics based on MapReduce on serverless platforms

This paper investigates the application of serverless computing in conjunction with the MapReduce framework, particularly in machine learning (ML) tasks. The MapReduce programming model has been widely used to process large-scale datasets by simplifying parallel and distributed data processing. This study explores how the combination of these two technologies can provide more efficient and cost-effective ML solutions. Through a detailed analysis of serverless environments and the MapReduce framework, this paper shows how the combination can advance the fields of cloud computing and machine learning. The experimental part includes the implementation of Map-Reduce model on a serverless platform, exploring the impact of different parameter settings on performance and improving efficiency by optimizing the data processing flow. In addition, the paper analyzes the use of memory and CPU resources and derives the relationship between dataset size, memory consumption and processor configuration and execution time. Through these experiments and analyses, this paper provides an empirical basis and theoretical support for the optimization of cloud computing frameworks.

Design of an Iterative Method for MapReduce Scheduling Using Deep Reinforcement Learning and Anomaly Detection

Due to high complexities within distributed computing environments, there's a critical need for advanced scheduling frameworks that are capable of optimizing MapReduce systems. Current approaches have static policies that limit their capability to adapt to changing system dynamics and workload variations for different cloud scenarios. To overcome these issues, this study introduces a robust MapReduce framework empowered by intelligent scheduling algorithms, tailored to enhance system efficiency and resilience levels. The framework introduces three novel scheduling models: Deep Reinforcement Learning for Dynamic Job Scheduling (DRDJS), Anomaly Detection-driven Adaptive Scheduling (ADAS), and Cluster-based Job Categorization and Scheduling (CJCS). DRDJS utilizes deep reinforcement learning to dynamically generate optimal scheduling policies based on multiple metrics that include historical job data, system metrics, and workload characteristics. This adaptive approach leads to significant reductions in job completion times, outperforming static scheduling methods by up to 30%. Next, ADAS leverages anomaly detection to prioritize critical tasks and efficiently allocate resources in response to anomalies, resulting in a notable improve scheduling speed by up to 40%. Finally, CJCS employs clustering algorithms to categorize jobs based on their resource requirements and execution profiles, enabling more accurate resource allocation and reducing average job completion times by up to 25%. By integrating these methods into a unified scheduling framework, the proposed solution addresses the variability in job characteristics and system performance, thereby enhancing the overall throughput and stability of MapReduce operations.

Agriculture data analysis using parallel k-nearest neighbour classification algorithm

A cost-effective and effective agriculture management system is created by utilizing data analytics (DA), internet of things (IoT), and cloud computing (CC). Geographic information system (GIS) technology and remote sensing predictions give users and stakeholders access to a variety of sensory data, including rainfall patterns and weather-related information (such as pressure, humidity, and temperatures). They have unstructured format for sensory data. The current systems do a poor job of analysing such data since they cannot effectively balance speed and memory usage. An effective categorization model (ECM) on agriculture management system is proposed to address this research difficulty. First, a classification technique called priority-based k-nearest neighbour (KNN) is provided to categorize unstructured multi-dimensional data into a structured form. Additionally, the Hadoop MapReduce (HMR) framework is used to do classification utilizing a parallel approach. Data from real-time IoT sensors used in agriculture is the subject of experiments. The suggested approach significantly outperforms previous approaches that are computing time, memory efficiency, model accuracy, and speedup.

IoT Forensic Cyber Activities Detection and Prevention with Automated Machine Learning Model

The Internet of Things (IoT) has been deployed in a vast range of applications with exponential increases in data size and complexity. Existing forensic techniques are not effective for the accuracy and detection rate of security issues in IoT forensics. Cyber forensic comprises huge volume constraints that are processing huge volumes of data in the Information and Communication Technology (ICT) comprised of IoT devices and platforms. Trust blockchain is effective technology those are utilized to assess the tamper-proof records in all transaction in the IoT environment. With the implementation of trust blockchain the record and transaction are processed with a distributed ledger that is managed by the network nodes. The challenge associated with the trust blockchain in IoT forensics is cost and security. To achieve significant cost-effectiveness organizations, need to evaluate the risks and benefits associated with IoT forensics in the trust blockchain technology. In this paper, developed a Block Chain Enabled Cyber-Physical system with distributed storage. The developed Blockchain model is termed as Integrated Hadoop Blockchain Forensic Machin Learning (IHBF-ML). The IHBF-ML model uses the Hadoop Distributed File System (HDFS) with cyberspace to improve security. Within the IHBF-ML model, IoT data communication is established with the smart contract. The smart contract-based blockchain process uses the Machine Learning model integrated with Cat Boost classification model for anomaly detection. Cost in IoT forensic is minimized with the parallel processing of the data through MapReduce Framework for the traffic translation, extraction, and analysis of the dynamic feature traffic from the IoT environment. The experimental analysis stated that constructed IHBF-ML model reduces the cost by ~25% than the other conventional blockchain Ethereum and EOS.

Cost minimization approach with hybrid optimization based task scheduling in Geo-distributed cloud

ABSTRACT To manage the huge data, cloud computing necessitates a significant number of disc I/O, network bandwidth and CPU cycles. To handle the massive volumes of data, the programming paradigm known as data flow integrates the tasks to a graph structure. Task scheduling is one of the main areas of current study that primarily aims to allocate the tasks to resources. It is essential to organise the tasks in a way which saves time and makes the most use of available resources. In recent years, scholars have offered different job scheduling techniques. This work intends to introduce a novel cost minimisation for big data processing with respect to task allocation or scheduling. Initially, the big data is processed under Map Reduce framework. This allocation processes is taken as the optimisation issue, which is solved under the consideration of minimisation factors like execution cost, makespan, communication cost, and energy consumption and risk. For solving this issue, a new hybrid optimisation algorithm known as Hybrid bald eagle and Archimedes Optimisation algorithm (HBE-ArCA) is developed. The betterment of HBE-ArCA is proven regarding cost, makespan and so on.

Blockchain-Enabled Secure Data Sharing with Honey Encryption and DSNN-Based Key Generation

Ensuring data confidentiality is a critical requirement for modern security systems globally. Despite the implementation of various access-control policies to enhance system security, significant threats persist due to insecure and inadequate access management. To address this, Multi-Party Authorization (MPA) systems employ multiple authorities for authorization and authentication, utilizing blockchain technology to store and access data securely, ensuring immutable and trusted audit trails. In this work, we propose a hybrid key-generation approach called the Identity and Attribute-Based Honey Encryption (IABHE) Algorithm combined with Deep Spiking Neural Network (DSNN) denoted by IABHE+DSNN for secure data sharing in a multi-party blockchain-based system. This approach incorporates various entities and multiple security functionalities to ensure data security. The data-sharing process involves several steps: initialization, authentication, initial registration, data protection, validation, and data sharing. Data protection is executed within the MapReduce framework, with data encryption performed using IABHE and key generation managed by DSNN. Experimental results demonstrate that the proposed IABHE+DSNN approach achieves a decryption time of 10.786 s, an encryption time of 15.765 s, and a key complexity of 0.887, outperforming existing methods.

Assessment of soil fertility in Xinjiang oasis cotton field based on big data techniques

Assessing soil fertility through traditional methods has faced challenges due to the vast amount of meteorological data and the complexity of heterogeneous data. In this study, we address these challenges by employing the K-means algorithm for cluster analysis on soil fertility data and developing a novel K-means algorithm within the Hadoop framework. Our research aims to provide a comprehensive analysis of soil fertility in the Shihezi region, particularly in the context of oasis cotton fields, leveraging big data techniques. The methodology involves utilizing soil nutrient data from 29 sampling points across six round fields in 2022. Through K-means clustering with varying K values, we determine that setting K to 3 yields optimal cluster effects, aligning closely with the actual soil fertility distribution. Furthermore, we compare the performance of our proposed K-means algorithm under the MapReduce framework with traditional serial K-means algorithms, demonstrating significant improvements in operational speed and successful completion of large-scale data computations. Our findings reveal that soil fertility in the Shihezi region can be classified into four distinct grades, providing valuable insights for agricultural practices and land management strategies. This classification contributes to a better understanding of soil resources in oasis cotton fields and facilitates informed decision-making processes for farmers and policymakers alike.

Bigdata clustering and classification with improved fuzzy based deep architecture under MapReduce framework

The current state of economic, social ideas, and the advancement of cutting-edge technology are determined by the primary subjects of the contemporary information era, big data. People are immersed in a world of information, guided by the abundance of data that penetrates every element of their surroundings. Smart gadgets, the IoT, and other technologies are responsible for the data’s explosive expansion. Organisations have struggled to store data effectively throughout the past few decades. This disadvantage is related to outdated, expensive, and inadequately large storage technology. In the meanwhile, large data demands innovative storage techniques supported by strong technology. This paper proposes the bigdata clustering and classification model with improved fuzzy-based Deep Architecture under the Map Reduce framework. At first, the pre-processing phase involves data partitioning from the big dataset utilizing an improved C-Means clustering procedure. The pre-processed big data is then handled by the Map Reduce framework, which involves the mapper and reducer phases. In the mapper phase. Data normalization takes place, followed by the feature fusion approach that combines the extracted features like entropy-based features and correlation-based features. In the reduction phase, all the mappers are combined to produce an acceptable feature. Finally, a deep hybrid model, which is the combination of a DCNN and Bi-GRU is used for the classification process. The Improved score level fusion procedure is used in this case to obtain the final classification result. Moreover, the analysis of the proposed work has proved to be efficient in terms of classification accuracy, precision, recall, FNR, FPR, and other performance metrics.

Multi-objective hybrid optimized task scheduling in cloud computing under big data perspective

The new and rising paradigm of cloud computing offers customers various possibilities of task computation based on their desires and choices. Customers receive services from cloud computing systems as a utility. Customers are enthusiastic about low-cost service availability and task completion times that are kept to be minimum. To achieve client fulfilment, the service provider must schedule the jobs to the right resources if the cloud server gets many user requests. The rapid growth in data volume necessitates petabytes processing of data each day. Unstructured, semi-structured, and structured data are all described in terms of their rapid growth and availability. In order to make correct and timely decisions, it must be processed appropriately. In this research, we present BWUJS (Black Widow Updated Jellyfish Search), a multi-objective hybrid optimization-based task scheduling algorithm. This work considers task generation from the Bigdata perspective. The clustering of tasks is performed via the Map Reduce framework with an Improved K-means clustering model. After task clustering, the task priority estimation is performed. Finally, the scheduling is performed via BWJSU based on certain constraints like priority, makespan, completion time, resource utilization, and degree of imbalance.

Cost-aware optimal resource provisioning Map-Reduce scheduler for hadoop framework

<p><span lang="EN-US">Distributed data processing model has been one of the primary components in the case of data-intensive applications; furthermore, due to advancements in technologies, there has been a huge volume of data generation of diverse nature. Hadoop map reduce framework is responsible for adopting the ease of deployment mechanism in an open-source framework. The existing Hadoop MapReduce framework possesses high makespan time and high Input/Output overhead and it mainly affects the cost of a model. Thus, this research work presents an optimized cost aware resource provisioning MapReduce model also known as the cost-effective resource provisioning MapReduce (CRP-MR) model. CRP-MR model introduces the two integrated approaches to minimize the cost; at first, this model presents the optimal resource optimization and optimal Input/Output optimization cleansing in the Hadoop MapReduce (HMR) scheduler. CRP-MR is evaluated considering the bioinformatics dataset and CRP-MR performs better than the existing model. </span></p>

A new service composition method in the cloud‐based Internet of things environment using a grey wolf optimization algorithm and MapReduce framework

SummaryCloud computing is quickly becoming a common commercial model for software delivery and services, enabling companies to save maintenance, infrastructure, and labor expenses. Also, Internet of Things (IoT) apps are designed to ease developers' and users' access to networks of smart services, devices, and data. Although cloud services give nearly infinite resources, their reach is constrained. Designing coherent and organized apps is made possible by integrating the cloud and IoT. Expanding facilities by combining services is a critical component of this technology. Various services may be presented in this environment based on the user's demands. Considering their Quality of Service (QoS) attributes, discovering the appropriate available atomic services to construct the needed composite service with their collaboration in an orchestration model is an NP‐hard issue. This article suggests a service composition method using Grey Wolf Optimization (GWO) and MapReduce framework to compose services with optimized QoS. The simulation outcomes illustrate cost, availability, response time, and energy‐saving improvements through the suggested approach. Comparing the suggested technique to three baseline algorithms, the average gain is a 40% improvement in energy savings, a 14% decrease in response time, an 11% increase in availability, and a 24% drop in cost.

H-mrk-means: Enhanced Heuristic mrk-means for Linear Time Clustering of Big Data Using Hybrid Meta-heuristic Algorithm

Big data is generally derived with a large volume and combined categories of attributes like categorical and numerical. Among them, [Formula: see text]-prototypes have been adopted into MapReduce structure, and thus, it provides a better solution for the huge range of data. However, [Formula: see text]-prototypes need to compute all distances among every data point and cluster centres. Moreover, the computations of distances are redundant as data points are often present in similar clusters after fewer iterations. Nowadays, to cluster huge-scale datasets, one of the efficient solutions is [Formula: see text]-means. However, [Formula: see text]-means is not intrinsically appropriate to execute in MapReduce due to the iterative nature of this technique. Moreover, for every iteration, [Formula: see text]-means should perform an independent MapReduce job but, it leads to higher Input/Output (I/O) overhead at every iteration. This research paper presents a novel enhanced linear time clustering for handling big data called Heuristic mrk-means (H-mrk-means) using optimized [Formula: see text]-means on the MapReduce model. In order to manage big data that is time series in nature, the sampling and MapReduce framework are adopted, which utilize different machines for processing data. Before initiating the main clustering process, a sampling process is adopted to get the noteworthy information. The two main phases of the developed method are the map phase (divide and conquer) and the reduce phase (final clustering). In the map phase, the data are divided into diverse chunks that should be stored in assigned machines. In the reduce phase, data clustering is performed. Here, the cluster centroid of data is tuned with the help of hybrid Tunicate-Deer Hunting Optimization (T-DHO) algorithm by attaining a newly derived objective function. This type of optimal tuning of solution enhances the efficiency of clustering when compared over normal iterative [Formula: see text]-means and mrk-means clustering. The experimental evaluation on varied counts of chunks using the proposed H-mrk-means has attained higher quality of clustering results and faster execution times evaluated with other clustering approaches.

Optimization enabled LeNet for big data classification using MapReduce framework on COVID-19 data

ABSTRACT The big data clustering is a requisite for generating the data in the digitalised globe. The old-fashioned clustering approaches are not large sized and highly unorganised big data. Thus, to obtain the efficiency of big data clustering, a new architecture is required. This work introduces the Jaya African Vulture Optimization Algorithm-based LeNet (JAVO-based LeNet) for big data classification on COVID-19. The big data clustering is detected by using the LeNet model, which is tuned by the JAVO algorithm. Here, the JAVO algorithm is the integration of the Jaya algorithm and AVOA algorithm. Further, using the Deep Embedded Clustering (DEC) technique for partitioning the smaller data to improve the performance of big data classification. The classification process for COVID-19 is established in the MapReduce (MR) framework. For each mapper, the pre-processing is completed based on Min-Max normalisation and the feature fusion is performed based on the Hellinger distance, which is measured with a deep residual network (DRN). Finally, the classification process for COVID-19 is done by considering the fused data acquired from each mapper with LeNet, which is tuned by the JAVO algorithm. Furthermore, the analysis of the proposed method attains the testing accuracy of 0.948, specificity of 0.939 and sensitivity of 0.950.

Implementation of machine learning techniques with big data and IoT to create effective prediction models for health informatics

As a result of the availability of healthcare data in sheer size, big data analytics has to grow regularly in this industry to ensure new and effective opportunities. This is helpful in providing early prevention, prediction, and detection of disease, thus helping in the enhancement of the overall life quality of the individuals. Likewise, in this paper, a machine learning-based big data analytics model is developed for predicting multi-diseases to provide a better decision support system for various healthcare applications. This developed framework utilizes the MapReduce framework, where the map phase performs feature extraction and the reduce phase performs feature selection for the purpose of handling and processing big data. The required healthcare data is collected from external web sources. In the map phase, the statistical features and the Principal Component Analysis (PCA) features are extracted. In the reduction phase, the optimal features are selected with the aid of the developed Hybrid Flower Pollination Bumblebees Optimization Algorithm (HFPBOA). Then, the Ensemble Learning (EL) model is developed to predict the multi-diseases. Moreover, the parameters present in the EL classifiers are optimized by using the same HFPBOA. The final prediction output is obtained by averaging the weight function between the outputs of the NN, KNN, and fuzzy classifier. Thus, the offered model attains 40.1%, 28.7%, 23.6%, and 10.5% improved than SSA-EL, DOA-EL, BOA-EL, and FA-EL respectively in terms of best value. The effectiveness computed for the developed multi-disease prediction framework is guaranteed by comparing the results among the recently developed prediction approaches.