MapReduce Framework Research Articles

Ensemble Model for Stock Price Forecasting: MapReduce Framework for Big Data Handling: An Optimal Trained Hybrid Model for Classification

A number of authors have focused on this study to examine how huge data are perceived. A novel big data classification paradigm is introduced by the work’s preprocessing, feature extraction and classification techniques. Data normalization is carried out at the preprocessing stage. The MapReduce framework is then utilized to manage the massive data. Statistical features (mean, median, min/max and SD), higher-order statistical features (skewness, kurtosis and enhanced entropy), and correlation-based features are all extracted prior to classification. The Bi-LSTM and deep maxout hybrid classification model classifies the data during the reduction stage. To assure classification accuracy, training will also be deployed by the new Hybrid Butterfly Positioned Coot Optimization (HBPCO) algorithm. The proposed method’s accuracy of 97.45% beats the methods of NN (85.13%), CNN (83.78%), RNN (78.37%), Bi-LSTM (82.43%) and SVM (87.83%).

A Comparative Study on Hadoop MapReduce and Apache Spark Framework for Big Data Analytics

A Comparative Study on Hadoop MapReduce and Apache Spark Framework for Big Data Analytics

Taylor jellyfish search algorithm enabled feature selection and deep learning for big data classification using mapreduce framework

ABSTRACT Nowadays, the increasing growth of technology contributes various organisations to storing a huge amount of records using web servers in the form of big data. The big database possesses a vast amount of relevant and irrelevant features, and the less significant features are difficult to remove which makes the process time-consuming and more complex. Hence, this paper proposes an optimisation algorithm, named the Taylor Jellyfish Search Optimisation (TaylorJSO) for selecting the relevant features. Here, an optimisation-based deep learning approach called Taylor Jellyfish Search Optimisation-Deep Maxout Network (TaylorJSO-DMN) is introduced to classify big data more effectively by selecting optimal features. In this model, the input database is preprocessed using quantile normalisation and uses the oversampling of imbalanced classes for data augmentation. In the MapReduce framework, the selection of optimal features is performed in the mapper phase, whereas in the reducer phase, the DMN is used for the classification of big data. The DMN is trained using the introduced TaylorJSO algorithm to improve the classification performance of big data classification. Moreover, the TaylorJSO-DMN model attained greater performance with a maximum of 92.5% accuracy, 91.6% sensitivity, and 92.9% specificity than other traditional approaches.

Optimized RMDL with Transfer Learning for Sentiment Classification in the MapReduce Framework

A core task in sentiment analysis is sentiment categorization, and it is crucial to understand user feelings based on their remarks in social media or product evaluations. Due to ambiguous phrases, refusal words, and other factors, categorizing sentiment presents several challenging issues. The objective of this research is to develop a hybrid optimization-based deep learning model and MapReduce framework-based sentiment categorization approach. The review document is taken from a dataset and used in this case with the MapReduce methodology. MapReduce is a software framework and programming model for analyzing massive volumes of data that consists of two phases, mapper and reducer. BERT tokenization and aspect term extraction are executed in the mapper phase, whereas sentiment analysis is performed in the reducer stage utilizing random multimodal deep learning (RMDL) with transfer learning and AlexNet and ResNet 50 as pre-trained models. In addition, the exponential coot political algorithm (ECPA) is offered as an optimization algorithm for weight optimization in RMDL. The ECPA is obtained by combining the exponential weighted moving average model (EWMA) with the coot algorithm, as well as a political optimizer (PO). The proposed ECPA_RMDL model has acquired 90.9% precision, 89.7% recall, and 89.9% f-measure.

MapReduce based on serverless platforms

This study intends to investigate the application of the MapReduce (MR) framework based on serverless computing in big data processing. By combining the MapReduce model with serverless computing, efficient data processing is achieved. In this framework, the phases of Map task execution, reduce task execution, etc. are accomplished through stateless serverless functions, and data storage is realized with the help of cloud storage platforms (e.g., OSS). In this paper, the author introduces the basic theory of MR, the basic theory of serverless computing, describes the framework implementation process, and discusses the role of OSS in distributed computing. The outcomes of the trial indicate the average execution time of the framework for a WordCount task on 100 pieces of TOEFL English reading data is 6.81 seconds. The discussion analyzes the frameworks advantages (high elasticity, resource utilization) and disadvantages (cold start latency, unsuitable for long time tasks). Future research directions encompass performance optimization, long-time task processing, state management, etc. In summary, this study provides valuable insights for the practical application of serverless computing in big data processing.

Securing large-scale data processing: Integrating lightweight cryptography in MapReduce

<p>In today’s rapidly evolving digital landscape, the imperative of data security stands paramount. With the proliferation of sensitive information being stored and transmitted online, the necessity for robust encryption algorithms has grown exponentially. However, the suitability of traditional encryption methods in resource-constrained settings, like mobile devices and cloud computing, remains a concern due to their computational intensity. To address this, researchers have introduced a novel category of encryption algorithms known as lightweight cryptography algorithms. These cryptographic solutions are designed to offer robust security while minimizing computational demands, thus striking a harmonious balance between security and efficiency. While lightweight cryptography algorithms present a promising solution, their adequacy for applications demanding exceptionally high security, particularly within Big Data environments, warrants careful consideration. In this study, we presented a novel approach involving the utilization of lightweight cryptography algorithms within the MapReduce framework. By subjecting these algorithms to rigorous experimentation, we assessed their performance using software-oriented metrics from various dimensions.</p>

A novel three-factor authentication and optimal mapreduce frameworks for secure medical big data transmission over the cloud with shaxecc

A novel three-factor authentication and optimal mapreduce frameworks for secure medical big data transmission over the cloud with shaxecc

3D Interior Design System Model Based on Computer Virtual Reality Technology

Globally, data volume is increases exponentially with increase in the proliferation with Cloud Computing. MapReduce is emerged as the prominent solution for the unprecedented growth in the efficient manner as it process both structured and unstructured data. The dynamic landscape of Virtual Reality has seen a significant shift towards technology-driven approaches, with data analytics and personalized learning becoming increasingly important. This paper introduces an innovative framework that leverages the power of Hadoop and MapReduce to elevate 3D virtual reality experiences within diverse VR Cloud settings. This paper presents the development of an efficient Cache-Based MapReduce framework (CMF) where Cache algorithms are effectively used to process queries on large-scale cloud-based data. The Hadoop System processes data in single-node Hadoop Clusters (Pseudo-distributed) as well as heterogeneous Hadoop Clusters (fully distributed nodes) within Amazon Web Services (AWS). The Hadoop System process the data in the single node Hadoop Cluster (Pseudo-distributed) heterogeneous Hadoop Cluster (fully distributed node) in the Amazon Web Services (AWS). The experimental analysis is evaluated for the SmallGutenberg and LargeGutenberg database. The developed model achieves the average reduction in job of 48.01% with reduction in execution time of 51.99%. The CMF of 7-node, 9-node, 15-node and 20-node reduction in execution time is measured as 49.91%, 51.38%, 54.71% and 45.29% respectively.

Efficient hybrid oversampling and intelligent undersampling for imbalanced big data classification

Imbalanced classification is a well-known challenge faced by many real-world applications. This issue occurs when the distribution of the target variable is skewed, leading to a prediction bias toward the majority class. With the arrival of the Big Data era, there is a pressing need for efficient solutions to solve this problem. In this work, we present a novel resampling method called SMOTENN that combines intelligent undersampling and oversampling using a MapReduce framework. Both procedures are performed on the same pass over the data, conferring efficiency to the technique. The SMOTENN method is complemented with an efficient implementation of the neighborhoods related to the minority samples. Our experimental results show the virtues of this approach, outperforming alternative resampling techniques for small- and medium-sized datasets while achieving positive results on large datasets with reduced running times.

Big data clustering using fuzzy based energy efficient clustering and MobileNet V2

Big data analytics (BDA) is a systematic way to analyze and detect various patterns, relationships, and trends in vast amounts of data. Big data analysis and processing require significant effort, techniques, and equipment. The Hadoop framework software uses the MapReduce approach to do large-scale data analysis using parallel processing in order to generate results as soon as possible. Due to the traditional algorithm’s longer execution time and difficulty in processing big amounts of data, this is one of the main issues. Clusters are highly correlated inside each other but are not highly correlated with one another. The technique of effectively allocating limited resources is known as an optimization algorithm for clustering. For processing large amounts of data with several dimensions, the conventional optimization approach is insufficient. By using a fuzzy method, this can be prevented. In this paper, we proposed Fuzzy based energy efficient clustering approach to enhance the clustering mechanism. In summary, Fuzzy based energy efficient clustering introduces a function that measures the distance between the cluster center and the instance, which aids in improved clustering, and we then present the MobileNet V2 model to improve efficiency and speed up computation. To enhance the method’s performance and reduce its time complexity, the distributed database simulates the shared memory space and parallelizes on the MapReduce framework on the Hadoop cloud computing platform. The proposed approach is evaluated using performance metrics such as Accuracy, Precision, Adjusted Rand Index (ARI), Recall, F1-Score, and Normalized Mutual Information (NMI). The experimental findings indicate that the proposed approach outperforms the existing techniques in terms of clustering accuracy.

Solving Traveling Salesman Problem Using Parallel River Formation Dynamics Optimization Algorithm on Multi-core Architecture Using Apache Spark

According to Moore’s law, computer processing hardware technology performance is doubled every year. To make effective use of this technological development, the algorithmic solutions have to be developed at the same speed. Consequently, it is necessary to design parallel algorithms to be implemented on parallel machines. This helps to exploit the multi-core environment by executing multiple instructions simultaneously on multiple processors. Traveling Salesman (TSP) is a challenging non-deterministic-hard optimization problem that has exponential running time using brute-force methods. TSP is concerned with finding the shortest path starting with a point and returning to that point after visiting the list of points, provided that these points are visited only once. Meta-heuristic optimization algorithms have been used to tackle TSP and find near-optimal solutions in a reasonable time. This paper proposes a parallel River Formation Dynamics Optimization Algorithm (RFD) to solve the TSP problem. The parallelization technique depends on dividing the population into different processors using the Map-Reduce framework in Apache Spark. The experiments are accomplished in three phases. The first phase compares the speedup, running time, and efficiency of RFD on 1 (sequential RFD), 4, 8, and 16 cores. The second phase compares the proposed parallel RFD with three parallel water-based algorithms, namely the Water Flow algorithm, Intelligent Water Drops, and the Water Cycle Algorithm. To achieve fairness, all algorithms are implemented using the same system specifications and the same values for shared parameters. The third phase compares the proposed parallel RFD with the reported results of metaheuristic algorithms that were used to solve TSP in the literature. The results demonstrate that the RFD algorithm has the best performance for the majority of problem instances, achieving the lowest running times across different core counts. Our findings highlight the importance of selecting the most suitable algorithm and core count based on the problem characteristics to achieve optimal performance in parallel optimization.

A MapReduce-Based Ensemble Learning Method with Multiple Classifier Types and Diversity for Condition-based Maintenance with Concept Drifts

Condition-based maintenance (CBM) is to collect a huge amount of production data stream continuously from sensors or IoT devices attached to machines to forecast the time when to maintain machines or replace components. However, as conditions of machines change dynamically with time owing to machine aging or malfunction, the concept of capturing the foresting pattern from the data stream could drift unpredictably so that it is hard to find a robust forecasting method with high precision. Therefore, this work proposes an ensemble learning method with multiple classifier types and diversity for CBM in manufacturing industries. Aside from manipulating data diversity from previous works, the proposed method includes the features of multiple classifier types, dynamic weight adjusting, and data-based adaption to concept drifts for offline learning models, to promote the precision of the forecasting model and precisely detect and adapt to concept drifts. With these features, the proposed method requires powerful computing resources to efficiently respond to practical CBM applications in manufacturing industries. Therefore, furthermore, the implementation of the proposed method based on the MapReduce framework is proposed to increase computational efficiency. Simulation shows that the proposed method can detect and adapt to all concept drifts with a high precision rate.

In-Storage Computing for Hadoop MapReduce Framework: Challenges and Possibilities

Solid State Drives (SSDs) were initially developed as faster storage devices intended to replace conventional magnetic Hard Disk Drives (HDDs). However, high computational capabilities enable SSDs to be computing nodes, not just faster storage devices. Such capability is generally called ”In-Storage Computing (ISC)”. Today’s Hadoop MapReduce framework has become a de facto standard for big data processing. This paper explores In-Storage Computing challenges and opportunities for the Hadoop MapReduce framework. For this, we integrate a Hadoop MapReduce system with ISC SSD devices that implement the Hadoop Mapper inside real SSD firmware. This offloads Map tasks from the host MapReduce system to the ISC SSDs. We additionally optimize the host Hadoop system to make the best use of our proposed ISC Hadoop system. Experimental results demonstrate our ISC Hadoop MapReduce system achieves a remarkable performance gain (2.3 faster) as well as significant energy savings (11.5 lower) compared to a typical Hadoop MapReduce system. Further, the experiment suggests such ISC augmented systems can provide a very promising computing model in terms of a system scalability.

An Adaptive Salp-Stochastic-Gradient-Descent- Based Convolutional LSTM With MapReduce Framework for the Prediction of Rainfall

An Adaptive Salp-Stochastic-Gradient-Descent- Based Convolutional LSTM With MapReduce Framework for the Prediction of Rainfall

A powerful heuristic method for generating efficient database systems

Heuristic functions are an integral part of MapReduce software, both in Apache Hadoop and Spark. If the heuristic function performs badly, the load in the reduce part will not be balanced and access times spike. To investigate this problem closer, we run an optimal database program with numerous different heuristic functions on database. We will leverage the Amazon elastic MapReduce framework. The paper investigates on general purpose, implementation, and evaluation of heuristic algorithm for generating optimal database system, checksum, and special heuristic functions. With the analysis, we present the corresponding runtime results. For the coding part, the records counting part is hasty and can only work for local Hadoop part, it can be debugged and optimized for general purpose implement on Hadoop and Spark and turn into an effective performance monitor tool. As mentioned before, there are strange issue, also the performance of BLAKE2s is unexpectedly slow in that it’s widely accepted the performance of BLAKE2s is much better than MD5 and SHA256, we would like to figure out why the common-sense performance of heuristics is deferent from what we got in distributed frameworks.

Diabetes classification using MapReduce-based capsule network

Big data analytics is a complex exploratory process to uncover hidden data information from vast collections of data. It often provides enormous information from diverse sources and the use of analytics provides confined knowledge from the collected noisy data. In the case of diabetes data, there exist a massive collection of patient data that relates to significant information on patient health and its critical nature. In order of validating and analysing the data to get desired information about a patient and their health risk from the vast collection of data, the study uses bigdata based deep learning analytics. This study uses a Deep Learning Model namely capsule network (CapsNet) is executed on a MapReduce framework. The CapsNet present in the MapReduce framework enables the classification of instances via proper regulations. This model after suitable training with the training dataset enables optimal classification of instances to detect the nature of the risk of a patient. The validation conducted on the test dataset shows that the proposed CapsNets-based MapReduce model obtains increased accuracy, recall, and F-score than the conventional MapReduce and deep learning models.

Hybrid metaheuristic model based performance-aware optimization for map reduce scheduling

Because of the rapid rise in the count of corporations using cloud-dependent infrastructure as the foundation for big data storing and analysis. The fundamental difficulty in scheduling big data services in cloud-dependent systems is ensuring the shortest possible makespan while simultaneously reducing the number of resources being used. We have created a new, secure map reduce scheduling method that functions as follows. Initially, the cloud architecture is designed and the tasks are generated. In the pre-processing phase, the huge set of tasks was processed by the map-reduce scheduling framework. Afterward, the optimal task scheduling task is conducted which utilizes a hybrid algorithm named Tunicate Combined Moth Flame Algorithm (TCMFA), which provides better task scheduling via providing optimal makespan, execution time, and security. This proposed TCMFA is the hybridization of both Moth Flame Optimization (MFO) and Tunicate Swarm Algorithm (TSA). The error rate of the TCMFA gets reduced to 320 approximately over other conventional methods such as RSA, ACO, GHO, BTS, OWPSO, BES, PRO, SOA, COOT, TSA & MFO which proves the accuracy of our TCMFA and makes it more efficient and secure for optimal map-reduce scheduling.

Modeling of class imbalance handling with optimal deep learning enabled big data classification model

Big data is the amount of data that surpasses the ability to process the data of a system concerning memory usage and computation time. It is commonly applied in several domains like healthcare, education, social networks, e-commerce, etc., as they have progressively obtained a massive quantity of input data. A major research problem is big data analytics, which can be carried out using expert systems and deep structured architectures. Besides, data wrangling and class imbalance data handling are challenging issues that need to be resolved in big data analytics. Class imbalance data degrade the performance of the classification model, which remains a challenging process due to the heterogeneous and complex structure of the comparatively huge datasets. Thus, the research focused on presenting a Class Imbalance Handling with Optimal Deep Learning Enabled Big Data Classification (CIHODL-BDC) framework. The core perception of the CIHODL-BDC framework helps to classify the big data in the Hadoop MapReduce framework. To accomplish this, the presented CIHODL-BDC model initially performs a data wrangling process is performed to alter the unrefined data into a useful layout. Next, the CIHODL-BDC model handles the class imbalance problem using a grey wolf optimizer (GWO) with Synthetic Minority Oversampling (SMOTE) technique. Besides, the Adam optimizer procedure with the Bidirectional Long Short Term Memory (BiLSTM) approach is performed to categorize the big data. The result analysis of the proposed CIHODL-BDC model is evaluated by two standard datasets. The simulation outcomes revealed the elevated performance of the CIHODL-BDC approach over existing methods.

A Novel Nodesets-Based Frequent Itemset Mining Algorithm for Big Data using MapReduce

Due to the rapid growth of data from different sources in organizations, the traditional tools and techniques that cannot handle such huge data are known as big data which is in a scalable fashion. Similarly, many existing frequent itemset mining algorithms have good performance but scalability problems as they cannot exploit parallel processing power available locally or in cloud infrastructure. Since big data and cloud ecosystem overcomes the barriers or limitations in computing resources, it is a natural choice to use distributed programming paradigms such as Map Reduce. In this paper, we propose a novel algorithm known as A Nodesets-based Fast and Scalable Frequent Itemset Mining (FSFIM) to extract frequent itemsets from Big Data. Here, Pre-Order Coding (POC) tree is used to represent data and improve speed in processing. Nodeset is the underlying data structure that is efficient in discovering frequent itemsets. FSFIM is found to be faster and more scalable in mining frequent itemsets. When compared with its predecessors such as Node-lists and N-lists, the Nodesets save half of the memory as they need only either pre-order or post-order coding. Cloudera's Distribution of Hadoop (CDH), a MapReduce framework, is used for empirical study. A prototype application is built to evaluate the performance of the FSFIM. Experimental results revealed that FSFIM outperforms existing algorithms such as Mahout PFP, Mlib PFP, and Big FIM. FSFIM is more scalable and found to be an ideal candidate for real-time applications that mine frequent itemsets from Big Data.

Evaluation of Responsive With Big Data Technologies

The volume of produced digital data & everyday usage is increasing as a result of rising man-machine interaction, process automation, and falling hardware and software costs.
 The vast amount of digital information produced per second worldwide in unstructured, structured, and semi-structured formats is referred to as big data in this context.
 Big data analytics is a new topic that has inspired researchers all over the world to create, develop, and implement a variety of mechanisms, technologies, architectures, and areas for evaluating the enormous amount of data created every day. Traditional database management systems struggle to make sense of the massive amounts of data that make up "big data." The paper describes a few analyses, including word count, sentiment analysis, and responsive analysis.
 Responsiveness is crucial for improving the system, identifying vulnerabilities, and ensuring that tasks are distributed fairly. Responsive is important data that can be used for making wise decisions.
 Responsive is significant for both strengths and weaknesses, not just when it identifies flaws. If the responsive analysis is performed incorrectly, the analysis's conclusion will likewise be incorrect. As a result, the system as a whole will be inaccurate because the pattern that was discovered will also be incorrect. We will utilize the Map-Reduce framework to create this suggested system for responsive analysis, and Hadoop will be used for storage.