MapReduce Research Articles

Security issues analysis based on big data in cloud computing

This study will investigate the security problems that are linked with the Hadoop ecosystem, Big Data, Map Reduce, and cloud computing. The objective of this research is to analyze these concerns. Specifically, the primary emphasis is placed on the security concerns that are linked with big data in cloud computing. Big data applications have a lot to offer organizations, companies, and a broad range of industries, no matter how big or little. There have been issues with Hadoop and cloud computing security, therefore we also look at several possible solutions to these difficulties. The field of cloud computing security is experiencing fast growth in several areas, including computer security, network security, information security, and data privacy. Data, apps, and the infrastructure related to them are greatly helped by cloud computing's use of rules, technologies, controls, and big data tools to keep everything safe. Researchers also anticipate that cloud computing, big data, and their associated benefits will represent the most exciting new areas of study.

Cyber-attack Detection and Mitigation Process under Big Data Consideration: Improved Recursive Feature Elimination-based Feature Selection

Due to the rapid growth of network technology, huge volume and distinct data sent via networks is expanding constantly. The situation shows how complex and dense cyber attacks and hazards are developing. Due to the rapid advancement in network density, cyber security specialists find it difficult to monitor all network activity. Due to frequent and sophisticated cyber attacks, it is becoming more challenging to detect and identify abnormalities in network events. The use of deep learning provides a variety of tools and strategies for automated cyber-attack detection as well as quick attack-type prediction as well as evaluation. This work introduces a novel cyber-attack detection and mitigation process under the following phases including preprocessing, feature extraction via the Map Reduce framework that handles the big data, feature selection, attack detection and mitigation. The Improved Normalisation process is achieved on the preprocessing phase. The work is examined from a big data perspective; hence Map Reduce framework is utilised for this. As a result, the framework will manage the feature extraction process, where features including statistical features, raw features, improved correlation-based features, and info gain-based features will be extracted. Following feature extraction, the Improved Recursive Feature Elimination procedure is processed that selects the relevant features. The hybrid detection model, which combines Recurrent Neural Networks (RNN) Deep and Belief Networks (DBN) is used to detect the attacks. Once an attack has been detected, the attacker must be mitigated. To accomplish this, an improved BAIT-based mitigation procedure is used. The two datasets used in this work are, namely, Intrusion Detection Systems (IDS) 2018 Intrusion CSVs (CSE-CIC-IDS2018) and UNSW_NB15. Finally, the suggested model and the alternative methods are contrasted using a variety of measures such as accuracy, sensitivity, specificity, precision, FDR, FNR and FPR.

Input/output optimization scheduler for cloud-based map reduce framework

Hadoop MapReduce (HMR) provides the most common MapReduce (MR) framework, and it is available as open source. MR is a famous computational framework for evaluating unstructured, and semi-structured big data and executing applications in the past ten years. Memory and input/output (I/O) overhead are just two of the many problems affecting the current HMR scheduler system. This study aims to improve systems resource use including the processing of data in real-time by creating a memory I/O optimized scheduler (MIOOS) for HMR. The disk I/O seek can be reduced by using MIOOS, which analyzes the entire memory management. Additionally, the MIOOS makespan approach is used to reduce the occurrence of problems in intermediary tasks. Both the MIOOS approach and the current approach are assessed by using complex scientific workflow applications with extreme task inter-dependencies. Further, the comparison study demonstrates that the MIOOS framework outdoes the current approach regarding makespan and overall memory usage.

Cost minimization approach with hybrid optimization based task scheduling in Geo-distributed cloud

ABSTRACT To manage the huge data, cloud computing necessitates a significant number of disc I/O, network bandwidth and CPU cycles. To handle the massive volumes of data, the programming paradigm known as data flow integrates the tasks to a graph structure. Task scheduling is one of the main areas of current study that primarily aims to allocate the tasks to resources. It is essential to organise the tasks in a way which saves time and makes the most use of available resources. In recent years, scholars have offered different job scheduling techniques. This work intends to introduce a novel cost minimisation for big data processing with respect to task allocation or scheduling. Initially, the big data is processed under Map Reduce framework. This allocation processes is taken as the optimisation issue, which is solved under the consideration of minimisation factors like execution cost, makespan, communication cost, and energy consumption and risk. For solving this issue, a new hybrid optimisation algorithm known as Hybrid bald eagle and Archimedes Optimisation algorithm (HBE-ArCA) is developed. The betterment of HBE-ArCA is proven regarding cost, makespan and so on.

Bigdata clustering and classification with improved fuzzy based deep architecture under MapReduce framework

The current state of economic, social ideas, and the advancement of cutting-edge technology are determined by the primary subjects of the contemporary information era, big data. People are immersed in a world of information, guided by the abundance of data that penetrates every element of their surroundings. Smart gadgets, the IoT, and other technologies are responsible for the data’s explosive expansion. Organisations have struggled to store data effectively throughout the past few decades. This disadvantage is related to outdated, expensive, and inadequately large storage technology. In the meanwhile, large data demands innovative storage techniques supported by strong technology. This paper proposes the bigdata clustering and classification model with improved fuzzy-based Deep Architecture under the Map Reduce framework. At first, the pre-processing phase involves data partitioning from the big dataset utilizing an improved C-Means clustering procedure. The pre-processed big data is then handled by the Map Reduce framework, which involves the mapper and reducer phases. In the mapper phase. Data normalization takes place, followed by the feature fusion approach that combines the extracted features like entropy-based features and correlation-based features. In the reduction phase, all the mappers are combined to produce an acceptable feature. Finally, a deep hybrid model, which is the combination of a DCNN and Bi-GRU is used for the classification process. The Improved score level fusion procedure is used in this case to obtain the final classification result. Moreover, the analysis of the proposed work has proved to be efficient in terms of classification accuracy, precision, recall, FNR, FPR, and other performance metrics.

Multi-objective hybrid optimized task scheduling in cloud computing under big data perspective

The new and rising paradigm of cloud computing offers customers various possibilities of task computation based on their desires and choices. Customers receive services from cloud computing systems as a utility. Customers are enthusiastic about low-cost service availability and task completion times that are kept to be minimum. To achieve client fulfilment, the service provider must schedule the jobs to the right resources if the cloud server gets many user requests. The rapid growth in data volume necessitates petabytes processing of data each day. Unstructured, semi-structured, and structured data are all described in terms of their rapid growth and availability. In order to make correct and timely decisions, it must be processed appropriately. In this research, we present BWUJS (Black Widow Updated Jellyfish Search), a multi-objective hybrid optimization-based task scheduling algorithm. This work considers task generation from the Bigdata perspective. The clustering of tasks is performed via the Map Reduce framework with an Improved K-means clustering model. After task clustering, the task priority estimation is performed. Finally, the scheduling is performed via BWJSU based on certain constraints like priority, makespan, completion time, resource utilization, and degree of imbalance.

Cost-aware optimal resource provisioning Map-Reduce scheduler for hadoop framework

<p><span lang="EN-US">Distributed data processing model has been one of the primary components in the case of data-intensive applications; furthermore, due to advancements in technologies, there has been a huge volume of data generation of diverse nature. Hadoop map reduce framework is responsible for adopting the ease of deployment mechanism in an open-source framework. The existing Hadoop MapReduce framework possesses high makespan time and high Input/Output overhead and it mainly affects the cost of a model. Thus, this research work presents an optimized cost aware resource provisioning MapReduce model also known as the cost-effective resource provisioning MapReduce (CRP-MR) model. CRP-MR model introduces the two integrated approaches to minimize the cost; at first, this model presents the optimal resource optimization and optimal Input/Output optimization cleansing in the Hadoop MapReduce (HMR) scheduler. CRP-MR is evaluated considering the bioinformatics dataset and CRP-MR performs better than the existing model. </span></p>

Optimization enabled LeNet for big data classification using MapReduce framework on COVID-19 data

ABSTRACT The big data clustering is a requisite for generating the data in the digitalised globe. The old-fashioned clustering approaches are not large sized and highly unorganised big data. Thus, to obtain the efficiency of big data clustering, a new architecture is required. This work introduces the Jaya African Vulture Optimization Algorithm-based LeNet (JAVO-based LeNet) for big data classification on COVID-19. The big data clustering is detected by using the LeNet model, which is tuned by the JAVO algorithm. Here, the JAVO algorithm is the integration of the Jaya algorithm and AVOA algorithm. Further, using the Deep Embedded Clustering (DEC) technique for partitioning the smaller data to improve the performance of big data classification. The classification process for COVID-19 is established in the MapReduce (MR) framework. For each mapper, the pre-processing is completed based on Min-Max normalisation and the feature fusion is performed based on the Hellinger distance, which is measured with a deep residual network (DRN). Finally, the classification process for COVID-19 is done by considering the fused data acquired from each mapper with LeNet, which is tuned by the JAVO algorithm. Furthermore, the analysis of the proposed method attains the testing accuracy of 0.948, specificity of 0.939 and sensitivity of 0.950.

MapReduce Algorithm on a Serverless Platform

Using the MapReduce (MR) programming technique, large-scale data processing tasks can be divided into tasks that are easier to manage and independent. The serverless implementation's performance is better than the non-serverless MR model when the model parameters are altered in the experiment. Optimizing the use of the MR model on the serverless platform can be achieved by taking into account the relationship between implementation efficiency and platform settings. Additionally, it can act as a source of inspiration for future serverless hardware support configurations. Also, the serverless platform demonstrates how it improves the effectiveness of resource utilization in machine learning training. The intended result can then be achieved by combining the results of these concurrently operating jobs on server clusters. This work adapts MR, a popular big data processing framework, to the serverless platform, emphasizing realization simulation principles and services, and then uses the results in a word count experiment. The experiment uses a word count of about eleven thousand words to evaluate how well MR is implemented on Alibaba Cloud. It is validated for execution time on the platform with varying Central Processing Unit (CPU) core counts, memory configurations, and worker counts. By testing several platform configurations, it is found that the memory configuration has very little impact on the model's execution time, while the size of the CPU core has a considerable impact on reaction time relative to the number of workers.

Improving big data analytics data processing speed through map reduce scheduling and replica placement with HDFS using genetic optimization techniques

Big Data Analytics (BDA) is an unavoidable technique in today’s digital world for dealing with massive amounts of digital data generated by online and internet sources. It is kept in repositories for data processing via cluster nodes that are distributed throughout the wider network. Because of its magnitude and real-time creation, big data processing faces challenges with latency and throughput. Modern systems such as Hadoop and SPARK manage large amounts of data with their HDFS, Map Reduce, and In-Memory analytics approaches, but the migration cost is higher than usual. With Genetic Algorithm-based Optimization (GABO), Map Reduce Scheduling (MRS) and Data Replication have provided answers to this challenge. With multi objective solutions provided by Genetic Algorithm, resource utilization and node availability improve processing performance in large data environments. This work develops a novel creative strategy for enhancing data processing performance in big data analytics called Map Reduce Scheduling Based Non-Dominated Sorting Genetic Algorithm (MRSNSGA). The Hadoop-Map Reduce paradigm handles the placement of data in distributed blocks as a chunk and their scheduling among the cluster nodes in a wider network. Best fit solutions with high latency and low accessing time are extracted from the findings of various objective solutions. Experiments were carried out as a simulation with several inputs of varied location node data and cluster racks. Finally, the results show that the speed of data processing in big data analytics was enhanced by 30–35% over previous methodologies. Optimization approaches developed to locate the best solutions from multi-objective solutions at a rate of 24–30% among cluster nodes.

MapReduce based on serverless platforms

This study intends to investigate the application of the MapReduce (MR) framework based on serverless computing in big data processing. By combining the MapReduce model with serverless computing, efficient data processing is achieved. In this framework, the phases of Map task execution, reduce task execution, etc. are accomplished through stateless serverless functions, and data storage is realized with the help of cloud storage platforms (e.g., OSS). In this paper, the author introduces the basic theory of MR, the basic theory of serverless computing, describes the framework implementation process, and discusses the role of OSS in distributed computing. The outcomes of the trial indicate the average execution time of the framework for a WordCount task on 100 pieces of TOEFL English reading data is 6.81 seconds. The discussion analyzes the frameworks advantages (high elasticity, resource utilization) and disadvantages (cold start latency, unsuitable for long time tasks). Future research directions encompass performance optimization, long-time task processing, state management, etc. In summary, this study provides valuable insights for the practical application of serverless computing in big data processing.

A Study on Hadoop and Map Reduce (MR) Architecture

Apache Hadoop is an open-source programming system for dispersed capacity and disseminated preparing of huge informational indexes on PC bunches worked from item equipment. Every one of the modules in Hadoop are outlined with a principal presumption that equipment disappointments are normal and ought to be naturally taken care of by the structure. The center of Apache Hadoop comprises of a capacity part and a preparing part which is otherwise called Hadoop circulated record framework (HDFS) and Map Reduce separately. Hadoop parts records into expansive squares and disseminates them crosswise over hubs in a bunch.

Aiding secure data retrieval incorporated with parallelization technique in cloud

<p>Cloud data owners prefer to outsource data because of ease in maintenance. Data confidentiality of this outsourced sensitive data is a major task. The searchable encryption technique helps to carry out searches on encrypted data without decrypting it. In the data outsourcing environment, volume of data is increasing rapidly. Hence the time required to build the index and to carry out searches is also increasing exponentially. This makes it more difficult to build a system which is efficient, reliable and can cope up with growing data. In this paper, a parallelization technique to build the index on outsourced data is proposed. This technique minimizes the time required to construct the index. It also supports secure ranked retrieval using bucketization technique. The buckets are formed using Hadoop map reduce framework which achieves significant efficiency. The proposed method prune the keyword dataset, which helps in significant reduction in the size of index. Through extensive experiments using standard dataset, the performance of the system is validated. The experimental results show that the proposed system requires less time for index construction and hence improves retrieval efficiency.</p>

Rapid Recover Map Reduce (RR-MR): Boosting failure recovery in Big Data applications

Rapid Recover Map Reduce (RR-MR): Boosting failure recovery in Big Data applications

Map Reduce approach for road accident data analysis using data mining techniques

Map Reduce approach for road accident data analysis using data mining techniques

A Novel Nodesets-Based Frequent Itemset Mining Algorithm for Big Data using MapReduce

Due to the rapid growth of data from different sources in organizations, the traditional tools and techniques that cannot handle such huge data are known as big data which is in a scalable fashion. Similarly, many existing frequent itemset mining algorithms have good performance but scalability problems as they cannot exploit parallel processing power available locally or in cloud infrastructure. Since big data and cloud ecosystem overcomes the barriers or limitations in computing resources, it is a natural choice to use distributed programming paradigms such as Map Reduce. In this paper, we propose a novel algorithm known as A Nodesets-based Fast and Scalable Frequent Itemset Mining (FSFIM) to extract frequent itemsets from Big Data. Here, Pre-Order Coding (POC) tree is used to represent data and improve speed in processing. Nodeset is the underlying data structure that is efficient in discovering frequent itemsets. FSFIM is found to be faster and more scalable in mining frequent itemsets. When compared with its predecessors such as Node-lists and N-lists, the Nodesets save half of the memory as they need only either pre-order or post-order coding. Cloudera's Distribution of Hadoop (CDH), a MapReduce framework, is used for empirical study. A prototype application is built to evaluate the performance of the FSFIM. Experimental results revealed that FSFIM outperforms existing algorithms such as Mahout PFP, Mlib PFP, and Big FIM. FSFIM is more scalable and found to be an ideal candidate for real-time applications that mine frequent itemsets from Big Data.

MR-AT: Map Reduce based Apriori Technique for Sequential Pattern Mining using Big Data in Hadoop

One of the most well-known and widely implemented data mining methods is Apriori algorithm which is responsible for mining frequent item sets. The effectiveness of the Apriori algorithm has been improved by a number of algorithms that have been introduced on both parallel and distributed platforms in recent years. They are distinct from one another on account of the method of load balancing, memory system, method of data degradation, and data layout that was utilised in their implementation. The majority of the issues that arise with distributed frameworks are associated with the operating costs of handling distributed systems and the absence of high-level parallel programming languages. In addition, when using grid computing, there is constantly a possibility that a node will fail, which will result in the task being re-executed multiple times. The MapReduce approach that was developed by Google can be used to solve these kinds of issues. MapReduce is a programming model that is applied to large-scale distributed processing of data on large clusters of commodity computers. It is effective, scalable, and easy to use. MapReduce is also utilised in cloud computing. This research paper presents an enhanced version of the Apriori algorithm, which is referred to as Improved Parallel and Distributed Apriori (IPDA). It is based on the scalable environment referred as Hadoop MapReduce, which was used to analyse Big Data. Through the generation of split-frequent data regionally and the early elimination of unusual data, the proposed work has its primary objective to reduce the enormous demands placed on available resources as well as the reduction of the overhead communication that occurs whenever frequent data are retrieved. The paper presents the results of tests, which demonstrate that the IPDA performs better than traditional apriori and parallel and distributed apriori in terms of the amount of time required, the number of rules created, and the various minimum support values.

Enhancing Data Engineering Frameworks for Scalable Real-Time Marketing Solutions

This report discusses improvements to the data engineering frameworks for cost-efficient real-time marketing environment, with the applications in mobile advertising and smart city analytics. The proposed framework combines the processing of time-series data with map-reduce (M-R) systems in order to compensate for inefficiencies in the execution of temporal queries and the processing of real-time data streams. It enables effective behavioral targeting in mobile marketing and helps integrate various domains’ real-time data for smart city utilization. Specific examples that illustrate further refinements in the algorithms are showcased through case study and experimental evidence where gains in throughput, data reliability, and system effectiveness are quantified in real-time Marketing and Smart City applications.

CREATING A HIGH-PERFORMANCE HADOOP CLUSTER CONSIST OF WEAK COMPUTERS WITHIN THE ORGANIZATION

Knowledge of practical methods of processing large amounts of data is akin to black magic. There are many tools and techniques for scalable data processing, in particular, caching (for example, using the mem cached program), replication, partitioning and, of course, Map Reduce/Hadoop. Hadoop is an open source framework that implements the Map Reduce distribution and reduction algorithm, which underlies Google's approach to organizing queries to distributed datasets that make up the Internet. This article is intended for programmers, architects and project managers involved in processing large amounts of data offline. It will describe how to get a copy of Hadoop, how to organize a cluster and how to write analysis programs. We will start by applying Hadoop in the default configuration to solving a few simple tasks, for example, analyzing changes in the frequency of occurrence of words in the corpus of documents; this will help to understand the basic ideas of Hadoop and MapReduce. Next, we will move on to the basic concepts of MapReduce applications developed using Hadoop, and along the way we will study the components of the framework, the application of Hadoop to a wide range of data analysis tasks and numerous examples of Hadoop in action. Keywords: big data, cluster, Hadoop.

Database System Based on 3Dmax Big Data Mining Technology

INTRODUCTION: This project intends to study the mining method of FP-growth frequent items in 3Dmax big data under the Hadoop framework and combined with the Map Reduce development model. Firstly, the transaction database is selected according to the frequency of each transaction and the corresponding projection library is generated. Then the obtained image database is distributed on each node computer. Then, under the guidance of the node machine, the projection is divided into different regions to produce several smaller sub-databases. The method is parallelized by using node machine to generate local frequency items. Finally, all the local frequency sets are merged into one complete frequency set. This method does not need to generate as many FP trees as the regular FP-growth method. This method can overcome the computational failure problem caused by the limited memory of a single computer by the conventional FP-Growth method and other methods. At the same time, because the sublibraries of partitions are similar in size, the load distributed to each node machine is more balanced. The effectiveness of the algorithm is improved.