MapReduce Environment Research Articles

Modelling of healthcare data analytics using optimal machine learning model in big data environment

AbstractRecent advances in wireless networking, big data technologies, namely Internet of Things (IoT) 5G networks, health care big data analytics, and other technologies in artificial intelligence (AI) and wearables, have supported the progression of intellectual disease diagnosis methods. Medical data covers all patient data such as pharmacy texts, electronic health reports (EHR), prescriptions, study data from medical journals, clinical photographs, and diagnostic reports. Big data is a renowned method in the healthcare sector, with beneficial datasets that are highly difficult, voluminous, and rapid for healthcare providers for interpreting and computing using prevailing tools. This study combines concepts like deep learning (DL) and big data analytics in medical field. This article develops a new healthcare data analytics using optimal machine learning model in big data environment (HDAOML‐BDE) technique. The presented HDAOML‐BDE technique mainly aims to examine the healthcare data for disease detection and classification in the big data environment. For handling big data, the HDAOML‐BDE technique uses Hadoop MapReduce environment. In addition, the HDAOML‐BDE technique uses manta ray foraging optimization‐based feature selection (MRFO‐FS) technique to reduce high dimensionality problems. Moreover, the HDAOML‐BDE method uses relevance vector machine (RVM) model for the healthcare data environment. Furthermore, the arithmetic optimization algorithm (AOA) is utilized for the parameter tuning of the RVM classifier. The simulation results of the HDAOML‐BDE technique are tested on a healthcare dataset, and the outcomes portray the improved performance of the HDAOML‐BDE strategy over recent approaches in different measures.

The Application of Wireless Network Technology in the Capacity Building of Anhui Industry Associations Based on the Characteristics of Big Data

Recent advances of wireless networks and communication technologies, integrated into the considerable development in the field of environmental policies and public participation, have resulted in the development of wireless network-enabled big data environment in the capacity building of Anhui industry association. At the same time, effective task scheduling in the cloud server based big data environment is essential to schedule the tasks accurately and rapidly. There are thousands of jobs to be executed by the resources available on cloud data centers to achieve minimum time, high performance, and the proper utilization of CPU and resources. The necessity to fulfil user requirements is the main reason of having studies for optimizing the cloud computing of big data in terms of latency, bandwidth, execution time, and resource utilization. Therefore, this work presents a crow search optimization-based task scheduling scheme (CSO-TSS) for capacity building of Anhui industry association on wireless network-enabled big data environment. The proposed CSO-TSS technique mainly intends to schedule the tasks exist in the wireless network-enabled big data environment. Besides, the CSO-TSS technique is executed on a MapReduce environment in order to proficiently handle the big data. In addition, the CSO-TSS technique derives an objective function intending to maximize resource utilization and minimize execution time of the tasks. For examining the improved task scheduling performance of the CSO-TSS technique, a wide range of simulations were carried out and the results are investigated under several aspects. The comparative result analysis stated the better outcomes of the CSO-TSS technique over the recent approaches in terms of different measures.

Handling uncertainty using optimal clustering with rough sets‐based rule generation model for data classification

AbstractIn recent times, MapReduce has become a popular tool for handling big data. At the same time, uncertainty is related to arbitrariness, fuzziness, ambiguity, irregularity and incomplete knowledge. In RS theory, the uncertainty behaviour of the data in the dataset of interest is managed by using upper and lower approximate sets and classification accuracy. The RS model is integrated with data clustering technique for optimal outcomes. With this motivation, this study designs an Optimal Clustering with RS‐Based Rule Generation Model (OC‐RSRGM) for data classification on MapReduce environment. The OC‐RSRGM technique aims to generate an optimal set of rules using RST for the data classification process and it involves a two‐stage process namely Optimal Fuzzy c‐Means Clustering (OFCM) and RSRGM‐based rule generation with classification. The OFCM technique is derived to eradicate the local optimal problem of the FCM (Fuzzy c‐Means) model using Barnacles Mating Optimizer (BMO). It provides the decision‐makers with all the information needed to design appropriate mechanisms to support their decision‐making activities. The Hadoop MapReduce tool is used to handle big data. The proposed method combines an FCM, BMO, RS theory to accomplish effective decision‐making. The OC‐RSRGM technique can be employed to continuous value dataset where data point does not offer any class details and it might be uncertain. To validate the performance of OC‐RSRGM technique, a detailed experimental analysis is carried out to highlights the betterment of OC‐RSRGM technique. The proposed OC‐RSRGM technique has obtained an effective outcome with the CT of 5.43 s.

Computational intelligence based sustainable computing with classification model for big data visualization on map reduce environment

In recent years, the researchers have perceived the modifications or transformations motivated by the presence of big data on the definition, complexity, and future direction of the real world optimization problems. Big Data visualization is mainly based on the efficient computer system for ingesting actual data and producing graphical representation for understanding large quantity of data in a fraction of seconds. At the same time, clustering is an effective data mining tool used to analyze big data and computational intelligence (CI) techniques can be employed to solve big data classification process. In this aspect, this study develops a novel Computational Intelligence based Clustering with Classification Model for Big Data Visualization on Map Reduce Environment, named CICC-BDVMR technique. The proposed CICC-BDVMR technique intends to perform effective BDV using the clustering and data classification processes on the Map Reduce environment. For clustering process, a grasshopper optimization algorithm (GOA) with kernelized fuzzy c-means (KFCM) technique is used to cluster the big data and the GOA is mainly utilized to determine the initial cluster centers of the KFCM technique. GOA is a recently proposed metaheuristic algorithm inspired by the swarming behaviour of grasshoppers. This algorithm has been shown to be efficient in tackling global unconstrained and constrained optimization problems. Based on the modified GOA, an effective kernel extreme learning machine model for financial stress prediction was created. Besides, big data classification process takes place using the Ridge Regression (RR) and the parameter optimization of the RR model is carried out via the Red Colobuses Monkey (RCM) algorithm. The design of GOA and RCM algorithms for parameter optimization processes for big data classification shows the novelty of the study. A wide ranging simulation analysis is carried out using benchmark big datasets and the comparative results reported the enhanced outcomes of the CICC-BDVMR technique over the recent state of art approaches. The broad comparison research illustrates the CICC-BDVMR approach’s promising performance against contemporary state-of-the-art techniques. As a result, the CICC-BDVMR technique has been demonstrated to be an effective technique for visualising and classifying large amounts of data.

Automatic Taxonomy Generation and Incremental Evolution on Apache Spark Parallelization Framework

The term “Big Data” refers to a large volume of information usually in terabytes and petabytes. It includes both structured and unstructured data. Unstructured data is conventionally text-heavy, but may also contain data such as facts, dates and numbers. To use this unstructured information effectively, it needs to be processed and organized. Taxonomy is considered as a powerful way of organizing the information. For automatic taxonomy generation, various techniques have been proposed in the past. However, the substantial nature of big data presently crosses the processing abilities of traditional techniques. Thus, to meet this challenge an extensible and scalable technique is required to potentially accelerate the process of taxonomy generation and its evolution upon arrival of new data, hence catering large amount of unstructured big data. This paper proposes a technique for both the taxonomy generation and evolution on Apache Spark infrastructure. The proposed technique is evaluated on a text dataset of a computing domain. The evaluation results show that the technique presented in this paper outperformed the existing techniques in terms of time and quality metrics. The time and quality-based evaluation showed that the use of MapReduce environment has resolved the scalability issues of current taxonomy generation and evolution process.

MapReduce-based big data classification model using feature subset selection and hyperparameter tuned deep belief network

In recent times, big data classification has become a hot research topic in various domains, such as healthcare, e-commerce, finance, etc. The inclusion of the feature selection process helps to improve the big data classification process and can be done by the use of metaheuristic optimization algorithms. This study focuses on the design of a big data classification model using chaotic pigeon inspired optimization (CPIO)-based feature selection with an optimal deep belief network (DBN) model. The proposed model is executed in the Hadoop MapReduce environment to manage big data. Initially, the CPIO algorithm is applied to select a useful subset of features. In addition, the Harris hawks optimization (HHO)-based DBN model is derived as a classifier to allocate appropriate class labels. The design of the HHO algorithm to tune the hyperparameters of the DBN model assists in boosting the classification performance. To examine the superiority of the presented technique, a series of simulations were performed, and the results were inspected under various dimensions. The resultant values highlighted the supremacy of the presented technique over the recent techniques.

An Intelligent Metaheuristic Binary Pigeon Optimization-Based Feature Selection and Big Data Classification in a MapReduce Environment

Big Data are highly effective for systematically extracting and analyzing massive data. It can be useful to manage data proficiently over the conventional data handling approaches. Recently, several schemes have been developed for handling big datasets with several features. At the same time, feature selection (FS) methodologies intend to eliminate repetitive, noisy, and unwanted features that degrade the classifier results. Since conventional methods have failed to attain scalability under massive data, the design of new Big Data classification models is essential. In this aspect, this study focuses on the design of metaheuristic optimization based on big data classification in a MapReduce (MOBDC-MR) environment. The MOBDC-MR technique aims to choose optimal features and effectively classify big data. In addition, the MOBDC-MR technique involves the design of a binary pigeon optimization algorithm (BPOA)-based FS technique to reduce the complexity and increase the accuracy. Beetle antenna search (BAS) with long short-term memory (LSTM) model is employed for big data classification. The presented MOBDC-MR technique has been realized on Hadoop with the MapReduce programming model. The effective performance of the MOBDC-MR technique was validated using a benchmark dataset and the results were investigated under several measures. The MOBDC-MR technique demonstrated promising performance over the other existing techniques under different dimensions.

Sentiment Weighted Word Embedding for Big Text Data

Sentiment analysis is the practice of eliciting a sentiment orientation of people's opinions (i.e. positive, negative and neutral) toward the specific entity. Word embedding technique like Word2vec is an effective approach to encode text data into real-valued semantic feature vectors. However, it fails to preserve sentiment information that results in performance deterioration for sentiment analysis. Additionally, big sized textual data consisting of large vocabulary and its associated feature vectors demands huge memory and computing power. To overcome these challenges, this research proposed a MapReduce based Sentiment weighted Word2Vec (MSW2V), which learns the sentiment and semantic feature vectors using sentiment dictionary and big textual data in a distributed MapReduce environment, where memory and computing power of multiple computing nodes are integrated to accomplish the huge resource demand. Experimental results demonstrate the outperforming performance of the MSW2V compared to the existing distributed and non-distributed approaches.

Efficient k-dominant skyline query over incomplete data using MapReduce

Skyline queries are extensively incorporated in various real-life applications by filtering uninteresting data objects. Sometimes, a skyline query may return so many results because it cannot control the retrieval conditions especially for high-dimensional datasets. As an extension of skyline query, the k-dominant skyline query reduces the control of the dimension by controlling the value of the parameter k to achieve the purpose of reducing the retrieval objects. In addition, with the continuous promotion of Bigdata applications, the data we acquired may not have the entire content that people wanted for some practically reasons of delivery failure, no power of battery, accidental loss, so that the data might be incomplete with missing values in some attributes. Obviously, the k-dominant skyline query algorithms of incomplete data depend on the user definition in some degree and the results cannot be shared. Meanwhile, the existing algorithms are unsuitable for directly used to the incomplete big data. Based on the above situations, this paper mainly studies k-dominant skyline query problem over incomplete dataset and combines this problem with the distributed structure like MapReduce environment. First, we propose an index structure over incomplete data, named incomplete data index based on dominate hierarchical tree (ID-DHT). Applying the bucket strategy, the incomplete data is divided into different buckets according to the dimensions of missing attributes. Second, we also put forward query algorithm for incomplete data in MapReduce environment, named MapReduce incomplete data based on dominant hierarchical tree algorithm (MR-ID-DHTA). The data in the bucket is allocated to the subspace according to the dominant condition by Map function. Reduce function controls the data according to the key value and returns the k-dominant skyline query result. The effective experiments demonstrate the validity and usability of our index structure and the algorithm.

Security Recommendatios And Security Issues Associated With Big Data In Cloud Computing

In this paper, we discuss security surveillance for Big data, cloud computing, Map Reduce and Hadoop environment. The focal point is on security surveillance in cloud computing that are corresponding with big data. Big data applications are a admirable beneficent to organizations, business, companies and many large scale and small scale industries.We also talk about various attainable solutions for the issues in Hadoop and cloud computing security. Cloud computing security is being blossom at a rapid pace which incorporate with computer security, network security, information security, and data privacy.Cloud computing plays a very crucial role in keep safe data, applications and the related infrastructure with the help of technologies, policies,controls , and big data tools. Moreover, Cloud computing, big data and its applications, advantages are likely to illustrated the most hopeful new boundaries in science.

MapReduce Implementation of a Multinomial and Mixed Naive Bayes Classifier

This study presents an efficient way to deal with discrete as well as continuous values in Big Data in a parallel Naïve Bayes implementation on Hadoop's MapReduce environment. Two approaches were taken: (i) discretizing continuous values using a binning method; and (ii) using a multinomial distribution for probability estimation of discrete values and a Gaussian distribution for probability estimation of continuous values. The models were analyzed and compared for performance with respect to run time and classification accuracy for varying data sizes, data block sizes, and map memory sizes.

A Hybrid Model for Near-Duplicate Image Detection in MapReduce Environment

A Hybrid Model for Near-Duplicate Image Detection in MapReduce Environment

Positive and negative association rule mining in Hadoop\u2019s MapReduce environment

In this paper, we present a Hadoop implementation of the Apriori algorithm. Using Hadoop’s distributed and parallel MapReduce environment, we present an architecture to mine positive as well as negative association rules in big data using frequent itemset mining and the Apriori algorithm. We also analyze and present the results of a few optimization parameters in Hadoop’s MapReduce environment as it relates to this algorithm. The results are presented based on the number of rules generated as well as the run-time efficiency. We find that, a higher amount of parallelization, which means larger block sizes, will increase the run-time efficiency of the Hadoop implementation of the Apriori algorithm.

A privacy protection-oriented parallel fully homomorphic encryption algorithm in cyber physical systems

Cyber physical system (CPS) is facing enormous security challenges because of open and interconnected network and the interaction between cyber components and physical components, the development of cyber physical systems is constrained by security and privacy threats. A feasible solution is to combine the fully homomorphic encryption (FHE) technique to realize the efficient operation of ciphertext without decryption. However, most current homomorphic encryption algorithms only support limited data types, making it difficult to be widely applied in actual environment. To address this limitation, we propose a parallel fully homomorphic encryption algorithm that supports floating-point numbers. The proposed algorithm not only expands the data types supported by the existing fully homomorphic encryption algorithms, but also utilizes the characteristics of multi-nodes in cloud environment to conduct parallel encryption through simultaneous group-wise ciphertext computations. The experimental results show that, in a 16-core 4-node cluster with MapReduce environment, the proposed encryption algorithm achieves the maximum speed-up exceeding 5, which not only solves the limited application problem of the existing fully homomorphic encryption algorithm, but also meets the requirements for the efficient homomorphic encryption of floating-point numbers in cloud computing environment.

An enhanced hadoop heartbeat mechanism for MapReduce task scheduler using dynamic calibration

MapReduce is a popular programming model for processing large-scale datasets in a distributed environment and is a fundamental component of current cloud computing and big data applications. In this paper, a heartbeat mechanism for MapReduce Task Scheduler using Dynamic Calibration (HMTS-DC) is proposed to address the unbalanced node computation capacity problem in a heterogeneous MapReduce environment. HMTS-DC uses two mechanisms to dynamically adapt and balance tasks assigned to each compute node: 1) using heartbeat to dynamically estimate the capacity of the compute nodes, and 2) using data locality of replicated data blocks to reduce data transfer between nodes. With the first mechanism, based on the heartbeats received during the early state of the job, the task scheduler can dynamically estimate the computational capacity of each node. Using the second mechanism, unprocessed Tasks local to each compute node are reassigned and reserved to allow nodes with greater capacities to reserve more local tasks than their weaker counterparts. Experimental results show that HMTS-DC performs better than Hadoop and Dynamic Data Placement Strategy (DDP) in a dynamic environment. Furthermore, an enhanced HMTS-DC (EHMTS-DC) is proposed by incorporating historical data. In contrast to the slow start property of HMTS-DC, EHMTS-DC relies on the historical computation capacity of the slave machines. The experimental results show that EHMTS-DC outperforms HMTS-DC in a dynamic environment.

Load balancing in join algorithms for skewed data in MapReduce systems

Join is an essential tool for data analysis which collected from different data sources. MapReduce has emerged as a prominent programming model for processing of massive data. However, traditional join algorithms based on MapReduce are not efficient when handling skewed data. The presence of data skew in input data leads to considerable load imbalance and performance degradation. This paper proposes a new skew-insensitive method, called fine-grained partitioning for skew data (FGSD) which can improve the load balancing for reduce tasks. The proposed method considers the properties of both input and output data through a proposed stream sampling algorithm. FGSD introduces a new approach for distribution of input data which leads to efficiently handling redistribution and join product skew. The experimental results confirm that our solution can not only achieve higher balancing performance, but also reduce the execution time of a job with varying degrees of the data skew. Furthermore, FGSD does not require any modification to the MapReduce environment and is applicable to complex join.

Big Data Analytics based on PANFIS MapReduce

In this paper, a big data analytic framework is introduced for processing high-frequency data stream. This framework architecture is developed by combining an advanced evolving learning algorithm namely Parsimonious Network Fuzzy Inference System (PANFIS) with MapReduce parallel computation, where PANFIS has the capability of processing data stream in large volume. Big datasets are learnt chunk by chunk by processors in MapReduce environment and the results are fused by rule merging method, that reduces the complexity of the rules. The performance measurement has been conducted, and the results are showing that the MapReduce framework along with PANFIS evolving system helps to reduce the processing time around 22 percent in average in comparison with the PANFIS algorithm without reducing performance in accuracy.

Comparison of Hive's query optimisation techniques

The ever increasing size of data sets in this big data era has forced data analytics to be moved from traditional RDBMS systems to distributed technologies like Hadoop. Since data analysts are more familiar with SQL than the MapReduce programming paradigm, HiveQL was built on Hadoop's MapReduce framework. Traditional RDBMS query optimisation techniques used in the rule-based optimiser (RBO) of Hive do not perform well in the MapReduce environment, hence, the correlation optimiser (CRO) and cost-based optimisers (CBOs) were developed. These optimisers perform query optimisations taking the MapReduce execution framework into account. In this work, the three optimisers, RBO, CRO, and CBO are compared. Queries with common intra-query operations are found to be better optimised with CRO.

Improved classification of large imbalanced data sets using rationalized technique: Updated Class Purity Maximization Over_Sampling Technique (UCPMOT)

The huge variety of NoSQL Big Data has tossed a need for new pathways to store, process and analyze it. The quantum of data created is inconceivable along with a mixed breath of unknown veracity and creative visualization. The new trials of frameworks help to find substantial unidentified values from massive data sets. They have added an exceptional dimension to the pre-processing and contextual conversion of the data sets for needful analysis. In addition, handling of ambitious imbalanced data sets has acknowledged an intimation of alarm. Traditional classifiers are unable to discourse the precise need of grouping for such data sets. Over_sampling of the minority classes help to improve the performance. Updated Class Purity Maximization Over_Sampling Technique (UCPMOT) is a rationalized technique proposed to handle imbalanced data sets using exclusive safe-level based synthetic sample creation. It addresses the multi-class problem in alignment to a newly induced method namely lowest versus highest. The projected technique experiments with several data sets from the UCI repository. The underlying bed of mapreduce environment encompasses the distributed processing approach on Apache Hadoop framework. Several classifiers help to authorize the classification results using parameters like F-measure and AUC values. The experimental conclusions quote the dominance of UCPMOT over the benchmarking techniques.

Efficient data distribution and results merging for parallel data clustering in mapreduce environment

Clustering data consists in partitioning it into clusters such that there is a strong similarity between data in the same cluster and a weak similarity between data in different clusters. With the significant increase in data volume, the clustering process becomes an expensive task in terms of computation. Therefore, several solutions have been proposed to overcome this issue using parallelism with the MapReduce paradigm. The proposed solutions in the literature aim to optimize the execution time while keeping the clustering quality close or identical to the sequential execution. One of the commonly used parallel clustering strategies when using the MapReduce framework consists in partitioning data and processing each partition separately. The results obtained from each partition are merged to obtain the final clusters configuration. Using a random data distribution strategy and an inappropriate merging technique will lead to an inaccurate final centroids and a rather average clustering quality. Hence, in this paper we propose a parallel scheme for partitional clustering algorithms based on MapReduce with a non-conventional data distribution and results merging strategies to improve the clustering quality. With this solution, in addition to optimizing the execution time, we exploit the parallel environment to enhance the clustering quality. The experimental results demonstrate the effectiveness and scalability of our solution in comparison with other recently proposed works. We also proposed an application of our approach to the community detection problem. The results demonstrate the ability of our approach to provide effective and relevant results.