Hadoop MapReduce Research Articles

STA/LTA trigger algorithm implementation on a seismological dataset using Hadoop MapReduce

<span>In this work we implemented STA/LTA trigger algorithm, which is widely used in seismic detection, using Hadoop MapReduce. This<br />implementation allows to find out how effective it is in this type of tasks as well as to accelerate the detection process by reducing the processing time. We tested our implementation on a seismological dataset of 14 broadband seismic stations and compare it with the traditional one. The results show that MapReduce decreased the processing time by 34% compared to the traditional implementation.</span>

A Hybrid Multilingual Fuzzy-Based Approach to the Sentiment Analysis Problem Using SentiWordNet

Sentiment Analysis or in particular social network analysis (SNA) is a new research area which is increased explosively. This domain has become a very active research issue in data mining and natural language processing. Sentiment analysis (opinion mining) consists in analyzing and extracting emotions, opinions or attitudes from product’s reviews, movie's reviews, etc., and classify them into classes such as positive, negative and neutral, or extract the degree of importance (polarity). In this paper, we propose a new hybrid approach for classifying tweets into classes based on fuzzy logic and a lexicon based approach using SentiWordnet. Our approach consists in classifying tweets according to three classes: positive, negative or neutral, using SentiWordNet and the fuzzy logic with its three important steps: Fuzzification, Rule Inference/aggregation, and Defuzzification. The dataset of tweets to classify and the result of the classification are stored in the Hadoop Distributed File System (HDFS), and we use the Hadoop MapReduce for the application of our proposal.

Improved Hadoop Cluster Performance by Dynamic Load and Resource Aware Speculative Execution and Straggler Node Detection

The big data is one of the fastest growing technologies, which can to handle huge amounts of data from various sources, such as social media, web logs, banking and business sectors etc. In order to pace with the changes in the data patterns and to accommodate the requirements of big data analytics, the platform for storage and processing such as Hadoop, also requires great advancements. Hadoop, an open source project executes the big data processing job in map and reduce phases and follows master-slave architecture. A Hadoop MapReduce job can be delayed if one of its many tasks is being assigned to an unreliable or congested machine. To solve this straggler problem, a novel algorithm design of speculative execution schemes for parallel processing clusters, from an optimization perspective, under different loading conditions is proposed. For the lightly loaded case, a task cloning scheme, namely, the combined file task cloning algorithm, which is based on maximizing the overall system utility, a straggler detection algorithm is proposed based on a workload threshold. The detection and cloning of tasks assigned with the stragglers only will not be enough to enhance the performance unless cloning of tasks is allocated in a resource aware method. So, a method is proposed which identifies and optimizes the resource allocation by considering all possible aspects of cluster performance balancing. One main issue arises due to the pre configuration of distinct map and reduce slots based on the number of files in the input folder. This can cause severe under-utilization of slot as map slots might not be fully utilized with respect to the input splits. To solve this issue, an alternative technique of Hadoop Slot Allocation is introduced in this paper by keeping the efficient management of slots model. The combine file task cloning algorithm combines the files which are less than the size of a single data block and executes them in the highly performing data node. On implementing these efficient cloning and combining techniques on a heavily loaded cluster after detecting the straggler, machine is found to reduce the elapsed time of execution to an average of 40%. The detection algorithm improves the overall performance of the heavily loaded cluster by 20% of the total elapsed time in comparison with the native Hadoop algorithm.

GPU-based efficient join algorithms on Hadoop

The growing data has brought tremendous pressure for query processing and storage, so there are many studies that focus on using GPU to accelerate join operation, which is one of the most important operations in modern database systems. However, existing GPU acceleration join operation researches are not very suitable for the join operation on big data. Based on this, this paper speeds up nested loop join, hash join and theta join, combining Hadoop with GPU, which is also the first to use GPU to accelerate theta join. At the same time, after the data pre-filtering and pre-processing, using Map-Reduce and HDFS in Hadoop proposed in this paper, the larger data table can be handled, compared to existing GPU acceleration methods. Also with Map-Reduce in Hadoop, the algorithm proposed in this paper can estimate the number of results more accurately and allocate the appropriate storage space without unnecessary costs, making it more efficient. The rigorous experiments show that the proposed method can obtain 1.5 to 2 times the speedup, compared to the traditional GPU acceleration equi join algorithm. And in the synthetic data set, the GPU version of the proposed method can get 1.3 to 2 times the speedup, compared to CPU version.

MapReduce Implementation of a Multinomial and Mixed Naive Bayes Classifier

This study presents an efficient way to deal with discrete as well as continuous values in Big Data in a parallel Naïve Bayes implementation on Hadoop's MapReduce environment. Two approaches were taken: (i) discretizing continuous values using a binning method; and (ii) using a multinomial distribution for probability estimation of discrete values and a Gaussian distribution for probability estimation of continuous values. The models were analyzed and compared for performance with respect to run time and classification accuracy for varying data sizes, data block sizes, and map memory sizes.

Enhancing Hadoop Performance in Homogenous and Heterogeneous Big Data Environments by Dynamic Slot Configuration

Hadoop is one of the most famous platform solutions for processing large volume and scale of data in parallel processing in Cloud computing. A Hadoop system can be characterized based on three main factors: cluster, workload and user. Each of these factors can be described as either heterogeneous or homogenous, which reflects the heterogeneity degree of the Hadoop systemThe objective of this proposed research work is to investigate the degree of influence of heterogeneity for each of these factors on the performance of Hadoop based on different schedulers. Three schedulers are considered with different levels of Hadoop heterogeneity and are tested and analyzed: the first algorithm considered is the FIFO (First in First out), the second is the Fair sharing, and the final is the COSHH (Classification and Optimization based Scheduler for Heterogeneous Hadoop). Performance issues are related to Hadoop schedulers and comparative performance analysis between different cases of jobs submission. These jobs are processed in different homogenous or heterogeneous data environments and under fixed or reconfigurable slot between map and reduce tasks for Hadoop MapReduce java programming clustering model. The results showed that when assigning tunable knob between map and reduce tasks under certain schedulers like FIFO algorithm, the performance enhanced significantly especially in cases of heterogeneity environment where the workload decreased significantly and the utilization of computational resources increase was obvious.

A Bigdata processing with Hadoop Map Reduce in Cloud Systems

A Bigdata processing with Hadoop Map Reduce in Cloud Systems

Cloud Computing-Based Socially Important Locations Discovery on Social Media Big Datasets

Socially important locations are places which are frequently visited by social media users in their social media lifetime. Discovering socially important locations provides valuable information, such as which locations are frequently visited by a social media user, which locations are common for a social media user group, and which locations are socially important for a group of urban area residents. However, discovering socially important locations is challenging due to huge volume, velocity, and variety of social media datasets, inefficiency of current interest measures and algorithms on social media big datasets, and the need of massive spatial and temporal calculations for spatial social media analyses. In contrast, cloud computing provides infrastructure and platforms to scale compute-intensive jobs. In the literature, limited number of studies related to socially important locations discovery takes into account cloud computing systems to scale increasing dataset size and to handle massive calculations. This study proposes a cloud-based socially important locations discovery algorithm of Cloud SS-ILM to handle volume and variety of social media big datasets. In particular, in this study, we used Apache Hadoop framework and Hadoop MapReduce programming model to scale dataset size and handle massive spatial and temporal calculations. The performance evaluation of the proposed algorithm is conducted on a cloud computing environment using Turkey Twitter social media big dataset. The experimental results show that using cloud computing systems for socially important locations discovery provide much faster discovery of results than classical algorithms. Moreover, the results show that it is necessary to use cloud computing systems for analyzing social media big datasets that could not be handled with traditional stand-alone computer systems. The proposed Cloud SS-ILM algorithm could be applied on many application areas, such as targeted advertisement of businesses, social media utilization of cities for city planners and local governments, and handling emergency situations.

Reinforced K-means Hadoop Mapreduce Methodology for Rainfall Prediction

Reinforced K-means Hadoop Mapreduce Methodology for Rainfall Prediction

Optimizing parameters in algorithm trading using map reduce on Indian stock exchange (Sensex)

In the stock market , stock price movements depend on more number of parameters here we are focusing on how to optimize the multiple parameters based on the running result in algorithm trading on historical data. It is a time consuming task in such a large search space for parameters as well as the huge volume of historical data. In this paper a new technique is realized how to optimize multiple parameters using Hadoop MapReduce, this technique utilizes the parallel processing capability. Here we provide complete procedure on how the method is realized and the configurations needed to be considered in the method.

A Fast and Scalable Workflow for SNPs Detection in Genome Sequences Using Hadoop Map-Reduce.

Next generation sequencing (NGS) technologies produce a huge amount of biological data, which poses various issues such as requirements of high processing time and large memory. This research focuses on the detection of single nucleotide polymorphism (SNP) in genome sequences. Currently, SNPs detection algorithms face several issues, e.g., computational overhead cost, accuracy, and memory requirements. In this research, we propose a fast and scalable workflow that integrates Bowtie aligner with Hadoop based Heap SNP caller to improve the SNPs detection in genome sequences. The proposed workflow is validated through benchmark datasets obtained from publicly available web-portals, e.g., NCBI and DDBJ DRA. Extensive experiments have been performed and the results obtained are compared with Bowtie and BWA aligner in the alignment phase, while compared with GATK, FaSD, SparkGA, Halvade, and Heap in SNP calling phase. Experimental results analysis shows that the proposed workflow outperforms existing frameworks e.g., GATK, FaSD, Heap integrated with BWA and Bowtie aligners, SparkGA, and Halvade. The proposed framework achieved 22.46% more efficient F-score and 99.80% consistent accuracy on average. More, comparatively 0.21% mean higher accuracy is achieved. Moreover, SNP mining has also been performed to identify specific regions in genome sequences. All the frameworks are implemented with the default configuration of memory management. The observations show that all workflows have approximately same memory requirement. In the future, it is intended to graphically show the mined SNPs for user-friendly interaction, analyze and optimize the memory requirements as well.

Massive Power Information Processing Scheme Based on MongoDB

In order to solve the problem that the power consumption data presents a trend of sea quantification, and the huge power consumption data brings tremendous pressure to the system, especially slows down the speed of data query and gives users poor experience in using it, this paper proposes to use one of the non-relational databases (NoSQL) to store power consumption data instead of the traditional relational databases. To solve the problem that the collected electricity data can not meet the demand of page display, Hadoop MapReduce is proposed to preprocess the electricity data. Distributed architecture is adopted for storage and processing. MongoDB cluster and Hadoop cluster are overlapped and deployed. Combined with MongoDB’s powerful storage capacity and Hadoop MapReduce’s analytical and computational capability, a set of high-availability and high-performance data storage and pre-processing scheme for power-consuming universities is constructed.

Network Function Virtualization Resource Allocation Based on Joint Benders Decomposition and ADMM

Network function virtualization (NFV) has emerged as a new technology to reduce the cost of hardware deployment. It is an architecture that using virtualized functions run on the virtual machine to achieve services instead of using specific hardware. Although NFV brings more opportunities to enhance the flexibility and efficiency of the network, resource allocation problems should be well taken into consideration. In this paper, we investigate the virtual network function (VNF) resource allocation problem to minimize the network operation cost for different services. Both setting the VNF instances for each virtual machine and allocating the traffic volume in the network are considered. The problem is formulated as a mixed integer programming problem. Although it can be solved in a centralized fashion which requires a central controller to collect information from all virtual machines, it is not practical for large-scale networks. Thus, we propose a distributed iteration algorithm to achieve the optimal solution. The proposed algorithm framework is developed based on the joint Benders decomposition and alternating direction method of multipliers (ADMM), which allows us to deal with integer variables and decompose the original problem into multiple subproblems for each virtual machine. Furthermore, we describe the detail implementation of our algorithm to run on a computer cluster using the Hadoop MapReduce software framework. Finally, the simulation results indicate the effectiveness of the algorithm.

Similarity Measurement Technique for Measuring the Performance of Page Rank Algorithm Based On Hadoop

In this century big data manipulation is a challenging task in the field of web mining because content of web data is massively increasing day by day. Using search engine retrieving efficient, relevant and meaningful information from massive amount of Web Data is quite impossible. Different search engine uses different ranking algorithm to retrieve relevant information easily. A new page ranking algorithm is presented based on synonymous word count using Hadoop MapReduce framework named as Similarity Measurement Technique (SMT). Hadoop MapReduce framework is used to partition Big Data and provides a scalable, economical and easier way to process these data. It stores intermediate result for running iterative jobs in the local disk. In this algorithm, SMT takes a query from user and parse it using Hadoop and calculate rank of web pages. For experimental purpose wiki data file have been used and applied page rank algorithm (PR), improvised page rank algorithm (IPR) and proposed SMT method to calculate page rank of all web pages and compare among these methods. Proposed method provides better scoring accuracy than other approaches and reduces theme drift problem.

Fine-grained data-locality aware MapReduce job scheduler in a virtualized environment

Big data overwhelmed industries and research sectors. Reliable decision making is always a challenging task, which requires cost-effective big data processing tools. Hadoop MapReduce is being used to store and process huge volume of data in a distributed environment. However, due to huge capital investment and lack of expertise to set up an on-premise Hadoop cluster, big data users seek cloud-based MapReduce service over the Internet. Mostly, MapReduce on a cluster of virtual machines is offered as a service for a pay-per-use basis. Virtual machines in MapReduce virtual cluster reside in different physical machines and co-locate with other non-MapReduce VMs. This causes to share IO resources such as disk and network bandwidth, leading to congestion as most of the MapReduce jobs are disk and network intensive. Especially, the shuffle phase in MapReduce execution sequence consumes huge network bandwidth in a multi-tenant environment. This results in increased job latency and bandwidth consumption cost. Therefore, it is essential to minimize the amount of intermediate data in the shuffle phase rather than supplying more network bandwidth that results in increased service cost. Considering this objective, we extended multi-level per node combiner for a batch of MapReduce jobs to improve makespan. We observed that makespan is improved up to 32.4 $$\%$$ by minimizing the number of intermediate data in shuffle phase when compared to classical schedulers with default combiners.

A decentralised framework for efficient storage and processing of big data using HDFS and IPFS

Big data revolution emerged with greater opportunities as well as challenges. Some of the major challenges include capturing, storing, transferring, analysing, processing and updating these large and complex datasets. Traditional data handling techniques cannot manage this fast growing data. Apache Hadoop is one of the best technologies which can address the challenges involved in big data handling. Hadoop is a centralised, distributed data storage model. InterPlanetary file system (IPFS) is an emerging technology which can provide a decentralised distributed storage. By integrating both these technologies, we can create a better framework for the distributed storage and processing of big data. In the proposed work, we formulated a model for big data placement, replication and processing by combining the features of Hadoop and IPFS. Hadoop distributed file system and IPFS jointly handle the data placement and replication tasks and the programming framework MapReduce in Hadoop handle the data processing task. The experimental result shows that the proposed framework can achieve cost-effective storage as well as faster processing of big data.

A decentralised framework for efficient storage and processing of big data using HDFS and IPFS

Big data revolution emerged with greater opportunities as well as challenges. Some of the major challenges include capturing, storing, transferring, analysing, processing and updating these large and complex datasets. Traditional data handling techniques cannot manage this fast growing data. Apache Hadoop is one of the best technologies which can address the challenges involved in big data handling. Hadoop is a centralised, distributed data storage model. InterPlanetary file system (IPFS) is an emerging technology which can provide a decentralised distributed storage. By integrating both these technologies, we can create a better framework for the distributed storage and processing of big data. In the proposed work, we formulated a model for big data placement, replication and processing by combining the features of Hadoop and IPFS. Hadoop distributed file system and IPFS jointly handle the data placement and replication tasks and the programming framework MapReduce in Hadoop handle the data processing task. The experimental result shows that the proposed framework can achieve cost-effective storage as well as faster processing of big data.

Encroachment in Data Processing using Big Data Technology

the nature of big data is now growing and information is present all around us in different kind of forms. The big data information plays crucial role and it provides business value for the firms and its benefits sectors by accumulating knowledge. This growth of big data around all the concerns is high and challenge in data processing technique because it contains variety of data in enormous volume. The tools which are built on the data mining algorithm provides efficient data processing mechanisms, but not fulfill the pattern of heterogeneous, so the emerging tools such like Hadoop MapReduce, Pig, SPARK, Cloudera, Impala and Enterprise RTQ, IBM Netezza and Apache Giraphe as computing tools and HBase, Hive, Neo4j and Apache Cassendra as storage tools useful in classifying, clustering and discovering the knowledge. This study will focused on the comparative study of different data processing tools, in big data analytics and their benefits will be tabulated.

Research of efficiency of technologies for data processing in tasks of big data analysis

The paper compares the performance of technologies for big data processing applied for anomaly detection in a database of a billing system. Experiments have been conducted to evaluate the performance for processing big data of two technologies such as Hadoop MapReduce and Apache Spark. The datasets and configuration of the applications used in the experiments are described. The experiment results are presented as a description of the relation between calculation time, data size and imbalance in datasets. Based on the obtained results, a decision making system for choosing a technology for processing data depending on the characteristics of the datasets is proposed.

Effective Feature Selection Using Hybrid GA-EHO for Classifying Big Data SIoT

Several novel applications and services of networking for the IoT are supported by the Social Internet of Things (SIoT) in a more productive and powerful way. SIoTs are the recent hot topics rather than other extensions of IoTs. In this research, the authors have extracted the Big Data SIoT using the well-known model named MapReduce framework. Moreover, the unwanted data and noise from the database are reduced using the Gabor filter, and the big databases are mapped and reduced using the Hadoop MapReduce (HMR) technique for improving the efficiency of the proposed GA-EHO. Furthermore, the feature selection using GA-EHO is processed on the filtered dataset. The implementation of the proposed system is done by using some machine learning classifiers for classifying the data and the efficiency is predicted for the proposed work. From the simulation results, the specificity, maximum accuracy, and sensitivity of the proposed GA-EHO are produced about 87.88%, 99.1%, and 81%. Also, the results are compared with other existing techniques.