Hadoop Tools Research Articles

Understand the working of Sqoop and hive in Hadoop

In past decades, the structured and consistent data analysis has seen huge success. It is a challenging task to analyse the multimedia data which is in unstructured format. Here the big data defines the huge volume of data that can be processed in distributed format. The big data can be analysed by using the hadoop tool which contains the Hadoop Distributed File System (HDFS) storage space and inbuilt several components are there. Hadoop manages the distributed data which is placed in the form of cluster analysis of data itself. In this, it shows the working of Sqoop and Hive in hadoop. Sqoop (SQL-to-Hadoop) is one of the Hadoop component that is designed to efficiently imports the huge data from traditional database to HDFS and vice versa. Hive is an open source software for managing large data files that is stored in HDFS. To show the working, here we are taking the application Instagram which is a most popular social media. In this analyze the data that is generated from Instagram that can be mined and utilized by using Sqoop and Hive. By this, prove that sqoop and hive can give results efficiently. This paper gives the details of sqoop and hive working in hadoop.

Identification of Multiple Attacks in Cloud Environment using Big Data

Since the Cloud environment is prone to many attacks we are implementing a Big Data based centralized log analysis system to identify the network traffic occurred by attackers through DDOS, SQL Injection, and Brute Force attacks. The log file is automatically transmitted to the centralized cloud server where big data is initiated and uses a tool called Hadoop to process the huge amount of log files that are being sent to big data. If an attacker attacks any files then it will be compared with the attack dataset that is maintained here to detect the attacks that are being used by the attacker. This system also helps in storing the information of all registered users, and their files which get uploaded to the cloud server and downloaded from the server and IP addresses in order to view any attacks that may occur in the future. All this stored information is maintained securely using SQL which is a backend process. Thus we are implementing a system that delivers a very high performance as well as very efficient results in categorizing the attacks. Since the Hadoop tool is being used in the system, this system is able to increase its scalability and achieves a faster detection of attacks like DDOS, SQL Injection, and Brute Force attacks. This system can play a vital role in various organizations to safeguard their privacy. Data integrity, Data confidentiality, and non-repudiation can be achieved by using this system. Finally, we can say that this system avoids any faults that occurred in the previous system that was invented before this.

New and Existing Approaches Reviewing of Big Data Analysis with Hadoop Tools

Everybody is connected with social media like (Facebook, Twitter, LinkedIn, Instagram…etc.) that generate a large quantity of data and which traditional applications are inadequate to process. Social media are regarded as an important platform for sharing information, opinion, and knowledge of many subscribers. These basic media attribute Big data also to many issues, such as data collection, storage, moving, updating, reviewing, posting, scanning, visualization, Data protection, etc. To deal with all these problems, this is a need for an adequate system that not just prepares the details, but also provides meaningful analysis to take advantage of the difficult situations, relevant to business, proper decision, Health, social media, science, telecommunications, the environment, etc. Authors notice through reading of previous studies that there are different analyzes through HADOOP and its various tools such as the sentiment in real-time and others. However, dealing with this Big data is a challenging task. Therefore, such type of analysis is more efficiently possible only through the Hadoop Ecosystem. The purpose of this paper is to analyze literature related analysis of big data of social media using the Hadoop framework for knowing almost analysis tools existing in the world under the Hadoop umbrella and its orientations in addition to difficulties and modern methods of them to overcome challenges of big data in offline and real –time processing. Real-time Analytics accelerates decision-making along with providing access to business metrics and reporting. Comparison between Hadoop and spark has been also illustrated.

Designing parallel computing using raspberry pi clusters for IoT servers on apache Hadoop

The community has widely used the IoT tool because it makes it easy for users in various sectors. The implementation of IoT does not only require connectivity and devices but also requires a server. Servers that are accessed by many devices will have more and more computing processes on the server. The computing process on the server requires a high-performance but small-sized server device that is easy to connect with other devices. Raspberry Pi, known as an IoT server, is a computer with a small size, cheap, and very energy efficient. The parallel machine was built based on Raspberry Pi using parallel computing technology to have high performance. Nowadays, one of the parallel computing tools is Apache Hadoop. This research using clusters method with four pieces of Raspberry Pi with parallel computing technology using Apache Hadoop tools. Testing is done using Siege tools. The number of requests that used to test web server performance in this research was 10, 50, 100, and 150. The most optimal results are on requests by 50 with an increase of about 260% from a single node to cluster in the measurement of transactions, transaction rates, and throughput. Testing with 150 requests can only increase by 50% to 60%.

Hadoop Framework for Big Data Analytics to Analyze Attacks on Business Data

Big Data consist large volumes of data sets with various formats i.e., structured, unstructured and semi structured. Big Data requires security because day by day attackers attack on it in different manner. Big Data Security Analytics analyses Big Data for finding various threats and complex attacks. By increasing the number of targeting attacks on data and one side rapid growing of data, it is too difficult to analyze accurately. The Security Analytics Systems are used the untrusted data. So, strong security analytical tools are required to analyze the data. The organizations and industries exchange the data through networks dynamically, so this may become more vulnerable for data misusing and theft. Attackers are more advanced in the attacking on data that the existing security mechanisms are not identified before damaging. At present, the collecting and analyzing various attacks is major challenging task for Security Analytics Systems, to take suitable decision. In this research paper, we have addressed about Hadoop tool that how it analyses Big Data and how Big Data Security Analytics is applied to analyze the various threats and securing the business data more accurately.

Similarity Based Prediction System using Machine Learning Algorithms in Big Data Analytics

Big Data is a noteworthy environment to maintain the diversity of the huge amount of data. The big data utilizes machine learning algorithms to process large datasets which comes from various places such as histories, weblogs, and data repositories, large datasets and data warehousing, etc. In an existing method, most of the data mining approaches might not be able to maintain the large dataset. Using datamining, the big data are having lack of compatibility with database systems and analysis tools; large dataset clustering and analyzing is a big issue in big data. For this reason, the research work uses machine learning algorithms which are implemented in the Hadoop tool to collect and process the large amount of data which is structured, semi-structured or unstructured in a reasonable amount of time. Also, it gives more accurate prediction system and accurate information. Using Machine Learning Algorithm computational cost and complexities is minimized. The overall research work is implemented in the Hadoop tool with the help of the python programming language and it is compared with some existing algorithms. The proposed work tested with suitable parameters such as accuracy, Kappa T and Kappa M.

Big Data Hadoop Tools and Technologies: A Review

The word ‘big data’ represents a huge volume of data which can be in a structured, semi-structured, unstructured form. Big Data is generated in a very large amount making it difficult to process using traditional techniques. In order to process these generated data traditional big data management system is inefficient to manage the large amount of data. Hadoop is a framework which is designed to process the large data sets and provides high performance, fault tolerance from a single server to thousands of machines. In this paper we describe a detailed review of big data and Hadoop along with a comparison of various tools/technologies of big data.

Identifying Requirements for Big Data Analytics and Mapping to Hadoop Tools

Big data is being generating in a wide variety of formats at an exponential rate. Big data analytics deals with processing and analyzing voluminous data to provide useful insight for guided decision making. The traditional data storage and management tools are not well-equipped to handle big data and its application. Apache Hadoop is a popular open-source platform that supports storage and processing of extremely large datasets. For the purposes of big data analytics, Hadoop ecosystem provides a variety of tools. However, there is a need to select a tool that is best suited for a specific requirement of big data analytics. The tools have their own advantages and drawbacks over each other. Some of them have overlapping business use cases however they differ in critical functional areas. So, there is a need to consider the trade-offs between usability and suitability while selecting a tool from Hadoop ecosystem. This paper identifies the requirements for Big Data Analytics (BDA) and maps tools of the Hadoop framework that are best suited for them. For this, we have categorized Hadoop tools according to their functionality and usage. Different Hadoop tools are discussed from the users’ perspective along with their pros and cons, if any. Also, for each identified category, comparison of Hadoop tools based on important parameters is presented. The tools have been thoroughly studied and analyzed based on their suitability for the different requirements of big data analytics. A mapping of big data analytics requirements to the Hadoop tools has been established for use by the data analysts and predictive modelers.

Optimal Algorithms for Preprocessing of Server Side Web Logs Using Parallel Computing and HBASE

Now a day’s use of internet has been increased tremendously. Every day internet users generate 2.5 quintillion bytes of data from various sources, and thus leads to Big data analytics. Web usage mining is the type of web mining activity that involves discovering user access pattern from web log data. Web usage mining has three phases such as Data Preprocessing, Data Discovery and Data Evaluation. In this paper we have mainly focused on Data preprocessing. Data preprocessing is an important phase of Web usage mining required to unstructured, heterogeneous and unwanted (noisy) nature of log data. In general, two types of logs ie., server-side logs and client side logs are used for web usability analysis. Preprocessing consists of four phases, Data Extraction, Data Cleaning, User identification, Session Identification and Path completion. This paper presents a specific data preprocessing case using hadoop tool for Vizhamurasu News site. In this work, server-side logs are considered to experiment the proposed preprocessing algorithms. The existing preprocessing algorithms are efficient but that are not scalable because when we increasing size of log file and also take much more computation time compared to proposed parallel computing techniques.

The Study on the Analysis System of Effective Attack Patterns in Kendo Match using Hadoop-based Hive

Objectives: To analyze the effective attack pattern of Kendo for the use of effective training through Hive. Methods/ Statistical Analysis: The qualified three referees of Kendo competition analyze a video image of a real contest to extract the attack pattern, and store the extracted data in a database by pattern and in HDFS through the classification process. After purifying the stored data through the Mapping and Reducing, the results are extracted by using HiveQL, one of the sub-project ecosystems of Hadoop. Findings: In this paper, by using Hadoop and analysis tool Hive, a big data analysis system for performance enhancement of Kendo athlete is proposed. Experiment subjects were implemented for one game, but after collecting and purifying the recorded historical data over several years, various types of information will be collected. For example, many types of information can be collected such as changes in the age-specific, player-specific pattern, the contextual pattern, and so on by using such information effectively. Then, it may provide a pattern of the Kendo training. Improvements/Applications: In the future research, the methods of collecting data by image analysis techniques and the coaching environment for research and Kendo training should be studied continuously.

CREDIT CARD DATA PROCESSING AND E-STATEMENT GENERATION WITH USE OF HADOOP

The Bigdata for most of part is known as virtually wealthy storage and processing .This paper is related to basic knowledge of Bigdata and Hadoop environment. Now a day's various organizations deal with problems like data sharing ,data privacy and secure data transfer where they take lots of time for all this work , so to avoid this we are using Hadoop. Hadoop is used for large storage and parallel processing , we can say it also works in distributed fashion. There are different hadoop tools available like hive, sqoop, pig and more which we will discuss later.

Processing Solutions for Big Data in Astronomy

This paper gives a simple introduction to processing solutions applied to massive amounts of data. It proposes a general presentation of the Big Data paradigm. The Hadoop framework, which is considered as the pioneering processing solution for Big Data, is described together with YARN, the integrated Hadoop tool for resource allocation. This paper also presents the main tools for the management of both the storage (NoSQL solutions) and computing capacities (MapReduce parallel processing schema) of a cluster of machines. Finally, more recent processing solutions like Spark are discussed. Big Data frameworks are now able to run complex applications while keeping the programming simple and greatly improving the computing speed.

A Hyperconnected Manufacturing Collaboration System Using the Semantic Web and Hadoop Ecosystem System

With the explosive growth of digital data communications in synergistic operating networks and cloud computing service, hyperconnected manufacturing collaboration systems face the challenges of extracting, processing, and analyzing data from multiple distributed web sources. Although semantic web technologies provide the solution to web data interoperability by storing the semantic web standard in relational databases for processing and analyzing of web-accessible heterogeneous digital data, web data storage and retrieval via the predefined schema of relational / SQL databases has become increasingly inefficient with the advent of big data. In response to this problem, the Hadoop Ecosystem System is being adopted to reduce the complexity of moving data to and from the big data cloud platform. This paper proposes a novel approach in a set of the Hadoop tools for information integration and interoperability across hyperconnected manufacturing collaboration systems. In the Hadoop approach, data is “Extracted” from the web sources, “Loaded” into a set of the NoSQL Hadoop Database (HBase) tables, and then “Transformed” and integrated into the desired format model with Hive's schema-on-read. A case study was conducted to illustrate that the Hadoop Extract-Load-Transform (ELT) approach for the syntax and semantics web data integration could be adopted across the global smartphone value chain.

An Imminent Approach for Genome Sequence and Analysis using Map Reduce

The recent trend of BigData in Healthcare is overpowering and necessity increasing rapidly because of its data type diversity in addition to its volume, managing speed and leads to improving care even at the lowest cost. Cancer prevails as a challenging issue because of its different mutations. Identification of the each tumor‟s root for mutations and mapping of their evolution of genetics that leads to growth in the conflict against the cancer disease, “GenomeAnalysis “plays an important role. In order to accumulate and categorize the enormous revenue of information from genome analysis, research field coalesced with a data Platform ApacheHadoop supporting parallelization, composability for extremely huge upsurge in activity of sequencing data. By aggregating all aids of BigData Analytics Tools and EHR, this proposal presents a study about how to incorporate the Hadoop Tool integrated with GATK(Genome Analysis Tool Kit) through MapReduce to map cancer genomic data problems with the conscious of financially low cost and high speed of accessing data.

English

More Social Networks used as a platform for millions of users to share their thoughts and opinions about different aspects so dataset become larger. For example Twitter is a rich source of information for decision making using Sentiment analysis. Sentiment analysis to provide better decision making given to particular person, product or any service. Sentiment analysis and Hadoop tool has been used to process the dataset. Sentiment analysis over the Twitter offers organizations a fast and effective way to monitor the publics’ feelings towards their brands. It was focuses to predict the polarity of words and then classify them into positive and negative feelings with the aim of identifying attitude and opinions that are expressed in any form or language. The proposed method includes various pre-processing steps before feeding the text to the classifier and Map reduce algorithm used for getting accurate decision about product from different perception of feedback from users. Experimental results show that the proposed technique overcomes the previous limitations and achieves higher accuracy when compared to similar technique.