Apache Pig Research Articles

Analysis of data processing efficiency with use of Apache Hive and Apache Pig in Hadoop environment

The aim of this paper is the analysis of data processing efficiency with use of Apache Hive and Apache Pig in Hadoop environment. The analysis was based on comparison between both mentioned tools with use of large data set, represented by 28 million records. Research was provided with use of scripts and queries destined for Apache Hive and Apache Pig, and then executed 10 times on environment brought by created virtual machine. Those methods were performed on the same data sets for 16 times according to previously prepared research scenarios. As the conclusion, authors had observed that Apache Hive is more efficient tool, than Apache Pig.

Apache Hadoop based effective sentiment analysis on demonetization and covid-19 tweets

In Recent, Twitter is the well-known public Network acquires a huge number of tweets. Sentiment analysis in twitter data are tremendously valuable in social media observing as it allows getting an overview of extensive global opinion in certain issue. This data are utilized for industrial, government, social and economic approaches by analyzing the tweets as per the requirement of the user. Processing and storing these data are more complicated to analyze. Hadoop is a distributed environment which process with Big and Huge variety of dataset which supports processing components that collectively called Hadoop Ecosystem. In this paper, regular tweets are analyzed by sentiment analysis technique in Hadoop Eco system. Dataset are taken from Kaggle data repository. This research has done by Apache Pig in Demonetization and Covid 19 twitter dataset.

Performance testing in lexical analysis on latest Twitter trends for enterprise network using PIG

Twitter usage is outgrowing each day and so is the ever changing trend. The data flow keeps on increasing each day through tweets and thus analysis on the latest trends are done by using most widely used framework for processing large datasets-Apache Hadoop. This paper presents an overview of Twitter Trend analysis using Apache PIG in Hadoop framework and its performance will be tested using lexical analysis. Hadoop is preferred over other frameworks due to its robustness, scalability, cost effectiveness and high speed which makes parallel processing over distributed network easy. The paper also shows results obtained through lexical analysis done on the Hadoop cluster to show the efficiency of the program.

Hadoop Based Generic Template for Performing Sentiment Analysis Using Apache PIG

Every day over 2.5 quintillion data is generated using various channels like online surveys, transactional data tracking, social media monitoring, etc. Out of these majority of the data is generated using social media platforms. This raw data contains information that can be used for industrial, economic, social and business purposes. To facilitate this, sentiment analysis has become a prospect for various tech-based industry giants to review and analyze their products. Hadoop has been established as one of the best tools for storing, processing, and streaming data in the market. In this paper, we present a generic approach to performing sentiment analysis using Apache PIG which classifies the given data taken from a dataset to either positive or negative to get the people’s sentiment over an object or an issue.

Analysis of Crime Rates of Different States in India Using Apache Pig in HDFS Environment

Background: In this astronomically immense world tremendous amount of data engendering in every minute from the different domain which is referred to Big Data. In the last few years the data is incrementing day by day across the world. This Research fixates on the analysis of malefaction rates of 5 different states year wise, all the analysis is done utilizing Apache Pig. Methods: The goal of the work is to analyze the astronomically immense malefaction data and find the estimate number of malefaction transpires in sundry states. This is done in Apache pig environment utilizing “Pig Latin” as language. A short code is indicted in Pig Latin which is utilized to load and process the data into Map reduce environment, afterwards the result are obtained with the detail of minimum and maximum mapper and reducer timing. Results: The data is visualized into graphs to make analysis to analyze the variation of malefaction rates in distinct states. After analyzing the malefaction against women, murder cases are very high in 2006-2010 as compared to other year groups whereas abducting and rape cases incremented perpetually from 2001 to 2014 respectively. Similarly all the reports regarding to different malefaction rates are visualized above by utilizing graphs. Conclusion: Various results are found with sundry queries and everything is represented graphically for better understanding and comparison. This avails us to find which state is affected by which crime. The expeditiousness of Apache pig can additionally be optically discerned as this immensely colossal crime data processed in short time with precision.

The research of social processes at the university using big data

The volume of information in the 21st century is growing at a rapid pace. Big data technologies are used to process modern information. This article discusses the use of big data technologies to implement monitoring of social processes. Big data has its characteristics and principles, which reflect here. In addition, we also discussed big data applications in some areas. Particular attention in this article pays to the interactions of big data and sociology. For this, there consider digital sociology and computational social sciences. One of the main objects of study in sociology is social processes. The article shows the types of social processes and their monitoring. As an example, there is implemented monitoring of social processes at the university. There are used following technologies for the realization of social processes monitoring: products 1010data (1010edge, 1010connect, 1010reveal, 1010equities), products of Apache Software Foundation (Apache Hive, Apache Chukwa, Apache Hadoop, Apache Pig), MapReduce framework, language R, library Pandas, NoSQL, etc. Despite this, this article examines the use of the MapReduce model for social processes monitoring at the university.

Bigdata Analysis on Airline Delay and Cancellation

In recent years, there has been a tremendous increase in the number of air passengers. These increasing figures have enhanced problems, ultimately causing flight delays. Delay in flights is recognized not only the usual factor but also a bothersome experience offered to customers. These delays lead to loss of money to the travelers. Additionally, it also affects the business value and status of the aviation industry. The delays can be caused due to many factors like weather conditions, technical problem, air traffic etc. These concerned parameter makes it important in carrying out analysis based on past trends of delays which can be further divided into daily, monthly and yearly intervals. This project makes use of big data technologies like Apache Pig, hive and Hadoop Distributed File System (HDFS). These technologies would be helpful in determining delay problems faced by the aviation industry where an appropriate decision can be implemented and furthermore would be benefiting customers and industrialists of the aviation domain.

Equi-Depth Histogram Construction Methodology for Big Data Tools

In recent decades, countless data sources such as social media, machines, and networks are constantly pushing data into the digital world. The size of the data has been growing exponentially. To understand the statistical information of data query optimization, equi-depth histograms are essential. In this paper, we present approximate equi-depth histogram construction for big data using both Apache Pig Scripts and Java Web Interface interacting with Apache Hadoop. We use equi-depth histogram construction with quality guarantees for big data approaches and implement them with Apache Hadoop Map-Reduce and Apache Pig user-defined functions. We introduce a prototype implementation of the construction of the approximate equi-depth histogram from the Java Server Face page using Apache Hadoop jobs and the Hadoop Distributed Files System, and we evaluate these methods using the demonstration. We explain Apache Pig Scripts techniques to create equi-depth histograms using big data. The results indicate that our system provides the capability of writing multiple jobs using Apache Pig, and programmers can make use of the advantages of Apache Pig to create histograms and eliminate the complex implementation of Map-Reduce jobs.

Twitter data analysis using hadoop ecosystems and apache zeppelin

The day-to-day life of the people doesn't depend only on what they think, but it is affected and influenced by what others think. The advertisements and campaigns of the favourite celebrities and mesmerizing personalities influence the way people think and see the world. People get the news and information at lightning speed than ever before. The growth of textual data on the internet is very fast. People express themselves in various ways on the web every minute. They make use of various platforms to share their views and opinions. A huge amount of data is being generated at every moment on this process. Being one of the most important and well-known social media of the present time, millions of tweets are posted on Twitter every day. These tweets are a source of very important information and it can be made use for business, small industries, creating government policies, and various studies can be performed by using it. This paper focuses on the location from where the tweets are posted and the language in which the tweets are written. These details can be effectively analysed by using Hadoop. Hadoop is a tool that is used to analyze distributed big data, streaming data, timestamp data and text data. With the help of Apache Flume, the tweets can be collected from Twitter and then sink in the HDFS (Hadoop Distributed File System). These raw data then analyzed using Apache Pig and the information available can be made use for social and commercial purposes. The result will be visualized using Apache Zeppelin.

Pipeline provenance for cloud‐based big data analytics

SummaryProvenance is information about the origin and creation of data. In data science and engineering related with cloud environment, such information is useful and sometimes even critical. In data analytics, it is necessary for making data‐driven decisions to trace back history and reproduce final or intermediate results, even to tune models and adjust parameters in a real‐time fashion. Particularly, in cloud, users need to evaluate data and pipeline trustworthiness. In this paper, we propose a solution: LogProv, toward realizing these functionalities for big data provenance, which needs to renovate data pipelines or some of big data software infrastructure to generate structured logs for pipeline events, and then stores data and logs separately in cloud space. The data are explicitly linked to the logs, which implicitly record pipeline semantics. Semantic information can be retrieved from the logs easily since they are well defined and structured beforehand. We implemented and deployed LogProv in Nectar Cloud,* associated with Apache Pig, Hadoop ecosystem, and adopted Elasticsearch to provide query service. LogProv was evaluated and empirically case studied. The results show that LogProv is efficient since the performance overhead is no more than 10%; the query can be responded within 1 second; the trustworthiness is marked clearly; and there is no impact on the data processing logic of original pipelines.

Simulation of Performance Analysis of MongoDB, PIG, HIVE Storage, Map Reduce, Spark and Yarn

Nowadays there are a variety of the size or volume, complexity, variety, rate of growth or veracity of information. The companies have achieved an outstanding stage in order to handle the data. The cause is that the traditional techniques and analytical devices have failed to do this job. Big Data is always increasing rapidly. It is not possible to determine with respect to its size. Hadoop is capable to evaluate the big size data. Hadoop has been considered a framework. It has been applied to process the big data sets across numerous clusters. The Tools Hadoop, Map Reduce etc. are capable to manage this huge amount of data are. Along with this the Apache Hive, No SQL are also this kind of tolls. Information extraction has been considered essential. Its cause is that there is rapid growth of unstructured text data. Thus, it has been considered a computationally intensive and MapReduce and parallel database management systems. These are applied to evaluate the huge size of information. This paper has familiarized big data tools such as pache hive and Apache pig. here the comparison of hive and pig has been made based on some parameters. After making comparison it has been come to know that the hive performs better as compare to pig. Major difference in Hadoop MapReduce and Spark lies in way of processing. Spark is capable to do it in-memory. However, Hadoop MapReduce need to read from and write to the disk. Thus, the speed of processing is different. Spark is 100 times faster as compare to MapReduce

Web Server log Analysis for Unstructured data Using Apache Flume and Pig

Web Server log Analysis for Unstructured data Using Apache Flume and Pig

An Overview of Apache Pig and Apache Hive

Ever since the enhancement of technology has taken place, the data is growing at an alarming rate. The most prominent factor of data growth is the “Social Media”, leads to the origination of a tremendous amount of data called Big Data. Big Data is a term used for data sets that are extremely large in size as well as complicated to store and process using traditional database processing applications. A saviour to deal with Big Data is “Hadoop” and two major components of Hadoop which are HDFS (Distributed Storage) and Map Reduce(Parallel Processing). Apache Pig and Hive is an essential part of the Hadoop Ecosystem. This paper covers an overview of both Apache Pig and Hive with their architecture. As Hadoop, no doubt is doing tremendously great work by storing and processing the huge volume of data but there are more frameworks now a days to increase the efficiency of Hadoop framework which are basically seen as the layers of Hadoop or a part of Apache Hadoop project. And that is why this paper includes the two most important layers namely Apache Pig and Apache Hive.

Using the Big Data in the human resources management systems

Measures like setting a destination, evaluation of performances, planning of the labour force about business, personnel recruitment, charging of personnel, informations about personnel, analytical of labour force and reporting, etc., are considered as vital problems of the human resources management. In this study, these criteria in a variety of data were collected with a large repository Apache Hadoop Distributed File System file system owned. Data entry and analysis were used Apache Pig and Java programming languages. The aim of the study is to help ‘the owners of business’ evaluate the abundance of data and to get rid of the management complexity via an application which is the biggest problem of management system in human resources. Keywords: Big Data, management of human resources, Hadoop, Pig, map-reduce.

Integración de herramientas para la toma de decisiones en la congestión vehicular

Este estudio tiene como finalidad presentar un análisis de la utilización e integración de herramientas tecnológicas que ayudan a tomar decisiones en situaciones de congestión vehicular. La ciudad de Quito-Ecuador es considerada como un caso de estudio para el trabajo realizado. La investigación se presenta en función del desarrollo de una aplicación, haciendo uso de herramientas Big Data (Apache Flume, Apache Hadoop, Apache Pig), que permiten el procesamiento de gran cantidad de información que se requiere recolectar, almacenar y procesar. Uno de los aspectos innovadores de la aplicación es el uso de la red social Twitter como fuente de origen de datos. Para esto se utilizó su interfaz de programación de aplicaciones (Application Programming Interface, API), la cual permite tomar datos de esta red social en tiempo real e identificar puntos probables de congestión. Este estudio presenta resultados de pruebas realizadas con la aplicación, durante un período de 9 meses.

Health data analytics using scalable logistic regression with stochastic gradient descent

As wearable medical sensors continuously generate enormous data, it is difficult to process and analyse. This paper focuses on developing scalable sensor data processing architecture in cloud computing to store and process body sensor data for healthcare applications. Proposed architecture uses big data technologies such as Apache Flume, Apache Pig and Apache HBase to collect and store huge sensor data in the Amazon web service. Apache Mahout implementation of MapReduce-based online stochastic gradient descent algorithm is used in the logistic regression to develop the scalable diagnosis model. Cleveland heart disease database (CHDD) is used to train the logistic regression model. Wearable body sensors are used to get the blood pressure, blood sugar level and heart rate of the patient to predict the heart disease status. Proposed prediction model efficiently classifies the heart disease with the accuracy of training and validation sample is 81.99% and 81.52%, respectively.

A new architecture of Internet of Things and big data ecosystem for secured smart healthcare monitoring and alerting system

Wearable medical devices with sensor continuously generate enormous data which is often called as big data mixed with structured and unstructured data. Due to the complexity of the data, it is difficult to process and analyze the big data for finding valuable information that can be useful in decision-making. On the other hand, data security is a key requirement in healthcare big data system. In order to overcome this issue, this paper proposes a new architecture for the implementation of IoT to store and process scalable sensor data (big data) for health care applications. The Proposed architecture consists of two main sub architectures, namely, Meta Fog-Redirection (MF-R) and Grouping and Choosing (GC) architecture. MF-R architecture uses big data technologies such as Apache Pig and Apache HBase for collection and storage of the sensor data (big data) generated from different sensor devices. The proposed GC architecture is used for securing integration of fog computing with cloud computing. This architecture also uses key management service and data categorization function (Sensitive, Critical and Normal) for providing security services. The framework also uses MapReduce based prediction model to predict the heart diseases. Performance evaluation parameters such as throughput, sensitivity, accuracy, and f-measure are calculated to prove the efficiency of the proposed architecture as well as the prediction model.

PERFORMANCE COMPARISON OF HADOOP MAPREDUCE AND APACHE PIG

In the recent era data has been growing at an exponential rate. This data can be either Structured or Semi-Structured or Unstructured and needs to be processed and analyzed carefully to get new insights. There are various tools or frameworks available for this purpose; few of them are Apache Hadoop MapReduce, Apache Pig,Apache Hive, Apache Spark,Tez etc. These tools are widely adopted for managing and processing BigData. Hadoop MapReduce provides low level of abstraction whereas Pig provides high level of abstraction. In this paper, we discuss the major architectural component differences between MapReduce and Pig and conduct detailed experiments to compare their performances with different inputs.

An Efficient Storage and Retrieval of DICOM Objects using Big Data Technologies

DICOM is a global Information technology standard for electronic medical image. Meta data information and images are stored in a single file with dcm extension. A single DICOM file, sizes from MBs to GBs based on the study. PACS uses RDBMS to store and retrieve DICOM data. Replacing RDBMS with Big data technologies will help to handle DICOM data efficiently. All Indian government hospitals stores around 1400PB data collectively. Thus DICOM qualifies as a problem for handling big data. Storing and retrieving these big data from large repositories are highly complex and challenging. Applying big data techniques for handling DICOM data will helps to save many patient lives and improve areas like research, treatment methods, patient similarity searching, disease progression monitoring, clinical follow-up, case studies, training and learning, expertise sharing and helps to understand different patterns in the medical image data archive in a secured way. This paper presents an extensive survey on selective big data technologies for DICOM data storage and retrieval and also analyses the performance of Apache Pig, Hive and Spark while performing storage and retrieval of DICOM data. Keywords: Big data, RDBMS, PACS, DICOM, Storage

Opinion Mining of Twitter Data using Hadoop and Apache Pig

Opinion Mining of Twitter Data using Hadoop and Apache Pig