Hadoop Research Articles

Big data storage and management in SaaS applications

As an important service model for advanced computing, SaaS uses a defined protocol that manages services and applications. The popularity of advanced computing has reached a level that has led to the generation of large data sets, which is also called Big data. Big data is evolving with great velocity, large volumes, and great diversity. Such an amplification of data has brought into question the existing database tools in terms of their capabilities. Previously, storage and processing of data were simple tasks; however, it is now one of the biggest challenges in the industry. Experts are paying close attention to big data. Designing a system capable of storing and analyzing such data in order to extract meaningful information for decision-making is a priority. The Apache Hadoop, Spark, and NoSQL databases are some of the core technologies that are being used to solve these issues. This paper contributes to the solutions to the issues of big data storage and processing. It presents an analysis of the current technologies in the industry that could be useful in this context. Efforts have been focused on implementing a novel Trinity model, which is built using the lambda architecture with the following technologies: Hadoop, Spark, Kafka, and MongoDB.

Frequent Itemsets Mining for Big Data: A Comparative Analysis

Abstract Itemset mining is a well-known exploratory data mining technique used to discover interesting correlations hidden in a data collection. Since it supports different targeted analyses, it is profitably exploited in a wide range of different domains, ranging from network traffic data to medical records. With the increasing amount of generated data, different scalable algorithms have been developed, exploiting the advantages of distributed computing frameworks, such as Apache Hadoop and Spark. This paper reviews Hadoop- and Spark-based scalable algorithms addressing the frequent itemset mining problem in the Big Data domain through both theoretical and experimental comparative analyses. Since the itemset mining task is computationally expensive, its distribution and parallelization strategies heavily affect memory usage, load balancing, and communication costs. A detailed discussion of the algorithmic choices of the distributed methods for frequent itemset mining is followed by an experimental analysis comparing the performance of state-of-the-art distributed implementations on both synthetic and real datasets. The strengths and weaknesses of the algorithms are thoroughly discussed with respect to the dataset features (e.g., data distribution, average transaction length, number of records), and specific parameter settings. Finally, based on theoretical and experimental analyses, open research directions for the parallelization of the itemset mining problem are presented.

Source camera identification: a distributed computing approach using Hadoop

The widespread use of digital images has led to a new challenge in digital image forensics. These images can be used in court as evidence of criminal cases. However, digital images are easily manipulated which brings up the need of a method to verify the authenticity of the image. One of the methods is by identifying the source camera. In spite of that, it takes a large amount of time to be completed by using traditional desktop computers. To tackle the problem, we aim to increase the performance of the process by implementing it in a distributed computing environment. We evaluate the camera identification process using conditional probability features and Apache Hadoop. The evaluation process used 6000 images from six different mobile phones of the different models and classified them using Apache Mahout, a scalable machine learning tool which runs on Hadoop. We ran the source camera identification process in a cluster of up to 19 computing nodes. The experimental results demonstrate exponential decrease in processing times and slight decrease in accuracies as the processes are distributed across the cluster. Our prediction accuracies are recorded between 85 to 95% across varying number of mappers.

A Comparison of HDFS Compact Data Formats: Avro Versus Parquet

In this paper, file formats like Avro and Parquet are compared with text formats to evaluate the performance of the data queries. Different data query patterns have been evaluated. Cloudera’s open-source Apache Hadoop distribution CDH 5.4 has been chosen for the experiments presented in this article. The results show that compact data formats (Avro and Parquet) take up less storage space when compared with plain text data formats because of binary data format and compression advantage. Furthermore, data queries from the column based data format Parquet are faster when compared with text data formats and Avro.

Application of feedback control on the cloud-based web simulation system

This article illustrated the development of a web simulation system based on Matlab distributed parallel cluster. The web simulation system construct a visual and controllable simulation interface using javascript and Asynchronous JavaScript and XML(AJAX) technology. In order to realize the Matlab resources dynamic allocation and management, elastic resources management method has been adopted. To achieve the multi-user concurrent access, the resource management system has been designed by adopting principle of feedback control and principle of Apache Hadoop Yet Another Resource Negotiator(YARN). The feasibility of this system demonstrated through application of simulation and practical use.

Cloud Forensics : Isolating Cloud Instance

Abstract—Cloud computing has been the trending model for storing, accessing and modifying the data over the Internet in the recent years. Rising use of the cloud has generated a new concept related to the cloud which is cloud forensics. Cloud forensics can be defined as investigating for evidence over the cloud, so it can be viewed as a combination of both cloud computing and digital forensics. Many issues of applying forensics in the cloud have been addressed. Isolating the location of the incident has become an essential part of forensic process. This is done to ensure that evidence will not be modified or changed. Isolating an instant in the cloud computing has become even more challenging, due to the nature of the cloud environment. In the cloud, the same storage or virtual machine have been used by many users. Hence, the evidence is most likely will be overwritten and lost. The proposed solution in this paper is to isolate a cloud instance. This can be achieved by marking the instant that reside in the servers as "Under Investigation". To do so, cloud file system must be studied. One of the well-known file systems used in the cloud is Apache Hadoop Distributed File System (HDFS). Thus, in this paper the methodology used for isolating a cloud instance would be based on the HDFS architecture.
 Keywords: cloud computing; digital forensics; cloud forensics

English

English

Hadoop on Named Data Networking

The Named Data Networking (NDN) architecture retrieves content by names rather than connecting to specific hosts. It provides benefits such as highly efficient and resilient content distribution, which fit well to data-intensive distributed computing. This paper presents and discusses our experience in modifying Apache Hadoop, a popular MapReduce framework, to operate on an NDN network. Through this first-of-its-kind implementation process, we demonstrate the feasibility of running an existing, large, and complex piece of distributed software commonly seen in data centers over NDN. We show advantages such as simplified network code and reduced network traffic which are beneficial in a data center environment. There are also challenges faced by NDN, that are being addressed by the community, which can be magnified under data center traffic. Through detailed evaluation, we show a reduction of 16% for overall data transmission between Hadoop nodes while writing data with default replication settings. Preliminary results also show promise for in-network caching of repeated reads in distributed applications. We also show that overall performance is currently slower under NDN, and we identify challenges and opportunities for further NDN improvements.

Big Data’s Tools for Internet Data Analytics: Modelling of System Dynamics

In this paper, an application based on Apache Hadoop is deployed to gather, store and analyse the data from the internet, especially online and social media. Nowadays, this application is a common tool for media analysis. In our case, it is used to assist in the modelling of system dynamics. Basically, There are several tools that will be used, such as for file system, data crawling from the Internet, data indexing, data storage, and data analytics. The selection of technology is as the industrial trend. Surely, this is not the best approach, but as another perspective for modelling of system dynamics. A system dynamics model is developed to study the profitability of the telecommunication company and how the complaint or negative sentiment will impact to their profits. The clustering analytics are used to identify the components of the system. In continuation of the improvement process, the clustering analytics will be used not only as a one time effort. It runs periodically to develop a better model of the system. Sentiment analysis tool is used as the input for one of the components which is compliant components. The sentiment is sourced from online and social media. Manual investigation and analytics of internet data is required in developing the relationships between the components.

Hadoop on Named Data Networking

In today's data centers, clusters of servers are arranged to perform various tasks in a massively distributed manner: handling web requests, processing scientific data, and running simulations of real-world problems. These clusters are very complex, and require a significant amount of planning and administration to ensure that they perform to their maximum potential. Planning and configuration can be a long and complicated process; once completed it is hard to completely re-architect an existing cluster. In addition to planning the physical hardware, the software must also be properly configured to run on a cluster. Information such as which server is in which rack and the total network bandwidth between rows of racks constrain the placement of jobs scheduled to run on a cluster. Some software may be able to use hints provided by a user about where to schedule jobs, while others may simply place them randomly and hope for the best. Every cluster has at least one bottleneck that constrains the overall performance to less than the optimal that may be achieved on paper. One common bottleneck is the speed of the network: communication between servers in a rack may be unable to saturate their network connections, but traffic flowing between racks or rows in a data center can easily overwhelm the interconnect switches. Various network topologies have been proposed to help mitigate this problem by providing multiple paths between points in the network, but they all suffer from the same fundamental problem: it is cost-prohibitive to build a network that can provide concurrent full network bandwidth between all servers. Researchers have been working on developing new network protocols that can make more efficient use of existing network hardware through a blurring of the line between network layer and applications. One of the most well-known examples of this is Named Data Networking (NDN), a data-centric network architecture that has been in development for several years. While NDN has received significant attention for wide-area Internet, a detailed understanding of NDN benefits and challenges in the data center environment has been lacking. The Named Data Networking architecture retrieves content by names rather than connecting to specific hosts. It provides benefits such as highly efficient and resilient content distribution, which fit well to data-intensive distributed computing. This paper presents and discusses our experience in modifying Apache Hadoop, a popular MapReduce framework, to operate on an NDN network. Through this first-of-its-kind implementation process, we demonstrate the feasibility of running an existing, large, and complex piece of distributed software commonly seen in data centers over NDN. We show advantages such as simplified network code and reduced network traffic, which are beneficial in a data center environment. There are also challenges faced by NDN that are being addressed by the community, which can be magnified under data center traffic. Through detailed evaluation, we show a reduction of 16% for overall data transmission between Hadoop nodes while writing data with default replication settings. Preliminary results also show promise for in-network caching of repeated reads in distributed applications. We show that while overall performance is currently slower under NDN, there are challenges and opportunities for further NDN improvements.

Application of a Hadoop-based Distributed System for Offline Processing of Power Quality Disturbances

Electric power quality is a critical issue for electric utilities and their customers and identification of the power quality disturbances is an important task in power system monitoring and protection. Offline processing of power quality disturbances provides an economic alternative for electric distribution companies, not capable of buying enough number of power quality analyzers for monitoring the disturbances online. Due to the wide frequency range of the disturbances which may happen in a power system, a high sampling rate is necessary for digital processing of the disturbances. Therefore, a large volume of data must be processed for this purpose for each node of an electric distribution network and such a processing has not yet been practical. However, thanks to the rapid developments of digital processors and computer networks, processing big databases is not so hard today. Apache Hadoop is an open-source software framework that allows for the distributed processing of large datasets using simple programming models. In this paper, application of Hadoop distributed computing software for offline processing of power quality disturbances is proposed and it is shown that this application makes such a processing possible and leads to a very cheaper system with widespread usage, compared to the power quality analyzers.

Big Data Diagnosis Enhances Innovative Winning Formula in the World of Sports

Objectives: The central theme of this research paper is to analyze how sports have been benefited with the use of Big Data and to have a deep insight of the working of the Map Reduce algorithm which is responsible for handling and analyzing Big Data. Methods/Statistical Analysis: The research work is based on mining and filtering huge self-constructed database comprising records relevant to Indian cricketers after March 1980. The Apache Hadoop framework Hortonworks Sandbox 2.2.0 has been utilized for extracting information as per requirements from the database. The backbone behind working of the Apache Hadoop framework is the MapReduce algorithm which operates in three different phases and provides results depending on scripts and queries written in Apache Hive. Findings: With the advent of wearable and sensors enabled technologies, it has been easier for the analysts collect data from different sources. Many popular companies like Adidas and Nike are engaged in making wearable technologies that show real time stats of a player like his speed, heart rate, etc. These wearable devices can be used during training sessions to get a better picture about player’s fitness in real time. A team can analyze the tactics followed by opponents and predict how they are going to play in the next match with the help of Big Data. But it is not that the data is important, but what matters is that what organizations do with that data. The research paper concentrates on the methods that can be adopted to get the best out of the gathered data. The research work also proves that the MapReduce algorithm is far better than conventional data mining algorithms in terms of efficiency and timing. Application/Improvements: Big Data has brought a revolution in almost every sector and sports are no exemption. Forbes has mentioned that Big Data is the first ever use of statistics and data to make personal decisions in professional sports. The Apache Hadoop framework is also efficient in handling semi-structured and unstructured data. Keywords: Apriori Algorithm, Big Data, Hadoop Framework, Map-Reduce Algorithm, Sports

Experiences with the Twitter Health Surveillance (THS) System.

Social media has become an important platform to gauge public opinion on topics related to our daily lives. In practice, processing these posts requires big data analytics tools since the volume of data and the speed of production overwhelm single-server solutions. Building an application to capture and analyze posts from social media can be a challenge simply because it requires combining a set of complex software tools that often times are tricky to configure, tune, and maintain. In many instances, the application ends up being an assorted collection of Java/Scala programs or Python scripts that developers cobble together to generate the data products they need. In this paper, we present the Twitter Health Surveillance (THS) application framework. THS is designed as a platform to allow end-users to monitor a stream of tweets, and process the stream with a combination of built-in functionality and their own user-defined functions. We discuss the architecture of THS, and describe its implementation atop the Apache Hadoop Ecosystem. We also present several lessons learned while developing our current prototype.

Design a degradation condition monitoring system scheme for rolling bearing using EMD and PCA

PurposeRolling bearings based on rotating machinery are one of the most widely used in industrial applications because of their low cost, high performance and robustness. The purpose of this paper is to describe how to identify degradation condition of rolling bearing and predict its fault time in big data environment in order to achieve zero downtime performance and preventive maintenance for the rolling bearing.Design/methodology/approachThe degradation characteristic parameters of rolling bearings including intrinsic mode energy and failure frequency were, respectively, extracted from the pre-processed original vibration signals using EMD and Hilbert transform. Then, Spearman’s rank correlation coefficient and PCA were used to obtain the health index of the rolling bearing so as to detect the appearance of degradations. Furthermore, the degradation condition of the rolling bearings might be identified through implementing the monotonicity analysis, robustness analysis and degradation analysis of the health index.FindingsThe effectiveness of the proposed method is verified by a case study. The result shows that the proposed method can be applied to monitor the degradation condition of the rolling bearings in industrial application.Research limitations/implicationsFurther experiment remains to be done so as to validate the effectiveness of the proposed method using Apache Hadoop when massive sensor data are available.Practical implicationsThe paper proposes a methodology for rolling bearing condition monitoring representing the steps that need to be followed. Real-time sensor data are utilized to find the degradation characteristics.Originality/valueThe result of the work presented in this paper form the basis for the software development and implementation of condition monitoring system for rolling bearings based on Hadoop.

MarDRe: efficient MapReduce-based removal of duplicate DNA reads in the cloud.

This article presents MarDRe, a de novo cloud-ready duplicate and near-duplicate removal tool that can process single- and paired-end reads from FASTQ/FASTA datasets. MarDRe takes advantage of the widely adopted MapReduce programming model to fully exploit Big Data technologies on cloud-based infrastructures. Written in Java to maximize cross-platform compatibility, MarDRe is built upon the open-source Apache Hadoop project, the most popular distributed computing framework for scalable Big Data processing. On a 16-node cluster deployed on the Amazon EC2 cloud platform, MarDRe is up to 8.52 times faster than a representative state-of-the-art tool. Source code in Java and Hadoop as well as a user's guide are freely available under the GNU GPLv3 license at http://mardre.des.udc.es . rreye@udc.es.

An REA Ontology-Based Model for Mapping Big Data to Accounting Information Systems Elements

ABSTRACT The term “Big Data” refers to massive volumes of data that grow at an increasing rate and encompass complex data types such as audio and video. While the applications of Big Data and analytic techniques for business purposes have received considerable attention, it is less clear how external sources of Big Data relate to the transaction processing-oriented world of accounting information systems. This paper uses the Resource-Event-Agent Enterprise Ontology (REA) (McCarthy 1982; International Standards Organization [ISO] 2007) to model the implications of external Big Data sources on business transactions. The five-phase REA-based specification of a business transaction as defined in ISO (2007) is used to formally define associations between specific Big Data elements and business transactions. Using Big Data technologies such as Apache Hadoop and MapReduce, a number of information extraction patterns are specified for extracting business transaction-related information from Big Data. We also present a number of analytics patterns to demonstrate how decision making in accounting can benefit from integrating specific external Big Data sources and conventional transactional data. The model and techniques presented in this paper can be used by organizations to formalize the associations between external Big Data elements in their environment and their accounting information artifacts, to build architectures that extract information from external Big Data sources for use in an accounting context, and to leverage the power of analytics for more effective decision making.

Minutiae-based fingerprint matching decomposition: Methodology for big data frameworks

Fingerprint recognition, and in particular minutiae-based matching methods, are ever more deeply implanted into many companies and institutions. As the size of their identification databases grows, there is a need of flexible, reliable structures for fingerprint recognition systems. In this paper, we propose a generic decomposition methodology for minutiae-based matching algorithms that splits the calculation of the matching scores into lower level steps that can be carried out in parallel in a flexible manner. The decomposition allows to adapt any minutiae-based algorithm to frameworks such as MapReduce or Apache Spark. General and specific guidelines to enhance the performance of the adapted matching algorithms are also described. The proposal is evaluated over two matching algorithms, two Big Data frameworks (Apache Hadoop and Apache Spark) and two large-scale fingerprint databases, with promising results concerning the identification time, in addition to the reliability, scalability, distribution and availability capabilities that are provided by such underlying frameworks.

Using machine learning and big data approaches to predict travel time based on historical and real-time data from Taiwan electronic toll collection

As the technology in automation and computation advances, traffic data can be easily collected from multiple sources, such as sensors and surveillance cameras. To extract value from the huge volumes of available data requires the capability to process and extract patterns in large datasets. In this paper, a machine learning method embedded within a big data analytics platform is constructed by using random forests method and Apache Hadoop to predict highway travel time based on data collected from highway electronic toll collection in Taiwan. Various prediction models are then developed for highway travel time based on historical and real-time data to provide drivers with estimated and adjusted travel time information.

HIPI Based Biometric probe image retrieval using Big Image data

Everyday Terabyte of image data is generated through the social media which can generate large Biometric data. Storing and processing Big biometric Image data is a big problem. This paper proposes a model and method using hadoop image processing interface to store the large Biometric images using Hipi image Bundel to form a Hadoop cluster, and shows how it can be process using HIPI combined with OpenCv library. Proposed model uses the hadoop distributed file system for storing these images, and uses already available image processing functions from OpenCV by integrating it with the HIPI. Paper uses the cloudera manager to manage the Hadoop cluster. Keywords: HIPI; Cloludera; HDFS; Hipi Image Bundle; Cull;OpenCV;

Querying Archetype-Based Electronic Health Records Using Hadoop and Dewey Encoding of openEHR Models.

Archetype-based Electronic Health Record (EHR) systems using generic reference models from e.g. openEHR, ISO 13606 or CIMI should be easy to update and reconfigure with new types (or versions) of data models or entries, ideally with very limited programming or manual database tweaking. Exploratory research (e.g. epidemiology) leading to ad-hoc querying on a population-wide scale can be a challenge in such environments. This publication describes implementation and test of an archetype-aware Dewey encoding optimization that can be used to produce such systems in environments supporting relational operations, e.g. RDBMs and distributed map-reduce frameworks like Hadoop. Initial testing was done using a nine-node 2.2 GHz quad-core Hadoop cluster querying a dataset consisting of targeted extracts from 4+ million real patient EHRs, query results with sub-minute response time were obtained.