Scalable Hadoop Research Articles

Online Web query system for various frequency distributions of bus passengers in Taichung city of Taiwan

It is highly desirable that traffic controllers or city residents can discern regular patterns and promptly detect irregularities or abnormal events in a public transportation system. This study proposes a web-based information system that allows users to study the travel behaviour of bus passengers from various perspectives. The system uses data from the comprehensive set of Taichung City Bus Riding Records between 2015 and 2016. However, it can provide the same functionality to any other similar bus transportation system by using the appropriate data. It should be emphasised that the system can provide the frequency distributions not only of passenger trips between two stops but also of the passenger volume for a given segment of any route. Owing to the increased computational and storage-capacity requirements of the proposed system, the scalable Hadoop MapReduce programming model was used. Furthermore, bus companies can use the system to design better service plans, such as more flexible bus schedules and more convenient routes, to meet passenger demand as well as reduce operation cost and energy consumption. The authors believe that the proposed system can make a valuable contribution to public welfare.

A high performance query analytical framework for supporting data-intensive climate studies

Climate observations and model simulations produce vast amounts of data. The unprecedented data volume and the complexity of geospatial statistics and analysis requires efficient analysis of big climate data to investigate global problems such as climate change, natural disasters, diseases, and other environmental issues. This paper introduces a high performance query analytical framework to tackle these challenges by leveraging Hive and cloud computing technologies. With this framework, we propose grid transformation, a new perspective for complex climate analysis that applies a series of atomic transformations to terabytes of climate data using SQL-style query (HiveQL). Specifically, we introduce four types of grid transformations (temporal, spatial, local, and arithmetic) to support a broad range of climate analyses, from the basic spatiotemporal aggregation to more sophisticated anomaly detection. Each query is processed as MapReduce tasks in a highly scalable Hadoop cluster as the parallel processing engine. Big climate data are directly stored and managed in a Hadoop Distributed File System without any data format conversion. A prototype is developed to evaluate the feasibility and performance of the framework. Experimental results show that complex and data-intensive climate analysis can be conducted using intuitive SQL queries with good flexibility and performance. This research provides a building block and practical insights in establishing a cyberinfrastructure that provides a high performance and collaborative environment for data-intensive geospatial applications in climate science.

MapReduce: A Technical Review

MapReduce, a programming model, allows parallel processing of large amount of data sets where various data mining techniques are not quite useful. It’s Map and Reduce functions can be customized by the developers according to their application. This paper gives an idea of MapReduce, its advantages and disadvantages. This paper also focuses on how MapReduce is used, how map and reduce computations are customized, implemented in several scenarios such as in medical field to generate medical reports by processing large medical data sets, stream processing and workflow scheduling in multi core processors, in distributed environment, for processing distributed data sets by using pilot abstractions. We also represent how MapReduce used for deduplication of files to save disk space in data centers. MapReduce based Pre-Post (MRPre-Post) a parallel data mining algorithm is adapted in Hadoop platform to achieve scalability. MapReduce is implemented in vHadoop (Virtual Hadoop), a scalable hadoop virtual cluster to process machine learning algorithms. The scenarios discussed in this paper help developers and researchers how to customize and use MapReduce in their applications.

A Cost Effective Virtual Cluster with Hadoop Framework for Big Data Analytics

Big data processing is currently becoming increasingly important research field in computer technology professionals due to the continuous growth of the amount of data generated by various fields. However, the processing of large-scale research data requires cluster technology infrastructure which causes huge investments for educational institutions. Hadoop is an open-source framework that allows for distributed storage and processing of very large data sets on computer clusters built from commodity hardware. This technology is being widely used for the analysis of large datasets. This paper focuses on proposes a low cost scalable hadoop virtual cluster platform and the performance of hadoop irtual cluster. We first describe the design and implementation of a virtual datacenter using hadoop framework. Then we perform a set of experiments to investigate the performance of virtual datacenter with standard datasets. For experiment and evaluation, the Cloudera's distribution of Apache Hadoop (CDH) is installed and configured on the DIU Cloud and Big Data Lab as a prototype implementation of a virtual data center and tested with 40 workstations. In this paper, we use TPC BENCHMARK ™ DS” by-Transaction Processing Performance Council (TPC) benchmarks for evaluation virtual data center performance. The contributions of this paper, is to design model and implement a cost effective elastic virtual data center with hadoop framework and resource utilizations for educational institutions to provide high performance for distributed and parallel processing; as well as, Identify the bottlenecks of this systems e.g. bandwidth of network connectivity with all nodes in the lab.

COSMOS: Towards an integrated and scalable service for analysing social media on demand

The growing number of people using social media to publish their opinions, share expertise, make social connections and promote their ideas to an international audience is creating data on an epic scale. This enables social scientists to conduct research into ethnography, discourse analysis and analysis of social interactions, providing insight into today's society, which is largely augmented by social computing. The tools available for such analysis are often proprietary and expensive, and often non-interoperable, meaning the rapid marshalling of large data-sets through a range of analyses is arduous and difficult to scale. The collaborative online social media observatory (COSMOS), an integrated social media analysis tool is presented, developed for open access within academia. COSMOS is underpinned by a scalable Hadoop infrastructure and can support the rapid analysis of large data-sets and the orchestration of workflows between tools with limited human effort. We describe an architecture and scalability results for the computational analysis of social media data, and comment on the storage, search and retrieval issues associated with massive social media data-sets. We also provide an insight into the impact of such an integrated on-demand service in the social science academic community.

Scripting for large-scale sequencing based on Hadoop

The large volumes of data generated by modern sequencing experiments present significant challenges in their manipulation and analysis. Traditional approaches are often found to be complicated to scale. We describe our ongoing work on SeqPig, a tool that facilitates the use of the Pig Latin distributed scripting language to manipulate, analyze and query sequencing data applying the advances motivated by the “big data revolution” in data-intensive activities. SeqPig provides access to popular data formats and implements a number of custom sequencing-specific functions. Most importantly, it grants users access to the scalable Hadoop platform from a high level scripting language