Apache Hadoop Platform Research Articles

Experimental Characteristics Study of Data Storage Formats for Data Marts Development within Data Lakes

One of the most popular methods for building analytical platforms involves the use of the concept of data lakes. A data lake is a storage system in which the data are presented in their original format, making it difficult to conduct analytics or present aggregated data. To solve this issue, data marts are used, representing environments of stored data of highly specialized information, focused on the requests of employees of a certain department, the vector of an organization’s work. This article presents a study of big data storage formats in the Apache Hadoop platform when used to build data marts.

Analysis of Big Data Storage Tools for Data Lakes based on Apache Hadoop Platform

When developing large data processing systems, the question of data storage arises. One of the modern tools for solving this problem is the so-called data lakes. Many implementations of data lakes use Apache Hadoop as a basic platform. Hadoop does not have a default data storage format, which leads to the task of choosing a data format when designing a data processing system. To solve this problem, it is necessary to proceed from the results of the assessment according to several criteria. In turn, experimental evaluation does not always give a complete understanding of the possibilities for working with a particular data storage format. In this case, it is necessary to study the features of the format, its internal structure, recommendations for use, etc. The article describes the features of both widely used data storage formats and the currently gaining popularity.

RETRACTED ARTICLE: Smart traffic management system in metropolitan cities

Now-a-days, traffic congestion is one of the troubles in metro cities. It is mainly due to escalate in vehicle, population etc. Due to the raise in vehicle, the road is incapable of handling the traffic. Consequently it is very important to revamp the traffic management system. The advanced traffic management system intention is to ameliorate the safety and efficiency of the transportation system. Advanced traffic management system permit opportunities for new methods of evaluations and continuing assessment is provided. A few scenarios were analysed in the traffic issues and better performance is acquired in this research. The traffic map in New York City is taken as a database model and to discover the shortest path an advanced Dijkstra’s algorithm is applied. The advanced form of Hadoop known as Apache spark is used. Spark is the open standard and it is easy to program as it has tons of high level operators which can be flexible in-memory data processing that enables batch, real-time, and advanced analytics on the Apache Hadoop platform. R-tool is an open source programming language and software environment for statistical computing is also employed. By using time forecasting algorithm, for each scenarios the vehicle speed, count, collusion, time, etc. is calculated and the Map Analysis is done with good performance by using R-tool. The overall performance of the system that enhances the traffic control efficiency is 96.23%

Employing differential privacy for big data security

Big data frequently contains huge amounts of personal identifiable information and therefore the protection of user’s privacy becomes a challenge. Lots of researches had been carried out on securing big data, but still limited in efficient privacy management and data sensitivity. This paper designs a mechanism that employs Differential Privacy for protection of personal data, which enforces privacy and access restriction level. The Differential Privacy technique acts on the request by introducing a minimum distortion to the information provided by the database system. The mechanism, DP-Data was implemented with Python scripting and Java Programming languages, Mysql and VmWare on Apache Hadoop platform. To test the effectiveness of DP-Data, a medical dataset with 1,048,576 instances and 12 attributes was employed. It was evaluated based on its utility, scalability, accuracy, sensitivity, specificity and processing time. The results indicated accuracy of 95.80 %, sensitivity of 93.60 %, specificity of 98.00 % and 0.40 ms processing time with high utility and good scalability which shows that the time it takes to preserve a data of 5000 tuples or less are almost similar. From these results, the application of differential privacy in solving privacy issue proved a high level of efficiency. Hence, the deployment of a secure big data framework that is based on access restriction and preserved level of privacy poses a higher level of protection of user’s privacy in comparison with other techniques.Keywords: Big Data, Privacy Preservation, Differential Privacy

До питання автоматизації проектування робочих процесів на основі алгебро-алгоритмічного та онтологічного інструментарію

The article presents the concept of a system for automated workflow design based on algebra-algorithmic and ontological tools. The previously developed applied ontology for program design has been expanded with new concepts related to the workflow design. The existing approaches are analysed and architecture for system that under development is proposed. The work of the system is illustrated by the example of the deve­lopment of the Apache Oozie workflow for analyzing large amounts of data on the distributed Apache Hadoop platform. It is shown that the combination of ontology and algebra-algorithmic tools provides significant potential for adaptation, optimization, integration and modification. In addition, the high level of abstraction provided by these tools facilitates the understanding of the designed workflows by both people and software agents, simplifying teamwork, workflow sharing and reuse. The results of a quantitative assessment of the proposed tools indicate that they can contribute to a significant increase in development productivity and reduce labor costs. Problems in programming 2019; 1: 37-47

Medical Big Data Warehouse: Architecture and System Design, a Case Study: Improving Healthcare Resources Distribution.

The huge increases in medical devices and clinical applications which generate enormous data have raised a big issue in managing, processing, and mining this massive amount of data. Indeed, traditional data warehousing frameworks can not be effective when managing the volume, variety, and velocity of current medical applications. As a result, several data warehouses face many issues over medical data and many challenges need to be addressed. New solutions have emerged and Hadoop is one of the best examples, it can be used to process these streams of medical data. However, without an efficient system design and architecture, these performances will not be significant and valuable for medical managers. In this paper, we provide a short review of the literature about research issues of traditional data warehouses and we present some important Hadoop-based data warehouses. In addition, a Hadoop-based architecture and a conceptual data model for designing medical Big Data warehouse are given. In our case study, we provide implementation detail of big data warehouse based on the proposed architecture and data model in the Apache Hadoop platform to ensure an optimal allocation of health resources.

DIME: a novel framework for de novo metagenomic sequence assembly.

The recently developed next generation sequencing platforms not only decrease the cost for metagenomics data analysis, but also greatly enlarge the size of metagenomic sequence datasets. A common bottleneck of available assemblers is that the trade-off between the noise of the resulting contigs and the gain in sequence length for better annotation has not been attended enough for large-scale sequencing projects, especially for the datasets with low coverage and a large number of nonoverlapping contigs. To address this limitation and promote both accuracy and efficiency, we develop a novel metagenomic sequence assembly framework, DIME, by taking the DIvide, conquer, and MErge strategies. In addition, we give two MapReduce implementations of DIME, DIME-cap3 and DIME-genovo, on Apache Hadoop platform. For a systematic comparison of the performance of the assembly tasks, we tested DIME and five other popular short read assembly programs, Cap3, Genovo, MetaVelvet, SOAPdenovo, and SPAdes on four synthetic and three real metagenomic sequence datasets with various reads from fifty thousand to a couple million in size. The experimental results demonstrate that our method not only partitions the sequence reads with an extremely high accuracy, but also reconstructs more bases, generates higher quality assembled consensus, and yields higher assembly scores, including corrected N50 and BLAST-score-per-base, than other tools with a nearly theoretical speed-up. Results indicate that DIME offers great improvement in assembly across a range of sequence abundances and thus is robust to decreasing coverage.

Application of HADOOP to Store and Process Big Data Gathered from an Urban Water Distribution System

Information technology has become an integral part of municipal water distribution systems (WDS). Various types of sensors, e.g., smart water meters, usually work in real-time mode delivering a huge amount of data. Big data must be stored in appropriate databases. Along with the development of data mining tools, the analysis of big data is very important for the management of WDS. Valuation of NoSQL databases for water data is currently in its very early stages. In this paper, the Apache Hadoop platform is investigated with respect to a possible database solution based on NoSQL. We present comparative experiments evaluating the performance of the Hadoop and MySQL databases.

Market Basket Analysis algorithms with MapReduce

The MapReduce approach has been popular in computing large scale data since Google implemented its platform on Google Distributed File Systems (GFS) followed by Amazon Web Service (AWS) providing the Apache Hadoop platform in inexpensive computing nodes. Map/Reduce motivates to redesign and convert the existing sequential algorithms to MapReduce as restricted parallel programming so that the paper proposes Market Basket Analysis algorithm with MapReduce as well as apriority property. Two algorithms are proposed by adapting an existing Apriori‐algorithm and building a simple algorithm that sorts data sets and converts it to (key, value) pairs to fit with MapReduce. It is executed on Amazon EC2 Map/Reduce platform. The experimental results show that the Apriori‐algorithm does not perform as well as the simple algorithm. Using the simple algorithm, the code with Map/Reduce increases the performance by adding more nodes, but at a certain point there is a bottleneck that does not allow further performance gain. It is believed that the operations of distributing, aggregating, and reducing data in Map/Reduce, cause the bottleneck. WIREs Data Mining Knowl Discov 2013, 3:445–452. doi: 10.1002/widm.1107This article is categorized under: Algorithmic Development > Association Rules Application Areas > Business and Industry Fundamental Concepts of Data and Knowledge > Big Data Mining