Hadoop File System Research Articles

Big Data and Java are integrated with machine learning

This in-depth investigation explores the revolutionary nexus of Java, Big Data, and Machine Learning (ML), clarifying the innovations and synergies that result from their integration. The voyage commences with a summary of the past, following the growth from Java's fundamental function in high-level development to the revolutionary influence of Big Data frameworks such as Apache Hadoop and Apache Spark. The story then delves into the fundamentals of machine learning in Java, highlighting its adaptability, rich library support, and crucial role in building strong ML pipelines. The investigation delves into the transformative power of Big Data frameworks, highlighting the distributed file system of Hadoop and the in-memory processing capabilities of Spark. We observe the significant effects of this convergence on a variety of industries through real-world case studies, from e-commerce personalized suggestions to fraud detection in banking. The insights gained from these implementations highlight how crucial it is to use ML models with ethical considerations, interdisciplinary cooperation, and ongoing learning. The following sections cover the nuances of data preprocessing, including the use of Java in ETL workflows, scalable feature engineering using Big Data frameworks, and data quality assurance via transformation and cleansing. ML model deployment is the major focus, along with an exploration of the Java runtime environment, micro services architecture, and crucial aspects of model robustness monitoring and maintenance. The investigation concludes with a focus on case studies and success stories that demonstrate the real-world effects of this convergence in sectors like e-commerce, finance, and healthcare. These real-world examples highlight the accomplishments of companies like Netflix, Uber, Airbnb, and others and provide insightful information about how well integration works to accomplish a range of business objectives.

Forensic implications of stacked file systems

While file system analysis is a cornerstone of forensic investigations and has been extensively studied, certain file system classes have not yet been thoroughly examined from a forensic perspective. Stacked file systems, which use an underlying file system for data storage instead of a volume, are a prominent example. With the growth of cloud infrastructure and big data, it is increasingly likely that investigators will encounter distributed stacked file systems, such as MooseFS and the Hadoop File System, that employ this architecture. However, current standard models and tools for file system analysis fall short of addressing the complexities of stacked file systems. This paper highlights the forensic challenges and implications associated with stacked file systems, discussing their unique characteristics in the context of forensic analyses. We provide insights through three analyses of different stacked file systems, illustrating their operational details and emphasizing the necessity of understanding this file system category during forensic investigations. For this purpose, we present general considerations that must be made when dealing with the analysis of stacked file systems.

Big Data Security on Hadoop Open Source Frame for Healthcare Data Management using One-Time-Pad Encryption Algorithm

The study elicited knowledge about the factors associated with one-time pad encryption/decryption with big data in healthcare; formulate an assembled algorithms model for one-time pad encryption; design and implement the system and evaluating the system performance with the view implementing big data security on Hadoop open-source framework for healthcare data. Literature was sourced to investigate the factors associated with healthcare security attacks and various consequences of breach of data. An assembled algorithm model was formulated using mathematical theory of one-time pad encryption and a model was designed using Universal Modelling Language (UML) and implemented using python programming language, Distributed File System of Hadoop, Yet Another Resource Negotiator called YARN; encryption time and decryption time was adopted for the performance metrics deployed for the evaluation of the developed system. The result showed that as the size of the files increased, the encryption/decryption time keeps increasing as well. While carryout the algorithm evaluation, two different values (file sizes) were used for testing on the Hadoop framework.Securing the healthcare (Ebola) big-data, it was observed that OTP encryption/decryption performed better compared to AES encryption/decryption in term of computational processing time of the healthcare big-data considered. Considering before/after downloading, it was observed that there was need for authentication for another level of security towards securing healthcare records on HDFS. The study concluded that, big data analytics on Hadoop is ideal for today’s big healthcare data and also that One Time Pad encryption algorithm is sufficient to provide needed big healthcare data security.

Memory aware optimized Hadoop MapReduce model in cloud computing environment

<p>In the last decade, data analysis has become one of the popular tasks due to enormous growth in data every minute through different applications and instruments. MapReduce is the most popular programming model for data processing. Hadoop constitutes two basic models i.e., Hadoop file system (HDFS) and MapReduce, Hadoop is used for processing a huge amount of data whereas MapReduce is used for data processing. Hadoop MapReduce is one of the best platforms for processing huge data in an efficient manner such as processing web logs data. However, existing model This research work proposes memory aware optimized Hadoop MapReduce (MA-OHMR). MA-OHMR is developed considering memory as the constraint and prioritizes memory allocation and revocation in mapping, shuffling, and reducing, this further enhances the job of mapping and reducing. Optimal memory management and I/O operation are carried out to use the resource inefficiently manner. The model utilizes the global memory management to avoid garbage collection and MA-OHMR is optimized on the makespan front to reduce the same. MA-OHMR is evaluated considering two datasets i.e., simple workload of Wikipedia dataset and complex workload of sensor dataset considering makespan and cost as an evaluation parameter.</p>

AdMap: a framework for advertising using MapReduce pipeline

There is a vast collection of data for consumers due to tremendous development in digital marketing. For their ads or for consumers to validate nearby services which already are upgraded to the dataset systems, consumers are more concerned with the amount of data. Hence there is a void formed between the producer and the client. To fill that void, there is the need for a framework which can facilitate all the needs for query updating of the data. The present systems have some shortcomings by a vast number of information that each time lead to decision tree-based approach. A systematic solution to the automated incorporation of data into a Hadoop distributed file system (HDFS) warehouse (Hadoop file system) includes a data hub server, a generic data charging mechanism and a metadata model. In our model framework, the database would be able to govern the data processing schema. In the future, as a variety of data is archived, the datalake will play a critical role in managing that data. To order to carry out a planned loading function, the setup files immense catalogue move the datahub server together to attach the miscellaneous details dynamically to its schemas.

AdMap: a framework for advertising using MapReduce pipeline

There is a vast collection of data for consumers due to tremendous development in digital marketing. For their ads or for consumers to validate nearby services which already are upgraded to the dataset systems, consumers are more concerned with the amount of data. Hence there is a void formed between the producer and the client. To fill that void, there is the need for a framework which can facilitate all the needs for query updating of the data. The present systems have some shortcomings by a vast number of information that each time lead to decision tree-based approach. A systematic solution to the automated incorporation of data into a Hadoop distributed file system (HDFS) warehouse (Hadoop file system) includes a data hub server, a generic data charging mechanism and a metadata model. In our model framework, the database would be able to govern the data processing schema. In the future, as a variety of data is archived, the datalake will play a critical role in managing that data. To order to carry out a planned loading function, the setup files immense catalogue move the datahub server together to attach the miscellaneous details dynamically to its schemas.

Contributions to Hadoop File System Architecture by Revising the File System Usage Along with Automatic Service

The usage of unstructured data is becoming obvious by companies and social media is raised heavily from past decade. The sharing of images, audio, and video content by the individual user and corporate can be observed everywhere. The current work focused on the Hadoop framework revision contributions so as to improve the performance of the eco system in the context of space and time parameters. The architecture basically provides the usage of Hadoop Distributed File System (HDFS) and MapReduce (MR). We are proposing certain revision contributions so that the process of importing and processing of the tasks can get the benefit of time and space usage in the effective and efficient manner. The work provides the service running in two different ways which reduces the time requirements of the cluster management. In the distributed environment, this revision helps in the reduction of waiting time for the start of the service. The other context we have focused on, the local file system handler in the data storage and processing of the data, the provision of using the file system according to the proposed architecture, will handle the CPU context switch while performing the import and export process in the running of the jobs. The outcome of the work is revision architecture to reflect the service initiation by all the machines in the cluster and file system revision approach to minimize the CPU context switch, while performing the storage and processing relevant aspects of the Hadoop cluster.

Using HDFS to Load, Search, and Retrieve Data from Local Data Nodes

Abstract: By utilizing the Hadoop environment, data may be loaded and searched from local data nodes. Because the dataset's capacity may be vast, loading and finding data using a query is often more difficult. We suggest a method for dealing with data in local nodes that does not overlap with data acquired by script. The query's major purpose is to store information in a distributed environment and look for it quickly. In this section, we define the script to eliminate duplicate data redundancy when searching and loading data in a dynamic manner. In addition, the Hadoop file system is available in a distributed environment. Keywords: HDFS; Hadoop distributed file system; replica; local; distributed; capacity; SQL; redundancy

An Intelligent Approach to Resource Allocation on Heterogeneous Cloud Infrastructures

Cloud computing systems are rapidly evolving toward multicloud architectures supported on heterogeneous hardware. Cloud service providers are widely offering different types of storage infrastructures and multi-NUMA architecture servers. Existing cloud resource allocation solutions do not comprehensively consider this heterogeneous infrastructure. In this study, we present a novel approach comprised of a hierarchical framework based on genetic programming to solve problems related to data placement and virtual machine allocation for analytics applications running on heterogeneous hardware with a variety of storage types and nonuniform memory access. Our approach optimizes data placement using the Hadoop File System on heterogeneous storage devices on multicloud systems. It guarantees the efficient allocation of virtual machines on physical machines with multiple NUMA (nonuniform memory access) domains by minimizing contention between workloads. We prove that our solutions for data placement and virtual machine allocation outperform other state-of-the-art approaches.

FASTA/Q data compressors for MapReduce-Hadoop genomics: space and time savings made easy

BackgroundStorage of genomic data is a major cost for the Life Sciences, effectively addressed via specialized data compression methods. For the same reasons of abundance in data production, the use of Big Data technologies is seen as the future for genomic data storage and processing, with MapReduce-Hadoop as leaders. Somewhat surprisingly, none of the specialized FASTA/Q compressors is available within Hadoop. Indeed, their deployment there is not exactly immediate. Such a State of the Art is problematic.ResultsWe provide major advances in two different directions. Methodologically, we propose two general methods, with the corresponding software, that make very easy to deploy a specialized FASTA/Q compressor within MapReduce-Hadoop for processing files stored on the distributed Hadoop File System, with very little knowledge of Hadoop. Practically, we provide evidence that the deployment of those specialized compressors within Hadoop, not available so far, results in better space savings, and even in better execution times over compressed data, with respect to the use of generic compressors available in Hadoop, in particular for FASTQ files. Finally, we observe that these results hold also for the Apache Spark framework, when used to process FASTA/Q files stored on the Hadoop File System.ConclusionsOur Methods and the corresponding software substantially contribute to achieve space and time savings for the storage and processing of FASTA/Q files in Hadoop and Spark. Being our approach general, it is very likely that it can be applied also to FASTA/Q compression methods that will appear in the future.AvailabilityThe software and the datasets are available at https://github.com/fpalini/fastdoopc

Replica-aware data recovery performance improvement for Hadoop system with NVM

The non-volatile memory (NVM) is the promising device to store data and accelerate big data analysis due to its excellent I/O performance. However, we find that simply replacing hard disk drive (HDD) with NVM cannot bring the expected performance improvement. In this paper, we take the data recovery issue in Hadoop file system (HDFS) as a case study to investigate how to take advantage of the performance of NVM. We analyze the data recovery mechanism in HDFS and find that the configuration of replication tasks in the DataNode can affect the data recovery significantly. We conduct extensive analysis and experiments tuning the configuration and also get some interesting findings. With the new configuration, we increase the data recovery performance from 17 to 71%. We can also improve the execution performance of MapReduce jobs from 28 to 59% through optimized configuration. We also find that the sudden data recovery brings disordered network resource competition, which reduces the performance of MapReduce jobs. Hence, We present a priority-aware multi-stage data recovery method. This improves the performance by 32.5% in addition for the MapReduce jobs.

Acquiring and storing Internet of Things data into Hadoop

This paper looks at issues, related to the use of Big Data and Internet of Things. Those two modern paradigms are essential in the ongoing digital business transformation of our world. Our society and activities create new products, services and improve the efficiency of the traditional ways of production, business, and life. We use one of the most popular software libraries, operating on big data sets – the Apache Hadoop, and experiment with data, generated by IoT. We investigate the impact on performance by varying parameters of Internet of Things data like number of messages and rates write of those messages in the system. Our experiments use the MQTT protocol and demonstrate that there are ways to achieve better effectiveness by using tools to automate data streams, and optimization of data sizes than direct store of data in the Hadoop file system.

An Enhanced Multiple Linear Regression Model for Seasonal Rainfall Prediction

Aims & Background: India is a country which has exemplary climate circumstances comprising of different seasons and topographical conditions like high temperatures, cold atmosphere, and drought, heavy rainfall seasonal wise. These utmost varieties in climate make us exact weather prediction is a challenging task. Majority people of the country depend on agriculture. Farmers require climate information to decide the planting. Weather prediction turns into an orientation in farming sector to deciding the start of the planting season and furthermore quality and amount of their harvesting. One of the variables are influencing agriculture is rainfall. Objectives & Methods: The main goal of this project is early and proper rainfall forecasting, that helpful to people who live in regions which are inclined natural calamities such as floods and it helps agriculturists for decision making in their crop and water management using big data analytics which produces high in terms of profit and production for farmers. In this project, we proposed an advanced automated framework called Enhanced Multiple Linear Regression Model (EMLRM) with MapReduce algorithm and Hadoop file system. We used climate data from IMD (Indian Metrological Department, Hyderabad) in 1901 to 2002 period. Results: Our experimental outcomes demonstrate that the proposed model forecasting the rainfall with better accuracy compared with other existing models. Conclusion: The results of the analysis will help the farmers to adopt effective modeling approach by anticipating long-term seasonal rainfall.

Enhancing Proportional IO Sharing on Containerized Big Data File Systems

Big Data platforms recently employ resource management systems, such as YARN, Mesos, and Google Borg, to provision computational resources. These systems adopt containerization to share the computing resources in a multi-tenant setting with low performance overhead and interference. However, it may be observed that tenants often interfere with each other on the underlying Big Data File Systems (BDFS), e.g., Hadoop File System, which have been widely deployed as a persistent layer in current data centers. A solution with systematic generality is to containerize BDFS itself to isolate and allocate its IO sources to multiple tenants. To this end, we conduct analysis on the ineffectiveness of proportionally sharing BDFS IO resource via containerization. This ineffectiveness is due to the scheduler of containerization in “pseudo-starvation” status, in which most of IO requests are backlogged in BDFS rather than in containerization scheduler. Without enough backlogged IO requests, existing schedulers might have to maximize device utilization rather than enforce proportional sharing policy. To resolve this ineffectiveness issue, we develop a cross-layer system called BDFS-Container , which containerizes BDFS at the Linux block IO level. Central to BDFS-Container, we propose and design a proactive IOPS throttling-based mechanism named IOPS Regulator , which achieves a trade-off between maximizing IO utilization and accurately proportional IO sharing. The evaluation results show that our method can improve proportionally sharing BDFS IO resources by 74.4 percent on average.

WITHDRAWN: Comparative Research on Active Learning of Big Aata based on Mapreduce and Spark

Abstract The big data era is coming, and it is changing itself in science, engineering, medicine, healthcare, finance, business, and ultimately in our society. Traditional data may not be able to handle large amounts of analytical data. Big data can appear in three formats: structured, unstructured, and semi-structured. Map Reduce and Spark are the two most popular open-source important frameworks for large-scale data analysis. The performance of Map Reduce and Spark will vary depending on the application being implemented. The Map Reduce program model is a Hadoop configuration used to store large data formats stored in the Hadoop File System (HDFS) in big data. This is an integral part of the Hadoop framework that contains its core parts. Map Reduce has failed those programs for real-time data processing as it is designed to do volume processing on large volumes of data. Apache Spark is a data processing framework that can quickly handle tasks on large datasets, and whether on its own or in series with other distributed computing tools, across multiple computers and data processing tasks can be distributed. Apache Spark is a data processing framework that can quickly handle tasks on large datasets. Whether on its own or in series with other distributed computing tools, across multiple computers and data processing tasks can be distributed. The Map Reduce is two times larger than the spark. Thus, the Spark model quickly deploys and Apache spark, proving that data is more effective than a Map Reduce.

Sandbox security model for Hadoop file system

Extensive usage of Internet based applications in day to day life has led to generation of huge amounts of data every minute. Apart from humans, data is generated by machines like sensors, satellite, CCTV etc. This huge collection of heterogeneous data is often referred as Big Data which can be processed to draw useful insights. Apache Hadoop has emerged has widely used open source software framework for Big Data Processing and it is a cluster of cooperative computers enabling distributed parallel processing. Hadoop Distributed File System is used to store data blocks replicated and spanned across different nodes. HDFS uses an AES based cryptographic techniques at block level which is transparent and end to end in nature. However cryptography provides security from unauthorized access to the data blocks, but a legitimate user can still harm the data. One such example was execution of malicious map reduce jar files by legitimate user which can harm the data in the HDFS. We developed a mechanism where every map reduce jar will be tested by our sandbox security to ensure the jar is not malicious and suspicious jar files are not allowed to process the data in the HDFS. This feature is not present in the existing Apache Hadoop framework and our work is made available in github for consideration and inclusion in the future versions of Apache Hadoop.

Automated Deployment of a Spark Cluster with Machine Learning Algorithm Integration

Abstract The vast amount of data stored nowadays has turned big data analytics into a very trendy research field. The Spark distributed computing platform has emerged as a dominant and widely used paradigm for cluster deployment and big data analytics. However, to get started up is still a task that may take much time when manually done, due to the requisites that all nodes must fulfill. This work introduces LadonSpark, an open-source and non-commercial solution to configure and deploy a Spark cluster automatically. It has been specially designed for easy and efficient management of a Spark cluster with a friendly graphical user interface to automate the deployment of a cluster and to start up the distributed file system of Hadoop quickly. Moreover, LadonSpark includes the functionality of integrating any algorithm into the system. That is, the user only needs to provide the executable file and the number of required inputs for proper parametrization. Source codes developed in Scala, R, Python, or Java can be supported on LadonSpark. Besides, clustering, regression, classification, and association rules algorithms are already integrated so that users can test its usability from its initial installation.

Loading, searching and retrieving data from local data nodes on HDFS

The loading and searching the data from data with in local data nodes by using the Hadoop environment. In general the loading and searching data by using a query is more complex, because the capacity of the dataset may large. We propose a technique to handle the data in local nodes without overlapping and data retrieved by script. The main task of the query is to store the information on distributed environment and searching the without any delay. Here we define the script to avoid the redundancy of the duplicate while searching and loading the data in dynamic mechanism. And also provide the Hadoop file system in distributed environment. The apache script is used to loading and searching the information instead of the SQL mechanism. We improve the performance of query execution and graph theory. The query can split into three parts to search the data individually and combined the results in execution. Here we used the replica concept to store the data at time of executing query in Hadoop file system. The script is executed on the locating environment of Hadoop file system.

Research on Online Monitoring Data Storage of Intelligent Substation Based on Hadoop

With the development of intelligent substation information integration, power monitoring data has grown exponentially. The characteristics and data monitoring mechanism of intelligent substation monitoring data are studied. Combined with Hadoop cloud computing technology, the reliable storage and online query method of online monitoring data of intelligent substation based on Hadoop is studied. The substation online monitoring and collection of massive power equipment data is redundantly stored in the distributed file system of Hadoop, and the index table structure of the online monitoring data is optimized and stored in a distributed database to realize rapid query of massive monitoring data. A Hadoop-based online monitoring data platform is established to conduct benchmark tests, sequencing tests, and online monitoring data read and write performance tests to meet the reliable storage and high-speed processing needs of large-scale online monitoring data of smart grid substations.

Several Adaptive Replica Synchronization Approaches for Distributed file System

This survey paper explained the different approaches of synchronization of replicas of files placed on distributed systems. The survey tells some older and latest techniques of synchronization. Some techniques are by the interference of metadata servers and some are without any intrusion of MDS. In former technique SS storage servers are used for synchronization among replicas. To maximize the performance, scalability and reliability CEPH is a distributed file system. It makes distinction between meta data and data management by object storage file system run on object file systems. Excellent I/O and metadata management is done on CEPH. Commodity servers and disks are used for multitier distributed systems. Performance reliability, I/O rate, workload in writes operations and less overhead in synchronization are the main focus while synchronization of replicas. Hadoop and Google file system are the distributive file systems. Hadoop ensures the better input and output performance with minimal synchronization in replicas, data intensive applications and provides fault tolerance. Some strategies are used for data intensive applications. Parallel file system is type of distributed file system. Analysis enforces the best performance on small and large input output requests. Pattern direct and layout replication technique is one of the most optimized techniques for parallel file system. Data access performance, reliability, data consistency, centralized synchronization, less workload, less overhead is the main focus of all the techniques. Some other file systems like SOFA and frangipani do focus on data consistency and reduce of bandwidth.