Hadoop File Research Articles

Forensic implications of stacked file systems

While file system analysis is a cornerstone of forensic investigations and has been extensively studied, certain file system classes have not yet been thoroughly examined from a forensic perspective. Stacked file systems, which use an underlying file system for data storage instead of a volume, are a prominent example. With the growth of cloud infrastructure and big data, it is increasingly likely that investigators will encounter distributed stacked file systems, such as MooseFS and the Hadoop File System, that employ this architecture. However, current standard models and tools for file system analysis fall short of addressing the complexities of stacked file systems. This paper highlights the forensic challenges and implications associated with stacked file systems, discussing their unique characteristics in the context of forensic analyses. We provide insights through three analyses of different stacked file systems, illustrating their operational details and emphasizing the necessity of understanding this file system category during forensic investigations. For this purpose, we present general considerations that must be made when dealing with the analysis of stacked file systems.

Memory aware optimized Hadoop MapReduce model in cloud computing environment

<p>In the last decade, data analysis has become one of the popular tasks due to enormous growth in data every minute through different applications and instruments. MapReduce is the most popular programming model for data processing. Hadoop constitutes two basic models i.e., Hadoop file system (HDFS) and MapReduce, Hadoop is used for processing a huge amount of data whereas MapReduce is used for data processing. Hadoop MapReduce is one of the best platforms for processing huge data in an efficient manner such as processing web logs data. However, existing model This research work proposes memory aware optimized Hadoop MapReduce (MA-OHMR). MA-OHMR is developed considering memory as the constraint and prioritizes memory allocation and revocation in mapping, shuffling, and reducing, this further enhances the job of mapping and reducing. Optimal memory management and I/O operation are carried out to use the resource inefficiently manner. The model utilizes the global memory management to avoid garbage collection and MA-OHMR is optimized on the makespan front to reduce the same. MA-OHMR is evaluated considering two datasets i.e., simple workload of Wikipedia dataset and complex workload of sensor dataset considering makespan and cost as an evaluation parameter.</p>

Generative Adversarial Networks (GAN) and HDFS-Based Realtime Traffic Forecasting System Using CCTV Surveillance

The most crucial component of any smart city traffic management system is traffic flow prediction. It can assist a driver in selecting the most efficient route to their destination. The digitalization of closed-circuit television (CCTV) systems has resulted in more effective and capable surveillance imaging systems for security applications. The number of automobiles on the world’s highways has steadily increased in recent decades. However, road capacity has not developed at the same rate, resulting in significantly increasing congestion. The model learning mechanism cannot be guided or improved by prior domain knowledge of real-world problems. In reality, symmetrical features are common in many real-world research objects. To mitigate this severe situation, the researchers chose adaptive traffic management to make intelligent and efficient use of the current infrastructure. Data grow exponentially and become a complex item that must be managed. Unstructured data are a subset of big data that are difficult to process and have volatile properties. CCTV cameras are used in traffic management to monitor a specific point on the roadway. CCTV generates unstructured data in the form of images and videos. Because of the data’s intricacy, these data are challenging to process. This study proposes using big data analytics to transform real-time unstructured data from CCTV into information that can be shown on a web dashboard. As a Hadoop-based architectural stack that can serve as the ICT backbone for managing unstructured data efficiently, the Hadoop Distributed File System (HDFS) stores several sorts of data using the Hadoop file storage system, a high-performance integrated virtual environment (HIVE) tables, and non-relational storage. Traditional computer vision algorithms are incapable of processing such massive amounts of visual data collected in real-time. However, the inferiority of traffic data and the quality of unit information are always symmetrical phenomena. As a result, there is a need for big data analytics with machine learning, which entails processing and analyzing vast amounts of visual data, such as photographs or videos, to uncover semantic patterns that may be interpreted. As a result, smart cities require a more accurate traffic flow prediction system. In comparison to other recent methods applied to the dataset, the proposed method achieved the highest accuracy of 98.21%. In this study, we look at the construction of a secure CCTV strategy that predicts traffic from CCTV surveillance using real-time traffic prediction analysis with generative adversarial networks (GAN) and HDFS.

AdMap: a framework for advertising using MapReduce pipeline

There is a vast collection of data for consumers due to tremendous development in digital marketing. For their ads or for consumers to validate nearby services which already are upgraded to the dataset systems, consumers are more concerned with the amount of data. Hence there is a void formed between the producer and the client. To fill that void, there is the need for a framework which can facilitate all the needs for query updating of the data. The present systems have some shortcomings by a vast number of information that each time lead to decision tree-based approach. A systematic solution to the automated incorporation of data into a Hadoop distributed file system (HDFS) warehouse (Hadoop file system) includes a data hub server, a generic data charging mechanism and a metadata model. In our model framework, the database would be able to govern the data processing schema. In the future, as a variety of data is archived, the datalake will play a critical role in managing that data. To order to carry out a planned loading function, the setup files immense catalogue move the datahub server together to attach the miscellaneous details dynamically to its schemas.

AdMap: a framework for advertising using MapReduce pipeline

There is a vast collection of data for consumers due to tremendous development in digital marketing. For their ads or for consumers to validate nearby services which already are upgraded to the dataset systems, consumers are more concerned with the amount of data. Hence there is a void formed between the producer and the client. To fill that void, there is the need for a framework which can facilitate all the needs for query updating of the data. The present systems have some shortcomings by a vast number of information that each time lead to decision tree-based approach. A systematic solution to the automated incorporation of data into a Hadoop distributed file system (HDFS) warehouse (Hadoop file system) includes a data hub server, a generic data charging mechanism and a metadata model. In our model framework, the database would be able to govern the data processing schema. In the future, as a variety of data is archived, the datalake will play a critical role in managing that data. To order to carry out a planned loading function, the setup files immense catalogue move the datahub server together to attach the miscellaneous details dynamically to its schemas.

Contributions to Hadoop File System Architecture by Revising the File System Usage Along with Automatic Service

The usage of unstructured data is becoming obvious by companies and social media is raised heavily from past decade. The sharing of images, audio, and video content by the individual user and corporate can be observed everywhere. The current work focused on the Hadoop framework revision contributions so as to improve the performance of the eco system in the context of space and time parameters. The architecture basically provides the usage of Hadoop Distributed File System (HDFS) and MapReduce (MR). We are proposing certain revision contributions so that the process of importing and processing of the tasks can get the benefit of time and space usage in the effective and efficient manner. The work provides the service running in two different ways which reduces the time requirements of the cluster management. In the distributed environment, this revision helps in the reduction of waiting time for the start of the service. The other context we have focused on, the local file system handler in the data storage and processing of the data, the provision of using the file system according to the proposed architecture, will handle the CPU context switch while performing the import and export process in the running of the jobs. The outcome of the work is revision architecture to reflect the service initiation by all the machines in the cluster and file system revision approach to minimize the CPU context switch, while performing the storage and processing relevant aspects of the Hadoop cluster.

Using HDFS to Load, Search, and Retrieve Data from Local Data Nodes

Abstract: By utilizing the Hadoop environment, data may be loaded and searched from local data nodes. Because the dataset's capacity may be vast, loading and finding data using a query is often more difficult. We suggest a method for dealing with data in local nodes that does not overlap with data acquired by script. The query's major purpose is to store information in a distributed environment and look for it quickly. In this section, we define the script to eliminate duplicate data redundancy when searching and loading data in a dynamic manner. In addition, the Hadoop file system is available in a distributed environment. Keywords: HDFS; Hadoop distributed file system; replica; local; distributed; capacity; SQL; redundancy

An Intelligent Approach to Resource Allocation on Heterogeneous Cloud Infrastructures

Cloud computing systems are rapidly evolving toward multicloud architectures supported on heterogeneous hardware. Cloud service providers are widely offering different types of storage infrastructures and multi-NUMA architecture servers. Existing cloud resource allocation solutions do not comprehensively consider this heterogeneous infrastructure. In this study, we present a novel approach comprised of a hierarchical framework based on genetic programming to solve problems related to data placement and virtual machine allocation for analytics applications running on heterogeneous hardware with a variety of storage types and nonuniform memory access. Our approach optimizes data placement using the Hadoop File System on heterogeneous storage devices on multicloud systems. It guarantees the efficient allocation of virtual machines on physical machines with multiple NUMA (nonuniform memory access) domains by minimizing contention between workloads. We prove that our solutions for data placement and virtual machine allocation outperform other state-of-the-art approaches.

FASTA/Q data compressors for MapReduce-Hadoop genomics: space and time savings made easy

BackgroundStorage of genomic data is a major cost for the Life Sciences, effectively addressed via specialized data compression methods. For the same reasons of abundance in data production, the use of Big Data technologies is seen as the future for genomic data storage and processing, with MapReduce-Hadoop as leaders. Somewhat surprisingly, none of the specialized FASTA/Q compressors is available within Hadoop. Indeed, their deployment there is not exactly immediate. Such a State of the Art is problematic.ResultsWe provide major advances in two different directions. Methodologically, we propose two general methods, with the corresponding software, that make very easy to deploy a specialized FASTA/Q compressor within MapReduce-Hadoop for processing files stored on the distributed Hadoop File System, with very little knowledge of Hadoop. Practically, we provide evidence that the deployment of those specialized compressors within Hadoop, not available so far, results in better space savings, and even in better execution times over compressed data, with respect to the use of generic compressors available in Hadoop, in particular for FASTQ files. Finally, we observe that these results hold also for the Apache Spark framework, when used to process FASTA/Q files stored on the Hadoop File System.ConclusionsOur Methods and the corresponding software substantially contribute to achieve space and time savings for the storage and processing of FASTA/Q files in Hadoop and Spark. Being our approach general, it is very likely that it can be applied also to FASTA/Q compression methods that will appear in the future.AvailabilityThe software and the datasets are available at https://github.com/fpalini/fastdoopc

Replica-aware data recovery performance improvement for Hadoop system with NVM

The non-volatile memory (NVM) is the promising device to store data and accelerate big data analysis due to its excellent I/O performance. However, we find that simply replacing hard disk drive (HDD) with NVM cannot bring the expected performance improvement. In this paper, we take the data recovery issue in Hadoop file system (HDFS) as a case study to investigate how to take advantage of the performance of NVM. We analyze the data recovery mechanism in HDFS and find that the configuration of replication tasks in the DataNode can affect the data recovery significantly. We conduct extensive analysis and experiments tuning the configuration and also get some interesting findings. With the new configuration, we increase the data recovery performance from 17 to 71%. We can also improve the execution performance of MapReduce jobs from 28 to 59% through optimized configuration. We also find that the sudden data recovery brings disordered network resource competition, which reduces the performance of MapReduce jobs. Hence, We present a priority-aware multi-stage data recovery method. This improves the performance by 32.5% in addition for the MapReduce jobs.

Analysis of Big Data Using Two Mapper Files in Hadoop

Nowadays, a large amount of valuable uncertain data is easily available in many real-life applications. Many industries and government organizations can exploit this data to extract valuable information. This information can help the managers to enhance their strategies and optimize their plans in making decisions. In fact, various private companies and governments have launched programs with investments and funds in order to maximize profits and optimize resources. This vast amount of data is called big data. The analysis of big data is important for future growth. This paper depicts big data analytics through experimental results. In this paper, data for New York stock exchange has been analyzed using two mapper files in Hadoop. For each year, the calculation of maximum and minimum price of every stock exchange and the average stock price is done.

Acquiring and storing Internet of Things data into Hadoop

This paper looks at issues, related to the use of Big Data and Internet of Things. Those two modern paradigms are essential in the ongoing digital business transformation of our world. Our society and activities create new products, services and improve the efficiency of the traditional ways of production, business, and life. We use one of the most popular software libraries, operating on big data sets – the Apache Hadoop, and experiment with data, generated by IoT. We investigate the impact on performance by varying parameters of Internet of Things data like number of messages and rates write of those messages in the system. Our experiments use the MQTT protocol and demonstrate that there are ways to achieve better effectiveness by using tools to automate data streams, and optimization of data sizes than direct store of data in the Hadoop file system.

An Enhanced Multiple Linear Regression Model for Seasonal Rainfall Prediction

Aims & Background: India is a country which has exemplary climate circumstances comprising of different seasons and topographical conditions like high temperatures, cold atmosphere, and drought, heavy rainfall seasonal wise. These utmost varieties in climate make us exact weather prediction is a challenging task. Majority people of the country depend on agriculture. Farmers require climate information to decide the planting. Weather prediction turns into an orientation in farming sector to deciding the start of the planting season and furthermore quality and amount of their harvesting. One of the variables are influencing agriculture is rainfall. Objectives & Methods: The main goal of this project is early and proper rainfall forecasting, that helpful to people who live in regions which are inclined natural calamities such as floods and it helps agriculturists for decision making in their crop and water management using big data analytics which produces high in terms of profit and production for farmers. In this project, we proposed an advanced automated framework called Enhanced Multiple Linear Regression Model (EMLRM) with MapReduce algorithm and Hadoop file system. We used climate data from IMD (Indian Metrological Department, Hyderabad) in 1901 to 2002 period. Results: Our experimental outcomes demonstrate that the proposed model forecasting the rainfall with better accuracy compared with other existing models. Conclusion: The results of the analysis will help the farmers to adopt effective modeling approach by anticipating long-term seasonal rainfall.

Enhancing Proportional IO Sharing on Containerized Big Data File Systems

Big Data platforms recently employ resource management systems, such as YARN, Mesos, and Google Borg, to provision computational resources. These systems adopt containerization to share the computing resources in a multi-tenant setting with low performance overhead and interference. However, it may be observed that tenants often interfere with each other on the underlying Big Data File Systems (BDFS), e.g., Hadoop File System, which have been widely deployed as a persistent layer in current data centers. A solution with systematic generality is to containerize BDFS itself to isolate and allocate its IO sources to multiple tenants. To this end, we conduct analysis on the ineffectiveness of proportionally sharing BDFS IO resource via containerization. This ineffectiveness is due to the scheduler of containerization in “pseudo-starvation” status, in which most of IO requests are backlogged in BDFS rather than in containerization scheduler. Without enough backlogged IO requests, existing schedulers might have to maximize device utilization rather than enforce proportional sharing policy. To resolve this ineffectiveness issue, we develop a cross-layer system called BDFS-Container , which containerizes BDFS at the Linux block IO level. Central to BDFS-Container, we propose and design a proactive IOPS throttling-based mechanism named IOPS Regulator , which achieves a trade-off between maximizing IO utilization and accurately proportional IO sharing. The evaluation results show that our method can improve proportionally sharing BDFS IO resources by 74.4 percent on average.

WITHDRAWN: Comparative Research on Active Learning of Big Aata based on Mapreduce and Spark

Abstract The big data era is coming, and it is changing itself in science, engineering, medicine, healthcare, finance, business, and ultimately in our society. Traditional data may not be able to handle large amounts of analytical data. Big data can appear in three formats: structured, unstructured, and semi-structured. Map Reduce and Spark are the two most popular open-source important frameworks for large-scale data analysis. The performance of Map Reduce and Spark will vary depending on the application being implemented. The Map Reduce program model is a Hadoop configuration used to store large data formats stored in the Hadoop File System (HDFS) in big data. This is an integral part of the Hadoop framework that contains its core parts. Map Reduce has failed those programs for real-time data processing as it is designed to do volume processing on large volumes of data. Apache Spark is a data processing framework that can quickly handle tasks on large datasets, and whether on its own or in series with other distributed computing tools, across multiple computers and data processing tasks can be distributed. Apache Spark is a data processing framework that can quickly handle tasks on large datasets. Whether on its own or in series with other distributed computing tools, across multiple computers and data processing tasks can be distributed. The Map Reduce is two times larger than the spark. Thus, the Spark model quickly deploys and Apache spark, proving that data is more effective than a Map Reduce.

Sandbox security model for Hadoop file system

Extensive usage of Internet based applications in day to day life has led to generation of huge amounts of data every minute. Apart from humans, data is generated by machines like sensors, satellite, CCTV etc. This huge collection of heterogeneous data is often referred as Big Data which can be processed to draw useful insights. Apache Hadoop has emerged has widely used open source software framework for Big Data Processing and it is a cluster of cooperative computers enabling distributed parallel processing. Hadoop Distributed File System is used to store data blocks replicated and spanned across different nodes. HDFS uses an AES based cryptographic techniques at block level which is transparent and end to end in nature. However cryptography provides security from unauthorized access to the data blocks, but a legitimate user can still harm the data. One such example was execution of malicious map reduce jar files by legitimate user which can harm the data in the HDFS. We developed a mechanism where every map reduce jar will be tested by our sandbox security to ensure the jar is not malicious and suspicious jar files are not allowed to process the data in the HDFS. This feature is not present in the existing Apache Hadoop framework and our work is made available in github for consideration and inclusion in the future versions of Apache Hadoop.

The File System Recommendations to Reduce the Space and Time Parameters in Hadoop File Storage and Map Reduce Processing of Big Data Applications

The study of Hadoop Distributed File System (HDFS) and Map Reduce (MR) are the key aspects of the Hadoop framework. The big data scenarios like Face Book (FB) data processing or the twitter analytics such as storing the tweets and processing the tweets is other scenario of big data which can depends on Hadoop framework to perform the storage and processing through which further analytics can be done. The point here is the usage of space and time in the processing of the above-mentioned huge amounts of the data definitely leads to higher amounts of space and time consumption of the Hadoop framework. The problem here is usage of huge amounts of the space and at the same time the processing time is also high which need to be reduced so as to get the fastest response from the framework. The attempt is important as all the other eco system tools also depends on HDFS and MR so as to perform the data storage and processing of the data and alternative architecture so as to improve the usage of the space and effective utilization of the resources so as to reduce the time requirements of the framework. The outcome of the work is faster data processing and less space utilization of the framework in the processing of MR along with other eco system tools like Hive, Flume, Sqoop and Pig Latin. The work is proposing an alternative framework of the HDFS and MR and the name we are assigning is Unified Space Allocation and Data Processing with Metadata based Distributed File System (USAMDFS).

An HADOOP File Systems Based Data Analysis of Flood Prediction

One of the most well-known uses of Artificial Intelligence which has noticed an enormous development within the digital era is actually Machine Learning Techniques in which the method scientific studies and also increases the overall performance of its via progressive learning with no explicit programming. It is popular within many programs certainly one of them becoming a weather condition prediction. Image distinction, as well as feature extraction, are regarded as to become essentially the most popularly pre-owned techniques finished utilizing machine mastering procedure. With this proposed method, a hybrid model is developed for predicting rainfall by using feature extraction methods that have been proposed by us. The unit was created in such a manner it fetches a sequence of pictures originating from a data source as well as different info regarding earlier rainfalls wearing a particular region. The pictures are actually pre-processed as well as additional segmented for option extraction. The segmented pictures are then categorized via the Random Forest algorithm in which the sequence of pictures is actually validated frame by frame. The effectiveness of the suggested design is actually evaluated and it is kept in a sent out HADOOP File Systems (HDFS) for faster retrieval of information. It’s found that this suggested model provides greater results. The functionality of this unit tends to be more precise because the unit has an iterative method for characteristic extraction inside classifying pictures. The suggested item is actually incorporated by using an aware process to be able to attain a warning or an alert to the individuals in a space properly prior to a flood really hits.

CSFC: A New Centroid Based Clustering Method to Improve the Efficiency of Storing and Accessing Small Files in Hadoop

In day to day life, the computer plays a major role, due to this advancement of technology collection of data from various fields are increasing. A large amount of data is produced by various fields for every second and is not easy to process. This large amount of data is called as Big data. A large number of small files also considered as Big data. It's not easy to process and store the small files in Hadoop. In the existing methods Merging technologies and Clustering Techniques are used to combine smaller files to large files up to 128 MB before sending it to HDFS in Hadoop. In the Proposed system CSFC (Clustering Small Files based on Centroid) Clustering Technique is used without mentioning the number of Clusters previously because if the clusters are mentioned before, all the files are clubbed within the limited number of clusters. In proposing system clusters are generated by depending on the number of related files in the dataset. The relevant files are combined up to 128 MB in a cluster. If any file is not relevant to the existing cluster or if the memory size reached 128MB then-new cluster will be generated and the file will be stored. It is easy to process the related files, comparing two irrelevant files. By using this method fetching data from the data node, it produces efficient result when comparing with other clustering techniques.

Loading, searching and retrieving data from local data nodes on HDFS

The loading and searching the data from data with in local data nodes by using the Hadoop environment. In general the loading and searching data by using a query is more complex, because the capacity of the dataset may large. We propose a technique to handle the data in local nodes without overlapping and data retrieved by script. The main task of the query is to store the information on distributed environment and searching the without any delay. Here we define the script to avoid the redundancy of the duplicate while searching and loading the data in dynamic mechanism. And also provide the Hadoop file system in distributed environment. The apache script is used to loading and searching the information instead of the SQL mechanism. We improve the performance of query execution and graph theory. The query can split into three parts to search the data individually and combined the results in execution. Here we used the replica concept to store the data at time of executing query in Hadoop file system. The script is executed on the locating environment of Hadoop file system.