Hadoop Research Articles

GENERALIZED DATASET OF GEOLOGICAL AND GEOPHYSICAL INFORMATION ON THE EASTERN SECTOR OF THE RUSSIAN ARCTIC FOR MACHINE LEARNING – BASED ANALYSIS

The article presents a practical approach to the geological and geophysical spatial data collection and preliminary processing to use in machine learning models for geophysical applications. According to the established principles for estimating efforts in data analysis, which are confirmed by the results of surveys among specialists, this stage is viewed as major time and resource-consuming, amounting up to 80% in total volume of data analysis for a hypothesis testing project. The paper focuses on creating a consistent data set that integrates geological and geophysical information on a given region. We consider problems of different sources in the geodata representation to be related to their format (vector/raster), scale, type of attribute information (quantitative/qualitative) and their availability. The algorithm formalization and synthesis for combining geospatial data and converting them into quantitative vectors is a critical aspect. Combining various data draws on the concept of neighborhood fitting in with the data selection techniques and data consolidation strategy. The paper presents the general architecture of the software and hardware complex which includes a module for data collection and transformation in Python using the Pandas library, a data storage system based on the PostgreSQL DBMS (Database Management System) with the PostGIS extension. It is shown that for the considered class of problems in geophysics, it is sufficient to use a relational DBMS for data storing and processing. If the problem dimension increases, it is proposed to use the Big Data technology based on Apache Hadoop for scaling the system. A practical application of the proposed approach is demonstrated as results of data collection for the Caucasus region and eastern sector of the Russian Arctic. Based on the prepared data, experiments were carried out using machine learning models for recognition of locations of potential strong earthquakes and for sensitivity estimation of several geophysical features of these regions. The article presents the experimental results and evaluation of their efficiency.

Parallel power load abnormalities detection using fast density peak clustering with a hybrid canopy-K-means algorithm

Parallel power loads anomalies are processed by a fast-density peak clustering technique that capitalizes on the hybrid strengths of Canopy and K-means algorithms all within Apache Mahout’s distributed machine-learning environment. The study taps into Apache Hadoop’s robust tools for data storage and processing, including HDFS and MapReduce, to effectively manage and analyze big data challenges. The preprocessing phase utilizes Canopy clustering to expedite the initial partitioning of data points, which are subsequently refined by K-means to enhance clustering performance. Experimental results confirm that incorporating the Canopy as an initial step markedly reduces the computational effort to process the vast quantity of parallel power load abnormalities. The Canopy clustering approach, enabled by distributed machine learning through Apache Mahout, is utilized as a preprocessing step within the K-means clustering technique. The hybrid algorithm was implemented to minimise the length of time needed to address the massive scale of the detected parallel power load abnormalities. Data vectors are generated based on the time needed, sequential and parallel candidate feature data are obtained, and the data rate is combined. After classifying the time set using the canopy with the K-means algorithm and the vector representation weighted by factors, the clustering impact is assessed using purity, precision, recall, and F value. The results showed that using canopy as a preprocessing step cut the time it proceeds to deal with the significant number of power load abnormalities found in parallel using a fast density peak dataset and the time it proceeds for the k-means algorithm to run. Additionally, tests demonstrate that combining canopy and the K-means algorithm to analyze data performs consistently and dependably on the Hadoop platform and has a clustering result that offers a scalable and effective solution for power system monitoring.

Evading Cyber-Attacks on Hadoop Ecosystem: A Novel Machine Learning-Based Security-Centric Approach towards Big Data Cloud

The growing industry and its complex and large information sets require Big Data (BD) technology and its open-source frameworks (Apache Hadoop) to (1) collect, (2) analyze, and (3) process the information. This information usually ranges in size from gigabytes to petabytes of data. However, processing this data involves web consoles and communication channels which are prone to intrusion from hackers. To resolve this issue, a novel machine learning (ML)-based security-centric approach has been proposed to evade cyber-attacks on the Hadoop ecosystem while considering the complexity of Big Data in Cloud (BDC). An Apache Hadoop-based management interface “Ambari” was implemented to address the variation and distinguish between attacks and activities. The analyzed experimental results show that the proposed scheme effectively (1) blocked the interface communication and retrieved the performance measured data from (2) the Ambari-based virtual machine (VM) and (3) BDC hypervisor. Moreover, the proposed architecture was able to provide a reduction in false alarms as well as cyber-attack detection.

Implementing Privacy Preserving Techniques in Distributed Big Data Framework on Cloud Computing

Cloud-based deployment architectures have emerged as the preferred model for Big Data operations due to their scalability, flexibility, and cost-effectiveness. However, this shift raises new security concerns as data is no longer directly under the user's control. Securing Big Data in cloud environments is crucial for widespread adoption, but developing a comprehensive security plan is challenging without a thorough analysis of potential vulnerabilities.To address this, a novel security-by-design framework for Big Data deployment on cloud computing, termed Big Cloud, is proposed in this article. The framework leverages systematic security analysis methodologies and automated assessment tools to map domain-specific security knowledge to design best practices. Validation of the framework was conducted through an Apache Hadoop stack use case, demonstrating its effectiveness in enhancing security awareness and reducing design time.The study also evaluates the framework's strengths and limitations, identifying key challenges in the Big Cloud domain.

A method to enhance Apache Spark performance based on data segmentation and configuration parameters settings

When using modern big data processing tools, there is a problem of increasing the productivity of using modern frameworks in the context of effective setting of various configuration parameters. The object of the research is computational processes of processing big data with the use of technologies of high-performance frameworks. The subject is methods and approaches to the effective setting of configuration parameters of frameworks in the conditions of limitations of virtualization environments and local resources. The purpose of the study is to improve the performance of Apache Spark and Apache Hadoop deployment modes based on a combined approach that includes preprocess segmentation of input data and setting of basic and additional configuration parameters that take into account the limitations of the virtual environment and local resources. Achieving the set goal involves the following tasks: create a synthesized set of WordCount test data for using input data segmentation methods. Determine the composition of general and specific Apache Spark and Apache Hadoop configuration parameters that most affect the performance of frameworks in Spark Standalone and Hadoop Yarn (FIFO) deployment modes. Justify changes in the values of the configuration parameters (accepted by default) by setting the level of parallelism, the number of partitions of the input file according to the number of processor cores, the number of tasks assigned to each core and the system executor. Conduct experimental research to substantiate theoretical results and prove their use in practice. Methods. The research used the following methods: statistical analysis; a method of generating test data based on defined segmentation characteristics with arbitrary volumes of data; a systematic approach for comprehensive evaluation and analysis of performance of frameworks based on selected configuration parameters. The results. On the basis of the developed system of parameters for evaluating the performance of the studied frameworks, experiments were carried out, which include: the application of the method of segmentation of input data based on the division of the input file into paragraphs (lines) for different values of the ranges of the number of words and the number of letters in each word; setting the main parameters and specific ones, in particular, partitioning and parallelism, taking into account the characteristics of the virtual environment and the local resource. According to the obtained results, a detailed analysis of the use of the proposed methods to improve the performance of the studied frameworks with recommendations for choosing the optimal values of data segmentation parameters and configuration parameters was carried out. You are snowmen. The obtained results of the experiments allow us to conclude that the use of the proposed methods of setting the configuration parameters of Spark and Hadoop will increase the processing productivity: for small files (0.5–1 GB) on average up to 25–30%, for large ones (1.5–2.5 GB ) – up to 10–20% on average. At the same time, the average value of the execution time of one task decreased by 10-15% for files of different sizes and with different number of words in a line.

Identification of Influential Nodes in Social Network: Big Data - Hadoop

Software development and associated data is the most critical factor these days. Currently, people are living in an internet world where data and related artifacts are major sets of information these days. The data is correlated with real-world data. The analysis of large datasets was done as part of the experimental analysis. The dataset for online social media like Facebook and Twitter was taken for the identification of influential nodes. The analysis of the dataset provides an overview and observation of the dataset for Facebook or Twitter. Here, in the current activity, an overview of cloud computing and big data technologies are discussed along with effective methods and approaches to resolve the problem statement. Particularly, big data technologies such as Hadoop provided by Apache for processing and analysis of Gigabyte(GB) or petabyte(PB) scale datasets are discussed for processing data in distributed and parallel data fashion. Here, the processing of large datasets is done by big data technology by implementing Apache Hadoop in online social media.

Improving Hadoop MapReduce performance on heterogeneous single board computer clusters

Over the past decade, Apache Hadoop has become a leading framework for big data processing. Single board computer (SBC) clusters, predominantly adopting Raspberry Pi (RPi), have been employed to explore the potential of MapReduce processing in terms of low power and cost because, capital costs aside, power consumption has also become a primary concern in many industries. After building SBC clusters, it is prevalent to consider adding more nodes, particularly newer generation SBCs, to the existing clusters or replacing old (or inactive) nodes with new ones to improve performance, inevitably causing heterogeneous SBC clusters. The Hadoop framework on these heterogeneous SBC clusters creates challenging new problems due to computing resource discrepancies in each node. Native Hadoop does not carefully consider the heterogeneity of the cluster nodes. Consequently, heterogeneous SBC Hadoop clusters result in significant performance variation or, more critically, persistent node failures. This paper proposes a new Hadoop Yet Another Resource Negotiator (YARN) architecture design to improve MapReduce performance on heterogeneous SBC Hadoop clusters with tight computing resources. We newly implement two main scheduling policies on Hadoop YARN based on the correct computing resource information that each SBC node provides: (1) two (master-driven vs. slave-driven) MapReduce task scheduling frameworks to determine more effective processing modes and (2) ApplicationMaster (AM) and reduce task distribution mechanisms to provide the best Hadoop performance by minimizing performance variation. Thus, the proposed Hadoop framework makes the best use of the performance-frugal SBC Hadoop cluster by intelligently distributing MapReduce tasks to each node. To our knowledge, the proposed framework is the first redesigned Hadoop YARN architecture to address various challenging problems particularly on heterogeneous SBC Hadoop clusters for big data processing. The extensive experiments with Hadoop benchmarks demonstrate that the redesigned framework performs better performance than the native Hadoop by an average of 2.55× and 1.55× under I/O intensive and CPU-intensive workloads, respectively.

Towards automatic labeling of exception handling bugs: A case study of 10 years bug-fixing in Apache Hadoop

Towards automatic labeling of exception handling bugs: A case study of 10 years bug-fixing in Apache Hadoop

A Metaheuristic Framework with Experience Reuse for Dynamic Multi-Objective Big Data Optimization

Dynamic multi-objective big data optimization problems (DMBDOPs) are challenging because of the difficulty of dealing with large-scale decision variables and continuous problem changes. In contrast to classical multi-objective optimization problems, DMBDOPs are still not intensively explored by researchers in the optimization field. At the same time, there is lacking a software framework to provide algorithmic examples to solve DMBDOPs and categorize benchmarks for relevant studies. This paper presents a metaheuristic software framework for DMBDOPs to remedy these issues. The proposed framework has a lightweight architecture and a decoupled design between modules, ensuring that the framework is easy to use and has enough flexibility to be extended and modified. Specifically, the framework now integrates four basic dynamic metaheuristic algorithms, eight test suites of different types of optimization problems, as well as some performance indicators and data visualization tools. In addition, we have proposed an experience reuse method, speeding up the algorithm’s convergence. Moreover, we have implemented parallel computing with Apache Spark to enhance computing efficiency. In the experiments, algorithms integrated into the framework are tested on the test suites for DMBDOPs on an Apache Hadoop cluster with three nodes. The experience reuse method is compared to two restart strategies for dynamic metaheuristics.

A distributed data processing scheme based on Hadoop for synchrotron radiation experiments.

With the development of synchrotron radiation sources and high-frame-rate detectors, the amount of experimental data collected at synchrotron radiation beamlines has increased exponentially. As a result, data processing for synchrotron radiation experiments has entered the era of big data. It is becoming increasingly important for beamlines to have the capability to process large-scale data in parallel to keep up with the rapid growth of data. Currently, there is no set of data processing solutions based on the big data technology framework for beamlines. Apache Hadoop is a widely used distributed system architecture for solving the problem of massive data storage and computation. This paper presents a set of distributed data processing schemes for beamlines with experimental data using Hadoop. The Hadoop Distributed File System is utilized as the distributed file storage system, and Hadoop YARN serves as the resource scheduler for the distributed computing cluster. A distributed data processing pipeline that can carry out massively parallel computation is designed and developed using Hadoop Spark. The entire data processing platform adopts a distributed microservice architecture, which makes the system easy to expand, reduces module coupling and improves reliability.

Construction of large scale English teaching corpus and e-learning system based on Apache Hadoop algorithm

Currently, the contradiction between traditional English teaching and the need for dynamic learning is becoming increasingly prominent. Therefore, the organic integration of new teaching aids with traditional English teaching is conducive to achieving a new concept design and innovation in English teaching. On this basis, a large-scale English teaching corpus was established by combining MapReduce with Apache Hadoop algorithm. At the same time, information theory and methods based on the MapReduce computing model are proposed and applied to the parallelism analysis of big data, effectively solving practical problems that are difficult to expand certain datasets. Secondly, this article will design a new algorithm, Salsa20, based on the construction characteristics of Apache Hadoop. By utilizing the parallel characteristics of this algorithm, we can parallelize the Apache Hadoop algorithm, improve the system's data processing ability, ensure the security of data in MapReduce operation mode, and ensure its efficient data processing ability. In addition, the comprehensive and systematic use of large-scale corpora to establish an English teaching system based on them can effectively improve some of the problems that arise in English teaching, greatly enhance the teaching effectiveness of English courses.

KS-SDN-DDoS: A Kafka streams-based real-time DDoS attack classification approach for SDN environment

Software-Defined Networking (SDN) is a modern networking architecture that segregates control logic from data plane and supports a loosely coupled architecture. It provides flexibility in this advanced networking paradigm for any changes. Further, it controls the complete network in a centralized using controller(s). However, it comes with several security issues: Exhausting bandwidth and flow tables, Distributed Denial of Service (DDoS) attacks, etc. DDoS is a powerful attack for Internet-based applications and services, traditional and SDN paradigms. In the case of the SDN environment, attackers frequently target the central controller(s). This paper proposes a Kafka Streams-based real-time DDoS attacks classification approach for the SDN environment, named KS-SDN-DDoS. The KS-SDN-DDoS has been designed using highly scalable H2O ML techniques on the two-node Apache Hadoop Cluster (AHC). It consists of two modules: (i) Network Traffic Capture (NTCapture) and (ii) Attack Detection and Traffic Classification (ADTClassification). The NTCapture is deployed on the two nodes Apache Kafka Streams Cluster (AKSC-1). It captures incoming network traffic, extracts and formulates attributes, and publishes significant network traffic attributes on the Kafka topic. The ADTClassification is deployed on the two nodes Apache Kafka Streams Cluster (AKSC-2). It consumes network flows from the Kafka topic, classifies it based on the ten attributes, and publishes it to the decision Kafka topic. Further, it saves attributes with outcome to the Hadoop Distributed File System (HDFS). The KS-SDN-DDoS approach is designed and validated using the recent “DDoS Attack SDN dataset”. The result shows that the proposed system gives better classification accuracy (100%).

Big Data Analytic Tools Usage among Academic Libraries in Tanzania

This study examines the implementation of Big Data analytics in Tanzanian University libraries. Employing a qualitative approach within the interpretive paradigm, the study conducted interviews to collect data from 12 library staff across various University libraries in Tanzania. The qualitative data collected was analyzed using Atlasi.ti 7, focusing on emerging themes. The findings established that most University libraries in Tanzania utilize social networking sites for marketing their products and services. However, only a few academic libraries employ analytics tools such as Apache Hadoop, MongoDB, Apache Spark, Tableau, Zoho Analytics, Apache Storm, and others due to insufficient skills. The study concludes that library administrators could greatly benefit from integrating Big Data analytics into their daily operations. Recommendations include developing Big Data analytics policies to ensure proper handling of user data and offering training on the use of such analytic tools to librarians to equip them with necessary skills and techniques.

Big Data Analytic Tools Usage among Academic Libraries in Tanzania

This study examines the implementation of Big Data analytics in Tanzanian University libraries. Employing a qualitative approach within the interpretive paradigm, the study conducted interviews to collect data from 12 library staff across various University libraries in Tanzania. The qualitative data collected was analyzed using Atlasi.ti 7, focusing on emerging themes. The findings established that most University libraries in Tanzania utilize social networking sites for marketing their products and services. However, only a few academic libraries employ analytics tools such as Apache Hadoop, MongoDB, Apache Spark, Tableau, Zoho Analytics, Apache Storm, and others due to insufficient skills. The study concludes that library administrators could greatly benefit from integrating Big Data analytics into their daily operations. Recommendations include developing Big Data analytics policies to ensure proper handling of user data and offering training on the use of such analytic tools to librarians to equip them with necessary skills and techniques.

Big Data and Java are integrated with machine learning

This in-depth investigation explores the revolutionary nexus of Java, Big Data, and Machine Learning (ML), clarifying the innovations and synergies that result from their integration. The voyage commences with a summary of the past, following the growth from Java's fundamental function in high-level development to the revolutionary influence of Big Data frameworks such as Apache Hadoop and Apache Spark. The story then delves into the fundamentals of machine learning in Java, highlighting its adaptability, rich library support, and crucial role in building strong ML pipelines. The investigation delves into the transformative power of Big Data frameworks, highlighting the distributed file system of Hadoop and the in-memory processing capabilities of Spark. We observe the significant effects of this convergence on a variety of industries through real-world case studies, from e-commerce personalized suggestions to fraud detection in banking. The insights gained from these implementations highlight how crucial it is to use ML models with ethical considerations, interdisciplinary cooperation, and ongoing learning. The following sections cover the nuances of data preprocessing, including the use of Java in ETL workflows, scalable feature engineering using Big Data frameworks, and data quality assurance via transformation and cleansing. ML model deployment is the major focus, along with an exploration of the Java runtime environment, micro services architecture, and crucial aspects of model robustness monitoring and maintenance. The investigation concludes with a focus on case studies and success stories that demonstrate the real-world effects of this convergence in sectors like e-commerce, finance, and healthcare. These real-world examples highlight the accomplishments of companies like Netflix, Uber, Airbnb, and others and provide insightful information about how well integration works to accomplish a range of business objectives.

Development of a Low-Cost Distributed Computing Pipeline for High-Throughput Cotton Phenotyping.

In this paper, we present the development of a low-cost distributed computing pipeline for cotton plant phenotyping using Raspberry Pi, Hadoop, and deep learning. Specifically, we use a cluster of several Raspberry Pis in a primary-replica distributed architecture using the Apache Hadoop ecosystem and a pre-trained Tiny-YOLOv4 model for cotton bloom detection from our past work. We feed cotton image data collected from a research field in Tifton, GA, into our cluster's distributed file system for robust file access and distributed, parallel processing. We then submit job requests to our cluster from our client to process cotton image data in a distributed and parallel fashion, from pre-processing to bloom detection and spatio-temporal map creation. Additionally, we present a comparison of our four-node cluster performance with centralized, one-, two-, and three-node clusters. This work is the first to develop a distributed computing pipeline for high-throughput cotton phenotyping in field-based agriculture.

A Study on Hadoop and Map Reduce (MR) Architecture

Apache Hadoop is an open-source programming system for dispersed capacity and disseminated preparing of huge informational indexes on PC bunches worked from item equipment. Every one of the modules in Hadoop are outlined with a principal presumption that equipment disappointments are normal and ought to be naturally taken care of by the structure. The center of Apache Hadoop comprises of a capacity part and a preparing part which is otherwise called Hadoop circulated record framework (HDFS) and Map Reduce separately. Hadoop parts records into expansive squares and disseminates them crosswise over hubs in a bunch.

Large-scale recommender systems using Hadoop and collaborative filtering: a comparative study

With the rapid advancements in internet technologies over the past two decades, the amount of information available online has exponentially increased. This data explosion has led to the development of recommender systems, designed to understand individual preferences and provide personalized recommendations for desirable new content. These systems act as helpful guides, assisting users in discovering relevant and appealing information tailored to their specific tastes and interests. This study's primary objective is to assess and contrast the latest methods utilized in recommender systems within a distributed system architecture that relies on Hadoop. Our analysis will focus on collaborative filtering and will be conducted using a large dataset. We have implemented the algorithms using Python and PySpark, enabling the processing of large datasets using Apache Hadoop and Spark. The studied approaches have been implemented on the MovieLens dataset and compared using the following evaluation metrics: RMSE, precision, recall, and F1 score. Their training times have also been compared.

Is cloud computing the future of bioinformatics?

Bioinformatics is defined as the application of computational and analytical tools to collect and interpret biological data. With the advancement in technology and biological research, a vast amount of data is being produced, necessitating large-scale storage and processing capabilities. Sequencing instruments lack sufficient storage capacity. Cloud computing offers an efficient solution to these challenges, with low management costs. Cloud computing involves storing and accessing data “over the Internet,” providing a scalable and flexible system for addressing storage and processing problems. It allows anytime, anywhere access to applications, data sharing, and resources, removing the need for researchers to manage computing clusters. In the field of bioinformatics, cloud computing can provide services such as probe sequencing, quality control, and reporting for next-generation sequencing (NGS) data and other bioinformatics tools. Over the years, cloud computing has become integral to many bioinformatics solutions. Major companies such as Microsoft, Google, and Amazon offer cloud-based platforms, and some cloud-based bioinformatics systems include Apache Hadoop, Dryad, Azure, and BioNimbus. However, concerns have been raised regarding data management, infrastructure, and environmental performance. Overall, cloud computing has the potential to revolutionize bioinformatics, making it more accessible to researchers and professionals. In this research paper, we explore how the introduction of cloud computing has impacted the field of bioinformatics.

Beaf:BD – A Blockchain Enabled Authentication Framework for Big Data

The widespread utilization of Internet-based applications in our daily routines has resulted in enormous amounts of data being generated every minute. This data is not only produced by humans but also by various machines such as sensors, satellites, CCTV, etc. For many organizations, Apache Hadoop is the solution for handling big data. Big data refers to the extensive set of dissimilar data that can be processed to derive meaningful insights. For its security needs, Hadoop relies on trusted third-party security providers such as Kerberos. Kerberos has several security vulnerabilities. The focus of this paper is to eliminate security issues, particularly dictionary attacks and single points of failure, by proposing a model based on blockchain technology and threshold cryptography.In comparison to other existing schemes, the proposed approach offers superior computational overhead and storage requirements while maintaining the system's security level.