Hadoop Ecosystem Research Articles

The Future of Accounting: Efficacy of Big Data on Accountant’s Functions in the Accounting Information Systems

Before the advent of big data, accounting operations were performed mainly using historical and structured data. The emergence, however, introduced the use of big data to enhance and complement functions in the accounting information system, which improves the real-time and effective management of accounting data for varying uses of stakeholders. This study examined the efficacy of big data on accountant’s functions in the accounting information system in Nigeria. The study utilised survey research design method by administering structured Likert scaled questionnaire. This study was rooted in resource-based view theory and agency theory. The targeted population and the sample size was 283, which comprised accountants with experience in information systems. A purposive sampling technique was adopted to determine the respondent. The collected data were analysed using descriptive statistics and regression analysis. The empirical findings revealed that big data proxied with prescriptive analytics, predictive analytics, machine learning, and the Hadoop ecosystem had significant but negative effect on accountant’s functions in accounting information systems in Nigeria. This suggests that although these technologies are relevant and influential, their integration may be challenging or counterproductive to accountants' traditional roles. The negative effect could be due to factors like complexity, the need for new skill sets, inefficiencies in system adaptation, or resistance to change, which may hinder the effectiveness of accountants in fully leveraging these technologies within their work processes. This study concluded that while big data has the potential to significantly enhance accounting information systems, its integration into the accounting profession in Nigeria has been problematic. It was recommended that organizations should invest in comprehensive training programs for accountants, focusing on developing the necessary skills to handle big data tools effectively.

Security issues analysis based on big data in cloud computing

This study will investigate the security problems that are linked with the Hadoop ecosystem, Big Data, Map Reduce, and cloud computing. The objective of this research is to analyze these concerns. Specifically, the primary emphasis is placed on the security concerns that are linked with big data in cloud computing. Big data applications have a lot to offer organizations, companies, and a broad range of industries, no matter how big or little. There have been issues with Hadoop and cloud computing security, therefore we also look at several possible solutions to these difficulties. The field of cloud computing security is experiencing fast growth in several areas, including computer security, network security, information security, and data privacy. Data, apps, and the infrastructure related to them are greatly helped by cloud computing's use of rules, technologies, controls, and big data tools to keep everything safe. Researchers also anticipate that cloud computing, big data, and their associated benefits will represent the most exciting new areas of study.

Evading Cyber-Attacks on Hadoop Ecosystem: A Novel Machine Learning-Based Security-Centric Approach towards Big Data Cloud

The growing industry and its complex and large information sets require Big Data (BD) technology and its open-source frameworks (Apache Hadoop) to (1) collect, (2) analyze, and (3) process the information. This information usually ranges in size from gigabytes to petabytes of data. However, processing this data involves web consoles and communication channels which are prone to intrusion from hackers. To resolve this issue, a novel machine learning (ML)-based security-centric approach has been proposed to evade cyber-attacks on the Hadoop ecosystem while considering the complexity of Big Data in Cloud (BDC). An Apache Hadoop-based management interface “Ambari” was implemented to address the variation and distinguish between attacks and activities. The analyzed experimental results show that the proposed scheme effectively (1) blocked the interface communication and retrieved the performance measured data from (2) the Ambari-based virtual machine (VM) and (3) BDC hypervisor. Moreover, the proposed architecture was able to provide a reduction in false alarms as well as cyber-attack detection.

Predictive modelling of MapReduce job performance in cloud environments using machine learning techniques

Within the Hadoop ecosystem, MapReduce stands as a cornerstone for managing, processing, and mining large-scale datasets. Yet, the absence of efficient solutions for precise estimation of job execution times poses a persistent challenge, impacting task allocation and distribution within Hadoop clusters. In this study, we present a comprehensive machine learning approach for predicting the execution time of MapReduce jobs, encompassing data collection, preprocessing, feature engineering, and model evaluation. Leveraging a rich dataset derived from comprehensive Hadoop MapReduce job traces, we explore the intricate relationship between cluster parameters and job performance. Through a comparative analysis of machine learning models, including linear regression, decision tree, random forest, and gradient-boosted regression trees, we identify the random forest model as the most effective, demonstrating superior predictive accuracy and robustness. Our findings underscore the critical role of features such as data size and resource allocation in determining job performance. With this work, we aim to enhance resource management efficiency and enable more effective utilisation of cloud-based Hadoop clusters for large-scale data processing tasks.

Development of a Low-Cost Distributed Computing Pipeline for High-Throughput Cotton Phenotyping.

In this paper, we present the development of a low-cost distributed computing pipeline for cotton plant phenotyping using Raspberry Pi, Hadoop, and deep learning. Specifically, we use a cluster of several Raspberry Pis in a primary-replica distributed architecture using the Apache Hadoop ecosystem and a pre-trained Tiny-YOLOv4 model for cotton bloom detection from our past work. We feed cotton image data collected from a research field in Tifton, GA, into our cluster's distributed file system for robust file access and distributed, parallel processing. We then submit job requests to our cluster from our client to process cotton image data in a distributed and parallel fashion, from pre-processing to bloom detection and spatio-temporal map creation. Additionally, we present a comparison of our four-node cluster performance with centralized, one-, two-, and three-node clusters. This work is the first to develop a distributed computing pipeline for high-throughput cotton phenotyping in field-based agriculture.

An EFSM-based model for testing security issues in the Hadoop ecosystem

An EFSM-based model for testing security issues in the Hadoop ecosystem

An EFSM-based Model for Testing Security Issues in Hadoop Ecosystem

An EFSM-based Model for Testing Security Issues in Hadoop Ecosystem

Big Data in Medical Field

Large volumes of structured, unstructured, and semi-structured data have been produced in recent years by numerous global institutions; this heterogeneous data is collectively known as big data. Because several sources are known to provide large volumes of heterogeneous data, the health industry sector has to deal with the challenge of managing big data. In the healthcare industry, numerous big-data analytics technologies and approaches have been created to manage these enormous volumes of data. The influence of big data on healthcare, as well as the range of applications and technologies available within the Hadoop ecosystem, will all be covered in this paper. The fragmentation of healthcare data, ethical concerns, usability challenges, security and privacy concerns, and other issues are some of the barriers preventing the successful application of big data in healthcare, according to this research.

Streamlining Stock Price Analysis: Hadoop Ecosystem for Machine Learning Models and Big Data Analytics

The rapid growth of data in various industries has led to the emergence of big data analytics as a vital component for extracting valuable insights and making informed decisions. However, analyzing such massive volumes of data poses significant challenges in terms of storage, processing, and analysis. In this context, the Hadoop ecosystem has gained substantial attention due to its ability to handle large-scale data processing and storage. Additionally, integrating machine learning models within this ecosystem allows for advanced analytics and predictive modeling. This article explores the potential of leveraging the Hadoop ecosystem to enhance big data analytics through the construction of machine learning models and the implementation of efficient data warehousing techniques. The proposed approach of optimizing stock price by constructing machine learning models and data warehousing empowers organizations to derive meaningful insights, optimize data processing, and make data-driven decisions efficiently.

An Empirical Evaluation of Columnar Storage Formats

Columnar storage is a core component of a modern data analytics system. Although many database management systems (DBMSs) have proprietary storage formats, most provide extensive support to open-source storage formats such as Parquet and ORC to facilitate cross-platform data sharing. But these formats were developed over a decade ago, in the early 2010s, for the Hadoop ecosystem. Since then, both the hardware and workload landscapes have changed. In this paper, we revisit the most widely adopted open-source columnar storage formats (Parquet and ORC) with a deep dive into their internals. We designed a benchmark to stress-test the formats' performance and space efficiency under different workload configurations. From our comprehensive evaluation of Parquet and ORC, we identify design decisions advantageous with modern hardware and real-world data distributions. These include using dictionary encoding by default, favoring decoding speed over compression ratio for integer encoding algorithms, making block compression optional, and embedding finer-grained auxiliary data structures. We also point out the inefficiencies in the format designs when handling common machine learning workloads and using GPUs for decoding. Our analysis identified important considerations that may guide future formats to better fit modern technology trends.

Big data: "Navigating the Hadoop ecosystem: Unraveling the potential of big data”

Big data refers to extremely large data sets that can be analyzed to reveal patterns, trends, and associations, particularly those relating to hu-man behavior and interactions. This data is too large and complex to be processed using traditional data processing methods, necessitating the use of specialized software and systems to manage and analyze it. This paper discusses about the Big Data Architecture, Hadoop Ecosystem, HDFS, Map reduce, Yarn, Hive, and many other components of Hadoop ecosystem.

Development of Fault Detection Systems: The Hadoop Ecosystem Implementation

Semiconductor manufacturing relies on a long and complex production line for wafer fabrication. In a modern semiconductor fabrication plant (fab), a wafer manufacturing line is composed of more than 1,000 manufacturing tools and an overhead hoist system for delivering wafer lots among the tools. Novel technologies, such as Internet of Things (IoT) and software engineering technologies, are adopted in semiconductor manufacturing such that large amounts of data can be collected from manufacturing tools. Also, these tools and the overhead hoist system should cooperate well with each other. Due to the data-processing capacity limitation in a current fault detection and classification (FDC) system, the diagnosis efficiency is limited such that some undesired events cannot be detected in time, leading to significant economic loss. With the booming development of big data technology, this work conducts a study on a new FDC framework based on a Hadoop ecosystem to deal with the data-processing limitation and improve diagnosis efficiency. Also, a migration path is presented such that the current FDC system can be smoothly migrated to a Hadoop ecosystem-based one without shutting down a wafer fabrication line. Experimental results show that the proposed FDC framework can run safely and stably.

An Efficient Feature Subset Selection with Fuzzy Wavelet Neural Network for Data Mining in Big Data Environment

Big data refers to the massive quality of data being generated at a drastic speed from various heterogeneous sources namely social media, mobile devices, internet transactions, networked devices, and sensors. Several data mining (DM) and machine learning (ML) models have been presented for the extraction of knowledge from Big Data. Since the big datasets include numerous features, feature selection techniques are essential to eliminate unwanted and unrelated features which degrade the classification efficiency. The adoption of DM tools for big data environments necessitates remodeling the algorithm. In this aspect, this paper presents an intelligent feature subset selection with fuzzy wavelet neural network (FSS-FWNN) for big data classification. The FSS-FWNN technique incorporates Hadoop Ecosystem tool for handling big data in an effectual way. Besides, the FSS-FWNN technique involves three processes namely preprocessing, feature selection, and classification. In addition, quasi-oppositional chicken swarm optimization (QOCSO) technique is employed for the feature selection process and the FWNN technique is applied for the classification process. The design of QOCSO algorithm as an FS technique for big data classification shows the novelty of the work and the feature subset selection process considerably enhances the classification performance. An extensive set of simulations is carried out and the results are reviewed in terms of several evaluation factors in order to analyse the improvement of the FSS-FWNN approach. The experimental findings demonstrated that the FSS-FWNN approach outperformed the most current algorithms.

A Framework for Attribute-Based Access Control in Processing Big Data with Multiple Sensitivities

There is an increasing demand for processing large volumes of unstructured data for a wide variety of applications. However, protection measures for these big data sets are still in their infancy, which could lead to significant security and privacy issues. Attribute-based access control (ABAC) provides a dynamic and flexible solution that is effective for mediating access. We analyzed and implemented a prototype application of ABAC to large dataset processing in Amazon Web Services, using open-source versions of Apache Hadoop, Ranger, and Atlas. The Hadoop ecosystem is one of the most popular frameworks for large dataset processing and storage and is adopted by major cloud service providers. We conducted a rigorous analysis of cybersecurity in implementing ABAC policies in Hadoop, including developing a synthetic dataset of information at multiple sensitivity levels that realistically represents healthcare and connected social media data. We then developed Apache Spark programs that extract, connect, and transform data in a manner representative of a realistic use case. Our result is a framework for securing big data. Applying this framework ensures that serious cybersecurity concerns are addressed. We provide details of our analysis and experimentation code in a GitHub repository for further research by the community.

A MapReduce Based Approach for Secure Batch Satellite Image Encryption

The overarching goal of this research was to examine the state of satellite imagery security in relation to its deteriorating form due to rising demand. The most common approaches to safeguarding satellite images during transmission across transmission networks, which are not protected by standard encryption, are the focus of this investigation. Since satellite imagery can be encrypted both in transit and while stored on a computer’s hard drive, we put the suggested Image Encryption System to the test by applying it to a collection of satellite photos. Concurrently encrypting data and running MapReduce jobs is key to the study methodology employed. This will be carried out in the Hadoop ecosystem, where an innovative method of analysing random numbers for use in Image encryption will be put to the test. The encryption was processed using MapReduce in the Hadoop ecosystem. Images were saved as BMP files with added security metadata. The evaluation of experiments was based on four (4) indicators. It was found that the processing time for batch encryption calculations grew in proportion to the amount of computations. All cluster, map, and reduction processes were put to the test using encrypted images, exposing load balancing difficulties and inefficiencies. Histogram analysis, the basis of an image encryption technique, provides evidence that the encrypted pixel values are consistent. Therefore, compared to other methods, such as a histogram or information entropy, this one is superior. Because of how it was crafted, it can withstand even the most sophisticated attacks without being compromised.

Design and Development of a Big Data Platform for Disease Burden Based on the Spark Engine.

This study attempts to build a big data platform for disease burden that can realize the deep coupling of artificial intelligence and public health. This is a highly open and shared intelligent platform, including big data collection, analysis, and result visualization. Based on data mining theory and technology, the current situation of multisource data on disease burden was analyzed. Putting forward the disease burden big data management model, functional modules, and technical framework, Kafka technology is used to optimize the transmission efficiency of the underlying data. This will be an efficient and highly scalable data analysis platform through embedding embedded Sparkmlib in the Hadoop ecosystem. With the concept of "Internet + medical integration," the overall architecture design of the big data platform for disease burden management was proposed based on the Spark engine and Python language. The main system composition and application scenarios are given at four levels: multisource data collection, data processing, data analysis, and the application layer, according to application scenarios and use requirements. The big data platform of disease burden management helps to promote the multisource convergence of disease burden data and provides a new path for the standardized paradigm of disease burden measurement. Provide methods and ideas for the deep integration of medical big data and the formation of a broader standard paradigm.

Handling Big Microarray Data: A Novel Approach to Design Accurate Fuzzy-Based Medical Expert System

The genes data produced by microarray experiments is complex in terms of dimensions and samples. It consumes a lot of computation power and time when it is processed for a disease analysis while working with an expert system. At the same time, data can help doctors identify a patient’s health condition if it is presented in a meaningful way and processed on time. Several methods have been proposed to reduce the dimensions of medical microarray data and optimize its search space with minimal accuracy loss. However, the discretization of continuous gene-values in the process of dimension reduction is failed to preserve the inherent meaning of genes. Also, ensuring high accuracy and interpretability in the reduction process may result in extra processing time, which is unfavorable for time-critical applications. To overcome these issues, in this paper, we propose a dimension reduction method in conjunction with an expert system optimization approach, while keeping an accuracy-interpretability-speedy tradeoff in mind. To accomplish this, we use a fuzzy rough set on f-information to identify meaningful genes without changing their original values. We propose a conditionally guided particle swarm optimization for faster knowledge acquisition, where the velocity is adjusted based on a predefined update probability, resulting in a faster search. A big data processing architecture is designed using the Hadoop ecosystem along with a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MapReduce</i> -equivalent algorithm of the proposed method for speedy processing, enabling parallel processing on microarray data to reduce dimensions and perform classification through knowledge extraction. The proposed method is thoroughly tested on eleven microarray datasets by considering accuracy-interpretability-speed tradeoff. The results show that the proposed method is effective in identifying disease-causing genes while also understanding the patient’s genetic profile with only a few operations and a small amount of CPU time. Statistical tests are also run to validate the proposed method’s efficacy in comparison to other methods.

HOW HEALTHCARE INDUSTRY CAN LEVERAGE BIG DATA ANALYTICS TECHNOLOGY AND TOOLS FOR EFFICIENT MANAGEMENT

Recently, huge quantities of organised, unstructured, and semi structured data are being produced by different systems and machines around the world. Such a huge volume of heterogenous data set that typically cannot be managed by traditional system is described as big data. The healthcare sector continues to be confronted by the importance of managing the fundamental information created by numerous sources, known for creating high volumes of heterogeneous information. Different big data analytics programs, as well as strategies, have already been created for managing these substantial quantities of information in the healthcare market. In this particular paper, we go over the effect of big data on healthcare, as well as different methods offered in the Hadoop ecosystem for managing it. We also examine the conceptual structure of big data analytics for healthcare, which involves the information collecting historical past of various limbs, the genome database, electronic health records, text/imagery, and then the medical conclusions support program.

Дослідження даних у промислових data-mining проєктах в епоху генерації великих даних

Introduction. The review material is based mainly on business intelligence (BI) solutions designed for tasks with corporate data. But all the main aspects of working with data discussed in the work are also used on data processing platforms (Data Science Platform). Many BI vendors have expanded the capabilities of their systems to perform more advanced analytics, including Data Science. They added the phrase “Data Science” to their marketing research, and the term “advanced analytics” lost some popularity in relation to corporate data. The Data Science Platform provides a comprehensive set of tools for use by advanced users who traditionally work with data. Capabilities that allow you to connect to multi-structured data across different types of storage platforms, both on-premises and in the cloud, and the infrastructure architecture of a modern BI analytics platform enable high-performance workloads, including business intelligence. It uses distributed architecture, massively parallel processing, data virtualization, in-memory computing, etc. The combination of traditional relational data processing with calculations on the well-known Apache Hadoop software infrastructure, which integrates a number of components of the Hadoop ecosystem (Apache Hive, HBase, Spark, Solr, etc.) with the necessary target functions, allows you to create a fully functional platform for storing and processing structured and non-structures data. Purpose. A review of data processing problems and an analysis of the use of world-class mathematical apparatus and tools for obtaining knowledge from information were carried out. Methods. The paper describes the use of Data Mining methods in big data processing tasks, as well as methods of business, recommendation and predictive analytics. Result. The study suggests that machine learning-enhanced master data management (MDM), data quality, data preparation, and data catalogs will converge into a single, modern Enterprise Information Management (EIM) platform applicable to most new analytics projects. The results of the analysis of the process of identifying useful data can be useful to researchers and developers of modern platforms for processing and researching data in various spheres of society. Conclusion. A review of data processing problems and an analysis of the use of world-class mathematical apparatus and tools for obtaining knowledge from information were carried out. It is shown that a high-quality solution to the problems of working with first-level data indicated in this review will be provided by data research in modern analytical platforms. Successful penetration into their essence at the level of obtaining knowledge using machine learning and artificial intelligence algorithms will make it possible to predict future results in managed objects (processes) and make informed decisions.

THE STUDY OF HADOOP ECOSYSTEM

The Hadoop Ecosystem is a platform or framework that addresses big data issues; it is neither a programming language nor a service. You can think of it as a collection of services that includes consuming, storing, analysing, and managing data. This article defines the concept of big data, which is a collection of enormous data sets that typical computational methods cannot handle. Hadoop is a system created to process large amounts of data. Businesses use Hadoop as their platform for processing big data. In a distributed computing context, Hadoop is an open source, Java-based programming platform that facilitates the processing and archiving of very massive data collections. By solving the challenges that are typically encountered when managing Big Data, it supports Big Data analytics. Hadoop may fail. Keywords: Hadoop, hadoop ecosystem, Distributed Computing, Business Intelligence