Hadoop Framework Research Articles

Scheduling of Big Data Workflows in the Hadoop Framework with Heterogeneous Computing Cluster

Scheduling of Big Data Workflows in the Hadoop Framework with Heterogeneous Computing Cluster

Data-driven diabetes management: a statistical assessment information system leveraging big data

ABSTRACT Amidst the escalating global burden of diabetes and its multifaceted complications, there arises an urgent need for robust healthcare information systems tailored to address the complexities of diabetic patient management. This paper introduces a novel statistical assessment healthcare information system engineered to meticulously analyze diabetes-related data by harnessing the power of big data technologies. Diabetes, affecting millions worldwide, exerts substantial strain on healthcare infrastructures, necessitating innovative solutions to optimize patient outcomes and treatment efficacy. Leveraging advancements in data processing and analytics alongside vast medical datasets, this study pioneers a statistical assessment model explicitly crafted for diabetes analysis within a big data paradigm. The model’s efficacy is rigorously evaluated using key performance metrics such as accuracy and F-measure, within the Hadoop framework. Remarkably, the results demonstrate a significant enhancement in performance compared to conventional methodologies, affirming the model’s potential to revolutionize the accuracy and efficiency of diabetes analysis within a big data milieu. Moving forward, further research avenues include refining the model’s algorithms to accommodate diverse data sources and expanding its applicability to encompass a broader spectrum of diabetes-related factors.

Improving Hadoop MapReduce performance on heterogeneous single board computer clusters

Over the past decade, Apache Hadoop has become a leading framework for big data processing. Single board computer (SBC) clusters, predominantly adopting Raspberry Pi (RPi), have been employed to explore the potential of MapReduce processing in terms of low power and cost because, capital costs aside, power consumption has also become a primary concern in many industries. After building SBC clusters, it is prevalent to consider adding more nodes, particularly newer generation SBCs, to the existing clusters or replacing old (or inactive) nodes with new ones to improve performance, inevitably causing heterogeneous SBC clusters. The Hadoop framework on these heterogeneous SBC clusters creates challenging new problems due to computing resource discrepancies in each node. Native Hadoop does not carefully consider the heterogeneity of the cluster nodes. Consequently, heterogeneous SBC Hadoop clusters result in significant performance variation or, more critically, persistent node failures. This paper proposes a new Hadoop Yet Another Resource Negotiator (YARN) architecture design to improve MapReduce performance on heterogeneous SBC Hadoop clusters with tight computing resources. We newly implement two main scheduling policies on Hadoop YARN based on the correct computing resource information that each SBC node provides: (1) two (master-driven vs. slave-driven) MapReduce task scheduling frameworks to determine more effective processing modes and (2) ApplicationMaster (AM) and reduce task distribution mechanisms to provide the best Hadoop performance by minimizing performance variation. Thus, the proposed Hadoop framework makes the best use of the performance-frugal SBC Hadoop cluster by intelligently distributing MapReduce tasks to each node. To our knowledge, the proposed framework is the first redesigned Hadoop YARN architecture to address various challenging problems particularly on heterogeneous SBC Hadoop clusters for big data processing. The extensive experiments with Hadoop benchmarks demonstrate that the redesigned framework performs better performance than the native Hadoop by an average of 2.55× and 1.55× under I/O intensive and CPU-intensive workloads, respectively.

Assessment of soil fertility in Xinjiang oasis cotton field based on big data techniques

Assessing soil fertility through traditional methods has faced challenges due to the vast amount of meteorological data and the complexity of heterogeneous data. In this study, we address these challenges by employing the K-means algorithm for cluster analysis on soil fertility data and developing a novel K-means algorithm within the Hadoop framework. Our research aims to provide a comprehensive analysis of soil fertility in the Shihezi region, particularly in the context of oasis cotton fields, leveraging big data techniques. The methodology involves utilizing soil nutrient data from 29 sampling points across six round fields in 2022. Through K-means clustering with varying K values, we determine that setting K to 3 yields optimal cluster effects, aligning closely with the actual soil fertility distribution. Furthermore, we compare the performance of our proposed K-means algorithm under the MapReduce framework with traditional serial K-means algorithms, demonstrating significant improvements in operational speed and successful completion of large-scale data computations. Our findings reveal that soil fertility in the Shihezi region can be classified into four distinct grades, providing valuable insights for agricultural practices and land management strategies. This classification contributes to a better understanding of soil resources in oasis cotton fields and facilitates informed decision-making processes for farmers and policymakers alike.

Comparative study on satellite based image encryption methods: a survey

The availability of high-resolution satellite images increases with advancements in remote sensing technology. These satellite images are used in various earth observation applications such as disaster management, military applications, weather forecasting, land use and cover, and many more. Satellite images have large volumes stored in memory devices. These satellite images are transmitted to the ground station for processing and analysis. In these cases, images are vulnerable to privacy issues. As technology advances, onboard processing of satellite images using intelligent systems processes the images faster. A model such as field programmable gate arrays (FPGA) is used in onboard processing to process satellite images. However, images are susceptible to faults induced by harsh radiation environments in space. Encryption is one of the most assured methods to provide privacy to satellite images. Hence, encryption of satellite images during processing, storage, and transmission is the present rising demand. There are various encryption methods implemented using algorithms such as advanced encryption standard (AES), homomorphic, advanced encryption standard-counter (AES-CTR), and chaotic maps. Concurrent processing and encryption of images using MapReduce with Hadoop Framework perform the task faster. The focus of this paper is a comparative study of the various encryption methods used in recent years.

Survey on MapReduce Scheduler Algorithms in Hadoop Framework

Hadoop serves as a robust framework tailored for the storage and processing of vast data volumes across clusters. Its foundation lies in the Hadoop Distributed File System (HDFS) for data storage, complemented by the MapReduce paradigm for data processing. MapReduce, functioning as a parallel computing framework, operates adeptly on distributed clusters to manage large-scale data processing tasks. Within this framework, scheduling emerges as a pivotal aspect, influencing the overall system's performance and efficiency. The essence of scheduling in MapReduce lies in enhancing performance, minimizing response times, and optimizing resource allocation. This paper undertakes a systematic exploration of existing scheduling algorithms, offering a fresh classification and detailed examination of each category. Furthermore, the analysis delves into the core principles, objectives, as well as the strengths and weaknesses of these scheduling algorithms.

HE-Gaston algorithm for frequent subgraph mining with hadoop framework

Graph mining contributes a key role in data mining and as the size of the data increases, it becomes complicated. Identifying the interesting subgraphs in the graph is a commonly researched issue, where the subgraphs denote the commonly occurring pattern exhibiting a particular structure. Frequent Subgraph Mining (FSM) is an important task for exploratory data analysis on graph data.Though many techniques are proposed for FSM, the large dimension of the data makes FSM complex. This research proposes a novel technique for performing FSM in a Hadoop framework. Here, FSM is carried out using the proposed Holoentropy Gaston algorithm (HE-Gaston), which is developed by incorporating the Holoentropy support measure instead of the Recurrent support measure in the Recurrent-Gaston (R-Gaston) technique. Here, the weblog files are considered for FSM and are fed to the Spark framework, which encompasses a master and several slaves. The slave nodes generate the frequent subgraphs based on the Holoentropy support measure and the generated frequent subgraphs are applied to the master node which produces the final frequent subgraphs by utilizing an aggregate Holoentropy support measure. Further, the HE-Gaston shows that it recorded execution time of 43 ms, memory of 2.161 MB, and number of structures mined as 117.

Social Media-based Polling Data Processing using MapReduce on the Hadoop Framework

The processing of data obtained from a polling system holds significant importance as the results can serve as a reference for addressing community issues. With the escalating use of social media, Indonesia ranks fifth globally in Twitter usage. When dealing with substantial data sets, processing speed becomes a concern, prompting the author to develop a more time-efficient system for handling polling data gathered through social media. Hadoop emerges as a prime candidate for this purpose due to its two main modules: Hadoop Distributed File System (HDFS) for distributed storage and MapReduce for algorithmic computation. Through experimentation using both the wordcount program with MapReduce on Hadoop and the version without MapReduce, it was observed that MapReduce outperforms the latter in terms of data processing speed. On average, employing MapReduce on Hadoop resulted in a 1.3 times faster data processing rate compared to the method without MapReduce.

Big data clustering using fuzzy based energy efficient clustering and MobileNet V2

Big data analytics (BDA) is a systematic way to analyze and detect various patterns, relationships, and trends in vast amounts of data. Big data analysis and processing require significant effort, techniques, and equipment. The Hadoop framework software uses the MapReduce approach to do large-scale data analysis using parallel processing in order to generate results as soon as possible. Due to the traditional algorithm’s longer execution time and difficulty in processing big amounts of data, this is one of the main issues. Clusters are highly correlated inside each other but are not highly correlated with one another. The technique of effectively allocating limited resources is known as an optimization algorithm for clustering. For processing large amounts of data with several dimensions, the conventional optimization approach is insufficient. By using a fuzzy method, this can be prevented. In this paper, we proposed Fuzzy based energy efficient clustering approach to enhance the clustering mechanism. In summary, Fuzzy based energy efficient clustering introduces a function that measures the distance between the cluster center and the instance, which aids in improved clustering, and we then present the MobileNet V2 model to improve efficiency and speed up computation. To enhance the method’s performance and reduce its time complexity, the distributed database simulates the shared memory space and parallelizes on the MapReduce framework on the Hadoop cloud computing platform. The proposed approach is evaluated using performance metrics such as Accuracy, Precision, Adjusted Rand Index (ARI), Recall, F1-Score, and Normalized Mutual Information (NMI). The experimental findings indicate that the proposed approach outperforms the existing techniques in terms of clustering accuracy.

File Monitoring System based on Hadoop Framework for Image Data Processing

File Monitoring System based on Hadoop Framework for Image Data Processing

Database System Based on 3Dmax Big Data Mining Technology

INTRODUCTION: This project intends to study the mining method of FP-growth frequent items in 3Dmax big data under the Hadoop framework and combined with the Map Reduce development model. Firstly, the transaction database is selected according to the frequency of each transaction and the corresponding projection library is generated. Then the obtained image database is distributed on each node computer. Then, under the guidance of the node machine, the projection is divided into different regions to produce several smaller sub-databases. The method is parallelized by using node machine to generate local frequency items. Finally, all the local frequency sets are merged into one complete frequency set. This method does not need to generate as many FP trees as the regular FP-growth method. This method can overcome the computational failure problem caused by the limited memory of a single computer by the conventional FP-Growth method and other methods. At the same time, because the sublibraries of partitions are similar in size, the load distributed to each node machine is more balanced. The effectiveness of the algorithm is improved.

A Novel Data Management Scheme in Cloud for Micromachines

In cyber-physical systems (CPS), micromachines are typically deployed across a wide range of applications, including smart industry, smart healthcare, and smart cities. Providing on-premises resources for the storage and processing of huge data collected by such CPS applications is crucial. The cloud provides scalable storage and computation resources, typically through a cluster of virtual machines (VMs) with big data tools such as Hadoop MapReduce. In such a distributed environment, job latency and makespan are highly affected by excessive non-local executions due to various heterogeneities (hardware, VM, performance, and workload level). Existing approaches handle one or more of these heterogeneities; however, they do not account for the varying performance of storage disks. In this paper, we propose a prediction-based method for placing data blocks in virtual clusters to minimize the number of non-local executions. This is accomplished by applying a linear regression algorithm to determine the performance of disk storage on each physical machine hosting a virtual cluster. This allows us to place data blocks and execute map tasks where the data blocks are located. Furthermore, map tasks are scheduled based on VM performance to reduce job latency and makespan. We simulated our ideas and compared them with the existing schedulers in the Hadoop framework. The results show that the proposed method improves MapReduce performance in terms of job latency and makespan by minimizing non-local executions compared to other methods taken for evaluation.

Adaptive Weighted Support Vector Machine classification method for privacy preserving in cloud over big data using hadoop framework

Adaptive Weighted Support Vector Machine classification method for privacy preserving in cloud over big data using hadoop framework

HQDCNet: Hybrid Quantum Dilated Convolution Neural Network for detecting covid-19 in the context of Big Data Analytics.

Medical care services are changing to address problems with the development of big data frameworks as a result of the widespread use of big data analytics. Covid illness has recently been one of the leading causes of death in people. Since then, related input chest X-ray image for diagnosing COVID illness have been enhanced by diagnostic tools. Big data technological breakthroughs provide a fantastic option for reducing contagious Covid disease. To increase the model's confidence, it is necessary to integrate a large number of training sets, however handling the data may be difficult. With the development of big data technology, a unique method to identify and categorise covid illness is now found in this research. In order to manage incoming big data, a massive volume of chest x-ray images is gathered and analysed using a distributed computing server built on the Hadoop framework. In order to group identical groups in the input x-ray images, which in turn segments the dominating portions of an image, the fuzzy empowered weighted k-means algorithm is then employed. A hybrid quantum dilated convolution neural network is suggested to classify various kinds of covid instances, and a Black Widow-based Moth Flame is also shown to improve the performance of the classifier pattern. The performance analysis of COVID-19 detection makes use of the COVID-19 radiography dataset. The suggested HQDCNet approach has an accuracy of 99.01. The experimental results are evaluated in Python using performance metrics such as accuracy, precision, recall, f-measure, and loss function.

Building an Effective Classifier for Phishing Web Pages Detection: A Quantum-Inspired Biomimetic Paradigm Suitable for Big Data Analytics of Cyber Attacks.

To combat malicious domains, which serve as a key platform for a wide range of attacks, domain name service (DNS) data provide rich traces of Internet activities and are a powerful resource. This paper presents new research that proposes a model for finding malicious domains by passively analyzing DNS data. The proposed model builds a real-time, accurate, middleweight, and fast classifier by combining a genetic algorithm for selecting DNS data features with a two-step quantum ant colony optimization (QABC) algorithm for classification. The modified two-step QABC classifier uses K-means instead of random initialization to place food sources. In order to overcome ABCs poor exploitation abilities and its convergence speed, this paper utilizes the metaheuristic QABC algorithm for global optimization problems inspired by quantum physics concepts. The use of the Hadoop framework and a hybrid machine learning approach (K-mean and QABC) to deal with the large size of uniform resource locator (URL) data is one of the main contributions of this paper. The major point is that blacklists, heavyweight classifiers (those that use more features), and lightweight classifiers (those that use fewer features and consume the features from the browser) may all be improved with the use of the suggested machine learning method. The results showed that the suggested model could work with more than 96.6% accuracy for more than 10 million query-answer pairs.

A novel design and application of spatial data management platform for natural resources

Owing to the continuous development of earth observation technology in recent years, substantial amount of remote sensing image and natural resources business data have been generated. However, the traditional method of managing remote sensing image and natural resources data can no longer be used for organizing and managing big data. Along with the increasing requirements of data services, methods to efficiently organize large-scale remote sensing image and natural resource data are currently a hot topic With this background, we herein combine the Hadoop framework with the GIS technology to build a set of independent frameworks for the distributed data managing and service querying in order to enable the quick access and mass storage of large volumes of remote sensing image and natural resource data and to resolve the problem of big-data management. Additionally, to effectively improve the efficiency of query and service of remote sensing images and natural resource data, we use the services, such as those of land approval, regional planning application, and ground traffic analysis, to optimize the data storage model. Then we apply the model to solve the technical problems of large-scale data service release in a distributed computing environment. The results shows that, in terms of data management, the designed platform changed the traditional folder-type or common file-type system storage mode and solved the difficult problem of managing and applying massive remote sensing images and natural resources data. In terms of the query efficiency, compared with the single-node computing mode, the query efficiency of the platform for approved but not provided for use lands increased by 98.9%Tthe statistical efficiency of project-compliant land increased by 51.3%, and the analysis efficiency of land flow increased by 75.2%, and the data query service efficiency significantly improved. This study provides a solution to achieve the effective application of large-scale data and, in particular, contributes toward the application of remote sensing images and natural resource data.

Rapid identification of chronic kidney disease in electronic health record database using computable phenotype combining a common data model.

Rapid identification of chronic kidney disease in electronic health record database using computable phenotype combining a common data model.

Research on Network Resource Collection and Distribution Strategies in Big Data-Driven Multimedia Teaching

The multimedia teaching information data generated in the era of big data are increasing at a huge scale, but the computing environment used before is difficult to meet users’ requirements for data integration. Multimedia information has the characteristics of diversity and structure diversity, which lead to the situation that the storage and processing methods are too complicated and time-consuming to varying degrees. This brings great challenges to big data analysis. Therefore, the fusion of this part of the data information requires a large-scale data parallel framework environment to ensure that multiple data calculation instructions can be executed each time, improve the range of problem solving, and essentially improve the complexity of multimedia teaching information. The main index of large-scale data fusion is computing speed. Aiming at the performance defects of the open source computing framework Hadoop, this paper intends to establish a new computing model. This model uses distributed data sets to replace the previous data structures and solve some problems in the process of data integration, thereby significantly improving the computation efficiency and reducing resource usage. Starting from the network’s perception of energy resources and considering that the randomness of energy available to nodes in the energy harvesting cognitive relay different systems will lead to different data volumes such as throughput and resource utilization, this paper takes the network’s throughput as the optimization network object for resource collection and allocation research. Experiment description: compared with the previous resource management, this model can reduce the total consumption of the system; compared with the single base station mode, the proposed method can extend the network life cycle to a greater extent; meanwhile, it ensures the use of users. Based on the stability analysis of the data, an optimal solution is obtained that can maximize the stable throughput of cognitive users and effectively improve resource collection and allocation.

ILLUSTRATING A SCALABLE ARCHITECTURE-POWERED DISEASE PREDICTION USING MACHINE LEARNING TECHNIQUES

Healthcare information systems typically collect, store and manage various kinds of data such as illness details, clinical history, essential body parameters, health insurance plans, and other related data towards enabling data processing and analytics to arrive at better decision making with all the clarity and alacrity. To reduce the mortality rate due to heart diseases, it is essential to predict the presence of disease in its budding stage itself. Manual extraction of the useful knowledge from historical data is practically tedious and time-consuming. Machine learning (ML) algorithms are being used to detect and predict something useful out of both historical and current data. Despite the applicability of machine learning algorithms for prediction, the accuracy of prediction is significantly influenced by features used for prediction. Moreover, to meet the needs of evolving data sizes, suitable technologies for data storage also become essential. Based on these two aspects, a comparative analysis has been performed for feature selection using four filter methods, namely, correlation measure, information gain, gain ratio and relief. Further, a scalable architecture using Hadoop framework has been proposed to enable the machine learning algorithms to handle larger datasets while performing prediction task. The impact of the proposed architecture on the performance of machine learning algorithm has been evaluated with benchmark dataset and found to have improved scalability and accuracy.

A Parallel High-Utility Itemset Mining Algorithm Based on Hadoop

High-utility itemset mining (HUIM) can consider not only the profit factor but also the profitable factor, which is an essential task in data mining. However, most HUIM algorithms are mainly developed on a single machine, which is inefficient for big data since limited memory and processing capacities are available. A parallel efficient high-utility itemset mining (P-EFIM) algorithm is proposed based on the Hadoop platform to solve this problem in this paper. In P-EFIM, the transaction-weighted utilization values are calculated and ordered for the itemsets with the MapReduce framework. Then the ordered itemsets are renumbered, and the low-utility itemsets are pruned to improve the dataset utility. In the Map phase, the P-EFIM algorithm divides the task into multiple independent subtasks. It uses the proposed S-style distribution strategy to distribute the subtasks evenly across all nodes to ensure load-balancing. Furthermore, the P-EFIM uses the EFIM algorithm to mine each subtask dataset to enhance the performance in the Reduce phase. Experiments are performed on eight datasets, and the results show that the runtime performance of P-EFIM is significantly higher than that of the PHUI-Growth, which is also HUIM algorithm based on the Hadoop framework.