Hadoop MapReduce Research Articles

Input/output optimization scheduler for cloud-based map reduce framework

Hadoop MapReduce (HMR) provides the most common MapReduce (MR) framework, and it is available as open source. MR is a famous computational framework for evaluating unstructured, and semi-structured big data and executing applications in the past ten years. Memory and input/output (I/O) overhead are just two of the many problems affecting the current HMR scheduler system. This study aims to improve systems resource use including the processing of data in real-time by creating a memory I/O optimized scheduler (MIOOS) for HMR. The disk I/O seek can be reduced by using MIOOS, which analyzes the entire memory management. Additionally, the MIOOS makespan approach is used to reduce the occurrence of problems in intermediary tasks. Both the MIOOS approach and the current approach are assessed by using complex scientific workflow applications with extreme task inter-dependencies. Further, the comparison study demonstrates that the MIOOS framework outdoes the current approach regarding makespan and overall memory usage.

Predictive modelling of MapReduce job performance in cloud environments using machine learning techniques

Within the Hadoop ecosystem, MapReduce stands as a cornerstone for managing, processing, and mining large-scale datasets. Yet, the absence of efficient solutions for precise estimation of job execution times poses a persistent challenge, impacting task allocation and distribution within Hadoop clusters. In this study, we present a comprehensive machine learning approach for predicting the execution time of MapReduce jobs, encompassing data collection, preprocessing, feature engineering, and model evaluation. Leveraging a rich dataset derived from comprehensive Hadoop MapReduce job traces, we explore the intricate relationship between cluster parameters and job performance. Through a comparative analysis of machine learning models, including linear regression, decision tree, random forest, and gradient-boosted regression trees, we identify the random forest model as the most effective, demonstrating superior predictive accuracy and robustness. Our findings underscore the critical role of features such as data size and resource allocation in determining job performance. With this work, we aim to enhance resource management efficiency and enable more effective utilisation of cloud-based Hadoop clusters for large-scale data processing tasks.

Agriculture data analysis using parallel k-nearest neighbour classification algorithm

A cost-effective and effective agriculture management system is created by utilizing data analytics (DA), internet of things (IoT), and cloud computing (CC). Geographic information system (GIS) technology and remote sensing predictions give users and stakeholders access to a variety of sensory data, including rainfall patterns and weather-related information (such as pressure, humidity, and temperatures). They have unstructured format for sensory data. The current systems do a poor job of analysing such data since they cannot effectively balance speed and memory usage. An effective categorization model (ECM) on agriculture management system is proposed to address this research difficulty. First, a classification technique called priority-based k-nearest neighbour (KNN) is provided to categorize unstructured multi-dimensional data into a structured form. Additionally, the Hadoop MapReduce (HMR) framework is used to do classification utilizing a parallel approach. Data from real-time IoT sensors used in agriculture is the subject of experiments. The suggested approach significantly outperforms previous approaches that are computing time, memory efficiency, model accuracy, and speedup.

Improving Hadoop MapReduce performance on heterogeneous single board computer clusters

Over the past decade, Apache Hadoop has become a leading framework for big data processing. Single board computer (SBC) clusters, predominantly adopting Raspberry Pi (RPi), have been employed to explore the potential of MapReduce processing in terms of low power and cost because, capital costs aside, power consumption has also become a primary concern in many industries. After building SBC clusters, it is prevalent to consider adding more nodes, particularly newer generation SBCs, to the existing clusters or replacing old (or inactive) nodes with new ones to improve performance, inevitably causing heterogeneous SBC clusters. The Hadoop framework on these heterogeneous SBC clusters creates challenging new problems due to computing resource discrepancies in each node. Native Hadoop does not carefully consider the heterogeneity of the cluster nodes. Consequently, heterogeneous SBC Hadoop clusters result in significant performance variation or, more critically, persistent node failures. This paper proposes a new Hadoop Yet Another Resource Negotiator (YARN) architecture design to improve MapReduce performance on heterogeneous SBC Hadoop clusters with tight computing resources. We newly implement two main scheduling policies on Hadoop YARN based on the correct computing resource information that each SBC node provides: (1) two (master-driven vs. slave-driven) MapReduce task scheduling frameworks to determine more effective processing modes and (2) ApplicationMaster (AM) and reduce task distribution mechanisms to provide the best Hadoop performance by minimizing performance variation. Thus, the proposed Hadoop framework makes the best use of the performance-frugal SBC Hadoop cluster by intelligently distributing MapReduce tasks to each node. To our knowledge, the proposed framework is the first redesigned Hadoop YARN architecture to address various challenging problems particularly on heterogeneous SBC Hadoop clusters for big data processing. The extensive experiments with Hadoop benchmarks demonstrate that the redesigned framework performs better performance than the native Hadoop by an average of 2.55× and 1.55× under I/O intensive and CPU-intensive workloads, respectively.

An Efficient Approach to Extract and Store Big Semantic Web Data Using Hadoop and Apache Spark GraphX

The volume of data is growing at an astonishingly high speed. Traditional techniques for storing and processing data, such as relational and centralized databases, have become inefficient and time-consuming. Linked data and the Semantic Web make internet data machine-readable. Because of the increasing volume of linked data and Semantic Web data, storing and working with them using traditional approaches is not enough, and this causes limited hardware resources. To solve this problem, storing datasets using distributed and clustered methods is essential. Hadoop can store datasets because it can use many hard disks for distributed data clustering; Apache Spark can be used for parallel data processing more efficiently than Hadoop MapReduce because Spark uses memory instead of the hard disk. Semantic Web data has been stored and processed in this paper using Apache Spark GraphX and the Hadoop Distributed File System (HDFS). Spark's in-memory processing and distributed computing enable efficient data analysis of massive datasets stored in HDFS. Spark GraphX allows graph-based semantic web data processing. The fundamental objective of this work is to provide a way for efficiently combining Semantic Web and big data technologies to utilize their combined strengths in data analysis and processing. First, the proposed approach uses the SPARQL query language to extract Semantic Web data from DBpedia datasets. DBpedia is a hugely available Semantic Web dataset built on Wikipedia. Secondly, the extracted Semantic Web data was converted to the GraphX data format; vertices and edges files were generated. The conversion process is implemented using Apache Spark GraphX. Third, both vertices and edge tables are stored in HDFS and are available for visualization and analysis operations. Furthermore, the proposed techniques improve the data storage efficiency by reducing the amount of storage space by half when converting from Semantic Web Data to a GraphX file, meaning the RDF size is around 133.8 and GraphX is 75.3. Adopting parallel data processing provided by Apache Spark in the proposed technique reduces the required data processing and analysis time. This article concludes that Apache Spark GraphX can enhance Semantic Web and Big Data technologies. We minimize data size and processing time by converting Semantic Web data to GraphX format, enabling efficient data management and seamless integration.

Cost-aware optimal resource provisioning Map-Reduce scheduler for hadoop framework

<p><span lang="EN-US">Distributed data processing model has been one of the primary components in the case of data-intensive applications; furthermore, due to advancements in technologies, there has been a huge volume of data generation of diverse nature. Hadoop map reduce framework is responsible for adopting the ease of deployment mechanism in an open-source framework. The existing Hadoop MapReduce framework possesses high makespan time and high Input/Output overhead and it mainly affects the cost of a model. Thus, this research work presents an optimized cost aware resource provisioning MapReduce model also known as the cost-effective resource provisioning MapReduce (CRP-MR) model. CRP-MR model introduces the two integrated approaches to minimize the cost; at first, this model presents the optimal resource optimization and optimal Input/Output optimization cleansing in the Hadoop MapReduce (HMR) scheduler. CRP-MR is evaluated considering the bioinformatics dataset and CRP-MR performs better than the existing model. </span></p>

Modelling of healthcare data analytics using optimal machine learning model in big data environment

AbstractRecent advances in wireless networking, big data technologies, namely Internet of Things (IoT) 5G networks, health care big data analytics, and other technologies in artificial intelligence (AI) and wearables, have supported the progression of intellectual disease diagnosis methods. Medical data covers all patient data such as pharmacy texts, electronic health reports (EHR), prescriptions, study data from medical journals, clinical photographs, and diagnostic reports. Big data is a renowned method in the healthcare sector, with beneficial datasets that are highly difficult, voluminous, and rapid for healthcare providers for interpreting and computing using prevailing tools. This study combines concepts like deep learning (DL) and big data analytics in medical field. This article develops a new healthcare data analytics using optimal machine learning model in big data environment (HDAOML‐BDE) technique. The presented HDAOML‐BDE technique mainly aims to examine the healthcare data for disease detection and classification in the big data environment. For handling big data, the HDAOML‐BDE technique uses Hadoop MapReduce environment. In addition, the HDAOML‐BDE technique uses manta ray foraging optimization‐based feature selection (MRFO‐FS) technique to reduce high dimensionality problems. Moreover, the HDAOML‐BDE method uses relevance vector machine (RVM) model for the healthcare data environment. Furthermore, the arithmetic optimization algorithm (AOA) is utilized for the parameter tuning of the RVM classifier. The simulation results of the HDAOML‐BDE technique are tested on a healthcare dataset, and the outcomes portray the improved performance of the HDAOML‐BDE strategy over recent approaches in different measures.

Improving big data analytics data processing speed through map reduce scheduling and replica placement with HDFS using genetic optimization techniques

Big Data Analytics (BDA) is an unavoidable technique in today’s digital world for dealing with massive amounts of digital data generated by online and internet sources. It is kept in repositories for data processing via cluster nodes that are distributed throughout the wider network. Because of its magnitude and real-time creation, big data processing faces challenges with latency and throughput. Modern systems such as Hadoop and SPARK manage large amounts of data with their HDFS, Map Reduce, and In-Memory analytics approaches, but the migration cost is higher than usual. With Genetic Algorithm-based Optimization (GABO), Map Reduce Scheduling (MRS) and Data Replication have provided answers to this challenge. With multi objective solutions provided by Genetic Algorithm, resource utilization and node availability improve processing performance in large data environments. This work develops a novel creative strategy for enhancing data processing performance in big data analytics called Map Reduce Scheduling Based Non-Dominated Sorting Genetic Algorithm (MRSNSGA). The Hadoop-Map Reduce paradigm handles the placement of data in distributed blocks as a chunk and their scheduling among the cluster nodes in a wider network. Best fit solutions with high latency and low accessing time are extracted from the findings of various objective solutions. Experiments were carried out as a simulation with several inputs of varied location node data and cluster racks. Finally, the results show that the speed of data processing in big data analytics was enhanced by 30–35% over previous methodologies. Optimization approaches developed to locate the best solutions from multi-objective solutions at a rate of 24–30% among cluster nodes.

Performance Comparison of Apache Spark and Hadoop for Machine Learning based iterative GBTR on HIGGS and Covid-19 Datasets

In the realm of distributed computing frameworks, such as Apache Spark and MapReduce Hadoop, the efficacy of these frameworks varies across diverse applications and algorithms contingent upon distinctive evaluation metrics and critical parameters. This research paper diligently scrutinizes the extant body of research that compares these two frameworks concerning said evaluation metrics and parameters. Subsequently, it conducts empirical investigations to authenticate the performance of these frameworks in the context of an iterative Gradient Boosting Tree Regression (GBTR) algorithm. Remarkably, the comparative analyses in previous studies encompass a spectrum of iterative machine learning regression and classification techniques, batch processing, SQL, and Graph processing algorithms. Furthermore, numerous investigations have explored the application of machine learning algorithms encompassing logistic regression, Page Rank, K-Means, KNN, and the HiBench suite. This paper presents the comparison between the two distributed computing platforms on iterative GBTR for classification task on the HIGGS dataset from the physics domain and for the regression task on the Covid-19 dataset from the healthcare domain. The empirical findings corroborate that Apache Spark exhibits superior execution speed in iterative tasks when the available physical memory significantly exceeds the dataset size. Conversely, Hadoop outperforms Spark when dealing with substantial datasets or constrained physical memory resources.

Ballot Counting System Prototype with Hadoop Platform using Map and Reduce

Indonesia is among the nations practicing a democratic system, where the head of state is elected through general elections. During these elections, every citizen possessing a National Identity Card (KTP) holds equal voting rights. Counting votes accurately amidst the vast number of ballots poses a challenge. The ultimate objective is to develop a vote counting implementation using Hadoop, utilizing a form C1 image as input. The numerical data from the C1 form is processed into INT data types employing artificial neural networks (ANN) generated through the Restricted Boltzmann Machine (RBM), trained with MNIST data. These ANN models are uploaded onto a website for image storage in the database. The website's data is then crawled using Apache Nutch, and the obtained results are processed through Hadoop's MapReduce algorithm. Testing reveals efficient performance, with the recapitulation of results using Apache Nutch and Hadoop employing multimode and fair scheduler scheduling algorithms taking one minute and twenty-five seconds. Conversely, Hadoop configured with a single-node setup and FIFO scheduling algorithm achieves a quicker time of 67 seconds.

A scalable MapReduce-based design of an unsupervised entity resolution system.

Traditional data curation processes typically depend on human intervention. As data volume and variety grow exponentially, organizations are striving to increase efficiency of their data processes by automating manual processes and making them as unsupervised as possible. An additional challenge is to make these unsupervised processes scalable to meet the demands of increased data volume. This paper describes the parallelization of an unsupervised entity resolution (ER) process. ER is a component of many different data curation processes because it clusters records from multiple data sources that refer to the same real-world entity, such as the same customer, patient, or product. The ability to scale ER processes is particularly important because the computation effort of ER increases quadratically with data volume. The Data Washing Machine (DWM) is an already proposed unsupervised ER system which clusters references from diverse data sources. This work aims at solving the single-threaded nature of the DWM by adopting the parallelization nature of Hadoop MapReduce. However, the proposed parallelization method can be applied to both supervised systems, where matching rules are created by experts, and unsupervised systems, where expert intervention is not required. The DWM uses an entropy measure to self-evaluate the quality of record clustering. The current single-threaded implementations of the DWM in Python and Java are not scalable beyond a few 1,000 records and rely on large, shared memory. The objective of this research is to solve the major two shortcomings of the current design of the DWM which are the creation and usage of shared memory and lack of scalability by leveraging on the power of Hadoop MapReduce. We propose Hadoop Data Washing Machine (HDWM), a MapReduce implementation of the legacy DWM. The scalability of the proposed system is displayed using publicly available ER datasets. Based on results from our experiment, we conclude that HDWM can cluster from 1,000's to millions of equivalent references using multiple computational nodes with independent RAM and CPU cores.

A Comparative Study on Hadoop MapReduce and Apache Spark Framework for Big Data Analytics

A Comparative Study on Hadoop MapReduce and Apache Spark Framework for Big Data Analytics

In-Storage Computing for Hadoop MapReduce Framework: Challenges and Possibilities

Solid State Drives (SSDs) were initially developed as faster storage devices intended to replace conventional magnetic Hard Disk Drives (HDDs). However, high computational capabilities enable SSDs to be computing nodes, not just faster storage devices. Such capability is generally called ”In-Storage Computing (ISC)”. Today’s Hadoop MapReduce framework has become a de facto standard for big data processing. This paper explores In-Storage Computing challenges and opportunities for the Hadoop MapReduce framework. For this, we integrate a Hadoop MapReduce system with ISC SSD devices that implement the Hadoop Mapper inside real SSD firmware. This offloads Map tasks from the host MapReduce system to the ISC SSDs. We additionally optimize the host Hadoop system to make the best use of our proposed ISC Hadoop system. Experimental results demonstrate our ISC Hadoop MapReduce system achieves a remarkable performance gain (2.3 faster) as well as significant energy savings (11.5 lower) compared to a typical Hadoop MapReduce system. Further, the experiment suggests such ISC augmented systems can provide a very promising computing model in terms of a system scalability.

IDLTM-DMT: Intelligent Deep Learning based Trust Management with Decision Making Tool for Healthcare Internet of Things and Big Data Environment with Neutrosophic Set Analysis

Over the last few years development of Internet of Things (IoT) devices and communication technologies have resulted in the massive generation of health-related data. In the context of healthcare, IoT offers several advantages, including being able to observe patients very closely and using data for analytics. A major challenging issue that exists in the usage of IoT and big data in the medical field is security. As healthcare data is highly vulnerable and becomes a target for attacks, there are significant privacy issues related to the usage of big data analytics. Besides, implementing new data analysis tools and strategies for handling big data decision-making is a major issue. The capability to examine this amount of data is a significant aspect of big data in health care. For resolving these issues, this paper presents a new intelligent deep learning-based trust management with decision making tool (IDLTM-DMT) for IoT healthcare big data environments, incorporating Neutrosophic Set Analysis (NSA). The proposed IDLTM-DMT model enables IoT devices to gather healthcare data. The IDLTM-DMT model involves a DL based bidirectional long short-term memory (BiLSTM) model for vulnerability detection and thereby identifies the malicious traffic in the Network. Hadoop MapReduce is used for handling big data and a decision-making tool using Deep Stacked Auto Encoder (DSAE) is used for the classification of diseases that exist in big data. To optimize the DSAE model's hyperparameters and improve classification performance, the Sandpiper Optimization (SPO) Algorithm is employed. Neutrosophic Set Analysis is integrated to manage the indeterminacy and inconsistency of the data, enhancing the decision-making process. Extensive experimental analysis is conducted on the EEG Eye State Dataset, with results analyzed using various performance measures. The findings indicate that the proposed method achieves improved accuracy compared to existing methods, demonstrating the effectiveness of incorporating Neutrosophic Set Analysis in IoT healthcare big data environments.

Modeling of class imbalance handling with optimal deep learning enabled big data classification model

Big data is the amount of data that surpasses the ability to process the data of a system concerning memory usage and computation time. It is commonly applied in several domains like healthcare, education, social networks, e-commerce, etc., as they have progressively obtained a massive quantity of input data. A major research problem is big data analytics, which can be carried out using expert systems and deep structured architectures. Besides, data wrangling and class imbalance data handling are challenging issues that need to be resolved in big data analytics. Class imbalance data degrade the performance of the classification model, which remains a challenging process due to the heterogeneous and complex structure of the comparatively huge datasets. Thus, the research focused on presenting a Class Imbalance Handling with Optimal Deep Learning Enabled Big Data Classification (CIHODL-BDC) framework. The core perception of the CIHODL-BDC framework helps to classify the big data in the Hadoop MapReduce framework. To accomplish this, the presented CIHODL-BDC model initially performs a data wrangling process is performed to alter the unrefined data into a useful layout. Next, the CIHODL-BDC model handles the class imbalance problem using a grey wolf optimizer (GWO) with Synthetic Minority Oversampling (SMOTE) technique. Besides, the Adam optimizer procedure with the Bidirectional Long Short Term Memory (BiLSTM) approach is performed to categorize the big data. The result analysis of the proposed CIHODL-BDC model is evaluated by two standard datasets. The simulation outcomes revealed the elevated performance of the CIHODL-BDC approach over existing methods.

MR-AT: Map Reduce based Apriori Technique for Sequential Pattern Mining using Big Data in Hadoop

One of the most well-known and widely implemented data mining methods is Apriori algorithm which is responsible for mining frequent item sets. The effectiveness of the Apriori algorithm has been improved by a number of algorithms that have been introduced on both parallel and distributed platforms in recent years. They are distinct from one another on account of the method of load balancing, memory system, method of data degradation, and data layout that was utilised in their implementation. The majority of the issues that arise with distributed frameworks are associated with the operating costs of handling distributed systems and the absence of high-level parallel programming languages. In addition, when using grid computing, there is constantly a possibility that a node will fail, which will result in the task being re-executed multiple times. The MapReduce approach that was developed by Google can be used to solve these kinds of issues. MapReduce is a programming model that is applied to large-scale distributed processing of data on large clusters of commodity computers. It is effective, scalable, and easy to use. MapReduce is also utilised in cloud computing. This research paper presents an enhanced version of the Apriori algorithm, which is referred to as Improved Parallel and Distributed Apriori (IPDA). It is based on the scalable environment referred as Hadoop MapReduce, which was used to analyse Big Data. Through the generation of split-frequent data regionally and the early elimination of unusual data, the proposed work has its primary objective to reduce the enormous demands placed on available resources as well as the reduction of the overhead communication that occurs whenever frequent data are retrieved. The paper presents the results of tests, which demonstrate that the IPDA performs better than traditional apriori and parallel and distributed apriori in terms of the amount of time required, the number of rules created, and the various minimum support values.

A Novel Data Management Scheme in Cloud for Micromachines

In cyber-physical systems (CPS), micromachines are typically deployed across a wide range of applications, including smart industry, smart healthcare, and smart cities. Providing on-premises resources for the storage and processing of huge data collected by such CPS applications is crucial. The cloud provides scalable storage and computation resources, typically through a cluster of virtual machines (VMs) with big data tools such as Hadoop MapReduce. In such a distributed environment, job latency and makespan are highly affected by excessive non-local executions due to various heterogeneities (hardware, VM, performance, and workload level). Existing approaches handle one or more of these heterogeneities; however, they do not account for the varying performance of storage disks. In this paper, we propose a prediction-based method for placing data blocks in virtual clusters to minimize the number of non-local executions. This is accomplished by applying a linear regression algorithm to determine the performance of disk storage on each physical machine hosting a virtual cluster. This allows us to place data blocks and execute map tasks where the data blocks are located. Furthermore, map tasks are scheduled based on VM performance to reduce job latency and makespan. We simulated our ideas and compared them with the existing schedulers in the Hadoop framework. The results show that the proposed method improves MapReduce performance in terms of job latency and makespan by minimizing non-local executions compared to other methods taken for evaluation.

Memory aware optimized Hadoop MapReduce model in cloud computing environment

<p>In the last decade, data analysis has become one of the popular tasks due to enormous growth in data every minute through different applications and instruments. MapReduce is the most popular programming model for data processing. Hadoop constitutes two basic models i.e., Hadoop file system (HDFS) and MapReduce, Hadoop is used for processing a huge amount of data whereas MapReduce is used for data processing. Hadoop MapReduce is one of the best platforms for processing huge data in an efficient manner such as processing web logs data. However, existing model This research work proposes memory aware optimized Hadoop MapReduce (MA-OHMR). MA-OHMR is developed considering memory as the constraint and prioritizes memory allocation and revocation in mapping, shuffling, and reducing, this further enhances the job of mapping and reducing. Optimal memory management and I/O operation are carried out to use the resource inefficiently manner. The model utilizes the global memory management to avoid garbage collection and MA-OHMR is optimized on the makespan front to reduce the same. MA-OHMR is evaluated considering two datasets i.e., simple workload of Wikipedia dataset and complex workload of sensor dataset considering makespan and cost as an evaluation parameter.</p>

A Novel Political Optimizer-Based Feature Selection with an Optimal Machine Learning Model for Financial Crisis Prediction

In today’s digital environment, business intelligence advances make it difficult to stay competitive and up to date on business trends. Decision-making in the financial industry is increasingly being powered by big data and machine learning. A decision-making process may be thought of as any sequence of processes that an individual goes through in order to select the option or course of action that is most suitable to meet their needs and necessities. The ability to anticipate the onset of a financial crisis is a significant economic phenomenon. A nation’s economic development and strength can be gauged by its capacity to provide an accurate assessment of the number of failed firms and the frequency with which they fail. The economics of the globe have been ravaged by recent global crises like as the COVID-19 pandemic and other recent environmental, financial, and economic disasters, which have marginalized efforts to construct a maintainable economy and civilization. The health and growth of a nation’s economy can be determined by precisely estimating the number of enterprises that will fail and the number that will succeed. Historically, there have been numerous strategies for constructing a successful financial crisis prediction (FPC) method. Effectively predicting business failures is a gauge of a country’s economic health. Several strategies are available for effective FCP. Classification performance, forecast accuracy, and legality are insufficient for practical use. Several of the suggested methods work for some issues. The specific dataset is not expandable. To improve classification, design a good prediction model adaptable to several datasets. An effective financial crisis prediction method (FPC) requires the right qualities. ML models can also be used to classify a company’s financial health. This research presents political optimizer-based feature selection (POFS) with optimal cascaded deep forest (OCDF) for FCP in big data environments. Hadoop Map Reduce handles huge datasets. POFS reduces computing complexity by handling feature selection. POFS is an original FCP algorithm categorization using OCDF. SFO is used to optimize CDF model parameters. A thorough simulation study was performed to evaluate POFS performance on benchmark datasets OCDFs. The results confirmed the POFS-OCDF method’s superiority over state-of-the-art approaches. With an outstanding sensitivity of 0.912, specificity of 0.953, accuracy of 0.944, F-score of 0.930, and Matthews correlation coefficient (MCC) of 0.912, the proposed POFS-OCDF technique has shown optimum results. The experimental results demonstrated that the POFS-OCDF technique outperformed other recently developed strategies on a variety of criteria. As previously stated, Sunflower optimization (SFO) is also used to tune the Cascaded Deep Forest (CDF) parameters. A detailed simulation analysis is performed based on the benchmark dataset to evaluate the higher classification efficiency of the POFS-OCDF technique. The invention of the POFS algorithm for FCP exemplifies the work’s originality.

SMOTE-Based Homogeneous Prediction for Aging-Related Bugs in Cloud-Oriented Software

Software aging is the process caused by Aging-Related Bugs (ARBs) which leads to the depletion of resources and degradation of performance in the long run. ARBs are difficult to find and replicate in future studies as they are less in number, thus prediction of ARB is necessary to save cost and time in the testing phase. ARBs are present in low proportion as compared to non-ARBs known as the class Imbalance problem resulting in insufficient training dataset for prediction models. In this study, Synthetic Minority Oversampling Technique (SMOTE) is applied along with homogeneous cross-project ARB prediction to reduce the effect of imbalance problem in software. SMOTE is oversampling of the minority instances synthetically to balance the dataset and improve the capability of defect prediction models. Homogeneous cross-project prediction is implemented where the datasets are different but the distribution of metric sets of both training and testing datasets is similar. The experiment is conducted on five cloud-oriented software such as Cassandra, Hive, Storm, Hadoop HDFS and Hadoop Mapreduce. The novelty of this study is the combination of SMOTE and homogeneous cross-project defect prediction for ARBs in cloud-oriented software. The comparative analysis is also conducted to understand the difference between SMOTE and non-SMOTE results with the help of machine learning classifiers. The result conveys that SMOTE is an efficient method to address class imbalance problem in ARB prediction.