Sophisticated methods for noise filtering, subgroup discovery, and classification in big data analysis

Abstract

Big Data is a popular research area where a vast amount of data is created, replicated, and consumed by society. The quality of the data used directly influences big data knowledge discovery. The existence of noise is the most prevalent problem influencing data quality. The following techniques were developed to reduce noise in data with a distributed setting: Homogenous Ensemble for Big Data (HME-BD) and Heterogeneous Ensemble for Big Data (HTE-BD). In this article, the performance of HTE-BD is improved further by developing Enhanced HTE-BD (EHTE-BD), which combines Logistic Regression based Support Vector Machine (LR-SVM) in conjunction with RF, LR, and KNN to reduce noisy data. Furthermore, the Multi-Objective Evolutionary Fuzzy Method for Subgroup Discovery throughout Big Data (MEFASD-BD) was used to resolve the multi-objective optimization challenge, and the Non-Dominated Sorting Genetic Algorithm-II (NSGA-II) was utilized to handle the rising dimensionality issue through subgroup discovery. To address the NSGA-II’s slow convergence rate, an Improved Multi-Objective Meta-Heuristic Fuzzy approach for discovering subgroups in big data is described, that contains a meta-heuristic method for subgroup discovery known as the Multi-Objective Differential Search Algorithm (MODSA). It selects the most relevant subgroups from vast amounts of data, reducing the data’s dimensionality. The Fuzzy Deep Neural Network (FDNN) classifier assesses the main subgroups. By removing noisy data and selecting the most relevant subgroups, the performance of FDNN in classifying vast amounts of data is improved.