Big Data Datasets Research Articles

Developing Big Data anomaly dynamic and static detection algorithms: AnomalyDSD spark package

BackgroundAnomaly detection is the process of identifying observations that differ greatly from the majority of data. Unsupervised anomaly detection aims to find outliers in data that is not labeled, therefore, the anomalous instances are unknown. The exponential data generation has led to the era of Big Data. This scenario brings new challenges to classic anomaly detection problems due to the massive and unsupervised accumulation of data. Traditional methods are not able to cop up with computing and time requirements of Big Data problems. MethodsIn this paper, we propose four distributed algorithm designs for Big Data anomaly detection problems: HBOS_BD, LODA_BD, LSCP_BD, and XGBOD_BD. They have been designed following the MapReduce distributed methodology in order to be capable of handling Big Data problems. ResultsThese algorithms have been integrated into an Spark Package, focused on static and dynamic Big Data anomaly detection tasks, namely AnomalyDSD. Experiments using a real-world case of study have shown the performance and validity of the proposals for Big Data problems. ConclusionsWith this proposal, we have enabled the practitioner to efficiently and effectively detect anomalies in Big Data datasets, where the early detection of an anomaly can lead to a proper and timely decision.

The role of Big Data and Data Science in the context of information security and cybersecurity

ObjectivesIn the area of the fastest growing fields of ICT technology which determine the successive stages of progress in the field of online electronic banking, it is necessary to disseminate standards for conducting financial operations carried out in the so-called cloud as well as using the large data sets located in the so-called Big Data platforms.Material and methodsFor the purposes of this article, research has been carried out on the broad issue of Big Data Analytics applications in the context of social, normative, economic, environmental, climate aspects. A questionnaire survey was conducted. The method of analysis of literature sources concerning the subject of the article was applied.ResultsCurrently emerging trends in the development of technology based on Big Data dataset platforms usually allow to perform multidimensional calculations and reported results of analyzes in real time. The analyzes conducted on huge data sets allow for comprehensive, multi-aspect risk assessments at the level of the whole entity. During the SARS-CoV-2 (Covid-19) coronavirus pandemic, there was an increase in the importance of analytics based on Big Data Analytics. During the pandemic, there was also an increase in the digitisation and internetisation of business processes. This also resulted in an increase in the scale of cybercrime development.ConclusionsThe technological revolution currently taking place, defined as Industry 4.0, is motivated by the development of the following factors: Big Data database technologies, cloud computing, machine learning, Internet of Things, artificial intelligence, Business Intelligence and other technologies of advanced data processing Data Mining. On the basis of the development new technological solutions, in recent years there has been a dynamic development of Big Data Analytics.

A novel efficient Rank-Revealing QR matrix and Schur decomposition method for big data mining and clustering (RRQR-SDM)

Big Data is an emerging technology with enormous potential to develop business and its administration. Due to the enormous volume, efficient data mining and clustering methods are crucial to extracting meaningful insights and patterns from large-scale datasets. Problems may arise from the need to analyze, capture, share, store, and visualize the data. Several methods have already been proposed for mining knowledge from big data. It is practically inefficient or impossible to handle these massive data using the proposed methods in a single machine because big data are frequently acquired from dispersed locations and stored on several machines. Matrix decomposition is one of the critical strategies to retrieve knowledge from diverse, noisy, huge data generated by modern applications and stored in dispersed locations. This study proposes a novel approach called the Rank-Revealing QR Matrix and Schur Decomposition Method (RRQR-SDM) specifically designed for big data mining and clustering tasks. The RRQR-SDM is designed to reveal the rank of the data matrix in a computationally efficient manner by using a modified QR decomposition, eliminating the need for expensive Singular Value Decomposition (SVD) computations. The proposed RRQR-SDM method offers several advantages over existing approaches. Firstly, exploiting the inherent low-rank structure reduces the computational complexity associated with large-scale datasets. By revealing the rank of the input matrix, it enables dimensionality reduction and efficient data compression. Secondly, the Schur decomposition enhances the interpretability of the data by providing a clear separation between the relevant and irrelevant components. This feature makes the RRQR-SDM method particularly suitable for data mining and clustering tasks where identifying the most significant features is essential. To evaluate the performance of the RRQR-SDM method, extensive experiments were conducted on various big data datasets. The results demonstrate that the proposed method outperforms state-of-the-art computational efficiency and clustering accuracy techniques.

Explainable machine learning models for Medicare fraud detection

As a means of building explainable machine learning models for Big Data, we apply a novel ensemble supervised feature selection technique. The technique is applied to publicly available insurance claims data from the United States public health insurance program, Medicare. We approach Medicare insurance fraud detection as a supervised machine learning task of anomaly detection through the classification of highly imbalanced Big Data. Our objectives for feature selection are to increase efficiency in model training, and to develop more explainable machine learning models for fraud detection. Using two Big Data datasets derived from two different sources of insurance claims data, we demonstrate how our feature selection technique reduces the dimensionality of the datasets by approximately 87.5% without compromising performance. Moreover, the reduction in dimensionality results in machine learning models that are easier to explain, and less prone to overfitting. Therefore, our primary contribution of the exposition of our novel feature selection technique leads to a further contribution to the application domain of automated Medicare insurance fraud detection. We utilize our feature selection technique to provide an explanation of our fraud detection models in terms of the definitions of the selected features. The ensemble supervised feature selection technique we present is flexible in that any collection of machine learning algorithms that maintain a list of feature importance values may be used. Therefore, researchers may easily employ variations of the technique we present.

ChatGPT and Big Data: Enhancing Text-to-Speech Conversion

Text-to-speech (TTS) conversion is a crucial technology for various applications, including accessibility, education, and entertainment. With the rapid growth of big data, TTS conversion systems face new challenges in terms of data size and diversity. In this paper, we propose to use the state-of-the-art language model ChatGPT to enhance TTS conversion for big data. We first introduce the background of TTS conversion and big data, and then review the existing TTS conversion systems and their limitations. Next, we describe the architecture and training of ChatGPT, and how it can be applied to TTS conversion. Finally, we evaluate the performance of the ChatGPT-based TTS conversion system on a large-scale real-world big data dataset, and compare it with the existing TTS systems. Our experimental results demonstrate that ChatGPT can significantly improve the quality and efficiency of TTS conversion for big data.

InterCriteria Analysis as a tool for analyzing Big Data datasets: Case study of 2021 national statistics of Bulgarian system of higher education

In the paper, the intuitionistic fuzzy sets based InterCriteria Analysis (ICA) and big data systems are applied to the 2021 dataset of the National Statistical Institute with respect to the Bulgarian system of higher education. The results are produced using the Big Data platform Hortonworks and the ICA software ICrAData. Discussion is presented on the discovered relationships regarding tendencies between enrolled and graduated students, and teaching staff.

The Longitudinal IntermediaPlus (2014–2016): A Case Study in Structuring Unstructured Big Data

Abstract This article details the novel structure developed to handle, harmonize and document big data for reuse and long-term preservation. ‘The Longitudinal IntermediaPlus (2014–2016)’ big data dataset is uniquely rich: it covers an array of German online media extendable to cross-media channels and user information. The metadata file for this dataset, and its documentation, were recently deposited as its own MySQL database called charmstana_sample_14-16.sql (https://data.gesis.org/sharing/#!Detail/10.7802/2030) (cs16) and is suitable for generating descriptive statistics. Analogous to the ‘Data View’ in spss, the charmstana_analysis (ca) contains the dataset’s numerical values. Both the cs16 and ca MySQL files are needed to conduct analysis on the full database. The research challenge was to process large-scaled datasets into one longitudinal, big-data data source suitable for academic research, and according to fair principles. The authors review four methodological recommendations that can serve as a framework for solving big-data structuring challenges, using the harmonization software CharmStats.

Prediction of drug response in major depressive disorder using ensemble of transfer learning with convolutional neural network based on EEG

Major Depressive Disorder (MDD) is one of the leading causes of disability worldwide. Prediction of response to Selective Serotonin Reuptake Inhibitors (SSRIs) antidepressants in patients with MDD is necessary for preventing side effects of mistreatment. In this study, a deep Transfer Learning (TL) strategy based on powerful pre-trained convolutional neural networks (CNNs) in the big data datasets is developed for classification of Responders and Non-Responders (R/NR) to SSRI antidepressants, using 19-channel Electro-encephalography (EEG) signal acquired from 30 MDD patients in the resting state. Multiple time–frequency images are obtained from each EEG channel using Continuous Wavelet Transform (CWT) for feeding into pre-trained CNN models that are VGG16, Xception, DenseNet121, MobileNetV2 and InceptionResNetV2. Our plan is to adapt and fine-tune the weights of networks to the target task with the small-sized dataset. Finally, to improve the recognition performance, an ensemble method based on majority voting of outputs of five mentioned deep TL architectures has been developed. Results indicate that the best performance among basic models achieved by DenseNet121 with accuracy, sensitivity and specificity of 95.74%, 95.56% and 95.64%, respectively. An Ensemble of these basic models created to surpass the accuracy obtained by each individual basic model. Our experiments show that ensemble model can gain accuracy, sensitivity and specificity of 96.55%, 96.01% and 96.95%, respectively. Therefore, proposed ensemble of TL strategy of pre-trained CNN models based on WT images obtained from EEG signal can be used for antidepressants treatment outcome prediction with a high accuracy.

An IoT based big data framework using equidistant heuristic and duplex deep neural network for diabetic disease prediction

A number of strategies empowered by Internet of Thing (IoT) have been utilized for the prediction of several dreadful diseases like diabetic for which ceaseless and real-time tracing system is an exceptionally predominant one. Wearable medical devices with sensor have consistently producing abundant data volume referred to as big data. With higher speed of data creation, it becomes cumbersome to accumulate, operate and analyze such abundant data volume during emergency. Deep Learning (DL) approaches have been largely made use of perceiving patterns, categorizing objects and the prediction of diabetic diseases at an early stage. Even so, DLs are fundamentally modest in evaluation precisely when the diabetic disease data size is abundant. To achieve the expectations of DLs for big data applications from an electronic device, the evaluation procedure for diabetic disease prediction must be speeded up, so that early analysis can be arrived at. In this work, a method called Equidistant Heuristic and Duplex Deep Neural Network (EH-DDNN) for early diabetic disease prediction is proposed. First, with the Big Data dataset as input, Equidistant Heuristic Pruning (EHP) algorithm is presented for feature selection. The EHP splits the input data matrix into rows and columns separately. By utilizing the notion of conditional non-alignment assessment and heuristics techniques, EHP, exploits neighbourhood evaluations into sub-division while reducing the communication time and overhead, thus enormously correlating computations. This in turn removes the irrelevant as well as redundant features, therefore resulting with fewer features easier for early prediction. Next, with the inherent features, a Duplex Deep Neural Network (DDNN) is designed for early prediction analyses a fusion of nonlinear processing features and linear response for stockpiling abundant data volume. Experiments are conducted and validation is performed on the benchmark datasets, Diabetes Data Set from UCI repository and Pima Indians Diabetes Disease dataset. Comparative analysis of diabetic disease prediction time, diabetic disease prediction overhead and ROC curve analysis are made.

Mining of High-Utility Patterns in Big IoT-based Databases

When focusing on the general area of data mining, high-utility itemset mining (HUIM) can be defined as an offset of frequent itemset mining (FIM). It is known to emphasize more factors critically, which gives HUIM its intrinsic edge. Due to the flourishing development of the IoT technique, the uncertainty patterns mining is also attractive. Potential high-utility itemset mining (PHUIM) is introduced to reveal valuable patterns in an uncertainty database. Unfortunately, even though the previous methods are all very effective and powerful to mine, the potential high-utility itemsets quickly. These algorithms are not specifically designed for a database with an enormous number of records. In the previous methods, uncertainty transaction datasets would be load in the memory ultimately. Usually, several pre-defined operators would be applied to modify the original dataset to reduce the seeking time for scanning the data. However, it is impracticable to apply the same way in a big-data dataset. In this work, a dataset is assumed to be too big to be loaded directly into memory and be duplicated or modified; then, a MapReduce framework is proposed that can be used to handle these types of situations. One of our main objectives is to attempt to reduce the frequency of dataset scans while still maximizing the parallelization of all processes. Through in-depth experimental results, the proposed Hadoop algorithm is shown to perform strongly to mine all of the potential high-utility itemsets in a big-data dataset and shows excellent performance in a Hadoop computing cluster.

Cooperative co-evolution for feature selection in Big Data with random feature grouping

A massive amount of data is generated with the evolution of modern technologies. This high-throughput data generation results in Big Data, which consist of many features (attributes). However, irrelevant features may degrade the classification performance of machine learning (ML) algorithms. Feature selection (FS) is a technique used to select a subset of relevant features that represent the dataset. Evolutionary algorithms (EAs) are widely used search strategies in this domain. A variant of EAs, called cooperative co-evolution (CC), which uses a divide-and-conquer approach, is a good choice for optimization problems. The existing solutions have poor performance because of some limitations, such as not considering feature interactions, dealing with only an even number of features, and decomposing the dataset statically. In this paper, a novel random feature grouping (RFG) has been introduced with its three variants to dynamically decompose Big Data datasets and to ensure the probability of grouping interacting features into the same subcomponent. RFG can be used in CC-based FS processes, hence called Cooperative Co-Evolutionary-Based Feature Selection with Random Feature Grouping (CCFSRFG). Experiment analysis was performed using six widely used ML classifiers on seven different datasets from the UCI ML repository and Princeton University Genomics repository with and without FS. The experimental results indicate that in most cases [i.e., with naïve Bayes (NB), support vector machine (SVM), k-Nearest Neighbor (k-NN), J48, and random forest (RF)] the proposed CCFSRFG-1 outperforms an existing solution (a CC-based FS, called CCEAFS) and CCFSRFG-2, and also when using all features in terms of accuracy, sensitivity, and specificity.

Adaptive Fuzzy Map Approach for Accruing Velocity of Big Data Relies on Fireflies Algorithm for Decentralized Decision Making

Velocity and volume are two important factors that affect the accuracy of streaming data during the transfer process in Big data applications. This paper presents an Adaptive Fuzzy Map Approach that Relies on Fireflies Algorithm for Accruing Velocity of Big Data and Decentralized Decision Making. A key advantage of the Firefly algorithm is the providing of a small number of iterations comparing to the other methods, which minimize the execution time. Furthermore, the Firefly algorithm is significant to the fuzzy logic system to get its inputs. In addition to the Firefly algorithm, Kalman filter is used to scale the distances of Big data datasets, where it generates output by assigning the match and mismatch. This work used a real dataset to extract variables and values through fuzzification function and be able to coexist as categorical data. After 10 dependent runs that are dealing with certain parameters to be available on aspects of velocity and volume of Big data existing in two parameters Goal and Dimension, the meaningful aspect scale by minimizes the randomness parameter by approximately 1.6%. The other aspect is decision making that is gained through exploration and exploitation that is covered by attraction base and attraction_min parameters. The evaluation has been made by making a comparison between the proposed Adaptive Fuzzy Map Approach and ANOVA model based on the variables like travelled time, road, speed, and distance, which showed clear enhancement produced by the proposed Adaptive Fuzzy Map Approach in terms of the accruing velocity of Big Data.

Hybrid neural networks for big data classification

Two new hybrid neural architectures combining morphological neurons and perceptrons are introduced in this paper. The first architecture, called Morphological - Linear Neural Network (MLNN) consists of a hidden layer of morphological neurons and an output layer of classical perceptrons has the capability of extracting features. The second architecture, called Linear-Morphological Neural Network (LMNN) is composed of one or several perceptron layers as a feature extractor, it is then followed by an output layer of morphological neurons for non-linear classification. Both architectures are trained by stochastic gradient descent. One of the main contributions of this paper is to show that the morphological layer offers a greater capacity to extract features than the perceptron layer. This claim is supported both theoretically and experimentally. We prove that the morphological layer possesses a greater capacity per computation unit to segment the 2D input space than the perceptron layer. In other words, adding more hyper-boxes produces more response regions than adding hyperplanes. From an empirical point of view, we test the two new models on 25 standard datasets at low dimensionality and one big data dataset. The result is that MLNN requires a lesser number of learning parameters than the other tested architectures while achieving better accuracies.

From Big to Smart Data: Iterative ensemble filter for noise filtering in Big Data classification

The quality of the data is directly related to the quality of the models drawn from that data. For that reason, many research is devoted to improve the quality of the data and to amend errors that it may contain. One of the most common problems is the presence of noise in classification tasks, where noise refers to the incorrect labeling of training instances. This problem is very disruptive, as it changes the decision boundaries of the problem. Big Data problems pose a new challenge in terms of quality data due to the massive and unsupervised accumulation of data. This Big Data scenario also brings new problems to classic data preprocessing algorithms, as they are not prepared for working with such amounts of data, and these algorithms are key to move from Big to Smart Data. In this paper, an iterative ensemble filter for removing noisy instances in Big Data scenarios is proposed. Experiments carried out in six Big Data datasets have shown that our noise filter outperforms the current state-of-the-art noise filter in Big Data domains. It has also proved to be an effective solution for transforming raw Big Data into Smart Data.

DPASF: a flink library for streaming data preprocessing

BackgroundData preprocessing techniques are devoted to correcting or alleviating errors in data. Discretization and feature selection are two of the most extended data preprocessing techniques. Although we can find many proposals for static Big Data preprocessing, there is little research devoted to the continuous Big Data problem. Apache Flink is a recent and novel Big Data framework, following the MapReduce paradigm, focused on distributed stream and batch data processing.In this paper, we propose a data stream library for Big Data preprocessing, named DPASF, under Apache Flink. The library is composed of six of the most popular and widely used data preprocessing algorithms. It contains three algorithms for discretization, and three algorithms for performing feature selection.ResultsThe algorithms have been tested using two Big Data datasets. Experimental results show that preprocessing can not only reduce the size of the data, but also maintain or even improve the original accuracy in a short period of time.ConclusionDPASF contains algorithms that are useful when dealing with Big Data data streams. The preprocessing algorithms included in the library are able to tackle Big Datasets efficiently and to correct imperfections in the data.

D2D Task Offloading: A Dataset-Based Q&A

D2D interactions are promoted both in the context of traffic and computation offloading. Applications with functionalities that are executed in parallel in more than one mobile device have been proposed while the idea of using another mobile device as a relay is going to be part of 5G. In this work, we first present the main types of D2D applications, then we discuss the characteristics of task offloading and the quality of experience in D2D ecosystems. After that, we list five popular questions regarding the ability of other users to be helpful in terms of resources, connectivity, availability, incentives, and security. To answer the listed questions, we use two mobile big data datasets, one experimental study, and arguments from our experience and the literature.

Evaluating associative classification algorithms for Big Data

BackgroundAssociative Classification, a combination of two important and different fields (classification and association rule mining), aims at building accurate and interpretable classifiers by means of association rules. A major problem in this field is that existing proposals do not scale well when Big Data are considered. In this regard, the aim of this work is to propose adaptations of well-known associative classification algorithms (CBA and CPAR) by considering different Big Data platforms (Spark and Flink).ResultsAn experimental study has been performed on 40 datasets (30 classical datasets and 10 Big Data datasets). Classical data have been used to find which algorithms perform better sequentially. Big Data dataset have been used to prove the scalability of Big Data proposals. Results have been analyzed by means of non-parametric tests. Results proved that CBA-Spark and CBA-Flink obtained interpretable classifiers but it was more time consuming than CPAR-Spark or CPAR-Flink. In this study, it was demonstrated that the proposals were able to run on Big Data (file sizes up to 200 GBytes). The analysis of different quality metrics revealed that no statistical difference can be found for these two approaches. Finally, three different metrics (speed-up, scale-up and size-up) have also been analyzed to demonstrate that the proposals scale really well on Big Data.ConclusionsThe experimental study has revealed that sequential algorithms cannot be used on large quantities of data and approaches such as CBA-Spark, CBA-Flink, CPAR-Spark or CPAR-Flink are required. CBA has proved to be very useful when the main goal is to obtain highly interpretable results. However, when the runtime has to be minimized CPAR should be used. No statistical difference could be found between the two proposals in terms of quality of the results except for the interpretability of the final classifiers, CBA being statistically better than CPAR.

Hybrid Parallel Linguistic Fuzzy Rules with Canopy MapReduce for Big Data Classification in Cloud

With the increasing availability of large amount of information and the benefits related to data processing, big data have gained large significance in recent years. With scalable nature of data, big data applications are processed using MapReduce programming model. However, the application of rule-based models in datasets is not straightforward and big data are not classified in an efficient manner. To overcome the above-mentioned problems, parallel linguistic fuzzy rule with canopy MapReduce (LFR-CM) framework is introduced. LFR-CM framework classifies big data using canopy MapReduce function for information sharing in cloud with higher classification accuracy and lesser time consumption. It comprises three steps for efficient classification in cloud environment. Initially, it constructs the fuzzy knowledge base (KB) from the big data training set where linguistic fuzzy rules are constructed. The second step in LFR-CM framework has three operations. The first operation is map function used in parallel manner through every cloud user without transmitting any data to other cloud user nodes. The second operation is processing of data through the map function across all additional cloud user nodes. The third operation is reduce function deployed by each cloud user through the partitioned information. Finally, by this way, the data classification is performed with higher classification accuracy and lesser time consumption. LFR-CM framework is implemented and evaluated on Amazon EC2 cloud big data datasets and compared with the other classification system that utilizes MapReduce in terms of the runtime, classification time, classification accuracy and input/output cost. Based on the results observed from the study, LFR-CM framework is more efficient than the existing methods.

SMOTE-BD: An Exact and Scalable Oversampling Method for Imbalanced Classification in Big Data

The volume of data in today's applications has meant a change in the way Machine Learning issues are addressed. Indeed, the Big Data scenario involves scalability constraints that can only be achieved through intelligent model design and the use of distributed technologies. In this context, solutions based on the Spark platform have established themselves as a de facto standard.
 In this contribution, we focus on a very important framework within Big Data Analytics, namely classification with imbalanced datasets. The main characteristic of this problem is that one of the classes is underrepresented, and therefore it is usually more complex to find a model that identifies it correctly. For this reason, it is common to apply preprocessing techniques such as oversampling to balance the distribution of examples in classes.
 In this work we present SMOTE-BD, a fully scalable preprocessing approach for imbalanced classification in Big Data. It is based on one of the most widespread preprocessing solutions for imbalanced classification, namely the SMOTE algorithm, which creates new synthetic instances according to the neighborhood of each example of the minority class. Our novel development is made to be independent of the number of partitions or processes created to achieve a higher degree of efficiency. Experiments conducted on different standard and Big Data datasets show the quality of the proposed design and implementation.

A taxonomy of software-based and hardware-based approaches for energy efficiency management in the Hadoop

Apache Hadoop framework supports the storing and processing of big data datasets using simple programming models. Energy management has been recognized as one of the major issues in Hadoop, and many types of research have been conducted in this scope. However, despite the importance of this issue, there is no inclusive study about energy efficiency in Hadoop. In this paper, the techniques of energy efficiency in Hadoop are classified into two main categories. Moreover, the benefits and drawbacks of these methods and a systematic study of the conducted research are provided and examined in this paper. Another aim is to provide the visions for the descriptions of open issues and recommendations for future research.