Big Data Framework Research Articles

A COMPREHENSIVE STUDY ON BIG DATA FRAMEWORKS

With the advent of cloud computing technology, the generation of data from various sources has increased during the last few years. The current data processing technology must handle the enormous volumes of newly created data. Therefore, the studies in the literature have concentrated on big data, which has enormous volumes of almost unstructured data. Dealing with such data needs well-designed frameworks that fulfil developers’ needs and fit colourful purposes. Moreover, these frameworks can use for storing, processing, structuring, and analyzing data. The main problem facing cloud computing developers is selecting the most suitable framework for their applications. The literature includes many works on these frameworks. However, there is still a severe gap in providing comprehensive studies on this crucial area of research. Hence, this article presents a novel comprehensive comparison among the most popular frameworks for big data, such as Apache Hadoop, Apache Spark, Apache Flink, Apache Storm, and MongoDB. In addition, the main characteristics of each framework in terms of advantages and drawbacks are also deeply investigated in this article. Our research provides a comprehensive analysis of various metrics related to data processing, including data flow, computational model, overall performance, fault tolerance, scalability, interval processing, language support, latency, and processing speed. To our knowledge, no previous research has conducted a detailed study of all these characteristics simultaneously. Therefore, our study contributes significantly to the understanding of the factors that impact data processing and provides valuable insights for practitioners and researchers in the field

A survey on techniques, methods and security approaches in big data healthcare

A huge percentage of people especially in developed countries spend a good chunk of their wealth in managing their health conditions. In order to adequately administer healthcare, governments and various organizations have embraced advanced technology for automating the health industry. In recent past, electronic health records have largely been managed by Enterprise Resource Planning and legacy systems. Big data framework steadily emerge as the underlying technology in healthcare, which offers solutions that limits capacity of others systems in terms of storage and reporting. Automation through cloud services supported by storage of structured and unstructured health data in heterogeneous environment has improved service delivery, efficiency, medication, diagnosis, reporting and storage in healthcare. The argument support the idea that big data healthcare still face information security concern, for instance patient image sharing, authentication of patient, botnet, correlation attacks, man-in-the-middle, Distributed Denial of Service (DDoS), blockchain payment gateway, time complexities of algorithms, despite numerous studies conducted by scholars in security management for big data in smart healthcare. Some security technique include digital image encryption, steganography, biometrics, rule-based policy, prescriptive analysis, blockchain contact tracing, cloud security, MapReduce, machine-learning algorithms, anonymizations among others. However, most of these security solutions and analysis performed on structured and semi-structured data as opposed to unstructured data. This may affect the output of medical reporting of patients’ condition particularly on wearable devices and other examinations such as computerized tomography (CT) Scans among others. A major concern is how to identify inherent security vulnerabilities in big healthcare, which generate images for transmission and storage. Therefore, this paper conducted a comparative survey of solutions that specifically safeguards structured and unstructured data using systems that run on big data frameworks. The literature highlights several security advancements in cryptography, machine learning, anonymization and protocols. Most of these security frameworks lacks implementation evidence. A number of studies did not provide comprehensive performance metrics (accuracy, error, recall, precision) of the models besides using a single algorithm without validated justification. Therefore, a critique on the contribution, performance and areas of improvements discussed and summarized in the paper.

A new Apache Spark-based framework for big data streaming forecasting in IoT networks.

Analyzing time-dependent data acquired in a continuous flow is a major challenge for various fields, such as big data and machine learning. Being able to analyze a large volume of data from various sources, such as sensors, networks, and the internet, is essential for improving the efficiency of our society's production processes. Additionally, this vast amount of data is collected dynamically in a continuous stream. The goal of this research is to provide a comprehensive framework for forecasting big data streams from Internet of Things networks and serve as a guide for designing and deploying other third-party solutions. Hence, a new framework for time series forecasting in a big data streaming scenario, using data collected from Internet of Things networks, is presented. This framework comprises of five main modules: Internet of Things network design and deployment, big data streaming architecture, stream data modeling method, big data forecasting method, and a comprehensive real-world application scenario, consisting of a physical Internet of Things network feeding the big data streaming architecture, being the linear regression the algorithm used for illustrative purposes. Comparison with other frameworks reveals that this is the first framework that incorporates and integrates all the aforementioned modules.

MSVM: An Approach for Customer Churn Prediction using modified Support Vector Machine and various Machine Learning Techniques

Consumer attrition is a major issue across the world companies and that is one of their primary worries. Organizations, specifically in the telecommunication sector, are working to build technologies to forecast future customer churn due to the obvious direct impact on profits. It's essential to define the variables that lead to customer churn before making the required efforts to minimize churn. An important impact was the creation of an attrition estimation method that aids telecommunication corporations in predicting which consumers are willing to churn. The methodology proposed in this study employs mathematical methodologies on a big data framework to provide a unique strategy to feature design and evaluation. This research work has proposed a customer churn prediction using a modified Support Vector Machine Learning (mSVM) classifier. The significant contribution of this research is that we have changed the distance function of SVM in both training and testing. Similar machine learning algorithms are also validated on similar datasets such as Naïve Bayes (NB), K-nearest Neighbor (KNN) and Decision Tree (DT) with J48 classifiers. The BigML dataset has been used for detecting churn in the telecom industry in a real-time scenario. In an extensive experimental analysis, we demonstrate mSVM which produces higher accuracy over the traditional machine learning algorithm for different cross-validation methods.

A Low-Cost System Using a Big-Data Deep-Learning Framework for Assessing Physical Telerehabilitation: A Proof-of-Concept

The consolidation of telerehabilitation for the treatment of many diseases over the last decades is a consequence of its cost-effective results and its ability to offer access to rehabilitation in remote areas. Telerehabilitation operates over a distance, so vulnerable patients are never exposed to unnecessary risks. Despite its low cost, the need for a professional to assess therapeutic exercises and proper corporal movements online should also be mentioned. The focus of this paper is on a telerehabilitation system for patients suffering from Parkinson's disease in remote villages and other less accessible locations. A full-stack is presented using big data frameworks that facilitate communication between the patient and the occupational therapist, the recording of each session, and real-time skeleton identification using artificial intelligence techniques. Big data technologies are used to process the numerous videos that are generated during the course of treating simultaneous patients. Moreover, the skeleton of each patient can be estimated using deep neural networks for automated evaluation of corporal exercises, which is of immense help to the therapists in charge of the treatment programs.

Interminable Flows: A Generic, Joint, Customizable Resiliency Model for Big-Data Streaming Platforms

The examiner of cloud computing systems in the last few years observes that there is a trend of the emergence of a new Big Data framework almost every year. Since Hadoop was developed in 2007, new frameworks followed it such as Spark, Storm, Heron, Apex, Flink, Samza, Kafka...etc. Each framework is developed in a certain way to target and achieve certain objectives better than other frameworks do. However, there are few common functionalities and aspects that are shared between these frameworks. One vital aspect all these frameworks strive to achieve is better reliability and faster recovery time in case of failures. This is particularly crucial for streaming systems (compared to batch processing systems) where events are processed and monitored online in real time, and any delay in data delivery will cause a major inconvenience to the users. Another observation is that some reliability implementations are redundant between different frameworks. Encapsulating these implementations into one layer and make it shared between different applications will benefit more than one framework without the burden of re-implementing the same reliability approach in each single framework. These observations motivated us to present Warden, a generic, multi-framework, flexible, customizable, low overhead protocol to ensure the resiliency of streaming applications running on streaming Big Data frameworks. Most reliability protocols carry out one rigid fault tolerance approach targeted towards one system at a time. It is more challenging to provide a reliability approach that is pluggable in multiple Big Data frameworks at a time and can achieve low overheads comparable with single targeted framework approaches, yet is flexible and customizable by its users to make it tailored towards their objectives. The genericity is attained by providing an interface that can be used in different applications from different frameworks. The low overhead is achieved by providing faster application finish times with and without failures. The customizability is fulfilled by providing the users the options to choose between two delivery semantics (Exactly Once / At Most Once) combined with two fault tolerance guarantees (Crash Failures / Byzantine Failures). To the best of our knowledge, such approach was never tried on multiple streaming frameworks before. We built a prototype of Warden on Flink and Samza (with Kafka) streaming frameworks. Our evaluations highlight the effectiveness of our approach in the presence of failures and without failures compared to other fault tolerance techniques (such as checkpointing).

Framework of Big Data and Deep Learning for Simultaneously Solving Space Allocation and Signal Timing Problem

Framework of Big Data and Deep Learning for Simultaneously Solving Space Allocation and Signal Timing Problem

Development and Application of Visualization System of Gas Geological Dynamic Characteristics under Big Data Framework

In this study, coal and gas outbursts are the “biggest killer” of mine safety production. With the deepening of mining, the mine gas disaster is becoming more and more serious. From the perspective of big data, the establishment of a dynamic visualization system of gas geology integrating gas geology, gas drainage, dynamic outburst prevention, and other information can effectively improving the defects of gas disaster prevention and control in the coal industry, such as insufficient advance, lack of systematic identification methods and backward information collection methods, which is of great significance to improve the ability of gas hazard identification. Based on the precise detection of geology, structure, and gas, this paper proposes to use information technology, the Internet of things, big data analysis, and other technologies to comprehensively analyze the changes in gas occurrence and coal seam occurrence on the basis of the causes of mine gas geological outburst, fully consider the logical relationship between different factors and outburst and adopt the disciplinary advantages of grey theory, fault tree theory, BP neural network, and so on. The tree of coal and gas outburst accidents with general significance is constructed by 24 relatively independent factors. The input vector is determined as the matrix composed of eight main factors affecting and controlling outburst, including gas pressure, coal mechanical strength, comprehensive characteristic coefficient of coal fragmentation, the permeability coefficient of coal, comprehensive characteristic coefficient of coal seam bifurcation and combination, comprehensive characteristic coefficient of coal thickness and coal thickness change, fault complexity coefficients and interlayer sliding comprehensive characteristic coefficient. The geological data affecting coal and gas outbursts are analyzed and calculated scientifically so that the gas geological data can be updated in time, and the change of gas geological laws is presented dynamically, so as to guide the mine to predict the gas disaster more scientifically and reliably.

Performance Enhancement of Distributed System Using HDFS Federation and Sharding

In today's world, 2.5 exabytes of data are generated and processed by the IT industry and different organizations. Processing and managing such a massive volume of Big data is challenging. Hadoop(HDFS) is widely used framework for processing Bigdata. The limitation of HDFS is it can scale only the datanodes and not namenode where the metadata is managed. There is no scaling of name nodes in the present Hadoop architecture, so managing the metadata of the exabyte scale becomes an essential and challenging leading to tightly coupled block storage, namespace scalability, and performance bottleneck issues. The proposed approach dynamic federated metadata management (DFMM) architecture manages the metadata in the name node. DFMM configures multiple name nodes using HDFS federation technique, and metadata is partitioned into numerous name nodes using sharding technique. Sharding manages the metadata using locality-preserving hashing and consistent hashing methods. Locality-preserving hashing stores metadata in the most appropriate name node using the file's parent directory, and metadata is dynamically distributed among Name nodes to maintain load balancing. DFMM also maintains replicas for the high availability and reliability of data. Results showcase the superiority of the proposed DFMM architecture combined with the sharding technique is highly scalable and manages metadata efficiently, and can store up to 1 billion files with a namespace size of 400 GB, enhancing the throughput by 21% and is highly scalable as compared to existing techniques. The comparative study analysis of existing HDFS, and proposed DFMM architecture combined with sharding enhances the performance and manages the metadata more efficiently.

Big data analytics-based approach for robust, flexible and sustainable collaborative networked enterprises

Networked enterprises (NEs) in the current business are constantly under pressure from stakeholders and government restrictions to encourage ethical and transparent behavior in using natural resources, and their impacts on nearby and global ecosystems, people, and communities. In addition, NEs face vulnerable economical challenges including, market changes, personalized consumer trends, as well as, environmental and social restrictions. In this context, this paper addresses the problem of sustainable NEs vulnerabilities. To do so, a big data analytics-based approach is developed to drive sustainable NEs flexibility and robustness. More specifically, flexibility refers to the network’s ability to respond quickly to changes and risks. While robustness concerns the development of optimum and long-term strategies enabling the network to cope with severe environmental risks and economical costs. Moreover, even if the literature is rich with Big Data models and frameworks developed for sustainable enterprises, there is a real need to scale and extend existing models to cover all sustainability pillars (i.e., social, environmental, and economical) and sustainable value creation (SVC). Accordingly, flexibility and robustness coupling with big data analytics (BDA) levels (i.e. descriptive analytics, diagnostic analytics, predictive analytics, prescriptive analytics) will enable NEs to grow sustainability in order to create sustainable value. Finally, to demonstrate the applicability of the developed approach, the corporate environmental impact (CEI) database is used to evaluate the sustainable development goals (SDGs) of NEs. The obtained numerical results show the efficiency of our approach.

A Novel Approach to Design Distribution Preserving Framework for Big Data

In several fields like financial dealing, industry, business, medicine, et cetera, Big Data (BD) has been utilized extensively, which is nothing but a collection of a huge amount of data. However, it is highly complicated along with time-consuming to process a massive amount of data. Thus, to design the Distribution Preserving Framework for BD, a novel methodology has been proposed utilizing Manhattan Distance (MD)-centered Partition Around Medoid (MD–PAM) along with Conjugate Gradient Artificial Neural Network (CG-ANN), which undergoes various steps to reduce the complications of BD. Firstly, the data are processed in the pre-processing phase by mitigating the data repetition utilizing the map-reduce function; subsequently, the missing data are handled by substituting or by ignoring the missed values. After that, the data are transmuted into a normalized form. Next, to enhance the classification performance, the data’s dimensionalities are minimized by employing Gaussian Kernel (GK)-Fisher Discriminant Analysis (GK-FDA). Afterwards, the processed data is submitted to the partitioning phase after transmuting it into a structured format. In the partition phase, by utilizing the MD-PAM, the data are partitioned along with grouped into a cluster. Lastly, by employing CG-ANN, the data are classified in the classification phase so that the needed data can be effortlessly retrieved by the user. To analogize the outcomes of the CG-ANN with the prevailing methodologies, the NSL-KDD openly accessible datasets are utilized. The experiential outcomes displayed that an efficient result along with a reduced computation cost was shown by the proposed CG-ANN. The proposed work outperforms well in terms of accuracy, sensitivity and specificity than the existing systems.

Business marketing analysis of agricultural products on online platforms based on big data

In recent years, online platform shopping has gradually become a mainstream shopping method for the public. And with the continuous development of China’s poverty alleviation cause in rural areas, marketing of agricultural products has al-so been added to online marketing, but in the process of development, various unsuitable situations have emerged, and data cannot be combined with products, resulting in the further development of agricultural products in this marketing mode has been restricted. Nowadays has entered the big data era and the market-ing of big data has appeared in many industries. This paper, through a systematic review of domestic and foreign literature, uncovers a basic framework of big data for agricultural products in online platform marketing, explaining the basic defini-tion and characteristics of big data, and elaborating the application. Although there is logistic regression model to truly realize big data in agricultural products online platform. The application of big data in agricultural products online plat-form marketing is explained.

A Comprehensive Review on Big Data for Industries: Challenges and Opportunities

Technological advancements in large industries like power, minerals, and manufacturing are generating massive data every second. Big data techniques have opened up numerous opportunities to utilize massive datasets in several effective ways to improve the efficacy of related industries. This paper presents a review of big data technologies used in the power, mineral, and manufacturing industries for various purposes. We analyze the meta-data of the collected papers before reviewing and selecting papers by applying selection criteria and paper quality assessment strategy. Then we propose a taxonomy of big data application areas in the power, mineral, and manufacturing industries. We have studied current big data architectures and techniques implemented in industry sectors and have uncovered the big data research gaps in industry sectors. To address the gaps, we point out some relevant research questions and, to answer the questions, we make some future research recommendations that might explore interesting research ideas for building a big data-driven industry. As the careful use of big data benefits every other industry sector; hence, supportive big data frameworks need to be developed to speed up the big data analysis process. Proper multi-dimensional big data assessment is also needed to take into account for serving effective data analysis tasks. Industry automation is also heavily influenced by the proper utilization of big data. While an intelligent agent can make many processes and heavy production loads in the manufacturing industry, it can work in a risky environment such as mines efficiently. To train agents for working in a specific environment big data can be used.

A Secure Big Data Storage Framework Based on Blockchain Consensus Mechanism With Flexible Finality

Data security and integrity are becoming increasingly important as the volume of data being created and stored grows. A controlled third party that provides most of the existing big data security systems makes them susceptible to several security risks. By resolving current technology challenges, including scalability, non-tampering, trustworthiness, data governance, and transparency, blockchain technology plays a vital role and has a significant potential to safeguard personal information. Therefore, this work focuses on addressing real-time big data storage issues based on a transdisciplinary research approach. This study introduces a brand-new approach to big data storage security that leverages blockchain technology and applies highway protocol to generate new blocks. The proposed highway protocol works based on the flexible finality condition to overcome issues of baseline models. The highway allows blocks to run the consensus mechanism to configure security thresholds more freely. The proposed protocol also allows blocks with lower thresholds to reach finality more quickly than blocks requiring greater degrees of confidence. Therefore, the proposed big data framework can dynamically control data manipulation and continuously support individuals to participate in the data-sharing process. A comparison was performed with the number of data requests in terms of hit ratio, and a highway protocol provides better results than baseline models. The proposed model provides a data processing period of 13 to 30 ms and an energy consumption of 32 to 41 mJ.

A Fully Streaming Big Data Framework for Cyber Security Based on Optimized Deep Learning Algorithm

A Fully Streaming Big Data Framework for Cyber Security Based on Optimized Deep Learning Algorithm

AN OVERVIEW OF HADOOP JOB SCHEDULING ALGORITHMS FOR BIG DATA

Rapid advancements in Big data systems have occurred over the last several decades. The significant element for attaining high performance is "Job Scheduling" in Big data systems which requires more utmost attention to resolve some challenges of scheduling. To obtain higher performance when processing the big data, proper scheduling is required. Apache Hadoop is most commonly used to manage immense data volumes in an efficient way and also proficient in handling the issues associated with job scheduling. To improve performance of big data systems, we significantly analyzed various Hadoop job scheduling algorithms. To get an overall idea about the scheduling algorithm, this paper presents a rigorous background. This paper made an overview on the fundamental architecture of Hadoop Big data framework, job scheduling and its issues, then reviewed and compared the most important and fundamental Hadoop job scheduling algorithms. In addition, this paper includes a review of other improved algorithms. The primary objective is to present an overview of various scheduling algorithms to improve performance when analyzing big data. This study will also provide appropriate direction in terms of job scheduling algorithm to the researcher according to which characteristics are most significant.

A Distributed Big Data Analytics Architecture for Vehicle Sensor Data.

The unceasingly increasing needs for data acquisition, storage and analysis in transportation systems have led to the adoption of new technologies and methods in order to provide efficient and reliable solutions. Both highways and vehicles, nowadays, host a vast variety of sensors collecting different types of highly fluctuating data such as speed, acceleration, direction, and so on. From the vast volume and variety of these data emerges the need for the employment of big data techniques and analytics in the context of state-of-the-art intelligent transportation systems (ITS). Moreover, the scalability needs of fleet and traffic management systems point to the direction of designing and deploying distributed architecture solutions that can be expanded in order to avoid technological and/or technical entrapments. Based on the needs and gaps detected in the literature as well as the available technologies for data gathering, storage and analysis for ITS, the aim of this study is to provide a distributed architecture platform to address these deficiencies. The architectural design of the system proposed, engages big data frameworks and tools (e.g., NoSQL Mongo DB, Apache Hadoop, etc.) as well as analytics tools (e.g., Apache Spark). The main contribution of this study is the introduction of a holistic platform that can be used for the needs of the ITS domain offering continuous collection, storage and data analysis capabilities. To achieve that, different modules of state-of-the-art methods and tools were utilized and combined in a unified platform that supports the entire cycle of data acquisition, storage and analysis in a single point. This leads to a complete solution for ITS applications which lifts the limitations imposed in legacy and current systems by the vast amounts of rapidly changing data, while offering a reliable system for acquisition, storage as well as timely analysis and reporting capabilities of these data.

Retraction Note: MapReduce-based big data framework using modified artificial neural network classifier for diabetic chronic disease prediction

Retraction Note: MapReduce-based big data framework using modified artificial neural network classifier for diabetic chronic disease prediction

Development of a Rice Plant Disease Classification Model in Big Data Environment.

More than the half of the global population consume rice as their primary energy source. Therefore, this work focused on the development of a prediction model to minimize agricultural loss in the paddy field. Initially, rice plant diseases, along with their images, were captured. Then, a big data framework was used to encounter a large dataset. In this work, at first, feature extraction process is applied on the data and after that feature selection is also applied to obtain the reduced data with important features which is used as the input to the classification model. For the rice disease datasets, features based on color, shape, position, and texture are extracted from the infected rice plant images and a rough set theory-based feature selection method is used for the feature selection job. For the classification task, ensemble classification methods have been implemented in a map reduce framework for the development of the efficient disease prediction model. The results on the collected disease data show the efficiency of the proposed model.

A Framework to Build a Big Data Ecosystem Oriented to the Collaborative Networked Organization

A Collaborative Networked Organization (CNO) is a set of entities that operate in heterogeneous contexts and aim to collaborate to take advantage of a business opportunity or solve a problem. Big data allows CNOs to be more competitive by improving their strategy, management and business processes. To support the development of big data ecosystems in CNOs, several frameworks have been reported in the literature. However, these frameworks limit their application to a specific CNO manifestation and cannot conduct intelligent processing of big data to support decision making at the CNO. This paper makes two main contributions: (1) the proposal of a metaframework to analyze existing and future frameworks for the development of big data ecosystems in CNOs and (2) to show the Collaborative Networked Organizations–big data (CNO-BD) framework, which includes guidelines, tools, techniques, conceptual solutions and good practices for the building of a big data ecosystem in different kinds of Collaborative Networked Organizations, overcoming the weaknesses of previous issues. The CNO-BD framework consists of seven dimensions: levels, approaches, data fusion, interoperability, data sources, big data assurance and programmable modules. The framework was validated through expert assessment and a case study.