Big Data Processing Tools Research Articles

Drought Prediction using Geo-Spatial Big Data

The digital world with digital processing, requires large storage space. The continuous explosion of the data such as text, image, audio, video, data centers and backup data lead to several problem in both storage and retrieval process. In this paper drought analysis and prediction is done using big data processing tools such as Hadoop and hive which can increase high. Previously to analyze and predict drought, traditional techniques such as AVISO model is used which is complex to process, requires more processing time, cannot process huge data and also has more security issues like malware in the database, abuse of privileges, etc. The system proposed in this paper can process huge data and has more processing speed. Here, drought analysis and prediction is carried out. To analyze drought dataset with more than ten lakhs are processed and drought type is found using map-reduce algorithm which maps and reduces the data using numerical summarization. Drought types such as D0, D1, D2,D3 D4 are analyzed to obtain reduced output. The obtained drought type are clustered using hive. To predict drought, random forest algorithm acts as an predictor which creates multiple decision trees and finds the best split among them. Finally, the predicted output is visualized using the time series model. The tools used in this paper include Hadoop and hive which can process huge data and it is the solution of Big Data. Hadoop is an open-source software framework for storing data and processing them efficiently, even if the data size is very huge. Hadoop uses Hadoop Distributed File System(HDFS) for storage and MapReduce for processing the data. Hive is a query processing tool which is built on top of Hadoop. It is a Structured query language(SQL)-like language called HiveQL (HQL). In this paper hive is used to cluster the data obtained from MapReduce. Thus using Big Data improves performance more than 50% compared to traditional system

Industry 4.0 based process data analytics platform: A waste-to-energy plant case study

Industry 4.0 and Industrial Internet of Things (IIoT) technologies are rapidly fueling data and software solutions driven digitalization in many fields notably in industrial automation and manufacturing systems. Among the several benefits offered by these technologies, is the infrastructure for harnessing big-data, machine learning (ML) and cloud computing software tools, for instance in designing advanced data analytics platforms. Although, this is an area of increased interest, the information concerning the implementation of data analytics in the context of Industry4.0 is scarcely available in scientific literature. Therefore, this work presents a process data analytics platform built around the concept of industry 4.0. The platform utilizes the state-of-the-art IIoT platforms, ML algorithms and big-data software tools. The platform emphasizes the use of ML methods for process data analytics while leveraging big-data processing tools and taking advantage of the currently available industrial grade cloud computing platforms. The industrial applicability of the platform was demonstrated by the development of soft sensors for use in a waste-to-energy (WTE) plant. In the case study, the work studied data-driven soft sensors to predict syngas heating value and hot flue gas temperature. Among the studied data-driven methods, the neural network-based NARX model demonstrated better performance in the prediction of both syngas heating value and flue gas temperature. The modeling results showed that, in cases where process knowledge about the process phenomena at hand is limited, data-driven soft sensors are useful tools for predictive data analytics.

Research based on big data analysis of medical industry

In recent decades, institutions in every country and every place have produced a large amount of structured, unstructured and semi-structured data with different (heterogeneous) structures. The healthcare sector faces the need to manage big data from a variety of sources known for generating large amounts of heterogeneous data. In the medical and health industry, various big data analysis tools and technologies have been developed to deal with these massive data. In this article, we will discuss the impact of big data on healthcare and health, as well as the various big data processing tools available in the Hadoop system. We also explored the conceptual architecture of medical big data analysis, including data collection histories from different branches, genomic databases, electronic health records, text/image, and clinical decision support systems.

Distributed kernel gradient descent algorithm for minimum error entropy principle

Distributed learning based on the divide and conquer approach is a powerful tool for big data processing. We introduce a distributed kernel gradient descent algorithm for the minimum error entropy principle and analyze its convergence. We show that the L 2 error decays at a minimax optimal rate under some mild conditions. As a tool we establish some concentration inequalities for U-statistics which play pivotal roles in our error analysis.

The Rising Role of Big Data Analytics and IoT in Disaster Management: Recent Advances, Taxonomy and Prospects

The recent development of big data analytics (BDA) and the Internet of Things (IoT) technologies create a huge opportunity for both disaster management systems and disaster-related authorities (emergency responders, police, public health, and fire departments) to acquire state-of-the-art assistance and improved insights for accurate and timely decision-making. The motivation behind this research is to pave the way for effective utilization of the available opportunities that the BDA and IoT collaboratively offer to predict, understand and monitor disaster situations. Most of the conventional disaster management systems lack the support for multiple new data sources and real-time big data processing tools that can assist decision makers with quick and accurate results. This paper highlights the importance of BDA and IoT for disaster management and investigates recent studies directed towards the same. We classify a thematic taxonomy with several related attributes and inspect the prevalent solutions to propose a conceptual reference model for the deployment of BDA- and IoT-based disaster management environments. The reference model with its proposed integrated parameters can provide guidelines to harvest, transmit, manage, and analyze disaster data from various data sources to deliver updated and valuable information for disaster management. We also enumerate some important use cases from a disaster management perspective. Finally, we highlight the main research challenges that need to be addressed in such an important field of research.

New Approach based on Model Driven Engineering for Processing Complex SPARQL Queries on Hive

Semantic web technologies are increasingly used in different domains. The core technology of the Semantic Web is the RDF standard. Today with the growth of RDF data it requires systems capable of handling these large volumes of data and responding to very complex requests at the join level, Several RDF data processing systems have been proposed, but are not dedicated to handling complex SPARQL queries. in this paper we present a new approach based on model driven engineering for processing complex SPARQL queries using one of the big data processing tools named Hive. We evaluate our system using three datasets from LUBM Benchmark. The results of this evaluation show the performance, and the scalability of our approach, also give very interesting results when it is compared with existing works.

Big data processing tools: An experimental performance evaluation

Big Data is currently a hot topic of research and development across several business areas mainly due to recent innovations in information and communication technologies. One of the main challenges of Big Data relates to how one should efficiently handle massive volumes of complex data. Due to the notorious complexity of the data that can be collected from multiple sources, usually motivated by increasing data volumes gathered at high velocity, efficient processing mechanisms are needed for data analysis purposes. Motivated by the rapid growth in technology, development of tools, and frameworks for Big Data, there is much discussion about Big Data querying tools and, specifically, those that are more appropriated for specific analytical needs. This paper describes and evaluates the following popular Big Data processing tools: Drill, HAWQ, Hive, Impala, Presto, and Spark. An experimental evaluation using the Transaction Processing Council (TPC‐H) benchmark is presented and discussed, highlighting the performance of each tool, according to different workloads and query types.This article is categorized under: Technologies > Computer Architectures for Data Mining Fundamental Concepts of Data and Knowledge > Big Data Mining Technologies > Data Preprocessing Application Areas > Data Mining Software Tools

CONTROL MODEL OF DATA STREAM TRANSMITTED OVER A NETWORK BASED ON PROXYING TECHNOLOGY

Network traffic control model is described in the work, including data mining modules transmitted in the network as well as further qualified data analysis module based on artificial intelligence methods and means, namely recursive and convolutional neural networks. The topicality of work is proved by the intensive scientific research conducted in the field of information security, big data intelligent data processing tools, and stipulated by growing necessity to limit access of children to aggressive information, which can impact on the child’s psychoemotional state. Particular attention is paid in the article to proxy-server development for HTTP query receipt, search for a match in white and black lists and decision making as to data legality.

Performance evaluation of intrusion detection based on machine learning using Apache Spark

Nowadays, network intrusion is considered as one of the major concerns in network communications. Thus, the developed network intrusion detection systems aim to identify attacks or malicious activities in a network environment. Various methods have been already proposed for finding an effective and efficient solution to detect and prevent intrusion in the network, ensuring network security and privacy. Machine learning is an effective analysis framework to detect any anomalous events occurred in the network traffic flow. Based on this framework, the paper in hand evaluates the performance of four well-known classification algorithms; SVM, Naïve Bayes, Decision Tree and Random Forest using Apache Spark, a big data processing tool for intrusion detection in network traffic. The overall performance comparison is evaluated in terms of detection accuracy, building time and prediction time. Experimental results on UNSW-NB15, a recent public dataset for network intrusion detection, show an important advantage for Random Forest classifier among other well-known classifiers in terms of detection accuracy and prediction time, using the complete dataset with all 42 features.

Scaling up Bayesian variational inference using distributed computing clusters

In this paper we present an approach for scaling up Bayesian learning using variational methods by exploiting distributed computing clusters managed by modern big data processing tools like Apache Spark or Apache Flink, which efficiently support iterative map-reduce operations. Our approach is defined as a distributed projected natural gradient ascent algorithm, has excellent convergence properties, and covers a wide range of conjugate exponential family models. We evaluate the proposed algorithm on three real-world datasets from different domains (the Pubmed abstracts dataset, a GPS trajectory dataset, and a financial dataset) and using several models (LDA, factor analysis, mixture of Gaussians and linear regression models). Our approach compares favorably to stochastic variational inference and streaming variational Bayes, two of the main current proposals for scaling up variational methods. For the scalability analysis, we evaluate our approach over a network with more than one billion nodes and approx. 75% latent variables using a computer cluster with 128 processing units (AWS). The proposed methods are released as part of an open-source toolbox for scalable probabilistic machine learning (http://www.amidsttoolbox.com) Masegosa et al. (2017) [29].

A Real-time Security Situation Assessment System Designed for Cloud Platform

Security situation assessment is an effective way to analyze the situation of cloud platform, which helps administrator understand the current cloud platform security status and develop policies to reduce risks in time. However, the existing researches for security situation assessment mostly focus on network security situation assessment. The proposed methods for network are not so suitable for cloud platform. For the first time, we propose a security situation assessment system for cloud platform. In consideration of diversity, heterogeneous, multi-source of data in cloud platform security assessment, we use the cloud computing security indicator system to construct a multi-source heterogeneous data fusion method which is used to assess the situation of cloud platform. In detail, we utilize big data processing tools and real-time data processing tools to transform the multi-source and heterogeneous data of cloud platform into the situational indicators of each protection level, which are further used to calculate the overall security situation in real time. The accuracy and practicability of the security situation assessment system are verified on real-word cloud platform in our experiments.

Load feedback-based resource scheduling and dynamic migration-based data locality for virtual hadoop clusters in openstack-based clouds

With cloud computing technology becoming more mature, it is essential to combine the big data processing tool Hadoop with the Infrastructure as a Service (IaaS) cloud platform. In this study, we first propose a new Dynamic Hadoop Cluster on IaaS (DHCI) architecture, which includes four key modules: monitoring, scheduling, Virtual Machine (VM) management, and VM migration modules. The load of both physical hosts and VMs is collected by the monitoring module and can be used to design resource scheduling and data locality solutions. Second, we present a simple load feedback-based resource scheduling scheme. The resource allocation can be avoided on overburdened physical hosts or the strong scalability of virtual cluster can be achieved by fluctuating the number of VMs. To improve the flexibility, we adopt the separated deployment of the computation and storage VMs in the DHCI architecture, which negatively impacts the data locality. Third, we reuse the method of VM migration and propose a dynamic migration-based data locality scheme using parallel computing entropy. We migrate the computation nodes to different host(s) or rack(s) where the corresponding storage nodes are deployed to satisfy the requirement of data locality. We evaluate our solutions in a realistic scenario based on OpenStack. Substantial experimental results demonstrate the effectiveness of our solutions that contribute to balance the workload and performance improvement, even under heavy-loaded cloud system conditions.

SOR: An optimized semantic ontology retrieval algorithm for heterogeneous multimedia big data

Semantic information can express the search intentions of users, and this approach has become an important tool in the field of information retrieval. To support semantic-based multimedia retrieval in big data environment, this paper presents an optimized algorithm called semantic ontology retrieval (SOR), which uses big data processing tools to store and retrieve ontologies from heterogeneous multimedia data. First, the background of semantic extraction and ontology representation for multimedia big data are addressed. Second, the methodology of SOR, including the model definition and retrieval algorithm, is proposed. Third, for parallel processing SOR in distributed nodes, a MapReduce-based retrieval framework is presented. Finally, to achieve high retrieval precision and good user experience, a user feedback scheme is designed. The experimental results illustrate that SOR is suitable for semantic-based retrieval for heterogeneous multimedia big data.

Remote health care cyber‐physical system: quality of service (QoS) challenges and opportunities

There is a growing emphasis to find alternative non-traditional ways to manage patients to ease the burden on health care services largely fuelled by a growing demand from sections of population that is ageing. In-home remote patient monitoring applications harnessing technological advancements in the area of Internet of things (IoT), semantic web, data analytics, and cloud computing have emerged as viable alternatives. However, such applications generate large amounts of real-time data in terms of volume, velocity, and variety thus making it a big data problem. Hence, the challenge is how to combine and analyse such data with historical patient data to obtain meaningful diagnoses suggestions within acceptable time frames (considering quality of service (QoS)). Despite the evolution of big data processing technologies (e.g. Hadoop) and scalable infrastructure (e.g. clouds), there remains a significant gap in the areas of heterogeneous data collection, real-time patient monitoring, and automated decision support (semantic reasoning) based on well-defined QoS constraints. In this study, the authors review the state-of-the-art in enabling QoS for remote health care applications. In particular, they investigate the QoS challenges required to meet the analysis and inferencing needs of such applications and to overcome the limitations of existing big data processing tools.

Introduction of National Environmental Health Tracking Project in China

Introduction Multiple physical environmental pollutions have been demostrated to be an important driver of health damage for Chinese population. It is urgent to require an integrated response to control complex pollution to protect public health in China. However, the weak link between the current environmental monitoring and health surveillance could not effectively support the demands from data-based decision-making of government, abundent data of environmetnal health professional, and risk communication of public. Methods Thus we are proposing a long-term national environmental health tracking project to collect advance data integrating environmental hazards and health effects at county level. The framework of this tracking project consists of three parts: (1) tracking indicators and data surveillance; (2) data processing, including quality check, data cleaning, data storage, data analysis; (3) data application, sharing data with different stakeholders, such as supporting government for risk assessment and early warning, provideing full data sharing with researchers, and providing information access to the public. To realize the environmental health tracking, on one hand, we need to integrate the on-going surveillance work, as well as collect existed opening dataset invloving environmental health factors in China; on the other hand, we need build up facilitate cooperation with local CDCs and pilot surveillance sites at county level. Results Possible achievements after 5-10 years development of the integrated tracking system may include: (1) environmental health database; (2) big data processing technologies and tools; (3) data sharing services in different ways; (4) risk assessment and early warning system based on continous data surveillance. Conclusions This tracking project is an advanced network to fill the current gap between science and public health practice, aiming at improving better protection for public health.

The methods and tools for Big data processing in territorial administration systems

Подано визначення Великих даних та описано основні характеристики. Проаналізовано математичні засоби подання та опрацювання Великих даних та визначено їх обмеження. Визначено формальний опис Великих даних. Подано моделі асоціацій між сутностями та характеристиками для різних категорій Nosql баз даних. Розроблено метод обміну різнотипними даними та приведення реляційних даних до моделі "сутність-характеристика". Спроектовано схему даних регіону. Апробовано розроблені методи та алгоритми.

Extending Science Gateway Frameworks to Support Big Data Applications in the Cloud

Cloud computing offers massive scalability and elasticity required by many scientific and commercial applications. Combining the computational and data handling capabilities of clouds with parallel processing also has the potential to tackle Big Data problems efficiently. Science gateway frameworks and workflow systems enable application developers to implement complex applications and make these available for end-users via simple graphical user interfaces. The integration of such frameworks with Big Data processing tools on the cloud opens new opportunities for application developers. This paper investigates how workflow systems and science gateways can be extended with Big Data processing capabilities. A generic approach based on infrastructure aware workflows is suggested and a proof of concept is implemented based on the WS-PGRADE/gUSE science gateway framework and its integration with the Hadoop parallel data processing solution based on the MapReduce paradigm in the cloud. The provided analysis demonstrates that the methods described to integrate Big Data processing with workflows and science gateways work well in different cloud infrastructures and application scenarios, and can be used to create massively parallel applications for scientific analysis of Big Data.

Big Data and Large Scale Methods In Cloud Computing

In this paper I want to highlight what we believe to be the key technology dimensions for evaluating data management solutions. This paper offers and explore define what is meant by big data, We review analytics techniques for text, audio, video, and social media data, We make the case for new statistical techniques for big data, We highlight the expected future developments in big data analytics, Examines cloud data management architectures and Covers Big Data analytics and visualization.It not only considers data management and analytics for vast amounts of unstructured data but also Explores clustering, classification, and link analysis of Big Data. Large Scale and Big Data: Processing and Management provides readers with a central source of reference on the data management techniques currently available for large-scale data processing. that addresses the fundamental challenges associated with Big Data processing tools and techniques across a range of computing environments.The basic concepts and tools of large-scale Big Data processing and cloud computing can be view here and hope every one will get basic idea about these concepts. It also provides an overview of different programming models and cloud-based deployment models that examines the usage of advanced Big Data processing techniques in different domains, including semantic web, graph processing, and stream processing Big Data such as consistency management, privacy, and security.

SurvSurf: human retrieval on large surveillance video data

The volume of surveillance videos is increasing rapidly, where humans are the major objects of interest. Rapid human retrieval in surveillance videos is therefore desirable and applicable to a broad spectrum of applications. Existing big data processing tools that mainly target textual data cannot be applied directly for timely processing of large video data due to three main challenges: videos are more data-intensive than textual data; visual operations have higher computational complexity than textual operations; and traditional segmentation may damage video data's continuous semantics. In this paper, we design SurvSurf, a human retrieval system on large surveillance video data that exploits characteristics of these data and big data processing tools. We propose using motion information contained in videos for video data segmentation. The basic data unit after segmentation is called M-clip. M-clips help remove redundant video contents and reduce data volumes. We use the MapReduce framework to process M-clips in parallel for human detection and appearance/motion feature extraction. We further accelerate vision algorithms by processing only sub-areas with significant motion vectors rather than entire frames. In addition, we design a distributed data store called V-BigTable to structuralize M-clips' semantic information. V-BigTable enables efficient retrieval on a huge amount of M-clips. We implement the system on Hadoop and HBase. Experimental results show that our system outperforms basic solutions by one order of magnitude in computational time with satisfactory human retrieval accuracy.

A Framework for Fast and Efficient Cyber Security Network Intrusion Detection Using Apache Spark

Due to increase in internet based services, the size of network traffic data has become so large and complex that it is very difficult to process with the traditional data processing tools. Fast and efficient cyber security intrusion detection is a very challenging problem due to big and complex nature of network traffic data. A realistic cyber security intrusion detection system should be able to process large size of network traffic data as fast as possible in order to detect the malicious traffic as early as possible. This paper used Apache Spark, a big data processing tool for processing the large size of network traffic data. In this paper, we have proposed a framework in which first a well-known feature selection algorithm is employed for selecting the most important features and then classification based intrusion detection method is used for fast and efficient detection of intrusion in the massive network traffic. In this work, we have used two well-known feature selection algorithm, namely, correlation based feature selection and Chi-squared feature selection and five well known classification based intrusion detection methods, namely, Logistic regression, Support vector machines, Random forest, Gradient Boosted Decision trees & Naive Bayes. A real time DARPA's KDD’99 data set is used to validate the proposed framework and performance comparison of classification based intrusion detection schemes are evaluated in terms of training time, prediction time, accuracy, sensitivity and specificity.