Big Data Processing Systems Research Articles

Benchmarking big data systems: A survey

With the enormous growth on the availability and usage of Big Data storage and processing systems, it has become essential to assess the various performance aspects of these systems so that we can carefully understand their strong and weak aspects. In practice, currently, when an individual/enterprise aims to develop a Big Data storage and processing solution for harnessing the knowledge inside their data, they will get challenged by the availability of several frameworks from which they need to select. This is a challenging task which needs to directed by with good knowledge about various perspectives of such systems. Additionally, the choice normally vary from one scenario to another according to the essential needs of the application. In practice, there is no single benchmark study which can cover the different types of big data processing requirements, systems, application scenarios and metrics. Several benchmarks and benchmarking studies have been developed where each study focuses on some representative type of frameworks and only consider some aspects to cover. In this article, we provide a comprehensive survey and analysis of the state-of-the-art of benchmarking the different types of big data systems (e.g., NoSQL databases, Big SQL engines, Big Streaming engines, Big Graph Processing engines, Big Machine/Deep Learning engines). Additionally, we highlight some of the significant open challenges and missing requirements of current benchmarks of big data systems with suggestions of directions for future extensions and improvements.

GSV2SVF-an interactive GIS tool for sky, tree and building view factor estimation from street view photographs

Sky View Factor (SVF) is a commonly used indicator of urban geometry. The availability of street-level SVF measurements has been fairly limited due to the high costs of field survey. The Google Street View (GSV) serves a massive storage of panorama data that can be utilized to obtain SVF measurements. Yet, automatic extraction of SVFs from panoramas is a complicated process that involves multiple sophisticated computation technologies including machine learning, big image data processing, SVF estimation and geographic information systems (GIS), which constitute major hurdles for the end users. In this light, we developed an easy-to-use GIS-integrated tool (GSV2SVF) to streamline the workflow of extracting SVFs from GSV images and therefore making this vast treasure trove of information conveniently available to everyone at a mouse click. As by-products in addition to the SVF, the results obtained from each GSV panorama are accompanied with the tree view factor (TVF) and the building view factor (BVF), which together can provide a more holistic characterization of the outdoor built environment. GSV2SVF is freely available with source code at https://github.com/jian9695/GSV2SVF. A video is available at https://github.com/jian9695/GSV2SVF/blob/master/Video.mp4 and https://youtu.be/k00wCnuzuvE.

Hierarchical Analysis of Loops With Relaxed Abstract Transformers

Numerical computation is often involved in software of embedded control systems, cyber-physical systems, artificial neural network systems, big data processing systems, etc. Automatically discovering numerical loop invariants is fundamental for checking the safety of such software. Abstract interpretation provides a framework to automatically discover sound invariants but which may be not precise enough due to over-approximations. One major source of precision loss is due to the limited linear expressiveness of most widely used numerical abstract domains and the widening operation. This becomes more serious when analyzing all variables simultaneously as a whole for programs that involve nonlinear behaviors. Based on the observation that the dependency among variables in a loop can be hierarchical, in this article, we propose a hierarchical static analysis to analyze a loop by utilizing relaxed abstract transformers. The main idea is to first partition all variables involved in a loop into different hierarchical layers, then compute invariants over the variables layer by layer in a bottom-up manner. During the iterative process, the computed invariants over lower layer variables are then used to relax transfer functions when analyzing the higher layer variables. One benefit of our method lies in that it can generate linear invariants to soundly enclose nonlinear behaviors in a loop. Finally, we present encouraging experimental results on benchmark programs involving nonlinear behaviors.

Edge computing health model using P2P-based deep neural networks

Currently, significant amounts of IoT data are being collected as big data for deep neural network learning algorithms that are used to extract meaningful information from big data and applied to various areas. However, a neural network’s over-fitting problem and rising computational costs associated with high levels of time complexity are obstacles to deep learning computations. Because of these problems, response delays are prevalent in big data learning processes and data extraction processes that use deep neural networks, which exponentially increase the cost of data extraction. Therefore, the amount of computation must be controlled so that data processing models can be used practically, and various dedicated devices can be used to process big data, including methods such as distributed processing methods. In general, a P2P method is a typical distributed processing method. P2P is based on traffic distribution and cooperation, and edge computing is a method that increases efficiency by locally processing large volumes of data produced and only transmitting essential information. If P2P method-based edge computing techniques are used, an effective parallel processing model can be constructed, reducing the computing requirements of a central server and the load of a network. In this paper, we propose an edge computing health model using P2P-based deep neural networks. The proposed method is used to process health big data in edge computing. To construct the model, multiple edge nodes are required, and the edge node modules are deployed at locations where health data are produced and directly connected to the deep neural network model. At the edge nodes, a modularized deep neural network is constructed, thereby implementing a parallel big data processing system. In addition, a single server is required for gathering the results of a neural network model, and because the server only collects evaluation results and provides them to the users, the response time delay is improved. In this paper, to evaluate an improved response time, a regular server model and an edge computing health model are constructed separately, and positive numerical values are extracted from the experiments. The extraction results confirm that a combination of parallel processing models and deep neural network techniques can be used to distribute and process computing operations of big data size, ensuring an effective system for reducing response time delay.

A Survey on Geographically Distributed Big-Data Processing Using MapReduce

Hadoop and Spark are widely used distributed processing frameworks for large-scale data processing in an efficient and fault-tolerant manner on private or public clouds. These big-data processing systems are extensively used by many industries, e.g., Google, Facebook, and Amazon, for solving a large class of problems, e.g., search, clustering, log analysis, different types of join operations, matrix multiplication, pattern matching, and social network analysis. However, all these popular systems have a major drawback in terms of locally distributed computations, which prevent them in implementing geographically distributed data processing. The increasing amount of geographically distributed massive data is pushing industries and academia to rethink the current big-data processing systems. The novel frameworks, which will be beyond state-of-the-art architectures and technologies involved in the current system, are expected to process geographically distributed data at their locations without moving entire raw datasets to a single location. In this paper, we investigate and discuss challenges and requirements in designing geographically distributed data processing frameworks and protocols. We classify and study batch processing (MapReduce-based systems), stream processing (Spark-based systems), and SQL-style processing geo-distributed frameworks, models, and algorithms with their overhead issues.

Big SQL systems: an experimental evaluation

Recently, Big Data systems have been gaining increasing popularity on handling the massive amounts of data that are continuously generated in our digital world. While the Hadoop framework has pioneered the area of Big Data processing systems, it had clear performance limitations on providing the best performance of processing massive amounts of structured data. In addition, practically, many users of the big data systems face some challenges on dealing with the APIs and the low level programming abstractions of the Big Data System and they would prefer to use SQL (in which they are more proficient) as a high-level declarative language to express their tasks while leaving all of the execution optimization details to the backend engine. Thus, several systems have been designed and implemented to tackle these challenges by designing and implementing scalable query execution engines for processing massive structured data while supporting SQL interfaces. In this article, we present an extensive experimental study of four popular systems in this domain, namely, Apache Hive, SPARK SQL, Apache Impala and PrestoDB. In particular, we report and analyze the performance characteristics of these systems using three different benchmarks, namely, TPC-H, TPC-DS and TPCx-BB. Finally, we report a set of insights and important lessons that we have learned from conducting our experiments.

Load Balancing in Big Data Processing Systems

The article examines the load balancing methods used in the construction of big data processing systems. In the light of the widespread transition to Industry 4.0, the authors analyze the major trends in the development of science and technology, describe the existing balancing methods and suggest their recommendations to improve their effective use. The analysis has revealed the key features of load balancing algorithms. The study examines load balancers in terms of the four layers of the TCP/IP network model – data link, network, transport and application. Furthermore, the authors refer to the features of load balancing algorithms and formalize the relevant requirements. The study primarily focuses on the load balancing at the Internet layer. The authors suggest formulas for calculating the response time for various load balancing algorithms. A new method of load balancing is proposed for big data processing systems. The article presents a typical network topology. The solution involves the integration of large data methods into a load balancing system. To implement the load distribution in the server cluster, the authors use a processing cluster analyzing the server machines and managing the distribution of the load in the network, based on the received data. In the end, the authors suggest the load balancing algorithm.

D-Simplexed: Adaptive Delaunay Triangulation for Performance Modeling and Prediction on Big Data Analytics

Big Data processing systems (e.g., Spark) have a number of resource configuration parameters, such as memory size, CPU allocation, and the number of running nodes. Regular users and even expert administrators struggle to understand the mutual relation between different parameter configurations and the overall performance of the system. In this paper, we address this challenge by proposing a performance prediction framework, called d-Simplexed, to build performance models with varied configurable parameters on Spark. We take inspiration from the field of Computational Geometry to construct a d-dimensional mesh using Delaunay Triangulation over a selected set of features. From this mesh, we predict execution time for various feature configurations. To minimize the time and resources in building a bootstrap model with a large number of configuration values, we propose an adaptive sampling technique to allow us to collect as few training points as required. Our evaluation on a cluster of computers using WordCount, PageRank, Kmeans, and Join workloads in HiBench benchmarking suites shows that we can achieve less than 5% error rate for estimation accuracy by sampling less than 1% of data.

A Survey of Distributed Data Stream Processing Frameworks

Big data processing systems are evolving to be more stream oriented where each data record is processed as it arrives by distributed and low-latency computational frameworks on a continuous basis. As the stream processing technology matures and more organizations invest in digital transformations, new applications of stream analytics will be identified and implemented across a wide spectrum of industries. One of the challenges in developing a streaming analytics infrastructure is the difficulty in selecting the right stream processing framework for the different use cases. With a view to addressing this issue, in this paper we present a taxonomy, a comparative study of distributed data stream processing and analytics frameworks, and a critical review of representative open source (Storm, Spark Streaming, Flink, Kafka Streams) and commercial (IBM Streams) distributed data stream processing frameworks. The study also reports our ongoing study on a multilevel streaming analytics architecture that can serve as a guide for organizations and individuals planning to implement a real-time data stream processing and analytics framework.

Big Data Processing and Monitoring System based on Vehicle Data

Big Data Processing and Monitoring System based on Vehicle Data

Big Data in Smart-Cities: Current Research and Challenges

Smart-cities are an emerging paradigm containing heterogeneous network infrastructure, ubiquitous sensing devices, big-data processing and intelligent control systems. Their primary aim is to improve the quality of life of the citizens by providing intelligent services in a wide variety of aspects like transportation, healthcare, entertainment, environment, and energy. In order to provide such services, the role of big-data and its analysis is extremely important as it enables to obtain valuable insights into the large data generated by the smart-cities. In this article, we investigate the state-of-art research efforts directed towards big-data analytics in a smart-city context. Specifically, first we present a big-data centric taxonomy for the smart-cities to bring forth a generic overview of the importance of big-data paradigm in a smart-city environment. This is followed by the presentation of a top-level snapshot of the commonly used big-data analytical platforms. Due to the heterogeneity of data being collected by the smart-cities, often with conflicting processing requirements, suitable analytical techniques depending upon the data type are also suggested. In addition to this, a generic four-tier big-data framework comprising of the sensing hub, storage hub, processing hub and application hub is also proposed that can be applied in any smart-city context. This is complemented by providing the common big-data applications in a smart-city and presentation of ten selected case studies of smart-cities across the globe. Finally, the open challenges are highlighted in order to give future research directions.

Big Data in Smart-Cities: Current Research and Challenges

Smart-cities are an emerging paradigm containing heterogeneous network infrastructure, ubiquitous sensing devices, big-data processing and intelligent control systems. Their primary aim is to improve the quality of life of the citizens by providing intelligent services in a wide variety of aspects like transportation, healthcare, entertainment, environment, and energy. In order to provide such services, the role of big-data and its analysis is extremely important as it enables to obtain valuable insights into the large data generated by the smart-cities. In this article, we investigate the state-of-art research efforts directed towards big-data analytics in a smart-city context. Specifically, first we present a big-data centric taxonomy for the smart-cities to bring forth a generic overview of the importance of big-data paradigm in a smart-city environment. This is followed by the presentation of a top-level snapshot of the commonly used big-data analytical platforms. Due to the heterogeneity of data being collected by the smart-cities, often with conflicting processing requirements, suitable analytical techniques depending upon the data type are also suggested. In addition to this, a generic four-tier big-data framework comprising of the sensing hub, storage hub, processing hub and application hub is also proposed that can be applied in any smart-city context. This is complemented by providing the common big-data applications in a smart-city and presentation of ten selected case studies of smart-cities across the globe. Finally, the open challenges are highlighted in order to give future research directions.

Dynamic load balancing algorithm for large data flow in distributed complex networks

Abstract Information society brings convenience to people, but also produces a lot of data. Relational databases are not suitable for processing big data due to architecture defects. The most commonly used system to store and process large amounts of data is the NoSQL (Not only Structured Query Language) database. Obviously, it is very important to cooperate with these independent computers to accomplish processing tasks efficiently, which is the function of load balancing. This paper studies the commonly used NoSQL database and load balancing algorithms, and designs and implements a more efficient load balancing algorithm. By introducing the relationship between nodes and the children of their brother nodes, we reduce the height of the whole sorted binary tree. The time cost of the algorithm is reduced versus the commonly used weighted polling algorithm O(N) to O(log N), while the spatial cost remains unchanged. The equalization algorithm synthetically utilizes the characteristics of big data processing systems and has good performance. At the same time, the algorithm can quickly find the sub-optimal nodes when the optimal nodes have been occupied, so it is very suitable for load balancing in highly concurrent systems. Finally, the effectiveness of the proposed load balancing algorithm is verified by simulation.

High-Performance Geospatial Big Data Processing System Based on MapReduce

With the rapid development of Internet of Things (IoT) technologies, the increasing volume and diversity of sources of geospatial big data have created challenges in storing, managing, and processing data. In addition to the general characteristics of big data, the unique properties of spatial data make the handling of geospatial big data even more complicated. To facilitate users implementing geospatial big data applications in a MapReduce framework, several big data processing systems have extended the original Hadoop to support spatial properties. Most of those platforms, however, have included spatial functionalities by embedding them as a form of plug-in. Although offering a convenient way to add new features to an existing system, the plug-in has several limitations. In particular, while executing spatial and nonspatial operations by alternating between the existing system and the plug-in, additional read and write overheads have to be added to the workflow, significantly reducing performance efficiency. To address this issue, we have developed Marmot, a high-performance, geospatial big data processing system based on MapReduce. Marmot extends Hadoop at a low level to support seamless integration between spatial and nonspatial operations of a solid framework, allowing improved performance of geoprocessing workflow. This paper explains the overall architecture and data model of Marmot as well as the main algorithm for automatic construction of MapReduce jobs from a given spatial analysis task. To illustrate how Marmot transforms a sequence of operators for spatial analysis to map and reduce functions in a way to achieve better performance, this paper presents an example of spatial analysis retrieving the number of subway stations per city in Korea. This paper also experimentally demonstrates that Marmot generally outperforms SpatialHadoop, one of the top plug-in based spatial big data frameworks, particularly in dealing with complex and time-intensive queries involving spatial index.

Performance Analysis of IoT-Based Sensor, Big Data Processing, and Machine Learning Model for Real-Time Monitoring System in Automotive Manufacturing.

With the increase in the amount of data captured during the manufacturing process, monitoring systems are becoming important factors in decision making for management. Current technologies such as Internet of Things (IoT)-based sensors can be considered a solution to provide efficient monitoring of the manufacturing process. In this study, a real-time monitoring system that utilizes IoT-based sensors, big data processing, and a hybrid prediction model is proposed. Firstly, an IoT-based sensor that collects temperature, humidity, accelerometer, and gyroscope data was developed. The characteristics of IoT-generated sensor data from the manufacturing process are: real-time, large amounts, and unstructured type. The proposed big data processing platform utilizes Apache Kafka as a message queue, Apache Storm as a real-time processing engine and MongoDB to store the sensor data from the manufacturing process. Secondly, for the proposed hybrid prediction model, Density-Based Spatial Clustering of Applications with Noise (DBSCAN)-based outlier detection and Random Forest classification were used to remove outlier sensor data and provide fault detection during the manufacturing process, respectively. The proposed model was evaluated and tested at an automotive manufacturing assembly line in Korea. The results showed that IoT-based sensors and the proposed big data processing system are sufficiently efficient to monitor the manufacturing process. Furthermore, the proposed hybrid prediction model has better fault prediction accuracy than other models given the sensor data as input. The proposed system is expected to support management by improving decision-making and will help prevent unexpected losses caused by faults during the manufacturing process.

Real-time Big Data Processing System to Improve Semiconductor Production Efficiency in Smart Factory

Semiconductor production efficiency is closely related to the defect rate in the production process. The temperature and humidity control in the production line are very important because these affect the defect rate. So many smart factory of semiconductor production uses sensor. It is installed in the semiconductor process, which send huge amounts of data per second to a central server to carry out temperature and humidity control in each production line. However, big data processing systems that analyze and process large-scale data are subject to frequent delays in processing, and transmitted data are lost owing to bottlenecks and insufficient memory caused by traffic concentrated in the central server. In this paper, we propose a real-time big data processing system to improve semiconductor production efficiency. The proposed system consists of a production line collection system, task processing system and data storage system, and improves the productivity of the semiconductor manufacturing process by reducing data processing delays as well as data loss and discarded data.

A survey of state management in big data processing systems

State management and its use in diverse applications varies widely across big data processing systems. This is evident in both the research literature and existing systems, such as Apache Flink, Apache Samza, Apache Spark, and Apache Storm. Given the pivotal role that state management plays in various use cases, in this survey, we present some of the most important uses of state as an enabler, discuss the alternative approaches used to handle and implement state, propose a taxonomy to capture the many facets of state management, and highlight new research directions. Our aim is to provide insight into disparate state management techniques, motivate others to pursue research in this area, and draw attention to some open problems.

Enhancing the performance of distributed big data processing systems using Hadoop and Polybase

The approach to improvement of performance of distributed information systems based on sharing technologies of the Hadoop cluster and component of SQL Server PolyBase was considered. It was shown that the relevance of the problem, solved in the research, relates to the need for processing Big Data with different way of representation, in accordance with solving diverse problems of business projects. An analysis of methods and technologies of creation of hybrid data warehouses based on different data of SQL and NoSQL types was performed. It was shown that at present, the most common is the technology of Big Data processing with the use of Hadoop distributed computation environment. The existing technologies of organization and access to the data in the Hadoop cluster with SQL-like DBMS by using connectors were analyzed. The comparative quantitative estimates of using Hive and Sqoop connectors during exporting data to the Hadoop warehouse were presented. An analysis of special features of Big Data processing in the architecture of Hadoop-based distributed cluster computations was carried out. The features of Polybase technology as a component of SQL Server for organizing a bridge between SQL Server and Hadoop data of the SQL and NoSQL types were presented and described. The composition of the model computer plant based on the virtual machine for implementation of joint setting of PolyBase and Hadoop for solving test tasks was described. A methodological toolset for the installation and configuration of Hadoop and PolyBase SQL Server software was developed with consideration of constraints on computing capacities. Queries for using PolyBase and data warehouse Hadoop when processing Big Data were considered. To assess the performance of the system, absolute and relative metrics were proposed. For large volume of test data, the results of the experiments were presented and analyzed, which illustrated an increase in productivity of the distributed information system – query execution time and magnitude of memory capacity of temporary tables, created in this case. A comparative analysis of the studied technology with existing connectors with Hadoop cluster, which showed the advantage of PolyBase over connectors of Sqoop and Hive was performed. The results of the research could be used in the course of scientific and training experiments of organization when implementing the most modern IT-technologies.

Architecture of the Parallel Big Data Processing System for Security Monitoring of Internet of Things Networks

Internet-of-Things networks are applied in many areas of people life now. A cornerstone in a issue of a possibility of further distribution and use of these networks is the aspect of security support. However, the features of these networks complicate the use of traditional means and systems of computer protection in them. One of such features is the need to analyze very large volumes of data, heterogeneous by the nature, in real time and with the minimum computing expenses. Taking into account the features of computational capabilities of Internet-of-Things networks the architecture of the system for parallel big data processing based on the data processing technology named as Complex Event Processing and the parallel computing platform Hadoop is offered. The issues directly connected to the architecture of the system and with implementation of its principal components are considered. These components are: data collection component, data storage component, data normalization and analysis component, and data visualization component. An interconnection between components is provided by means of the Hadoop Distributed File System that is a basis for creation of the distributed data storage. The data collection component organizes the distributed data acquisition and their storage in the data storage component. The data normalization and analysis component transforms data to a uniform format and processes them by means of correlation rules. The data visualization component presents data in a graphical form more suitable for further perception by the operator. The results of the experimental evaluation of the system performance confirming a conclusion about its high performance are discussed.

A benchmark approach and its toolkit for online scheduling of multiple deadline-constrained workflows in big-data processing systems

As distributed systems such as clouds get increasingly popular in the use for big-data processing, there is a need of shifting research attention from minimizing the workflow completion time to satisfying the deadline constraints specified by the users and boosting the benefit of the resource providers. This paper focuses on the online deadline-constrained workflow scheduling problem of how to schedule a set of sequentially submitted workflows with deadline constraints to maximize the resource utilization as well as the success rate of meeting the deadlines. A discrete-event based simulator with a novel benchmark approach is proposed to ease the analysis of the problem. Extensive evaluation has been done to exhibit the effectiveness and significance of the proposed benchmark approach and the developed simulator.