Data Processing Platform Research Articles

A gray-box performance model for Apache Spark

Apache Spark is a powerful open source data processing platform. It is getting more and more popular with the growing need of processing massive amounts of data. A performance prediction model not only helps administrators to have a better understanding of system behavior, but also is useful in performance tuning. However, considering the complex application processing mechanism of Spark, it is not an easy job to model the relationship between system performance and configuration settings.In this paper, we present a gray-box performance model for Spark applications based on machine learning algorithms. Given a specific Spark application, the size of its input data and some key system parameters, this performance model is able to forecast its execution time according to history information. To achieve better accuracy, our model takes basic hardware information and the resource allocation strategy of Spark into consideration.In our experiments, result shows our gray-box model is better than typical black-box approaches in most of the cases. We consider this model is helpful for further researches on Apache Spark.

GFlink: An In-Memory Computing Architecture on Heterogeneous CPU-GPU Clusters for Big Data

The increasing main memory capacity and the explosion of big data have fueled the development of in-memory big data management and processing. By offering an efficient in-memory parallel execution model which can eliminate disk I/O bottleneck, existing in-memory cluster computing platforms (e.g., Flink and Spark) have already been proven to be outstanding platforms for big data processing. However, these platforms are merely CPU-based systems. This paper proposes GFlink, an in-memory computing architecture on heterogeneous CPU-GPU clusters for big data. Our proposed architecture extends the original Flink from CPU clusters to heterogeneous CPU-GPU clusters, greatly improving the computational power of Flink. Furthermore, we have proposed a programming framework based on Flink's abstract model, i.e., DataSet (DST), hiding the programming complexity of GPUs behind the simple and familiar high-level interfaces. To achieve high performance and good load-balance, an efficient JVM-GPU communication strategy, a GPU cache scheme, and an adaptive locality-aware scheduling scheme for three-stage pipelining execution are proposed. Extensive experiment results indicate that the high computational power of GPUs can be efficiently utilized, and the implementation on GFlink outperforms that on the original CPU-based Flink.

GPU-Accelerated High-Throughput Online Stream Data Processing

The Single Instruction Multiple Data (SIMD) architecture of Graphic Processing Units (GPUs) makes them perfect for parallel processing of big data. In this paper, we present the design, implementation and evaluation of G-Storm , a GPU-enabled parallel system based on Storm, which harnesses the massively parallel computing power of GPUs for high-throughput online stream data processing. G-Storm has the following desirable features: 1) G-Storm is designed to be a general data processing platform as Storm, which can handle various applications and data types. 2) G-Storm exposes GPUs to Storm applications while preserving its easy-to-use programming model. 3) G-Storm achieves high-throughput and low-overhead data processing with GPUs. 4) G-Storm accelerates data processing further by enabling Direct Data Transfer (DDT), between two executors that process data at a common GPU. We implemented G-Storm based on Storm 0.9.2 and tested it using three different applications, including continuous query, matrix multiplication and image resizing. Extensive experimental results show that 1) Compared to Storm, G-Storm achieves over 7× improvement on throughput for continuous query, while maintaining reasonable average tuple processing time. It also leads to 2.3× and 1.3× throughput improvements on the other two applications, respectively. 2) DDT significantly reduces data processing time.

Optimizing the performance of hadoop clusters through efficient cluster management techniques

The necessity for processing the huge data has become a critical task in the age of Internet, even though data processing has evolved into a next generation level still data processing and information extraction has many problems to solve. With the increase in data size retrieving useful information with a given span of time is a herculean task. The most optimal solution that has been adopted is usage of distributed computing environment supporting data processing involving suitable model architecture with large complex structure. Although processing has achieved good amount of improvement, efficiency, energy utilization and accuracy has been compromised. The research aims to propose an efficient environment for data processing with optimized energy utilization and increased performance. Hadoop environment common and popular among big data processing platform has been chosen as base for enhancement. Creating a multi node Hadoop cluster architecture on top of which an efficient cluster monitor is setup and an algorithm to manage efficiency of the cluster is formulated. Cluster monitor is incorporated with Zoo keeper, Yarn (Node and resource manager). Zoo keeper does the monitoring of cluster nodes of the distributed system and identifies critical performance problems. Yarn plays a vital role in managing the resources efficiently and controlling the nodes with the help of hybrid scheduler algorithm. Thus this integrated platform helps in monitoring the distributed cluster as well as improving the performance of the overall Big Data processing.

A Smartphone-Based System for Outdoor Data Gathering Using a Wireless Beacon Network and GPS Data: From Cyber Spaces to Senseable Spaces

Information and Communication Technologies (ICTs) and mobile devices are deeply influencing all facets of life, directly affecting the way people experience space and time. ICTs are also tools for supporting urban development, and they have also been adopted as equipment for furnishing public spaces. Hence, ICTs have created a new paradigm of hybrid space that can be defined as Senseable Spaces. Even if there are relevant cases where the adoption of ICT has made the use of public open spaces more “smart”, the interrelation and the recognition of added value need to be further developed. This is one of the motivations for the research presented in this paper. The main goal of the work reported here is the deployment of a system composed of three different connected elements (a real-world infrastructure, a data gathering system, and a data processing and analysis platform) for analysis of human behavior in the open space of Cardeto Park, in Ancona, Italy. For this purpose, and because of the complexity of this task, several actions have been carried out: the deployment of a complete real-world infrastructure in Cardeto Park, the implementation of an ad-hoc smartphone application for the gathering of participants’ data, and the development of a data pre-processing and analysis system for dealing with all the gathered data. A detailed description of these three aspects and the way in which they are connected to create a unique system is the main focus of this paper.

A Survey of Orbitrap All Ion Fragmentation Analysis Assessed by an R MetaboList Package to Study Small-Molecule Metabolites

Leukemia cell and melanoma tumor tissue extracts were studied for small (mostly m/z < 250) polar metabolites by LC-ESI-HRMSn analysis powered by a hybrid Quadrupole-Orbitrap. MS data were simultaneously acquired in fast polarity switching mode operating in MS1 and MS/MS (All Ion Fragmentation, AIF) full-scan analyses at high mass resolution. Positive metabolite assignments were achieved by AIF analysis considering at least two characteristic transitions. Targeted metabolite profiling was achieved by the relative quantification of 18 metabolites through spiking of their respective deuterated counterparts. Manual data processing of MS1 and AIF scans were compared for the accurate determination of natural metabolites and their deuterated analogs by chromatographic alignment and peak area integration. Evaluation of manual and automated (MetaboList R package) AIF data processing yielded comparable results. The versatility of AIF analysis also enabled the untargeted metabolite profiling of leukemia and melanoma samples in which 22 and 53 compounds were, respectively, identified outside those studied by labeling. The main limitation of this method was that low abundance metabolites with scan rates below 8 scans/peak could not be accurately quantified by AIF analysis. The combination of AIF analysis with MetaboList R package represents an opportunity to move towards automated, faster, and more global metabolomics approaches supported by an entirely flexible open source data processing platform freely available from Comprehensive R Archive Network (CRAN, https://CRAN.R-project.org/package=MetaboList ).

Geothermal play fairway analysis at a populated rifting basin area of Taiwan

Taiwan is located in the active mountain belts on the west side of the Pacific Rim with promising potentials for geothermal resources. The triangular shaped Yilan Plain, our study area chosen for a geothermal development pilot project, resides at the southwestern end of the Okinawa Trough behind the Ryukyu Arc-Trench where the Philippine Sea Plate is subducting beneath the Eurasian Plate with low-grade metamorphic rocks exposed in the surrounding mountain ranges.Play Fairway Analysis (PFA) is adapted to demonstrate important aspects, predicting geothermal potential through quantitative and statistical methods in a given type by overlapping each element layer into a composite favorability map with properly calculated weights accounting for uncertainty levels. Highlighted exploratory sweet spots on a regional scale can be identified with the easy-to-read map for decision-making stakeholders.Three components are selected for this analysis: they include (1) heat sources: with temperatures of hot springs, geothermal gradients, and silica geothermometry, (2) permeable fracture networks: with fault lines, drainage network, micro-earthquake activities, and dilation rate distribution, and (3) human activities: with population density, sediment thickness, and hazardous risk. Expert opinion on factor weighting for two geological components and Analytic Hierarchy Process (AHP) for the socioeconomic component are utilized to determine weightings representing the assigned confidence and uncertainty levels. ArcGIS software is used as the platform for data processing and map demonstration.The final map displayed the combined geothermal favorability from the two geological aspects blocked out with the riskier zones from the socioeconomic aspect. It indicates two potential prospects with one in the southwestern hillside where the Chingshui power plant resided and another in the southeastern portion of the Yilan Plain, which could interfere with certain human activities.

Performance Improvement of Heterogeneous Hadoop Clusters Using Query Optimization

The problem that has occurred as a result of the increased connection between the device and the system is creating information at an exponential rate that it is becoming increasingly difficult for a possible solution for processing. Therefore, creating a platform for such advanced level data processing, such as the need to increase the level of hardware and software with bright data. To improve the efficiency of the Hadoop Cluster in extensive data collection and analysis, we have proposed an algorithm system that meets the needs of protected discrimination data in Hadoop Clusters and improves performance and efficiency. The proposed paper aims to find out the effectiveness of the new algorithm, compare, consultation, and find out the best solution for improving the big data scenario is a competitive approach. The map reduction techniques from Hadoop will help maintain a close watch on the underlying or discriminatory Hadoop clusters with insights of results as expected from the luminosity.

Prediction of residential gross yields by using a deep learning method on large scale data processing framework

Purpose- Households, investors and companies who want to make an investment on residential properties are interested in sales prices and rental values that vary depending on regional factors, location and attributes of residential units. It is the preference of investors to buy a new house with higher rental income. Real estate developers and real estate consultants as well as the real estate investors are also interested in investigating relationship between gross yield rate and location, regional factors, attributes of residential units. The purpose of this study is to examine the relationship between attributes of the residential units, location of the units and the gross yield rate. Methodology - In this study, the prediction model of residential gross yield rates with the help of city, county, district, residential attributes information, was created by using LSTM, which is a deep learning method, on big data platform Spark. Findings- According to test results, it has been proven that gross yield rates could be estimated with high accurate model by the aid of Long short term memories. With this model, researchers can predict gross yield rate of any specific flat. Conclusion- The LSTM network has been built in this study shows that the residential gross yield rate could be estimated using city, county, district, number of rooms, number of bathrooms, floor number, total floor attributes. This study also shows that the Spark framework can be used to deal with the growing size of data in real estate and to develop deep learning applications on distributed data processing platforms.

Deep learning‐based real‐time fine‐grained pedestrian recognition using stream processing

Real-time recognition of pedestrian details can be very important in emergency situations for security reasons, such as traffic accidents identification from traffic video. However, this is challenging due to the needed accuracy of video data mining, and also the performance for real-time video processing. Here, the authors propose a solution for fine-grained pedestrian recognition in monitoring scenarios using deep learning and stream processing cloud computing, which is called DRPRS (deep learning-based real-time fine-grained pedestrian recognition using stream processing). The authors design an improved convolutional neural network (CNN) network called fine-CNN, which is a nine-layer neural network for detailed pedestrian recognition. In DRPRS, a pedestrian in a surveillance video is segmented and fine-grainedly recognised using improved single-shot detector and several fine-CNNs. DRPRS is supported by parallel mechanisms provided by Apache Storm stream processing framework. In addition, in order to further improve the recognition performance, a GPU-based scheduling algorithm is proposed to make full use of GPU resources in a cluster. The whole recognition process is deployed on a big video data processing platform to meet real-time requirements. DRPRS is extensively evaluated in terms of accuracy, fault tolerance, and performance, which show that the proposed approach is efficient.

Data processing platforms for electronic health records

As expectations for the data processing platforms to support processing for healthcare data, new services need to be deployed in healthcare services. One of them is Electronic health records. It raised its importance recently from the data become shareable, connection oriented, high available, and united. Therefore, a first motivation is to help healthcare organizations to define which the appropriate processing platform can support real time processing for huge electronic health records data. Although define different platforms that enhanced analytics and prediction methods of healthcare entities which produces a huge amount of data. A second possible motivation is to go beyond the limitations of traditional processing techniques that used in electronic health records. The big data processing is the cheaper solution that performing real-time analysis on the huge data. This paper identifies and discusses new data processing platforms that can be used in electronic health records.

Big Data Processing and Analytics Platform Architecture for Process Industry Factories

This paper describes the architecture of a cross-sectorial Big Data platform for the process industry domain. The main objective was to design a scalable analytical platform that will support the collection, storage and processing of data from multiple industry domains. Such a platform should be able to connect to the existing environment in the plant and use the data gathered to build predictive functions to optimize the production processes. The analytical platform will contain a development environment with which to build these functions, and a simulation environment to evaluate the models. The platform will be shared among multiple sites from different industry sectors. Cross-sectorial sharing will enable the transfer of knowledge across different domains. During the development, we adopted a user-centered approach to gather requirements from different stakeholders which were used to design architectural models from different viewpoints, from contextual to deployment. The deployed architecture was tested in two process industry domains, one from the aluminium production and the other from the plastic molding industry.

Gabriela.NET: Modular platform for 1D and 2D data acquisition, processing and presentation

Within this paper we describe the idea and implementation of the modular platform capable of acquisition and processing data from external measurement boards. The platform supports the data collection through the Ethernet network, fast and effective 1D and 2D data processing and real-time processed data presentation. The platform is implemented by .NET Framework as a desktop PC application and its modularity is reached by splitting particular functions into separate blocks. By a simple use-case we present the final functionality of the solution.

A Performance Comparison of Big Data Processing Platform Based on Parallel Clustering Algorithms

The performance of three typical big data processing platform: Hadoop, Spark and DataMPI are compared based on different parallel clustering algorithms: parallel K-means, parallel fuzzy K-means and parallel Canopy. Experiments are performed on different text as well as numeric dataset and clusters of different scale. The results show that: (1) for the same data set, when the memory of each node is 4GB, DataMPI can achieve about 60% performance improvement compared with Hadoop, and can achieve about 32% performance improvement compared with Spark; (2) in order to obtain a high clustering performance, a cluster with 6 nodes and 6GB memory of each node should be selected.

Joint Scheduling of Tasks and Network Flows in Big Data Clusters

As an increasing number of big data processing platforms like Hadoop, Spark, and Storm appear and normally share the resources in the data center, it has been important and challenging to schedule various jobs from these platforms onto the underlying data center resources such that the overall job completion time is minimized. To solve the problem, the existing work either focus on the task-level scheduling techniques, such as Quincy and delay scheduling, or focus on the network flow scheduling techniques, such as D3 and preemptive distributed quick. These works deal with the scheduling of tasks and network flows separately and cannot achieve optimal performance. The reason is that the task scheduling without regard of the available network bandwidths may generate the task placement that causes serious network congestions and thus leads to long data transmission time. In this paper, we propose the joint scheduling technique by coordinating the task placement and the scheduling of network flows arising from these tasks. We develop a software-defined network (SDN)-based online scheduling framework which selects the task placement based on the available bandwidth on the SDN switches and at meanwhile optimally allocates the bandwidth to each data flow. Comprehensive trace-driven simulations show that the joint scheduling technique can take full use of the network bandwidth and thus reduce the job completion time by 55% on average compared with the benchmark methods.

基于DPDK的用户态IPSec网关的研究和实现

本论文针对在使用IPSec网关时由于用户量不断增加、网络流量需求大的情况下所造成的时延、阻塞、吞吐量小等问题，分析了内核态IPSec解决方案处理流程中存在的不足，提出了基于DPDK在用户态实现IPSec网关的方法。针对传统方案处理开销大的问题提出利用思科的VPP作框架，DPDK作收包工具搭建平台实现数据包在用户态的处理。测试结果表明，基于DPDK的用户态IPSec解决方案与传统方案相比，性能得到了有效的提升。 This paper aims at the problems of delay, congestion, small throughput caused due to increasing users and more network traffic demand by using IPSec gateway, analyzes the shortcomings during the IPSec solution process, and puts forward the method of implementation of DPDK gateway based on IPSec in user space. Aiming at the problem of large overhead of traditional programs, we propose to build Cisco VPP as a platform and DPDK as a receiving packets tool to build data processing platform. The test results show that the performance of the IPSec solution of user space based on DPDK is improved effectively compared with the traditional solution.

Data processing platform for indoor localization framework

Nowadays the wireless localization of people or other subjects within buildings is a current topic. This article presents a complete system for collecting, processing and presenting localization data within buildings or open spaces. The system is based on applications and services that can be run either on a dedicated Microsoft Serves OS or on the MS Azure platform. The location data are obtained from a network of wireless devices deployed according to certain rules in the zone of interest.

The IoT for smart sustainable cities of the future: An analytical framework for sensor-based big data applications for environmental sustainability

The Internet of Things (IoT) is one of the key components of the ICT infrastructure of smart sustainable cities as an emerging urban development approach due to its great potential to advance environmental sustainability. As one of the prevalent ICT visions or computing paradigms, the IoT is associated with big data analytics, which is clearly on a penetrative path across many urban domains for optimizing energy efficiency and mitigating environmental effects. This pertains mainly to the effective utilization of natural resources, the intelligent management of infrastructures and facilities, and the enhanced delivery of services in support of the environment. As such, the IoT and related big data applications can play a key role in catalyzing and improving the process of environmentally sustainable development. However, topical studies tend to deal largely with the IoT and related big data applications in connection with economic growth and the quality of life in the realm of smart cities, and largely ignore their role in improving environmental sustainability in the context of smart sustainable cities of the future. In addition, several advanced technologies are being used in smart cities without making any contribution to environmental sustainability, and the strategies through which sustainable cities can be achieved fall short in considering advanced technologies. Therefore, the aim of this paper is to review and synthesize the relevant literature with the objective of identifying and discussing the state-of-the-art sensor-based big data applications enabled by the IoT for environmental sustainability and related data processing platforms and computing models in the context of smart sustainable cities of the future. Also, this paper identifies the key challenges pertaining to the IoT and big data analytics, as well as discusses some of the associated open issues. Furthermore, it explores the opportunity of augmenting the informational landscape of smart sustainable cities with big data applications to achieve the required level of environmental sustainability. In doing so, it proposes a framework which brings together a large number of previous studies on smart cities and sustainable cities, including research directed at a more conceptual, analytical, and overarching level, as well as research on specific technologies and their novel applications. The goal of this study suits a mix of two research approaches: topical literature review and thematic analysis. In terms of originality, no study has been conducted on the IoT and related big data applications in the context of smart sustainable cities, and this paper provides a basis for urban researchers to draw on this analytical framework in future research. The proposed framework, which can be replicated, tested, and evaluated in empirical research, will add additional depth to studies in the field of smart sustainable cities. This paper serves to inform urban planners, scholars, ICT experts, and other city stakeholders about the environmental benefits that can be gained from implementing smart sustainable city initiatives and projects on the basis of the IoT and related big data applications.

Optimizing Checkpointing Performance in Spark

Spark [1] is a cluster framework that performs in-memory computing. As with other distributed data processing platforms, fault tolerant plays an important role in the whole architecture. The Fault Tolerant of Spark contains Lineage and Checkpointing. The latter is expensive because doing checkpoint always causes RDD recomputation. In this paper, we analyze the workflow in the execution of the current design and propose alternatives to improve the performance of checkpointing, one of which is based on an existing approach. We evaluate our results in terms of application level throughput.

Applying intelligent data traffic adaptation to high-performance multiple big data analytics platforms

This paper introduces the integration of two data processing platforms, RHhadoop and SparkR, to carry out rapid big data retrieval and analytics using R programming, which can serve as part of business intelligence. Besides, it has developed the job scheduling optimization called Memory-Sensitive Heterogeneous Earliest Finish Time algorithm to enhance system throughput. However, the bottleneck of system throughput is definitely relevant to data traffic problem over network, especially a large amount of data exchange between distributed computing nodes within a cluster. The objective of this paper is to propose an intelligence approach to tackle the crucial problems of inefficient data traffic flow. Adaptive network-based fuzzy inference system along with particle swarm optimization has employed to tune the network-related parameters at computing nodes for improving network QoS and speed up data transportation significantly. In order to examine the computing efficiency, performance index has been evaluated for all of treatments in the experiment.