Data Processing Platform Research Articles

On the use of IoT and Big Data Technologies for Real-time Monitoring and Data Processing

Recent advances in pervasive technologies, such as wireless ad hoc networks and wearable sensor devices, allow the connection of everyday things to the Internet, commonly denoted as Internet of Things (IoT). IoT is seen as an enabler to the development of intelligent and context-aware services and applications. These services could dynamically react to the environment changes and users’ preferences. The main aim is to make users’ life more comfortable according to their locations, current requirements, and ongoing activities. However, handling dynamic and frequent context changes is a difficult task without a real-time event/data acquisition and processing platform. Big data and IoT technologies have been recently proposed for timely analysing information (i.e., data, events) streams. In this paper, we propose to combine IoT techniques with Big data technologies into a holistic platform for continuous and real-time data monitoring and processing. Preliminary experiments have been conducted and results are reported to show the usefulness of this platform in a real-case scenario.

Design and Implementation of an Expert Search System Using Academic Data in Big Data Processing Platforms

Design and Implementation of an Expert Search System Using Academic Data in Big Data Processing Platforms

SIproc: an open-source biomedical data processing platform for large hyperspectral images.

There has recently been significant interest within the vibrational spectroscopy community to apply quantitative spectroscopic imaging techniques to histology and clinical diagnosis. However, many of the proposed methods require collecting spectroscopic images that have a similar region size and resolution to the corresponding histological images. Since spectroscopic images contain significantly more spectral samples than traditional histology, the resulting data sets can approach hundreds of gigabytes to terabytes in size. This makes them difficult to store and process, and the tools available to researchers for handling large spectroscopic data sets are limited. Fundamental mathematical tools, such as MATLAB, Octave, and SciPy, are extremely powerful but require that the data be stored in fast memory. This memory limitation becomes impractical for even modestly sized histological images, which can be hundreds of gigabytes in size. In this paper, we propose an open-source toolkit designed to perform out-of-core processing of hyperspectral images. By taking advantage of graphical processing unit (GPU) computing combined with adaptive data streaming, our software alleviates common workstation memory limitations while achieving better performance than existing applications.

A New Memory MapReduce Framework for Higher Access to Resources

The demand for highly parallel data processing platform was growing due to an explosion in the number of massive-scale data applications both in academia and industry. MapReduce was one of the most meaningful solutions to deal with big data distributed computing, This paper was based on the work of Hadoop MapReduce. In the face of massive data computing and calculation process, MapReduce generated a lot of dynamic data, but these data were discarded after the task completed. Meanwhile, a large number of dynamic data were written to HDFS during task execution, caused much unnecessary IO cost. In this paper, we analyzed existing distributed caching mechanism and proposed a new Memory MapReduce framework that has a real-time response to read or write request from task nodes, maintain related information about cache data. After performance testing, we could clearly find MapReduce with cache significantly improved in IO performance.

Crowd cloud routing protocol based on opportunistic computing for wireless sensor networks

We proposed a crowd cloud routing protocol based on opportunistic computing to improve the data transmission efficiency, reliability, and reduce routing overhead in wireless sensor networks. Based on the analysis of the demand of big data processing in wireless sensor network, the data analysis and processing platform for wireless sensor network are designed based on the combination with the cloud computing. The cloud platform includes the main nodes, the nodes, and the core nodes. There are the engine and the drive between the wireless sensor network and the cloud server. Secondly, aiming at the problem of data transmission in the cloud platform, we design an opportunistic computing model which is suitable for wireless sensor networks to minimize the weight of routing management and network overhead. Then, we design an opportunistic calculation model to guarantee the data transmission scheme of the cloud platform. Finally, by eliminating the factors that may cause the link instability, the crowd cloud routing protocol is proposed. The experimental results show that the proposed crowd cloud routing protocol has the functions of real-time and reliability and reduces the cost of routing request.

Performance Modeling and Predictive Scheduling for Distributed Stream Data Processing

In a distributed stream data processing system, an application is usually modeled using a directed graph, in which each vertex corresponds to a data source or a processing unit, and edges indicate data flow. In this paper, we propose a novel predictive scheduling framework to enable fast and distributed stream data processing, which features topology-aware modeling for performance prediction and predictive scheduling. For prediction, we present a topology-aware method to accurately predict the average tuple processing time of an application for a given scheduling solution, according to the topology of the application graph and runtime statistics. For scheduling, we present an effective algorithm to assign threads to machines under the guidance of prediction results. To validate and evaluate the proposed framework, we implemented it based on a highly-regarded distributed stream data processing platform, Storm, and tested it with 3 representative applications: word count (stream version), log stream processing and continuous query. Extensive experimental results show 1) The topology-aware prediction method offers an average accuracy of 84.2 percent. 2) The predictive scheduling framework reduces the average tuple processing time by 24.9 percent on average, compared to Storm's default scheduler.

Dynamic Data Placement Strategy in MapReduce-styled Data Processing Platform

Data placement is one of the core technologies of the MapReduce-styled data processing platform, which contributes much to the data processing efficiency. A dynamic data placement strategy is proposed in this paper due to that existing data placement techniques are lack of the consideration of the computing load on the data storage node and reduce the ratio of the localized processing of the hot-spot data. The strategy takes the data accessing localization ratio, the remaining computing capacity of nodes as the new factors of the data placement decision. Performance evaluation results show that the proposed data placement strategy outperforms the original strategy in HDFS file system and the average job execution time is reduce by the maximum of 12%.

Smart Meter Data Analytics

Smart electricity meters have been replacing conventional meters worldwide, enabling automated collection of fine-grained (e.g., every 15 minutes or hourly) consumption data. A variety of smart meter analytics algorithms and applications have been proposed, mainly in the smart grid literature. However, the focus has been on what can be done with the data rather than how to do it efficiently. In this article, we examine smart meter analytics from a software performance perspective. First, we design a performance benchmark that includes common smart meter analytics tasks. These include offline feature extraction and model building as well as a framework for online anomaly detection that we propose. Second, since obtaining real smart meter data is difficult due to privacy issues, we present an algorithm for generating large realistic datasets from a small seed of real data. Third, we implement the proposed benchmark using five representative platforms: a traditional numeric computing platform (Matlab), a relational DBMS with a built-in machine learning toolkit (PostgreSQL/MADlib), a main-memory column store (“System C”), and two distributed data processing platforms (Hive and Spark/Spark Streaming). We compare the five platforms in terms of application development effort and performance on a multicore machine as well as a cluster of 16 commodity servers.

ChemInform Abstract: Metal—Organic Frameworks as Competitive Materials for Non‐Linear Optics

AbstractReview: 124 refs.

Integration and optimization of multiple big data processing platforms

Purpose – The purpose of this paper is to integrate and optimize a multiple big data processing platform with the features of high performance, high availability and high scalability in big data environment. Design/methodology/approach – First, the integration of Apache Hive, Cloudera Impala and BDAS Shark make the platform support SQL-like query. Next, users can access a single interface and select the best performance of big data warehouse platform automatically by the proposed optimizer. Finally, the distributed memory storage system Memcached incorporated into the distributed file system, Apache HDFS, is employed for fast caching query results. Therefore, if users query the same SQL command, the same result responds rapidly from the cache system instead of suffering the repeated searches in a big data warehouse and taking a longer time to retrieve. Findings – As a result the proposed approach significantly improves the overall performance and dramatically reduces the search time as querying a database, especially applying for the high-repeatable SQL commands under multi-user mode. Research limitations/implications – Currently, Shark’s latest stable version 0.9.1 does not support the latest versions of Spark and Hive. In addition, this series of software only supports Oracle JDK7. Using Oracle JDK8 or Open JDK will cause serious errors, and some software will be unable to run. Practical implications – The problem with this system is that some blocks are missing when too many blocks are stored in one result (about 100,000 records). Another problem is that the sequential writing into In-memory cache wastes time. Originality/value – When the remaining memory capacity is 2 GB or less on each server, Impala and Shark will have a lot of page swapping, causing extremely low performance. When the data scale is larger, it may cause the JVM I/O exception and make the program crash. However, when the remaining memory capacity is sufficient, Shark is faster than Hive and Impala. Impala’s consumption of memory resources is between those of Shark and Hive. This amount of remaining memory is sufficient for Impala’s maximum performance. In this study, each server allocates 20 GB of memory for cluster computing and sets the amount of remaining memory as Level 1: 3 percent (0.6 GB), Level 2: 15 percent (3 GB) and Level 3: 75 percent (15 GB) as the critical points. The program automatically selects Hive when memory is less than 15 percent, Impala at 15 to 75 percent and Shark at more than 75 percent.

LabVIEW-based impedance biosensing system for detection of avian influenza virus

In order to detect the multiple avian influenza viruses (AIVs) rapidly, specifically and sensitively, a LabVIEW and microelectrode array-based impedance biosensor was developed and demonstrated. A laptop with LabVIEW software was used to generate excitation signals at different frequencies with an audio card and measure the impedance of target viruses through a data acquisition card. The audio card of the laptop was used as a function generator, while a data acquisition card was used for data communication. A virtual instrument was programmed with LabVIEW to provide a platform for impedance measurement, data processing, and control. Six interdigitated microelectrodes were placed at the bottom of six wells on a microplate to form six sensors for different AIVs and controls. Then, AIV specific ligands were immobilized on the microelectrode surface to capture target viruses. To enhance the sensitivity, AIV specific aptamers conjugated gold nanoparticles and thiocyanuric acid were employed to form a network structure and used as an amplifier. Results of the measured impedance were compared with a commercial IM6 impedance analyzer, and the error was less than 5%. The developed biosensor was portable with the sensitivity and specificity for applications to on-site or in-field rapid screening of avian influenza viruses. Keywords: biosensor, impedance detection, avian influenza virus, LabVIEW DOI: 10.3965/j.ijabe.20160904.1704 Citation: Zhang B H, Wang R H, Wang Y X, Li Y B. LabVIEW-based impedance biosensing system for detection of avian influenza virus. Int J Agric & Biol Eng, 2016; 9(4): 116－122.

The MASSIF platform: a modular and semantic platform for the development of flexible IoT services

In the Internet of Things (IoT), data-producing entities sense their environment and transmit these observations to a data processing platform for further analysis. Applications can have a notion of context awareness by combining this sensed data, or by processing the combined data. The processes of combining data can consist both of merging the dynamic sensed data, as well as fusing the sensed data with background and historical data. Semantics can aid in this task, as they have proven their use in data integration, knowledge exchange and reasoning. Semantic services performing reasoning on the integrated sensed data, combined with background knowledge, such as profile data, allow extracting useful information and support intelligent decision making. However, advanced reasoning on the combination of this sensed data and background knowledge is still hard to achieve. Furthermore, the collaboration between semantic services allows to reach complex decisions. The dynamic composition of such collaborative workflows that can adapt to the current context, has not received much attention yet. In this paper, we present MASSIF, a data-driven platform for the semantic annotation of and reasoning on IoT data. It allows the integration of multiple modular reasoning services that can collaborate in a flexible manner to facilitate complex decision-making processes. Data-driven workflows are enabled by letting services specify the data they would like to consume. After thorough processing, these services can decide to share their decisions with other consumers. By defining the data these services would like to consume, they can operate on a subset of data, improving reasoning efficiency. Furthermore, each of these services can integrate the consumed data with background knowledge in its own context model, for rapid intelligent decision making. To show the strengths of the platform, two use cases are detailed and thoroughly evaluated.

Stream processing platforms for analyzing big dynamic data

Abstract Nowadays, data is produced in every aspect of our lives, leading to a massive amount of information generated every second. However, this vast amount is often too large to be stored and for many applications the information contained in these data streams is only useful when it is fresh. Batch processing platforms like Hadoop MapReduce do not fit these needs as they require to collect data on disk and process it repeatedly. Therefore, modern data processing engines combine the scalability of distributed architectures with the one-pass semantics of traditional stream engines. In this paper, we survey the current state of the art in scalable stream processing from a user perspective. We examine and describe their architecture, execution model, programming interface, and data analysis support as well as discuss the challenges and limitations of their APIs. In this connection, we introduce Piglet, an extended Pig Latin language and code generator that compiles (extended) Pig Latin code into programs for various data processing platforms. Thereby, we discuss the mapping to platform-specic concepts in order to provide a uniform view.

Big Data 2.0 Processing Systems: Taxonomy and Open Challenges

Data is key resource in the modern world. Big data has become a popular term which is used to describe the exponential growth and availability of data. In practice, the growing demand for large-scale data processing and data analysis applications spurred the development of novel solutions from both the industry and academia. For a decade, the MapReduce framework, and its open source realization, Hadoop, has emerged as a highly successful framework that has created a lot of momentum in both the research and industrial communities such that it has become the defacto standard of big data processing platforms. However, in recent years, academia and industry have started to recognize the limitations of the Hadoop framework in several application domains and big data processing scenarios such as large scale processing of structured data, graph data and streaming data. Thus, we have witnessed an unprecedented interest to tackle these challenges with new solutions which constituted a new wave of mostly domain-specific, optimized big data processing platforms. In this article, we refer to this new wave of systems as Big Data 2.0 processing systems. To better understand the latest ongoing developments in the world of big data processing systems, we provide a taxonomy and detailed analysis of the state-of-the-art in this domain. In addition, we identify a set of the current open research challenges and discuss some promising directions for future research.

OpenCL Implementation of a Parallel Universal Kriging Algorithm for Massive Spatial Data Interpolation on Heterogeneous Systems

In some digital Earth engineering applications, spatial interpolation algorithms are required to process and analyze large amounts of data. Due to its powerful computing capacity, heterogeneous computing has been used in many applications for data processing in various fields. In this study, we explore the design and implementation of a parallel universal kriging spatial interpolation algorithm using the OpenCL programming model on heterogeneous computing platforms for massive Geo-spatial data processing. This study focuses primarily on transforming the hotspots in serial algorithms, i.e., the universal kriging interpolation function, into the corresponding kernel function in OpenCL. We also employ parallelization and optimization techniques in our implementation to improve the code performance. Finally, based on the results of experiments performed on two different high performance heterogeneous platforms, i.e., an NVIDIA graphics processing unit system and an Intel Xeon Phi system (MIC), we show that the parallel universal kriging algorithm can achieve the highest speedup of up to 40× with a single computing device and the highest speedup of up to 80× with multiple devices.

THE DESIGN OF A HIGH PERFORMANCE EARTH IMAGERY AND RASTER DATA MANAGEMENT AND PROCESSING PLATFORM

Abstract. This paper summarizes the general requirements and specific characteristics of both geospatial raster database management system and raster data processing platform from a domain-specific perspective as well as from a computing point of view. It also discusses the need of tight integration between the database system and the processing system. These requirements resulted in Oracle Spatial GeoRaster, a global scale and high performance earth imagery and raster data management and processing platform. The rationale, design, implementation, and benefits of Oracle Spatial GeoRaster are described. Basically, as a database management system, GeoRaster defines an integrated raster data model, supports image compression, data manipulation, general and spatial indices, content and context based queries and updates, versioning, concurrency, security, replication, standby, backup and recovery, multitenancy, and ETL. It provides high scalability using computer and storage clustering. As a raster data processing platform, GeoRaster provides basic operations, image processing, raster analytics, and data distribution featuring high performance computing (HPC). Specifically, HPC features include locality computing, concurrent processing, parallel processing, and in-memory computing. In addition, the APIs and the plug-in architecture are discussed.

THE DESIGN OF A HIGH PERFORMANCE EARTH IMAGERY AND RASTER DATA MANAGEMENT AND PROCESSING PLATFORM

This paper summarizes the general requirements and specific characteristics of both geospatial raster database management system and raster data processing platform from a domain-specific perspective as well as from a computing point of view. It also discusses the need of tight integration between the database system and the processing system. These requirements resulted in Oracle Spatial GeoRaster, a global scale and high performance earth imagery and raster data management and processing platform. The rationale, design, implementation, and benefits of Oracle Spatial GeoRaster are described. Basically, as a database management system, GeoRaster defines an integrated raster data model, supports image compression, data manipulation, general and spatial indices, content and context based queries and updates, versioning, concurrency, security, replication, standby, backup and recovery, multitenancy, and ETL. It provides high scalability using computer and storage clustering. As a raster data processing platform, GeoRaster provides basic operations, image processing, raster analytics, and data distribution featuring high performance computing (HPC). Specifically, HPC features include locality computing, concurrent processing, parallel processing, and in-memory computing. In addition, the APIs and the plug-in architecture are discussed.

MapReduce program synthesis

By abstracting away the complexity of distributed systems, large-scale data processing platforms—MapReduce, Hadoop, Spark, Dryad, etc.—have provided developers with simple means for harnessing the power of the cloud. In this paper, we ask whether we can automatically synthesize MapReduce-style distributed programs from input–output examples. Our ultimate goal is to enable end users to specify large-scale data analyses through the simple interface of examples. We thus present a new algorithm and tool for synthesizing programs composed of efficient data-parallel operations that can execute on cloud computing infrastructure. We evaluate our tool on a range of real-world big-data analysis tasks and general computations. Our results demonstrate the efficiency of our approach and the small number of examples it requires to synthesize correct, scalable programs.

Big Data for Supporting Low-Carbon Road Transport Policies in Europe: Applications, Challenges and Opportunities

Abstract Big data is among the most promising research trends of the decade, drawing attention from every segment of the market and society. This paper provides the scientific community with a comprehensive overview of the applications of a data processing platform designed to harness the potential of big data in the field of road transport policies in Europe. This platform relies on datasets of driving and mobility patterns collected by means of navigation systems. Two datasets from conventional fuel vehicles collected with on-board GPS systems have been used to perform an initial pilot study and develop its core algorithms. They consist of 4.5 million trips and parking events recorded by monitoring 28,000 vehicles over one month. The presented analyses address: (1) large-scale mobility statistics, (2) potential of electric vehicles in replacing conventional fuel vehicles and related modal shift, (3) energy demand coming from electric vehicles, (4) smart design of the recharge infrastructure and Vehicle-to-Grid, and (5) real-world driving and evaporative emissions assessment and mapping. The developed methodology and the presented outcomes demonstrate the potential of big data for policy assessment and better governance, focusing on the challenges and on the huge opportunities offered for future developments. This paper ultimately aims to show how big data can inspire smart policies together with public and private investments to enable the large scale deployment of the next generation of green vehicles, offering an unprecedented opportunity to shape policies for future mobility and smart cities.

Multi-Objective Parametric Query Optimization

We propose a generalization of the classical database query optimization problem: multi-objective parametric query optimization (MPQ). MPQ compares alternative processing plans according to multiple execution cost metrics. It also models missing pieces of information on which plan costs depend upon as parameters. Both features are crucial to model query processing on modern data processing platforms. MPQ generalizes previously proposed query optimization variants such as multi-objective query optimization, parametric query optimization, and traditional query optimization. We show however that the MPQ problem has different properties than prior variants and solving it requires novel methods. We present an algorithm that solves the MPQ problem and finds for a given query the set of all relevant query plans. This set contains all plans that realize optimal execution cost tradeoffs for any combination of parameter values. Our algorithm is based on dynamic programming and recursively constructs relevant query plans by combining relevant plans for query parts. We assume that all plan execution cost functions are piecewise-linear in the parameters. We use linear programming to compare alternative plans and to identify plans that are not relevant. We present a complexity analysis of our algorithm and experimentally evaluate its performance.