Efficient Big Data Processing Research Articles

A Novel Efficient Big Data Processing Scheme for Feature Extraction in Electrical Discharge Machining

Electrical discharge machining (EDM) can machine hard conductive workpieces that are difficult to machine using traditional machining techniques. For monitoring the EDM process using virtual metrology (VM), probes with a very high sampling rate are needed to acquire the voltage and current signals of electrodes, thereby generating a huge amount of sensor data and raising a big data processing issue in extracting features from raw sensor data. This paper proposes a novel efficient big data processing scheme for feature extraction in EDM, called BEDPS, based on Spark and HDFS. A parallel gap-based wave detection mechanism using Spark is designed to efficiently detect effective machining waves from big machining raw data of EDM without the slow internode data communications in HDFS. A pre-loaded memory-based feature calculation mechanism is also developed to calculate the key features from machining-wave data in Spark in a parallel-processing manner. Testing results show that the proposed BEDPS is much more efficient in terms of total execution time in machining wave detection and feature calculation and more scalable in the data size that can be processed, compared to the existing system.

Harnessing the power of big data analytics in the cloud to support learning analytics in mobile learning environment

Technology enhanced learning (TEL) such as online learning environment with adaptive technologies has gained growing interest in recent past in the field of teaching and learning. In this context, mobile learning has got much momentum and is exemplified by diverse characteristics associated with the technologies and devices used, the enormous size of data generated throughout a learning session, and the interactions among the learners that occur outside the classroom. Consequently, sophisticated data analysis techniques are required to handle the intricacy of mobile learning and analyze the vast amount of datasets to enhance the learning experiences of mobile learners. This has led to the adoption of big data analytics for efficient processing of big learning data to add value to the mobile learning environments. Yet limited processing capability of the mobile devices is another key challenge faced by such big data analytics in mobile learning environments. To overcome this limitation, certain heavy computational parts could be offloaded to the cloud which can provide enough computation and storage resources. To this end, this paper presents a cloud based mobile learning framework that utilizes big data analytics technique to extract values from huge volume of mobile learners' data. Finally, we investigate learners' readiness and driving factors of mobile learning adoption in higher education institutions. In particular, we propose a hypothesized model for mobile learning adoption built on a locally extended technology acceptance model (TAM).

A large data processing algorithm for energy efficiency in a heterogeneous cluster

It is reportedi that the electricity cost to operate a cluster may well exceed its acquisition cost, and the processing of big data requires large scale cluster and long period. Therefore, energy efficient processing of big data is essential for the data owners and users. In this paper, we propose a novel algorithm MinBalance to processing I/O intensive big data tasks energy efficiently in heterogeneous cluster. In the former step, four greedy policies are used to select the proper nodes considering heterogeneity of the cluster. While in the latter step, the workloads of the selected nodes will be well balanced to avoid the energy wastes caused by waiting. MinBalance is a universal algorithm and cannot be affected by the data storage strategies. Experimental results indicate that MinBalance can achieve over 60% energy reduction for large sets over the traditional methods of powering down partial nodes.

Leveraging Cloud Based Big Data Analytics in Knowledge Management for Enhanced Decision Making in Organizations

In recent past, big data opportunities have gained much momentum to enhance knowledge management in organizations. However, big data due to its various properties like high volume, variety, and velocity can no longer be effectively stored and analyzed with traditional data management techniques to generate values for knowledge development. Hence, new technologies and architectures are required to store and analyze this big data through advanced data analytics and in turn generate vital real-time knowledge for effective decision making by organizations. More specifically, it is necessary to have a single infrastructure which provides common functionality of knowledge management, and flexible enough to handle different types of big data and big data analysis tasks. Cloud computing infrastructures capable of storing and processing large volume of data can be used for efficient big data processing because it minimizes the initial cost for the large-scale computing infrastructure demanded by big data analytics. This paper aims to explore the impact of big data analytics on knowledge management and proposes a cloud-based conceptual framework that can analyze big data in real time to facilitate enhanced decision making intended for competitive advantage. Thus, this framework will pave the way for organizations to explore the relationship between big data analytics and knowledge management which are mostly deemed as two distinct entities.

Automatic Scaling Hadoop in the Cloud for Efficient Process of Big Geospatial Data

Efficient processing of big geospatial data is crucial for tackling global and regional challenges such as climate change and natural disasters, but it is challenging not only due to the massive data volume but also due to the intrinsic complexity and high dimensions of the geospatial datasets. While traditional computing infrastructure does not scale well with the rapidly increasing data volume, Hadoop has attracted increasing attention in geoscience communities for handling big geospatial data. Recently, many studies were carried out to investigate adopting Hadoop for processing big geospatial data, but how to adjust the computing resources to efficiently handle the dynamic geoprocessing workload was barely explored. To bridge this gap, we propose a novel framework to automatically scale the Hadoop cluster in the cloud environment to allocate the right amount of computing resources based on the dynamic geoprocessing workload. The framework and auto-scaling algorithms are introduced, and a prototype system was developed to demonstrate the feasibility and efficiency of the proposed scaling mechanism using Digital Elevation Model (DEM) interpolation as an example. Experimental results show that this auto-scaling framework could (1) significantly reduce the computing resource utilization (by 80% in our example) while delivering similar performance as a full-powered cluster; and (2) effectively handle the spike processing workload by automatically increasing the computing resources to ensure the processing is finished within an acceptable time. Such an auto-scaling approach provides a valuable reference to optimize the performance of geospatial applications to address data- and computational-intensity challenges in GIScience in a more cost-efficient manner.

Performance Modeling and Analysis of a Hadoop Cluster for Efficient Big Data Processing

Performance Modeling and Analysis of a Hadoop Cluster for Efficient Big Data Processing

Hadoop based Feature Selection and Decision Making Models on Big Data

Objectives: A large amount of informative data is being captured and processed by today's organizations and is continuing to increase exponentially. It becomes computationally inaccurate to analyze such big data for decision making systems. Methods/Analysis: Hadoop, which is a working model based on the Map-Reduce framework with efficient computation and processing of Big Data. Findings: Most of the traditional classification algorithms have issues such as class imbalance and dimension reduction on Big Data. However, a large part of the data produced today are incomplete and inaccurate, so large organizations prefer relational databases to store their information, but the user query processing speed is very low. Unlike existing solutions that require a prior knowledge of classification accuracy for various types of data characteristics, which is impossible to obtain in practice. Applications/Improvement: In this paper, we have given a compared proposed model to different big data feature selection and classification models along with advantages and limitations.

A spatiotemporal indexing approach for efficient processing of big array-based climate data with MapReduce

ABSTRACTClimate observations and model simulations are producing vast amounts of array-based spatiotemporal data. Efficient processing of these data is essential for assessing global challenges such as climate change, natural disasters, and diseases. This is challenging not only because of the large data volume, but also because of the intrinsic high-dimensional nature of geoscience data. To tackle this challenge, we propose a spatiotemporal indexing approach to efficiently manage and process big climate data with MapReduce in a highly scalable environment. Using this approach, big climate data are directly stored in a Hadoop Distributed File System in its original, native file format. A spatiotemporal index is built to bridge the logical array-based data model and the physical data layout, which enables fast data retrieval when performing spatiotemporal queries. Based on the index, a data-partitioning algorithm is applied to enable MapReduce to achieve high data locality, as well as balancing the workload. The proposed indexing approach is evaluated using the National Aeronautics and Space Administration (NASA) Modern-Era Retrospective Analysis for Research and Applications (MERRA) climate reanalysis dataset. The experimental results show that the index can significantly accelerate querying and processing (~10× speedup compared to the baseline test using the same computing cluster), while keeping the index-to-data ratio small (0.0328%). The applicability of the indexing approach is demonstrated by a climate anomaly detection deployed on a NASA Hadoop cluster. This approach is also able to support efficient processing of general array-based spatiotemporal data in various geoscience domains without special configuration on a Hadoop cluster.

MetaCloudDataStorage Architecture for Big Data Security in Cloud Computing

The cloud is increasingly being used to store and process the big data. Many researchers have been trying to protect big data in cloud computing environment. Traditional security mechanisms using encryption are neither efficient nor suited to the task of protecting big data in the Cloud. In this paper, we first discuss about challenges and potential solutions for protecting big data in cloud computing. Second, we proposed MetaCloudDataStorage Architecture for protecting Big Data in Cloud Computing Environment. This framework ensures that efficient processing of big data in cloud computing environment and gains more business insights.

Enhancing Telco Service Quality with Big Data Enabled Churn Analysis: Infrastructure, Model, and Deployment

The penetration of mobile phones is nearly saturated in both developing and developed regions. In such a circumstance, how to prevent subscriber churn has become an important issue for today’s telecom operators, as the cost to acquire a new subscriber is much higher than that to retain an existing subscriber. In this paper, we propose to leverage the power of big data to mitigate the problem of subscriber churn and enhance the service quality of telecom operators. As the information hub, telecom operators have accumulated a huge volume of valuable data on subscriber behaviors, service usage, and network operations. To enable efficient big data processing, we first build a dedicated distributed cloud infrastructure that integrates both online and offline processing capabilities. Second, we develop a complete churn analysis model based on deep data mining techniques, and utilize inter-subscriber influence to improve prediction accuracy. Finally, we use real datasets obtained from a large telecom operator in China to verify the accuracy of our churn analysis models. The dataset contains the information of over 3.5 million subscribers, which generate over 600 million call detail records (CDRs) per month. The empirical results demonstrate that our proposed method can achieve around 90% accuracy for T + 1 testing periods and identify subscribers with high negative influence successfully.

Real-Time Big Data Analytical Architecture for Remote Sensing Application

The assets of remote senses digital world daily generate massive volume of real-time data (mainly referred to the term “Big Data”), where insight information has a potential significance if collected and aggregated effectively. In today’s era, there is a great deal added to real-time remote sensing Big Data than it seems at first, and extracting the useful information in an efficient manner leads a system toward a major computational challenges, such as to analyze, aggregate, and store, where data are remotely collected. Keeping in view the above mentioned factors, there is a need for designing a system architecture that welcomes both real-time, as well as offline data processing. Therefore, in this paper, we propose real-time Big Data analytical architecture for remote sensing satellite application. The proposed architecture comprises three main units, such as 1) remote sensing Big Data acquisition unit (RSDU); 2) data processing unit (DPU); and 3) data analysis decision unit (DADU). First, RSDU acquires data from the satellite and sends this data to the Base Station, where initial processing takes place. Second, DPU plays a vital role in architecture for efficient processing of real-time Big Data by providing filtration, load balancing, and parallel processing. Third, DADU is the upper layer unit of the proposed architecture, which is responsible for compilation, storage of the results, and generation of decision based on the results received from DPU. The proposed architecture has the capability of dividing, load balancing, and parallel processing of only useful data. Thus, it results in efficiently analyzing real-time remote sensing Big Data using earth observatory system. Furthermore, the proposed architecture has the capability of storing incoming raw data to perform offline analysis on largely stored dumps, when required. Finally, a detailed analysis of remotely sensed earth observatory Big Data for land and sea area are provided using Hadoop. In addition, various algorithms are proposed for each level of RSDU, DPU, and DADU to detect land as well as sea area to elaborate the working of an architecture.

Recent research advances in cloud computing and big data

Recent research advances in cloud computing and big data

Optimizing big data processing performance in the public cloud: opportunities and approaches

Today’s lightning fast data generation from massive sources is calling for efficient big data processing, which imposes unprecedented demands on the computing and networking infrastructures. State-of-the-art tools, most notably MapReduce, are generally performed on dedicated server clusters to explore data parallelism. For grass roots users or non-computing professionals, the cost of deploying and maintaining a large-scale dedicated server clusters can be prohibitively high, not to mention the technical skills involved. On the other hand, public clouds allow general users to rent virtual machines and run their applications in a pay-as-you-go manner with ultra-high scalability with minimal upfront costs. This new computing paradigm has gained tremendous success in recent years, becoming a highly attractive alternative to dedicated server clusters. This article discusses the critical challenges and opportunities when big data meet the public cloud. We identify the key differences between running big data processing in a public cloud and in dedicated server clusters. We then present two important problems for efficient big data processing in the public cloud, resource provisioning (i.e., how to rent VMs) and VM-MapReduce job/task scheduling (i.e., how to run MapReduce after the VMs are constructed). Each of these two questions have a set of problems to solve. We present solution approaches for certain problems, and offer optimized design guidelines for others. Finally, we discuss our implementation experiences.

Exploring Big Data with Helix

While much work has focused on efficient processing of Big Data, little work considers how to understand them. In this paper, we describe Helix, a system for guided exploration of Big Data. Helix provides a unified view of sources, ranging from spreadsheets and XML files with no schema, all the way to RDF graphs and relational data with well-defined schemas. Helix users explore these heterogeneous data sources through a combination of keyword searches and navigation of linked web pages that include information about the schemas, as well as data and semantic links within and across sources. At a technical level, the paper describes the research challenges involved in developing Helix, along with a set of real-world usage scenarios and the lessons learned.

하둡 기반 빅 데이터 기법을 이용한 웹 서비스 데이터 처리 설계 및 구현

데이터를 구조화하여 사용하는 관계형 데이터베이스가 현재까지 데이터 관리에 가장 많이 사용되고 있다. 그러나 관계형 데이터베이스는 데이터가 증가되면 데이터를 저장하거나 조회할 때 읽기, 쓰기 연산 수행에 제약 조건이 발생되어 서비스가 느려지는 현상이 나타난다. 또 새로운 업무가 추가되면 데이터베이스 내 데이터는 증가되고 결국 이를 해결하기 위해 하드웨어의 병렬 구성, CPU, 메모리, 네트워크 등 추가적인 인프라 구성을 필요로 하게 된다. 본 논문에서는 관계형 데이터베이스의 데이터 증가로 느려지는 웹 정보서비스 개선을 위해 기존 관계형 데이터베이스의 데이터를 하둡 HDFS로 전송하고 이를 일원화하여 데이터를 재구성한 후 사용자에게 하둡 데이터 처리로 대량의 데이터를 빠르고 안전하게 추출하는 모델을 구현한다. 본 시스템 적용을 위해 웹 기반 민원시스템과 비정형 데이터 처리인 이미지 파일 저장에 본 제안시스템을 적용하였다. 적용결과 관계형 데이터베이스 시스템보다 제안시스템 데이터 처리가 0.4초 더 빠른 결과를 얻을 수 있었고 기존 관계형 데이터베이스와 같은 대량의 데이터를 처리를 빅 데이터 기법인 하둡 데이터 처리로도 웹 정보서비스를 지원이 가능하였다. 또한 하둡은 오픈소스로 제공되어 소프트웨어 구매 비용을 줄여주는 장점이 있으며 기존 관계형 데이터베이스의 데이터 증가로 효율적인 대용량 데이터 처리를 요구하는 조직에게 도움을 줄 수 있을 것이다. Relational databases used by structuralizing data are the most widely used in data management at present. However, in relational databases, service becomes slower as the amount of data increases because of constraints in the reading and writing operations to save or query data. Furthermore, when a new task is added, the database grows and, consequently, requires additional infrastructure, such as parallel configuration of hardware, CPU, memory, and network, to support smooth operation. In this paper, in order to improve the web information services that are slowing down due to increase of data in the relational databases, we implemented a model to extract a large amount of data quickly and safely for users by processing Hadoop Distributed File System (HDFS) files after sending data to HDFSs and unifying and reconstructing the data. We implemented our model in a Web-based civil affairs system that stores image files, which is irregular data processing. Our proposed system's data processing was found to be 0.4 sec faster than that of a relational database system. Thus, we found that it is possible to support Web information services with a Hadoop-based big data processing technique in order to process a large amount of data, as in conventional relational databases. Furthermore, since Hadoop is open source, our model has the advantage of reducing software costs. The proposed system is expected to be used as a model for Web services that provide fast information processing for organizations that require efficient processing of big data because of the increase in the size of conventional relational databases.

Accordion: An Efficient Gear-Shifting for a Power-Proportional Distributed Data-Placement Method

Power-aware distributed file systems for efficient Big Data processing are increasingly moving towards power-proportional designs. However, current data placement methods for such systems have not given careful consideration to the effect of gear-shifting during operations. If the system wants to shift to a higher gear, it must reallocate the updated datasets that were modified in a lower gear when a subset of the nodes was inactive, but without disrupting the servicing of requests from clients. Inefficient gear-shifting that requires a large amount of data reallocation greatly degrades the system performance. To address this challenge, this paper proposes a data placement method known as Accordion, which uses data replication to arrange the data layout comprehensively and provide efficient gear-shifting. Compared with current methods, Accordion reduces the amount of data transferred, which significantly shortens the period required to reallocate the updated data during gear-shifting then able to improve the performance of the systems. The effect of this reduction is larger with higher gears, so Accordion is suitable for smooth gear-shifting in multigear systems. Moreover, the times when the active nodes serve the requests are well distributed, so Accordion is capable of higher scalability than existing methods based on the I/O throughput performance. Accordion does not require any strict constraint on the number of nodes in the system therefore our proposed method is expected to work well in practical environments. Extensive empirical experiments using actual machines with an Accordion prototype based on the Hadoop Distributed File System demonstrated that our proposed method significantly reduced the period required to transfer updated data, i.e., by 66% compared with an existing method.

Implementation of Hadoop and Map-Reduce for Efficient Processing of Big-Data

Implementation of Hadoop and Map-Reduce for Efficient Processing of Big-Data

A spatiotemporal compression based approach for efficient big data processing on Cloud

It is well known that processing big graph data can be costly on Cloud. Processing big graph data introduces complex and multiple iterations that raise challenges such as parallel memory bottlenecks, deadlocks, and inefficiency. To tackle the challenges, we propose a novel technique for effectively processing big graph data on Cloud. Specifically, the big data will be compressed with its spatiotemporal features on Cloud. By exploring spatial data correlation, we partition a graph data set into clusters. In a cluster, the workload can be shared by the inference based on time series similarity. By exploiting temporal correlation, in each time series or a single graph edge, temporal data compression is conducted. A novel data driven scheduling is also developed for data processing optimisation. The experiment results demonstrate that the spatiotemporal compression and scheduling achieve significant performance gains in terms of data size and data fidelity loss.

Parallel map projection of vector-based big spatial data: Coupling cloud computing with graphics processing units

The objective of this article is to present a framework that couples cloud and high-performance computing for the parallel map projection of vector-based big spatial data. The past few years have witnessed a tremendous growth of a variety of high-volume spatial data—i.e., big spatial data. Map projection is often needed, for example, when we apply these big spatial data into large-scale spatial analysis and modeling approaches that require a common coordinate system. However, due to the size of these data and algorithmic complexity of map projections, the transformation of big spatial data between alternative projections represents a pressing computational challenge. Recent advancement in cloud computing and high-performance computing offers a potential means of addressing this computational challenge. The parallel map projection framework presented in this study is based on a layered architecture that couples capabilities of cloud computing and high-performance computing accelerated by Graphics Processing Units. We use large LiDAR data as an example of vector-based big spatial data to investigate the utility of the parallel map projection framework. As experimental results reveal, the framework provides considerable acceleration for re-projecting vector-based big spatial data. Coupling high-performance and cloud computing, which complement to each other, is a suggested solution for the efficient processing and analysis of big spatial data.

Kerberos 기반 하둡 분산 파일 시스템의 안전성 향상방안

최근 스마트 기기 및 소셜 네트워크 서비스의 발달로 인해 데이터가 폭증하며 세계는 이른바 빅데이터 시대를 맞고 있다. 이에 이러한 데이터를 처리할 수 있는 새로운 기술인 빅데이터 처리기술은 클라우드 컴퓨팅 기술과 함께 주목받고 있으며, 가장 대표적인 기술이 바로 하둡이다. 하둡 분산 파일 시스템은 상용 리눅스 서버에서 실행되도록 설계된 오픈소스 프레임워크로서 수백 테라바이트 크기의 파일을 저장할 수 있다. 초기 하둡은 빅데이터 처리에 초점을 맞추어 보안이 거의 도입되지 않은 상태였으나 사용자가 빠르게 늘어남에 따라 하둡 분산 파일 시스템에 개인정보를 포함한 민감한 데이터가 많이 저장되면서, 2009년 커버로스와 토큰 시스템을 도입한 새로운 버전을 발표하였다. 그러나 이 시스템은 재전송 공격, 가장 공격 등이 가능하다는 취약점을 가진다. 따라서 본 논문에서는 하둡 분산 파일 시스템 보안 취약점을 분석하고, 이러한 취약점을 보완하면서 하둡의 성능을 유지할 수 있는 새로운 프로토콜을 제안한다. As the developments of smart devices and social network services, the amount of data has been exploding. The world is facing Big data era. For these reasons, the Big data processing technology which is a new technology that can handle such data has attracted much attention. One of the most representative technologies is Hadoop. Hadoop Distributed File System(HDFS) designed to run on commercial Linux server is an open source framework and can store many terabytes of data. The initial version of Hadoop did not consider security because it only focused on efficient Big data processing. As the number of users rapidly increases, a lot of sensitive data including personal information were stored on HDFS. So Hadoop announced a new version that introduces Kerberos and token system in 2009. However, this system is vulnerable to the replay attack, impersonation attack and other attacks. In this paper, we analyze these vulnerabilities of HDFS security and propose a new protocol which complements these vulnerabilities and maintains the performance of Hadoop.