Data-intensive Distributed Applications Research Articles

Privacy-aware genetic algorithm based data security framework for distributed cloud storage

An increased use of data driven applications and integrated systems have caused an accelerating expansion in data volumes and increase in the number of digital records, over the past few decades. Exponentially growing data volumes being processed by large-scale distributed data-intensive applications have placed an increasing pressure on the underlying storage services for timely and efficient storage and retrieval of the data. The use of cloud storage is among the best strategies to efficiently store growing volumes of data. However, outsourcing data to public cloud storage leads to the challenge of data confidentiality preservation. Data Confidentiality is among the top challenges associated with cloud storage which have contributed substantially as an inhibitor for cloud computing adoption all over the world and is considered a serious concern, especially in case of big data, where securing data in a timely and accurate manner is an arduous task. Our study aims to contribute in anti-cybercrime by protecting the confidentiality of sensitive growing data. We enunciate an optimized confidentially preserving framework on distributed cloud storage, that works for growing data with time-efficiency and minimum memory usage. Our framework uses a merger of Genetic Algorithm (GA), parallel data distribution, and privacy-aware selective encryption techniques. The experiments and comparative analysis depict that our proposed framework outperforms others under consideration, in terms of execution time, memory usage and network throughput respectively.

IS-HBase: An In-Storage Computing Optimized HBase with I/O Offloading and Self-Adaptive Caching in Compute-Storage Disaggregated Infrastructure

Active storage devices and in-storage computing are proposed and developed in recent years to effectively reduce the amount of required data traffic and to improve the overall application performance. They are especially preferred in the compute-storage disaggregated infrastructure. In both techniques, a simple computing module is added to storage devices/servers such that some stored data can be processed in the storage devices/servers before being transmitted to application servers. This can reduce the required network bandwidth and offload certain computing requirements from application servers to storage devices/servers. However, several challenges exist when designing an in-storage computing- based architecture for applications. These include what computing functions need to be offloaded, how to design the protocol between in-storage modules and application servers, and how to deal with the caching issue in application servers. HBase is an important and widely used distributed Key-Value Store. It stores and indexes key-value pairs in large files in a storage system like HDFS. However, its performance especially read performance, is impacted by the heavy traffics between HBase RegionServers and storage servers in the compute-storage disaggregated infrastructure when the available network bandwidth is limited. We propose an I n- S torage-based HBase architecture, called IS-HBase , to improve the overall performance and to address the aforementioned challenges. First, IS-HBase executes a data pre-processing module ( I n- S torage S can N er, called ISSN ) for some read queries and returns the requested key-value pairs to RegionServers instead of returning data blocks in HFile. IS-HBase carries out compactions in storage servers to reduce the large amount of data being transmitted through the network and thus the compaction execution time is effectively reduced. Second, a set of new protocols is proposed to address the communication and coordination between HBase RegionServers at computing nodes and ISSNs at storage nodes. Third, a new self-adaptive caching scheme is proposed to better serve the read queries with fewer I/O operations and less network traffic. According to our experiments, the IS-HBase can reduce up to 97% network traffic for read queries and the throughput (queries per second) is significantly less affected by the fluctuation of available network bandwidth. The execution time of compaction in IS-HBase is only about 6.31% – 41.84% of the execution time of legacy HBase. In general, IS-HBase demonstrates the potential of adopting in-storage computing for other data-intensive distributed applications to significantly improve performance in compute-storage disaggregated infrastructure.

A fault recovery protocol for brokers in centralized publish-subscribe systems targeting multiprocessor systems-on-chips

The publish-subscribe programming model has been an alternative to the design of data-intensive distributed applications in many domains. Recently, this model has been ported to the domain of Multiprocessor Systems-on-Chips, in which applications must use the underlying Network-on-Chip communication infrastructure effectively due to restrictions on the architecture such as low power consumption and limited memory size. In such a scenario, the publish-subscribe model fulfills some of these requirements while providing high-level access to the network hardware to programmers, thus contributing to software quality. However, the publish-subscribe model relies on a single process dedicated to orchestrating the communication at the application level, the broker. Should a broker process crash, the communication between associated nodes may experience delays, downtime, or even inconsistent data. In extreme cases, communication is definitively ruined. Thus, a recovery strategy for brokers in the publish-subscribe model becomes crucial when the application has safety requirements. In this work, we extend a publish-subscribe protocol to add redundancy to brokers’ sensitive data. Besides, we provide a recovery protocol to recover brokers in case of a failure. We also provide analytical models to estimate the communication overhead of our approach. We validate our approach in two distinct MPSoC platforms. The results show that our approach inserts a small memory footprint to the system while providing minimal system downtime during recovery.

Big data transfer optimization through adaptive parameter tuning

Obtaining optimal data transfer performance is of utmost importance to today’s data-intensive distributed applications and wide-area data replication services. Tuning application-layer protocol parameters such as pipelining, parallelism, and concurrency can significantly increase efficient utilization of the available network bandwidth as well as the end-to-end data transfer performance. However, determining the best settings for these parameters is a challenging problem, as network conditions can vary greatly between sites and over time. Poor protocol tuning can cause either under- or over-utilization of network resources and thus degrade transfer performance. In this paper, we present three novel algorithms for application-layer parameter tuning and transfer scheduling to maximize transfer throughput in wide-area networks. Our algorithms use heuristics to tune the level of control channel pipelining (for small file optimization), the number of parallel data streams per file (for large file optimization), and the number of concurrent file transfers to increase I/O throughput (for all types of files). The proposed algorithms improve the transfer throughput up to 10x compared to the baseline and 7x compared to the state-of-the-art solutions. We also propose adaptive tuning to adjust the values of parameters based on real-time observations. The results show that adaptive tuning can further improve transfer throughput by up to 24% compared to the heuristic approach.

Distributed document clustering analysis based on a hybrid method

Clustering is one of the recently challenging tasks since there is an ever-growing amount of data in scientific research and commercial applications. High quality and fast document clustering algorithms are in great demand to deal with large volume of data. The computational requirements for bringing such growing amount data to a central site for clustering are complex. The proposed algorithm uses optimal centroids for K-Means clustering based on Particle Swarm Optimization(PSO). PSO is used to take advantage of its global search ability to provide optimal centroids which aids in generating more compact clusters with improved accuracy. This proposed methodology utilizes Hadoop and MapReduce framework which provides distributed storage and analysis to support data intensive distributed applications. Experiments were performed on Reuter’s and RCV1 document dataset which shows an improvement in accuracy with reduced execution time.

타원곡선기반 하둡 분산 시스템의 초기 인증 프로토콜

Abstract Recently, the development of cloud computing technology is developed as soon as smartphones is increases, and increased that users want to receive big data service. Hadoop framework of the big data service is provided to hadoop file system and hadoop mapreduce supported by data-intensive distributed applications. But, smpartphone service using hadoop system is a very vulnerable state to data authentication. In this paper, we propose a initial authentication protocol of hadoop system assisted by smartphone service. Proposed protocol is combine symmetric key cryptography techniques with ECC algorithm in order to support the secure multiple data processing systems. In particular, the proposed protocol to access the system by the user Hadoop when processing data, the initial authentication key and the symmetric key instead of the elliptic curve by using the public key-based security is improved. Key Words : Elliptic Curve, Authentication, Hadoop Distribution, Big data, Cloud Computing

MR-ARM: A MAP-REDUCE ASSOCIATION RULE MINING FRAMEWORK

Association rule is one of the primary tasks in data mining that discovers correlations among items in a transactional database. The majority of vertical and horizontal association rule mining algorithms have been developed to improve the frequent items discovery step which necessitates high demands on training time and memory usage particularly when the input database is very large. In this paper, we overcome the problem of mining very large data by proposing a new parallel Map-Reduce (MR) association rule mining technique called MR-ARM that uses a hybrid data transformation format to quickly finding frequent items and generating rules. The MR programming paradigm is becoming popular for large scale data intensive distributed applications due to its efficiency, simplicity and ease of use, and therefore the proposed algorithm develops a fast parallel distributed batch set intersection method for finding frequent items. Two implementations (Weka, Hadoop) of the proposed MR association rule algorithm have been developed and a number of experiments against small, medium and large data collections have been conducted. The ground bases of the comparisons are time required by the algorithm for: data initialisation, frequent items discovery, rule generation, etc. The results show that MR-ARM is very useful tool for mining association rules from large datasets in a distributed environment.

Scheduling Algorithm for Workflow-Based Applications in Optical Grid

Grid is evolving to a more efficient global computing infrastructure by introducing optical network technology to support the advanced data-intensive distributed applications. Scheduling such data-intensive applications includes assigning tasks on computational resources, routing lightpaths, and assigning wavelength channels for data communication. The scheduling problem is NP-hard in the traditional grid system, and in optical grids, it is more complicated due to the character of optical networks. In this paper, we formulate the scheduling problem in optical grids and propose a novel scheduling algorithm which modifies the scheduling order according to actual importance of each task to search for a better solution. We call it the scheduled critical path (SCP) algorithm. We compare the scheduling results obtained by the SCP algorithm with the optimal results calculated by OPL studio software on a 3-node optical grid. To evaluate the performance of the proposed algorithm on more complicated systems, we construct a simulator which is able to schedule the application to the optical grid according to a certain scheduling algorithm. The simulation results prove the efficiency of the SCP algorithm.

IQ-Paths: Predictably High Performance Data Streams Across Dynamic Network Overlays

Overlay networks are a key vehicle for delivering network and processing resources to high performance applications. For shared networks, however, to consistently deliver such resources at desired levels of performance, overlays must be managed at runtime, based on the continuous assessment and prediction of available distributed resources. Data-intensive applications, for example, must assess, predict, and judiciously use available network paths, and dynamically choose alternate or exploit concurrent paths. Otherwise, they cannot sustain the consistent levels of performance required by tasks like remote data visualization, online program steering, and remote access to high end devices. The multiplicity of data streams occurring in complex scientific workflows or in large-scale distributed collaborations exacerbate this problem, particularly when different streams have different performance requirements. This paper presents IQ-Paths, a set of techniques and their middleware realization that implement self-regulating overlay streams for data-intensive distributed applications. Self-regulation is based on (1) the dynamic and continuous assessment of the quality of each overlay path, (2) the use of online network monitoring and statistical analyses that provide probabilistic guarantees about available path bandwidth, loss rate, and RTT, and (3) self-management, via an efficient packet routing and scheduling algorithm that dynamically schedules data packets to different overlay paths in accordance with their available bandwidths. IQ-Paths offers probabilistic guarantees for application-level specifications of stream utility, based on statistical predictions of available network bandwidth. This affords applications with the ability, for instance, to send control or steering data across overlay paths that offer strong guarantees for future bandwidth vs. across less guaranteed paths. Experimental results presented in this paper use IQ-Paths to better handle the different kinds of data produced by two high performance applications and one multimedia application: (1) a data-driven interactive high performance code with user-defined utility requirements, (2) an adaptive overlay version of the popular Grid-FTP application, and (3) a MPEG-4 Fine-Grained Scalable layered video streaming.

The Google file system

We have designed and implemented the Google File System, a scalable distributed file system for large distributed data-intensive applications. It provides fault tolerance while running on inexpensive commodity hardware, and it delivers high aggregate performance to a large number of clients. While sharing many of the same goals as previous distributed file systems, our design has been driven by observations of our application workloads and technological environment, both current and anticipated, that reflect a marked departure from some earlier file system assumptions. This has led us to reexamine traditional choices and explore radically different design points. The file system has successfully met our storage needs. It is widely deployed within Google as the storage platform for the generation and processing of data used by our service as well as research and development efforts that require large data sets. The largest cluster to date provides hundreds of terabytes of storage across thousands of disks on over a thousand machines, and it is concurrently accessed by hundreds of clients. In this paper, we present file system interface extensions designed to support distributed applications, discuss many aspects of our design, and report measurements from both micro-benchmarks and real world use.

Scalable Bulk Data Transfer in Wide Area Networks

Bulk data transfer in wide area networks (WAN) requires scalable and high network bandwidth. In this paper, we identify a number of the scalability limitations that affect the full utilization of peak theoretical network bandwidth. In addition, we study and classify different offered approaches to overcome some of the identified limitations and increase network bandwidth among Grid components in WAN. With these limitations in mind, we study and evaluate the scalability and flexibility of a UDP-based multiple-network-interface socket (MuniSocket) model in WAN. The MuniSocket model is a middleware layer between the distributed applications and the multiple networks and system resources available. MuniSocket utilizes existing system resources, network interface cards, and network links to provide a scalable, reliable and high bandwidth network solution for data-intensive distributed applications, thus eliminating most of the identified limitations.

Experimental studies using photonic data services at IGrid 2002

We describe an architecture for remote and distributed data intensive applications that integrates optical path services, network protocol services for high performance data transport, and data services for remote data analysis and distributed data mining. We also present experimental evidence using geoscience data that this architecture scales to long haul, high performance networks.