High Performance Cloud Research Articles

HybridMR: a new approach for hybrid MapReduce combining desktop grid and cloud infrastructures

SummaryThis paper introduces HybridMR, a novel model for the execution of MapReduce (MR) computation on hybrid computing environment. Using this model, high performance cloud resources and heterogeneous desktop personal computers (PCs) in Internet or Intranet can be integrated to form a hybrid computing environment. Thanks to HybridMR, the computation and storage capability of large scale desktop PCs can be fully utilized to process large scale datasets. HybridMR relies on two innovative solutions to enable such large scale data‐intensive computation. The first one is HybridDFS, which is a hybrid distributed file system. HybridDFS features reliable distributed storage that alleviates the volatility of desktop PCs, thanks to fault tolerance and file replication mechanism. The second innovation is a new node priority‐based fair scheduling (NPBFS) algorithm has been developed in HybridMR to achieve both data storage balance and job assignment balance by assigning each node a priority through quantifying CPU speed, memory size, and input and output capacity. In this paper, we describe the HybridMR, HybridDFS, and NPBFS. We report on performance evaluation results, which show that the proposed HybridMR not only achieves reliable MR computation, reduces task response time, and improves the performance of MR, but also reduces the computation cost and achieves a greener computing mode. Copyright © 2015 John Wiley & Sons, Ltd.

Applications for ultrascale computing

Studies of complex physical and engineering systems, represented by multi-scale and multi-physics computer simulations have an increasing demand for computing power, especially when the simulations of realistic problems are considered. This demand is driven by the increasing size and complexity of the studied systems or the time constraints. Ultrascale computing systems offer a possible solution to this problem. Future ultrascale systems will be large-scale complex computing systems combining technologies from high performance computing, distributed systems, big data, and cloud computing. Thus, the challenge of developing and programming complex algorithms on these systems is twofold. Firstly, the complex algorithms have to be either developed from scratch, or redesigned in order to yield high performance, while retaining correct functional behaviour. Secondly, ultrascale computing systems impose a number of non-functional cross-cutting concerns, such as fault tolerance or energy consumption, which can significantly impact the deployment of applications on large complex systems. This article discusses the state-of-the-art of programming for current and future large scale systems with an emphasis on complex applications. We derive a number of programming and execution support requirements by studying several computing applications that the authors are currently developing and discuss their potential and necessary upgrades for ultrascale execution.

계산과학 시뮬레이션을 위한 실시간 가상 클러스터 생성 및 I/O 성능 향상 기법

Computational science simulations have been used to enable discovery in a broad spectrum of application areas, these simulations show irregular demanding characteristics of computing resources from time to time. The adoption of virtualized high performance cloud, rather than CPU-centric computing platform (such as supercomputers), is gaining interest of interests mainly due to its ease-of-use, multi-tenancy and flexibility. Basically, provisioning a virtual cluster, which consists of a lot of virtual machines, in a real-time has a critical impact on the successful deployment of the virtualized HPC clouds for computational science simulations. However, the cost of concurrently creating many virtual machines in constructing a virtual cluster can be as much as two orders of magnitude worse than expected. One of the main factors in this bottleneck is the time spent to create the virtual images for the virtual machines. In this paper, we propose a novel technique to minimize the creation time of virtual machine images and improve I/O performance of the provisioned virtual clusters. We also confirm that our proposed technique outperforms the conventional ones using various sets of experiments.

A Stochastic Approach to Analysis of Energy-Aware DVS-Enabled Cloud Datacenters

With the increasing call for green cloud, reducing energy consumption has been an important requirement for cloud resource providers not only to reduce operating costs, but also to improve system reliability. Dynamic voltage scaling (DVS) has been a key technique in exploiting the hardware characteristics of cloud datacenters to save energy by lowering the supply voltage and operating frequency. This paper presents a novel stochastic framework for energy efficiency and performance analysis of DVS-enabled cloud. This framework uses virtual machine request arrival rate, failure rate, repair rate, and service rate of datacenter servers as model inputs. Based on a queuing-network-based analysis, this paper gives analytic solutions of three metrics. The proposed framework can be used to help the design and optimization of energy-aware high performance cloud systems.

Cyber Physical Sensors and Actuators for Privacy- and Cost-Aware Optimization of User-Generated Content Provisioning

Nowadays there is growing interest in evolving the distributed sensors concept from the more traditional one of enabling technology to monitor the surrounding physical environmenttowards Cyber Physical Systems (CPS) sensors and actuators, that is, as a suitable tool to measure/influence the cyber activity of possibly worldwide communities of users (e.g., any geotagged operation leaving a cyber footprint and any cyber physical incentive to stimulate activity as in crowdsensing). To leverage this novel perspective, we propose a framework to integrate at bestmultilayer CPS sensors and actuatorsas the basis for autonomic management operations on both physical and cyber worlds. In this paper the specific application domain target is peer-to-peer content sharing based on social identities and relationships, but we claim that the proposed CPS framework is of general applicability. In particular, our original middleware solution adopts CPS actuators to move users’ content temporarily from smart home environments to high-performance cloud resources to minimize the access time of a dynamically selected quota of contents. Then, based on social network sensors and connectivity/networking ones hosted at lightweight domestic Web servers, our CPS actuators can originally and dynamically move content back from the cloud to smart homes when appropriate, in order to both retain full ownership of user-generated content and reduce cloud hosting costs.

가상 스위치 구현 및 비교에 관한 연구

ABSTRACT Nowadays, most Internet servers run on virtual a network which connects each virtual machine, to process large amounts of data on single physical computer. The virtual machines for their own purposes are interconnected to other virtual server systems with various types of operating systems. Each virtual server may share one physical computer resources and operates at a service seamlessly to each other. Sometimes even the virtual machines are implemented using a method that lease the physical IT resources without purchasing physical server. Therefore, a virtual switch is needed to improve the performance when a large number of virtual machines operate on physical server. In this study, the implementation and investigation was performed for the virtual switches, OpenvSwitch and DPDKvSwitch, in order to provide the necessary high-performance cloud services to creative business. 키워드 : 가상머신, 가상네트워크, 고속화 가상스위치, 클라우드 서비스 Key word : Virtual machine, virtual network, accelerated virtual switch, cloud serviceJournal of the Korea Institute of Information andCommunication Engineering

An Implementation of High Performance Cloud Data Management

This paper is aimed to maximize computing resources and improve performances of CMEP by using REST style cloud management interfaces to operate data, using open authentication constructor to validate requests, designing schedule algorithms and distributing cloud computing based on executing and user’s purchase situation. Experiments showed that, high performance cloud data management would significantly shorten response time, expand erupt simultaneously count and scalability.

High-Throughput Encryption for Cloud Computing Storage System

In recent years Cloud computing has become the infrastructure which small and medium-sized businesses are increasingly adopting for their IT and computational needs. It provides a platform for high performance and throughput oriented computing, and massive data storage. Subsequently, novel tools and technologies are needed to handle this new infrastructure. One of the biggest challenges in this evolving field is Cloud storage security, and accordingly we propose new optimized techniques based on encryption process to achieve better storage system security. This paper proposes a symmetric block algorithm (CHiS-256) to encrypt Cloud data in efficient manner. Also, this paper presents a novel partially encrypted metadata-based data storage. The (CHiS-256) cipher is implemented as part of the Cloud data storage service to offer a secure, high-performance and throughput Cloud storage system. The results of our proposed algorithm are promising and show the methods to be advantageous in Cloud massive data storage and access applications.

Virtualizing high-end GPGPUs on ARM clusters for the next generation of high performance cloud computing

High performance cloud computing is behind the scene powering "the next big thing" as the mainstream accelerator for innovation in many areas. We describe here how to accelerate inexpensive ARM-based computing nodes with high-end GPGPUs hosted on x86_64 machines using the GVirtuS general-purpose virtualization service. We draw the vision of a possible next generation computing clusters characterized by highly heterogeneous parallelism heading to a lower electric power demanding, less heat producing and more environmental friendliness. Preliminary but promising performance data suggest that this solution could be considered as part of the foundations of the next generation of high performance cloud computing components.

Enabling Elasticity of Key-Value Stores in the Cloud Using Cost-Aware Live Data Migration

Key-Value stores play an important role in today’s large-scale, high-performance cloud applications. Elastic scaling and load rebalancing with low cost live data migration are critical to enabling the elasticity of Key/Value stores in the cloud. Learning how to reduce the migration cost is one difficult problem that cloud providers are trying to solve. Many existing works try to address this issue in non-virtual environments. These approaches, however, may not be well suited for cloud-based Key/Value stores. To address this challenge, the study tackles the data migration problem under a load rebalancing scenario. The paper builds a one cost model that could be aware of the underlying VM interference and trade-off between migration time and performance impact. A cost-aware migration algorithm is designed to utilize the cost model and balance rate to guide the choice of possible migration actions. Our evaluation using Yahoo! Cloud Serving Benchmark shows the effectiveness of the approach.  基金项目: 国家自然科学基金(61173003, 61100068); 国家重点基础研究发展计划(973)(2009CB320704); 国家高技术研究发展 计划(863)(2012AA011204); 国家科技支撑计划(2012BAH05F02) 收稿时间: 2012-07-10; 修改时间: 2012-10-19; 定稿时间: 2012-12-03 1404 Journal of Software 软件学报 Vol.24, No.6, June 2013

Service-Oriented Development of Workflow-Based Semantic Reasoning Applications

The modern Semantic Web scenarios require reasoning algorithms to be flexible, modular, and highly-configurable. A solid approach, followed in the design of the most currently existing reasoners, is not sufficient when dealing with today's challenges of data analysis across multiple sources of heterogeneous data or when the data amount grows to the “Big Data” sizes. The “reasoning as a workflow” concept has attracted a lot of attention in the design of new-generation Semantic Web applications, offering a lot of opportunities to improve both flexibility and scalability of the reasoning process. Considering a single workflow component as a service offers a lot of opportunities for a reasoning algorithm to target a much wider range of potentially enabled Semantic Web use cases by taking benefits of a service-oriented and component-based implementation. We introduce a technique for developing service-oriented Semantic Reasoning applications based on the workflow concept. We also present the Large Knowledge Collider - a software platform for developing workflow-based Semantic Web applications, taking advantages of on-demand high performance computing and cloud infrastructures.

Design of High Performance Cloud Storage Platform Based on Cheap PC Clusters Using MongoDB and Hadoop

To lay the foundation for the high performance private cloud storage platform, this paper proposes a new cloud storage structure with horizontal scalability using MongoDB and Hadoop. MongoDB is a powerful NOSQL database which is used to construct the cloud storage platform. In certain scenarios, the map-reduce provided by MongoDB can not meet the need of the complex data analysis, especially for the mass complex unstructured data such as videos and documents. This paper introduce the key technologies in MongoDB and Hadoop, then aggregate the advantages of them to build a high performance private cloud storage infrastructure based on cheap personal computer clusters. This infrastructure combines the high horizontal scalability of MongoDB and the high-performance analysis capability from Hadoop.

PM&E-CP: Performance Monitor and Evaluator for Cloud Platforms

Cloud computing is an emerging infrastructure paradigm that promises to eliminate the need for companies to maintain expensive computing hardware. However, performance evaluation for Cloud computing platforms still remains an open issue. In this paper, we present an integrated performance evaluation middleware, which is aiming to provide users an easy-to-use toolkit to evaluate their Cloud system’s runtime performance. In addition, the proposed system introduces a novel resource working model by using queue theory, which provides us a quantitative technique to evaluate the dynamic performance of virtual resources at runtime. Meanwhile, it supports a real-time performance monitoring and profiling service for high-performance Cloud applications. Massive experiments are conducted to investigate the effectiveness of the proposed system, and the results indicate that its configurable feature is very useful when users are conducting performance comparing under different cases.

EScience Infrastructure for sharing neuroimaging data and running validated analysis pipelines on a high performance cloud

eScience Infrastructure for sharing neuroimaging data and running validated analysis pipelines on a high performance cloud

High performance Cloud data mining algorithm and Data mining in Clouds

We describe the design and implementation of a high performance cloud that we have used to archive, analyze and mine large distributed data sets. By a cloud, we mean an infrastructure that provides resources and/or services over the Internet. A storage cloud provides storage services, while a compute cloud provides compute services. High-performance can be reasonably intended as a intermediate step of high- performance data mining activities over large-scale amounts of data, while still keeping unaltered the primary and self-contained focus of achieving effectiveness and efficiency in these task themselves. In this paper we propose an algorithm to mine the data from the cloud using sector/sphere framework and association rules. We also describe the programming paradigm supported by the Sphere compute cloud and Association rules. Sector and Sphere are discussed for analyzing large data sets using computer clusters connected with wide area high performance networks. Data mining is the process of analyzing data from different perspectives and summarizing it into useful information. Mining association rules is one of the most important aspects in data mining.

THUNDER: helping underfunded NPO’s distribute electronic resources

As federal funding in many public non-profit organizations (NPO’s) seems to be dwindling, it is of the utmost importance that efforts are focused on reducing operating costs of needy organizations, such as public schools. Our approach for reducing organizational costs is through the combined benefits of a high performance cloud architecture and low-power, thin-client devices. However, general-purpose private cloud architectures are not easily deployable by average users, or even those with some computing knowledge. For this reason, we propose a new vertical cloud architecture, which is focused on ease of deployment and management, as well as providing organizations with cost-efficient virtualization and storage, and other organization-specific utilities. We postulate that if organizations are provided with on-demand access to electronic resources in a way that is cost-efficient, then the operating costs may be reduced, such that the user experience and organizational efficiency may be increased. In this paper we discuss our private vertical cloud architecture called THUNDER. Additionally, we introduce a number of methodologies that could enable needy non-profit organizations to decrease costs and also provide many additional benefits for the users. Specifically, this paper introduces our current implementation of THUNDER, details about the architecture, and the software system that we have designed to specifically target the needs of underfunded organizations.

Personalized cloud-based bioinformatics services for research and education: use cases and the elasticHPC package

BackgroundBioinformatics services have been traditionally provided in the form of a web-server that is hosted at institutional infrastructure and serves multiple users. This model, however, is not flexible enough to cope with the increasing number of users, increasing data size, and new requirements in terms of speed and availability of service. The advent of cloud computing suggests a new service model that provides an efficient solution to these problems, based on the concepts of "resources-on-demand" and "pay-as-you-go". However, cloud computing has not yet been introduced within bioinformatics servers due to the lack of usage scenarios and software layers that address the requirements of the bioinformatics domain.ResultsIn this paper, we provide different use case scenarios for providing cloud computing based services, considering both the technical and financial aspects of the cloud computing service model. These scenarios are for individual users seeking computational power as well as bioinformatics service providers aiming at provision of personalized bioinformatics services to their users. We also present elasticHPC, a software package and a library that facilitates the use of high performance cloud computing resources in general and the implementation of the suggested bioinformatics scenarios in particular. Concrete examples that demonstrate the suggested use case scenarios with whole bioinformatics servers and major sequence analysis tools like BLAST are presented. Experimental results with large datasets are also included to show the advantages of the cloud model.ConclusionsOur use case scenarios and the elasticHPC package are steps towards the provision of cloud based bioinformatics services, which would help in overcoming the data challenge of recent biological research. All resources related to elasticHPC and its web-interface are available at http://www.elasticHPC.org.

Evaluation of messaging middleware for high-performance cloud computing

Cloud computing is posing several challenges, such as security, fault tolerance, access interface singularity, and network constraints, both in terms of latency and bandwidth. In this scenario, the...

INTRODUCING THE EXTREMELY HETEROGENEOUS ARCHITECTURE

The computer industry is moving towards two extremes: extremely high-performance high-throughput cloud computing, and low-power mobile computing. Cloud computing, while providing high performance, is very costly. Google and Microsoft Bing spend billions of dollars each year to maintain their server farms, mainly due to the high power bills. On the other hand, mobile computing is under a very tight energy budget, but yet the end users demand ever increasing performance on these devices. This trend indicates that conventional architectures are not able to deliver high-performance and low power consumption at the same time, and we need a new architecture model to address the needs of both extremes. In this paper, we thus introduce our Extremely Heterogeneous Architecture (EHA) project: EHA is a novel architecture that incorporates both general-purpose and specialized cores on the same chip. The general-purpose cores take care of generic control and computation. On the other hand, the specialized cores, including GPU, hard accelerators (ASIC accelerators), and soft accelerators (FPGAs), are designed for accelerating frequently used or heavy weight applications. When acceleration is not needed, the specialized cores are turned off to reduce power consumption. We demonstrate that EHA is able to improve performance through acceleration, and at the same time reduce power consumption. Since EHA is a heterogeneous architecture, it is suitable for accelerating heterogeneous workloads on the same chip. For example, data centers and clouds provide many services, including media streaming, searching, indexing, scientific computations. The ultimate goal of the EHA project is two-fold: first, to design a chip that is able to run different cloud services on it, and through this design, we would be able to greatly reduce the cost, both recurring and non-recurring, of data centers\clouds; second, to design a light-weight EHA that runs on mobile devices, providing end users with improved experience even under tight battery budget constraints.

Special issue on Remote visualization

The visualization of huge datasets may be required by a broad spectrum of scientific applications encompassing, among others, the medicine, chemistry, architecture and aero-space fields. Unfortunately, in some cases, visualization devices might be unable to locally store such data. Datasets could be either too large (thus requiring to be stored in dedicated data centers) or might need specialized hardware to be processed (like high performance computers or cloud computing architectures). Some applications could even prevent, e.g., for privacy and/or security issues, a data transfer from data centers to (client) visualization devices. Under these circumstances, only representations of data, rather than data themselves, could be sent to the client side. Remote visualization has been often regarded as an interesting solution to the above problems, as data can be remotely processed where resources are available and only their representations are to be managed by the visualization devices. Although with the remote visualization paradigm it can be possible to minimize the resources required on the client side, it might also involve several issues basically related to the underlying network channel. In fact, network bandwidth, latency and interactive frame rates are the main parameters to be taken into account when data processing and data visualization are two logically and physically separated processes. A lot of work is known in the literature mainly related to the reduction of the network load from data centers to client devices. Compression scheme have been used to reduce the amount of data to be transmitted, both considering flows of