Virtual Cluster Research Articles

Deployment and Analysis of a Hybrid Shared/Distributed-Memory Parallel Visualization Tool for 3-D Oil Reservoir Grid on OpenStack Cloud Computing

The main goal of oil reservoir management is to provide more efficient, price-effective and environmentally more secure oil production. Oil production management includes an accurate characterization of the reservoir and strategies that involve interactions between reservoir data and human assessment. Hence, it is important to graphically visualize and handle massive data sets of oil and gas pressure / saturation levels to help decision makers in statistical analysis, history matching and recovery of hydrocarbons of the reservoir. In this article, we experimentally study the parallelization of intensive computation for a 3-D (three dimensional) oil reservoir data visualization tool. For this tool, we develop and implement a transformation and lighting model to visualize and react with the grid. Herein, we propose a hybrid (shared memory and distributed memory) parallelization technique to adapt with the data processing scalability. We tested these implementations on OpenStack Cloud Virtual Cluster. Our results indicate that although the virtual platform adds overhead for running parallel implementations, utilizing knowledge of the VM location on the compute host and network traffic among VMs to deploy the virtual environment can achieve significant performance enhancements. Hybrid parallel implementation using large data size can achieve $70\times $ speedup over serial execution without owning a costly HPC infrastructure as the conventional parallel processing deployment model.

Coworking As Innovative Educational Content in Modern Higher Education

The definition of “coworking” is just beginning to enter the orbit of Russian and foreign scientific research. Coworking spaces as innovative educational content in higher education are being created in Russian universities. The purpose of the study is the justification, development, and testing of a model of electronic coworking environment in the system of higher pedagogical education. At that, coworking environment is considered as an educational virtual cluster at which a basic platform for educational content is created with all educational products attached to it. The following methods of research were used to solve the tasks: design and modeling; the study of coworking environment products, chat conversations, questionnaires; and mathematical methods of result processing (quantitative and qualitative analysis of research results). The results of the study have revealed the advantages of coworking environment: successful socialization, minimization of time and economic resources, expansion of business partnerships, mutual assistance and a comfortable atmosphere. Analysis of studies on the criterion of coworking productivity revealed an increase in student mobility and an increase in the index satisfaction and communication effectiveness. Testing of the electronic coworking model proves its benefits and prospects of its introduction into the system of higher pedagogical education. The results can be used in social psychology, pedagogy, and practice.

Large-scale HPC deployment of Scalable CyberInfrastructure for Artificial Intelligence and Likelihood Free Inference (SCAILFIN)

The NSF-funded Scalable CyberInfrastructure for Artificial Intelligence and Likelihood Free Inference (SCAILFIN) project aims to develop and deploy artificial intelligence (AI) and likelihood-free inference (LFI) techniques and software using scalable cyberinfrastructure (CI) built on top of existing CI elements. Specifically, the project has extended the CERN-based REANA framework, a cloud-based data analysis platform deployed on top of Kubernetes clusters that was originally designed to enable analysis reusability and reproducibility. REANA is capable of orchestrating extremely complicated multi-step workflows, and uses Kubernetes clusters both for scheduling and distributing container-based workloads across a cluster of available machines, as well as instantiating and monitoring the concrete workloads themselves. This work describes the challenges and development efforts involved in extending REANA and the components that were developed in order to enable large scale deployment on High Performance Computing (HPC) resources. Using the Virtual Clusters for Community Computation (VC3) infrastructure as a starting point, we implemented REANA to work with a number of differing workload managers, including both high performance and high throughput, while simultaneously removing REANA’s dependence on Kubernetes support at the workers level.

Implementation of multi node Hadoop virtual cluster on open stack cloud environments

Implementation of multi node Hadoop virtual cluster on open stack cloud environments

The Method of Data Collection Based on Multiple Mobile Nodes for Wireless Sensor Network

In traditional wireless sensor networks (WSN), data is transmitted to base stations through multiple-hop. However, multi-hop transmission will cause problems such as energy holes, uneven energy consumption, and unreliable data transmission. Aiming at the defects of traditional networks, this paper proposes a scheme for collaborative data collection using multiple mobile nodes (MN) as sink nodes. In this scenario, an important issue is how to reasonably plan the data collection path of each mobile node under a series of constraints such as energy. This paper studies routing strategy and path planning. Firstly, a dynamic clustering algorithm is used to cluster the randomly arranged sensor nodes, and then a sensor node with higher energy is manually arranged at the virtual cluster center generated by the clustering algorithm as a cluster head node to establish a data collection cluster. Then the monitoring area is divided into several parts according to the number of MN, so that the MN traverses the cluster head nodes in the respective monitoring areas for data collection. At the same time, in order to enable the MN to complete data collection within the energy limit and improve the path balance, a path-based path equalization algorithm (PEABR) is proposed to adjust the path of the MN, further reduce and equalize the path length of the MN to satisfy the constraint conditions, and optimize the path planning scheme. Finally, simulation was carried out by using Matlab simulator, and the simulation experiments which were performed in a laboratory environment verified the feasibility and effectiveness of the algorithm.

OLAP parallel query processing in clouds with C‐ParGRES

SummaryThe advent of big data technologies has changed the way many companies manage their data. Several companies moved their data to the cloud using the concept of database‐as‐a‐service (DBaaS). Moving databases to the cloud presents several challenges related to flexible and scalable management of data. Although some of these companies migrated to NoSQL databases, most still rely on relational databases in the cloud to manage data, especially data that is critical to the decision making process. Online analytical processing (OLAP) queries take a long time to be processed, thus demanding high‐performance capabilities from their associated database systems to get results in a feasible time. In this article, we propose a middleware solution that can be deployed in any cloud provider, named C‐ParGRES, which explores database replication and interquery and intraquery parallelism to efficiently support OLAP queries in the cloud. C‐ParGRES is an extension of ParGRES, an open‐source database cluster middleware for high‐performance OLAP query processing in clusters. C‐ParGRES exploits cloud capabilities such as on‐demand resource provisioning and elasticity. In addition, C‐ParGRES can create multiple and independent virtual clusters for different database and users. We evaluate C‐ParGRES with two real‐world OLAP applications, both from the Brazilian Institute of Geography and Statistics. Results show that C‐ParGRES is a cost‐effective solution for OLAP query processing in the cloud.

Validation of Dynamic Risk Analysis Supporting Integrated Operations Across Systems

Dynamic risk analysis (DRA) is a novel industrial approach that aims to capture changes in operational conditions over time and quantify their effect on risk. This aspect may be advantageous for providing insight into the causal factors that have substantial risk contributions and supporting decisions related to risk control. Some DRA methods were developed by the oil and gas industry to support the integration of work processes and the cooperation across virtual clusters, e.g., between offshore and onshore systems and/or oil company and supplier. However, DRA has not been extensively adopted and limited attention is given to its validity in practical applications. The objective of this article is to illustrate how this validity can be established based on common validation approaches for risk analysis. The case study focuses on a DRA method named risk barometer that was developed to support integrated operations across the oil and gas industrial systems. The outcome of this study may serve as a basis for the validation of other DRA methods, the use of DRA in practical cases, and ultimately the achievement of integrated operations (IO) capabilities.

Discovery of a potent and selective adenylyl cyclase type 8 agonist by docking-based virtual screening

Adenylyl cyclases (ACs), which are responsible for catalyzing the conversion of adenosine triphosphate (ATP) into the second messenger cyclic adenosine monophosphate (cAMP), play a critical role in cell signal transduction. In this study, a combined approach involving docking-based virtual screening, with the combination of homology modeling followed by an in-vitro, and cell-based biological assay have been performed for discovering a class of novel potent and selective isoform adenylyl cyclase type 8 (AC8) agonist. The computer-aided virtual screening was used to identify fourteen virtual cluster compounds as potential hits which were further subjected to rigorous bioassays. A novel hit compound VHC-7 (ethyl 3-(2,4-dichlorobenzyl)-2-oxoindoline-3-carboxylate) was identified as a highly potent selective AC8 agonist with EC50 value of 0.1052 ± 0.038 µM. Remarkably, the molecule herein reported can be explored further to discover greater number of hit compounds with better pharmacokinetic properties as well as to serve as a promising novel hit agonist of AC8 for the treatment of various central nervous system disorders and its associated diseases.

Improving MapReduce scheduler for heterogeneous workloads in a heterogeneous environment

SummaryBig data is largely influencing business entities and research sectors to be more data‐driven. Hadoop MapReduce is one of the cost‐effective ways to process large scale datasets and offered as a service over the Internet. Even though cloud service providers promise an infinite amount of resources available on‐demand, it is inevitable that some of the hired virtual resources for MapReduce are left unutilized and makespan is limited due to various heterogeneities that exist while offering MapReduce as a service. As MapReduce v2 allows users to define the size of containers for the map and reduce tasks, jobs in a batch become heterogeneous and behave differently. Also, the different capacity of virtual machines in the MapReduce virtual cluster accommodate a varying number of map/reduce tasks. These factors highly affect resource utilization in the virtual cluster and the makespan for a batch of MapReduce jobs. Default MapReduce job schedulers do not consider these heterogeneities that exist in a cloud environment. Moreover, virtual machines in MapReduce virtual cluster process an equal number of blocks regardless of their capacity, which affects the makespan. Therefore, we devised a heuristic‐based MapReduce job scheduler that exploits virtual machine and MapReduce workload level heterogeneities to improve resource utilization and makespan. We proposed two methods to achieve this: (i) roulette wheel scheme based data block placement in heterogeneous virtual machines, and (ii) a constrained 2‐dimensional bin packing to place heterogeneous map/reduce tasks. We compared heuristic‐based MapReduce job scheduler against the classical fair scheduler in MapReduce v2. Experimental results showed that our proposed scheduler improved makespan and resource utilization by 45.6% and 47.9% over classical fair scheduler.

CLOSURE: A cloud scientific workflow scheduling algorithm based on attack–defense game model

The multi-tenant coexistence service mode makes the cloud-based scientific workflow encounter the risks of being intruded. For this problem, we propose a CLoud scientific wOrkflow SchedUling algoRithm based on attack–defensE game model (CLOSURE). In the algorithm, attacks based on different operating system vulnerabilities are regarded as different “attack” strategies; and different operating system distributions in a virtual machine cluster executing the workflows are regarded as different “defense” strategies. The information of the attacker and defender is not balanced. In other words, the defender cannot obtain the information about the attacker’s strategies, while the attacker can acquire information about the defender’s strategies through a network scan. Therefore, we propose to dynamically switch the defense strategies during the workflow execution, which can weaken the network scan effects and transform the workflow security problem into an attack–defense game problem. Then, the probability distribution of the optimal mixed defense strategies can be achieved by calculating the Nash Equilibrium in the attack–defense game model. Based on this probability, diverse VMs are provisioned for workflow execution. Furthermore, a task-VM mapping algorithm based on dynamic Heterogeneous Earliest Finish Time (HEFT) is presented to accelerate the defense strategy switching and improve workflow efficiency. The experiments are conducted on both simulation and actual environment, experimental results demonstrate that compared with other algorithms, the proposed algorithm can reduce the attacker’s benefits by around 15.23%, and decrease the time costs of the algorithm by around 7.86%.

An Architecture Model for a Distributed Virtualization System

The Thesis is about an architecture model for a Distributed Virtualization System, which could expand a virtual execution environment from a single physical machine to several nodes of a cluster. With current virtualization technologies, computing power and resource usage of Virtual Machines (or Containers) are limited to the physical machine where they run. To deliver high levels of performance and scalability, cloud applications are usually partitioned in several Virtual Machines (or Containers) located on different nodes of a virtualization cluster. Developers often use that processing model because the same instance of the operating system is not available on each node where their components run. The proposed architecture model is suitable for new trends in software development because it is inherently distributed. It combines and integrates Virtualization and Distributed Operating Systems technologies with the benefits of both worlds, providing the same isolated instance of a Virtual Operating System on each cluster node. Although it requires the introduction of changes in existing operating systems, thousands of legacy applications would not require modifications to obtain their benefits. A Distributed Virtualization System is suitable to deliver high-performance cloud services with provider-class features, such as high-availability, replication, migration, and load balancing. Furthermore, it is able to concurrently run several isolated instances of different guest Virtual Operating Systems, allocating a subset of nodes for each instance and sharing nodes between them. Currently, a prototype is running on a cluster of commodity hardware provided with two kinds of Virtual Operating Systems tailored for internet services (web server) as a proof of concept.

Real-time stochastic optimal scheduling of large-scale electric vehicles: A multidimensional approximate dynamic programming approach

This paper studies the real-time charging optimization (RTCO) of large-scale electric vehicles (EVs), which is a multistage and multidimensional stochastic resource allocation problem. In order to handle the complex RTCO of large-scale EVs, a multidimensional approximate dynamic programming (ADP-RTCO) is designed for the sequent optimal decision makings. The hierarchy of ADP-RTCO contains two layers. In the upper layer, RTCO of large-scale EVs is formulated as a multidimensional energy storage problem by classifying EVs into several virtual EV clusters (EVCs). Then, temporal difference learning based policy iteration method is used to obtain the value function approximation for each EVC. In the lower layer, priority based reallocation algorithm is employed to obtain the detailed charging power for connected EVs. ADP-RTCO is designed with on-line learning ability to enhance its adaptability and robustness to ambiguous environment. Comprehensive simulation results demonstrate the optimality and robustness of ADP-RTCO in both cost-saving and load-flattening problems. Furthermore, ADP-RTCO can obtain higher-quality solution compared with other algorithms and is applicable for RTCO of large-scale EVs under uncertainties due to its high computation efficiency.

Service Characteristics-Oriented Joint ACB, Cell Selection, and Resource Allocation Scheme for Heterogeneous M2M Communication Networks

Heterogeneous machine-to-machine communication networks (HMCNs) are expected to provide ubiquitous connectivity for machine type communication devices (MTCDs) with reduced human intervention. Although various types of services will be supported for MTCDs, diverse quality of service requirements and network characteristics pose challenges and difficulties to the resource allocation, cell selection, and random access schemes of the HMCNs. In this paper, we propose a joint access class barring (ACB), cell selection, and resource allocation algorithm for HMCNs. To achieve efficient utilization of the resource blocks (RBs) in heterogeneous cells, we first propose a virtual clustering-based joint ACB and RB allocation sub-algorithm. Partitioning the MTCDs into virtual clusters, we formulate the joint ACB and RB allocation problem of the virtual clusters as RB utilization maximization problem. Through solving the optimization problem, we obtain the optimal ACB factor and RB allocation strategy. Given the optimal strategy, we then formulate the joint cell selection and power allocation problem of the MTCDs as a utility function maximization problem. As the formulated problem is a mixed nonlinear optimization problem, which cannot be solved conveniently, we transform the problem using relaxation method and solve it through applying an iterative algorithm and the Lagrange dual method. Simulation results demonstrate the effectiveness of the proposed algorithm.

Ismael: Using Machine Learning to Predict Acceptance of Virtual Clusters in Data Centers

Existing virtual network admission control algorithms targeting high utilization of data center infrastructure are computationally expensive or provide poor performance. In particular, existing algorithms have in common that they are oblivious to the past, i.e., requests are handled in a fire-and-forget manner, not taking into account information from previously solved instances. This can be inefficient and misses out on a basic optimization opportunity: as for any network optimization algorithm that faces repeating problem instances, it may be beneficial to learn from network states and the outcome of acceptance decisions of the past. In this paper, we propose Ismael, a machine learning framework for predicting the acceptance of virtual clusters, one of the most common virtual network abstractions in data centers. Ismael can be configured with, and learn from, different existing algorithms by combining fixed-size feature representations for graphs with a convolutional neural network or a fully connected deep neural network. We report on extensive simulations, which demonstrate that it is possible to mimic existing, computationally intensive admission control algorithms with an accuracy of up to 94 %, while significantly reducing runtime.

Dynamic ranking-based MapReduce job scheduler to exploit heterogeneous performance in a virtualized environment

“More data, more information.” Big data helps businesses and research communities to gain insights and increase productivity. Many public cloud service providers offer Hadoop MapReduce as a service based on pay-per-use via infrastructure as a service on clusters of virtual machines promising on-demand horizontal scaling. These clusters of virtual machines are launched in various physical machines across racks in cloud data centers. Such multi-tenancy negatively introduces performance heterogeneity for Hadoop virtual machines due to hardware heterogeneity and interference from co-located virtual machine. Performance heterogeneity largely affects MapReduce job latency and resource utilization of rented Hadoop virtual clusters. Default MapReduce schedulers assign map/reduce tasks assuming the hardware is homogeneous. Interference-aware schedulers perform by only observing the interference pattern generated by co-located virtual machines. These schedulers do not consider the heterogeneous performance of virtual machines. Therefore, we propose a dynamic ranking-based MapReduce job scheduler that places the map and reduces tasks based on a virtual machine’s performance rank to minimize job latency and improve resource utilization. Our proposed approach calculates the performance score for each virtual machine based on hardware heterogeneity and co-located virtual machine interference. Then, it ranks the virtual machines based on the map and reduce performance separately to place map and reduce tasks. To demonstrate our ideas, we have set a test bed with 29 virtual machines on eight physical machines with different configurations and capacities. We modify a default fair scheduler in Hadoop 2.x to incorporate our ideas and evaluate them with different workloads on the PUMA dataset. The proposed method is then compared against a default fair scheduler (resource-aware) and an interference-aware scheduler based on job latency and resource utilization. Finally, we argue in favor of our approach as it improves resource utilization by 30–65% and overall job latency by up to 30%.

Dynamic virtual cluster cloud security using hybrid steganographic image authentication algorithm

ABSTRACT Storing data in a third party cloud system causes serious problems on data confidentiality. Generally, encryption techniques provide data confidentiality but with limited functionality, which occurs due to unsupported actions of encryption operation in cloud storage space. Hence, developing a decentralized secure storage system with multiple support functions like encryption, encoding, and forwarding tends to get complicated, when the storage system spreads. This paper aims mainly on hiding image information using specialized steganographic image authentication (SSIA) algorithm in clustered cloud systems. The SSIA algorithm is applied to virtual elastic clusters in a public cloud platform. Here, the SSIA algorithm embeds the image information using blowfish algorithm and genetic operators. Initially, the blowfish symmetric block encryption is applied over the image and then the genetic operator is applied to re-encrypt the image information. The proposed algorithm provides an improved security than conventional blowfish algorithm in a clustered cloud system.

ADMC-MAC: Energy efficient adaptive MAC protocol for mission critical applications in WSN

Energy efficient wireless sensor networks play a very important role in mission critical applications, as in these applications it is difficult to deploy and change the nodes frequently or to change their battery. Some of the mission critical applications are healthcare applications, environmental monitoring and control applications (Tsunami alert, earthquake alert, volcanic eruption prone areas) and surveillance applications. These applications require quick response time along with power management, so that the energy efficiency can be achieved without having latency beyond a critical threshold limit. Several MAC protocols have been proposed by researchers with a preview to enhance energy efficiency but hey failed to fulfill the need of mission critical applications. An in-depth survey analysis has helped in exploring several MAC protocols for their compatibility towards mission critical scenarios. In this paper we propose a novel energy efficient contention based ADMC-MAC protocol. This protocol is more adaptable to mission critical applications by improving the data delivery performance based on traffic conditions considering the queue size of the node and also have improved energy efficient performance. This protocol overcomes the lacunas of standard MAC protocols. Two algorithms have been proposed in this protocol. The first algorithm, which is priority based, select the node as cluster head with high number of packets in queue and having maximum energy among the neighboring nodes forming virtual cluster. The second algorithm uses regression technique for deciding the duty cycle of selected node based on traffic conditions and residual energy of the node. Only the selected nodes take part in transmission of packets. So the packets overloading at intermediate nodes is reduced. This helps in increasing the overall efficiency of the system by enhancing energy saving up to 71.452% as compare to S-MAC protocol in mission critical scenarios. The packet delivery ratio is enhanced by 15.198 percent in mission critical scenarios along with considerable improvement in throughput. S-MAC is found to be the most appropriate existing MAC protocol which can be enhanced for mission critical scenarios. So the proposed protocol is enhanced S-MAC protocol implemented in NS-2.35 and the results are compared with standard S-MAC protocol and latest MC-MAC mission critical MAC protocol. It is found that the proposed protocol ADMC-MAC outperforms the latest MC-MAC protocol in terms of energy saving and also provides efficient data delivery performance.

Multipole expansion for magnetic structures: A generation scheme for a symmetry-adapted orthonormal basis set in the crystallographic point group

We propose a systematic method to generate a complete orthonormal basis set of multipole expansion for magnetic structures in arbitrary crystal structure. The key idea is the introduction of a virtual atomic cluster of a target crystal, on which we can clearly define the magnetic configurations corresponding to symmetry-adapted multipole moments. The magnetic configurations are then mapped onto the crystal so as to preserve the magnetic point group of the multipole moments, leading to the magnetic structures classified according to the irreducible representations of crystallographic point group. We apply the present scheme to pyrhochlore and hexagonal ABO3 crystal structures, and demonstrate that the multipole expansion is useful to investigate the macroscopic responses of antiferromagnets.

Queue-based and learning-based dynamic resources allocation for virtual streaming media server cluster of multi-version VoD system

Nowadays, video-on-demand (VoD) providers offer multiple-quality video streaming services to users, called as multi-version VoD. Unlike traditional VoD, multi-version VoD providers should consider to allocate bandwidth resource and transcoding computation resource simultaneously. However, most of existing resource allocation works only focused on cost reduction or bandwidth optimization, and they did not consider to allocate transcoding computation resources for multi-version VoD systems. Therefore, how to allocate bandwidth resource and transcoding computation resource simultaneously for multi-version VoD systems is still one major challenge. In this paper, we propose a queue-based and learning-based dynamic resources allocation strategy (QLRA) for virtual streaming media server cluster of multi-version VoD system. First, we analyze the user behavior habits and build the virtual streaming media server cluster as an M/G/n queue system. Based on queueing theory, we can allocate initial resources for virtual streaming media server cluster of multi-version VoD system. Second, taking the changes of the user arrival rate and the workload of multi-version VoD system as feedbacks, we introduce learning automaton to allocate resources dynamically for virtual streaming media server cluster. Third, we evaluate QLRA with other methods, and results show the correctness and effectiveness of our strategy.

High-revenue Online Provisioning for Virtual Clusters in Multi-tenant Cloud Data Center Network

The rapid development of cloud computing and high requirements of operators requires strong support from the underlying Data Center Networks. Therefore, the effectiveness of using resources in the data center networks becomes a point of concern for operators and material for research. In this paper, we discuss the online virtual-cluster provision problem for multiple tenants with an aim to decide when and where the virtual cluster should be placed in a data center network. Our objective is maximizing the total revenue for the data center networks under the constraints. In order to solve this problem, this paper divides it into two parts: online multi-tenancy scheduling and virtual cluster placement. The first part aims to determine the scheduling orders for the multiple tenants, and the second part aims to determine the locations of virtual machines. We first approach the problem by using the variational inequality model and discuss the existence of the optimal solution. After that, we prove that provisioning virtual clusters for a multi-tenant data center network that maximizes revenue is NP-hard. Due to the complexity of this problem, an efficient heuristic algorithm OMS (Online Multi-tenancy Scheduling) is proposed to solve the online multi-tenancy scheduling problem. We further explore the virtual cluster placement problem based on the OMS and propose a novel algorithm during the virtual machine placement. We evaluate our algorithms through a series of simulations, and the simulations results demonstrate that OMS can significantly increase the efficiency and total revenue for the data centers.