OpenStack-based Private Cloud Research Articles

GenoVault: a cloud based genomics repository

GenoVault is a cloud-based repository for handling Next Generation Sequencing (NGS) data. It is developed using OpenStack-based private cloud with various services like keystone for authentication, cinder for block storage, neutron for networking and nova for managing compute instances for the Cloud. GenoVault uses object-based storage, which enables data to be stored as objects instead of files or blocks for faster retrieval from different distributed object nodes. Along with a web-based interface, a JavaFX-based desktop client has also been developed to meet the requirements of large file uploads that are usually seen in NGS datasets. Users can store files in their respective object-based storage areas and the metadata provided by the user during file uploads is used for querying the database. GenoVault repository is designed taking into account future needs and hence can scale both vertically and horizontally using OpenStack-based cloud features. Users have an option to make the data shareable to the public or restrict the access as private. Data security is ensured as every container is a separate entity in object-based storage architecture which is also supported by Secure File Transfer Protocol (SFTP) for data upload and download. The data is uploaded by the user in individual containers that include raw read files (fastq), processed alignment files (bam, sam, bed) and the output of variation detection (vcf). GenoVault architecture allows verification of the data in terms of integrity and authentication before making it available to collaborators as per the user’s permissions. GenoVault is useful for maintaining the organization-wide NGS data generated in various labs which is not yet published and submitted to public repositories like NCBI. GenoVault also provides support to share NGS data among the collaborating institutions. GenoVault can thus manage vast volumes of NGS data on any OpenStack-based private cloud.

Uniform genomic data analysis in the NCI Genomic Data Commons

The goal of the National Cancer Institute’s (NCI’s) Genomic Data Commons (GDC) is to provide the cancer research community with a data repository of uniformly processed genomic and associated clinical data that enables data sharing and collaborative analysis in the support of precision medicine. The initial GDC dataset include genomic, epigenomic, proteomic, clinical and other data from the NCI TCGA and TARGET programs. Data production for the GDC started in June, 2015 using an OpenStack-based private cloud. By June of 2016, the GDC had analyzed more than 50,000 raw sequencing data inputs, as well as multiple other data types. Using the latest human genome reference build GRCh38, the GDC generated a variety of data types from aligned reads to somatic mutations, gene expression, miRNA expression, DNA methylation status, and copy number variation. In this paper, we describe the pipelines and workflows used to process and harmonize the data in the GDC. The generated data, as well as the original input files from TCGA and TARGET, are available for download and exploratory analysis at the GDC Data Portal and Legacy Archive (https://gdc.cancer.gov/).

Detection of DDoS Attacks in OpenStack-based Private Cloud Using Apache Spark

Security is a critical concern for cloud service providers. Distributed denial of service (DDoS) attacks are the most frequent of all cloud security threats, and the consequences of damage caused by DDoS are very serious. Thus, the design of an efficient DDoS detection system plays an important role in monitoring suspicious activity in the cloud. Real-time detection mechanisms operating in cloud environments and relying on machine learning algorithms and distributed processing are an important research issue. In this work, we propose a real-time detection of DDoS attacks using machine learning classifiers on a distributed processing platform. We evaluate the DDoS detection mechanism in an OpenStack-based cloud testbed using the Apache Spark framework. We compare the classification performance using benchmark and real-time cloud datasets. Results of the experiments reveal that the random forest method offers better classifier accuracy. Furthermore, we demonstrate the effectiveness of the proposed distributed approach in terms of training and detection time.

Energy-efficient and Dynamic Consolidation of Virtual Machines in OpenStack-based Private Cloud

Abstract Dynamic Virtual Machines (VMs) consolidation is an efficient technique for enhancing resource utilization and reducing energy consumption by the physical servers. Over-utilized host affects the degradation of resource utilization and VM performance. Determining when it is best to migrate VMs from an over-utilized host to an under-utilized host is the aim of dynamic Live VM Migration (LVM). It influences the resource utilization and Quality of Services (QoS) offered by cloud providers. In the OpenStack cloud, LVM is done manually, and it uses first-fit as a default migration technique, which is not efficient for resource utilization. The paper proposes a novel approach for the dynamic consolidation of VMs in the OpenStack cloud. The CPU utilization, RAM utilization, and the number of instances of each host are monitored over a period of time, and overload detection of a host is carried out using the SVM classification model. The Load balancing consolidation is performed based on the SVM classification result. Further, an energy-efficient consolidation achieved to optimize the energy of physical servers. We carry out the performance analysis of the proposed work in the OpenStack-based real-time testbed of five-node. The result reveals the fair distribution of load within the servers and the significant benefits of energy-efficient VM consolidation resulting up to 15% energy savings.

Distributed Denial of Service (DDoS) Attacks Detection System for OpenStack-based Private Cloud

Cloud computing technique is commonly used by fast growing Internet-based applications. Moving to cloud computing results in reducing the cost of managing and maintaining IT infrastructure. Different organizations controls the cloud resources over internet using networking protocols and standards. This makes IT infrastructure distributed in nature but controlled centrally, which opens the door to the attackers for intrusions. Attacks as Distributed Denial of Service (DDoS) is one of the most popular intrusions in private cloud which cause a degradation of services being reduced or denial of services. In this work the sole focus is on DDoS attack which specifically targeted towards detection of bandwidth flooding and connection flooding. Such attacks target the network layer of the cloud set up with invalid requests and make it denial to legitimate requests. Thus, the entire cloud set up becomes vulnerable and can be disrupted by these DDoS attacks. To overcome this a cloud operating system which has an integrated firewall with DDoS detection system is required. Here a system with OpenStack integrated firewall and raw socket programming for monitoring the network traffic is proposed. Based on the dataset generated in controlled DDoS attack environment, algorithms such as Decision tree, K nearest neighbor (KNN), Naive Bayes and Deep Neural Network (DNN) algorithms are compared against the trained model. Eventually, DDoS attacks are detected, and administrator of the private cloud is notified.

An overview of the DII-HEP OpenStack based CMS data analysis

An OpenStack based private cloud with the Cluster File System has been built and used with both CMS analysis and Monte Carlo simulation jobs in the Datacenter Indirection Infrastructure for Secure High Energy Physics (DII-HEP) project. On the cloud we run the ARC middleware that allows running CMS applications without changes on the job submission side. Our test results indicate that the adopted approach provides a scalable and resilient solution for managing resources without compromising on performance and high availability.To manage the virtual machines (VM) dynamically in an elastic fasion, we are testing the EMI authorization service (Argus) and the Execution Environment Service (Argus-EES). An OpenStackplugin has been developed for Argus-EES.The Host Identity Protocol (HIP) has been designed for mobile networks and it provides a secure method for IP multihoming. HIP separates the end-point identifier and locator role for IP address which increases the network availability for the applications. Our solution leverages HIP for traffic management.This presentation gives an update on the status of the work and our lessons learned in creating an OpenStackbased cloud for HEP.