Amazon Elastic Compute Cloud Research Articles

Cloud Colonography: Distributed Medical Testbed over Cloud

Cloud Colonography is proposed in this paper, using different types of cloud computing environments. Databases from the Computed Tomographic Colonography (CTC) screening tests among several hospitals are explored. These networked databases are going to be available in the near future via cloud computing technologies. Associated Multiple Databases (AMD) was developed in this study to handle multiple CTC databases. When AMD is used for assembling databases, it can achieve very high classification accuracy. The proposed AMD has the potential to play a role of the core classifier in the cloud computing framework. AMD for multiple institutions databases yields high detection performance of polyps using Kernel principal component analysis (KPCA). Two cases in the proposed cloud platform are private and public. We adapted a university cluster as a private platform, and Amazon Elastic Compute Cloud (EC2) as a public. The computation time, memory usage, and running costs were compared using three representative databases between private and public cloud environments. The proposed parallel processing modules improved the computation time, especially in the public cloud environments. The successful development of a cloud computing environment that handles large amounts of data will make Cloud Colonography feasible for a new health care service.

A proactive defense method for the stealthy EDoS attacks in a cloud environment

SummaryCloud computing technology provides flexibility to Cloud Service Provider (CSP) for providing the cloud resources based on the users' requirements. In on‐demand pricing model, the attackers exploit this feature and cause unwanted scaling‐up of the cloud resources without any intent to pay for them. The associated cost for the unpaid malicious usage burdens the CSP, and over a long period, economic losses occur at the CSP end. Thus, the resources and services offered by the CSP become unsustainable, and the attack is termed as Economic Denial‐of‐Sustainability (EDoS) attack. The existing defense approaches for EDoS attacks are reactive. Thus, the associated attack detection/mitigation cost is high; consequently, the approaches are not suitable for the Small and Medium Enterprises (SMEs). The aim of this paper is to detect and mitigate, internal and external, stealthy EDoS attacks proactively. The attack is detected using average CPU utilization threshold and utility function (in terms of cost for the utilized cloud computing resources) and mitigated using virtual firewalls. Amazon Elastic Compute Cloud (Amazon EC2) is used to evaluate the performance of the proposed approach. The proposed approach accurately detects the EDoS attack and mitigates its effect as well with reduced cost. It is observed that the approach provides competitive response time, victim service downtime, and attack reporting time. Thus, the overall performance is improved.

Smart Home Automation using ESP8266 and Internet of Things

By means of using the latest technologies such as cloud computing, Internet of Things, embedded systems etc. this paper proposes a working model for smart home automation. The advancement in field of information technology has led to numerous practical applications which can help the humans to lead a better life where regular chores can be handled via wireless communications instead of practical effort. The proposed model is projected as one of the cheapest ways available to automate a home and transform it into a smart home using ESP8266 microcontroller. The model provides a mechanism to remotely monitor and control the home devices by using a user interface which will be interlinked to Amazon Elastic Compute Cloud (EC2). The paper provides a background of the enabling technologies along with the proposed model. The paper concludes by providing the result of implementation and its challenges.

Amazon Elastic Compute Cloud (EC2) versus In-House HPC Platform: A Cost Analysis

While High Performance Computing (HPC) centers continuously evolve to provide more computing power to their users, we observe a wish for the convergence between Cloud Computing (CC) and High Performance Computing (HPC) platforms, with the commercial hope to see Cloud Computing (CC) infrastructures to eventually replace in-house facilities. If we exclude the performance point of view where many previous studies highlight a non-negligible overhead induced by the virtualization layer at the heart of every Cloud middleware when running a HPC workload, the question of the real cost-effectiveness is often left aside with the intuition that, most probably, the instances offered by the Cloud providers are competitive from a cost point of view. In this article, we wanted to assert (or infirm) this intuition by analyzing what composes the Total Cost of Ownership (TCO) of an in-house HPC facility operated internally since 2007. This Total Cost of Ownership (TCO) model is then used to compare with the induced cost that would have been required to run the same platform (and the same workload) over a competitive Cloud IaaS offer. Our approach to address this price comparison is three-fold. First we propose a theoretical price-performance model based on the study of the actual Cloud instances proposed by one of the major Cloud IaaS actors: Amazon Elastic Compute Cloud (EC2). Then, based on the HPC facility TCO analysis we propose a hourly price comparison between our in-house cluster and the equivalent EC2 instances. Finally, based on the experimental benchmarking on the local cluster and on the Cloud instances we propose an update of the former theoretical price model to reflect the real system performance. The results obtained advocate in general for the acquisition of an in-house HPC facility, which balances the common intuition in favor of Cloud Computing platforms, would they be provided by the reference Cloud provider worldwide.

SecFact: Secure Large-scale QR and LU Factorizations

We are now in the big data era. Due to the emerging various systems and applications, such as the Internet of Things, cyber-physical systems, smart cities, smart healthcare, we are able to collect more data than ever before. On the other hand, it makes it very difficult to analyze such massive data in order to advance our science and engineering fields. We note that QR and LU factorizations are two of the most fundamental mathematical tools for data analysis. However, conducting QR or LU factorization of an $m\times n$ m × n matrix requires computational complexity of $\mathcal{O}(m^2n)$ O ( m 2 n ) . This incurs a formidable challenge in efficiently analyzing large-scale data sets by normal users or small companies on traditional resource-limited computers. To overcome this limitation, industry and academia propose to employ cloud computing that can offer abundant computing resources. This, however, obviously raises security concerns and hence a lot of users are reluctant to reveal their data to the cloud. To this end, we propose two secure outsourcing algorithms for efficiently performing large-scale QR and LU factorizations, respectively. We implement the proposed algorithms on the Amazon Elastic Compute Cloud (EC2) platform and a laptop. The experiment results show significant time saving for the user.

A One-Penny Imputed Genome from Next-Generation Reference Panels

Genotype imputation is commonly performed in genome-wide association studies because it greatly increases the number of markers that can be tested for association with a trait. In general, one should perform genotype imputation using the largest reference panel that is available because the number of accurately imputed variants increases with reference panel size. However, one impediment to using larger reference panels is the increased computational cost of imputation. We present a new genotype imputation method, Beagle 5.0, which greatly reduces the computational cost of imputation from large reference panels. We compare Beagle 5.0 with Beagle 4.1, Impute4, Minimac3, and Minimac4 using 1000 Genomes Project data, Haplotype Reference Consortium data, and simulated data for 10k, 100k, 1M, and 10M reference samples. All methods produce nearly identical accuracy, but Beagle 5.0 has the lowest computation time and the best scaling of computation time with increasing reference panel size. For 10k, 100k, 1M, and 10M reference samples and 1,000 phased target samples, Beagle 5.0's computation time is 3× (10k), 12× (100k), 43× (1M), and 533× (10M) faster than the fastest alternative method. Cost data from the Amazon Elastic Compute Cloud show that Beagle 5.0 can perform genome-wide imputation from 10M reference samples into 1,000 phased target samples at a cost of less than one US cent per sample.

A Local Approximation Approach for Processing Time-Evolving Graphs

To efficiently process time-evolving graphs where new vertices and edges are inserted over time, an incremental computing model, which processes the newly-constructed graph based on the results of the computation on the outdated graph, is widely adopted in distributed time-evolving graph computing systems. In this paper, we first experimentally study how the results of the graph computation on the local graph structure can approximate the results of the graph computation on the complete graph structure in distributed environments. Then, we develop an optimization approach to reduce the response time in bulk synchronous parallel (BSP)-based incremental computing systems by processing time-evolving graphs on the local graph structure instead of on the complete graph structure. We have evaluated our optimization approach using the graph algorithms single-source shortest path (SSSP) and PageRankon the Amazon Elastic Compute Cloud(EC2), a central part of Amazon.com’s cloud-computing platform, with different scales of graph datasets. The experimental results demonstrate that the local approximation approach can reduce the response time for the SSSP algorithm by 22% and reduce the response time for the PageRank algorithm by 7% on average compared to the existing incremental computing framework of GraphTau.

An MCDM method for cloud service selection using a Markov chain and the best-worst method

Due to the increasing number of cloud services, service selection has become a challenging decision for many organisations. It is even more complicated when cloud users change their preferences based on the requirements and the level of satisfaction of the experienced service. The purpose of this paper is to overcome this drawback and develop a cloud broker architecture for cloud service selection by finding a pattern of the changing priorities of User Preferences (UPs). To do that, a Markov chain is employed to find the pattern. The pattern is then connected to the Quality of Service (QoS) for the available services. A recently proposed Multi Criteria Decision Making (MCDM) method, Best Worst Method (BWM), is used to rank the services. We show that the method outperforms the Analytic Hierarchy Process (AHP). The proposed methodology provides a prioritized list of the services based on the pattern of changing UPs. The methodology is validated through a case study using real QoS performance data of Amazon Elastic Compute (Amazon EC2) cloud services.

Uncovering the Security Implications of Cloud Multi-Tenancy with Bolt

Cloud providers routinely schedule multiple applications per physical host to increase efficiency. The resulting interference on shared resources often leads to performance degradation and, more importantly, security vulnerabilities. Interference can leak important information about an application, ranging from a services placement to confidential data, such as private keys. We present Bolt, a practical system that accurately detects the type and characteristics of applications sharing a cloud platform based on the interference an adversary sees on shared resources. Bolt leverages online data mining techniques that only require 2-5 seconds for detection. In a multi-user study on Amazon Elastic Compute Cloud (EC2), Bolt correctly identifies the characteristics of 385 out of a set of 436 diverse workloads. Extracting this information enables a wide spectrum of previously impractical cloud attacks, including denial of service (DoS) attacks that increase tail latency by 140X, as well as resource freeing attacks (RFAs), and co-residency attacks. Finally, we show that, while advanced isolation mechanisms such as cache partitioning lower detection accuracy, they are insufficient to eliminate these vulnerabilities altogether. To do so, one must either disallow core sharing or allow it only between threads of the same application, leading to significant inefficiencies and performance penalties.

Reducing and Analyzing the PHAT Survey with the Cloud

Abstract We discuss the technical challenges we faced and the techniques we used to overcome them when reducing the Panchromatic Hubble Andromeda Treasury (PHAT) photometric data set on the Amazon Elastic Compute Cloud (EC2). We first describe the architecture of our photometry pipeline, which we found particularly efficient for reducing the data in multiple ways for different purposes. We then describe the features of EC2 that make this architecture both efficient to use and challenging to implement. We describe the techniques we adopted to process our data, and suggest ways these techniques may be improved for those interested in trying such reductions in the future. Finally, we summarize the output photometry data products, which are now hosted publicly in two places in two formats. They are in simple fits tables in the high-level science products on MAST, and on a queryable database available through the NOAO Data Lab.

Applying big data paradigms to a large scale scientific workflow: Lessons learned and future directions

The increasing amounts of data related to the execution of scientific workflows has raised awareness of their shift towards parallel data-intensive problems. In this paper, we deliver our experience combining the traditional high-performance computing and grid-based approaches with Big Data analytics paradigms, in the context of scientific ensemble workflows. Our goal was to assess and discuss the suitability of such data-oriented mechanisms for production-ready workflows, especially in terms of scalability. We focused on two key elements in the Big Data ecosystem: the data-centric programming model, and the underlying infrastructure that integrates storage and computation in each node. We experimented with a representative MPI-based iterative workflow from the hydrology domain, EnKF-HGS, which we re-implemented using the Spark data analysis framework. We conducted experiments on a local cluster, a private cloud running OpenNebula, and the Amazon Elastic Compute Cloud (AmazonEC2). The results we obtained were analysed to synthesize the lessons we learned from this experience, while discussing promising directions for further research.

Probabilistic Optimization of Resource Distribution and Encryption for Data Storage in the Cloud

In this paper, we develop a decentralized probabilistic method for performance optimization of cloud services. We focus on Infrastructure-as-a-Service where the user is provided with the ability of configuring virtual resources on demand in order to satisfy specific computational requirements. This novel approach is strongly supported by a theoretical framework based on tail probabilities and sample complexity analysis. It allows not only the inclusion of performance metrics for the cloud but the incorporation of security metrics based on cryptographic algorithms for data storage. To the best of the authors’ knowledge this is the first unified approach to provision performance and security on demand subject to the Service Level Agreement between the client and the cloud service provider. The quality of the service is guaranteed given certain values of accuracy and confidence. We present some experimental results using the Amazon Web Services, Amazon Elastic Compute Cloud service to validate our probabilistic optimization method.

A framework for big data analytics approach to failure prediction of construction firms

This study explored use of big data analytics (BDA) to analyse data of a large number of construction firms to develop a construction business failure prediction model (CB-FPM). Careful analysis of literature revealed financial ratios as the best form of variable for this problem. Because of MapReduce’s unsuitability for iteration problems involved in developing CB-FPMs, various BDA initiatives for iteration problems were identified. A BDA framework for developing CB-FPM was proposed. It was validated by using 150,000 datacells of 30,000 construction firms, artificial neural network, AmazonElastic ComputeCloud, Apache Spark and the R software. The BDA CB-FPM was developed in eight seconds while the same process without BDA was aborted after nine hours without success. This shows the issue of not wanting to use large dataset to develop CB-FPM due to tedious duration is resolvable by applying BDA technique. The BDA CB-FPM largely outperformed an ordinary CB-FPM developed with a dataset of 200 construction firms, proving that use of larger sample size with the aid of BDA, leads to better performing CB-FPMs. The high financial and social cost associated with misclassifications (i.e. model error) thus makes adoption of BDA CB-FPMs very important for, among others, financiers, clients and policy makers.

Efficient Secure Outsourcing of Large-Scale Sparse Linear Systems of Equations

Solving large-scale sparse linear systems of equations (SLSEs) is one of the most common and fundamental problems in big data, but it is very challenging for resource-limited users. Cloud computing has been proposed as a timely, efficient, and cost-effective way of solving such expensive computing tasks. Nevertheless, one critical concern in cloud computing is data privacy. Specifically, clients’ SLSEs usually contain private information that should remain hidden from the cloud for ethical, legal, or security reasons. Many previous works on secure outsourcing of linear systems of equations (LSEs) have high computational complexity, and do not exploit the sparsity in the LSEs. More importantly, they share a common serious problem, i.e., a huge number of memory I/O operations. This problem has been largely neglected in the past, but in fact is of particular importance and may eventually render those outsourcing schemes impractical. In this paper, we develop an efficient and practical secure outsourcing algorithm for solving large-scale SLSEs, which has low computational and memory I/O complexities and can protect clients’ privacy well. We implement our algorithm on Amazon Elastic Compute Cloud, and find that the proposed algorithm offers significant time savings for the client (up to 74 percent) compared to previous algorithms.

Cost-driven provisioning and execution of a computing-intensive service on the Amazon EC2

The decision of migrating a service to a cloud-based system must take into consideration many different aspects. Among them, economical costs is one of the most important. This paper describes how a computing-intensive service, based on a bag-of-tasks approach, has been migrated from a grid infrastructure to the Amazon Elastic Compute Cloud (EC2) infrastructure. A cost-based model for the evaluation of the economic costs of providing the service on the cloud has also been proposed, which considers computing costs as well as storage and platform deployment costs. The model includes a wide range of different instance types and purchasing policies provided by the Amazon EC2, as well as the deadline and problem size provided by the service user. The paper also shows how the proposed cost-based model is integrated into the framework used for the service deployment and execution, making possible the interaction with the Amazon Web services for hiring the required cloud resources and efficiently use them for the execution of the service requests.

Parallel Secure Outsourcing of Large-scale Nonlinearly Constrained Nonlinear Programming Problems

Nonlinearly constrained nonlinear programming (NLC-NLP) problems arise in various optimal decision-making fields, such as financial engineering, urban planning, supply chain management, and power system control. These real-world NLC-NLP problems are usually large-scale because they need to take into account massive variables and constraints. Solving NLC-NLP problems with common algorithms, e.g., the gradient projection method (GPM), generally incurs very high time complexity. This results in the inevitable inability of solving large-scale NLC-NLP problems by using general-purpose computers within a feasible time. To address this challenge, a cost-effective solution is to adopt cloud computing, but this raises security concerns since real-world NLC-NLP problems always carry sensitive information. Previous secure outsourcing algorithms try to protect sensitive information, but still let cloud service tenants bear heavy computation workload. To fill the gap, we develop a practical secure outsourcing algorithm for solving large-scale NLC-NLP problems with the GPM. To be more prominent, we also parallelize the secure outsourcing algorithm for accelerating computations and avoiding possible memory overflowing. We implement the proposed algorithm on the Amazon Elastic Compute Cloud (EC2) and a laptop, and notice that it can significantly reduce the tenant's computing time even for large-scale NLC-NLP problems.

The Great Automatic Grammatizator: writing, labour, computers

What does it mean when we say that computers can ‘write’ and how are recent developments in neural networks and machine learning changing this capacity? This article examines the long-standing literary fear of authorship being replaced by machines while also interrogating the labour and credit implications that sit behind widely used structures of authorship in a technological age. The argument makes reference to one work of computer-generated writing – Johannes Helden & Hakan Jonson’s Evolution [2014] – and to one software paradigm (a character-based recurrent neural networks for language acquisition trained on the corpus of the journal Textual Practice). I here argue that unless we conceive more broadly of the criteria for ‘authorship’ as a labour function, and unless we take seriously the need to see textual production as social production, hybridized (but predominantly) machine identities will come to dominate a literary landscape.

Cloud-based Geophysical Inversion Modeling Using GNU Octave and MatlabMPI on Amazon EC2

Computer modeling and simulation can be very demanding in terms of computational resources. Cloud computing has opened up new avenues for the scientific researchers with limited resources to do complicated simulation. In order to investigate whether cloud computing is suitable for modeling and simulation, we first describe how to build a cloud-based modeling and simulation platform using GNU Octave and MatlabMPI on the Amazon Elastic Compute Cloud (EC2). Then, we evaluation its performance taking the geophysical inversion modeling as an example. The results show that the cloud-based modeling and simulation platform is suitable for basic modeling for free. It can provide much more higher performance with acceptable price. Furthermore, we can cut down the cost by employing the Spot instance without losing the computation performance.

GATE Monte Carlo simulation of dose distribution using MapReduce in a cloud computing environment

The GATE Monte Carlo simulation platform has good application prospects of treatment planning and quality assurance. However, accurate dose calculation using GATE is time consuming. The purpose of this study is to implement a novel cloud computing method for accurate GATE Monte Carlo simulation of dose distribution using MapReduce. An Amazon Machine Image installed with Hadoop and GATE is created to set up Hadoop clusters on Amazon Elastic Compute Cloud (EC2). Macros, the input files for GATE, are split into a number of self-contained sub-macros. Through Hadoop Streaming, the sub-macros are executed by GATE in Map tasks and the sub-results are aggregated into final outputs in Reduce tasks. As an evaluation, GATE simulations were performed in a cubical water phantom for X-ray photons of 6 and 18MeV. The parallel simulation on the cloud computing platform is as accurate as the single-threaded simulation on a local server and the simulation correctness is not affected by the failure of some worker nodes. The cloud-based simulation time is approximately inversely proportional to the number of worker nodes. For the simulation of 10million photons on a cluster with 64 worker nodes, time decreases of 41× and 32× were achieved compared to the single worker node case and the single-threaded case, respectively. The test of Hadoop's fault tolerance showed that the simulation correctness was not affected by the failure of some worker nodes. The results verify that the proposed method provides a feasible cloud computing solution for GATE.

Build and Evaluate a Free Virtual Cluster on Amazon Elastic Compute Cloud for Scientific Computing

Scientific computing requires a huge amount of computing resources, but not all the scientific researchers have an access to sufficient high-end computing systems. Currently, Amazon provides a free tier account for cloud computing which could be used to build a virtual cluster. In order to investigate whether it is suitable for scientific computing, we first describe how to build a free virtual cluster using StarCluster on Amazon Elastic Compute Cloud (EC2). Then, we perform a comprehensive performance evaluation of the virtual cluster built before. The results show that a free virtual cluster is easily built on Amazon EC2 and it is suitable for the basic scientific computing. It is especially valuable for scientific researchers, who do not have any HPC or cluster, to develop and test their prototype system of scientific computing without paying anything, and move it to a higher performance virtual cluster when necessary by choosing more powerful instance on Amazon EC2.